Data processing methods, devices, and chip products applied to distributed training systems

Abstract

This application provides a data processing method, apparatus, and chip product for distributed training systems, relating to the field of distributed training technology. The method includes: in a distributed training system where a distributed training task is executed in parallel by t groups of computing nodes, and the checkpoint data corresponding to each group of nodes is divided into t shards, with each shard stored on a computing node within that group; for a first computing node in at least one group of nodes, determining at least one computing node from the distributed training system as at least one target computing node corresponding to the first computing node; and copying the shard data stored on the first computing node to the at least one target computing node for backup storage. This application, by utilizing computing nodes in the distributed training system to back up shard data, improves the fault tolerance and training state recovery efficiency of the distributed training system.

Description

Technical Field

[0001] This application relates to the field of distributed training technology, and in particular to a data processing method, apparatus and chip product applied to a distributed training system. Background Technology

[0002] Distributed training is a method for training machine learning models that distributes the training process across multiple computing nodes in a distributed system to achieve parallel training and improve the training efficiency of machine learning models.

[0003] Distributed training relies on the collaborative work of multiple computing nodes, but it is susceptible to interruptions due to various hardware and software issues affecting these nodes. To address this problem, relevant technologies employ the method of periodically checking the training status of each computing node. Specifically, for each computing node, the system periodically stores the data it processes (such as model parameters and parameter gradients) in its own memory and writes the data processed by each node to a persistent storage system.

[0004] Taking the use of DP (Data Parallel) technology as an example, the aforementioned multiple computing nodes are divided into multiple computer node groups. Multiple computer node groups are used to train machine learning models in parallel. For example, each group of computer node groups uses the same model parameters in the machine learning model, and for any group of computer node groups, the memory of each computer node stores the same complete model parameters (i.e., checkpoints).

[0005] When using FSDP (Fully Shared Data Parallel) technology, although each group of peer nodes is still used to train the same model parameters in the machine learning model, for any group of peer nodes, each peer node stores different shard data in its memory. The shard data stored by each peer node can be combined to form the checkpoint corresponding to a single peer node in the DP technology.

[0006] During fault recovery, since reading checkpoints from memory is faster than reading them from persistent storage, each compute node prioritizes reading checkpoints from its own memory to quickly resume distributed training. However, for FSDP technology, if a compute node fails and cannot provide sharded data, because different sharded data are stored in the memory of the same group of peer nodes, the technology cannot recover complete checkpoints from the compute node's memory. Instead, it needs to recover complete checkpoints from persistent storage, which significantly reduces the efficiency of checkpoint recovery and is detrimental to improving the training fault tolerance of the distributed training system. Summary of the Invention

[0007] This application provides a data processing method, apparatus, and chip product for use in a distributed training system. The technical solutions provided in this application include the following:

[0008] According to one aspect of the embodiments of this application, a data processing method for a distributed training system is provided. The distributed training system includes t computing node groups, which are used to execute distributed training tasks in parallel. All computing nodes in the t computing node groups are divided into at least one equi-node group, each equi-node group including t computing nodes from different computing node groups. The checkpoint data corresponding to each equi-node group is divided into t data shards, which are non-overlapping. Each of the t computing nodes in the equi-node group stores one of the t data shards. The checkpoint data is used to indicate the state of the t computing nodes in the equi-node group when participating in the execution of the distributed training task, where t is an integer greater than 1. The method includes:

[0009] For the first computing node in the at least one group of equal nodes, at least one computing node different from the first computing node is determined from the distributed training system as at least one target computing node corresponding to the first computing node.

[0010] The sharded data stored on the first computing node is copied to at least one target computing node corresponding to the first computing node for backup storage.

[0011] According to one aspect of the embodiments of this application, a data processing apparatus for a distributed training system is provided. The distributed training system includes t computing node groups, which are used to execute distributed training tasks in parallel. All computing nodes in the t computing node groups are divided into at least one equi-node group, each equi-node group including t computing nodes from different computing node groups. The checkpoint data corresponding to each equi-node group is divided into t fragments, which are non-repeating. Each of the t computing nodes in the equi-node group stores one fragment of the t fragments. The checkpoint data is used to indicate the state of the t computing nodes in the equi-node group when participating in the execution of the distributed training task, where t is an integer greater than 1. The apparatus includes:

[0012] The target node determination module is used to determine, for the first computing node in the at least one equipotential node group, at least one computing node different from the first computing node in the distributed training system, as at least one target computing node corresponding to the first computing node.

[0013] The sharded data backup module is used to copy the sharded data stored on the first computing node to at least one target computing node corresponding to the first computing node for backup storage.

[0014] According to one aspect of the embodiments of this application, a chip product is provided, wherein a computer program is stored in the chip product, the computer program being loaded and executed by a processor to implement the above-described data processing method applied to a distributed training system.

[0015] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described data processing method applied to a distributed training system.

[0016] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the above-described data processing method applied to a distributed training system.

[0017] According to one aspect of the embodiments of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the data processing method described above for a distributed training system.

[0018] The technical solutions provided in the embodiments of this application may include the following beneficial effects.

[0019] In a distributed training system, if the checkpoint data corresponding to a given node group is divided into t data shards, and each shard is stored in one of the nodes in that node group, then the shard data stored in one node can be copied to other nodes for backup. This ensures that there are nodes in the distributed training system capable of recovering the shard data corresponding to that node, improving the system's fault tolerance. Furthermore, it eliminates the need to recover the shard data from persistent storage, as retrieving data from a node is faster than from persistent storage, thus improving the efficiency of shard data retrieval. This, in turn, improves the efficiency of training state recovery and ultimately, the system's fault recovery efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a distributed training system provided in one possible implementation of this application;

[0022] Figure 2 This is a schematic diagram illustrating data parallelism provided in one possible implementation of this application;

[0023] Figure 3 This is a schematic diagram of pipeline parallelism provided in one possible implementation of this application;

[0024] Figure 4 This is a schematic diagram of tensor parallelism provided in one possible implementation of this application;

[0025] Figure 5 This is a schematic diagram of hybrid parallelism provided in one possible implementation of this application;

[0026] Figure 6 This is a schematic diagram illustrating training fault tolerance provided in one possible implementation of this application;

[0027] Figure 7 This is a schematic diagram of a method for generating checkpoint data provided in one possible implementation of this application;

[0028] Figure 8 This is a schematic diagram illustrating the use of checkpoint data for training fault tolerance in one possible implementation of this application;

[0029] Figure 9 This is a distributed schematic diagram of checkpoint data provided in one possible implementation of this application;

[0030] Figure 10 This is a schematic diagram of the distribution of fragmented data provided in one possible implementation of this application;

[0031] Figure 11 This is a schematic diagram of a method for reading checkpoint data in a fault replacement scenario provided in one possible implementation of this application;

[0032] Figure 12 This is a flowchart of a data processing method for a distributed training system provided in one possible implementation of this application;

[0033] Figure 13 This is a schematic diagram of fragmented data backup storage provided in one possible implementation of this application;

[0034] Figure 14 This is a schematic diagram of a method for reading fragmented data in a fault replacement scenario provided in one possible implementation of this application;

[0035] Figure 15 This is a schematic diagram of data re-partitioning provided in one possible implementation of this application;

[0036] Figure 16 This is a flowchart of a data processing method for a distributed training system provided in another possible implementation of this application;

[0037] Figure 17 This is a block diagram of a data processing apparatus for a distributed training system provided in one possible implementation of this application;

[0038] Figure 18 This is a block diagram of a data processing apparatus for a distributed training system provided in another possible implementation of this application;

[0039] Figure 19 This is a simplified structural block diagram of a computer device provided in one possible implementation of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0041] Please refer to Figure 1 This illustrates a schematic diagram of a distributed training system provided in one possible implementation of this application.

[0042] The distributed training system 100 can refer to a distributed system for implementing distributed training of machine learning models. A machine learning model is an algorithm that automatically learns and generates prediction results based on given input data (also called features). For example, a machine learning model can be a neural network model built upon a neural network. Distributed training is a method for training machine learning models by distributing the training process across multiple computing nodes in a distributed system to achieve parallel training and improve the training efficiency of the machine learning model. A distributed system is a system composed of a group of computing nodes that communicate through a network and coordinate their work to complete a common task. In the embodiments of this application, implementing distributed training of a machine learning model can be referred to as a distributed training task.

[0043] In one example, the distributed training system 100 may include multiple computing nodes 101. Each computing node 101 can be used to process data independently to provide computing resources. Multiple computing nodes 101 can run in parallel. Multiple computing nodes 101 are used to execute distributed training tasks, such as multiple computing nodes 101 can cooperate to complete the execution of distributed training tasks.

[0044] Computing node 101 may be implemented as a server. Exemplarily, computing node 101 may be implemented as at least one of the following: a server equipped with a Central Processing Unit (CPU), a GPU server equipped with a Graphics Processing Unit (GPU), a server equipped with a General-Purpose Computing on Graphics Processing Units (GPGPU), a server equipped with a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a server equipped with a Tensor Processing Unit (TPU), a Field Programmable Gate Array (FPGA), a server equipped with Tensor Cores, or a server equipped with a Neural Processing Unit (NPU).

[0045] In one example, the distributed training system 100 may further include at least one backup node 102. Backup node 102 may refer to a backup computing node 101. Optionally, computing node 101 participates in executing the distributed training task, while backup node 102 does not participate in executing the distributed training task. For example, the computing node that does not execute the task can be designated as backup node 102, and the computing node that participates in executing the task can be designated as computing node 101. Backup node 102 can be used to replace computing node 101 and continue executing the task corresponding to computing node 101 in the event of a failure of computing node 101. The capabilities and implementation of backup node 102 are the same as those of computing node 101, so that backup node 102 can directly replace computing node 101.

[0046] Optionally, multiple computing nodes 101 can communicate with each other, and computing nodes 101 and backup nodes 102 can also communicate with each other. For example, a communication link is established between each computing node 101, enabling communication between them; a communication link is established between each backup node 102 and each computing node 101, enabling communication between them. The communication link can be implemented through a network, such as a wired network or a wireless network.

[0047] For example, the network architecture of the distributed training system 100 is a leaf-spine network architecture, which includes leaf switches 103 and spine switches 104. Leaf switches 103 can be implemented as Layer 2 switches for connecting compute nodes 101 or backup nodes 102. Spine switches 104 can be implemented as Layer 3 switches responsible for routing and forwarding, and are connected to leaf switches 103. Some compute nodes 101 can form communication links only through leaf switches 103, while some compute nodes 101 can form communication links simultaneously through leaf switches 103 and spine switches 104. For example, compute nodes H0 and H1 can form a communication link only through leaf switch 0, while compute nodes H0 and H4 can form a communication link through leaf switch 0, spine switch 104, and leaf switch N. This application embodiment does not limit the value of N, which can be set and adjusted according to actual usage requirements. This application embodiment also does not limit the number of switch layers; for example, the distributed training system 100 may also include switches with 3 or more layers.

[0048] Optionally, compute nodes 101 and standby nodes 102 connected to the same leaf switch 103 belong to the same network domain. Two compute nodes 101 connected to different leaf switches 103 belong to different network domains; that is, compute nodes 101 within the same network domain are connected to the same leaf switch 103. In this implementation, a network domain can refer to a set of compute nodes with a security boundary. Trust relationships are established between compute nodes within the same network domain, so that accessing other compute nodes within the domain no longer requires permission from the accessed compute node. For example, compute nodes H0, H1, H2, and H3, and standby nodes B1 and B2, can form network domain 0 through leaf switch 0.

[0049] In one example, the distributed training system 100 also includes a master computing node ( Figure 1 (Not shown in the diagram), denoted as Training Master, is responsible for coordinating and scheduling computing nodes 101 and distributed training tasks, enabling multiple computing nodes 101 to collaboratively execute distributed training tasks. The master computing node ensures that all computing nodes 101 work collaboratively by centrally managing and allocating tasks, thereby optimizing training efficiency. Optionally, computing nodes 101 and standby nodes 102 can be collectively referred to as workers.

[0050] The master computing node can be implemented as at least one of the following: a server equipped with a CPU, a GPU server, a server equipped with a GPGPU, a server equipped with a DSP, an ASIC, a server equipped with a TPU, an FPGA, a server equipped with a Tensor Core, or a server equipped with an NPU. For example, a computing node 101 can be randomly selected from the distributed training system 100 as the master computing node, or a separate master computing node can be set up outside the distributed training system 100; this embodiment does not limit this approach.

[0051] Optionally, the master compute node can also be responsible for the monitoring and control of distributed training tasks, the partitioning of sample data for machine learning models, the status monitoring and control of compute node 101, network configuration management, and fault log analysis. In a feasible example, the master compute node can be implemented as a functional module and deployed on compute node 101.

[0052] Optionally, each computing node 101 may be equipped with multiple computing devices, including at least one of the following: CPU, GPU, GPGPU, DSP, ASIC, TPU, FPGA, Tensor Core, and NPU. For example, if computing node 101 is implemented as a GPU server, multiple GPUs may be configured on computing node 101, and each GPU may be implemented as a GPU card (i.e., a chip), which is the computing device. Each computing node 101 may run multiple sub-processes to utilize the multiple computing devices on computing node 101 to accelerate computing tasks.

[0053] In one example, distributed training of a machine learning model is an iterative process, with each iteration corresponding to a stage of distributed training tasks. The distributed training system 100 completes the iterative training of the machine learning model by sequentially executing the distributed training tasks of each stage. Distributed training typically requires multiple parallel techniques to improve training efficiency; based on the different degrees of parallelism of these techniques, the distributed training task can be divided into subtasks with varying degrees of parallelism.

[0054] For example, the parallel techniques used in distributed training may include at least one of the following:

[0055] Data Parallelism (DP): This involves dividing multiple computing nodes 101 into multiple computing node groups, and splitting the sample data used to train the machine learning model into multiple data subsets. Different data subsets are assigned to different computing node groups, and each computing node group has the same complete model parameters. Each computing node group consists of the same number of computing nodes, and each computing node group includes at least one computing node. For example, refer to... Figure 2 The sample data 201 is divided into two subsets, each processed by a separate group of computing nodes to achieve distributed training of the machine learning model. Each subtask can then be: to fully train the machine learning model using the subset of data.

[0056] Pipeline Parallel (PP): This involves partitioning a machine learning model across layers, distributing the data (such as model parameters) of some model layers across different computational nodes, and distributing the data of other model layers across different computational nodes. The model layers are the foundational data structures of the machine learning model, and different model layers can perform different data processing operations, such as convolution, pooling, normalization, linear transformation, and logistic regression. For example, see [reference needed]. Figure 3The model parameters of some model layers of the machine learning model are distributed to computing node 302, and the model parameters of the remaining model layers are distributed to computing node 303. Computing nodes 302 and 303 complete the distributed training of the machine learning model based on sample data 301. At this time, each subtask can be: training a portion of the model layers of the machine learning model using sample data.

[0057] Tensor Parallel (TP): This involves partitioning a machine learning model within its layers, that is, splitting the data (such as model parameters) of the same model layer across different computation nodes. For example, refer to... Figure 4 For each model layer of the machine learning model, the model parameters of the model layer are evenly distributed across computing nodes 402 and 403. Computing nodes 402 and 403 are simultaneously trained based on sample data 401. Here, PP and TP belong to model parallelism (ModelParallel). Each subtask can then be: training a subset of the model parameters of the model layer of the machine learning model using the sample data.

[0058] Hybrid Parallel (3D Parallel): Simultaneous application of DP, PP, and TP. For example, see reference... Figure 5 Taking DP=N, PP=4, and TP=8 as an example, the sample data is split into N data subsets according to the parallelism of DP. Each data subset is used to train a complete machine learning model. That is, the multiple computing nodes 101 in the distributed training system 100 are divided into N computing node groups, and each computing node group is used to process a data subset. For any complete machine learning model, it is split according to the parallelism of PP, so that different model layers are assigned to different computing nodes 101. For example, taking computing node group 501 as an example, the model parameters of the four model layers of the machine learning model can be assigned to the four computing nodes 502 in computing node group 501. For each model layer, it can be further split according to the parallelism of TP, so that different model parameters are assigned to different computing devices (such as GPUs) on the same computing node 101. For example, taking computing node 502 as an example, the model parameters of model layer 1 can be divided into 8 parts, and each part is assigned to a computing device 503. The values ​​in the above embodiments are merely exemplary and illustrative. This application does not limit the number of computing node groups, the number of computing nodes in the computing node groups, or the number of computing devices used on the computing nodes.

[0059] In the distributed training of machine learning models, sample data is sharded and distributed across different computing node groups for training. After each computing node group completes a training cycle, they need to exchange training state data (such as model parameters, parameter gradients, etc.) and update the overall model parameters of the machine learning model to complete the entire distributed training. For example, before the next training cycle begins, the training state of each computing node group needs to be synchronized to ensure consistency, thus achieving distributed training of the machine learning model. For instance, regarding model parameters, before the next training cycle begins, the updated overall model parameters of the machine learning model are synchronized to each computing node group to ensure consistency, thereby achieving parallel training of the machine learning model parameters.

[0060] Optionally, during distributed training, each computing device (such as a GPU) has a unique identifier, often referred to as a rank. The identifier of the computing device on each computing node is called the local rank, while each computing device in the entire distributed training system has a global identifier, often referred to as the global rank.

[0061] In one example, in the event of an interruption in the execution of a distributed training task, a training fault tolerance scheme can be employed to resume the task. This scheme primarily addresses how to promptly detect problems and take appropriate measures to recover from failures or network latency in a distributed training system, ensuring its continued operation.

[0062] For example, the aforementioned primary computing node can also be used to achieve training fault tolerance for distributed training by coordinating and scheduling the backup node 102. Training fault tolerance can refer to the process of handling faulty nodes that cause the execution of distributed training tasks to be interrupted, in order to restore the execution of distributed training tasks. A faulty node refers to a computing node that cannot continue to participate in the execution of distributed training tasks; that is, after a computing node fails, it can be adjusted to be a faulty node.

[0063] Optionally, the technical solutions for training fault tolerance include at least one of the following:

[0064] Failure Node Replacement: This involves isolating failed nodes in the distributed training system and replacing them with standby nodes to resume the execution of distributed training tasks. When a standby node is reassigned to participate in the distributed training task, it is updated as the new computing node.

[0065] Resilient fault tolerance: This involves isolating faulty nodes in a distributed training system and allowing the distributed training task to continue execution after scaling down or scaling up. Scaling down refers to reducing the number of computing nodes participating in the distributed training task, while scaling up refers to adding computing nodes using backup nodes. Scaling down and scaling up are performed at the computing node group level. For example, each computing node group deploys a complete machine learning model (i.e., model parameters) capable of independently processing sample data; therefore, scaling down and scaling up can be performed according to computing node groups without affecting the complete training of the machine learning model in other computing node groups.

[0066] For example, refer to Figure 6 The distributed training system 600 includes computing nodes H0 to H3, and a backup node B1. Computing nodes H0 and H1 form computing node group 1, and computing nodes H2 and H3 form computing node group 2. In the event of a failure (i.e., becoming a faulty node) in computing node H3, the training master selects backup node B1 to replace computing node H3 and isolates computing node H3. Backup node B1 and computing node H2 recover from computing node group 2 to continue executing the subtasks corresponding to computing node group 2.

[0067] After restoring compute node group 2 using backup node B1 and compute node H2, if compute node H2 also fails, since there are no backup nodes in the distributed training system 600, the training master will control compute node group 2 to cancel the execution of the corresponding subtasks for training scaling down. This means isolating compute node H2 and stopping the use of backup node B1 (which has now been converted into a new compute node) to resume the execution of the distributed training task. Specifically, compute node H2 is updated to be a backup node, and backup node B1 is also restored to a backup node.

[0068] Optionally, if two new spare nodes appear in the distributed training system 600 after training scaling down, the two new spare nodes can be combined into a computing node group for training scaling up to compensate for the impact of the lack of computing node group 2 on the execution efficiency of the distributed training task.

[0069] In one example, checkpointing is an important mechanism for training fault tolerance. Checkpointing is used to periodically save the complete training state of a machine learning model. A checkpoint can be an "archive" of all training states of the machine learning model at a certain point in time, or it can be the training state of each computing node at a certain point in time (such as a memory checkpoint described below). This embodiment of the application does not limit this approach. Checkpointing allows the distributed training system to resume distributed training from the most recent checkpoint, rather than starting from scratch, should a problem be encountered or the distributed training needs to be interrupted during distributed training.

[0070] In this embodiment, checkpoints can also be referred to as checkpoint data, which can be used to indicate the state of computing nodes participating in the execution of distributed training tasks. Checkpoint data contains all intermediate training states of the machine learning model. For example, checkpoint data includes at least one of the following: model parameters, parameter gradients, number of iterations, sample data numbers, and optimizer states (such as learning rate, model data backup, first-order momentum, second-order momentum, etc.).

[0071] Optionally, the checkpoint usage process can be as follows: During the execution of the distributed training task, the distributed training system periodically checks the training state. If the execution of the distributed training task is interrupted, the distributed training task is resumed based on the most recent checkpoint.

[0072] For example, for each computing node, the control computing node stores its corresponding training state in its own memory (such as host memory). The training state corresponding to each computing node includes the training state corresponding to each computing device on that computing node. Then, the training state stored in the memory of each computing node is asynchronously written to the persistent storage system. The asynchronous persistence process does not affect the execution of the distributed training task, thereby improving the execution efficiency of the distributed training task.

[0073] The "archive" of the training state of a computing node at a certain point in time can be referred to as a memory checkpoint. In this embodiment, checkpoint data can refer to memory checkpoints. Optionally, the computing node's memory may only store the latest checkpoint data.

[0074] Persistent storage systems are used to persistently store data. For example, a persistent storage system can be at least one of the following: a distributed file system, a distributed database, or an object storage service, such as a storage disk, NAS (Network Attached Storage), CPFS (Cloud Parallel File System), etc.

[0075] For example, refer to Figure 7 Taking GPU servers as an example, the distributed training system includes N computing nodes 701, each computing node 701 equipped with at least one GPU card 7011. Based on the size of the training state corresponding to the GPU card 7011 (e.g., represented as a Tensor), a contiguous shared memory segment (video memory, i.e., GPU memory) is allocated on the host memory 7012 of the computing node 701. Figure 7 The process involves several steps: First, the shared memory buffer (shared memory buffer). Second, the tensors in the GPU card 7011's memory (i.e., GPU Memory) are copied directly to the shared memory in byte form, forming memory checkpoints. Each GPU card 7011 executes this synchronously. Finally, the processes on each compute node 701 asynchronously write their respective memory checkpoints from the shared memory to the persistent storage system 702. This process does not interfere with the execution of the distributed training task. Here, `synccopy` refers to synchronous copying, `PCIe` refers to the bus, and `async persist` refers to asynchronous persistence.

[0076] Optionally, refer to Figure 8 The distributed training system 800 includes four computing nodes: computing node 1 to computing node 4. The fault tolerance principle based on memory checkpoints is as follows: for each stage (step) of the distributed training task, after executing the corresponding subtask, the four computing nodes write their respective training states from GPU memory to their respective host memory. For example, if the execution of the Nth stage of the distributed training task is interrupted, the four computing nodes can read the checkpoint data of the (N-1)th stage stored in their respective host memory into their respective GPU memory, and then restart the distributed training task from the Nth stage.

[0077] The checkpoint data for stage N-1 can be denoted as state(N-1), which represents the memory checkpoint corresponding to the execution of the distributed training task in stage N-1 by the computing node. Optionally, "execution of the distributed training task in stage N-1" can mean that each computing node has completed the execution of the distributed training task in stage N-1, and the state data of each computing node group has been synchronized. For example, the checkpoint data for stage N-1 of a computing node records the state data of that computing node when the distributed training task in stage N-1 has been completed and the state data of each computing node group has been synchronized. That is, for each stage of the distributed training task, the checkpoint data of each computing node group is substantially the same, and for each computing node, the checkpoint data of the computing node is substantially the same as the checkpoint data of its peer nodes. This is the data redundancy of the data parallelism strategy. The position of a computing node in its computing node group is the same as the position of its peer nodes in the same computing node group.

[0078] In one example, building upon dynamic programming (DP), the designers proposed FSDP (Fully Shared Data Parallel). The core idea of ​​FSDP is to fully shard the weights and gradients (i.e., checkpoint data) of the machine learning model and distribute them across multiple computing nodes (such as GPUs) to reduce the memory footprint of individual nodes. During the forward propagation phase, each computing node computes the gradients of its local weight shard. Then, the gradients across all computing nodes are synchronized and averaged to ensure that updates to each weight shard are based on the entire mini-batch of sample data. After gradient synchronization, each computing node updates its local weight shard. This ensures consistent updates to the overall machine learning model even if each computing node only holds a portion of the model.

[0079] In a distributed training system, each group of computing nodes processes the same model parameters. However, when storing checkpoint data, instead of storing a complete copy of the checkpoint data for each node in the same group, the complete checkpoint data is divided into multiple fragments, with each node storing one fragment. When the checkpoint data is needed, the fragments are aggregated (e.g., by obtaining parameters through all-gather communication) for subsequent processing, such as the forward and backward propagation phases in distributed training.

[0080] For example, compared to DP technology, although the groups of equidistant nodes in FSDP technology are still used to train the same model parameters in the machine learning model, for any group of equidistant nodes, each equidistant node stores different shard data in its memory. The shard data stored by each equidistant node can be combined to form the checkpoint data corresponding to a single equidistant node in DP technology.

[0081] In other words, for any set of peer nodes, the FSDP technology divides the checkpoint data of a single peer computing node in the DP technology into multiple data shards, and each peer node only stores one of these multiple data shards, instead of storing the complete checkpoint data, thereby reducing the redundancy of checkpoint data and thus reducing the storage pressure on computing nodes.

[0082] For example, refer to Figure 9 and Figure 10 This paper explains the FSDP technology by comparing it with the DP technology.

[0083] like Figure 9 As shown, when using the DP (Dynamic Programming) technique, the distributed training system 900 includes t groups of computing nodes (i.e., groups of computing nodes divided based on the DP technique described above). These t groups of computing nodes are used to execute distributed training tasks in parallel, where t is the degree of parallelism of the distributed training system 900, and the t-th group of computing nodes can be denoted as DP(t). Each group of computing nodes includes at least one computing node, and the t groups can include all computing nodes in the distributed training system 900. For example, if the distributed training system 900 includes a total of n computing nodes, and these n nodes only include those participating in the distributed training task and exclude spare nodes, then these n computing nodes are evenly divided into t groups of computing nodes, meaning each group includes the same number of computing nodes. Each group of computing nodes is used to train the complete parameters of the machine learning model using a portion of the training data. For any given group of computing nodes, each computing node in the group can be used to process a portion of the parameters of the machine learning model, such as each computing node processing the parameters of one or more model layers in the learning model.

[0084] All computing nodes in a group of t computing nodes can be divided into at least one equilateral node group, and each equilateral node group includes t computing nodes, with each t computing node coming from a different computing node group. For example, if the t computing node groups contain a total of n computing nodes, then these n computing nodes are completely divided into at least one equilateral node group. Optionally, the t computing nodes are arranged in the same position within their respective computing node groups. Alternatively, the t computing nodes each process the same subtask. Alternatively, the t computing nodes each store the same checkpoint data. t is an integer greater than 1.

[0085] For example, the first computing node in each computing node group can be combined into an equilateral node group. Each first computing node stores the same complete checkpoint data, and the checkpoint data stored in any first computing node can completely restore the training state of the first computing node.

[0086] Figure 9 Each box in the table represents a compute node. A compute node with a full black fill indicates that it stores complete checkpoint data. The compute nodes (i.e., equivalence node groups) in each column store the same checkpoint data.

[0087] like Figure 10 As shown, when using FSDP technology, the distributed training system 900 still includes t groups of computing nodes, which are used to execute distributed training tasks in parallel. The computing nodes in the t groups can also be divided into at least one equilateral node group. However, for any equilateral node group within the at least one equilateral node group, the checkpoint data corresponding to that equilateral node group is divided into t data shards. These t data shards are unique, meaning there is no duplicate data among the t data shards. Each of the t computing nodes in the equilateral node group stores one shard from the t data shards.

[0088] The checkpoint data corresponding to the equinode group is used to indicate the state of the t compute nodes in the equinode group when participating in the distributed training task, similar to the checkpoint data of a single compute node. The t compute nodes in the equinode group have the same state when participating in the distributed training task. The shard data corresponding to the equinode group can be used to indicate a partial state of the t compute nodes in the equinode group when participating in the distributed training task; that is, the shard data stored by a compute node can only recover a partial training state of that compute node, without needing to recover the complete training state.

[0089] Optionally, the aforementioned t data segments can be obtained by equally dividing the checkpoint data corresponding to the equipotential node group according to the parallelism t of the distributed training system 900.

[0090] For any computing node in the equidistant node group, it is necessary to aggregate t data fragments to that computing node in order to recover the state it was in when participating in distributed training based on the t data fragments. Since each of the t computing nodes in the equidistant node group stores only one fragment of the t data fragments, there are no computing nodes in the distributed training system that store the same data fragments, which means there is zero redundancy.

[0091] For example, the first compute node in each compute node group can be combined into an equipotential node group. The checkpoint data corresponding to this equipotential node group is divided into t data fragments, and each first compute node stores one fragment of data from the t fragments, and the fragments stored by each first compute node are different (i.e., different and incomplete checkpoint data).

[0092] Figure 10 Each box in the table represents a compute node, and the black fill in the compute node indicates the fragmented data stored in that compute node. It can be seen that the fragmented data stored in the compute nodes (i.e., equidistant node groups) in each column can be combined to form a complete checkpoint dataset.

[0093] Optionally, in a fault recovery scenario, the distributed training system prioritizes reading checkpoint data from the memory of the computing nodes, and secondly reads checkpoint data from the persistent storage system to resume the execution of the distributed training task.

[0094] For distributed training systems employing dynamic programming (DP) technology, in the event of a computing node failure, checkpoint data can be recovered from the memory of its peer nodes. For example, refer to... Figure 9 If computing node 901 in computing node group DP(1) fails and cannot be recovered, computing node 904 in computing node group DP(2) is the equivalent node of computing node 901, and the checkpoint data of computing node 901 can be recovered from computing node 904. If computing node 902 in computing node group DP(2) fails and cannot be recovered, computing node 905 in computing node group DP(1) is the equivalent node of computing node 902, and the checkpoint data of computing node 902 can be recovered from computing node 905. If computing node 903 in computing node group DP(t) fails and cannot be recovered, computing node 906 in computing node group DP(2) is the equivalent node of computing node 903, and the checkpoint data of computing node 903 can be recovered from computing node 906.

[0095] For example, refer to Figure 11The distributed training system 1100 includes four computing nodes and one standby node. If computing node 4 fails and cannot be recovered (referred to as the failed node), the distributed training system 1100 interrupts the execution of the Nth stage training task. Each computing node's host memory stores its own checkpoint data state(N-1) for the Nth stage. The primary computing node replaces computing node 4 with standby node 1. If the host memory of computing node 4 is not corrupted or state(N-1) has not been cleared, and state(N-1) can still be provided, the primary computing node can directly copy state(N-1) from computing node 4 to standby node 1 to resume distributed training.

[0096] For distributed training systems using FSDP technology, when a computing node fails and cannot provide sharded data, since different sharded data are stored in the same group of peer nodes, and there are no computing nodes storing the same sharded data in the distributed training system, it is impossible to recover the complete checkpoint data from the memory of the computing node. Instead, the complete checkpoint data needs to be recovered from the persistent storage system. This greatly reduces the efficiency of checkpoint data recovery and is not conducive to improving the training fault tolerance capability of the distributed training system.

[0097] For example, refer to Figure 10 If computing node 901 in computing node group DP(1) fails and cannot be recovered, although computing node 904 in computing node group DP(2) is an equivalent node of computing node 901, the sharded data stored by computing node 904 is different from the sharded data stored by computing node 901 (at this time, there is no computing node in distributed system 900 that stores the same sharded data as computing node 901), then the sharded data of computing node 901 cannot be recovered from computing node 904 (nor can the sharded data of computing node 901 be recovered from other computing nodes in distributed training system 900). If computing node 902 in computing node group DP(2) fails and cannot be recovered, and computing node 905 in computing node group DP(1) is an equivalent node of computing node 902, but the sharded data stored by computing node 905 is different from the sharded data stored by computing node 902, then the sharded data of computing node 902 cannot be recovered from computing node 905. If computing node 903 in computing node group DP(t) fails and cannot be recovered, computing node 906 in computing node group DP(2) is the quasi-node of computing node 903. However, the fragmented data stored in computing node 906 is different from the fragmented data stored in computing node 903, so the fragmented data of computing node 903 cannot be recovered from computing node 906.

[0098] To address this issue, this application provides a data processing method for distributed training systems. This method copies the sharded data stored in a computing node to other computing nodes in the distributed training system for data backup. This ensures that there are computing nodes in the distributed training system capable of recovering the sharded data corresponding to the current computing node, thereby improving the training fault tolerance of the distributed training system. Furthermore, it eliminates the need to recover the sharded data from a persistent storage system, improving the efficiency of sharded data retrieval, which in turn improves the efficiency of training state recovery in the distributed training system, and consequently, the fault recovery efficiency of the distributed training system.

[0099] The following will use method embodiments to describe in detail the technical solutions provided in the embodiments of this application. For content not described in the method embodiments, please refer to the above embodiments, which will not be repeated here.

[0100] Please refer to Figure 12 It illustrates a flowchart of a data processing method for a distributed training system provided in one possible implementation of this application, wherein the execution entity of each step of the method can be... Figure 1 The distributed training system 100 shown is a master computing node. The method may include at least one of the following steps (1201-1202).

[0101] Step 1201: For the first computing node in at least one group of equipotential nodes, determine at least one computing node different from the first computing node from the distributed training system as at least one target computing node corresponding to the first computing node.

[0102] In this embodiment, the distributed training system can be the aforementioned distributed training system employing FSDP technology. For example, the distributed training system includes t groups of computing nodes, which are used to execute distributed training tasks in parallel. All computing nodes in the t groups are divided into at least one equivalence node group, each equivalence node group containing t computing nodes from different computing node groups. The checkpoint data corresponding to each equivalence node group is divided into t data shards, which are mutually exclusive. Each of the t computing nodes in the equivalence node group stores one shard of the t data shards.

[0103] The aforementioned first computing node can be any computing node in at least one equilateral node group. The target computing node corresponding to the first computing node is any computing node in the distributed training system that is different from the first computing node, and it is used to back up the sharded data stored by the first computing node. For example, the target computing node can be a computing node in the equilateral node group to which the first computing node belongs, or it can be a computing node in the computing node group to which the first computing node belongs, or it can be a backup node in the distributed training system. This application embodiment does not limit this.

[0104] Optionally, the memory of each computing node in the distributed training system can form a distributed memory, meaning that data can be exchanged between the memory of each computing node. The sharded data stored on each computing node can be backed up and stored through this distributed memory.

[0105] Optionally, the first computing node periodically updates its stored shard data. For example, during the execution of each stage of the distributed training task, the first computing node periodically updates its stored shard data, a process identical to periodically updating checkpoint data. For example, refer to... Figure 13 The distributed training system 1300 comprises eight compute nodes, each representing a physical server. Each compute node has a GPU device (for simplicity, only one GPU device is shown per compute node; in actual deployment, multiple GPU devices may exist per compute node). During distributed training, the GPU devices on each compute node periodically copy the portion of the state they are responsible for recording to the host memory, forming shard data. The portion of state recorded by each GPU device is different. If compute nodes Node-0 and Node-4 are peer nodes, then shard data 0 stored in compute node-0 and shard data 4 stored in compute node-4 can form a complete checkpoint data set, used to indicate the complete state of compute nodes Node-0 and Node-4 when participating in the distributed training task.

[0106] Optionally, the number of at least one target computing node is less than the parallelism t of the distributed training system to prevent the memory pressure of the distributed training system under FSDP technology from exceeding that under DP technology, thus eliminating the advantages of FSDP technology. This application does not limit the specific number of at least one target computing node. For example, each computing node may correspond to at least two target computing nodes.

[0107] In one example, how the target compute node is determined depends on the redundancy strategy of the distributed memory. For instance, this redundancy strategy may include at least one of the following: a multi-replica redundancy strategy, or an erasure coding strategy. The working mechanism of distributed memory ensures that the sharded data maintains the overall reliability of the checkpoint data even when some compute nodes are unavailable.

[0108] The multi-replica redundancy strategy involves backing up and storing multiple replicas of each data shard in distributed memory, meaning at least one target compute node is selected to back up and store the data shards. The erasure coding strategy involves adding m data shards for every t data shards. Any t data shards from the t+m data shards can be restored to the checkpoint data corresponding to the equivalence node group; that is, m target compute nodes are selected to store the m data shards.

[0109] For example, taking a multi-replica redundancy strategy as an example, the process of determining the target computing node may include at least one of the following:

[0110] 1. Based on the identification information of the first computing node, obtain at least one hash value; based on the at least one hash value, determine at least one computing node different from the first computing node from the distributed training system, as at least one target computing node corresponding to the first computing node.

[0111] The identification information of the first computing node is used to uniquely identify the first computing node. For example, the identification information of the first computing node can be its name, ID (Identity Document), number, etc. Optionally, the identification information of the first computing node can also be the global rank of the computing device (such as a GPU device) on the first computing node; this embodiment of the application does not limit this.

[0112] The hash value can be a value obtained through a hash function. A hash function is a function that transforms an input of arbitrary length into an output of fixed length. For example, the master computing node inputs the identification information of the first computing node into the hash function, and the hash function outputs a hash value. Optionally, when the number of at least two target computing nodes is two or more, two or more different hash functions can be adaptively set to obtain at least one target computing node corresponding to the first computing node.

[0113] For example, the master computing node obtains a list of computing nodes in the distributed training system, such as [C1, C2, ..., Ct]. If the first computing node is Ck, the identifier information of the target node can be calculated using the hash function target = hash(k) % t. hash() is a hash algorithm, such as k*p, MD5, SHA1, where p is a large prime number.

[0114] Optionally, the computing node whose identification information matches the hash value of the first computing node is determined as the target computing node of the first computing node. For example, the identification information of the target computing node of the first computing node is the same as the hash value of the first computing node.

[0115] The embodiments of this application determine the computing nodes used for backing up and storing sharded data through a hash function, which enables the distributed memory to be used in a balanced manner, thus improving the rationality of the use of distributed memory.

[0116] 2. Randomly select at least one computing node from the other computing nodes in the distributed training system, excluding the first computing node, as at least one target computing node corresponding to the first computing node.

[0117] That is, the master computing node can also randomly select at least one other computing node from the distributed training system as at least one target computing node corresponding to the first computing node.

[0118] Optionally, the master computing node may also randomly select at least one other computing node from the network domain to which the first computing node belongs as at least one target computing node corresponding to the first computing node. The method for determining the target computing node in this application embodiment is not limited and can be set and adjusted according to actual usage requirements.

[0119] In one example, after the master computing node determines at least one target computing node corresponding to the first computing node, it also stores the correspondence between the first computing node and the at least one target computing node corresponding to the first computing node in the mapping relationship data. The mapping relationship data is used to indicate at least one target computing node corresponding to each computing node in at least one equivalence node group.

[0120] The mapping relationship data can record the shard data stored on each computing node and the target computing node to which it is backed up. For example, the mapping relationship data includes the identification information of the first computing node that is bound to the storage and the identification information of at least one target computing node corresponding to the first computing node.

[0121] Optionally, the mapping relationship data also records the storage location of each data shard, i.e., the computing node where each data shard is stored. This application embodiment does not limit the format of the mapping relationship data. For example, the mapping relationship data can be implemented as a table, denoted as a mapping relationship table. For instance, the mapping relationship table uses the identification information of computing nodes as elements to construct the correspondence between computing nodes and their corresponding target computing nodes.

[0122] For example, refer to Figure 13The master compute node 1301 maintains a mapping table 1302, which records the compute nodes where each data shard is stored and the target compute nodes where each data shard is backed up. For data shards stored in Node-0, backups are stored in Node-1 and Node-7, which are the target compute nodes corresponding to Node-0. For data shards stored in Node-5, backups are stored in Node-3 and Node-6, which are the target compute nodes corresponding to Node-5. Optionally, the master compute node 1301 can be any one of the eight compute nodes, or it can be a compute node other than the eight compute nodes; this embodiment does not limit this.

[0123] The embodiments of this application can clarify the backup path of fragmented data by maintaining mapping relationship data, which is beneficial for the rapid reading and storage of fragmented data.

[0124] In one example, after the first compute node generates sharded data in the host memory, it sends a query to the master compute node requesting information on which target compute nodes correspond to the first compute node. Upon receiving the query, the master compute node determines the target compute nodes corresponding to the first compute node in real time. After determining the target compute nodes, the master compute node sends a notification to the first compute node, informing it of the target compute nodes and storing the mapping relationship between the first compute node and at least one corresponding target compute node in mapping relationship data.

[0125] In one example, the master compute node maintains preset mapping relationship data. Upon receiving a query from the first compute node, it determines at least one target compute node corresponding to the first compute node by querying the mapping relationship data. Then, it sends a notification message to the first compute node.

[0126] For example, refer to Figure 13 Taking compute node Node-0 as an example, with the master compute node 1301 maintaining a preset mapping table 1302, after generating shard data 0 in the host memory, compute node Node-0 sends a query message to the master compute node 1301. The master compute node 1301, by querying the mapping table 1302, determines that the target compute nodes corresponding to Node-0 include Node-1 and Node-7. The master compute node 1301 then sends a notification message to compute node Node-0 to inform it that its corresponding target compute nodes include Node-1 and Node-7.

[0127] Step 1202: Copy the sharded data stored in the first computing node to at least one target computing node corresponding to the first computing node for backup storage.

[0128] Optionally, the master compute node, through the aforementioned notification information, controls the first compute node to copy the sharded data stored in the first compute node to at least one target compute node corresponding to the first compute node for backup storage. For example, after receiving the notification information from the master compute node, the first compute node determines at least one target compute node based on the identification information recorded in the notification information, and then copies the sharded data stored in the host memory of the first compute node to at least one target compute node corresponding to the first compute node for backup storage.

[0129] In one example, the master compute node controls the first compute node to use RDMA (Remote Direct Memory Access) technology to asynchronously copy the fragmented data stored on the first compute node to the memory of at least one target compute node corresponding to the first compute node for backup storage.

[0130] RDMA technology uses dedicated hardware to read and write data in the memory of computing nodes and communicates directly with external systems. It can quickly migrate data from the local system to the memory of a remote system without requiring processors, buses, or overall scheduling by the operating system and protocol stack, thus not increasing the load on the operating system or processor. RDMA technology meets the requirements of high bandwidth and low latency, significantly reducing the transmission time of fragmented data and improving the backup efficiency of fragmented data. Optionally, the aforementioned distributed memory can be based on RDMA technology for data interaction.

[0131] For example, for two computing nodes that need to communicate using RDMA technology, network cards and hardware devices that support RDMA, as well as corresponding software and drivers, such as operating system-level drivers and application-level software, can be installed on the two computing nodes. These software and drivers can handle the details of RDMA communication, such as data transmission and reception, so that data can be transmitted between the two computing nodes using RDMA technology.

[0132] Optionally, using RDMA technology, fragmented data stored in the memory of the first computing node can be directly copied to the memory of the target computing node. For example, the master computing node can act as the sender of the first computing node and the receiver of the target computing node. The first computing node, as the sender, can use dedicated RDMA hardware to send the fragmented data to the target computing node. The target computing node, as the receiver, can use dedicated RDMA hardware to store the fragmented data sent by the first computing node into its memory. Optionally, the first computing node can also use dedicated RDMA hardware to directly send the fragmented data to the memory of the target computing node; this embodiment of the application does not limit this approach.

[0133] For example, refer to Figure 13 The master compute node 1301 can control compute node Node-0 to use RDMA technology to asynchronously copy the fragmented data 0 stored in compute node Node-0 to the memory of compute node Node-1 and compute node Node-7 for backup storage.

[0134] Optionally, the sharded data of a compute node is referred to as backup sharded data after being copied to the target compute node. For example, sharded data 0 of compute node Node-01 is referred to as backup sharded data 0-2 after being copied to compute node Node-1, and sharded data 0 of compute node Node-01 is referred to as backup sharded data 0-1 after being copied to compute node Node-7.

[0135] In one example, for any target compute node, the number of backup shard data stored in the target compute node is less than or equal to a quantity threshold, and the quantity threshold is less than t. Here, backup shard data refers to shard data stored in the target compute node other than the shard data corresponding to the target compute node.

[0136] In other words, the backup shard data stored in the target compute node can be shard data copied from other compute nodes to that target compute node. The shard data corresponding to the target compute node is the shard data generated by the computing device of the target compute node. For example... Figure 13 The fragmented data 0 generated by the computing node Node-0.

[0137] The aforementioned quantity threshold can be set and adjusted according to actual usage requirements, and this application embodiment does not limit this. For example, when the aforementioned quantity threshold is m (m is less than t), each target computing node can store at most m backup shard data, that is, each target computing node can be a target computing node for at most m different computing nodes.

[0138] Since the memory occupied by all shard data (including backup shard data) under FSDP technology is the same as the memory occupied by all checkpoint data under DP technology when the quantity threshold is t, the quantity threshold can be set to be less than t. This embodiment of the application avoids the failure of the memory reduction effect of FSDP technology by limiting the number of backup shard data stored in the target computing node.

[0139] In summary, the technical solution provided in this application, during the parallel execution of distributed training tasks using t computing node groups in a distributed training system, if the checkpoint data corresponding to each node group is divided into t fragments, and each fragment is stored in one computing node within that node group, then by copying the fragment data stored in one computing node to other computing nodes for data backup, a computing node capable of recovering the fragment data corresponding to that computing node exists within the distributed training system. This improves the training fault tolerance capability of the distributed training system. Furthermore, it eliminates the need to recover the fragment data corresponding to that computing node from the persistent storage system. Since retrieving data from a computing node is faster than retrieving data from persistent storage, this improves the efficiency of fragment data retrieval, thereby improving the training state recovery efficiency of the distributed training system and ultimately enhancing its fault recovery efficiency.

[0140] In some embodiments, after the first computing node copies the sharded data to at least one target computing node corresponding to the first computing node, if the first computing node fails, the sharded data of the first computing node can be recovered from at least one target computing node corresponding to the first computing node to resume distributed training. Therefore, embodiments of this application may further include the following:

[0141] 1. In the event of a failure of the first computing node, the backup node corresponding to the computing node group to which the first computing node belongs shall be determined as the replacement node corresponding to the first computing node.

[0142] Optionally, the backup node corresponding to the computing node group to which the first computing node belongs can be a backup node set in the computing node group to which the first computing node belongs, a backup node in the network domain to which the computing node group to which the first computing node belongs, or any backup node in the distributed training system. This application embodiment does not limit this.

[0143] In this embodiment, computing nodes participate in executing distributed training tasks, while standby nodes do not. However, standby nodes can be used to replace a computing node in the event of a computing node failure and continue participating in the distributed training task. Idle computing nodes (i.e., computing nodes without subtasks) can also be set as standby nodes.

[0144] In one example, if the first compute node fails and cannot continue to participate in the distributed training task, the backup node corresponding to the compute node group to which the first compute node belongs is determined as the replacement node corresponding to the first compute node.

[0145] If the first computing node fails and cannot continue to participate in the distributed training task, the first computing node can be identified as the faulty node, and the replacement node corresponding to the first computing node can be identified as the new computing node. The new computing node is used to replace the faulty node and continue to participate in the distributed training task. For example, the replacement node corresponding to the first computing node can be used to replace the first computing node to continue to execute the subtask corresponding to the first computing node.

[0146] For example, if the distributed training task in the p+1th stage is interrupted, the replacement node corresponding to the first computing node can inherit the training state (such as model parameters, parameter gradients, etc.) of the first computing node after completing the distributed training task in the pth stage, so that the distributed training system can re-execute the distributed training task in the p+1th stage, where p is an integer.

[0147] For example, refer to Figure 6 In the event of an interruption in the execution of a distributed training task, the training master uses a backup node 1 to replace the faulty computing node H3. Therefore, the backup node 1 can be set as the replacement node for computing node H3.

[0148] During the execution of distributed training tasks in a distributed training system, the master computing node can be used to continuously monitor the execution process of the distributed training tasks. In the event of an interruption in the execution of the distributed training tasks, the master computing node enters a fault diagnosis process. If the fault is unrecoverable, the master computing node will identify the faulty node from among multiple computing nodes.

[0149] Optionally, the execution of a distributed training task may be interrupted due to at least one of the following: hardware failure, software failure, network interruption, insufficient memory, or excessive CPU resource usage. For software failures and network interruptions, the distributed training system can directly repair the fault and resume the execution of the distributed training task. For hardware failures such as damaged computing nodes or damaged computing devices (e.g., GPU cards), the distributed training system cannot directly repair them.

[0150] For example, the failure of any computing node (such as a GPU server) in a distributed training system may lead to the interruption of the distributed training task; the failure of any computing device (such as a GPU card) in a distributed training system may also lead to the interruption of the distributed training task. In the embodiments of this application, a computing node that cannot continue to participate in the execution of the distributed training task due to a failure is called a faulty node, such as a computing node with a damaged GPU card, a computing node with a damaged hard drive, a computing node with a power outage, a computing node with a crashed operating system, or a computing node that has crashed. Optionally, in the event of an interruption in the execution of the distributed training task, the master computing node may identify one or more faulty nodes, and the aforementioned faulty nodes may refer to any faulty node identified by the master computing node.

[0151] In one example, the fault detection method includes at least one of the following:

[0152] (1) The master computing node can determine the specific cause of the failure by analyzing the logs of the distributed training system, such as the time, location and cause of the failure, so as to identify the faulty node.

[0153] (2) Each computing node will send its own status data to the master computing node. The master computing node can identify the faulty node by parsing the status data of each computing node.

[0154] (3) The master computing node can identify faulty nodes by executing a detection script. If a computing node fails the test, it can be identified as a faulty node.

[0155] (4) When a computing node fails, it will actively send an error message to the master computing node, which can then identify the faulty node based on the error message.

[0156] (5) Computing nodes can identify faulty nodes through a heartbeat mechanism.

[0157] (6) The master computing node can monitor the status of each computing node in real time by monitoring the status indicators of the distributed training system (such as CPU utilization, network latency, etc.) to identify the faulty node.

[0158] Optionally, the faulty node needs to be isolated so as not to affect the normal operation of other computing nodes. The isolation methods for the faulty node include at least one of the following: functional module isolation, resource isolation, thread-level isolation, service degradation, and circuit breaker pattern.

[0159] If the first computing node fails but recovers and can continue to participate in the distributed training task, the step of determining the backup node corresponding to the computing node group to which the first computing node belongs as the replacement node for the first computing node is not executed. That is, after the first computing node recovers its training state, it is controlled to continue participating in the distributed training task without determining the replacement node for the first computing node.

[0160] 2. Copy the backup shard data corresponding to the first computing node from the target computing node corresponding to the first computing node to the replacement node corresponding to the first computing node. The backup shard data corresponding to the first computing node is the shard data copied from the first computing node to the target computing node corresponding to the first computing node.

[0161] Optionally, after a backup node is determined to be a replacement node for the first computing node, the replacement node sends a query to the primary computing node, requesting a list of target computing nodes corresponding to the first computing node. The primary computing node queries the mapping relationship data based on the query to determine at least one target computing node corresponding to the first computing node. The primary computing node then sends a notification to the replacement node, informing it of the target computing nodes corresponding to the first computing node.

[0162] The primary compute node, through notification information, controls the replacement node corresponding to the first compute node to copy the backup shard data corresponding to the first compute node from the target compute node corresponding to the first compute node to the replacement node corresponding to the first compute node. In other words, the replacement node corresponding to the first compute node can determine at least one target compute node corresponding to the first compute node based on the notification information, and then copy the backup shard data corresponding to the first compute node from the target compute node corresponding to the first compute node to the replacement node corresponding to the first compute node.

[0163] Optionally, the replacement node corresponding to the first computing node can copy the backup shard data corresponding to the first computing node from any one of the at least one target computing nodes corresponding to the first computing node to the replacement node corresponding to the first computing node.

[0164] In one example, the primary compute node can control the replacement node corresponding to the first compute node to use RDMA technology to copy the backup shard data corresponding to the first compute node from the memory of the target compute node corresponding to the first compute node to the memory of the replacement node corresponding to the first compute node.

[0165] For example, refer to Figure 14If compute node Node-5 fails and cannot continue to participate in the distributed training task, standby node 1303 can be designated as the replacement node for compute node Node-5. Standby node 1303 sends a query message to the primary compute node 1301. The primary compute node 1301 queries the mapping table 1302 based on the query message and determines that the target compute nodes corresponding to compute node Node-5 include compute nodes Node-3 and Node-6. The primary compute node 1301 then sends a notification message to standby node 1303 to inform it that the target compute nodes corresponding to compute node Node-5 include compute nodes Node-3 and Node-6.

[0166] For example, the backup shard data of compute node Node-5 in compute node Node-3 is: backup shard data 5-2, and the backup shard data of compute node Node-5 in compute node Node-6 is: backup shard data 5-1. Backup shard data 5-2 and backup shard data 5-2 are the same as shard data 5 stored in compute node Node-5.

[0167] The standby node 1303 can use RDMA technology to read backup fragment data 5-2 from computing node Node-3, or it can read backup fragment data 5-1 from Node-6. This application embodiment does not limit this.

[0168] In one example, if the memory of the first compute node still supports providing sharded data, the sharded data stored in the first compute node is copied to the replacement node corresponding to the first compute node.

[0169] For example, if the primary compute node detects that the memory of the primary compute node still supports the provision of sharded data, it directly controls the replacement node to copy the sharded data stored in the primary compute node to the corresponding replacement node. This saves the query steps for mapping relationship data, thereby improving the recovery efficiency of sharded data.

[0170] Optionally, in this case, the primary compute node can also control the replacement node to copy the backup shard data corresponding to the first compute node from the target compute node corresponding to the first compute node to the replacement node corresponding to the first compute node. This application embodiment does not limit this aspect.

[0171] If the memory of the first compute node cannot provide the sharded data, the master compute node controls the replacement node to copy the backup sharded data corresponding to the first compute node from the target compute node corresponding to the first compute node to the replacement node corresponding to the first compute node.

[0172] Optionally, the inability of the first compute node's memory to provide checkpoint data may be due to at least one of the following: the first compute node's memory is corrupted, or the fragment data in the first compute node's memory is erased. For example, an operating system kernel crash or a restart of the first compute node can cause fragment data to be corrupted and thus unavailable.

[0173] This application embodiment supports the replacement node to recover sharded data from different computing nodes in scenarios where memory fails or does not fail, which helps improve the fault recovery capability of the distributed training system.

[0174] In this embodiment, for any computing node, it is necessary to aggregate the shard data of the computing node and the shard data of its peer nodes to recover the complete checkpoint data corresponding to the computing node. Then, based on the complete checkpoint data, the training state of the computing node can be recovered to resume the distributed training task. This embodiment may also include the following:

[0175] 3. Determine the first set of all nodes from at least one set of all nodes.

[0176] The aforementioned first equi-node group is the equi-node group to which the first computing node belongs. The fragmented data stored by each of the t computing nodes in the first equi-node group can be combined to obtain the complete checkpoint data of the t computing nodes in the first equi-node group.

[0177] 4. Obtain the fragmented data stored in the remaining computing nodes in the first equivalence node group, excluding the first computing node.

[0178] Optionally, the master computing node can control the replacement node of the first computing node to directly obtain the shard data stored by the remaining computing nodes from the memory of the remaining computing nodes, or it can control the replacement node of the first computing node to obtain the backup shard data stored by the remaining computing nodes from the memory of the target computing node of the remaining computing nodes, as the shard data stored by the remaining computing nodes. This application embodiment does not limit this.

[0179] For example, refer to Figure 14 In the case that Node-5 and Node-1 are peer nodes, the standby node 1303 will also copy the shard data 1 stored in Node-1 from the memory of Node-1 to the memory of the standby node 1303.

[0180] 5. Control the replacement node to participate in the distributed training task based on the backup shard data corresponding to the first computing node and the shard data stored in the remaining computing nodes.

[0181] Optionally, the replacement node can adjust its training state to be the same as that of the first computing node based on the backup shard data corresponding to the first computing node and the shard data stored in the remaining computing nodes, thereby replacing the first computing node to participate in the execution of distributed training tasks.

[0182] For example, refer to Figure 14 The backup node 1303 can restore the training state (N+1) of Node-5 after completing the distributed training task of the N-1 stage based on the shard data 1 and the backup shard data 5-1. Then, based on the state (N+1), the training state is adjusted to be the same as the training state of Node-5 after completing the training task of the N-1 stage, thereby replacing Node-5 to resume the execution of the training task of the N stage.

[0183] Optionally, for each computing device in the standby node 1303, after the standby node 1303 recovers from the training state state(N+1), the computing device reads the corresponding training state data in the training state state(N+1) to resume the execution of the distributed training task.

[0184] This application embodiment can directly restore the complete training state of the replacement node by gathering all the shard data required for the replacement node to resume the distributed training task through the memory of the computing node, without having to restore the shard data from the persistent storage system. This is beneficial to improving the efficiency of shard data acquisition, thereby improving the efficiency of training state restoration of the distributed training system.

[0185] In one example, in the event of a failure in the first computing node, embodiments of this application may further include at least one of the following:

[0186] 1. If the backup shard data corresponding to the second computing node is stored in the first computing node, copy the backup shard data corresponding to the second computing node to the replacement node corresponding to the first computing node.

[0187] In this scenario, the first compute node serves as the target compute node for the second compute node. The backup shard data corresponding to the second compute node consists of shard data copied from the second compute node to the first compute node.

[0188] For example, the master compute node controls the replacement node corresponding to the first compute node to copy the backup shard data corresponding to the second compute node from the memory of the first compute node to the memory of the replacement node corresponding to the first compute node, so as to complete the backup storage of the shard data stored on the second compute node again.

[0189] 2. If the backup shard data corresponding to the second computing node is stored in the first computing node, select the third computing node from the distributed training system; copy the backup shard data corresponding to the second computing node to the third computing node.

[0190] In this scenario, a third computing node replaces the first computing node as the new target computing node for the second computing node. The third computing node is different from both the first and second computing nodes.

[0191] For example, the master computing node controls the replacement node corresponding to the first computing node to copy the backup shard data corresponding to the second computing node from the memory of the first computing node to the memory of the third computing node, so as to complete the backup storage of the shard data stored on the second computing node again.

[0192] In the event of a computing node failure, this embodiment of the application helps to ensure the training fault tolerance capability of the distributed training system by backing up and storing the backup shard data in the computing node again.

[0193] Optionally, after the first computing node is replaced by a standby node, the primary computing node can replace the first computing node in the mapping relationship data with the standby node to update the mapping relationship data. During distributed training, the primary computing node updates the mapping relationship data in real time according to the changes in the sharded data to ensure that the mapping relationship data accurately records the correspondence between computing nodes and their target computing nodes.

[0194] For example, refer to Figure 14 When Node-5 is replaced by standby node 1303, the primary computing node 1301 replaces the identification information of Node-5 in the mapping table 1302 with the identification information of standby node 1303.

[0195] In summary, the technical solution provided by the embodiments of this application can quickly restore the shard data by using the backup shard data stored in the target node when the computing node fails, without having to restore the shard data corresponding to the computing node from the persistent storage system. This is beneficial to improving the efficiency of shard data acquisition, thereby improving the efficiency of training state recovery of the distributed training system, and further improving the fault recovery efficiency of the distributed training system.

[0196] In some embodiments, after the first computing node copies the sharded data to at least one target computing node corresponding to the first computing node, if the distributed training system experiences training scaling down or scaling up, it is necessary to re-shard the sharded data and re-back up and store the re-sharded sharded data. Therefore, embodiments of this application may further include the following:

[0197] 1. For any equi-node group in at least one equi-node group, when the number of computing nodes included in the equi-node group changes from t to s, the re-sharding instruction information of the equi-node group is obtained according to s. The re-sharding instruction information is used to indicate that t shards of data are re-divided into s s shards of data, where s is a positive integer.

[0198] Distributed training systems scale down or up training based on groups of computing nodes. Therefore, for each group of equal nodes, the number of computing nodes in the group also decreases or increases synchronously.

[0199] For example, when the number of computing node groups is changed from t to s, the number of computing nodes included in each equivalence node group changes from t to s.

[0200] In one example, with the standby node set up to build a scaling node group, the primary compute node changes t compute node groups to s compute node groups, and the number of compute nodes included in the unequal node groups changes from t to s.

[0201] The system comprises s compute node groups, including expansion node groups and t compute node groups. The expansion node groups are used to increase the parallelism of the distributed training system in executing distributed training tasks. One compute node from the expansion node group is added to each existing node group.

[0202] The number of compute nodes in the expansion node group is the same as the number of compute nodes in the compute node group. The expansion node group can consist of spare nodes, which can be used to increase the parallelism of the distributed training system in executing distributed training tasks by increasing the size of the compute node groups participating in the execution of distributed training tasks. Each compute node group can be used to train a complete machine learning model, and as the number of compute node groups participating in the execution of distributed training tasks increases, the execution efficiency of distributed training tasks also improves.

[0203] Optionally, when the expansion node group includes at least two backup nodes, the at least two backup nodes belong to the same network domain. This minimizes the communication links between the backup nodes in the expansion node group, thereby maximizing the efficiency of the expansion node group in executing the corresponding subtasks.

[0204] Optionally, if the primary computing node fails and there is no backup node to replace it, the computing node group to which the primary computing node belongs is removed from participating in the distributed training task, the t computing node groups are changed to s computing node groups, and the number of computing nodes included in the alternative node groups is changed from t to s.

[0205] The s compute node groups include t-1 compute node groups excluding the one to which the first compute node group belongs. Compute nodes in each equivalence node group that belong to the same compute node group as the first compute node group are removed and designated as standby nodes. The first compute node is isolated.

[0206] In this embodiment, the resharding indication information can be used to indicate the storage areas of s data s in the distributed memory. The master computing node can reshard the t data ...

[0207] Optionally, the storage area of ​​the fragmented data in the distributed memory can be characterized by a first parameter and a second parameter. The first parameter indicates the storage location of the fragmented data in the distributed memory, and the second parameter indicates the storage length of the fragmented data. The distributed memory consists of the memory of each computer node in the distributed system.

[0208] For example, refer to Figure 15 The equilateral node group 1500 includes t compute nodes, each storing one piece of data. Shard data 1 to shard data t constitute a complete checkpoint data set. Compute node Node-1 stores shard data 1. The first parameter of shard data 1 indicates that it is stored in compute node Node-1, and the second parameter indicates its length, i.e., the length of memory occupied by shard data 1 in compute node Node-1. Based on the first and second parameters, shard data 1 can be determined from compute node Node-1.

[0209] Optionally, the first parameter of shard data 1 can be the identification information of the computing node Node-1, such as the global rank, and the second parameter of shard data 1 can be the storage length of shard data 1.

[0210] In one example, if s data fragments correspond to a first parameter and a second parameter respectively in the resharding indication information, then the process of obtaining the resharding indication information may include the following:

[0211] (1) Based on the total storage length of s and t data segments, obtain the second parameter of each of the s data segments.

[0212] Optionally, the total storage length of the t data segments is divided by s to obtain the second parameter of each of the s data segments, which is the storage length of each individual data segment in the s data segments. The total storage length of the t data segments can be obtained by summing the storage lengths of the individual t data segments.

[0213] (2) Based on the second parameters of each of the s data segments and the storage locations of the t data segments in the distributed memory, the first parameters of each of the s data segments are obtained.

[0214] Optionally, for the first fragment of s fragmented data, based on the storage length of the first fragmented data, it can be determined which fragments of t fragmented data the first fragmented data was divided from. Then, based on the storage location of these fragmented data, the storage location of the first fragmented data, i.e., the first parameter, can be determined.

[0215] For the second data segment out of s data segments, based on the storage lengths of the first and second data segments, we can determine which data segments out of t data segments the second data segment was derived from. Then, based on the storage locations of these data segments, we can determine the storage location of the second data segment, i.e., the first parameter. Similarly, we can obtain the first parameter for each of the s data segments.

[0216] Optionally, the storage addresses of the s data fragments can be unified to the same dimension, so the first parameter of each of the s data fragments can also be the starting storage location of each of the s data fragments in the distributed memory. For example, for the s-th data fragment, the sum of the first parameters of the first s-1 data fragments is the second parameter of the s-th data fragment. This application does not limit this aspect.

[0217] For example, refer to Figure 15 When the number of computing nodes included in the equal-position computing node 1500 changes from t to t+1, the total storage length of the fragmented data 1 to fragmented data N can be divided by t+1 to obtain the first parameter of the t+1 fragmented data.

[0218] For fragment 1 of t+1 data fragments, since the storage length of fragment 1 is less than the storage length of the original fragment 1, fragment 1 can be obtained by partitioning the original fragment 1. Therefore, the second parameter of fragment 1 can be the identification information of the computing node Node-1, or the starting storage address of the original fragment 1 in the memory of computing node Node-1, or the starting storage address of the original fragment 1 in the distributed memory, or the starting storage address of fragment 1 in the distributed memory. Optionally, the starting storage address of the distributed memory can be 0.

[0219] For shard data 2 of t+1 shard data, shard data 2 is obtained by combining part of the original shard data and part of the original shard data 2. Therefore, the second parameter of shard data 2 can be the identification information of computing nodes Node-1 and Node2, or the starting storage address of the original shard data 1 and the original shard data 2 in the distributed memory, or the starting storage address of shard data 2 in the distributed memory.

[0220] This application embodiment enables accurate reading of fragmented data by using a first parameter and a second parameter to indicate the storage area of ​​fragmented data in distributed memory. This facilitates accurate automatic repartitioning of fragmented data without having to repartition the complete checkpoint data consisting of t fragmented data offline in the persistent storage system, and then restore the repartitioned fragmented data back to the computing nodes. This improves the efficiency of fragmented data repartitioning and thus enhances the performance of the distributed training system.

[0221] 2. Based on the resharding prompt, control the s computing nodes in the equal node group to obtain one fragment of data from the s fragments of data from the t computing nodes.

[0222] Optionally, the master compute node controls s compute nodes in the equal node group to obtain one fragment of data from s fragments of data from t compute nodes, based on the resharding prompt information.

[0223] For example, for any one of the s compute nodes, the master compute node determines the first and second parameters corresponding to the compute node from the resharding prompt information; and sends the first and second parameters corresponding to the compute node to the compute node. The first and second parameters corresponding to the compute node are used to instruct the compute node to obtain the s ...

[0224] Based on the first and second parameters, the compute nodes can determine the storage area of ​​the sharded data in distributed memory (such as the compute node where the data is stored), and then read the data in that storage area into the memory of the compute node, thus completing the repartitioning of one sharded data. After all t sharded data corresponding to t compute nodes have been read, the t sharded data is automatically repartitioned into s s sharded data.

[0225] For example, refer to Figure 15Given that t pieces of data are divided into s pieces, compute node Node-2 needs to store piece 2 from the s pieces. Piece 2 is derived partly from the original piece 1 and partly from the original piece 2. Therefore, compute node Node-2 needs to read a portion of the original piece 1 data from Node-1 and a portion of the original piece 2 data from itself to combine them into piece 2.

[0226] Optionally, after each of the s compute nodes reads one of the s ...

[0227] This application's embodiments utilize data reading to achieve natural repartitioning of fragmented data, a simple and easy-to-implement method. Furthermore, by repartitioning the fragmented data, the integrity of the fragmented data and its stable updates can be ensured, thereby improving the stability of the distributed training system.

[0228] In summary, the technical solution provided in this application simulates the storage area of ​​the repartitioned fragmented data by using repartitioning instruction information and controlling the computing nodes to read data according to the repartitioning instruction information, thereby realizing automatic repartitioning of fragmented data without having to repartition the complete checkpoint data composed of t fragmented data offline in the persistent storage system and then restore the repartitioned fragmented data to the computing nodes. This helps to improve the repartitioning efficiency of fragmented data and thus improves the performance of the distributed training system.

[0229] In addition, the technical solutions provided in this application are applicable to training fault-tolerant scenarios such as fault node replacement, training scaling down, and training scaling up, and have good universality and high versatility.

[0230] In some embodiments, if a first condition is met, a backup node is selected from at least one backup node included in the distributed training system to construct a scaling node group. The first condition includes at least one of the following:

[0231] 1. The number of computing node groups in a distributed training system is less than the number threshold.

[0232] Optionally, the aforementioned quantity threshold may refer to the number of computing node groups in the distributed training system in the initial state, and the distributed training system when executing the first stage of the distributed training task can be defined as the distributed training system in the initial state.

[0233] For example, if the number of computing node groups in the initial state of the distributed training system is t, then t can be determined as the number threshold. For instance, if the initial distributed training system includes N computing node groups, and this number is reduced to N-1 computing node groups due to computing node failure and the lack of replacement nodes, the number of computing node groups in the distributed training system becomes less than the number threshold. In this case, a new node group can be built using backup nodes to supplement the distributed training system, thereby improving the efficiency of the distributed training system in executing distributed training tasks and restoring the efficiency of distributed training task execution.

[0234] Optionally, in cases where a scaling-up node group needs to be built due to scaling down, the scaling-up can be carried out after the distributed training system has been scaled down and a phase of training tasks has been performed; alternatively, the scaling-up node group can be built to carry out the scaling-up when it is determined that the distributed training system needs to be scaled down. This application embodiment does not limit this.

[0235] Optionally, the aforementioned quantity threshold can also refer to a preset value, which can be set and adjusted according to actual usage requirements. This preset value is used to ensure that the distributed training system is used as much as possible to execute distributed training tasks, and to reserve sufficient spare nodes for fault tolerance. For example, 80% of the maximum number of computing node groups that the distributed training system can build for distributed training tasks can be determined as the quantity threshold, with the remaining computing nodes set as spare nodes.

[0236] 2. A new backup node appears in the distributed training system.

[0237] Optionally, the distributed training system may include not only the standby nodes set up in the initial state, but also newly emerging standby nodes. If the number of new standby nodes in the same network domain meets the requirements for constructing an expansion node group, then the new standby nodes in the same network domain can be combined into an expansion node group.

[0238] For example, the process of determining a new backup node may include the following:

[0239] (1) In the case that the fault node group to which the fault node belongs in the distributed training system is canceled from executing the corresponding subtask, the computing nodes in the fault node group other than the fault node are set as standby nodes.

[0240] A fault node group can refer to a group of computing nodes that includes faulty nodes. For example, refer to... Figure 6 If the faulty node group containing compute node H2 is canceled from executing the corresponding subtask, compute node H2 can be adjusted to be a standby node, so that compute node H2 is a newly added standby node.

[0241] If computing node H1 also fails and there is no backup node to replace it, computing node H0 can also be adjusted to become a new backup node. In this case, computing nodes H0 and H2 can be built into an expanded node group to avoid excessive decrease in the execution efficiency of distributed training tasks.

[0242] (2) In the case that a computing node used to perform other tasks besides distributed training tasks in a distributed training system is released, the released computing node is set as a standby node.

[0243] Other tasks may refer to tasks different from the distributed training task. The release of a computing node may mean that the computing node is no longer occupied by other tasks, such as the computing node being canceled from executing other tasks, or the computing node being automatically released after completing other tasks. This application embodiment does not limit this.

[0244] For example, refer to Figure 6 If the distributed training system 600 has a second computing node that is currently being used to perform other tasks, this second computing node can be set as a new standby node in the distributed training system 600 after it is automatically released from its other tasks. Similarly, if the distributed training system 600 has a second computing node that is being used by a user and cannot participate in the distributed training task, this second computing node can be set as a new standby node in the distributed training system 600 after the user has finished using it.

[0245] (3) In the event of a new computing node in the distributed training system, the new computing node shall be set as a standby node.

[0246] In other words, after adding a computing node from outside the distributed training system to the distributed training system, that computing node can be set as a new standby node. For example, refer to... Figure 6 When a new computing node is connected to the distributed training system 600, the new computing node can be set as a new backup node in the distributed training system 600.

[0247] 3. Enter the preset time period.

[0248] Optionally, the preset time period can be set and adjusted according to actual usage needs, and this application embodiment does not limit this. For example, a time period with low computing node utilization can be set as the preset time period. For instance, after entering the time period from 0:00 to 3:00 AM, a spare node can be selected from at least one spare node included in the distributed training system to be set as a new computing node to build at least one expansion node group.

[0249] In one example, if the multiple computing nodes included in the above distributed training system are divided into t computing node groups, the number of nodes s in each computing node group can be obtained first, where s is a positive integer; from at least one spare node, s spare nodes are selected to construct an expansion node group.

[0250] The number of nodes, 's', refers to the number of computing nodes in a computing node group, which can be determined based on the parallel techniques applied in the distributed training system. For example, dynamic programming (DP) can be used to determine the number of computing node groups, while parallel processing (TP) and parallel processing (PP) can be used to determine the number of nodes, 's', within a computing node group.

[0251] For example, refer to Figure 6 Each computing node group includes 2 computing nodes. If the fault node group to which the fault node H2 belongs is canceled from executing the corresponding subtask, the main computing node can cross network domains and find 2 backup nodes in other network domains to serve as new computing nodes. Based on these 2 new computing nodes, an expanded node group is built to replace the fault node group to which the fault node H2 belongs in executing the corresponding subtask, so as to maintain the execution efficiency of the distributed training task.

[0252] Optionally, after constructing the expansion node group, the master computing node can update the expansion node group to a computing node group, then count the total number of computing node groups in the distributed training system, divide the distributed training task into a total number of subtasks, and finally execute the total number of subtasks in parallel through all computing node groups to resume the execution of the distributed training task.

[0253] In summary, the technical solutions provided in this application can restore the execution efficiency of distributed training tasks by scaling down the distributed training system and then scaling it up again through backup nodes. Alternatively, the distributed training system can be scaled up through backup nodes to accelerate the execution of distributed training tasks, thereby improving the stability and performance of the distributed training system.

[0254] In some embodiments, reference Figure 16 The embodiments of this application may also include the following contents.

[0255] 1. Practice pull-ups.

[0256] Distributed training systems execute distributed training tasks for machine learning models to initiate distributed training.

[0257] For example, refer to Figure 10The distributed training system 900 can execute distributed training tasks in parallel through t computing node groups. For example, each computing node group trains a complete machine learning model based on a portion of sample data to obtain training status (such as model parameters, parameter gradients, etc.). The distributed training system 900 then aggregates the training statuses corresponding to the t computing node groups to update the model parameters of the machine learning model, obtaining the updated model parameters. Finally, the updated model parameters are fully distributed to the t computing node groups to carry out the next round of distributed training until the training of the machine learning model is completed.

[0258] Optionally, after completing each stage of the distributed training task, the distributed training system 900 controls each computing node to store the training state it is responsible for recording into its respective host memory, forming sharded data. For the same peer node group, the sharded data stored by t computing nodes in the peer node group can form a complete checkpoint data, which can be used to indicate the state of t computing nodes when participating in the distributed training task. Different sharded data can be used to indicate different parts of the state of t computing nodes when participating in the distributed training task. In addition, the sharded data corresponding to each computing node is also synchronized to the persistent storage system for persistence.

[0259] Optionally, for any computing node in the distributed training system, the sharded data stored on that computing node is copied to at least two target computing nodes for backup storage. For example, refer to... Figure 13 The sharded data 5 stored on compute node Node-5 is copied to compute nodes Node-3 and Node-6 for backup storage.

[0260] In a distributed training system, the memory of each computing node constitutes distributed memory. Therefore, reading sharded data from a computing node can be understood as reading sharded data from distributed memory. Compared to retrieving sharded data from a persistent storage system, reading sharded data from distributed memory is much faster.

[0261] 2. Training monitoring.

[0262] The master compute node continuously monitors the distributed training task and exits its execution once the task is completed. Optionally, the master compute node also monitors the status of each compute node in the distributed training system.

[0263] For example, refer to Figure 13 The master computing node 1301 in the distributed training system 1300 can continuously monitor the status of the eight computing nodes in the distributed training system 1300.

[0264] 3. Fault diagnosis.

[0265] In the event of an interruption in the execution of a distributed training task, that is, in the event of a failure in the distributed training system that causes the interruption of the distributed training task, the main computing node enters the fault diagnosis process.

[0266] For example, refer to Figure 13 In the event of an interruption in the execution of a distributed training task, the master computing node 1301 in the distributed training system 1300 can analyze the cause of the failure based on the logs of the distributed training system 1300 to determine whether the failure is repairable. If the failure is repairable, proceed to step 4; if the failure is not repairable, proceed to step 5.

[0267] The master computing node 1301 can also identify faulty nodes based on the logs of the distributed training system 1300.

[0268] 4. Troubleshooting.

[0269] If the fault is repairable, the master compute node can directly guide the distributed training system to resume executing the distributed training task after the fault is repaired, and then proceed to step 1 after the fault is repaired.

[0270] During the fault repair process, for each equinode group, each computing node in the equinode group aggregates the fragmented data stored by each computing node in the equinode group to obtain the complete checkpoint data corresponding to the equinode group, and then resumes the execution of the distributed training task based on the complete checkpoint data.

[0271] For example, refer to Figure 13 In the event that the execution of the distributed training task is interrupted due to a network interruption of a computing node, the master computing node 1301 can directly control each peer node group in the distributed training system 1300 after restoring the network of the computing node, collect the fragmented data from the host memory of each computing node in the peer node group to obtain complete checkpoint data, and then resume the execution of the distributed training task.

[0272] Optionally, for each computing node, each computing device (such as a CPU) on the computing node reads the corresponding data from the sharded data in the host memory to restore the corresponding training state.

[0273] 5. Find a backup node.

[0274] If the fault is irreparable, for each faulty node, the primary compute node determines the first network domain to which the faulty node belongs, and then searches for a backup node within that first network domain. If there are backup nodes in the first network domain, and there are enough backup nodes to replace all faulty nodes in the first network domain, the selected backup node is set as the new compute node to replace all faulty nodes in the first network domain. This allows the backup node to perform the corresponding subtasks in place of the faulty nodes, thereby resuming the execution of the distributed training task.

[0275] For reading sharded data, if the host memory of the failed node is still readable (i.e., still provides sharded data), the sharded data is copied from the host memory of the failed node to the host memory of the new compute node. The new compute node then adjusts its training state to the same state as before the failed node's failure based on the sharded data, and then proceeds to step 1. If the host memory of the failed node fails (i.e., cannot provide checkpoint data), backup sharded data is copied from the host memory of the target compute node of the failed node to the host memory of the new compute node. The new compute node then adjusts its training state to the same state as before the failed node's failure based on the backup sharded data, and then proceeds to step 1.

[0276] For example, refer to Figure 13 If only compute node Node-5 fails in the distributed training system 1300, standby node 1303 can be set as the new compute node to replace compute node Node-5. While compute node Node-5 can still provide sharded data, the primary compute node 1301 controls the standby node 1303 to use RDMA technology to read the sharded data from the host memory of compute node Node-5 into the host memory of the standby node 1303, thus controlling the distributed training system 1300 to restart the distributed training of the machine learning model.

[0277] If compute node Node-5 is unable to provide sharded data, the primary compute node 1301 controls the standby node 1303 to use RDMA technology to read the backup sharded data 5-1 from the host memory of the target compute node (i.e., compute node Node-5) of compute node-5 into the host memory of the standby node 1303, thereby controlling the distributed training system 1300 to restart the distributed training of the machine learning model.

[0278] After compute node Node-5 is replaced by standby node 1303, the primary compute node 1301 isolates compute node Node-5. Optionally, if there are not enough standby nodes in the first network domain, proceed to step 6.

[0279] 6. Try reducing the size.

[0280] If there are not enough spare nodes in the first network domain, the master computing node guides the distributed training system to scale down, that is, directly controls the failure node group to which the failure node belongs to cancel the execution of the corresponding subtask.

[0281] For example, refer to Figure 6 If computing node H2 also fails and cannot be recovered, and there is no backup node in the network domain to which computing node H2 belongs, the master computing node can control the distributed training system 600 to exclude the group of failed nodes to which computing node H2 belongs from the execution process of the distributed training task, so as to achieve scaling down.

[0282] Optionally, if the distributed training system successfully scales down, for any equi-node group in the distributed training system, when the number of computing nodes in the equi-node group changes from t to s, the master computing node obtains the resharding instruction information for the equi-node group based on s. This resharding instruction information instructs the t s shards to be re-divided into s s s shards. The master computing node then, based on the resharding instruction information, controls the s computing nodes in the equi-node group to each retrieve one s s s s s s s s shards from the t computing nodes, thus completing the resharding of the data.

[0283] After the data fragments are re-partitioned, the data fragments can be backed up and stored again. Then, proceed to step 1, which is to restore the distributed training task through the scaled-down distributed training system. The execution efficiency of the distributed training task is reduced.

[0284] 7. Try expanding the capacity.

[0285] After the distributed training system is scaled down, if there are enough spare nodes in another network domain, the primary compute node can attempt to build a scaling-up node group using the spare nodes in that other network domain to scale up the distributed training system and restore the execution efficiency of the distributed training tasks. Specifically, the primary compute node sets the selected spare node as the new compute node to build the scaling-up node group.

[0286] For example, refer to Figure 6 If the faulty node group to which computing node H2 belongs is excluded from the execution of the distributed training task, two backup nodes can be found in other network domains and set as new computing nodes to build an expanded node group to make up for the impact of the absence of the faulty node group to which computing node H2 belongs on the execution efficiency of the distributed training task.

[0287] Optionally, if the distributed training system is successfully scaled up, for any equivalence node group in the distributed training system, when the number of computing nodes in the equivalence node group changes from t to s, the master computing node obtains the resharding instruction information for the equivalence node group based on s. The resharding instruction information is used to instruct the t s shards of data to be re-divided into s s s shards of data. The master computing node then controls the s computing nodes in the equivalence node group, based on the resharding instruction information, to each obtain one s s s s s s s s s shards of data from the t computing nodes, thereby completing the resharding of the data.

[0288] After the data shards are re-partitioned, the data shards are backed up and stored again. Then, the process moves to step 1, which is to restore the distributed training task through the expanded distributed training system. The execution efficiency of the distributed training task returns to normal.

[0289] In summary, the technical solution provided in this application, during the parallel execution of distributed training tasks using t computing node groups in a distributed training system, if the checkpoint data corresponding to each node group is divided into t fragments, and each fragment is stored in one computing node within that node group, then by copying the fragment data stored in one computing node to other computing nodes for data backup, a computing node capable of recovering the fragment data corresponding to that computing node exists within the distributed training system. This improves the training fault tolerance capability of the distributed training system. Furthermore, it eliminates the need to recover the fragment data corresponding to that computing node from the persistent storage system. Since retrieving data from a computing node is faster than retrieving data from persistent storage, this improves the efficiency of fragment data retrieval, thereby improving the training state recovery efficiency of the distributed training system and ultimately enhancing its fault recovery efficiency.

[0290] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0291] refer to Figure 17 This diagram illustrates a block diagram of a data processing apparatus for a distributed training system, provided in one possible implementation of this application. The apparatus has the functionality to implement the method examples described above; this functionality can be implemented in hardware or by hardware executing corresponding software. The apparatus can be a computer device (such as a master computing node) as described above, or it can be located within a computer device. Figure 17 As shown, the device 1700 includes: a target node determination module 1701 and a sharded data backup module 1702.

[0292] The target node determination module 1701 is used to determine, for the first computing node in the at least one equivalence node group, at least one computing node different from the first computing node in the distributed training system, as at least one target computing node corresponding to the first computing node.

[0293] The sharded data backup module 1702 is used to copy the sharded data stored on the first computing node to at least one target computing node corresponding to the first computing node for backup storage.

[0294] In some embodiments, the target node determination module 1701 is configured to:

[0295] Based on the identification information of the first computing node, at least one hash value is obtained; based on the at least one hash value, at least one computing node different from the first computing node is determined from the distributed training system as at least one target computing node corresponding to the first computing node;

[0296] Alternatively, at least one computing node may be randomly selected from the other computing nodes in the distributed training system besides the first computing node, as at least one target computing node corresponding to the first computing node.

[0297] In some embodiments, such as Figure 18 As shown, the device 1700 further includes a correspondence storage module 1703.

[0298] The correspondence storage module 1703 is used to store the correspondence between the first computing node and at least one target computing node corresponding to the first computing node in the mapping relationship data, wherein the mapping relationship data is used to indicate at least one target computing node corresponding to each computing node in the at least one equivalence node group.

[0299] In some embodiments, for any target computing node, the number of backup shard data stored in the target computing node is less than or equal to a quantity threshold, the quantity threshold being less than t, wherein the backup shard data refers to shard data stored in the target computing node other than the shard data corresponding to the target computing node.

[0300] In some embodiments, the distributed training system further includes at least one backup node, which does not participate in executing the distributed training task; such as Figure 18 As shown, the device 1700 further includes: a replacement node determination module 1704 and a fragmented data recovery module 1705.

[0301] The replacement node determination module 1704 is used to determine the backup node corresponding to the computing node group to which the first computing node belongs as the replacement node corresponding to the first computing node in the event of a failure of the first computing node.

[0302] The shard data recovery module 1705 is used to copy the backup shard data corresponding to the first computing node from the target computing node corresponding to the first computing node to the replacement node corresponding to the first computing node, wherein the backup shard data corresponding to the first computing node is the shard data copied from the first computing node to the target computing node corresponding to the first computing node.

[0303] In some embodiments, such as Figure 18 As shown, the device 1700 further includes a fragmented data copying module 1706.

[0304] The shard data copying module 1706 is used to copy the backup shard data corresponding to the second computing node to the replacement node corresponding to the first computing node when the first computing node stores the backup shard data corresponding to the second computing node.

[0305] Alternatively, the shard data copy module 1706 is further configured to select a third computing node from the distributed training system when the first computing node stores backup shard data corresponding to the second computing node; and copy the backup shard data corresponding to the second computing node to the third computing node.

[0306] In some embodiments, the fragmented data recovery module 1705 is configured to:

[0307] If the memory of the first computing node still supports providing the sharded data, the sharded data stored in the first computing node is copied to the replacement node corresponding to the first computing node.

[0308] If the memory of the first computing node cannot provide the shard data, the step of copying the backup shard data corresponding to the first computing node from the target computing node corresponding to the first computing node to the replacement node corresponding to the first computing node is executed.

[0309] In some embodiments, the replacement node determination module 1704 is used to determine the backup node corresponding to the computing node group to which the first computing node belongs as the replacement node corresponding to the first computing node when the first computing node fails and cannot continue to participate in the execution of the distributed training task.

[0310] In some embodiments, the replacement node determination module 1704 is further configured to, when the first computing node fails and continues to participate in the execution of the distributed training task after recovery, not to execute the step of determining the backup node corresponding to the computing node group to which the first computing node belongs as the replacement node corresponding to the first computing node.

[0311] In some embodiments, such as Figure 18 As shown, the device 1700 further includes: an equipotential node group determination module 1707, a fragmented data aggregation module 1708, and a training task recovery module 1709.

[0312] The equipotential node group determination module 1707 is used to determine a first equipotential node group from the at least one equipotential node group, wherein the first equipotential node group is the equipotential node group to which the first computing node belongs.

[0313] The sharded data aggregation module 1708 is used to obtain the sharded data stored by the remaining computing nodes in the first equidistant node group, excluding the first computing node.

[0314] The training task recovery module 1709 is used to control the replacement node to participate in the execution of the distributed training task based on the backup shard data corresponding to the first computing node and the shard data stored in the remaining computing nodes.

[0315] In some embodiments, such as Figure 18 As shown, the device 1700 further includes: an instruction information acquisition module 1710 and a fragmented data re-division module 1711.

[0316] The instruction information acquisition module 1710 is used to obtain resharding instruction information of the at least one equi-node group when the number of computing nodes included in the equi-node group changes from t to s, based on the s. The resharding instruction information is used to indicate that the t shards of data be re-divided into s s shards of data, where s is a positive integer.

[0317] The data repartitioning module 1711 is used to control the s computing nodes in the equal node group to obtain one fragment of data from the t computing nodes according to the repartitioning prompt information.

[0318] In some embodiments, the s data s fragments correspond to a first parameter and a second parameter respectively in the resharding indication information. The first parameter is used to indicate the storage location of the data s fragments in the distributed memory, and the second parameter is used to indicate the storage length of the data s fragments. The distributed memory is composed of the memory of each computer node in the distributed system.

[0319] The instruction information acquisition module 1710 is used to obtain the second parameters of each of the s data segments based on the total storage length of the s and t data segments; and to obtain the first parameters of each of the s data segments based on the second parameters of each of the s data segments and the storage locations of the t data segments in the distributed memory.

[0320] In some embodiments, the fragmented data re-partitioning module 1711 is configured to:

[0321] For any one of the s computing nodes, determine the first parameter and the second parameter corresponding to the computing node from the resharding prompt information;

[0322] The first and second parameters corresponding to the computing node are sent to the computing node. The first and second parameters corresponding to the computing node are used to instruct the computing node to obtain the s ...

[0323] In some embodiments, the distributed training system further includes at least one backup node, which does not participate in executing the distributed training task; such as Figure 18 As shown, the device 1700 further includes: an equal node group adjustment module 1712.

[0324] The equivalence node group adjustment module 1712 is used to change the t computing node groups to the s computing node groups when the spare nodes are set to be used to build an expansion node group, and to change the number of computing nodes included in the equivalence node groups from t to s, wherein the s computing node groups include the expansion node groups, and the expansion node groups are used to increase the parallelism of the distributed training system in executing the distributed training task.

[0325] Alternatively, the equivalence node group adjustment module 1712 is used to cancel the participation of the computing node group to which the first computing node belongs in the execution of the distributed training task when the first computing node fails and there is no backup node to replace it, change the t computing node groups to the s computing node groups, and change the number of computing nodes included in the equivalence node group from t to s.

[0326] In some embodiments, the shard data backup module 1702 is used to control the first computing node to use RDMA technology to asynchronously copy the shard data stored by the first computing node to the memory of at least one target computing node corresponding to the first computing node for backup storage.

[0327] In summary, the technical solution provided in this application, during the parallel execution of distributed training tasks using t computing node groups in a distributed training system, if the checkpoint data corresponding to each node group is divided into t fragments, and each fragment is stored in one computing node within that node group, then by copying the fragment data stored in one computing node to other computing nodes for data backup, a computing node capable of recovering the fragment data corresponding to that computing node exists within the distributed training system. This improves the training fault tolerance capability of the distributed training system. Furthermore, it eliminates the need to recover the fragment data corresponding to that computing node from the persistent storage system. Since retrieving data from a computing node is faster than retrieving data from persistent storage, this improves the efficiency of fragment data retrieval, thereby improving the training state recovery efficiency of the distributed training system and ultimately enhancing its fault recovery efficiency.

[0328] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0329] Please refer to Figure 19 This is a simplified structural block diagram of a computer device 1900 provided in one possible implementation of this application. The computer device 1900 can be implemented as the aforementioned computing node, standby node, or master computing node. The computer device 1900 can be used to implement the data processing method for a distributed training system provided in the above embodiments.

[0330] Typically, computer device 1900 includes a processor 1901 and a memory 1902.

[0331] Processor 1901 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1901 may also include an AI processor for handling computational operations related to machine learning.

[0332] The memory 1902 may include one or more computer-readable storage media, which may be non-transitory. The memory 1902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1902 is used to store a computer program configured to be executed by one or more processors to implement the data processing method described above for a distributed training system.

[0333] Those skilled in the art will understand that Figure 19 The structure shown does not constitute a limitation on the computer device 1900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0334] Optionally, the computer device 1900 may be a server, a server cluster, an artificial intelligence computing cluster, a cloud computing cluster, etc., wherein the artificial intelligence computing cluster may also be referred to as an intelligent computing cluster or a smart computing cluster, and this application does not limit it in this regard.

[0335] In some embodiments, a chip product is also provided, wherein a computer program is stored therein, the computer program being loaded and executed by a processor to implement the data processing method described above for use in a distributed training system.

[0336] In some embodiments, a computer-readable storage medium is also provided, wherein a computer program is stored therein, which, when executed by a processor of a computer device, implements the above-described data processing method applied to a distributed training system.

[0337] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0338] In some embodiments, a computer program product is also provided, the computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the data processing method described above for a distributed training system.

[0339] It should be understood that "multiple" as used herein refers to two or more. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order between steps. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this approach.

[0340] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method applied to a distributed training system, characterized in that, The distributed training system includes t computing node groups, which are used to execute distributed training tasks in parallel. All computing nodes in the t computing node groups are divided into at least one equi-node group, each equi-node group containing t computing nodes from different computing node groups. The checkpoint data corresponding to each equi-node group is divided into t unique data shards. Each of the t computing nodes in the equi-node group stores one shard from the t unique data shards. The checkpoint data is used to indicate the state of the t computing nodes in the equi-node group when participating in the execution of the distributed training task, where t is an integer greater than 1. The method includes: For the first computing node in the at least one group of equal nodes, at least one computing node different from the first computing node is determined from the distributed training system as at least one target computing node corresponding to the first computing node. The sharded data stored on the first computing node is copied to at least one target computing node corresponding to the first computing node for backup storage.

2. The method according to claim 1, characterized in that, The step of determining at least one computing node different from the first computing node from the distributed training system as at least one target computing node corresponding to the first computing node includes: Based on the identification information of the first computing node, at least one hash value is obtained; based on the at least one hash value, at least one computing node different from the first computing node is determined from the distributed training system as at least one target computing node corresponding to the first computing node; or, From the distributed training system, at least one computing node is randomly selected from the other computing nodes besides the first computing node as at least one target computing node corresponding to the first computing node.

3. The method according to claim 1, characterized in that, After determining at least one computing node different from the first computing node from the distributed training system as at least one target computing node corresponding to the first computing node, the method further includes: The correspondence between the first computing node and at least one target computing node corresponding to the first computing node is stored in the mapping relationship data, which is used to indicate at least one target computing node corresponding to each computing node in the at least one equivalence node group.

4. The method according to claim 3, characterized in that, For any target computing node, the number of backup shard data stored in the target computing node is less than or equal to a number threshold, the number threshold being less than t, wherein the backup shard data refers to shard data stored in the target computing node other than the shard data corresponding to the target computing node.

5. The method according to any one of claims 1 to 4, characterized in that, The distributed training system also includes at least one backup node, which does not participate in executing the distributed training task; After copying the sharded data stored on the first computing node to at least one target computing node corresponding to the first computing node for backup storage, the method further includes: In the event of a failure of the first computing node, the backup node corresponding to the computing node group to which the first computing node belongs is determined as the replacement node corresponding to the first computing node. The backup shard data corresponding to the first computing node is copied from the target computing node corresponding to the first computing node to the replacement node corresponding to the first computing node, wherein the backup shard data corresponding to the first computing node is the shard data copied from the first computing node to the target computing node corresponding to the first computing node.

6. The method according to claim 5, characterized in that, The method further includes: If the backup fragment data corresponding to the second computing node is stored in the first computing node, the backup fragment data corresponding to the second computing node is copied to the replacement node corresponding to the first computing node. or, If the backup shard data corresponding to the second computing node is stored in the first computing node, a third computing node is selected from the distributed training system; the backup shard data corresponding to the second computing node is copied to the third computing node.

7. The method according to claim 5, characterized in that, The step of copying the backup shard data corresponding to the first computing node from the target computing node corresponding to the first computing node to the replacement node corresponding to the first computing node includes: If the memory of the first computing node still supports providing the sharded data, the sharded data stored in the first computing node is copied to the replacement node corresponding to the first computing node. If the memory of the first computing node cannot provide the shard data, the step of copying the backup shard data corresponding to the first computing node from the target computing node corresponding to the first computing node to the replacement node corresponding to the first computing node is executed.

8. The method according to claim 5, characterized in that, In the event of a failure of the first computing node, determining the backup node corresponding to the computing node group to which the first computing node belongs as the replacement node for the first computing node includes: If the first computing node fails and is unable to continue participating in the distributed training task, the backup node corresponding to the computing node group to which the first computing node belongs shall be determined as the replacement node corresponding to the first computing node.

9. The method according to claim 8, characterized in that, The method further includes: If the first computing node fails but continues to participate in the distributed training task after recovery, the step of determining the backup node corresponding to the computing node group to which the first computing node belongs as the replacement node corresponding to the first computing node is not executed.

10. The method according to claim 5, characterized in that, The method further includes: A first equivalence node group is determined from the at least one equivalence node group, wherein the first equivalence node group is the equivalence node group to which the first computing node belongs; Obtain the fragmented data stored by the remaining computing nodes in the first equidistant node group, excluding the first computing node; The replacement node is controlled to participate in the distributed training task based on the backup shard data corresponding to the first computing node and the shard data stored in the remaining computing nodes.

11. The method according to any one of claims 1 to 4, characterized in that, After copying the sharded data stored on the first computing node to at least one target computing node corresponding to the first computing node for backup storage, the method further includes: For any of the at least one equi-node groups, when the number of computing nodes included in the equi-node group changes from t to s, the re-sharding indication information of the equi-node group is obtained according to s. The re-sharding indication information is used to indicate that the t shards of data are re-divided into s s shards of data, where s is a positive integer. According to the resharding instruction information, control the s computing nodes in the equal node group to obtain one of the s ...

12. The method according to claim 11, characterized in that, The s data s fragments correspond to a first parameter and a second parameter respectively in the resharding indication information. The first parameter is used to indicate the storage location of the data s fragments in the distributed memory, and the second parameter is used to indicate the storage length of the data s fragments. The distributed memory is composed of the memory of each computer node in the distributed training system. The step of obtaining the repartitioning indication information of the equipotential node group based on s includes: Based on the total storage length of the s and t data segments, the second parameter of each of the s data segments is obtained; Based on the second parameters of each of the s data segments and the storage locations of the t data segments in the distributed memory, the first parameters of each of the s data segments are obtained.

13. The method according to claim 12, characterized in that, The step of controlling the s computing nodes in the equal node group to obtain one fragment of data from the t computing nodes according to the re-sharding instruction information includes: For any one of the s computing nodes, determine the first parameter and the second parameter corresponding to the computing node from the resharding indication information; The first and second parameters corresponding to the computing node are sent to the computing node. The first and second parameters corresponding to the computing node are used to instruct the computing node to obtain the s ...

14. The method according to claim 11, characterized in that, The distributed training system further includes at least one backup node, which does not participate in executing the distributed training task; the method further includes: When the spare node is set up to build an expansion node group, the t computing node groups are changed to the s computing node groups, and the number of computing nodes included in the equal node groups is changed from t to s. The s computing node groups include the expansion node groups, which are used to increase the parallelism of the distributed training system in executing the distributed training task. or, If the first computing node fails and there is no backup node to replace it, the computing node group to which the first computing node belongs is removed from participating in the distributed training task, the t computing node groups are changed to the s computing node groups, and the number of computing nodes included in the s node groups is changed from t to s.

15. The method according to any one of claims 1 to 4, characterized in that, The step of copying the sharded data stored on the first computing node to at least one target computing node corresponding to the first computing node for backup storage includes: The first computing node is controlled to use Remote Direct Memory Access (RDMA) technology to asynchronously copy the fragmented data stored on the first computing node to the memory of at least one target computing node corresponding to the first computing node for backup storage.

16. A data processing device for a distributed training system, characterized in that, The distributed training system includes t computing node groups, which are used to execute distributed training tasks in parallel. All computing nodes in the t computing node groups are divided into at least one equi-node group, each equi-node group containing t computing nodes from different computing node groups. The checkpoint data corresponding to each equi-node group is divided into t data shards, which are unique. Each of the t computing nodes in each equi-node group stores one shard from the t data shards. The checkpoint data is used to indicate the state of the t computing nodes in the equi-node group when participating in the distributed training task, where t is an integer greater than 1. The device includes: The target node determination module is used to determine, for the first computing node in the at least one equipotential node group, at least one computing node different from the first computing node in the distributed training system, as at least one target computing node corresponding to the first computing node. The sharded data backup module is used to copy the sharded data stored on the first computing node to at least one target computing node corresponding to the first computing node for backup storage.

17. A chip product, characterized in that, The chip product stores a computer program, which is loaded and executed by a processor to implement the data processing method for a distributed training system as described in any one of claims 1 to 15.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, which is loaded and executed by the processor to implement the data processing method for a distributed training system as described in any one of claims 1 to 15.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the data processing method for a distributed training system as described in any one of claims 1 to 15.

20. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and a processor reads from and executes the computer program to implement the data processing method for a distributed training system as described in any one of claims 1 to 15.

Images