Fault-tolerant elastic training method and system supporting cross-parallel strategy efficient storage and reading

Abstract

The invention discloses a fault-tolerant elastic training method and system supporting cross-parallel strategy efficient storage and reading, and belongs to the field of artificial intelligence. The method comprises the steps that when it is detected that the number of available GPUs changes, the total number of the current available GPUs is determined; segmenting a target model based on the total number of the currently available GPUs, so that each GPU runs a corresponding model slice during training; and reading the parameter slices in the nodes to recover the check point parameters, and restarting the training task after initializing each model slice based on the check point parameters. According to the method, quick restart after part of equipment fails in the training process can be effectively realized, and computing resources can be fully utilized under the condition of ensuring the training precision.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a fault-tolerant elastic training method and system that supports efficient storage and reading across parallel strategies. Background Art

[0002] With the development of large-scale model training and the increase in the number of model parameters, a single graphics card cannot store all model parameters. Distributed deep learning has become an indispensable part of large-scale model training. When using large-scale clusters for model training, GPU / node failures are also a common problem. Therefore, it is necessary to regularly store the current model parameters during training to avoid data loss caused by failures.

[0003] (1) Existing distributed training model segmentation strategy.

[0004] Tensor Parallel (TP): Splits the parameters of the fully connected layers and attention layers in the Transformer (including model weights and activation values) across multiple devices for each parameter. Through the characteristics of block matrix multiplication, each device performs calculations independently, and introduces an additional communication operator (AllReduce) to synchronize the results and ensure computational consistency.

[0005] Pipeline Parallel (PP): This splits the model layers, using the Transformer layer as the basic unit to split the model across multiple devices. Each device is responsible for computing several layers, and additional communication operators (P2P Send / Recv) are introduced to complete the transmission of intermediate calculation results between devices.

[0006] ZeRO: Evenly distributes model weights and optimizer state across devices, retrieving the full set of parameters from other devices via a communication operator (AllGather) when needed. It consists of three levels, with progressively more parts distributed as the level increases. ZeRO1: Only distributes the optimizer state (mean and variance in the Adam optimizer). ZeRO2: In addition to ZeRO1, it also distributes the backpropagation gradient. ZeRO3: In addition to ZeRO2, it also distributes the model parameters.

[0007] Hybrid parallelism: Because different parallelization strategies have varying impacts on computing, communication, and video memory, hybrid parallelism is often combined during training to achieve better training efficiency. Existing distributed deep learning frameworks, such as Megatron and Torch Titan, generally support a hybrid of these strategies.

[0008] (2) Existing training fault tolerance schemes.

[0009] Distributed model storage: Stores the model parameters and optimizer state of the current device to disk during model training. If model training is interrupted for some reason (power outage or device damage), the next time training is restarted, it can be directly restored based on the model and optimizer state stored before the interruption.

[0010] Parallel strategy configuration after restart: The number of available devices in the cluster may change before and after restart. It is necessary to find a hybrid parallel strategy that adapts to the new cluster state.

[0011] Cross-strategy model loading: The model partitioning strategies before and after training restart may differ. If the strategy is the same, the corresponding device parameters can be directly loaded. If the strategy changes, the distributed storage parameters in the files corresponding to each device need to be spliced, and then partitioned according to the new parallel strategy to obtain the distributed parameters of each device under the new strategy and load them. This method mainly has two methods: offline processing and online processing. Offline processing generates a new distributed parameter file, while online processing performs all operations in memory and then directly loads.

[0012] (3) Existing model parameter storage medium.

[0013] Pickle: Python's pickle library is used to serialize and deserialize objects. It can convert complex data structures such as lists and dictionaries into byte streams for easy storage or transmission, and can be restored to their original state later. Many early models used this storage method, usually with the suffix .pt or .pth.

[0014] SafeTensors: SafeTensors is a new model weight storage format proposed by huggingface to address some security risks in traditional model storage formats (such as PyTorch's .pt or .pth files). Traditional model loading methods may inadvertently execute malicious code stored in the model file, which poses a threat to user security. SafeTensors avoids this risk by reading model weights in a way that does not involve code execution. And due to its optimized data structure and reading mechanism, SafeTensors can load large models faster than traditional methods, especially in resource-constrained environments.

[0015] However, the above-mentioned prior arts have the following shortcomings.

[0016] (1) The existing training system cannot fully utilize the computing power of the equipment when part of the cluster fails.

[0017] Existing training systems generally employ homogeneous model parallelism, meaning that the corresponding tensor parallelism, pipeline parallelism, and ZeRO partitioning dimensions are consistent across all devices in the cluster. To optimize communication efficiency, these different parallelization dimensions are often powers of 2. However, for clusters experiencing partial failures (e.g., a cluster with seven GPUs), traditional hybrid parallelism often fails to balance partitioning parameters, leading to training failures (or insufficient utilization of GPU computing power, resulting in low computational efficiency).

[0018] (2) The existing technology has a large overhead in training the fault tolerance part.

[0019] Fault-tolerant training requires continuous model storage during training. This involves transferring data from graphics memory to main memory and then to the file system. Existing systems lack performance optimization in this area, leading to additional time overhead for fault-tolerant training and impacting model training efficiency.

[0020] (3) The performance of existing technologies in cross-strategy reading is poor.

[0021] When performing data recovery across strategies, existing systems may need to switch strategies and read parameters from multiple different files and concatenate them (when the TP dimension decreases, such as from TP=8 to TP=4), which incurs additional memory copy overhead. Alternatively, they may read additional parameters (when the TP dimension increases, such as from TP=4 to TP=8) and then divide the parameters into the size required by the current device, which incurs redundant read overhead. In either case, there will be additional time overhead, and this additional overhead cannot be ignored when the model is large and the parallel strategy changes significantly. In addition, for distributed file systems, the reading efficiency of existing technologies is also limited and cannot fully utilize bandwidth. Summary of the Invention

[0022] The present invention discloses a fault-tolerant elastic training method and system that supports efficient storage and reading across parallel strategies, which can effectively achieve rapid restart after some equipment fails during training and can fully utilize computing resources while ensuring training accuracy.

[0023] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0024] A fault-tolerant elastic training method supporting efficient storage and reading across parallel strategies, the method comprising:

[0025] When a change in the number of available GPUs is detected, the total number of available GPUs is confirmed;

[0026] Slicing the target model based on the total number of currently available GPUs so that each GPU runs a corresponding model slice during training;

[0027] Read the parameter slice from the node to restore the checkpoint parameters, initialize each model slice based on the checkpoint parameters, and then restart the training task.

[0028] Furthermore, the process of detecting a decrease in the number of available GPUs includes:

[0029] A node is used as the master node to manage the process and store the IP address of each node; wherein the tasks in the management process include: monitoring the training task status of each GPU through a heartbeat detection mechanism;

[0030] When the heartbeat information of a training task is lost, the corresponding GPU is determined to be an unavailable GPU.

[0031] Furthermore, the target model is segmented based on the total number of currently available GPUs, including:

[0032] Based on the maximum tensor parallelism and the total number of currently available GPUs, the GPUs in the node device are divided into several TP groups; the number of GPUs in each TP group is an integer power of 2 and less than the maximum tensor parallelism;

[0033] Based on the maximum data parallelism value, each TP group is assigned to a different data parallel dimension, and the total number of devices in each data parallel dimension is kept as close as possible;

[0034] For each data parallel dimension, the target model layers are divided according to the number of GPUs in the TP group;

[0035] Load balancing among data parallel dimensions is achieved by assigning different numbers of micro-batches.

[0036] Furthermore, load balancing between data parallel dimensions is achieved by allocating different numbers of micro-batches, including:

[0037] Calculate the computational efficiency of each data parallel dimension Among them, LAYERS[j] represents the number of layers of the j-th PP Stage, SPLITS[j] represents the TP value corresponding to the PP Stage;

[0038] An integer linear programming algorithm is used to assign a local micro-batch number proportional to the computational efficiency CE to each data parallel dimension.

[0039] Furthermore, the process of storing the parameter slice in the node includes:

[0040] Each parameter is divided into global_splits parameter slices and an index is assigned to each parameter slice; wherein global_splits is not less than the minimum integer power of 2 of the total number of currently available GPUs;

[0041] Maintain a memory parameter buffer of the same size as the parameters on the GPU, and transfer the parameters from the video memory to the main memory just after the parameter slice is updated at the end of each global batch training;

[0042] Multi-process is used to implement the transfer of parameter slices from memory to the node's file system.

[0043] Furthermore, each parameter is divided into global_splits parameter slices, including:

[0044] Evenly divide the parameters in different data parallel dimensions to obtain the first parameter division result;

[0045] For each data parallel dimension, evenly split the first parameter segmentation result on different TP groups to obtain the second parameter segmentation result;

[0046] For each TP group, the data is divided equally according to the TP value of the TP group.

[0047] Furthermore, multiple processes are used to transfer parameter slices from memory to the node's file system, including:

[0048] At the beginning of training, all GPU parameters that need to be transmitted are scanned, and the total memory usage of the GPU parameters is calculated. When allocating the corresponding CPU area, a complete block of memory is allocated based on the total memory usage of the GPU parameters as shared memory. For each parameter, the offset and size of the parameter in the shared memory need to be recorded.

[0049] Maintain a storage process that is completely independent of the main process; when the storage process is first created, the shared memory path and various parameter information are provided. When storage is needed, the main process provides the storage information to the storage process, and the storage process reads data from the shared memory and stores the data on the disk.

[0050] Furthermore, after using multiple processes to implement the transfer of the file system from memory to the node, it also includes:

[0051] The parameter slice key-value pairs stored in each node are reordered based on the parameter slice indexes arranged from small to large, and the parameter slices with the same index are sorted from small to large according to the number of layers in the model.

[0052] A fault-tolerant and resilient training system that supports efficient storage and reading across parallel strategies, the system comprising:

[0053] A fault detection module is used to confirm the current total number of available GPUs when a change in the number of available GPUs is detected;

[0054] A strategy search module is configured to partition the target model based on the total number of currently available GPUs so that each GPU runs a corresponding model slice during training;

[0055] The training restart module is used to read the parameter slices in the node to restore the checkpoint parameters, initialize each model slice based on the checkpoint parameters, and then restart the training task.

[0056] A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement any of the above-mentioned fault-tolerant elastic training methods that support efficient storage and reading across parallel strategies.

[0057] Compared with the prior art, the present invention has at least the following beneficial effects.

[0058] 1) This paper proposes a flexible training system that supports automatic detection and restart when some GPUs / nodes fail. It can support more flexible model segmentation schemes while ensuring model training accuracy to cope with different cluster conditions and make full use of the available computing resources in the cluster.

[0059] 2) The present invention provides an efficient solution for automatically searching for heterogeneous model parallel strategies based on the available GPUs / nodes in the cluster, and can quickly update the strategy when the cluster environment changes.

[0060] 3) The present invention provides a set of slicing-based efficient real-time model parameter storage, dataset breakpoints, and cross-strategy storage and reading solutions that support different file systems.

[0061] 4) The system implemented by the present invention provides a system paradigm with three separate parts: data set processing, training, and storage. The performance of each part is optimized to maximize training efficiency and minimize the loss of training restarts. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a system overview diagram.

[0063] Figure 2 Schematic diagram of heterogeneous model parallelization.

[0064] Figure 3 This is a schematic diagram of distributed parameter storage and slicing. DETAILED DESCRIPTION

[0065] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0066] The present invention supports a fault-tolerant elastic training system that efficiently stores and reads across parallel strategies, such as Figure 1 As shown, it mainly includes two parts: process management and model training, and process management includes three modules: fault detection module, strategy search module and training restart module. Each part runs independently and is synchronously scheduled through shared memory and inter-process communication mechanism. Among them, the process management of the present invention is distributed process management, which adopts the Master-Worker architecture and communicates between Master-Workers. A master node serves as a management process, storing the IP address of each working node. The management process can start tasks for each node according to the IP address, including querying graphics card information, starting training scripts, generating files, etc. In addition, the management process also has the function of monitoring the status of training tasks through heartbeat information and providing key-value storage for information interaction for training tasks.

[0067] 1. Fault detection module.

[0068] The fault detection module is mainly used to detect restart conditions, that is, to monitor the training task status through heartbeat information and confirm the current total number of available GPUs when a change in the number of available GPUs is detected.

[0069] (1) The number of available GPUs decreases: Through the heartbeat detection mechanism, if there is a device failure and the current training cannot continue due to data loss, the management process will capture this information in time and end other ongoing tasks and restart.

[0070] (2) Increase in the number of available GPUs: The management process will regularly check the number of available GPUs in the current cluster. If a node or GPU is repaired and can be put back into use, the training will need to be restarted to improve training efficiency due to the increase in the number of available GPUs.

[0071] 2. Strategy search module.

[0072] When the number of available GPUs changes, this paper proposes a heterogeneous model parallel (HMP) sharding strategy that can perform heterogeneous model parallel sharding based on the new number of available GPUs. Compared with homogeneous model parallelism, this paper can support more complex model sharding and can use all devices for training in the event of partial device failure, without having to abandon some devices to meet the needs of homogeneous sharding.

[0073] Specifically, this strategy is based on and expands upon TP, PP, and ZeRO. Under this partitioning strategy, the TP and PP values ​​corresponding to each parameter are no longer fixed. TP can take values ​​of 1, 2, 4, or 8, and the PP stage is no longer evenly divided but can have any number of layers. A hybrid model parallel strategy can be expressed as follows:

[0074] HMP=[(SPLITS[0],LAYERS[0]),…,(SPLITS[k],LAYERS[k])]

[0075] Among them, LAYERS[i] represents the number of layers in the i-th PP Stage, SPLITS[i] represents the corresponding TP value, SPLITS[0]+…+SPLITS[k] is the total number of cards in this HMP strategy, and LAYERS[0]+…+LAYERS[k] is the total number of layers in the model.

[0076] Each data parallel dimension corresponds to an independent HMP strategy. For example, for a 32-layer model, the present invention can perform the following DP and HMP splits on a 7-card cluster: {DP=1, DP rank=0 HMP=[(1,5),(2,9),(4,18)]}, or Figure 2 As shown, {DP=2, DP rank=0HMP=[(1,11),(2,21)], DP rank=1HMP=[(4,32)]}.

[0077] In one embodiment, the present invention can also find a set of optimally efficient splittable solutions through greedy algorithms and integer linear programming.

[0078] 1) Get the maximum value of data parallelism and tensor parallelism.

[0079] When initializing elastic training, the present invention needs to obtain the maximum value DPmax of data parallelism (hereinafter referred to as DP) and the maximum value TPmax of tensor parallelism (hereinafter referred to as TP) to limit the size of the search space.

[0080] 2) TP group construction.

[0081] To improve communication efficiency, in most cases, this invention only considers intra-node TP groups, i.e., Tpmax <= 8. If TPmax = 8n (n > 1), n ​​adjacent nodes are merged into a single node. This invention then divides the devices on that node into multiple TP groups, with the number of GPUs in each TP group being an integer power of 2 and less than TPmax.

[0082] The following are three examples when TPmax=4 and the number of available GPU nodes is 8, 7, and 6 respectively:

[0083] Example 1: TPmax = 4, the included GPU number list = [0, 1, 2, 3, 4, 5, 6, 7], and the final TP group = [[0, 1, 2, 3], [4, 5, 6, 7]];

[0084] Example 2: TPmax = 4, the included GPU number list = [0, 1, 2, 3, 4, 5, 6], and the final TP group = [[0, 1, 2, 3], [4, 5], [6]];

[0085] Example 3: TPmax = 4, the included GPU number list = [0, 1, 2, 3, 4, 5], and the final TP group = [[0, 1, 2, 3], [4, 5]].

[0086] 3) TP group allocation.

[0087] After obtaining TP groups, the present invention assigns each TP group to a different data parallelism dimension (hereinafter referred to as a DP rank) to ensure that the total number of devices in each DP rank is as close as possible. The method is as follows: the number of GPUs in each TP group is recorded and sorted from largest to smallest, and then each TP group is assigned to the DP rank with the fewest devices. This greedy algorithm ensures that the number of devices in different DP ranks is as balanced as possible.

[0088] 4) Layer division.

[0089] For each TP group in a DP rank, we allocate layers proportionally based on the number of devices to balance the load between layers, resulting in a Hybrid Model Parallel (HMP) strategy for each DP rank. We use a simple integer linear programming algorithm to solve the layer partitioning problem. Here are some examples of layer partitioning, where STAGE_SIZE_LIST represents the number of devices in the TP group in the DP rank:

[0090] (a) STAGE_SIZE_LIST=[1,2,4], HMP=[(1,4),(2,9),(4,19)].

[0091] (b) STAGE_SIZE_LIST=[2,4], HMP=[(2,11),(4,21)].

[0092] 5) Micro batch allocation.

[0093] In one embodiment, the present invention further proposes to achieve load balancing among DP ranks by allocating different numbers of micro-batches.

[0094] Specifically, for each data parallel dimension and its corresponding strategy HMP = [(SPLITS[0], LAYERS[0]), ..., (SPLITS[k], LAYERS[k])], we can calculate the computational efficiency of this DP rank Then, each DP rank is assigned a local microbatch number (lmbn) proportional to the computational efficiency through the integer linear programming algorithm (ILP algorithm). The following is an example of allocating the number of microbatches for different data parallel dimensions based on HMP_LIST:

[0095] (a) DP = 2, total number of batches = 64.

[0096] HMP strategy (HMP_LIST) = [[(8, 32)], [(1, 4), (2, 9), (4, 19)]].

[0097] Compute capability list LBN_LIST = [35, 29].

[0098] (b) DP = 4, total number of batches = 64.

[0099] HMP_LIST=[[(8,32)],[(8,22)],[[(1,4),(2,9),(4,19)],[(2,11),(4,21)]].

[0100] LBN_LIST=[18,18,15,13].

[0101] Furthermore, for the case with fewer GPUs, layer partitioning and micro-batch allocation can be combined into one ILP problem to make the search more accurate.

[0102] In another embodiment, the policy search of the present invention also supports downward splitting of TP groups (8 to 4, 4 to 2, 2 to 1) to further balance the number of devices assigned to different DP ranks.

[0103] 3. Training restart module.

[0104] The training restart module is primarily used to launch a corresponding number of training tasks on each node based on node and available GPU information, with each task corresponding to an available GPU. Specifically, the training restart module first reads the checkpoint parameter slice data stored in the node file system, then initializes the model slice on each GPU based on the checkpoint parameter slice, and then restarts the training task. The checkpoint parameter slice is stored in the node's file system based on a distributed parameter storage strategy, a cross-strategy fast storage and read design, and parameter storage separation.

[0105] 1) Distributed parameter storage strategy.

[0106] The parameter splitting logic of heterogeneous model parallelism is as follows: when DP=m, for each parameter in the model, there is a corresponding tensor split list (hereinafter referred to as TP_LIST) = [TP[0], TP[1],…, TP[m-1]], TP[i] represents its TP value under the i-th data parallel dimension (hereinafter referred to as DP rank). The present invention first evenly splits the parameters at different DP ranks based on the ZeRO combined with TP, and then evenly splits them at each DP rank according to the TP value of the current rank. The specific parameter splitting diagram is shown in the figure below. Figure 3 shown.

[0107] 2) Cross-strategy fast storage and reading design.

[0108] Since different parameters correspond to different TP_LISTs, and the parallel strategies of heterogeneous models are complex and changeable, the system has specially designed a set of slice-based parameter storage strategies for efficient storage and reading of models. The core idea of ​​this solution is to divide the parameters on each GPU into slices of the same size, and under any strategy, the parameters of each GPU can be exactly spliced ​​together by an integer number of slices. The DP of the system of the present invention and the TP of each parameter are set to an integer power of 2, so the present invention can set a global split number (hereinafter represented by global_splits). Under given cluster conditions, the parameters on the GPU under any splitting strategy are integer multiples of the complete parameters 1 / (global_splits) (for example, for a cluster of 64 GPUs, the present invention is set to 64, and for a cluster of 240 GPUs, the present invention is set to 256). Before storing the parameters in the file system, each device first splits them into the size of the complete parameters 1 / (global_splits). This operation does not affect the total number of parameters stored in the model, but can ensure that each device does not read redundant parameters and perform slicing operations when reading, which will bring additional overhead. Under this storage strategy, each parameter is divided into global_splits slices, whose indexes are 0, 1, 2, ...global_splits-1, and distributed in the files corresponding to each device.

[0109] Considering the potential overhead of a single IO access in a distributed file system, this paper reorders the key-value pairs of parameters stored on each device, prioritizing them by their corresponding index from smallest to largest (0 to global_splits-1). For parameters with the same index, they are sorted by their number of layers in the model from smallest to largest. This ensures that when reading across policies, parameter slices can be read as seamlessly as possible, reducing IO times and the corresponding overhead.

[0110] 3) Parameter storage separation.

[0111] The storage of model parameters for each round includes two steps: from GPU memory to main memory, and from main memory to file system. The present invention hopes to minimize the impact on the training process time while ensuring that these two steps are carried out normally and without loss of accuracy.

[0112] (1)GPU memory->memory

[0113] Each time the data is stored, the present invention needs to transfer the data from the video memory to the internal memory. Although the time of a single transfer is short (generally within 5 seconds), the need to continuously store the parameters will waste a lot of time in long-term training, so this part of the overhead needs to be optimized.

[0114] The present invention has noticed that as the model grows, the time consumption for training the same number of data will become longer and longer, but the video memory of a single GPU has an upper limit, that is, the time for a single GPU to transfer model parameters to the memory has an upper limit. The present invention has found through experiments that in the process of model training, for a global batch that reaches a certain size, the overhead of transferring parameters from the GPU video memory to the memory will be less than the interval for updating the model parameters. Therefore, the system of the present invention maintains a memory parameter buffer of the same size as the parameters on the GPU (including model parameters and optimizer state), and transfers the video memory to the memory just after the update of the parameters at the end of each global batch training. This process will be completed before the parameters are updated and will not affect the training efficiency. In order to achieve this on the system, the present invention needs to allocate page-locked memory and use different computing flows to handle transmission and calculation respectively.

[0115] (2) Memory -> File System

[0116] Because the storage of the present invention is based on safetensors, the serialization process of safetensors triggers the global interpreter lock, which affects the efficiency of Python's multithreading. The present invention uses multiple processes to achieve this separation. And shared memory is needed to transfer data between multiple processes.

[0117] First, at the beginning of training, the system scans all GPU parameters that need to be transferred and calculates their total memory usage. When allocating the corresponding CPU area, it allocates a complete block of memory based on the total memory usage. For each parameter, its offset and size in the shared memory must be recorded for cross-process reading.

[0118] A storage process is also maintained. When it is first created, the shared memory path (under / dev / shm) and information about each parameter (parameter shape, data type, offset in shared memory, etc.) are provided. When storage is needed, the main process provides some information to be stored (number of training steps, number of trained data) to the storage process. The storage process reads data from the shared memory and stores it on disk. This process is completely independent of the main process and therefore does not affect training efficiency.

[0119] Through the above two optimizations, the system completes the separation of the training main process and the storage process. It will only occupy the training time of the main process during the first initialization. In other cases, it is completely independent of the main process and will not affect the training in any way.

[0120] Furthermore, to further reduce overhead, the present invention employs a data preprocessing separation strategy for the data preprocessing of the model training portion. Specifically, the system supports customizing the preprocessing logic for each piece of data and provides a buffer shared between processes (the buffer size is generally set to 8 times the size of each global batch). The time required to train a global batch of data is much longer than the time it takes to preprocess a global batch on the CPU. Therefore, during the training startup phase, the present invention simultaneously initiates a data preprocessing process to preprocess the data corresponding to the subsequent global batch and place it in a shared buffer. The training process can then directly retrieve data from this buffer, saving this portion of overhead.

[0121] The following experiments illustrate the accuracy correctness of the parallel verification of heterogeneous models, the consistency of accuracy before and after fault-tolerant recovery, and the efficiency of cross-strategy storage and reading of model parameters of the present invention.

[0122] This paper uses Llama2-7B, Llama2-13B, Llama2-34B, and Llama2-70B as comparison models, and conducts experiments on a 32-card NVIDIA H100-80GB GPU cluster. The accuracy correctness experiment uses the Llama2-7B model, and the cross-strategy storage and read model parameter performance experiment uses the Llama2-7B, Llama2-13B, Llama2-34B, and Llama2-70B models.

[0123] The accuracy correctness experiment of heterogeneous model parallelism tests the loss reduction of the same model parameters and the same number of training steps under 8, 7, 6, and 5 GPUs respectively.

[0124] The consistency experiment of accuracy before and after fault-tolerance recovery tested the loss convergence curve of the model in the scenario of fault-tolerance recovery with a change in the number of GPUs (8->7->6) and the scenario where the number of GPUs remains consistent (8).

[0125] The cross-strategy model parameter storage and reading performance experiment tested the time overhead of storing models and loading models across strategies under different numbers of GPUs and different models.

[0126] Experimental Comparison Results: Experimental results show that when loading the same model parameters, the loss values ​​for each training step are consistent across different GPU counts, demonstrating the feasibility of heterogeneous model parallelism. The loss curves for restarting training with varying GPU counts and continuing training are also completely consistent, demonstrating the correctness of the system's model storage and loading logic and its lack of negative impact on model convergence. The performance of cross-strategy model parameter storage and reading demonstrates that the system maintains high training recovery efficiency across various cluster scenarios.

[0127] In summary, the present invention proposes a new fault-tolerant elastic training system that supports efficient storage and reading across parallel strategies. Compared with traditional distributed training systems, this system can support more complex heterogeneous model parallel strategies on the basis of supporting conventional isomorphic model parallelism, and can better utilize computing resources for model training on clusters with partial failures. The distributed model parameter storage strategy provided by this system can also minimize the amount of data loaded and the number of loads on a single device after restart, and has high efficiency on various file systems. This system fully separates the parameter storage during fault-tolerant training from the training process, so that while ensuring the fault-tolerant function, it does not affect the efficiency of the training process.

[0128] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A fault-tolerant elastic training method that supports efficient storage and reading across parallel strategies, characterized in that: The method comprises: When a change in the number of available GPUs is detected, the total number of available GPUs is confirmed; Slicing the target model based on the total number of currently available GPUs so that each GPU runs a corresponding model slice during training; Read the parameter slice from the node to restore the checkpoint parameters, initialize each model slice based on the checkpoint parameters, and then restart the training task.

2. The method according to claim 1, characterized in that The process of detecting a decrease in the number of available GPUs includes: A node is used as the master node to manage the process and store the IP address of each node; wherein the tasks in the management process include: monitoring the training task status of each GPU through a heartbeat detection mechanism; When the heartbeat information of a training task is lost, the corresponding GPU is determined to be an unavailable GPU.

3. The method according to claim 1, characterized in that The target model is segmented based on the total number of currently available GPUs, including: Based on the maximum tensor parallelism and the total number of currently available GPUs, the GPUs in the node device are divided into several TP groups; the number of GPUs in each TP group is an integer power of 2 and less than the maximum tensor parallelism; Based on the maximum data parallelism value, each TP group is assigned to a different data parallel dimension, and the total number of devices in each data parallel dimension is kept as close as possible; For each data parallel dimension, the target model layers are divided according to the number of GPUs in the TP group; Load balancing among data parallel dimensions is achieved by assigning different numbers of micro-batches.

4. The method according to claim 3, characterized in that Load balancing across data parallel dimensions is achieved by allocating different numbers of micro-batches, including: Calculate the computational efficiency of each data parallel dimension Among them, LAYERS[j] represents the number of layers of the j-th PPStage, SPLITS[j] represents the TP value corresponding to the PP Stage; An integer linear programming algorithm is used to assign a local micro-batch number proportional to the computational efficiency CE to each data parallel dimension.

5. The method according to claim 3, characterized in that The process of storing the parameter slice in the node includes: Each parameter is divided into global_splits parameter slices and an index is assigned to each parameter slice; wherein global_splits is not less than the minimum integer power of 2 of the total number of currently available GPUs; Maintain a memory parameter buffer of the same size as the parameters on the GPU, and transfer the parameters from the video memory to the main memory just after the parameter slice is updated at the end of each global batch training; Multi-process is used to implement the transfer of parameter slices from memory to the node's file system.

6. The method according to claim 5, characterized in that Each parameter is divided into global_splits parameter slices, including: Evenly divide the parameters in different data parallel dimensions to obtain the first parameter division result; For each data parallel dimension, evenly split the first parameter segmentation result on different TP groups to obtain the second parameter segmentation result; For each TP group, the data is divided equally according to the TP value of the TP group.

7. The method according to claim 5, characterized in that Multiple processes are used to transfer parameter slices from memory to the node's file system, including: At the beginning of training, all GPU parameters that need to be transmitted are scanned, and the total memory usage of the GPU parameters is calculated. When allocating the corresponding CPU area, a complete block of memory is allocated based on the total memory usage of the GPU parameters as shared memory. For each parameter, the offset and size of the parameter in the shared memory need to be recorded. Maintain a storage process that is completely independent of the main process; when the storage process is first created, the shared memory path and various parameter information are provided. When storage is needed, the main process provides the storage information to the storage process, and the storage process reads data from the shared memory and stores the data on the disk.

8. The method according to claim 5, characterized in that After using multiple processes to implement the transfer of the file system from memory to the node, it also includes: The parameter slice key-value pairs stored in each node are reordered based on the parameter slice indexes arranged from small to large, and the parameter slices with the same index are sorted from small to large according to the number of layers in the model.

9. A fault-tolerant elastic training system that supports efficient storage and reading across parallel strategies, characterized in that: The system comprises: A fault detection module is used to confirm the current total number of available GPUs when a change in the number of available GPUs is detected; A strategy search module is configured to partition the target model based on the total number of currently available GPUs so that each GPU runs a corresponding model slice during training; The training restart module is used to read the parameter slices in the node to restore the checkpoint parameters, initialize each model slice based on the checkpoint parameters, and then restart the training task.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the fault-tolerant elastic training method that supports efficient storage and reading across parallel strategies as described in any one of claims 1 to 8.

Images