A large model training method and system supporting parallel hot switching

Abstract

The application discloses a large model training method and system supporting parallel hot switching, and belongs to the technical field of large model training. The method comprises the following steps: generating a logical calculation graph, wherein the logical calculation graph is used for representing a plurality of different parallel strategy combinations; compiling the logical calculation graph to generate a plurality of executable calculation graphs; wherein each executable calculation graph corresponds to a group of parallel strategy combinations; selecting an initialization graph for initializing a large model state from the plurality of executable calculation graphs, and arranging the execution order between the executable calculation graphs according to the hot switching cost between any two groups of parallel strategy combinations; and training the large model based on the initialization graph and the execution order between the executable calculation graphs. The application not only can ensure that the sequence calculation amount / workload in each group is approximately close, but also can enable the parallel strategy to be dynamically switched in the model training process and normally complete the gradient accumulation and update of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model training, and particularly relates to a large model training method and system supporting parallel hot switching. BACKGROUND

[0002] In recent years, large language models (LLM) represented by ChatGPT have attracted widespread attention. The performance improvement of LLM is due to the increase in model size, context, and data volume. For example, LLaMA3, an open-source LLM, has a maximum of 400 billion parameters, a context length of 128k, and a data volume of an astonishing 1.5 trillion tokens. However, these increases also pose more challenges to system optimization. On the one hand, when encountering larger model sizes or longer contexts, appropriate parallel strategies need to be used to balance memory usage and computing efficiency. On the other hand, the sequence length of real data is usually dynamic and mostly follows a long-tail distribution, so appropriate strategies are needed to handle dynamic sequences to avoid wasting computing resources.

[0003] When facing larger model sizes or longer contexts, existing technical solutions include Megatron-LM's hybrid parallel solution and DeepSpeed's Zero parallel + Ulysses solution.

[0004] Megatron-LM's hybrid parallel solution mainly includes data parallelism, sequence parallelism, model parallelism, and hybrid parallelism.

[0005] Data parallelism involves dividing a large amount of input data into multiple parts, with each device only responsible for processing part of the data. Additional communication costs are introduced between different devices to synchronize gradients. Under data parallelism, each device will have a complete set of model parameters. When memory is insufficient, the model needs to be further divided.

[0006] Sequence parallelism involves splitting a single sample data into multiple parts in the sequence dimension when the data is particularly long. Each device only processes a small part of the sequence slice. Additional communication (All-Gather + Reduce-Scatter / P2P) is introduced between different devices to obtain the overall sequence information. For example, Megatron-LM's Sequence Parallel and Context Parallel.

[0007] Model parallelism splits the model into multiple parts, so that each device only keeps part of the model parameters. According to the different dimensions of the split, model parallelism is divided into tensor parallelism and pipeline parallelism. Tensor parallelism mainly does the split of intra-layer parameters, and splits the fully connected layer and attention layer in the Transformer vertically to multiple devices, and introduces an additional communication operator (All-Reduce) to ensure calculation consistency. Pipeline parallelism: is to do the split of inter-layer parameters, to take the TransformerLayer as the basic unit to split the model horizontally to multiple devices, and each device is responsible for the calculation of part of the model layer, and introduces an additional communication operator (P2P) to complete the transmission of the intermediate calculation results.

[0008] Mixed parallelism is a combination of different parallel strategies that affect computation, communication, and memory, to achieve better training efficiency.

[0009] wherein, Figure 1 is an example diagram of the mixed parallelism scheme of Megatron-LM, which includes data parallelism, tensor parallelism and pipeline parallelism.

[0010] The Zero parallelism + Ulysses scheme of DeepSpeed mainly includes: Zero parallelism and sequence parallelism.

[0011] Zero parallelism also splits data and model parameters on different devices, but follows the data parallelism computing paradigm as a whole. In order to ensure the consistency of the calculation, additional communication (All-Gather+Reduce-Scatter) is required for the model parameters.

[0012] Sequence parallelism is to use the Ulysses scheme of DeepSpeed, which obtains the overall sequence information by using the communication operator (All-to-All) on the split sequence.

[0013] wherein, Figure 2 is an example diagram of the Zero parallelism + Ulysses scheme.

[0014] In the face of data sets with dynamically changing sequence length, the existing processing schemes are sequence padding (SequencePadding) and sequence packing (SequencePacking). Sequence padding uses special tokens to fill each unequal length data to the maximum length, but this will cause a lot of unnecessary redundant calculation. Sequence packing is to splice sequences of different lengths to a length close to the maximum length, and use special masks to ensure that the calculations between different sequences do not affect each other, thereby avoiding calculation waste.

[0015] wherein,Figure 3 is an example diagram of sequence padding and sequence packing for a dynamic sequence.

[0016] However, existing static parallel strategy combinations are not efficient for dynamic scenarios with varying sequence lengths. This is because the choice of parallel strategies such as data parallelism, sequence parallelism, and model parallelism results in different memory consumption, communication overhead, and execution efficiency when training large-scale Transformer models in a distributed manner. Existing systems such as Megatron-LM and DeepSpeed usually assume that the training load is constant, so they are designed to only use one fixed / static parallel strategy combination throughout the entire training process. However, existing datasets are mostly long-tailed distributed, and when encountering such dynamic scenarios with varying sequence lengths, the optimal parallel strategy required for different sequences may be different, and static parallel strategies may not be the optimal solution.

[0017] In addition, sequence padding can cause serious computational waste, and sequence packing not only causes computational imbalance, but also further causes parallel strategy imbalance.

[0018] For example, for scenarios with varying sequence lengths, sequence packing is usually used to avoid computational waste. At this time, the computation time of each sequence can be modeled as time seq =a·s 2 +b·s, and the time of each group of data can be modeled as time micro batch =a·∑s i 2 +b·∑s i =a·∑s i 2 +b·S. Where a, b, and S are constants, a and b are coefficients, and S is the maximum context length of the model, i.e., the length after packing; s is a variable corresponding to the sequence length of each data.

[0019] As can be seen from Figure 4 , for a complete 32k sequence, the computation is proportional to (32k)^2, and for 32 sequences of 1k packed together, the computation is proportional to 32*(1k)^2, which is obviously imbalanced in computation. As can be seen from Figure 5 , computational imbalance further leads to imbalance in data parallelism, model parallelism, and pipeline parallelism, further slowing down the overall execution efficiency. SUMMARY

[0020] In view of the above deficiencies of the prior art, the present application provides a large model training method and system supporting parallel hot switching, which can not only ensure that the sequence calculation amount / workload in each group is approximately close, but also enable the parallel strategy to be dynamically switched during the model training process and normally complete the gradient accumulation and update of the model.

[0021] To achieve the above object, the technical scheme of the present application includes the following contents.

[0022] A large model training method supporting parallel hot switching, the method comprising:

[0023] generating a logical computation graph, the logical computation graph being used to represent a plurality of different parallel strategy combinations;

[0024] compiling the logical computation graph to generate a plurality of executable computation graphs; wherein each executable computation graph corresponds to a group of parallel strategy combinations;

[0025] selecting an initialization graph for initializing the state of the large model from the plurality of executable computation graphs, and arranging the execution order between the executable computation graphs according to the hot switching cost between any two groups of parallel strategy combinations;

[0026] training the large model based on the initialization graph and the execution order between the executable computation graphs.

[0027] Further, the generating a logical computation graph comprises:

[0028] obtaining parallel strategies and grouping the parallel strategies according to the data sequence length processed by the parallel strategies;

[0029] binding the parameters and input variables of each group of parallel strategy combinations to a plurality of distributed states;

[0030] under a distributed tensor architecture, deducing the plurality of distributed states, and processing the differences in the intermediate operators during the deduction processes of different parallel strategies by inserting a fake operator, so as to represent different parallel strategies using one logical computation graph.

[0031] Further, the compiling the logical computation graph comprises:

[0032] inserting an fp32->bf16 type conversion operator and a gradient accumulation operator in the logical computation graph;

[0033] and / or,

[0034] merging adjacent communication operators and fusion operators;

[0035] and / or,

[0036] unnecessary pseudo-operators and type conversion operators, gradient accumulation operators, and gradient communication operators are pruned;

[0037] and / or,

[0038] all communication operators are replaced with set communication operators or point-to-point communication operators, and corresponding set communication operators or point-to-point communication operators are generated;

[0039] and / or,

[0040] the local topology of the logical computation graph is rearranged to achieve overlap of computation and communication.

[0041] Further, the initialization graph for initializing the state of the large model is selected from the plurality of executable computation graphs, including:

[0042] obtaining model splitting granularity corresponding to each group of parallel strategy combinations;

[0043] the executable computation graph corresponding to the parallel strategy combination with the smallest model splitting granularity is taken as the initialization graph for initializing the state of the large model.

[0044] Further, the execution order between the executable computation graphs is arranged according to the hot switching cost between any two groups of parallel strategy combinations, including:

[0045] grouping data in a global batch according to sequence length to correspond to each group of parallel strategy combinations;

[0046] computing the hot switching cost L = sum(V i (inter) ) / B (inter) + sum(V i (intra) ) / B (intra) between any two groups of parallel strategy combinations on the grouped data; wherein V i (inter) represents inter-machine communication volume, B (inter) represents inter-machine communication bandwidth, V i (intra) represents intra-machine communication volume, and B (intra) represents intra-machine communication bandwidth.

[0047] The execution order between the executable computation graphs is obtained by combining the hot switching cost between all parallel strategy combinations with the minimum hot switching overhead; wherein the hot switching overhead is obtained by optimizing communication overhead and memory copy overhead based on message merging and layout optimization.

[0048] Further, the hot switching cost between any two groups of parallel strategy combinations is computed on the grouped data, including:

[0049] For each device i, record the internal communication bandwidth B. (intra) Inter-machine communication bandwidth B (inter) And the current internal communication volume V i (intra) Inter-machine communication volume V i (inter) ;

[0050] Iterate through each model parameter / gradient slice and determine the set of devices S that own the slice and the set of devices D that need the model parameter / gradient slice based on the distributed state of the current policy and the target policy;

[0051] Iterate through each device dst in the device set D, and divide the devices in the device set S into intra-device devices S based on the differences between intra-device and inter-device devices. (intra) and machine room equipment S (inter) ;

[0052] In-machine equipment S (intra) If the model parameters / gradient slices required by device dst are available, then the on-machine device S will be used. (intra) As the sender of the model parameters / gradient slices, and performing intra-machine communication V i (intra) Update;

[0053] In-machine equipment S (intra) If the model parameters / gradient slices required by device dst are not available, then in the inter-machine device S (inter) Select the inter-machine device S with the smallest current data transmission volume. (inter) As the sender of the model parameters / gradient slices, and performing inter-machine communication V i (inter) Update.

[0054] Furthermore, the message merging process includes:

[0055] Data sent to the same device is merged into consecutive send buffers to generate merged send data;

[0056] and,

[0057] Data received from the same device is merged into consecutive receive buffers to generate merged receive data.

[0058] and,

[0059] The BatchedSendRecv primitive supports parallel transmission of merged send and receive data.

[0060] Furthermore, the layout optimization process includes:

[0061] The vertically segmented model parameters / gradient slices are arranged horizontally in the layout, and mathematical equivalence is ensured by changing the layout parameters of the general matrix multiplication kernel.

[0062] Furthermore, the training of the large model based on the execution order between the initialization graph and the executable computation graph includes:

[0063] The data within the global batch is grouped according to sequence length and bound to the corresponding executable computation graph. j The above is used to obtain data from each group. j Where j∈[1,n], and n represents the total number of executable computation graphs;

[0064] The model parameters and optimizer are initialized based on the initialization diagram, resulting in the initialized model parameters θ. t＝0 Where t is the number of iterations;

[0065] model parameters θ t and the current cumulative gradient in the t-th training round From the executable computation graph j-1 Hot-switch to executable computation graph j In, and based on the data j In the executable computation graph j During training, the cumulative gradient is obtained.

[0066] Based on cumulative gradient For model parameters θ t Update the model to obtain the model parameters θ. t+1 ;

[0067] Based on the model parameter θ t+1 If the large model does not meet the set requirements, and the number of iterations t+1 does not exceed the maximum number of iterations, let t = t+1, and re-execute the step of adjusting the model parameters θ. t and the current cumulative gradient in the t-th training round From the executable computation graph j-1 Hot-switch to executable computation graph j middle;

[0068] Based on the model parameter θ t+1 If the large model meets the set requirements, or if the number of iterations t+1 exceeds the maximum number of iterations, the output is based on the model parameters θ. t+1 The large model.

[0069] A large model training system supporting parallel hot switching, the system comprising:

[0070] a generating module configured to generate a logical computation graph, the logical computation graph being used to represent a plurality of different parallel strategy combinations;

[0071] a compiling module configured to compile the logical computation graph to generate a plurality of executable computation graphs; wherein each executable computation graph corresponds to a parallel strategy combination;

[0072] a deriving module configured to select an initialization graph for initializing a state of a large model from the plurality of executable computation graphs, and to arrange an execution order between the executable computation graphs according to a hot switching cost between any two parallel strategy combinations;

[0073] a training module configured to train the large model based on the initialization graph and the execution order between the executable computation graphs.

[0074] Compared with the prior art, the present application has at least the following beneficial effects.

[0075] 1) The training paradigm of parallel hot switching is first proposed: the present application proposes a brand-new training paradigm based on parallel hot switching. For the data of each global batch, the present application groups them according to the difference in their computation amount / workload, and uses the most suitable parallel strategy for each group. In any two strategy groups, the system of the present application automatically, efficiently and imperceptibly converts the model parameters and gradients, and completes the gradient accumulation between different strategies before the model is updated, so as to ensure that the training effect is not affected.

[0076] 2) Unified computation graph representation and compilation: existing systems, such as Megatron-LM and DeepSpeed, due to their complex system design, only support a fixed parallel strategy combination during training, so they cannot use different parallel strategies for different sequence loads. The present application designs a special graph compiler, which can represent a plurality of different parallel strategy combinations with a unified logical graph, and further compiles to generate a plurality of corresponding executable computation graphs, and shares the storage of the model state, so as to support complex parallel hot switching semantics.

[0077] 3) Parallel hot switching technology: given any two different parallel strategy combinations, the hot switching between them needs to exchange the parameters and gradients of the model in different devices, which inevitably introduces additional communication overhead. In order to solve this problem, the present application designs a hot switching planner, proposes a heuristic algorithm to find the optimal communication scheme between any two strategy groups, and introduces a series of optimization techniques for communication and memory copying to further reduce the switching overhead.

[0078] 4) Based on Graph Compiler and Hot Switch Planner, the present application realizes the HotSPa system. The HotSPa system supports a brand-new training paradigm of parallel hot switching. Compared with existing systems, an acceleration ratio of up to 2.99x can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 An example diagram for the hybrid parallel scheme of Megatron-LM.

[0080] Figure 2 An example diagram for the Zero parallel + Ulysses scheme.

[0081] Figure 3 An example diagram for sequence padding and sequence splicing on a dynamic sequence.

[0082] Figure 4 An example diagram for the same data amount but different calculation amount comparison under sequence splicing.

[0083] Figure 5 An example diagram for data parallelism, pipeline parallelism imbalance under sequence splicing.

[0084] Figure 6 An overview diagram of an efficient large model training system supporting parallel hot switching.

[0085] Figure 7 A schematic diagram for representing parallel strategies with DStates.

[0086] Figure 8 An example diagram for representing different parallel strategy combinations with unified logical diagrams.

[0087] Figure 9 An example diagram for converting logical diagrams into executable diagrams.

[0088] Figure 10 A schematic diagram for arranging executable diagrams and hot switching processes.

[0089] Figure 11 An example diagram for modeling the basic unit of communication of hot switching.

[0090] Figure 12 An example diagram for modeling hot switching.

[0091] Figure 13 An illustration of the hot switching algorithm.

[0092] Figure 14 An example diagram for the message merging optimization technique.

[0093] Figure 15 An example diagram for the layout optimization technique.

[0094] Figure 16 Data length distribution of the two datasets used (CommonCrawl and GitHub). DETAILED DESCRIPTION

[0095] The specific embodiments of the present application will be further described below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application, but are not used to limit the scope of the present application.

[0096] The technical solution proposed by the present application is shown in Figure 6 mainly divided into a compilation phase and a training phase.

[0097] I. Compilation phase

[0098] The compilation phase is mainly used for derivation, arrangement and initialization. The compilation phase is based on two core technologies, a hot switch planner (Hot Switch Planner) to derive the optimal communication scheme and the corresponding communication cost between any two groups of strategy combinations; and a graph compiler (Graph Compiler) to compile and generate a plurality of corresponding executable computing graphs according to a plurality of strategy combinations represented by a user-defined logic graph, arrange the execution order of these computing graphs according to the communication cost provided by the hot switch planner, and select an initialization graph for initializing the model state.

[0099] 1. Graph compiler (Graph Compiler)

[0100] The graph compiler (Graph Compiler) supports representing a plurality of different parallel strategy combinations simultaneously with a unified logic graph, and further compiles and generates a plurality of corresponding executable computing graphs, and shares the storage of the model state. Corresponding to three core steps / innovations:

[0101] (1) Logic computing graph (Logic Graph): A unified logic graph is used to represent a plurality of different parallel strategy combinations.

[0102] As shown in Figure 7As shown, the present application is based on a DTensor (Distributed Tensor) architecture, and uses DStates (Distributed States) to represent a set of parallel strategy combinations: Splits, Partial, and Duplicate are used to represent the splitting manner of parameters in different devices (where Splits represents the number of splits of a tensor in different dimensions, and each device has one of the slices; Partial represents that different devices have partial information of the same tensor; Duplicate represents that different devices repeatedly have the same tensor, i.e., no splitting is performed), and DeviceGroup is used to represent the mapping relationship between different parameter slices and devices.

[0103] In order to simultaneously represent multiple sets of distributed strategies, the present application innovatively binds one DTensor with multiple sets of DStates, and supports the derivation of multiple sets of distributed states on the entire logical computation graph.

[0104] Specifically, each parameter and input variable is simultaneously bound with multiple sets of DStates, and during the construction of the logical computation graph, each operator simultaneously derives the multiple sets of DStates, and automatically inserts intermediate operators to ensure the integrity of the functions.

[0105] Due to different parallel strategy combinations, the computation graphs obtained by deriving the DStates are not necessarily the same (for example, (a) in Figure 8 In order to be able to represent the multiple strategies with the same logical computation graph, the differences in the communication and other intermediate operators of different strategies are handled by automatically inserting dummy operators (for example, (b) in Figure 8 , so that the two computation graphs are uniformly represented by the same logical computation graph (for example, (c) in Figure 8 ). The specific explanations are as follows:

[0106] For example, taking the two strategies TP2 and DP2 represented in (a) in Figure 8 : TP2 represents splitting the model weights, so the intermediate result y obtained by preliminary calculation is also in a split state, and an additional communication operator comm needs to be introduced to obtain the complete result y; DP2 represents that the model weights are not split, so the complete result y can be directly calculated. Obviously, the computation graphs corresponding to TP2 and DP2 are not equal.

[0107] • In order to be able to merge the two computational graphs representing TP2 and DP2 in (a), a dummy operator is inserted in the computational graph of DP2, which is only used as a placeholder and has no computational meaning. At this time, in (b), the representation of the two computational graphs of TP2 and DP2 is consistent regardless of the splitting state.

[0108] • Further, due to the DTensor architecture, the splitting state DStates is bound to the tensors in the computational graph, so the present application can bind multiple sets of splitting states to the same tensor at the same time, thereby obtaining the unified logical graph in (c).

[0109] Through the above scheme, the present work supports the use of a unified logical graph to represent multiple sets of different parallel strategy combinations.

[0110] (2) Executable graphs (Exec Graphs): based on the unified logical graph, multiple sets of executable computational graphs are compiled and generated, each set of executable computational graphs corresponding to a set of independent parallel strategy combinations, which share the same model state storage.

[0111] The logical graph is only an abstract representation, in order to be able to compile and generate truly executable distributed computational graphs, the compiler will perform operations such as insertion, merging, pruning, replacement, rearrangement, etc. of operators.

[0112] • Insert type conversion operators, gradient accumulation operators.

[0113] • Merge adjacent communication operators (comm op) and fusion operators (fusion op).

[0114] • Prune unnecessary dummy operators (dummy op) and type conversion, gradient accumulation, gradient communication operators.

[0115] • Replace all communication operators (comm op) with collective communication operators or point-to-point communication operators, and generate corresponding collective communication operators or point-to-point communication operators.

[0116] • Rearrange the local topology of the computational graph to achieve overlap of computation and communication.

[0117] Figure 9 A specific example of converting a logical graph into two sets of executable graphs (two sets of parallel strategy combinations) is given. Among them, Figure 9 The top left corner in (a) is the logical computational graph, and the bottom left and right sides are the compilation and generation of two sets of executable computational graphs. The main difference here is the replacement of the communication operator in the step of generating executable computational Figure 1Simply replace the communication operator (composp) with the set communication operator All-Gather; and an executable computation will be generated. Figure 2 This requires replacing the communication operator (com op) with a more complex point-to-point communication operator (batch send recv op), as well as a split operator (split op) that processes the input of the communication and a concatenate operator (concatenate op) that processes the output of the communication.

[0118] (3) Orchestrate Exec Graphs: Based on the cost analysis provided by Hot Switch Planner, orchestrate the execution order of these executable graphs (a combination of parallel strategies) to minimize storage and communication overhead.

[0119] Initialization graph selection: Select the executable graph corresponding to the strategy that minimizes the storage of model states as the initialization graph. (Select the strategy that minimizes the granularity of model partitioning.) Figure 10 For example, strategies TP2 and PP2, which completely split the model, are selected as the initialization graph, while strategies DP4 and DP2,TP2 have some parameters that are repeated between devices and are not split due to the presence of DP.

[0120] Reordering the executable graph: Based on the cost of hot switching between any two sets of policies given by the Hot Switch Planner, determine the execution order of different policies to minimize the overall hot switching cost. Figure 10 For example, in the TP2PP2->DP4->DP2TP2 switch, only the first hot switch requires additional communication overhead; the rest can be handled locally.

[0121] Pruning & Gradient Accumulation: In addition to the initialization graph, unnecessary type casting operations and parameter update operations need to be pruned in other executable graphs to ensure that model updates are performed only on the initialization graph and gradient accumulation is performed only on other executable graphs, thus ensuring that the accuracy of the model is not affected.

[0122] 2. Hot Switch Planner (HotSPa).

[0123] HotSPa supports grouping data in a global batch by sequence length, each group uses different parallel strategies (corresponding to different executable computation graphs), and the gradients of different strategies are accumulated into the same gradient buffer to ensure that the model convergence is not affected.

[0124] Between the two different parallel strategies, HotSPa automatically hot switches the model's weights and gradients, and the core function of the hot switch planner is to derive the communication scheme with the minimum switching cost between any two different parallel strategies. Specifically, it includes the following two core steps:

[0125] The hot switching cost is defined as the total time spent from the start to the end of the hot switching, which is determined by the sending and receiving time of the weights and gradients, corresponding to the intra-machine communication volume V i (intra) Divided by the intra-machine communication bandwidth B (intra) Plus the inter-machine communication volume V i (inter) Divided by the inter-machine communication bandwidth B (intra) That is, the hot switching cost L = sum(V i (inter) ) / B (inter) +sum(V i (intra) ) / B (intra) .

[0126] (1) Model Hot Switching based on heuristic algorithm

[0127] There are multiple feasible solutions for the communication scheme: hot switching refers to the switching of distributed states (from src strategy -> dst strategy), which requires re-partitioning of the model's parameters and gradients in the entire cluster. It is a complex communication with many-to-many, and due to the existence of data parallelism (DP), there are multiple copies of the same model slice in different devices, so there are a large number of feasible solutions for the communication scheme (the sender of the model slice is not unique).

[0128] Basic unit of communication - model slice: For any model parameter or gradient, the whole can be seen as a global abstract ParamBlock. Each ParamBlock can be divided into multiple ParamSlices according to the distributed state (DStates) corresponding to the parallel strategy. Due to the existence of data parallelism (DP), the same ParamSlice can be owned by multiple devices at the same time. For any two sets of parallel strategies (such as src and dst strategies in Figure 11 ), the intersection of their ParamSlice division is defined as the basic unit of hot switching communication - model slice. Figure 11 is the basic unit of communication: the intersection of the ParamSlice division (left) of the two sets of parallel strategies (right).

[0129] Hot switching problem definition: Assuming hot switching from the current strategy to the target strategy, for any model slice, traverse each device (needed devices) in the target strategy that needs the slice, and from all devices (owned devices) that own it in the current strategy, select the most suitable sender.

[0130] Heuristic algorithm: Based on two basic principles, the present application proposes a heuristic search algorithm, so as to find the most suitable hot switching communication scheme.

[0131] Principle one: Intra-node communication is superior to inter-node communication. In a traditional GPU cluster, intra-node GPUs are linked for communication through NVLink, which has higher communication bandwidth than inter-node cross-machine communication through Infiniband or Ethernet. Therefore, if there are multiple different devices such as intra-machine or inter-machine that own the same model slice, the intra-node device is preferred as the sender.

[0132] Principle two: The connection link of the GPU is full duplex. Modern network links usually have independent communication bandwidth for data transmission and reception, so simultaneous data transmission and reception for any device will not affect communication efficiency. In fact, since the amount of data each device needs to receive is fixed (determined by the current strategy of switching), it cannot be reduced, only the data transmission amount of different devices can be as balanced as possible. That is, minimize the maximum value in the data transmission amount of all devices.

[0133] Based on the above two basic principles, the heuristic algorithm for deriving the hot switching communication scheme is as shown in Figure 13 , and the process is as follows:

[0134] (a) Step 1: For each device i, record the internal communication volume V. i (intra) Inter-machine communication volume V i (inter) .

[0135] (b) Step 2: Traverse each model parameter / gradient slice and determine the set of devices S (owner devices) that own the slice and the set of devices D (target devices) that need the slice based on the distributed state of the current policy and the target policy.

[0136] (c) Step 3: Iterate through each device dst in set D, and divide the devices in set S into intra-device devices S based on the differences between intra-device and inter-device devices. (intra) and machine room equipment S (inter) Based on principle one, priority should be given to internal equipment S. (intra) If empty, then consider the machine room equipment S. (inter) .

[0137] (d) Step 4: Based on principle two, greedily select the device with the smallest current data transmission volume from the candidate device set as the sender of the model slice, i.e., src←{V i (intra)or(inter) ,i∈S (inter)or(inter)}; simultaneously update the communication traffic corresponding to the sender, i.e.

[0138] (2) Optimize hot switching overhead: Message Fusion & Layout Optimization.

[0139] Hot-swapping incurs additional overhead, including communication overhead and memory copy overhead. Here, we use two techniques, message fusion and layout optimization, to perform targeted optimization.

[0140] Message merging: such as Figure 14As shown, this invention merges data sent to the same device into consecutive send buffers, and similarly merges data received from the same device into consecutive receive buffers. This allows for the merging of multiple p2p send / recv operations, reducing the number of p2p kernel calls, increasing the data size of a single communication, and improving bandwidth utilization and communication efficiency. Furthermore, the BatchedSendRecv primitive provided by NCCL supports parallel p2p send / recv transmission corresponding to different send / recv buffers.

[0141] Layout optimization: such as Figure 15 As shown, to avoid the high copy overhead caused by kernel memory access and data movement due to the introduction of contiguous and concat operators, this invention arranges the vertically partitioned weights / gradients in a horizontally partitioned layout. This allows most non-contiguous memory accesses to be converted into contiguous memory accesses, thus directly transforming most time-consuming contiguous and concat operators into cudaMemCpy, which has a very low memory access cost. In specific calculations, only the layout parameters of the gem kernel need to be changed to ensure mathematical equivalence. The original parameter W was used for vertical segmentation. Now, let the new parameter W′ be used for horizontal segmentation, and initialize W′ to W. T Therefore, during the calculation, the left-hand side of the equation needs to have layout = True set to transform W into W. T The right-hand side expression is set to layout=False to ensure mathematical equivalence.

[0142] II. Training Phase.

[0143] The training phase is mainly used for hot-switching training based on real-time read data and the computation graph obtained during the compilation phase.

[0144] by Figure 6 Taking the example of HotSPa, let's briefly introduce the overall training process. Here, DP, TP, and PP are abbreviations for data parallelism, tensor parallelism, and pipelined parallelism, respectively. DP2 indicates a data parallelism of 2, similarly TP2 indicates a tensor parallelism of 2, and PP2 indicates a pipelined parallelism of 2.

[0145] Step 1: First, group the data within the global batch and bind it to different computation graphs generated during the compilation phase. For example... Figure 6The Party has 3 groups Group 1: <TP2, PP2> for (16k~32k), Group 2: <DP2, TP2> for (4k~16k), Group 3: <dp4>for (0~4k). Each group corresponds to a parallel strategy combination. First, initialize the model parameters (Model Params) and the optimizer states (Optimizer States) based on the instantiation graph Graph1 (Instantiation graph). Note that the instantiation graph Graph1 is not executed at this time, but only initialized.

[0146] Step 2: Hot switch the model parameters (Params) from the instantiation graph Graph1 to Graph2.

[0147] Step 3: Apply for a gradient accumulation buffer (Grad Accumulation Buffer) on Graph2.

[0148] Step 4: Train on Graph2 based on the data in Group2, and accumulate the obtained gradient into the buffer.

[0149] Step 5: Hot switch the model parameters (Params) and the accumulated gradient (Grads) from Graph2 to Graph3.

[0150] Step 6: Train on Graph3 based on the data in Group3, and accumulate the obtained gradient into the buffer.

[0151] Step 7: Hot switch the accumulated gradient (Grads) from Graph3 to the instantiation graph Graph1. (Here, there is no need to switch the model parameters, because the initial model parameters are saved on the instantiation graph.)

[0152] Step 8: Train on Graph1 based on the data in Group1, and accumulate the obtained gradient into the buffer.

[0153] Step 9: All data in the global batch has been trained, and the accumulated gradient on the three groups has been obtained. Based on the gradient, perform an update on the model parameters on Graph1.

[0154] Note that, Figure 6 The process of only represents the training and updating of the model with one step of data.

[0155] Next, a specific experiment is used to illustrate the efficient large model training system provided by the present application to support parallel hot switching.

[0156] Experimental setup: In the experiment, the present application compares HotSPa and two existing distributed training systems, Megatron-LM (DP+TP+PP+SP) and DeepSpeed (Zero1 / 2 / 3+Ulysses), under different loads. Specifically:

[0157] On the experimental environment: 4 GPU servers, each server with 8 A800-80G, the communication bandwidth of the internal NVLink is 400 GB / s, and the inter-machine IB communication bandwidth is 200 GB / s.

[0158] On the model: LLaMA2, an open-source model, is selected, including three different sizes of parameter quantities: LLaMA2-7B, LLaMA2-13B, and LLaMA2-32B.

[0159] On the dataset: Two open-source and widely used CommonCrawl and GitHub are selected. The sequence length distribution of the two datasets is as shown in Figure 16 .

[0160] In the experimental setup: the context length is set to 4k-32k, and the number of samples corresponding to each global batch is fixed at 512. All systems use the sequence packing scheme, and HotSPa divides a global batch into 3-4 groups at most.

[0161] Experimental comparison effect: Experiments are conducted on different scales with 8-32 GPUs, model sizes of LLaMA2-7b-32b, and maximum sequence lengths of 4k-32k. On the GitHub and CommonCrawl datasets, HotSPa achieves a maximum speedup of 1.5x and 9.2x, respectively, relative to Megatron-LM, and a maximum speedup of 2.5x and 5x, respectively, relative to DeepSpeed.

[0162] Specifically:

[0163] 1. Under the static strategy, HotSPa and Megatron-LM / DeepSpeed perform consistently.

[0164] 2. In the CommonCrawl dataset, where the long-tail phenomenon is very serious, Megatron-LM causes the overall training time to increase by 2-4 times in order to support longer context training. The training time of HotSPa is not lengthened by this long-tail distribution, and the time is nearly constant, so a speedup of 2.99x can be achieved.

[0165] 3. In the GitHub dataset with a relatively large proportion of long sequences, HotSPa can also achieve a speedup of 1.5x relative to Megatron-LM by accelerating short sequences.

[0166] 4. When only using zero1, such as the experiment of LLaMA-7B, DeepSpeed Ulysses performs better than Megatron-LM, but once zero2 / 3 is used, in the scenario where the global batch is large, resulting in multiple gradient accumulations, each micro batch needs to do gradient reduce-scatter and parameter all-gather on the entire cluster, resulting in performance degradation, so HotSPa can obtain an acceleration ratio of 2.6x and 5x respectively on the two datasets compared with DeepSpeed.

[0167] In summary, HotSPa is the first large model distributed training system that supports dynamic hot switching of parallel strategies. Compared with existing systems such as Megatron-LM and DeepSpeed that only support static parallel strategies, HotSPa can more flexibly support and adapt to scenarios with dynamic changes in load, and can achieve higher training throughput in most existing datasets with uneven distribution of long and short sequences.

[0168] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the present application.

Claims

1. A large model training method supporting parallel hot switching, characterized in that, The method comprises: generating a logical computation graph, wherein the logical computation graph is used to represent a plurality of different parallel strategy combinations; compiling the logical computation graph to generate a plurality of executable computation graphs, wherein each executable computation graph corresponds to a parallel strategy combination; selecting an initialization graph for initializing a large model state from the plurality of executable computation graphs, and arranging an execution order between the executable computation graphs according to a hot switching cost between any two parallel strategy combinations; training the large model based on the initialization graph and the execution order between the executable computation graphs; wherein the generating of the logical computation graph comprises: obtaining parallel strategies, and grouping the parallel strategies according to a data sequence length processed by the parallel strategies; binding parameters and input variables of each parallel strategy combination to a plurality of distributed states; under a distributed tensor architecture, deducing the plurality of distributed states, and processing differences in intermediate operators in different parallel strategy deduction processes by inserting a dummy operator used as a placeholder, so as to represent different parallel strategies by using one logical computation graph; the selecting of the initialization graph for initializing the large model state from the plurality of executable computation graphs, and the arranging of the execution order between the executable computation graphs according to the hot switching cost between any two parallel strategy combinations, comprises: obtaining model splitting granularities corresponding to the parallel strategy combinations; taking an executable computation graph corresponding to a parallel strategy combination with the smallest model splitting granularity as the initialization graph for initializing the large model state; grouping data in a global batch processing according to sequence lengths, so as to correspond to the parallel strategy combinations of each group; L = sum(V i (inter) ) / B (inter) + sum(V i (intra) ) / B (intra) ; wherein V i (inter) represents inter-machine traffic, B (inter) represents inter-machine bandwidth, V i (intra) represents intra-machine traffic, and B (intra) represents intra-machine bandwidth. combining a minimum hot switching overhead to minimize the hot switching cost between all parallel strategy combinations, to obtain the execution order between the executable computation graphs; wherein the hot switching overhead is obtained by optimizing a communication overhead and a memory copy overhead based on message merging and layout optimization.

2. The method of claim 1, wherein, The compiling of the logical computation graph comprises: inserting an fp32->bf16 type conversion operator and a gradient accumulation operator in the logical computation graph; and / or, merging adjacent communication operators and fusion operators; and / or, pruning unnecessary dummy operators, type conversion operators, gradient accumulation operators and gradient communication operators; and / or, replacing all communication operators with a collective communication operator or a point-to-point communication operator, and generating a corresponding collective communication operator or point-to-point communication operator; and / or, rearranging a local topology of the logical computation graph to realize overlapping of computation and communication.

3. The method of claim 1, wherein, The calculation of the hot switching cost between any two parallel strategy combinations on grouped data comprises: For each device device i, record the intra-machine communication bandwidth B (intra) and the inter-machine communication bandwidth B (inter) and the current intra-machine traffic V i (intra) and the inter-machine traffic V i (inter) ; traversing each model parameter / gradient slice, and determining a device set S owning the slice and a device set D requiring the model parameter / gradient slice based on distributed states of a current strategy and a target strategy; traversing each device dst in the set of devices D, partitioning the devices in the set S into intra-machine devices S (intra) and inter-machine devices S (inter) according to intra- and inter-machine differences; In-device S (intra) Possessing a model parameter / gradient slice required by the in-device S (intra) As the sender of the model parameter / gradient slice, and performing in-device traffic V i (intra) update; In-device S (inter) Without owning the model parameter / gradient slice required by the device dst, then select the in-device S (inter) with the smallest current data transmission amount in the device (inter) As the sender of this model parameter / gradient slice, and carry out the in-device communication V i (inter) update.

4. The method of claim 1, wherein, the message merging process comprises: merging data sent to the same device into a continuous sending buffer to generate merged sending data; and merging data received by the same device into a continuous receiving buffer to generate merged receiving data. ​ The merged sending data and the merged receiving data are supported to be transmitted in parallel through a BatchedSendRecv primitive.

5. The method of claim 1, wherein, The layout optimization process comprises: The longitudinally split model parameter / gradient slices are arranged on the layout according to the transverse split, and the mathematical equivalence is ensured by changing the layout parameters of the general matrix multiplication kernel.

6. The method of claim 1, wherein, The training of the large model based on the execution order between the initialization graph and the executable computation graph comprises: Data in the global batch is grouped by sequence length and bound to the corresponding executable computation graph Graph j The data in each group Data j ; wherein j ∈ [1, n], n represents the total number of executable computation graphs; According to the initialization graph, the initialization of the model parameters and the optimizer is performed to obtain the initialized model parameters θ t＝0 ; wherein t is the number of iteration rounds. The model parameters θ t and the current accumulated gradient in the t-th round of training from the executable computation graph Graph j-1 hot-swap to the executable computation graph Graph j , and based on the data Data j train on the executable computation graph Graph j , to obtain the accumulated gradient According to the accumulated gradient The model parameters θ t are updated to obtain the model parameters θ t+1 ; In a case where the large model based on the model parameters θ t+1 does not meet the set requirement, and the iteration round number t+1 does not exceed the maximum iteration round number, let t=t+1, and re-execute the step of updating the model parameters θ t and the current cumulative gradient in the tth round of training. j-1 hot switching to the executable computation graph Graph j ; In a case where the large model based on the model parameter θ t+1 satisfies a set requirement, or a case where the iteration round number t+1 exceeds a maximum iteration round number, the large model based on the model parameter θ t+1 is output.

7. A large model training system supporting parallel hot switchover, characterized by, The system comprises: A generation module is configured to generate a logical computation graph, which is used to represent a plurality of different parallel strategy combinations; A compilation module is configured to compile the logical computation graph to generate a plurality of executable computation graphs, each of which corresponds to a parallel strategy combination; A derivation module is configured to select an initialization graph for initializing a large model state from the plurality of executable computation graphs, and arrange an execution order between the executable computation graphs according to a hot switching cost between any two parallel strategy combinations. A training module is configured to train the large model based on the initialization graph and the execution order between the executable computation graphs. The generation of the logical computation graph comprises: Parallel strategies are obtained, and the parallel strategies are grouped according to data sequence lengths processed by the parallel strategies; Parameters and input variables of each parallel strategy combination are bound to a plurality of distributed states; In a distributed tensor architecture, the plurality of distributed states are derived, and a difference in an intermediate operator in a derivation process of different parallel strategies is processed by inserting a dummy operator used as a placeholder, so that the different parallel strategies are represented by one logical computation graph; The selection of the initialization graph for initializing the large model state from the plurality of executable computation graphs, and the arrangement of the execution order between the executable computation graphs according to the hot switching cost between any two parallel strategy combinations, comprise: Model split granularities corresponding to the parallel strategy combinations are obtained; An executable computation graph corresponding to a parallel strategy combination with the smallest model split granularity is selected as the initialization graph for initializing the large model state; Data in a global batch are grouped according to sequence lengths to correspond to the parallel strategy combinations; L = sum(V i (inter) ) / B (inter) + sum(V i (intra) ) / B (intra) ; wherein V i (intra) represents inter-machine traffic, B (inter) represents inter-machine bandwidth, V i (intra) represents intra-machine traffic, and B (intra) represents intra-machine bandwidth. An execution order between the executable computation graphs is obtained by combining a hot switching overhead to minimize the hot switching cost between all parallel strategy combinations, wherein the hot switching overhead is obtained by optimizing a communication overhead and a memory copy overhead based on message merging and layout optimization.

Images