Large model training method and system supporting parallel hot switching

Abstract

The invention discloses a large model training method and system supporting parallel hot switching, and belongs to the technical field of large model training. The method comprises the following steps: generating a logic calculation graph, wherein the logic calculation graph is used for representing a plurality of groups of different parallel strategy combinations; compiling the logic calculation graph to generate a plurality of executable calculation graphs; wherein each executable calculation graph corresponds to one group of parallel strategy combination; selecting an initialization graph used for initializing the state of the large model from the plurality of executable calculation graphs, and arranging the execution sequence between the executable calculation graphs according to the hot switching cost between any two groups of parallel strategy combinations; and training a large model based on an execution sequence between the initialization graph and the executable calculation graph. According to the method, not only can the sequence calculation amount / working load in each group be ensured to be approximately close, but also the parallel strategy can be dynamically switched in the model training process, and gradient accumulation and updating of the model can be normally completed.

Description

Technical Field

[0001] The present invention relates to the technical field of large model training, and in particular to a large model training method and system supporting parallel hot switching. Background Art

[0002] In recent years, large language models (LLMs) represented by ChatGPT have attracted widespread attention. Their performance improvement is due to the growth of model size, context, and data volume. Taking the open source LLaMA3 as an example, the model has a maximum of 400 billion parameters, a context length of 128k, and the data volume has reached an astonishing 15 trillion tokens. But at the same time, these growths have also brought more challenges to system optimization. On the one hand, when encountering larger model sizes or longer contexts, it is necessary to use appropriate parallel strategies to balance video memory occupancy and computational efficiency; on the other hand, the sequence length of real data is usually dynamically changing, and most of them are long-tailed. Appropriate strategies are needed to handle dynamic sequences to avoid wasting computing resources.

[0003] When faced with larger model sizes or longer contexts, existing technical solutions include Megatron-LM's hybrid parallel solution and DeepSpeed's Zero parallel + Ulysses solution.

[0004] The hybrid parallel scheme of Megatron-LM mainly includes data parallelism, sequence parallelism, model parallelism and hybrid parallelism.

[0005] Data parallelism is to divide a large amount of input data into multiple parts, and each device is responsible for processing only part of the data. Different devices need to introduce additional communication costs to synchronize gradients. Under data parallelism, each device will have a complete set of model parameters. When the video memory is insufficient, the model needs to be further split.

[0006] Sequence parallelism is when a single piece of sample data is particularly long. It can be split into multiple parts in the sequence dimension. Each device only processes a small part of the sequence slice. Additional communication (All-Gather+Reduce-Scatter / P2P) is required between different devices to obtain the overall sequence information. For example, Megatron-LM's Sequence Parallel and Context Parallel.

[0007] Model parallelism divides the model into multiple parts so that each device only retains part of the model parameters. Depending on the dimension of division, model parallelism is divided into tensor parallelism and pipeline parallelism. Tensor parallelism mainly divides the parameters within the layer, vertically divides the fully connected layer and attention layer in the Transformer to multiple devices, and introduces additional communication operators (All-Reduce) to ensure calculation consistency. Pipeline parallelism: It divides the parameters between layers, divides the model horizontally to multiple devices with TransformerLayer as the basic unit, and each device is responsible for the calculation of part of the model layer, and introduces additional communication operators (P2P) to complete the transmission of intermediate calculation results.

[0008] Hybrid parallelism is because different parallel strategies have different effects on computing, communication, and video memory. The above parallel strategies are combined to form a hybrid parallel strategy to achieve better training efficiency.

[0009] in, Figure 1 This is an example diagram of the hybrid parallel scheme of Megatron-LM, which includes data parallelism, tensor parallelism, and pipeline parallelism.

[0010] DeepSpeed's Zero parallelism + Ulysses solution mainly includes: Zero parallelism and sequential parallelism.

[0011] Zero parallelism also divides data and model parameters on different devices, but follows the data parallel computing paradigm as a whole. In order to ensure the consistency of the calculation, additional communication of model parameters is required (All-Gather+Reduce-Scatter).

[0012] Sequence parallelism utilizes DeepSpeed's Ulysses solution, which obtains the overall sequence information by performing a communication operator (All-to-All) on the segmented sequence.

[0013] in, Figure 2 This is an example diagram of the Zero parallel + Ulysses solution.

[0014] When dealing with sequences with dynamically changing lengths in a dataset, existing processing solutions include sequence padding and sequence packing. Sequence padding uses a special token to fill each piece of data with different lengths to the maximum length, but this will cause a lot of unnecessary redundant calculations. Sequence packing splices sequences of different lengths to a length close to the maximum length, and uses a special mask to ensure that calculations between different sequences do not affect each other, thereby avoiding computational waste.

[0015] in, Figure 3 This is an example diagram of sequence filling and sequence splicing for a dynamic sequence.

[0016] However, the existing static parallel strategy combinations are not suitable for dynamic scenarios with changing sequence lengths and are not efficient. This is because when training large-scale Transformer models in a distributed manner, the choice of parallel strategies such as data parallelism, sequence parallelism, and model parallelism will result in different memory consumption, communication overhead, and execution efficiency. Existing systems, such as Megatron-LM and DeepSpeed, usually assume that the training load is constant, so they are designed to use only one fixed / static parallel strategy combination throughout the training process. However, most existing datasets are long-tailed. When encountering such dynamic scenarios with changing sequence lengths, the optimal parallel strategies required for different sequences may be different, and static parallel strategies are not necessarily the optimal solution.

[0017] In addition, sequence padding will cause serious computational waste, while sequence splicing will not only cause computational imbalance, but will also further lead to imbalance in parallel strategies.

[0018] For example, in scenarios where the length of sequences varies, people generally use sequence packing to avoid computational waste. In this case, the computation time of each sequence can be modeled as time seq =a·s 2 +b·s, the time of each set of data can be modeled as time micro batch =a·∑s i 2 +b·∑s i =a·∑s i 2 +b·S. Where a, b, and S are constants, a and b are coefficients, S is the maximum context length of the model, that is, the length after concatenation; s is a variable, corresponding to the sequence length of each data.

[0019] from Figure 4 As can be seen from the figure, for a complete 32k sequence, the amount of computation is proportional to (32k)^2, while for a sequence composed of 32 1k sequences, the amount of computation is proportional to 32*(1k)^2, which is obviously unbalanced. Figure 5 As can be seen in the figure, unbalanced computing will further lead to imbalance in data parallelism, model parallelism, and pipeline parallelism, further slowing down the overall execution efficiency. Summary of the invention

[0020] In view of the above-mentioned deficiencies in the prior art, the present invention provides a large model training method and system that supports parallel hot switching, which can not only ensure that the sequence calculation amount / workload within each group is roughly close, but also enable the parallel strategy to be dynamically switched during the model training process and normally complete the gradient accumulation and update of the model.

[0021] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0022] A large model training method supporting parallel hot switching, the method comprising:

[0023] Generate a logic calculation graph, wherein the logic calculation graph is used to represent multiple groups of different parallel strategy combinations;

[0024] Compiling the logic calculation graph to generate multiple executable calculation graphs; wherein each executable calculation graph corresponds to a set of parallel strategy combinations;

[0025] Select an initialization graph for initializing the state of the large model from multiple executable computation graphs, and arrange the execution order between the executable computation graphs according to the hot switching cost between any two sets of parallel strategy combinations;

[0026] The large model is trained based on the execution order between the initialization graph and the executable computation graph.

[0027] Furthermore, generating a logic calculation graph includes:

[0028] Acquire parallel strategies, and group the parallel strategies according to the length of the data sequence processed by the parallel strategies;

[0029] Bind the parameters and input variables of each set of parallel strategy combinations to multiple sets of distributed states;

[0030] Under a distributed tensor architecture, multiple groups of distributed states are derived, and the differences in intermediate operators in the derivation process of different parallel strategies are handled by inserting pseudo operators, so that different parallel strategies can be represented using a logical computation graph.

[0031] Furthermore, compiling the logic calculation graph includes:

[0032] Inserting a type conversion operator of fp32->bf16 and a gradient accumulation operator into the logic calculation graph;

[0033] and / or,

[0034] Merge adjacent communication operators and fusion operators;

[0035] and / or,

[0036] Prune unnecessary fake operators and type conversion operators, gradient accumulation operators, and gradient communication operators;

[0037] and / or,

[0038] Replace all communication operators with collective communication operators or point-to-point communication operators, and generate corresponding collective communication operators or point-to-point communication operators;

[0039] and / or,

[0040] Rearrange the local topology of the logical computation graph to achieve overlap of computation and communication.

[0041] Furthermore, the selecting of an initialization graph for initializing the state of the large model from a plurality of executable computation graphs includes:

[0042] Obtain the model segmentation granularity corresponding to each group of parallel strategy combinations;

[0043] The executable computational graph corresponding to the parallel strategy combination with the smallest model segmentation granularity is used as the initialization graph for initializing the state of the large model.

[0044] Furthermore, the step of arranging the execution order between executable computation graphs according to the hot switching cost between any two sets of parallel strategy combinations includes:

[0045] The data in a global batch are grouped according to the sequence length to correspond to the parallel strategy combination of each group;

[0046] Calculate the hot switching cost L = sum(V i (inter) ) / B (inter) +sum(V i (intra) ) / B (intra) ; Among them, V i (inter) represents the amount of inter-machine communication, B (inter) represents the inter-machine communication bandwidth, V i (intra) represents the amount of intra-machine communication, B (intra) Indicates the intra-machine communication bandwidth;

[0047] The hot switching cost between all parallel strategy combinations is minimized in combination with the hot switching overhead to obtain the execution order between executable computation graphs; wherein, the hot switching overhead is obtained by optimizing the communication overhead and the video memory copy overhead based on message merging and layout optimization.

[0048] Furthermore, the step of calculating the hot switching cost between any two sets of parallel strategy combinations on the grouped data includes:

[0049] For each device i, record the internal communication bandwidth B (intra) and inter-machine communication bandwidth B (inter) , and the current intra-machine communication volume V i (intra) and the amount of inter-machine communication V i (inter) ;

[0050] Traverse each model parameter / gradient slice, and determine the device set S that owns the slice and the device set D that needs the model parameter / gradient slice based on the distributed state of the current policy and the target policy;

[0051] Traverse each device dst in the device set D, and divide the devices in the device set S into intra-device devices S according to the difference between intra-device and inter-device (intra) And machine room equipment (inter) ;

[0052] In-machine equipment (intra) If the model parameters / gradient slices required by the device dst are available, the device S in the machine will be (intra) As the sender of the model parameters / gradient slices, and performs intra-machine communication V i (intra) Updates;

[0053] In-machine equipment (intra) If the model parameters / gradient slices required by device dst are not available, then the inter-machine device S (inter) Select the inter-machine device S with the smallest current data transmission volume (inter) As the sender of the model parameters / gradient slices, and conducts inter-machine communication volume V i (inter) Updates.

[0054] Furthermore, the message merging process includes:

[0055] Merge the data sent to the same device into continuous sending buffers to generate merged sending data;

[0056] and,

[0057] Merge the data received by the same device into continuous receive buffers to generate merged received data.

[0058] and,

[0059] The BatchedSendRecv primitive supports parallel transmission of combined send data and combined receive data.

[0060] Furthermore, the layout optimization process includes:

[0061] The vertically sliced ​​model parameters / gradient slices are laid out according to the horizontal slices, and the mathematical equivalence is ensured by changing the layout parameters of the general matrix multiplication kernel.

[0062] Furthermore, the training of the large model based on the execution order between the initialization graph and the executable computation graph includes:

[0063] Group the data in the global batch according to the sequence length and bind them to the corresponding executable graph j To get each set of data j ; Where j∈[1,n], n represents the total number of executable computational graphs;

[0064] Initialize the model parameters and optimizer according to the initialization graph to obtain the initialized model parameters θ t＝0 ; Where t is the number of iterations;

[0065] The model parameters θ t and the current cumulative gradient in the tth round of training From the executable computational graph Graph j-1 Hot switch to executable graph j and based on the data j In the executable computational graph Graph j Train on it and get the cumulative gradient

[0066] According to the cumulative gradient For the model parameters θ t Update and get the model parameters θ t+1 ;

[0067] Based on the model parameters θ t+1 If the large model does not meet the set requirements and the number of iterations t+1 does not exceed the maximum number of iterations, set t=t+1 and re-execute the method of changing the model parameters θ t and the current cumulative gradient in the tth round of training From the executable computational graph Graph j-1 Hot switch to executable graph j middle;

[0068] Based on the model parameters θ t+1 The large model meets the set requirements, or the number of iterations t+1 exceeds the maximum number of iterations, and the output is based on the model parameters θ t+1 Large model.

[0069] A large model training system supporting parallel hot switching, the system comprising:

[0070] A generating module, used for generating a logic calculation graph, wherein the logic calculation graph is used for representing a plurality of different parallel strategy combinations;

[0071] A compiling module, used to compile the logic calculation graph to generate multiple executable calculation graphs; wherein each executable calculation graph corresponds to a set of parallel strategy combinations;

[0072] The inference module is used to select an initialization graph for initializing the state of the large model from multiple executable computation graphs, and to arrange the execution order between the executable computation graphs according to the hot switching cost between any two sets of parallel strategy combinations;

[0073] A training module is used to train a large model based on the execution order between the initialization graph and the executable computation graph.

[0074] Compared with the prior art, the present invention has at least the following beneficial effects.

[0075] 1) The first parallel hot switching training paradigm is proposed: The present invention proposes a new training paradigm based on parallel hot switching. For each global batch of data, the present invention will group them according to the difference in their computing amount / workload, and use the most appropriate parallel strategy for each group. Between any two groups of strategies, the system of the present invention will automatically, efficiently and seamlessly convert model parameters and gradients, and complete the gradient accumulation between different strategies before the model is updated to ensure that the training effect is not affected.

[0076] 2) Unified computational graph representation and compilation: Existing systems, such as Megatron-LM and DeepSpeed, only support a fixed combination of parallel strategies during training due to their complex system design, and thus cannot use different parallel strategies for different sequence loads. The present invention designs a special graph compiler that can simultaneously represent multiple groups of different parallel strategy combinations with a unified logic graph, and further compile and generate corresponding multiple groups of executable computational graphs, and share the storage of model states, so as to support complex parallel hot switching semantics.

[0077] 3) Parallel hot switching technology: Given any two different sets of parallel strategy combinations, hot switching between them requires exchanging model parameters and gradients in different devices, which inevitably introduces additional communication overhead. To solve this problem, the present invention designs a hot switch planner, proposes a heuristic algorithm to find the optimal communication scheme between any two sets of strategies, and introduces a series of communication and memory copy optimization techniques to further reduce the switching overhead.

[0078] 4) Based on Graph Compiler and Hot Switch Planner, the present invention implements the HotSPa system. The HotSPa system supports a new parallel hot switch training paradigm. Compared with the existing system, a speedup ratio of up to 2.99x can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 An example diagram of the hybrid parallel scheme of Megatron-LM.

[0080] Figure 2 This is an example diagram of the Zero parallel + Ulysses solution.

[0081] Figure 3 This is an example diagram of sequence filling and sequence splicing for a dynamic sequence.

[0082] Figure 4 This is an example diagram comparing the same amount of data but different computational complexity under sequence splicing.

[0083] Figure 5 This is an example diagram of unbalanced data parallelism and pipeline parallelism under sequence splicing.

[0084] Figure 6 An overview of an efficient large model training system that supports parallel hot switching.

[0085] Figure 7 A schematic diagram showing the parallel strategy using DStates.

[0086] Figure 8 Example diagram representing different combinations of parallel strategies for a unified logic diagram.

[0087] Fig. 9 This is an example diagram of converting a logical diagram into an executable diagram.

[0088] Fig.10 A schematic diagram of the arrangement of executable graphs and the hot switching process.

[0089] Fig.11 Example diagram for modeling a hot-swappable communication primitive.

[0090] Fig.12 Example diagram for modeling hot switching.

[0091] Fig.13 This is a description of the hot switching algorithm.

[0092] Fig.14 Example diagram of the optimization technique for message merging.

[0093] Fig.15 Figure 1 is an example of layout optimization techniques.

[0094] Fig.16 The data length distribution diagram of the two datasets used (CommonCrawl and GitHub). DETAILED DESCRIPTION

[0095] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0096] The technical solution proposed by the present invention is as follows Figure 6 As shown, it is mainly divided into the compilation stage and the training stage.

[0097] 1. Compilation phase.

[0098] The compilation phase is mainly used for derivation, arrangement and initialization. The editing phase is based on two core technologies: the Hot Switch Planner deduces the optimal communication scheme and corresponding communication cost between any two sets of strategy combinations; the Graph Compiler compiles and generates multiple sets of corresponding executable computational graphs based on multiple sets of strategy combinations represented by user-defined logic graphs, arranges the execution order of these computational graphs based on the communication cost provided by the Hot Switch Planner, and selects the initialization graph used to initialize the model state.

[0099] 1. Graph Compiler

[0100] The graph compiler supports using a unified logic graph to represent multiple sets of different parallel strategy combinations at the same time, and further compiles and generates corresponding sets of executable computation graphs, and shares the storage of model states. Corresponding to three core steps / innovations:

[0101] (1) Logic Graph: A unified logic graph is used to represent multiple sets of different parallel strategy combinations.

[0102] like Figure 7As shown, the present invention is based on the DTensor (Distributed Tensor) architecture, and uses DStates (Distributed States) to represent a set of parallel strategy combinations: Splits, Partial, and Duplicate are used to represent the way parameters are sliced ​​in different devices (where Splits represents the number of slices of the tensor in different dimensions, and each device has one of the slices; Partial means that different devices have partial information of the same tensor; Duplicate means that different devices repeatedly have the same tensor, that is, no segmentation is performed), and DeviceGroup is used to represent the mapping relationship between different parameter slices and devices.

[0103] In order to simultaneously represent multiple sets of distributed strategies, the present invention innovatively binds a DTensor to multiple sets of DStates and supports the simultaneous derivation of multiple sets of distributed states on the entire logical computation graph.

[0104] Specifically, each parameter and input variable is bound to multiple sets of DStates at the same time. In the process of building the logical calculation graph, each operator will deduce these multiple sets of DStates at the same time and automatically insert intermediate operators to ensure the integrity of the function.

[0105] Due to different parallel strategy combinations, the computation graphs obtained by deriving DStates are not necessarily the same (e.g. Figure 8 In order to represent these multiple strategies with the same logical computation graph, dummy operators are automatically inserted to handle the differences between different strategies in intermediate operators such as communication (e.g. Figure 8 (b) in ), so that the two computation graphs can be represented uniformly by the same logical computation graph (such as Figure 8 The specific explanation is as follows:

[0106] ● Figure 8 Take the two strategies TP2 and DP2 in (a) as examples: TP2 means splitting the model weights, so the intermediate result y obtained by the initial calculation is also in a split state, and an additional communication operator comm needs to be introduced to obtain the complete result y; while DP2 means that the model weights are not split, so the complete result y can be directly calculated. Obviously, the calculation graphs corresponding to TP2 and DP2 are not equal.

[0107] ●In order to combine the two computation graphs representing TP2 and DP2 in (a), a dummy operator (dummy op) is inserted into the computation graph of DP2, which is only used as a placeholder and has no computational significance. At this time, in (b), regardless of the split state, the representation of the two computation graphs of TP2 and DP2 is the same.

[0108] ●Furthermore, due to the DTensor architecture, the split states DStates are bound to the tensors in the computational graph. Therefore, the present invention can bind multiple groups of split states to the same tensor at the same time, thereby obtaining a unified logic graph in (c).

[0109] Through the above scheme, this work supports the use of a unified logical graph to represent multiple groups of different parallel strategy combinations.

[0110] (2) Exec Graphs: Based on the unified logic graph compilation, multiple sets of executable computation graphs are generated. Each set of executable computation graphs corresponds to a set of independent parallel strategy combinations, and they share the same model state storage.

[0111] The logical graph is only an abstract representation. In order to compile and generate a truly executable distributed computing graph, the compiler will perform operations such as inserting, merging, pruning, replacing, and rearranging operators.

[0112] ●Insert the fp32->bf16 type conversion operator and gradient accumulation operator.

[0113] ●Merge adjacent communication operators (comm op) and fusion operators (fusion op).

[0114] ●Prune unnecessary dummy ops and type conversion, gradient accumulation, and gradient communication operators.

[0115] ●Replace all communication operators (comm op) with collective communication operators or point-to-point communication operators, and generate corresponding collective communication operators or point-to-point communication operators.

[0116] ●Rearrange the local topology of the computation graph to achieve overlap of computation and communication.

[0117] Fig. 9 A specific example of converting a logic graph into two sets of executable graphs (two sets of parallel strategy combinations) is given. Fig. 9 The upper left corner of the figure is the logical calculation graph, and the lower left corner and right side are the compilation and generation of two sets of executable calculation graphs. The main difference here is the step of replacing the communication operator to generate the executable calculation graph. Figure 1Just replace the communication operator (comm op) with the collective communication operator All-Gather; and generate executable calculations Figure 2 The communication operator (comm op) needs to be replaced with a more complex point-to-point communication operator (batch send recv op), a split operator (split op) for processing the communication input, and a concatenate operator (concatenate op) for processing the communication output.

[0118] (3) Orchestrate Exec Graphs: Based on the cost analysis given by the Hot Switch Planner, orchestrate the execution order of these executable graphs (parallel strategy combinations) to minimize storage and communication overhead.

[0119] Select the initialization graph: Select the executable graph corresponding to the strategy that minimizes the model state storage as the initialization graph. (Select the strategy that minimizes the granularity of the model segmentation. Fig.10 For example, the strategy TP2, PP2 that completely splits the model is selected as the initialization graph, while the two strategies DP4 and DP2, TP2 have some parameters that are repeated between devices and are not split due to the existence of DP);

[0120] Rearrange the executable graph: Based on the cost of hot switching between any two sets of strategies given by the Hot Switch Planner, determine the order in which different strategies are executed to minimize the overall hot switching cost. Fig.10 For example, TP2PP2->DP4->DP2TP2, only the first hot switch needs to introduce additional communication cost, and the rest only need local segmentation).

[0121] Pruning & Gradient Accumulation: Except for the initialization graph, the other executable graphs need to prune unnecessary type casting operators (type casting op), parameter update operators (update op), etc., so as to ensure that the model is updated only on the initialization graph, and only gradient accumulation is performed on other executable graphs to ensure that the accuracy of the model is not affected.

[0122] 2.Hot Switch Planner (HotSPa).

[0123] HotSPa supports grouping the data in a global batch according to the sequence length. Each group adopts a different parallel strategy (corresponding to a different executable computational graph). The gradients of different strategies are accumulated in the same gradient buffer to ensure that the model convergence is not affected.

[0124] HotSPa automatically hot switches the model weights and gradients between two different parallel strategies. The core function of the Hot Switch Planner is to derive the communication scheme with the lowest switching cost between any two different parallel strategies. Specifically, it is divided into the following two core steps:

[0125] The hot switching cost is defined as the total time from the start to the end of the hot switching, which is determined by the sending time and receiving time of the weights and gradients, corresponding to the intra-machine communication volume V i (intra) Divide by the internal communication bandwidth B (intra) Plus the inter-machine communication volume V i (inter) Divide by the inter-machine communication bandwidth B (intra) , that is, the hot switching cost L = sum(V i (inter) ) / B (inter) +sum(V i (intra) ) / B (intra) .

[0126] (1) Model Hot Switching based on heuristic algorithm

[0127] There are multiple feasible solutions for the communication scheme: hot switching refers to the switching of distributed state (from src strategy -> dst strategy), which requires the re-partitioning of the model parameters and gradients in the entire cluster. It is a many-to-many complex communication. Due to the existence of data parallelism (DP), the same model slice has multiple duplicate copies in different devices. Therefore, there are a large number of feasible solutions for the communication scheme (the sender of the model slice is not unique).

[0128] The basic unit of communication - model slice: For any model parameter or gradient, the whole can be regarded as a global abstract ParamBlock. Each ParamBlock is divided into multiple ParamSlices according to the distributed state (DStates) corresponding to the parallel strategy. Due to the existence of data parallelism (DP), the same ParamSlice may be owned by multiple devices at the same time. For any two sets of parallel strategies (such as Fig.11 The intersection of their ParamSlice partitions is defined as the basic unit of hot-swap communication - model slice. Fig.11 It is the basic unit of communication: the intersection (right side) of the ParamSlice partitions (left side) made by two sets of parallel strategies.

[0129] Hot switching problem definition: Assuming a hot switch from the current policy to the target policy, for any model slice, traverse each device that needs the slice in the target policy, and select the most suitable sender from all the devices that own it in the current policy.

[0130] Heuristic algorithm: Based on two basic principles, the present invention proposes a heuristic search algorithm to find the most suitable hot-swap communication solution.

[0131] Principle 1: Intra-node communication is better than inter-node communication. In traditional GPU clusters, GPUs in a node communicate via NVLink, which has a higher communication bandwidth than inter-node communication via Infiniband or Ethernet. Therefore, if multiple different devices within or between machines have the same model slice, the device within the node is given priority as the sender.

[0132] Principle 2: The GPU connection link is full-duplex. Modern network links usually have independent communication bandwidth for sending and receiving data, so for any device, sending and receiving data at the same time will not affect the communication efficiency. In fact, since the amount of data that each device needs to receive is fixed (determined by the current switching strategy), it is impossible to reduce it. The only way is to make the data sent by different devices as load-balanced as possible. That is, minimize the maximum value of the data sent by all devices.

[0133] Based on the above two basic principles, the heuristic algorithm derived by the hot switching communication scheme is as follows: Fig.13 As shown, the process is as follows:

[0134] (a) Step 1: For each device i, record the internal communication volume V i (intra) and the amount of inter-machine communication V i (inter) .

[0135] (b) Step 2: Traverse each model parameter / gradient slice, and determine the device set S (owner devices) that owns the slice and the device set D (target devices) that needs the slice based on the distributed state of the current policy and the target policy.

[0136] (c) Step 3: Traverse each device dst in the set D, and divide the devices in the set S into intra-device devices S according to the difference between intra-device and inter-device devices. (intra) And machine room equipment (inter) , based on principle 1, give priority to the in-machine equipment S (intra) If it is empty, consider the inter-machine device S (inter) .

[0137] (d) Step 4: Based on Principle 2, greedily select the device with the smallest current data transmission amount from the candidate device set as the sender of the model slice, that is, src←{V i (intra)or(inter) ,i∈S (inter)or(inter)}; At the same time, update the communication volume corresponding to the sender, that is,

[0138] (2) Optimize hot switching overhead: Message Fusion & Layout Optimization.

[0139] Hot switching will bring additional overhead, including communication overhead and memory copy overhead. Here, two technologies, message fusion and layout optimization, are used for targeted optimization.

[0140] Message Fusion: Fig.14As shown, the present invention merges the data sent to the same device into a continuous send buffer, and similarly, merges the data received from the same device into a continuous receive buffer, so that multiple p2p send / recv can be merged, the number of p2p kernel calls can be reduced, and the data size of a single communication can be increased, thereby improving bandwidth utilization and communication efficiency. In addition, the BatchedSendRecv primitive provided by NCCL supports p2p send / recv parallel transmission corresponding to different send / recv buffers.

[0141] Layout Optimization: Fig.15 As shown, in order to avoid the high copy overhead caused by the introduction of contiguous operators and concat operators, the weights / gradients that are vertically divided are also arranged in the horizontal division in layout in order to avoid the high copy overhead caused by the introduction of contiguous operators and concat operators. At this time, most of the non-contiguous memory accesses can be converted into continuous memory accesses, so that most of the time-consuming contiguous operators and concat operators can be directly converted into cudaMemCpy with very low memory access cost. In the specific calculation, it is only necessary to change the layout parameters of the gemm kernel to ensure mathematical equivalence, that is, Here the original parameter W is split vertically, and now the new parameter W′ is split horizontally, and W′ is initialized to W T . Then when calculating, the left side of the formula needs to set layout = True to convert W to W T , and the right side of the formula sets layout = False to ensure the mathematical equivalence of the calculation.

[0142] 2. Training phase.

[0143] The training phase is mainly used to perform hot-switch training based on the data read in real time and the computational graph obtained in the compilation phase.

[0144] by Figure 6 Taking the example of HotSPa as an example, we briefly introduce the overall training process of HotSPa. DP, TP, and PP are the abbreviations of data parallelism, tensor parallelism, and pipeline parallelism respectively. DP2 means data parallelism is 2, TP2 means tensor parallelism is 2, and PP2 means pipeline parallelism is 2.

[0145] Step 1: First, group the data in the global batch and bind them to different computational graphs generated during the compilation phase. For example Figure 6There are 3 groups in total Group1:<TP2,PP2> for(16k~32k),Group2:<DP2,TP2> for(4k~16k),Group3: <dp4>for(0~4k). Each group corresponds to a parallel strategy combination. First, initialize the model parameters (Model Params) and optimizer states (OptimizerStates) based on the initialization graph Graph1 (Instantiation graph). Note that the initialization graph Graph1 will not be executed at this time, only initialization is performed.

[0146] Step 2: Hot switch the model parameters (Params) from the initialization graph Graph1 to Graph2.

[0147] Step 3: Apply for a Grad Accumulation Buffer on Graph2.

[0148] Step 4: Based on the data in Group 2, train on Graph 2 and accumulate the obtained gradients into the buffer.

[0149] Step 5: Hot switch the model parameters (Params) and accumulated gradients (Grads) from Graph2 to Graph3.

[0150] Step 6: Based on the data in Group 3, train on Graph 3 and accumulate the obtained gradients into the buffer.

[0151] Step 7: Hot-switch the accumulated gradients (Grads) from Gprah3 to the initialization graph Graph1. (There is no need to switch the model parameters here, because the initialization graph stores the initial model parameters).

[0152] Step 8: Based on the data in Group 1, train on Graph 1 and accumulate the obtained gradients into the buffer.

[0153] Step 9: All data in the global batch has been trained, and the accumulated gradients on the three groups have been obtained. Based on the gradients, an update is performed on the model parameters on Graph1.

[0154] Notice, Figure 6 The process only means that the model training and updating are completed once with the data of one step.

[0155] The following is a specific experiment to illustrate that the present invention provides an efficient large model training system that supports parallel hot switching.

[0156] Experimental setup: In the experiment, the present invention compares HotSPa with two existing distributed training systems, Megatron-LM (DP+TP+PP+SP) and DeepSpeed ​​(Zero1 / 2 / 3+Ulysses), under different loads. Specifically:

[0157] In the experimental environment: there are 4 GPU servers, each with 8 A800-80G, the communication bandwidth of NVLink within the machine is 400GB / s, and the IB communication bandwidth between machines is 200GB / s.

[0158] In terms of model: the open source LLaMA2 was selected, including three different parameter sizes: LLaMA2-7B, LLaMA2-13B, and LLaMA2-32B.

[0159] In terms of datasets: we selected two open source and widely used datasets, CommonCrawl and GitHub. The sequence length distributions of these two datasets are as follows: Fig.16 shown.

[0160] In the experimental setting: the context length is set to 4k~32k, the number of samples corresponding to each global batch is fixed to 512, all systems use the sequence packing scheme uniformly, and HotSPa divides a global batch into 3~4 groups at most.

[0161] Experimental comparison results: Experiments were conducted on different scales with 8 to 32 GPUs, LLaMA2-7b to 32b models, and maximum sequence lengths of 4k to 32k. On the GitHub and CommonCrawl datasets, HotSPa achieved up to 1.5x and 9.2x acceleration ratios relative to Megatron-LM, and up to 2.5x and 5x acceleration ratios relative to DeepSpeed.

[0162] Specifically:

[0163] 1. Under the static strategy, HotSPa and Megaton-LM / DeepSpeed ​​perform the same.

[0164] 2. In the CommonCrawl dataset where the long-tail phenomenon is very serious, Megatron-LM's overall training time increases by 2-4 times in order to support training with longer contexts. However, HotSPa's training time is not prolonged by this long-tail distribution and remains almost unchanged, so a 2.99x acceleration ratio is ultimately achieved.

[0165] 3. In datasets such as GitHub where long sequences account for a relatively large proportion, HotSPa can also achieve a 1.5x acceleration ratio relative to Megatron-LM by accelerating short sequences.

[0166] 4. When only zero1 is used, such as in the LLaMA-7B experiment, DeepSpeed ​​Ulysses performs better than Megatron-LM. However, once zero2 / 3 is used, when the global batch is large and multiple gradient accumulations are performed, each micro batch must perform gradient reduce-scatter and parameter all-gather on the entire cluster, resulting in performance degradation. Therefore, HotSPa can achieve a 2.6x and 5x acceleration ratio on the two datasets respectively relative to DeepSpeed.

[0167] In summary, HotSPa proposed in the present invention is the first large-model distributed training system that supports dynamic hot switching of parallel strategies. Compared with existing systems that only support static parallel strategies such as Megatron-LM and DeepSpeed, HotSPa can more flexibly support and adapt to scenarios with dynamic load changes, and can achieve higher training throughput in most existing datasets with uneven distribution of long and short sequences.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A large model training method supporting parallel hot switching, characterized in that: The method comprises: Generate a logic calculation graph, wherein the logic calculation graph is used to represent multiple groups of different parallel strategy combinations; Compiling the logic calculation graph to generate multiple executable calculation graphs; wherein each executable calculation graph corresponds to a set of parallel strategy combinations; Select an initialization graph for initializing the state of the large model from multiple executable computation graphs, and arrange the execution order between the executable computation graphs according to the hot switching cost between any two sets of parallel strategy combinations; The large model is trained based on the execution order between the initialization graph and the executable computation graph.

2. The method according to claim 1, characterized in that The generating a logic calculation graph includes: Acquire parallel strategies, and group the parallel strategies according to the length of the data sequence processed by the parallel strategies; Bind the parameters and input variables of each set of parallel strategy combinations to multiple sets of distributed states; Under a distributed tensor architecture, multiple groups of distributed states are derived, and the differences in intermediate operators in the derivation process of different parallel strategies are handled by inserting pseudo operators, so that different parallel strategies can be represented using a logical computation graph.

3. The method according to claim 1, characterized in that Compiling the logic calculation graph includes: Inserting a type conversion operator of fp32->bf16 and a gradient accumulation operator into the logic calculation graph; and / or, Merge adjacent communication operators and fusion operators; and / or, Prune unnecessary fake operators and type conversion operators, gradient accumulation operators, and gradient communication operators; and / or, Replace all communication operators with collective communication operators or point-to-point communication operators, and generate corresponding collective communication operators or point-to-point communication operators; and / or, Rearrange the local topology of the logical computation graph to achieve overlap of computation and communication.

4. The method according to claim 1, characterized in that: The selecting an initialization graph for initializing the state of the large model from a plurality of executable computation graphs includes: Obtain the model segmentation granularity corresponding to each group of parallel strategy combinations; The executable computational graph corresponding to the parallel strategy combination with the smallest model segmentation granularity is used as the initialization graph for initializing the state of the large model.

5. The method according to claim 1, characterized in that The step of arranging the execution order of executable computation graphs according to the hot switching cost between any two sets of parallel strategy combinations includes: The data in a global batch are grouped according to the sequence length to correspond to the parallel strategy combination of each group; Calculate the hot switching cost L = sum(V i (inter) ) / B (inter) +sum(V i (inter) ) / B (inter) ; Among them, V (inter) represents the amount of inter-machine communication, B (inter) represents the inter-machine communication bandwidth, V i (inter) represents the amount of intra-machine communication, B (inter) Indicates the intra-machine communication bandwidth; The hot switching cost between all parallel strategy combinations is minimized in combination with the hot switching overhead to obtain the execution order between executable computation graphs; wherein, the hot switching overhead is obtained by optimizing the communication overhead and the video memory copy overhead based on message merging and layout optimization.

6. The method according to claim 5, characterized in that The step of calculating the hot switching cost between any two sets of parallel strategy combinations on the grouped data includes: For each device i, record the internal communication bandwidth B (intra) and inter-machine communication bandwidth B (inter) , and the current intra-machine communication volume V i (intra) and the amount of inter-machine communication V (inter) ; Traverse each model parameter / gradient slice, and determine the device set S that owns the slice and the device set D that needs the model parameter / gradient slice based on the distributed state of the current policy and the target policy; Traverse each device dst in the device set D, and divide the devices in the device set S into intra-device devices S according to the difference between intra-device and inter-device (intra) And machine room equipment (inter) ; In-machine equipment (intra) If the model parameters / gradient slices required by the device dst are available, the device S in the machine will be (intra) As the sender of the model parameters / gradient slices, and performs intra-machine communication V i (inter) Updates; In-machine equipment (inter) If the model parameters / gradient slices required by device dst are not available, then the inter-machine device S (inter) Select the inter-machine device S with the smallest current data transmission volume (inter) As the sender of the model parameters / gradient slices, and conducts inter-machine communication volume V i (inter) Updates.

7. The method according to claim 5, characterized in that The message merging process includes: Merging data sent to the same device into consecutive send buffers to generate merged send data; and, Merge the data received by the same device into consecutive receive buffers to generate merged received data, and, The BatchedSendRecv primitive supports parallel transmission of combined send data and combined receive data.

8. The method according to claim 5, characterized in that The layout optimization process includes: The vertically sliced ​​model parameters / gradient slices are laid out according to the horizontal slices, and the mathematical equivalence is ensured by changing the layout parameters of the general matrix multiplication kernel.

9. The method according to claim 1, characterized in that: The training of the large model based on the execution order between the initialization graph and the executable computation graph includes: Group the data in the global batch according to the sequence length and bind them to the corresponding executable graph j To get each set of data j ; Where j∈[1,n], n represents the total number of executable computational graphs; Initialize the model parameters and optimizer according to the initialization graph to obtain the initialized model parameters θ t＝0 ; Where t is the number of iterations; The model parameters θ t and the current cumulative gradient in the tth round of training From the executable computational graph Graph j-1 Hot switch to executable graph j and based on the data j In the executable computational graph Graph j Train on it and get the cumulative gradient According to the cumulative gradient For the model parameters θ t Update and get the model parameters θ t+1 ; Based on the model parameters θ t+1 If the large model does not meet the set requirements and the number of iterations t+1 does not exceed the maximum number of iterations, set t=t+1 and re-execute the method of changing the model parameters θ t and the current cumulative gradient in the tth round of training From the executable computational graph Graph j-1 Hot switch to executable graph j middle; Based on the model parameters θ t+1 The large model meets the set requirements, or the number of iterations t+1 exceeds the maximum number of iterations, and the output is based on the model parameters θ t+1 Large model.

10. A large model training system supporting parallel hot switching, characterized in that: The system comprises: A generating module, used for generating a logic calculation graph, wherein the logic calculation graph is used for representing a plurality of different parallel strategy combinations; A compiling module, used to compile the logic calculation graph to generate multiple executable calculation graphs; wherein each executable calculation graph corresponds to a set of parallel strategy combinations; The inference module is used to select an initialization graph for initializing the state of the large model from multiple executable computation graphs, and to arrange the execution order between the executable computation graphs according to the hot switching cost between any two sets of parallel strategy combinations; A training module is used to train a large model based on the execution order between the initialization graph and the executable computation graph.

Images