Method for estimating large language model training and reasoning time consumption

Abstract

The invention provides a method for estimating large language model training and reasoning time consumption, a hyper-parameter configuration reference is provided for large model training fine tuning, and a calculator can help a user to simplify the large model training process through automatic analysis and giving of appropriate parallel configuration requirements; a user does not need to manually research and try different configurations, but can select proper configurations according to suggestions of the calculator; according to the method, the computing power and time cost required by large model training are estimated, and a user can better understand and evaluate the cost and benefit of large model training by estimating the core index of large model training, so that the user can more easily try to use, and large model vertical application landing is promoted; gPU models and large models can be flexibly added, and the calculator not only can simulate training indexes under given configuration, but also can provide reference for a user to customize a model training plan. The user can adjust the configuration parameters in the training plan according to the estimation result of the calculator so as to achieve a better training effect and a better resource utilization rate.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method for estimating the time consumption of training and reasoning a large language model. Background Art

[0002] With the rapid development of deep learning technology, large language models are increasingly used in the field of natural language processing. However, the training and reasoning process of large models requires extremely high computing resources. How to efficiently use resources and accurately estimate training time and cost has become an urgent problem to be solved.

[0003] In terms of related technical background, large model training usually requires a lot of computing power and time, and is highly dependent on hardware resources (such as GPUs). In a distributed training environment, the choice of parallel configuration has an important impact on training efficiency and resource utilization. Therefore, how to choose the appropriate parallel configuration based on the characteristics of the large model and the hardware resources becomes the key to optimizing large model training.

[0004] Among the closest existing technologies, there are some methods for estimating the training time and resource requirements of large models. These methods are usually based on empirical formulas or simulation experiments, and output estimated training time and resource consumption by inputting information such as the number of model parameters and data size. Due to the large differences between different large models and the fact that the training process is affected by many factors, empirical formulas may not be able to fully and accurately reflect the actual situation, resulting in limited estimation accuracy; when faced with new large models or special training scenarios, empirical formulas may not be directly applicable, and data collection and formula fitting are required again. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings in the prior art and to propose a method for estimating the time consumption of training and reasoning a large language model.

[0006] To achieve the above purpose, the present invention adopts the following technical solution: a method for estimating the time consumption of training and reasoning of a large language model,

[0007] S1: Evaluation model parameters:

[0008] Determine the number of parameters of the target large language model, calculate the number of parameters of each component according to the architecture of the Transformer model, and finally derive the number of parameters of the overall model;

[0009] S2: Choose the appropriate degree of parallelism:

[0010] Choose the degree of parallelism for available computing resources during model training and fine-tuning, including the number of GPUs, memory size, and network bandwidth;

[0011] Including pipeline parallelism and tensor parallelism;

[0012] The pipeline parallelism refers to a method of decomposing a task into multiple consecutive subtasks and executing them in parallel through multiple processing units;

[0013] The tensor parallelism is a method for accelerating the training of large-scale models by dividing and distributing tensor computing tasks to multiple devices.

[0014] S3: Calculate video memory overhead:

[0015] Calculate the memory usage of each parameter during training, including weights, gradients, and optimizations for FP16 and FP32 precision. When pipeline parallelism or tensor parallelism is selected in step S2, evaluate the actual memory overhead of each GPU card and optimize memory allocation based on the impact of parallelism.

[0016] S4: Considering the time overhead of communication and calculation:

[0017] Calculate the amount of communication in each model training and evaluate the time cost of communication based on the model's tensor distribution, data transfer between GPUs, and network bandwidth;

[0018] Estimate the computational time of each forward propagation and backpropagation based on the model's structure, number of parameters, parallel strategy, and hardware computing power;

[0019] S5: Estimate the overall time required for model training and inference:

[0020] Based on the evaluations from step S1 to step S4, the overall time consumption of the model in the training and inference stages is calculated. Based on the model's hardware environment, parallelism settings, and communication requirements, the model's training and inference efficiency is estimated, and optimization suggestions are provided to shorten the training cycle and inference time.

[0021] Preferably, in step S2, the pipeline parallelization is specifically as follows:

[0022] S21-1: Use the PipeDream-flush pipeline scheduling algorithm to divide the model into several parts to ensure that the model can be trained in parallel through the pipeline;

[0023] S21-2: Calculate the parallelism of the pipeline and the number of microbatches, and ensure that the number of microbatches is at least greater than the pipeline length to fill the pipeline and reduce bubble time;

[0024] S21-3: Calculate the final batch size, where B is the product of data parallelism, number of gradient accumulation steps, number of microbatches, and microbatch size.

[0025] Preferably, in step S2, the tensor parallelization is specifically as follows:

[0026] S22-1: Optimize computation and communication efficiency by distributing different heads of the Attention layer to different GPUs to achieve tensor parallelism;

[0027] S22-2: Calculate the size of the Q, K, and V matrices of each head, and evaluate the computational and communication workload based on the model architecture and hardware parameters;

[0028] S22-3: Introduce tensor parallelism to optimize overall computing performance, and consider the limitations of GPU memory bandwidth and computing power on model parallelism;

[0029] S22-4: Determining the model parallelism is limited by the computing power and communication bandwidth of the GPU itself, and in practice is usually limited to the maximum parallelism within a single machine.

[0030] Preferably, in step S22-2, the calculation amount is specifically:

[0031] Computational amount = 2*3(Q, K, V)*minibatch size*token length = 2*3*B*s*h*(h / a).

[0032] Preferably, in step S22-2, the communication volume is specifically:

[0033] Communication volume = minibatch size * token length * hidden layer size / heads num * bytes per param = B * s * (h / a) * 2.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides targeted parallel configuration suggestions and training time estimates according to the characteristics of large models and hardware resource conditions, thereby improving the accuracy of estimation; the present invention fully considers the impact of parallel configuration in a distributed training environment on training efficiency and resource utilization, thereby making the estimation results closer to actual training conditions; the present invention allows users to flexibly add GPU models and large models, providing users with references for customized model training plans, thereby meeting the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of pipeline parallelism;

[0036] Figure 2 It is a schematic diagram of tensor parallelism;

[0037] Figure 3 This is the comparison picture of Experiment 1;

[0038] Figure 4 This is the comparison picture of Experiment 2;

[0039] Figure 5 This is the comparison figure for Experiment 3. DETAILED DESCRIPTION

[0040] In order to provide a further understanding of the purpose, structure, features, and functions of the present invention, the following detailed description is given in conjunction with the embodiments.

[0041] Please refer to Figure 1 ,The present invention provides a method for estimating the time consumption of training and reasoning of a large language model, S1: Evaluating the number of model parameters:

[0042] Determine the number of parameters of the target large language model, calculate the number of parameters of each component according to the architecture of the Transformer model, and finally obtain the number of parameters of the overall model, that is, the total number of model parameters.

[0043] Generally speaking, when evaluating the parameters of the Tranformer model, we consider the following components, and the corresponding calculation formula is as follows:

[0044] Word Embedding: h*v;

[0045] Indicates mapping each word to an h-dimensional vector; v is the size of the vocabulary, and h is the dimension of each word vector;

[0046] Self-Attention: 4*h*h;

[0047] Represents the weight matrix in the self-attention mechanism of each layer; the self-attention mechanism usually includes multiple weight matrices (such as query, key, value matrices), each of which has a size of h*h; therefore, the number of parameters of self-attention is 4*h*h, where 4 refers to the number of possible weight matrices in the self-attention mechanism;

[0048] Feed Forward (feedforward network): 8*h*h+5*h;

[0049] Represents the weight matrix and bias in the feedforward network; the feedforward network usually contains two fully connected layers, the first layer has 8*h*h parameters (usually the expanded dimension); the second layer has 5*h parameters (usually the bias term of a linear layer); therefore, the total number of parameters of the feedforward network is the sum of these two terms;

[0050] Position Embedding: h*s;

[0051] Represents the number of parameters for position encoding; the purpose of position encoding is to provide position information for each word, usually a vector of size s (s is the length of the sequence), and the encoding of each position is an h-dimensional vector, so the number of parameters for position encoding is

[0052] The total number of model parameters:

[0053] Among them: h*(v+s) is the parameter amount of word embedding and position encoding part; (12*h*h+5*h)*L is the parameter amount of self-attention mechanism and feedforward network part of each layer, multiplied by the number of layers L to calculate the parameter amount of multi-layer model.

[0054] S2: Choose the appropriate degree of parallelism:

[0055] Choose the degree of parallelism for available computing resources during model training and fine-tuning, including the number of GPUs, memory size, and network bandwidth;

[0056] Including pipeline parallelism and tensor parallelism;

[0057] The pipeline parallelism refers to a method of decomposing a task into multiple continuous subtasks and executing them in parallel through multiple processing units.

[0058] The pipeline parallelism is specifically as follows:

[0059] S21-1: Use the PipeDream-flush pipeline scheduling algorithm to divide the model into several parts to ensure that the model can be trained in parallel through the pipeline;

[0060] S21-2: Calculate the parallelism of the pipeline and the number of microbatches, and ensure that the number of microbatches is at least greater than the pipeline length to fill the pipeline and reduce bubble time;

[0061] For a pipeline of length p, the number of microbatches m must be at least greater than p, and the bubble time consumed by filling the pipeline is (p-1) / m. In order to make the proportion of bubble time small enough, the present invention sets m to be more than 4 times of p.

[0062] S21-3: Calculate the final batch size, where B is the product of data parallelism, number of gradient accumulation steps, number of microbatches, and microbatch size.

[0063] The final batch size (B) of the model is equal to the data parallelism (d) multiplied by the number of gradient accumulation steps (g) multiplied by the number of microbatches (m) multiplied by the microbatch size (b), that is, B = d*g*m*b, so too long pipeline steps will also cause B to be too large.

[0064] The tensor parallelism is a method for accelerating the training of large-scale models by dividing and distributing tensor computing tasks to multiple devices.

[0065] Tensor parallelism divides the model's layer structure more finely, and we can split multiple attention heads onto different GPUs.

[0066] The sizes of the Q and K matrices in each head are:

[0067] minibatch size*token length*key size;

[0068] The size of the V matrix is ​​the size of minibatch size*token length*value (just one of the heads).

[0069] The size of key / value is generally equal to the hidden layer size / number of heads. The size of the Q, K, and V parameter matrices on each head is:

[0070] hidden layer size*hidden layer size / head num.

[0071] S22-1: Optimize computation and communication efficiency by distributing different heads of the Attention layer to different GPUs to achieve tensor parallelism;

[0072] S22-2: Calculate the size of the Q, K, and V matrices of each head, and evaluate the computational and communication volume based on the model architecture and hardware parameters.

[0073] Computational amount = 2*3(Q, K, V)*minibatch size*token length = 2*3*B*s*h*(h / a).

[0074] Communication volume = minibatch size * token length * hidden layer size / heads num * bytes per param = B * s * (h / a) * 2.

[0075] S22-3: Introduce tensor parallelism to optimize overall computing performance, and consider the limitations of GPU memory bandwidth and computing power on model parallelism;

[0076] S22-4: Determining the model parallelism is limited by the computing power and communication bandwidth of the GPU itself, and in practice is usually limited to the maximum parallelism within a single machine.

[0077] Tensor parallelism (t) is introduced to ensure that computation time and communication time are as balanced as possible, which is conducive to overlapping computation and communication:

[0078] Let h'=h / t, and we get t≤3*h*simplex memory bandwidth / GPU computing power;

[0079] It can be seen that the model parallelism that different types of GPUs can support is actually limited by the computing power and communication bandwidth of the GPU itself; generally speaking, in order to ensure the availability of NVLink, model parallel communication is limited to a single machine.

[0080] S3: Calculate video memory overhead:

[0081] Calculate the memory usage of each parameter during training, including weights, gradients, and optimizations of FP16 and FP32 precision. When pipeline parallelism or tensor parallelism is selected in step S2, consider the impact of parallelism, evaluate the actual memory overhead of each GPU card, and optimize memory allocation.

[0082] S4: Considering the time overhead of communication and calculation:

[0083] Calculate the amount of communication in each model training and evaluate the time cost of communication based on the model's tensor distribution, data transfer between GPUs, and network bandwidth;

[0084] Estimate the computational time for each forward propagation and backpropagation based on the model's structure, number of parameters, parallel strategy, and hardware computing power.

[0085] In deep learning model (Transformer) training, common parallel methods include pipeline parallelism and tensor parallelism. The two parallel strategies are often used in combination to better utilize multiple GPUs or machines.

[0086] In pipeline parallelism, data transmission uses P2P (peer-to-peer) communication between GPUs, that is, when a GPU completes the calculation of a microbatch, it immediately passes its activation value (activation) and gradient (gradient) to the adjacent rank. Assume that the transmission volume of each microbatch is 2*h*h*(B / t)*8=16Bh 2 / t, where: h represents the hidden state dimension of each layer; B represents the batch size; t is the parallelism of the pipeline, that is, the number of devices the pipeline is divided into; h is the hidden state dimension processed by each GPU 8 is used to convert floating-point numbers to bytes.

[0087] During the computation of each microbatch, tensor parallelism involves data exchange between each GPU, which is specifically manifested in the following communication modes:

[0088] AllGather: Before each linear layer calculation of each Transformer layer, an AllGather operation is performed to allow each GPU to obtain the data of this layer on other devices.

[0089] For each forward calculation, the AllGather communication volume is:

[0090] 2*h*(h / t)*b*t=2bh 2 ,

[0091] Where: b represents the batch size; t is the degree of tensor parallelism

[0092] AllGather in Backward: During the back propagation process, an AllGather operation will also occur, and the communication volume is:

[0093] 2*h*(h / t)*b*t=2bh 2 ;

[0094] Reduce-Scatter: During the back-propagation process, each GPU needs to participate in the Reduce-Scatter operation, that is, each GPU aggregates the gradients in segments. The communication volume is: 2*h*(h / t)*b*2=4bh 2 / t.

[0095] Therefore, in each Transformer layer, communication operations occur four times per microbatch:

[0096] One AllGather is used before forward calculation;

[0097] One AllGather is used in backward calculation;

[0098] A Reduce-Scatter is used for backward calculation;

[0099] There are L / p layers on each GPU, where L is the number of layers of the Transformer model and p is the tensor parallelism. For each microbatch, the total communication volume is:

[0100] 16bh 2 (1+1 / t)Lm / p;

[0101] m is the number of microbatches; p is the degree of tensor parallelism; L is the number of model layers; t is the degree of pipeline; h is the hidden state dimension allocated to each GPU.

[0102] It can be seen that the traffic volume is mainly affected by the following factors:

[0103] Number of layers in the model: As the number of layers increases, the amount of communication increases;

[0104] Number of microbatches: More microbatches means higher communication volume;

[0105] Tensor parallelism: The greater the tensor parallelism, the smaller the amount of data that each GPU needs to transmit, but the total communication volume will increase;

[0106] Pipeline parallelism: When the pipeline parallelism is high, it can effectively share the communication burden and reduce the load on a single GPU.

[0107] The computational cost is usually related to the number of parameters on each GPU and the computational requirements of each layer. On each GPU, the number of parameters is assumed to be s / pt, where s is the total number of parameters of the model, p is the tensor parallelism, and t is the pipeline parallelism.

[0108] The cost of forward calculation is Where B is the batch size.

[0109] The cost of Backward calculation is

[0110] The main factors in computing overhead are:

[0111] The number of parameters in the model: more parameters means more computation;

[0112] Batch size: A larger batch size will increase the computational burden;

[0113] Tensor parallelism and pipeline parallelism: These two affect the computational burden of each GPU. S5: Estimate the overall time consumption of model training and inference:

[0114] Based on the evaluations from step S1 to step S4, the overall time consumption of the model in the training and inference stages is calculated. Based on the model's hardware environment, parallelism settings, and communication requirements, the model's training and inference efficiency is estimated, and optimization suggestions are provided to shorten the training cycle and inference time.

[0115] Specific experiments:

[0116] (1) Experiment 1

[0117] Calculation input:

[0118] Cluster: RTX3090; Bus Bandwidth (GB / s): 20; Per-host network bandwidth (Gb / s): 100

[0119] Model: GPT-3XL; Minibatch size: 32

[0120] Others: Optimization strategy: Full recomputation;

[0121] Tensor parallel degree: 1;

[0122] Pipeline parallel degree: 8;

[0123] Microbatch size: 2

[0124] intput: Total number of tokens: 1000;

[0125] Data parallel degree: 2; Number of epochs: 2

[0126] The calculation results are as follows:

[0127] 3090_GPT-3XL is the calculation result: Each iteration: 2.81s

[0128] Benchmark is the actual result: Each iteration: 2.96s

[0129] The error between actual and estimated time is 5%.

[0130] (2) Experiment 2

[0131] Calculation input:

[0132] Cluster: RTX3090; Bus Bandwidth (GB / s): 20; Per-host network

[0133] Bandwidth(Gb / s): 100

[0134] Model: GPT-3XL; Minibatch size: 32

[0135] Others: Optimization strategy: Full recomputation;

[0136] Tensor parallel degree: 4;

[0137] Pipeline parallel degree: 2;

[0138] Microbatch size: 2

[0139] intput: Total number of tokens: 1000;

[0140] Data parallel degree: 2; Number of epochs: 2

[0141] The calculation results are as follows:

[0142] 3090_GPT-3XL is the calculation result: Each iteration: 9.06s

[0143] Benchmark 2 is the actual result: Each iteration: 9.8547s

[0144] The error between actual and estimated time consumption is 8%.

[0145] (3) Experiment 3

[0146] Calculation input:

[0147] Cluster: RTX3090; Bus Bandwidth (GB / s): 20; Per-host network

[0148] Bandwidth(Gb / s): 100

[0149] Model: GPT-3XL; Minibatch size: 32

[0150] Others: Optimization strategy: Full recomputation;

[0151] Tensor parallel degree: 2;

[0152] Pipeline parallel degree: 4;

[0153] Microbatch size: 2

[0154] intput: Total number of tokens: 1000;

[0155] Data parallel degree: 2; Number of epochs: 2

[0156] The calculation results are as follows:

[0157] 3090_GPT-3XL is the calculation result: Each iteration: 5.52s

[0158] Benchmark 3 is the actual result: Each iteration: 5.4570s

[0159] The error between actual and estimated time consumption is 1%.

[0160] The overall experiment was repeated three times, with an average error of 4.6%. The difference in error between the estimated time and the actual time consumed was small, with an average error of 4.6%. The error ranged from 1% to 8%, indicating that the prediction accuracy of the computational model was relatively high, but the error increased slightly with the increase in parallelism (for example, in Experiment 2). Ultimately, these results show that under different configurations, the model's computational time estimate is closer to the actual time consumed, and the impact of optimization strategies and parallelism on performance is worthy of further in-depth analysis.

[0161] The method of the present invention provides users with estimates of core indicators of large model training, such as time overhead, video memory overhead, and resource utilization, by analyzing and simulating the parallel configuration requirements of large language models in the distributed training process. Compared with the prior art, the present invention has the following advantages:

[0162] 1. Strong targeting: The present invention provides targeted parallel configuration suggestions and training time estimation based on the characteristics of large models and hardware resources, thereby improving the accuracy of estimation.

[0163] 2. Considering parallel configuration: The present invention fully considers the impact of parallel configuration in a distributed training environment on training efficiency and resource utilization, so that the estimation results are closer to the actual training situation.

[0164] 3. Support for custom models: The present invention allows users to flexibly add GPU models and large models, providing users with references for custom model training plans, meeting the diverse needs of users.

[0165] In summary, the large model training calculator proposed in the present invention has made targeted improvements and innovations based on the relevant technical background and the closest existing technology, providing a more accurate and efficient method for time-consuming calculations of large model training and reasoning.

[0166] The present invention has been described by the above-mentioned relevant embodiments, however, the above-mentioned embodiments are only examples for implementing the present invention. It must be pointed out that the disclosed embodiments do not limit the scope of the present invention. On the contrary, changes and modifications made without departing from the spirit and scope of the present invention are all within the scope of patent protection of the present invention.

Claims

1. A method for estimating the time consumption of training and reasoning of a large language model, characterized by: S1: Evaluation model parameters: Determine the number of parameters of the target large language model, calculate the number of parameters of each component according to the architecture of the Transformer model, and finally obtain the number of parameters of the overall model; S2: Choose the appropriate degree of parallelism: Select the parallelism for available computing resources during model training and fine-tuning, including the number of GPUs, memory size, and network bandwidth; Including pipeline parallelism and tensor parallelism; The pipeline parallelism refers to a method of decomposing a task into multiple consecutive subtasks and executing them in parallel through multiple processing units; The tensor parallelism is a method for accelerating the training of large-scale models by dividing and distributing tensor computing tasks to multiple devices. S3: Calculate video memory overhead: Calculate the memory usage of each parameter during training, including weights, gradients, and optimizations for FP16 and FP32 precision. When pipeline parallelism or tensor parallelism is selected in step S2, evaluate the actual memory overhead of each GPU card and optimize memory allocation based on the impact of parallelism. S4: Considering the time overhead of communication and calculation: Calculate the amount of communication in each model training and evaluate the time cost of communication based on the model's tensor distribution, data transfer between GPUs, and network bandwidth; Estimate the computational time of each forward propagation and backpropagation based on the model's structure, number of parameters, parallel strategy, and hardware computing power; S5: Estimate the overall time required for model training and inference: Based on the evaluations from step S1 to step S4, the overall time consumption of the model in the training and inference stages is calculated. Based on the model's hardware environment, parallelism settings, and communication requirements, the model's training and inference efficiency is estimated, and optimization suggestions are provided to shorten the training cycle and inference time.

2. A method for estimating the time consumption of training and reasoning a large language model according to claim 1, characterized in that: In step S2, the pipeline parallelization is specifically as follows: S21-1: Use the PipeDream-flush pipeline scheduling algorithm to divide the model into several parts to ensure that the model can be trained in parallel through the pipeline; S21-2: Calculate the parallelism of the pipeline and the number of microbatches, and ensure that the number of microbatches is at least greater than the pipeline length to fill the pipeline and reduce bubble time; S21-3: Calculate the final batch size, where B is the product of data parallelism, number of gradient accumulation steps, number of microbatches, and microbatch size.

3. A method for estimating the time consumption of training and reasoning a large language model according to claim 1, characterized in that: In step S2, the tensor parallelization is specifically as follows: S22-1: Optimize computation and communication efficiency by distributing different heads of the Attention layer to different GPUs to achieve tensor parallelism; S22-2: Calculate the size of the Q, K, and V matrices of each head, and evaluate the computational and communication workload based on the model architecture and hardware parameters; S22-3: Introduce tensor parallelism to optimize overall computing performance, and consider the limitations of GPU memory bandwidth and computing power on model parallelism; S22-4: Determining the model parallelism is limited by the computing power and communication bandwidth of the GPU itself, and in practice is usually limited to the maximum parallelism within a single machine.

4. A method for estimating the time consumption of training and reasoning of a large language model as claimed in claim 3, characterized in that: In step S22-2, the calculation amount is specifically: Computational amount = 2*3 (Q, K, V) * minibatch size * token length = 2 * 3 * B * s * h * (h / a).

5. The method for estimating the time consumption of training and reasoning of a large language model according to claim 1, characterized in that: In step S22-2, the communication volume is specifically: l Communication amount = minibatch size * token length * hidden layer size / heads num * bytes per param = B * s * (h / a) * 2.

Images