Method, computing device, storage medium and computer program product for evaluating model training performance

Abstract

Embodiments of the present invention relate to a method, computing device, storage medium and computer program product for evaluating model training performance. The method includes: determining the deployment strategy of the model on multiple boards, the deployment strategy including data parallel strategy, tensor parallel strategy and pipeline parallel strategy; determining the shape of the operator at least based on the tensor parallel strategy; determining the proportion of cavitations corresponding to the pipeline parallel strategy at least based on the data parallel strategy and the pipeline parallel strategy; determining the inter-group attenuation between parallel data groups at least based on the tensor parallel strategy and the pipeline parallel strategy; and calculating the end-to-end consumption time required for model training at least based on the shape of the operator, the proportion of cavitations and the inter-group attenuation, so as to evaluate the model training performance based on the end-to-end consumption time. The embodiments of the present invention can comprehensively consider the deployment strategy of the model and the overhead of the computing operator and the communication operator when evaluating the model training performance, thereby improving the evaluation efficiency.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of large language models, and more specifically to a method, computing device, storage medium, and computer program product for evaluating model training performance. Background Art

[0002] Before deploying a large language model (LLM) to be trained on a large-scale cluster, the model training performance must be evaluated to determine whether performance optimization is needed. Generally speaking, the model training performance can be evaluated from the following aspects: the deployment strategy of the large language model on a large-scale cluster, the overhead of the computing operator and the communication operator, etc. However, in related technologies, different developers usually evaluate the above aspects separately, resulting in a lot of repeated evaluation and sorting work, which is relatively inefficient.

[0003] In summary, the shortcomings of the solutions used to evaluate model training performance in related technologies are: the lack of a systematic solution to evaluate model training performance, which makes it difficult to comprehensively consider the model's deployment strategy and the overhead of computing operators and communication operators when evaluating model training performance. Summary of the invention

[0004] In response to the above problems, the present invention provides a method, computing device, storage medium and computer program product for evaluating model training performance, so that the model deployment strategy and the overhead of computing operators and communication operators can be comprehensively considered when evaluating model training performance, thereby improving evaluation efficiency.

[0005] According to a first aspect of the present invention, a method for evaluating model training performance is provided, the method comprising: determining a deployment strategy of a model on multiple boards, the deployment strategy comprising a data parallel strategy, a tensor parallel strategy and a pipeline parallel strategy; determining a shape of an operator at least based on the tensor parallel strategy; determining a bubble ratio corresponding to the pipeline parallel strategy at least based on the data parallel strategy and the pipeline parallel strategy; determining inter-group attenuation between parallel data groups at least based on the tensor parallel strategy and the pipeline parallel strategy; and calculating the end-to-end consumption time required for model training at least based on the shape of the operator, the bubble ratio and the inter-group attenuation, so as to evaluate the model training performance based on the end-to-end consumption time.

[0006] In some embodiments, determining a deployment strategy of a model on multiple boards includes: obtaining model parameters of a model to be trained, wherein the model parameters include: sequence length, hidden layer size, hidden layer size of a feedback neural network, and number of attention heads; based on the model parameters, determining training parameters and a deployment strategy, wherein the training parameters include: micro-batch size and global batch size.

[0007] In some embodiments, determining the shape of the operator based at least on the tensor parallel strategy includes determining the shape of the operator based on a mini-batch size, a sequence length, a hidden layer size of the feedback neural network, and the tensor parallel strategy.

[0008] In some embodiments, determining the shape of the operator based at least on the tensor parallel strategy includes determining the shape of the operator based at least on a mini-batch size, a sequence length, a number of attention heads, and a tensor parallel strategy.

[0009] In some embodiments, determining the bubble ratio corresponding to the pipeline parallel strategy based at least on the data parallel strategy and the pipeline parallel strategy includes: determining the bubble ratio corresponding to the pipeline parallel strategy based on the data parallel strategy, the pipeline parallel strategy, the micro-batch size, and the global batch size.

[0010] In some embodiments, the operator includes a communication operator and a computing operator. In these embodiments, evaluating the model training performance based on the end-to-end consumption time also includes: for the computing operator, calculating the computing time corresponding to the computing operator to determine the performance upper limit of the computing operator; and for the communication operator, calculating the communication overhead corresponding to the communication operator to determine the performance upper limit of the communication operator.

[0011] In some embodiments, the computation time includes: computation instruction time and memory access time.

[0012] In some embodiments, calculating the communication overhead corresponding to the communication operator includes: determining the amount of data used for communication based at least on the shape and data type of the communication operator; determining the communication bandwidth based at least on the type and deployment strategy of the communication operator; and calculating the communication overhead based on the data amount and the communication bandwidth.

[0013] In some embodiments, the method of the first aspect of the present invention also includes: determining multiple candidate deployment strategies for deploying the model to be trained; for each of the multiple candidate deployment strategies, calculating the end-to-end consumption time required for the corresponding model training; and comparing the multiple end-to-end consumption times calculated for the multiple candidate deployment strategies to determine the best deployment strategy from the multiple candidate deployment strategies.

[0014] In some embodiments, determining the inter-group attenuation between parallel data groups includes: splitting the model based on the tensor parallel strategy to determine a first number after the model is split based on the tensor parallel strategy; splitting the model based on the pipeline parallel strategy to determine a second number after the model is split based on the pipeline parallel strategy; and determining a third number of the multiple boards; and determining the inter-group attenuation between the parallel data groups based on the first number, the second number and the third number.

[0015] According to a second aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the method of the first aspect of the present invention.

[0016] According to a third aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method of the first aspect of the present invention.

[0017] According to a fourth aspect of the present invention, there is provided a computer program product, wherein the computer program product is tangibly stored on a non-transitory computer readable medium and comprises machine executable instructions which, when executed, cause a machine to perform the steps in the method of the first aspect of the present invention.

[0018] The solution for evaluating model training performance provided by an embodiment of the present invention can comprehensively consider the deployment strategy of the model and the overhead of computing operators and communication operators, determine the theoretical performance upper limit of the model on a large-scale cluster, and realize the evaluation of model training performance.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0021] Figure 1 A schematic diagram of a system for evaluating model training performance according to an embodiment of the present invention is shown.

[0022] Figure 2 A flowchart of a method for evaluating model training performance according to an embodiment of the present invention is shown.

[0023] Figure 3 A schematic diagram showing the cavitation ratio corresponding to different pipeline parallel strategies.

[0024] Figure 4 A schematic diagram showing the execution process of forward propagation and back propagation during model training. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0027] As we all know, large language models (LLMs) are usually implemented based on the Transformer architecture. However, as the parameter scale of large language models continues to increase, the training scale of large language models also increases, so large language models need to be trained on large-scale clusters such as the Doka cluster. Usually, in order to improve training efficiency, before deploying the large language model to be trained on a large-scale cluster, the model training performance needs to be evaluated to determine whether performance optimization is needed.

[0028] Generally speaking, the model training performance can be evaluated from at least the following three aspects: first, the deployment strategy of the large language model on a large-scale cluster (also known as the "distributed training strategy"), which refers to the strategy used to split the large language model into multiple parts and distribute them to multiple boards in a large-scale cluster for training; second, the overhead of the computing operator, which can refer to the computational time of the algorithm on each board in a large-scale cluster; third, the overhead of the communication operator, which can refer to the communication time between multiple boards in a large-scale cluster.

[0029] However, in related technologies, model developers usually evaluate the deployment strategy of the model, and operator developers evaluate the overhead of the operator. In other words, the three aspects of evaluating model training performance are independently performed by different developers, resulting in a large amount of repeated evaluation and sorting work, which is relatively inefficient. In addition, the theoretical performance of the evaluation of operator overhead depends on the boards, interconnection structures between boards, servers, etc. provided by hardware manufacturers. When evaluated by operator developers, it may not reach the optimal state, making it difficult to reflect the actual performance.

[0030] In summary, the solutions used in related technologies to evaluate model training performance are inefficient and fail to comprehensively consider the model deployment strategy and the overhead of computing operators and communication operators.

[0031] In order to at least partially solve one or more of the above problems and other potential problems, an example embodiment of the present invention proposes a scheme for evaluating model training performance. In this scheme, by determining the deployment strategy of the model on multiple boards, the deployment strategy includes data parallel strategy, tensor parallel strategy and pipeline parallel strategy; at least based on the tensor parallel strategy, determine the shape of the operator; at least based on the data parallel strategy and the pipeline parallel strategy, determine the bubble ratio corresponding to the pipeline parallel strategy; at least based on the tensor parallel strategy and the pipeline parallel strategy, determine the inter-group attenuation between parallel data groups; and at least based on the shape of the operator, the bubble ratio and the inter-group attenuation, calculate the end-to-end consumption time required for model training, so that the deployment strategy of the model and the overhead of the computing operator and the communication operator can be comprehensively considered, and the theoretical performance upper limit of the model on a large-scale cluster can be determined to achieve the evaluation of the model training performance.

[0032] The following will be combined Figure 1 to Figure 2 , describes in detail a scheme for evaluating model training performance provided by an embodiment of the present invention

[0033] Figure 1 A schematic diagram of a system 100 for evaluating model training performance according to an embodiment of the present invention is shown. It should be understood that the system 100 may also include additional modules not shown and / or may omit the modules shown, and the scope of the present invention is not limited in this respect.

[0034] like Figure 1 As shown, the system 100 may include: a deployment strategy determination module 101, an operator shape determination module 102, a cavitation ratio determination module 103, a parallel data group attenuation determination module 104 and a model end-to-end consumption time determination module 105.

[0035] Regarding the deployment strategy determination module 101, it can be configured to: determine the deployment strategy of the model on multiple boards, wherein the deployment strategy can include a data parallel strategy, a tensor parallel strategy and a pipeline parallel strategy.

[0036] Regarding the operator shape determination module 102 , it may be configured to: determine the shape of the operator based at least on a tensor parallel strategy.

[0037] Regarding the cavitation ratio determination module 103, it can be configured to: determine the cavitation ratio corresponding to the pipeline parallel strategy based at least on the data parallel strategy and the pipeline parallel strategy.

[0038] The parallel data group attenuation determination module 104 may be configured to determine the group attenuation between parallel data groups based at least on the tensor parallel strategy and the pipeline parallel strategy.

[0039] Regarding the model end-to-end consumption time determination module 105, it can be configured to: calculate the end-to-end consumption time required for model training based on at least the shape of the operator, the bubble ratio and the inter-group attenuation.

[0040] Figure 2 A flow chart of a method 200 for evaluating model training performance according to an embodiment of the present invention is shown. It should be understood that the method 200 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0041] In step 201, a deployment strategy of the model on multiple boards is determined.

[0042] Regarding the deployment strategy, it may refer to a strategy for splitting the model into multiple parts to distribute them to multiple boards for training. That is to say, in an embodiment of the present invention, the deployment strategy may be related to factors such as the number of boards and the scale of the model. For example, a multi-card cluster for training a model may include a predetermined number of boards, such as 1024 boards. In this case, the model can be split into multiple parts based on the deployment strategy, and these parts can be distributed to these 1024 boards for training. According to an embodiment of the present invention, the deployment strategy may further include: a data parallel (DP) strategy, a tensor parallel (TP) strategy, and a pipeline parallel (PP) strategy.

[0043] Regarding the data parallel strategy, it refers to a strategy for distributing the data used for model training to different boards for parallel computing. In an embodiment of the present invention, the data parallel strategy may be related to the video memory occupancy of model parameters, optimizer parameters and gradient parameters. Specifically, the number of boards required to be occupied by each data parallel group (DP group) can be determined based on the video memory occupancy of model parameters, optimizer parameters and gradient parameters, and the number of data parallel groups can be further determined in combination with the total number of boards in the multi-card cluster. For example, according to the video memory occupancy of model parameters, optimizer parameters and gradient parameters, the 128 boards required to be occupied by each data parallel group are determined. For a multi-card cluster including 1024 boards, the data parallel strategy can indicate that the data used for model training is split into 8 data parallel groups, and these 8 data parallel groups are distributed to different boards for parallel computing.

[0044] The tensor parallel strategy refers to a strategy that splits the operations involved in the same layer of the model to be trained and distributes them to different boards for parallel operations. In other words, by adopting the tensor parallel strategy, the tensors involved in model training (for example, weight matrices or multidimensional arrays) can be split and distributed to different boards for parallel operations.

[0045] The pipeline parallel strategy refers to a strategy that distributes operations corresponding to different layers of the model to be trained to different boards for parallel operations. In other words, by adopting the pipeline parallel strategy, the model can be divided into multiple parts according to the layer order, and each part of the model can be distributed to different boards in sequence to form a pipeline for parallel operations.

[0046] In an embodiment of the present invention, the tensor parallel strategy and the pipeline parallel strategy can be determined based on the determined data parallel strategy. Generally, the product of the number of tensor parallel groups (TP groups) obtained by splitting the model based on the tensor parallel strategy and the number of pipeline parallel groups (PP groups) obtained by splitting the model based on the pipeline parallel strategy is equal to the number of boards occupied by a data parallel group. Taking the above example, since the number of boards occupied by each data parallel group is 128, the product of the number of tensor parallel groups and the number of pipeline parallel groups needs to be 128, for example, the number of tensor parallel groups is 8 and the number of pipeline parallel groups is 16; or the number of tensor parallel groups is 4 and the number of pipeline parallel groups is 32; or the number of tensor parallel groups is 16 and the number of pipeline parallel groups is 8. In some embodiments, the number of tensor parallel groups and the number of pipeline parallel groups can be further determined in combination with factors such as load balancing.

[0047] In summary, according to an embodiment of the present invention, the deployment strategy of the model to be trained on multiple boards can be determined based on at least the number of boards in a large-scale cluster, the scale of the model to be trained, etc., that is, the data parallel strategy, tensor parallel strategy and pipeline parallel strategy expected to be adopted by the model to be trained on multiple boards can be determined. For example, based on the size of a certain model to be trained and the number of available boards, it can be determined how to split the model to be trained into multiple parts and distribute them on multiple boards, so that the data parallel strategy, tensor parallel strategy and pipeline parallel strategy expected to be adopted on these multiple boards can be determined. In the example described above, the data parallel strategy of the model to be trained on multiple boards can be determined at least based on the number of boards and the scale of the model to be trained, and based on the determined data parallel strategy, the tensor parallel strategy and pipeline parallel strategy applicable to the model to be trained are selected.

[0048] According to an embodiment of the present invention, the deployment strategy may also include a sequence parallel (SP) strategy, that is, a strategy for performing column segmentation on the input sequence of the model for parallel operation. When the input sequence for model training is a long text sequence, the long text sequence may be segmented to distribute the long text sequence to multiple boards for processing, so as to reduce the sequence length processed by a single board.

[0049] Furthermore, in some embodiments of the present invention, determining a deployment strategy for a model to be trained on multiple boards may further include: obtaining model parameters of the model to be trained; and determining training parameters and a deployment strategy for the model to be trained based on the obtained model parameters.

[0050] Regarding model parameters, they refer to parameters that can be learned and adjusted in the model, and these parameters can be learned and optimized through training data. According to an embodiment of the present invention, model parameters may include: sequence length (SEQUENCE_LENGTH), hidden layer size (HIDDEN_SIZE), hidden layer size of feedback neural network (FFN_HIDDEN_SIZE), number of attention heads in multi-head attention mechanism (NHEADS), etc. In some embodiments of the present invention, model parameters may also include, for example, single attention head size (HEAD_SIZE).

[0051] Regarding training parameters, they refer to parameters that can affect the training performance (such as speed, effect) of the model. According to an embodiment of the present invention, the training parameters may include: micro batch size (MICRO_BATCH_SIZE), global batch size (GLOBAL_BATCH_SIZE), etc.

[0052] Regarding the micro-batch size, it refers to the number of batches processed by each board in each iteration during model training.

[0053] Regarding the global batch size, it refers to the number of batches that can be processed by one gradient update during model training.

[0054] From the above, it can be seen that the characteristics of the model to be trained can be determined by obtaining the model parameters of the model to be trained, and then the specific method for training the model can be determined based on these characteristics, for example, the training parameters for training the model, the deployment strategy for deploying the model on multiple boards, etc.

[0055] At step 202 , the shape of the operator is determined based at least on the tensor parallel strategy.

[0056] Regarding operators, they may include communication operators and computation operators.

[0057] Regarding the communication operator, it can be used for data interaction between boards during model training. According to an embodiment of the present invention, the communication operator may include, for example, an All-reduce operator, an All-gather operator, a Scatter operator, a Reduce-Scatter operator, a Send function, and a Recv function.

[0058] Regarding the calculation operator, it can be used to perform calculation tasks during the model training process. According to an embodiment of the present invention, the calculation operator can be used for pure density (Dense) calculation, such as linear (Linear) function calculation, matrix multiplication accumulation (MMA) calculation, etc. In some other embodiments, the calculation operator can be used for vector core (VCore) fusion calculation, such as root mean square normalization (RMSNorm) calculation, etc. In some other embodiments, the calculation operator can also be used for other complex calculations, such as attention calculation.

[0059] Regarding determining the shape of an operator, it refers to the dimension of the operator. Usually, an operator has an original shape, but after a tensor parallel strategy, the operator will be split into a new shape. According to an embodiment of the present invention, the shape of the operator can be determined based on the model parameters and the training parameters, and at least based on the determined tensor parallel strategy, that is, the shape of the split operator is determined.

[0060] For example, in one example, for the operator of the first linear calculation (or matrix multiplication accumulation calculation, or general matrix multiplication calculation) in the multi-layer perceptron MLP of the model, the original shape of the operator is related to the micro-batch size (MICRO_BATCH_SIZE) in the training parameters, the sequence length (SEQUENCE_LENGTH) in the model parameters, and the hidden layer size (FFN_HIDDEN_SIZE) of the feedback neural network. Specifically, the original shape of the operator can be expressed as: [MICRO_BATCH_SIZE, SEQUENCE_LENGTH, FFN_HIDDEN_SIZE]. Further, based on the tensor parallel strategy, the number (TP_SIZE) of the model after being split based on the tensor parallel strategy can be determined, and then the shape of the operator after the split can be determined as: [MICRO_BATCH_SIZE, SEQUENCE_LENGTH, FFN_HIDDEN_SIZE / TP_SIZE]. In other words, splitting the operator based on the tensor parallel strategy is essentially splitting the weight of the training. Therefore, after the operator is split based on the tensor parallel strategy, the shape of its weight will change from [HIDDEN_SIZE, FFN_HIDDEN_SIZE] to [HIDDEN_SIZE, FFN_HIDDEN_SIZE / TP_SIZE].

[0061] For example, in another example, for the shape of the operator of the query (Q) matrix of the attention mechanism used for model training, the original shape of the operator is related to the sequence length (SEQUENCE_LENGTH) in the model parameters, the number of attention heads (NHEADS) in the multi-head attention mechanism, and the size of a single attention head (HEAD_SIZE), and the micro-batch size (MICRO_BATCH_SIZE) in the training parameters. Specifically, the shape of the operator can be expressed as: [MICRO_BATCH_SIZE × NHEADS, SEQUENCE_LENGTH, HEAD_SIZE]. Further, based on the tensor parallel strategy, the number of splits (TP_SIZE) after the model is split based on the tensor parallel strategy is determined. On this basis, the shape of the split operator can then be determined as: [MICRO_BATCH_SIZE×NHEADS / TP_SIZE, SEQUENCE_LENGTH, HEAD_SIZE].

[0062] In some cases, the shape of the operator is also affected by the sequence parallel strategy. Therefore, according to some other embodiments of the present invention, the shape of the operator can also be determined based on the determined sequence parallel strategy. For example, in another example, for the shape of the operator used for root mean square normalization (RMSNorm) calculation, the original shape of the operator is related to the micro-batch size (MICRO_BATCH_SIZE) in the training parameters and the sequence length (SEQUENCE_LENGTH) and hidden layer size (HIDDEN_SIZE) in the model parameters. Specifically, the original shape of the operator can be expressed as: [MICRO_BATCH_SIZE, SEQUENCE_LENGTH, HIDDEN_SIZE]. Further, based on the sequence parallel strategy, the number (SP_SIZE) of splits of the model based on the sequence parallel strategy can be determined, and then the shape of the split operator can be determined as: [(MICRO_BATCH_SIZE × SEQUENCE_LENGTH) / SP_SIZE, HIDDEN_SIZE]. Taking the model as an example, the number of splits after the sequential parallel strategy is 8, that is, SP_SIZE=8, the shape of the operator after splitting is: [(MICRO_BATCH_SIZE× SEQUENCE_LENGTH) / 8, HIDDEN_SIZE].

[0063] As can be seen from the above, according to an embodiment of the present invention, the shape of the communication operator and the shape of the computing operator can be determined respectively based on at least the tensor parallel strategy. Further, based on the determined shape of the communication operator and the shape of the computing operator, the performance of the computing operator and the communication operator can be evaluated respectively. Specifically, in an embodiment of the present invention, for a computing operator, the computing time corresponding to the computing operator can be calculated to determine the performance upper limit of the computing operator; and for a communication operator, the communication overhead corresponding to the communication operator can be calculated to determine the performance upper limit of the communication operator.

[0064] Regarding the computational time consumption, it may include, for example, computational instruction time consumption, memory access time consumption, etc. According to an embodiment of the present invention, the performance upper limit of each computational operator can be determined by calculating the computational time consumption (also known as computational overhead) of each computational operator. For example, when the computational operator is an operator used for pure density calculation, the computational overhead of the operator can be calculated by constructing a cost model (Cost Model) to obtain the theoretical performance upper limit of the operator; when the computational operator is an operator used for virtual core fusion calculation, the computational instruction time consumption and memory access time consumption corresponding to the operator can be calculated to obtain the theoretical performance upper limit of the operator based on the computational instruction time consumption and memory access time consumption; when the computational operator is an operator used for other complex calculations, at least one of the computational instruction time consumption and memory access time consumption corresponding to the operator can be calculated based on the specific computational task to obtain the theoretical performance upper limit of the operator.

[0065] Regarding the communication overhead, it can be calculated based on the amount of data used for communication between boards and the communication bandwidth. According to some embodiments of the present invention, for example, the communication overhead can be calculated by dividing the amount of data used for communication between boards by the communication bandwidth. Among them, regarding the amount of data used for communication, it can be related to the model parameters and the shape of the operator, for example; regarding the communication bandwidth, it can be related to the type of communication operator, the algorithm to be implemented by the model, the deployment strategy of the model, and the interconnection structure between boards.

[0066] Specifically, in some embodiments of the present invention, calculating the communication overhead corresponding to a communication operator may include: determining the amount of data used for communication based at least on the shape and data type of the communication operator; determining the communication bandwidth based at least on the type and deployment strategy of the communication operator; and calculating the communication overhead based on the data amount and the communication bandwidth.

[0067] Take the scatter reduction operator performed after the forward calculation of the multilayer perceptron as an example. In this example, the amount of data used for communication can be determined by the model parameters and the shape of the operator. For example, the shape of the multilayer perceptron output of the model can be related to the sequence length (SEQUENCE_LENGTH) and hidden layer size (HIDDEN_SIZE) in the model parameters and the micro-batch size (MICRO_BATCH_SIZE) in the training parameters, that is, it can be expressed as: [MICRO_BATCH_SIZE, SEQUENCE_LENGTH, HIDDEN_SIZE]. On this basis, the amount of data used for communication can be determined at least based on the data type. For example, if the data type is bfloat16 (bf16), the amount of data used for communication can be calculated based on the following formula (1-1):

[0068] The amount of data used for communication = MICRO_BATCH_SIZE × SEQUENCE_LENGTH × HIDDEN_SIZE × 2 (BF16 2Byte) (1-1)

[0069] In the above example, the communication bandwidth can then be determined based on the type of communication operator, the algorithm to be implemented by the model, the deployment strategy of the model, and the interconnection structure between the boards. Regarding the communication bandwidth, it can refer to the bandwidth of the communication operator under a specific algorithm. For example, for the All-reduce operator, the specific algorithm corresponding to it can be the Ring algorithm, the Tree algorithm, or the CollNet algorithm. Generally, different types of communication operators, or different interconnection structures between boards, will result in different communication bandwidths. The specific data on the communication bandwidth can be determined, for example, based on the parameters measured by the hardware manufacturer providing the board for the above factors.

[0070] The communication overhead can then be calculated based on the following formula (1-2).

[0071] Communication overhead = amount of data used for communication / communication bandwidth (1-2)

[0072] In step 203, based at least on the data parallel strategy and the pipeline parallel strategy, a vacancy ratio corresponding to the pipeline parallel strategy is determined.

[0073] Regarding the bubbles corresponding to the pipeline parallel strategy, it refers to the jamming caused by waiting for instruction operands during the operation process when different layers of the model are allocated to different boards for parallel operation based on the pipeline parallel strategy. Usually, the bubble ratio (PP_BUBBLE_RATIO) corresponding to the pipeline parallel strategy, also known as the pipeline bubble rate, can be used to characterize the efficiency of model training based on the current pipeline parallel strategy. The higher the pipeline bubble rate, the greater the waste of board resources, that is, the lower the model training efficiency. The following will combine Figure 3 Further describe the impact of pipeline parallel strategy on the void ratio.

[0074] According to an embodiment of the present invention, the bubble ratio (PP_BUBBLE_RATIO) corresponding to the pipeline parallel strategy is at least related to the number (PP_SIZE) after the model is split based on the pipeline parallel strategy. For example, in one example, the number after the model is split based on the pipeline parallel strategy can be 4, that is, PP_SIZE = 4. In this case, the pipeline parallel has 4 stages, and each stage is assigned to a board. According to an embodiment of the present invention, the bubble ratio (PP_BUBBLE_RATIO) corresponding to the pipeline parallel strategy can then be determined based on the number (PP_SIZE) after the model is split based on the pipeline parallel strategy and the global accumulation (GA).

[0075] Regarding global accumulation (GA), it refers to the number of iterations required to repeat a gradient update during model training. According to an embodiment of the present invention, global accumulation (GA) may be related to a data parallel strategy and training parameters. Specifically, the global accumulation (GA) may be determined based on the number of splits (DP_SIZE) based on the data parallel strategy and the global batch size (GLOBAL_BATCH_SIZE) and micro-batch size (MICRO_BATCH_SIZE) in the training parameters. For example, the global accumulation (GA) may be determined based on the following formula (2):

[0076] GA = GLOBAL_BATCH_SIZE / (MICRO_BATCH_SIZE × DP_SIZE) (2)

[0077] On this basis, the bubble ratio (PP_BUBBLE_RATIO) corresponding to the pipeline parallel strategy can be determined based on the following formula (3):

[0078] PP_BUBBLE_RATIO = (PP_SIZE – 1) / (PP_SIZE – 1 + GA) (3)

[0079] Figure 3 A schematic diagram showing the cavitation ratio corresponding to different pipeline parallel strategies.

[0080] Figure 3 (a) and (b) in Figure 1 show the cases where GA=1 and GA=4, respectively. Figure 3 From (a) and (b), we can see that for the case of GA=1, that is, a gradient update in the model training process is repeated for only one iteration, and the bubble ratio PP_BUBBLE_RATIO corresponding to the pipeline parallel strategy is 3 / 4; while for the case of GA=4, that is, a gradient update in the model training process needs to be repeated for four iterations, and the bubble ratio PP_BUBBLE_RATIO corresponding to the pipeline parallel strategy is 3 / 7. Obviously, the bubble ratio of the latter is significantly smaller than that of the former. It can be seen that the pipeline parallel strategy can improve the utilization efficiency of the board.

[0081] As can be seen from the above, according to an embodiment of the present invention, the proportion of cavitations corresponding to the pipeline parallel strategy can be determined based at least on the data parallel strategy and the pipeline parallel strategy. Specifically, the proportion of cavitations corresponding to the pipeline parallel strategy can be determined based on the number of splits of the model based on the data parallel strategy, the number of splits of the model based on the pipeline parallel strategy, the micro-batch size in the training parameters, and the global batch size.

[0082] In step 204, based on at least the tensor parallel strategy and the pipeline parallel strategy, an inter-group attenuation between parallel data groups is determined.

[0083] When the model performs parallel operations on multiple boards based on the data parallel strategy, there is protocol communication between the boards, and the running time of the parallel data groups on different boards is not completely consistent. When synchronization is performed between the parallel data groups, waiting will occur, resulting in inter-group decay (DP_DECAY) between the parallel data groups. It should be understood that the inter-group decay between parallel data groups has a negative impact on the model training performance. Therefore, according to the inventive concept of the present invention, when evaluating the model training performance, the inter-group decay (DP_DECAY) between parallel data groups needs to be considered as one of the factors.

[0084] According to an embodiment of the present invention, the inter-group attenuation (DP_DECAY) between parallel data groups can be determined based on at least the tensor parallel strategy and the pipeline parallel strategy. For example, in an embodiment of the present invention, the inter-group attenuation (DP_DECAY) between parallel data groups can be determined based on: the number of models after being split based on the tensor parallel strategy (TP_SIZE), the number of models after being split based on the pipeline parallel strategy (PP_SIZE), and the number of boards (TOTAL_DEVICE_NUM). Specifically, the inter-group attenuation (DP_DECAY) between parallel data groups can be determined based on the following formula (4):

[0085] DP_DECAY = TOTAL_DEVICE_NUM / (TP_SIZE × PP_SIZE) (4)

[0086] In step 205, the end-to-end consumption time required for model training is calculated based on at least the shape of the operator, the bubble ratio and the inter-group attenuation, so as to evaluate the model training performance based on the end-to-end consumption time.

[0087] According to the inventive concept of the present invention, the purpose of the present invention is to systematically evaluate the training performance of a large language model on a large-scale cluster by comprehensively considering the deployment strategy of the model and the overhead of the computing operator and the communication operator. In order to achieve the above-mentioned purpose of the invention, in an embodiment of the present invention, further, the end-to-end consumption time (E2E_COST) required for model training can be calculated by the following formula (5).

[0088] E2E_COST = (LAYER_COST × LAYER_NUM) × GA / (1- PP_BUBBLE_RATIO) ×DP_DECAY (5)

[0089] Among them, LAYER_COST is the total time consumption of a single-layer subgraph of the model; LAYER_NUM is the number of layers of the model on a single board; GA is the global accumulation; PP_BUBBLE_RATIO is the bubble ratio corresponding to the pipeline parallel strategy; DP_DECAY is the inter-group attenuation between parallel data groups.

[0090] Regarding the total single-layer time consumption (LAYER_COST) of the model, it may refer to the sum of the computational time consumption (i.e., the computational operator cost) and the communication time consumption (i.e., the communication operator cost) of a single layer. The computational time consumption may include, for example, the root mean square normalization (RMSNorm) computational time consumption, the linear function computational time consumption, the attention layer computational time consumption, the multi-layer perceptron computational time consumption, etc.; the communication time consumption may include, for example, the scattering reduction operator time consumption, the full aggregation operator time consumption, etc. Regarding the calculation of the computational time consumption and the communication time consumption, the description in step 202 above may be referred to, and will not be repeated here.

[0091] Generally, the model training process can include two processes: forward propagation and backward propagation. Figure 4 A schematic diagram showing the execution process of forward propagation and back propagation during model training.

[0092] As described above, in an embodiment of the present invention, the shape of the operator may be determined based on a tensor parallel strategy, or a tensor parallel strategy. Figure 4 As shown, for example, the shape of the operator (RMSNorm_fwd) for RMS normalization calculation in forward propagation is determined based on the sequence parallel strategy; the shape of the operator (Linear_fwd) for linear function calculation in forward propagation, the operator (Attention_fwd) for attention calculation, and the like is determined based on the tensor parallel strategy. Similarly, the shape of the operator (RMSNorm_bwd) for RMS normalization calculation in reverse propagation is determined based on the sequence parallel strategy; the shape of the operator (Linear_ bwd) for linear function calculation in reverse propagation, the operator (Attention_ bwd) for attention calculation, and the like is determined based on the tensor parallel strategy. Then, the total single-layer time consumption of the model can be calculated according to the determined shape of the operator. That is, according to the inventive concept of the present invention, the shape of the operator for calculating the total single-layer time consumption of the model is based on the tensor parallel strategy, or the shape of the operator after the tensor parallel strategy is segmented.

[0093] Specifically, the total time consumption of a single layer of the model (LAYER_COST) can be determined based on the total time consumption of the forward calculation (FWD_COST) and the total time consumption of the reverse calculation (BWD_COST) performed on the single layer, that is, LAYER_COST = FWD_COST + BWD_COST. In other words, the total time consumption of a single layer of the model can be equal to the sum of the calculation time and communication time consumption related to the forward calculation plus the sum of the calculation time and communication time consumption related to the backward calculation.

[0094] For example, Figure 4Taking the forward propagation shown in the figure as an example, the total time consumption of the forward calculation (FWD_COST) is equal to the sum of the calculation time consumption related to the forward calculation (FWD_COMPUTATION_COST) and the communication time consumption related to the forward calculation (FWD_COMMUNICATION_COST). Among them, the calculation time consumption related to the forward calculation (FWD_COMPUTATION_COST) can be calculated by the following formula (6-1).

[0095] FWD_COMPUTATION_COST = RMSNorm_fwd time + Linear1_fwd time + Attention_fwd time + Linear2_fwd time + AddRMSNorm_fwd time + MLP_fwd time (6-1)

[0096] The communication cost associated with the forward calculation (FWD_COMMUNICATION_COST) can be calculated using the following formula (6-2).

[0097] FWD_COMMUNICATION_COST = 2×Reduce_scatter time + 2×All_gather time (6-2)

[0098] On this basis, the aforementioned computation time associated with the forward computation and the communication time associated with the forward computation may then be summed to calculate the total time consumption (FWD_COST) of the forward computation, that is, FWD_COST = FWD_COMPUTATION_COST + FWD_COMMUNICATION_COST.

[0099] Similarly, for Figure 4 For the back propagation shown in , the computational time (BWD_COMPUTATION_COST) related to the reverse calculation can be calculated using the following formula (7-1).

[0100] BWD_COMPUTATION_COST = RMSNorm_bwd time + Linear1_bwd time + Attention_bwd time + Linear2_bwd time + AddRMSNorm_bwd time + MLP_bwd time (7-1)

[0101] The communication cost (BWD_COMMUNICATION_COST) related to the reverse calculation can be calculated using the following formula (7-2).

[0102] BWD_COMMUNICATION_COST = 2×Reduce_scatter time + 2×All_gather time (7-2)

[0103] On this basis, BWD_COMPUTATION_COST and BWD_COMMUNICATION_COST can then be summed to calculate the total time consumed for the reverse calculation, that is, BWD_COST = BWD_COMPUTATION_COST + BWD_COMMUNICATION_COST.

[0104] Therefore, the total time consumption of a single layer of the model can be calculated according to the above formula: LAYER_COST = FWD_COST + BWD_COST.

[0105] Regarding the number of layers (LAYER_NUM) on a single board, it may refer to the number of layers allocated to a single board after the model is split based on the deployment strategy. According to an embodiment of the present invention, the number of layers (LAYER_NUM) on a single board may be related to the number (PP_SIZE) after the model is split based on the pipeline parallel strategy. For example, the number of layers (LAYER_NUM) on a single board may be equal to the total number of layers of the model divided by the number (PP_SIZE) after the model is split based on the pipeline parallel strategy.

[0106] Taking a large language model with a decoder-only Transformer architecture as an example, the total time consumption of a single layer of the model (LAYER_COST) can refer to the sum of the total time consumption of the forward calculation and the total time consumption of the reverse calculation of a layer of decoders on a single board. In this example, the number of layers on a single board (LAYER_NUM) can refer to the number of layers of the decoder on a single board, that is, the total number of layers of the decoder in the model divided by the number of layers after the model is split based on the pipeline parallel strategy (PP_SIZE).

[0107] It can be seen from the above that according to an embodiment of the present invention, the model training performance can be evaluated based at least on the shape of the operator obtained at steps 202 to 204, the bubble ratio corresponding to the pipeline parallel strategy, and the inter-group attenuation between parallel data groups, so that when evaluating the model training performance, the deployment strategy of the model and the overhead of the computing operator and the communication operator can be comprehensively considered, making the evaluation of the model training performance more accurate.

[0108] Further, according to another inventive concept of the present invention, according to the solution for evaluating model training performance provided by an embodiment of the present invention, for the same model to be trained, multiple different deployment strategies can be determined, and the model training performance can be evaluated for each deployment strategy, so as to determine the best deployment strategy from the multiple different deployment strategies. In other words, multiple candidate deployment strategies for the model to be trained on a large-scale cluster can be determined first; the model performance under each deployment strategy can be evaluated to determine the best deployment strategy for the model to be trained from the multiple candidate deployment strategies.

[0109] Specifically, the method for evaluating model training performance according to an embodiment of the present invention may also include: determining multiple candidate deployment strategies for deploying the model to be trained; for each of the multiple candidate deployment strategies, calculating the end-to-end consumption time required for the corresponding model training; and comparing the multiple end-to-end consumption times calculated for the multiple candidate deployment strategies to determine the best deployment strategy from the multiple candidate deployment strategies.

[0110] Regarding the optimal deployment strategy, in some embodiments, it may refer to the deployment strategy corresponding to the shortest end-to-end consumption time required for model training.

[0111] In summary, the present invention provides a solution that can systematically evaluate model training performance. Through the above embodiments, the present invention can comprehensively consider the model deployment strategy and the overhead of the computing operator and the communication operator when evaluating the model training performance, thereby improving the evaluation efficiency and accuracy.

[0112] In addition, the various processes and processing described herein, such as method 200, may be performed at a computing device. The computing device includes, for example: at least one processor (at least one graphics processor and at least one central processing unit); and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor. In some embodiments, method 200 may be implemented as a computer software program or program product, which is tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on a computing device via a read-only memory (ROM) and / or a communication unit. When the computer program is loaded into a random-access memory (RAM) and executed by a GPU and a GPGPU, one or more actions of method 200 described above may be performed.

[0113] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for executing various aspects of the present invention. The computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0114] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0115] These computer-readable program instructions can be provided to a central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the central processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0116] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0118] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0119] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for evaluating model training performance, characterized in that: include: Determine a deployment strategy of the model on multiple boards, wherein the deployment strategy includes a data parallel strategy, a tensor parallel strategy, and a pipeline parallel strategy; Determining the shape of operators based at least on the tensor parallel strategy, wherein the operators include communication operators and computation operators; At least based on the data parallel strategy and the pipeline parallel strategy, determine the cavitation proportion corresponding to the pipeline parallel strategy; Determining inter-group attenuation between parallel data groups based at least on the tensor parallel strategy and the pipeline parallel strategy; as well as Based at least on the shape of the operator, the cavitation ratio and the inter-group attenuation, the end-to-end consumption time required for model training is calculated so as to evaluate the model training performance based on the end-to-end consumption time.

2. The method according to claim 1, characterized in that The strategy for determining the deployment of the model on multiple boards includes: Obtaining model parameters of the model to be trained, wherein the model parameters include: sequence length, hidden layer size, hidden layer size of the feedback neural network, and number of attention heads; Based on the model parameters, training parameters and the deployment strategy are determined, wherein the training parameters include: micro-batch size and global batch size.

3. The method according to claim 2, characterized in that Determining the shape of the operator based at least on the tensor parallel strategy includes: The shape of the operator is determined based on the mini-batch size, the sequence length, the hidden layer size of the feedback neural network, and the tensor parallel strategy.

4. The method according to claim 2, characterized in that: Determining the shape of the operator based at least on the tensor parallel strategy includes: A shape of an operator is determined based on at least the mini-batch size, the sequence length, the number of attention heads, and the tensor parallelism strategy.

5. The method according to claim 2, characterized in that: Determining the cavitation ratio corresponding to the pipeline parallel strategy based at least on the data parallel strategy and the pipeline parallel strategy includes: Based on the data parallel strategy, the pipeline parallel strategy, the micro-batch size, and the global batch size, a bubble proportion corresponding to the pipeline parallel strategy is determined.

6. The method according to claim 1, characterized in that The operator includes a communication operator and a computing operator, wherein the evaluating the model training performance based on the end-to-end consumption time also includes: For the calculation operator, calculating the calculation time corresponding to the calculation operator to determine the performance upper limit of the calculation operator; and For the communication operator, a communication overhead corresponding to the communication operator is calculated to determine a performance upper limit of the communication operator.

7. The method according to claim 6, characterized in that The calculation time includes: calculation instruction time and memory access time.

8. The method according to claim 6, characterized in that Calculating the communication overhead corresponding to the communication operator includes: Determining the amount of data used for communication based at least on the shape and data type of the communication operator; determining the communication bandwidth based at least on the type of the communication operator and the deployment strategy; and The communication overhead is calculated based on the data volume and the communication bandwidth.

9. The method according to claim 1, characterized in that: Also includes: determining a plurality of candidate deployment strategies for deploying the model to be trained; For each of the multiple candidate deployment strategies, calculate the end-to-end consumption time required for the corresponding model training; as well as A plurality of end-to-end consumption durations calculated for a plurality of candidate deployment strategies are compared to determine an optimal deployment strategy from the plurality of candidate deployment strategies.

10. The method according to claim 1, characterized in that Determining the inter-group attenuation between parallel data groups involves: Splitting the model based on the tensor parallel strategy to determine a first number of the model after being split based on the tensor parallel strategy; Splitting the model based on the pipeline parallel strategy, and determining a second number of the model after the model is split based on the pipeline parallel strategy; and determining a third quantity of the plurality of boards; and An inter-group attenuation between the parallel data groups is determined based on the first number, the second number, and the third number.

11. A computing device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The computer program product is tangibly stored on a non-transitory computer readable medium and includes machine executable instructions which, when executed, cause a machine to perform the steps of any one of the methods of claims 1-10.

Images