Large model computing cluster service reliability evaluation and optimization method and device

Abstract

The invention relates to the technical field of large model computing clusters, in particular to a large model computing cluster service reliability evaluation and optimization method, device and equipment based on parallel policy scheduling and a computer storage medium. According to the large model computing cluster service reliability evaluation and optimization method, the flow line parallel behavior is taken as the core, the influence of the performance difference between different GPUs, the data transmission time and the uncertainty factors on the training time in the training process is comprehensively considered, the existing staggered and non-staggered flow line parallel strategies are combined, and the reliability of the large model computing cluster service is evaluated and optimized. According to the method, a model which fully considers inconsistency of training time and combines various pipeline parallel strategies is established, specific constraints of a training process are given for specific characteristics of large model training, and the model is solved through integer programming so as to realize efficient and reliable training of a large model computing cluster.

Description

Technical Field

[0001] The present invention relates to the technical field of large model computing clusters, and in particular to a large model computing cluster service reliability evaluation and optimization method, device, equipment and computer storage medium based on parallel strategy scheduling. Background Art

[0002] Since OpenAI launched ChatGPT, large language models have quickly become the focus of attention. Large models refer to neural network models that contain ultra-large-scale parameters (usually more than one billion). After a specific large model training process, large-scale data can be analyzed and complex tasks can be processed. Among them, the natural language processing model based on the Transformer architecture realizes the modeling of text of arbitrary length through the self-attention mechanism, which greatly improves the ability of language understanding. With the rapid development of natural language processing models based on the Transformer architecture, the number of parameters and training scale have increased exponentially. The huge model scale also puts forward certain requirements for the training process: large model training requires powerful computing resources, and traditional single-machine single-card training takes a long time to complete the training process, which can no longer meet the requirements of large model training. Through parallel computing, a faster and more stable training process for large models can be achieved. Therefore, designing an excellent parallel scheduling strategy is crucial for large model training.

[0003] Parallel strategy is an important part of large model training. By reasonably splitting and allocating the training data and model of the large model, the training process of multiple machines and multiple cards can be realized, and the training efficiency and stability can be improved. The natural language processing model based on Transformer consists of multiple encoding layers and decoding layers. Each encoding layer and decoding layer can be divided into two parts: feedforward neural network layer and attention layer. Parallel strategies can be divided into data parallelism and model parallelism. Data parallelism is a parallel strategy that splits the training data, and each training GPU retains the complete training model to complete a training together. Model parallelism is a parallel strategy that splits the Transformer layer. According to different splitting methods, it can be divided into pipeline parallelism and tensor parallelism. Pipeline parallelism is a strategy that splits the encoding layer and decoding layer to different GPUs to complete training, while tensor parallelism is a strategy that splits the weight matrix in the feedforward neural network layer and attention layer to different GPUs to complete training.

[0004] In the actual training process, each training data is sequentially forward-calculated on all Transformer layers, and then reversely calculated on all Transformer layers. Due to the limitation of the training order caused by the parallel strategy, a large number of GPU idle periods will be generated during training, and the resulting delay will greatly increase the entire training time. For the latest GPT-4 model, the actual utilization rate of the GPU during training is only 32%-36%.

[0005] At the same time, the existing parallel strategy sets the training time as a constant value, without considering the performance differences of different training GPUs and the time of training data transmission. This simple pipeline parallel strategy cannot cope with the uncertainty in the training process. Therefore, combined with the inspiration brought by the modeling method in industrial pipeline scheduling, by analyzing and combining different pipeline parallel strategies, the existing parallel strategy is optimized to achieve system service reliability guarantee in an uncertain environment. The existing large model computing cluster reliability evaluation and optimization methods based on parallel strategy scheduling still have the following challenges:

[0006] 1. Previous scheduling models modeled training time as one unit of forward computing time and two units of backward computing time due to recalculation. This is a relatively ideal condition and does not take into account the different training times that may occur when training different Transformer layers on different GPUs, the fluctuations in training time caused by randomness, and the increase in training time caused by hardware failure or aging.

[0007] 2. The existing model does not take into account the time spent on data transfer. The training time of training data on the GPU and the communication time between GPUs are of the same order of magnitude. Therefore, the impact of communication time on the total training time cannot be ignored when modeling.

[0008] 3. Although pipeline parallelism provides a scheduling solution, it is a very regular solution: each micro-batch of data enters the training cluster in sequence according to the number to complete the training. Existing research has not given an explanation for the rationality of this scheduling solution.

[0009] 4. The previous pipeline parallel strategy only considered the case where the number of GPUs and Transformer layers was 1:1. However, in actual training, the number of GPUs is often much higher than the number of Transformer layers. This means that almost each layer of Transformer will be divided into multiple GPUs to complete training. There is no corresponding optimization solution for the existing model.

[0010] In summary, the existing technology does not comprehensively consider the performance differences between different GPUs, data transfer time, and the impact of uncertainty factors on training time during training, which leads to the problems of inefficient and poor reliability of large model cluster training. Summary of the invention

[0011] To this end, the technical problem to be solved by the present invention is to overcome the problems of inefficiency and poor reliability of large model cluster training in the prior art.

[0012] In order to solve the above technical problems, the present invention provides a large model computing cluster service reliability evaluation and optimization method, comprising:

[0013] Mathematically model the large model computing cluster and define basic parameters and training tasks;

[0014] Modeling training time, which includes communication time caused by data transfer between GPUs and latency caused by network structures between GPUs;

[0015] Based on the training time, a first reliability index of the large model computing cluster service is defined as the probability that all micro-batch data complete the training task within the specified time, that is, the product of the probability that each micro-batch data completes a single task within the specified time;

[0016] An objective function is established based on the first reliability index of the large model computing cluster service, and related constraints are modeled to construct a deterministic optimization model for parallel strategy scheduling of the large model computing cluster;

[0017] Based on the deterministic optimization model, a mixed integer programming method is used to obtain an initial solution of the scheduling strategy.

[0018] Preferably, the mathematical modeling of the large model computing cluster and the definition of basic parameters and training tasks include:

[0019] Build a GPT model containing only the decoder;

[0020] Defining the basic parameters includes the initial micro-batch number, the number of Transformer layers, and the number of GPUs;

[0021] The training task is defined to ensure that all micro-batch data complete a forward and backward calculation in order on each Transformer layer.

[0022] Preferably, the training time also includes random time fluctuations.

[0023] Preferably, after obtaining the initial solution of the scheduling strategy based on the deterministic optimization model by using a mixed integer programming method, the method further includes:

[0024] generating a plurality of sets of actual training times based on an initial solution of the scheduling strategy;

[0025] The initial solution is evaluated according to the second reliability index, and the initial solution with the highest reliability is output as the optimal solution of the scheduling strategy.

[0026] Preferably, the second reliability indicator is defined as the ratio of the time when the forward calculation of the i-th micro-batch data at the first Transformer layer ends to the specified time when the forward calculation of the i-th micro-batch data at the j-Transformer layer ends under a certain number of experiments.

[0027] Preferably, the communication time is determined by the data volume of the training task and the computing rate of the GPU.

[0028] Preferably, the construction of a deterministic optimization model for parallel strategy scheduling of a large model computing cluster and modeling of related constraints includes:

[0029] Define the data size of each group of micro-batch data to be the same;

[0030] When defining the GPU to place the Transformer model, only the memory usage generated by storing the Transformer model is considered, and the memory usage generated by the Transformer model does not exceed the memory upper limit of the GPU;

[0031] Defines that the GPU can perform forward or backward computation and data transfer simultaneously;

[0032] Define that for the same micro-batch training task, the forward calculation and backward calculation must be performed on the same GPU;

[0033] Define that the GPU can only perform one forward or backward computing task at a time;

[0034] When the amount of data in each micro-batch is defined to be constant, the time required for forward and backward calculations is only related to the number of Transformer layers and GPU, and has nothing to do with the order of micro-batches;

[0035] When defining micro-batch data to complete the training of different Transformer layers between different GPUs, data transmission will be performed, and the time consumption of data transmission is divided into communication time and latency;

[0036] Define any training task. The forward calculation of the next layer of Transformer can be performed only after the forward calculation of the previous layer of Transformer is completed. The backward calculation can be performed only after all forward calculations are completed. The backward calculation of the previous layer of Transformer can be performed only after the backward calculation of the next layer of Transformer is completed.

[0037] The present invention also provides a large model computing cluster service reliability evaluation and optimization device, comprising:

[0038] The computing cluster modeling module is used to perform mathematical modeling on the large model computing cluster and define basic parameters and training tasks;

[0039] A training data modeling module, used to model the training time, wherein the training time includes the communication time caused by the GPU for data transmission and the delay caused by the network structure between the GPUs;

[0040] A reliability index definition module, used to define, based on the training time, a first reliability index of the large model computing cluster service as the probability that all micro-batch data complete the training task within the specified time, that is, the product of the probability that each micro-batch data completes a single task within the specified time;

[0041] The deterministic model is used to establish an objective function based on the first reliability index of the large model computing cluster service, and to model related constraints to construct a deterministic optimization model for parallel strategy scheduling of the large model computing cluster;

[0042] The initial solution generation module is used to obtain the initial solution of the scheduling strategy based on the deterministic optimization model using a mixed integer programming method.

[0043] The present invention also provides a large model computing cluster service reliability evaluation and optimization device, comprising:

[0044] Memory for storing computer programs;

[0045] A processor is used to implement the above-mentioned large model computing cluster service reliability evaluation and optimization method steps when executing the computer program.

[0046] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned large model computing cluster service reliability evaluation and optimization method are implemented.

[0047] The above technical solution of the present invention has the following advantages compared with the prior art:

[0048] In the process of large model training, training efficiency and cost are factors that must be considered, and reliability risks need to be paid special attention. The reliability of large model computing clusters directly affects their performance stability, security and long-term operation efficiency. Traditional scheduling strategies are difficult to cope with dynamic changes in the training process, and often ignore the overhead of inter-GPU communication time. The present invention aims to reduce the GPU window period and delay by optimizing the parallel scheduling strategy, improve the actual utilization of the GPU, especially the problem of low GPU utilization in the current large model training, and significantly improve the training efficiency by improving the scheduling strategy. By defining reliability indicators and incorporating communication overhead reliability constraints into the parallel training strategy, it can ensure that the training process meets the expected performance standards while reducing unexpected downtime and training costs caused by failures. Based on this, the present invention proposes a mixed integer programming optimization model for an efficient pipeline parallel training strategy, which comprehensively considers the communication overhead and training overhead in large model training, as well as the training time fluctuations caused by hardware aging and uncertainty. The optimized parallel strategy not only improves the training efficiency, but also reduces the additional costs caused by GPU idle time and delay, and reduces the overall training cost through more efficient resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0050] Figure 1 It is a flow chart for implementing a large model computing cluster service reliability evaluation and optimization method provided by the present invention;

[0051] Figure 2 It is the Transformer structure diagram;

[0052] Figure 3 It is a simple pipeline parallel flow chart;

[0053] Figure 4 It is a parallel flow chart of micro-batch pipeline;

[0054] Figure 5 It is a parallel flow chart of interleaved pipeline;

[0055] Figure 6 A flowchart of a large model computing cluster service reliability evaluation and optimization method provided by an embodiment of the present invention;

[0056] Figure 7 Optimize experimental results for ideal communication time;

[0057] Figure 8 Optimizing experimental results for critical communication time;

[0058] Fig. 9 Experimental results for double critical communication time optimization;

[0059] Fig.10 The results of data parallel experiments. DETAILED DESCRIPTION

[0060] The core of the present invention is to provide a method, device, equipment and computer storage medium for evaluating and optimizing the service reliability of a large model computing cluster, which effectively improves the efficiency and reliability of large model cluster training.

[0061] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0062] Please refer to Figure 1 , Figure 1 This is a flow chart of the implementation of a large model computing cluster service reliability evaluation and optimization method provided by the present invention; the present invention provides a large model computing cluster service reliability optimization method based on a mixed integer programming model to generate a reliability optimization solution for a large model training cluster, thereby ensuring efficient training of a large model computing cluster while minimizing training time. The specific operation steps are as follows:

[0063] S101: Perform mathematical modeling on the large model computing cluster and define basic parameters and training tasks;

[0064] S102: Modeling the training time, where the training time includes the communication time caused by data transfer by the GPU and the delay caused by the network structure between the GPUs;

[0065] S103: Based on the training time, define the first reliability index of the large model computing cluster service as the probability that all micro-batch data complete the training task within the specified time, that is, the product of the probability that each micro-batch data completes a single task within the specified time;

[0066] S104: establishing an objective function based on the first reliability index of the large model computing cluster service, modeling related constraints, and constructing a deterministic optimization model for parallel strategy scheduling of the large model computing cluster;

[0067] S105: Based on the deterministic optimization model, a mixed integer programming method is used to obtain an initial solution of the scheduling strategy.

[0068] Based on the above embodiments, this embodiment describes step S101 in detail:

[0069] Build a decoder-only GPT model.

[0070] The current mainstream model is the natural language processing model based on Transformer. Transformer can be divided into two parts: encoding part and decoding part. Figure 2 In the Transformer shown in the figure, the left side is the encoding part and the right side is the decoding part. The encoding part consists of multiple layers of encoders. The decoding part consists of multiple layers of decoders. Different encoders and decoders are the same in structure, but the weight matrix parameters are different. The encoder is mainly composed of the Attention layer and the Feed-Forward Neural Network (FFNN).

[0071] In addition to these two layers, the decoder also inserts an Encoder-DecoderAttention layer between the two layers. This layer can receive information from the encoder and help the decoder focus on the relevant parts of the input sentence. In addition, there is an Add&Normalize layer after each Attention layer and feedforward neural network layer of the encoder and decoder. Add represents the residual connection (Residual Connection) used to prevent network degradation, and Normalize represents Layer Normalization, which is used to normalize the activation values ​​of each layer.

[0072] In the actual training process, the word vector is encoded and then enters the encoding part. After the word vector is calculated in the first layer of encoder, the output is used as the new input to enter the next layer of encoder for calculation, and so on. The calculation is performed in sequence on all encoders to strengthen the output results. After all encoders have completed the calculation, the encoding part passes the calculated Key vector and Value vector to the encoder-decoder attention layer of each decoder. The final result of the previous output is used as the new input to enter the attention layer of the decoder to calculate the vector query. After calculation by each layer of decoder, the final output vector is obtained. Each element of the output vector is a floating point number. The linear layer is an ordinary fully connected neural network that can map the vector output by the decoder to a longer vector, which is called a logits vector. Each number in the logits vector represents the score of a word. The Softmax layer converts these scores into probabilities. Then the word corresponding to the number with the highest probability is selected and output as the final predicted word.

[0073] With further research, the structure of the Transformer model has changed to some extent. The current Transformer model can be divided into three categories according to its structure: a pure Encoder model containing only an Encoder, a pure Decoder model containing only a Decoder, and a traditional Encoder-Decoder model that has both.

[0074] The research of this invention mainly targets the GPT model which is a pure Decoder model, and the training data is sequentially calculated on the Decoder for forward and backward. The basic task of this invention is to ensure that each task completes a forward and backward calculation in order on each Transformer layer.

[0075] Define the basic parameters including the initial mini-batch size, the number of Transformer layers, and the number of GPUs;

[0076] The training task is defined as ensuring that all mini-batch data completes one forward and one backward calculation in order on each Transformer layer.

[0077] Based on the above embodiments, this embodiment describes step S102 in detail:

[0078] Pipeline parallelism is a scheduling method that divides the model into Transformer layers and assigns one or more layers of the model to a GPU. The data is forward- and backward-calculated on different GPUs in turn to complete a complete training task.

[0079] Simple pipeline parallel Figure 3 As shown in the figure, the naive pipeline parallelism does not further split the data. At the beginning and end of each layer of Transformer calculation, the GPUs of other layers are idle, that is, only one GPU can train the model at the same time. In the naive pipeline parallelism, as the number of parallel devices continues to increase, the proportion of idle GPUs continues to increase, and the utilization rate of GPUs gradually decreases. In addition, due to the communication overhead of data transmission between devices, if memory allows, parallel training of multiple GPUs is not as good as direct training of a single GPU.

[0080] To address the problem of low performance of the naive pipeline, we further process the data of a single batch and divide the completed training data into micro-batch training data so that the data can be trained on multiple GPUs at the same time, thereby improving the utilization of the GPU. Figure 4In the example, a set of training data is divided into four micro-batches, where F and B represent forward calculation and backward calculation respectively, and the numbers represent the micro-batch numbers. Under micro-batch pipeline parallelism, when micro-batch data 1 completes the forward calculation in Transformer layer 1, Transformer layer 1 can pass the forward calculation result of micro-batch data 1 to Transformer layer 2 while continuing to perform forward calculation on micro-batch data 2. Micro-batch data 1 can immediately perform forward calculation on Transformer layer 2, without having to wait for the forward calculation of Transformer layer 1 to be completed before starting the forward calculation of this layer as in traditional pipeline parallelism. Under micro-batch pipeline parallelism, multiple GPUs can perform training tasks at the same time, greatly improving training efficiency.

[0081] In this training mode, the utilization of the GPU increases with the number of micro-batches. When the number of micro-batches is infinite, the utilization of each GPU approaches 1, and the GPU idle time can be ignored. However, the method of increasing the number of micro-batches to increase the bubble score has high requirements for memory because it needs to store a large number of intermediate calculation variables. Therefore, in actual training, micro-batch data will not be cut indefinitely.

[0082] In addition, there is an interleaved 1F1B pipeline parallel model that can further improve GPU utilization. In the interleaved 1F1B pipeline parallel model, the number of consecutive Transformer layers assigned to each GPU is reduced, but alternate Transformer layers are assigned. Take 4 GPUs training 16-layer Transformer as an example: in non-interleaved pipeline parallel model, during training, Transformer layers 1, 2, 3, 4 may be assigned to GPU1, Transformer layers 5, 6, 7, 8 may be assigned to GPU2, Transformer layers 9, 10, 11, 12 may be assigned to GPU3, and Transformer layers 13, 14, 15, 16 may be assigned to GPU4. In staggered pipeline parallelism, during training, Transformer layers 1, 2, 9, and 10 are assigned to GPU1, Transformer layers 3, 4, 11, and 12 are assigned to GPU2, Transformer layers 5, 6, 13, and 14 are assigned to GPU3, and Transformer layers 7, 8, 15, and 16 are assigned to GPU4. Figure 5For example, each light-colored block represents the calculation of the first 8 layers, and the dark-colored block represents the calculation of the last 8 layers. In actual training, in non-interleaved training, the data completes 16 layers of training on GPU1, GPU2, GPU3, and GPU4 in turn, while in interleaved training, the data first completes the training of the first eight layers on GPU1, GPU2, GPU3, and GPU4, and then transfers the intermediate variables from GPU4 to GPU1 to complete the training of the last 8 layers in turn. In non-interleaved training, a single micro-batch of data needs to complete 4 layers of calculation on each GPU continuously, while in interleaved training, a single micro-batch of data only needs to complete 2 layers of calculation on each GPU continuously each time, which makes the idle time of the GPU in interleaved training only 50% of that in non-interleaved training. But at the same time, the communication frequency under interleaved scheduling has doubled, and accordingly, the communication time has also doubled. Therefore, the interleaved scheduling strategy is actually a way to exchange communication time for GPU utilization. In actual training, the impact of increased communication volume can be reduced by high-speed network bandwidth in multi-GPU servers.

[0083] The communication time is determined by the amount of data in the training task and the computing rate of the GPU.

[0084] Existing research on scheduling strategies lacks modeling of training time. In actual training, training time consists of two parts. One is the communication time caused by GPU data transmission, which is determined by the data volume of the training task and the computing rate of the GPU. The other is the delay caused by the network structure between GPUs. This part is a fixed value and is related to the specific performance of the hardware.

[0085] The communication time of data transfer is positively correlated with the amount of data communicated, and inversely proportional to the transmission rate. Therefore, the model of the present invention selects the micro-batch data volume C and the bandwidth v between GPUs as the relevant parameters of the communication time, and denotes the delay as α to perform relevant modeling of the data transfer time. The total time required for data transfer is:

[0086]

[0087] Based on the above embodiments, this embodiment describes step S103 in detail:

[0088] The present invention defines the service reliability of a large model computing cluster as the probability that the computing cluster completes a training task within a specified time. The present invention numbers the micro-batch data as i, the Transformer as j, the GPU as k, the micro-batch data set as [I], the Transformer set as [J], and the GPU set as [K]. A complete training process is when all micro-batch data complete a forward calculation and a backward calculation on each layer of Transformer. The time t when the forward calculation of micro-batch data i at Transformer layer j ends ij , the time τ at which the backward computation of micro-batch data i ends in Transformer layer j ij .

[0089] The specified time for the micro-batch data to complete the forward computation at Transformer layer j is recorded as The specified time for the micro-batch data to end the backward calculation at Transformer layer j is recorded as Therefore, the reliability of a single forward task is: The reliability of a single backward task is:

[0090] The overall reliability of the system forward calculation is R f , the overall reliability of the backward calculation is R b , considering the reliability of each task, the overall reliability of the training cluster is:

[0091]

[0092] However, when setting the prescribed time in units of layers and taking into account the reliability of all training tasks, it is meaningless to perform backward computing tasks in advance to release memory. The optimal strategy must be to perform all forward computing tasks first, and then all backward tasks, that is, the model will optimize the scheduling scheme according to the F-then-B (first perform all forward tasks, then perform backward tasks) scheduling strategy. However, the present invention does not want the model to optimize the scheduling scheme according to the F-then-B scheduling strategy. Therefore, under the premise of setting the prescribed time in units of layers, the reliability of all training tasks is taken into account, and setting a separate prescribed time for each training task is too cumbersome and meaningless. Therefore, the reliability index proposed by the present invention only considers whether each training task can complete the training process within the prescribed time, and no longer focuses on whether the training task can be completed on time at each layer, that is:

[0093]

[0094] Based on the above embodiments, this embodiment describes step S104 in detail:

[0095] The model of the present invention focuses on the degree of interleaving and scheduling strategies of pipeline parallel strategies to reduce the total time for large model training. The degree of interleaving mainly affects the ratio of communication time to computing time and thus affects the total training time. The scheduling strategy mainly affects the execution order of forward computing tasks and backward computing tasks. During the large model training process, the training data completes the forward calculation from the first Transformer layer to the last Transformer layer, and then the training data completes the backward calculation from the last Transformer layer to the first Transformer layer. During actual training, the training data will be cut into v equal micro-batch data, so that the vacuolation time during training becomes the original In combination with the research on the large model training process, the model assumptions of the present invention are as follows:

[0096] (1) The amount of data in each micro-batch is the same.

[0097] (2) Placing the Transformer model on the GPU will occupy a certain amount of memory. The specific memory usage is related to the Transformer layer. Only the memory usage generated by storing the Transformer model is considered, and the memory usage generated by intermediate variables is not considered. The memory usage generated by storing the Transformer model cannot exceed the memory limit of the GPU.

[0098] (3) GPU can perform forward or backward calculation and data transmission simultaneously.

[0099] (4) For the same micro-batch training task, the forward and backward computations must be performed on the same GPU.

[0100] (5) The GPU can only perform one forward or backward computing task at a time.

[0101] (6) When the amount of data is constant, the time required for forward and backward calculations is only related to the number of Transformer layers and the GPU, and has nothing to do with the order of micro-batches.

[0102] (7) When micro-batch data is transferred between different GPUs to complete the training of different Transformer layers, data transmission will be performed. The time consumption incurred by data transmission is divided into communication time and delay time. The communication time is related to the amount of micro-batch data and the communication rate between GPUs, and the delay time is a certain value.

[0103] (8) For any training task, the forward calculation of the next layer of Transformer can be performed only after the forward calculation of the previous layer of Transformer is completed, and the backward calculation can be performed only after the forward calculation is completed, and the backward calculation of the previous layer of Transformer can be performed only after the backward calculation of the next layer of Transformer is completed.

[0104] In a specific embodiment:

[0105] Define the time required for GPUk to complete the forward calculation of Transformer layer j as f jk , the time required for GPUk to complete the backward calculation of Transformer layer j is b jk , as a parameter for modeling related to computing time; define the memory capacity of the GPU as G and the memory space required by Transformerj as g j , as a parameter for memory usage related modeling.

[0106] In order to make the model meet the assumptions, a series of decision variables are set for modeling. ijk It is the core decision variable, which is used to decide whether the calculation of micro-batch data i in Transformer layer j is performed on GPUk. If the calculation of micro-batch data i in Transformer layer j is performed on GPUk, the decision variable takes the value of 1. Decision variable x jk It is used to determine whether GPUk needs to store the model of Transformer layer j. If it needs to be stored, the decision variable takes the value of 1. Decision variable y ij Used to determine whether micro-batch data i needs to be transferred after Transformer layer j is calculated. If data transfer is required, the value is 1. ij , τ ij They correspond to the time it takes for micro-batch data i to complete the forward calculation and backward calculation in Transformer layer j.

[0107] The optimization goal of the present invention is to minimize the total training time. In the last layer of Transformer, the backward calculation completion time of the last Transformer to complete the calculation in all micro-batch data is taken as the total training time of the model:

[0108]

[0109] For the above constraint (2), the present invention defines a bool variable x jk It is defined that when the calculation of any task at Transformer layer j needs to be performed on GPU k, it means that GPU k needs to store the model of Transformer layer j. At this time, xjk The value is 1:

[0110]

[0111] On any one GPU, the memory usage of storing all Transformer models does not exceed the GPU memory capacity limit:

[0112]

[0113] For the above constraint (4), each computing task is completed on a unique GPU:

[0114]

[0115] For the above constraint (5), in actual training, a GPU can only perform one training task at a time. ijk , B ijk is the completion time of the corresponding computing task on the specific GPU k. ijki′j′ , λ ijki′j′ Used to determine the order of two different tasks on the same GPU. When the decision variable takes the value of 1, it means that on GPUk, the calculation task of data i' in Transformer layer j' precedes the calculation task of data i in Transformer layer j. These decision variables are used to ensure that only one task is calculated on any GPU at the same time. ξ ijki′j Responsible for determining the order of the two forward computing tasks, Used to determine the order of two backward computing tasks, λ ijki′j′ It is to judge the forward computing task task ij and backward computation task i′j′ order of precedence.

[0116] About the decision variable ξ ijki′j′ , λ ijki′j′ , since the order of the two tasks is mutually exclusive, the decision variables ξ of the two identical tasks are determined ijki′j′ and i′j′kij , and λ ijki′j and λ i′j′kij , only one of the groups can get 1, and the other can get 0, that is:

[0117] ξ ijki′j′ +ξ i′j′kij ≤1

[0118]

[0119] λ ijki′j′ +λ i′j′kij ≤1

[0120] First, get the completion time of the computing task on a specific GPUk:

[0121]

[0122] The specific mathematical modeling of only one set of training data being calculated at the same time is as follows: First, determine whether the two computing tasks come from the same GPU. If the two computing tasks come from different GPUs, that is, (2-z ijk -z i′j′k ) is less than 1, the equation always holds true, and the model does not impose any time constraints on the two computing tasks. If the two computing tasks come from the same GPU, there must be a sequential relationship between them, and the end time of the first computing task must be earlier than the start time of the second computing task. The constraints between the two forward tasks on the same GPU are as follows:

[0123]

[0124] Assume two different tasks taskA and taskB on the same GPU. When taskA is later than taskB, that is, ξ ijki′j′ The value is 1, and the model performs the above judgment. The start time of taskA is earlier than the end time of taskB. When taskA is earlier than taskB, ξ ijki′j′ The value is 1, and the model makes the following judgment: the end time of taskA must be earlier than the start time of taskB. The constraints of the two backward tasks on the same GPU and the corresponding constraints of any forward task and any backward task on the same GPU are as follows:

[0125]

[0126]

[0127] For the above constraint (7), the present invention defines a bool variable y ij When the calculations of two adjacent layers of the same micro-batch data are not performed on the same GPU, a data transfer is required between the two calculations. In this case, y ij The value is 1:

[0128]

[0129] For the above constraint (8), for any training task, the forward calculation task of the next Transformer layer can only start after the previous Transformer layer completes the forward calculation task and completes the data transfer:

[0130]

[0131] For any training task, the backward computing task of the next Transformer layer is completed and the data is transferred, and then the backward computing task of the previous Transformer layer can start:

[0132]

[0133] For any training task, the backward calculation task can only be started after all forward calculation tasks are completed:

[0134]

[0135] Based on the above embodiments, this embodiment describes step S105 in detail:

[0136] The training time also includes random time fluctuations;

[0137] In actual training, due to various software and hardware problems, training tasks often cannot be completed on time. The timeout of a training task will cause the subsequent calculations and the calculation of the next task to be delayed, which will cause gaps in the originally continuous training tasks and significantly increase the total training time. This requires our model to consider the reliability of training under uncertain time.

[0138] In reliability modeling, the present invention changes the training time from the original determined value to a random variable. The random variable will produce a certain fluctuation based on the original determined value. The objective function is still the probability of completing the training task within the specified time, but the reliability evaluation method is changed to the end time t under a certain number of experiments. i1 Less than proportion.

[0139] In the reliability optimization model, we can obtain feasible scheduling strategies through other methods such as deterministic optimization models, and then further evaluate the reliability of the scheduling strategies through experiments. Figure 6 , after obtaining the initial solution of the scheduling strategy based on the deterministic optimization model using a mixed integer programming method, the method further includes:

[0140] generating a plurality of sets of actual training times based on an initial solution of the scheduling strategy;

[0141] The initial solution is evaluated according to the second reliability index, and the initial solution with the highest reliability is output as the optimal solution of the scheduling strategy.

[0142] The second reliability indicator is defined as the ratio of the time when the forward calculation of the i-th micro-batch data at the first Transformer layer ends to the specified time when the forward calculation of the i-th micro-batch data at the j-Transformer layer ends under a certain number of experiments.

[0143] Based on the above embodiments, this embodiment combines the data of GPT series large model training to experiment with the above mixed integer programming model. The experiment fully analyzes the correctness of the model under the fluctuation of training time and the efficiency of the traditional scheduling strategy. All numerical experiments in this experiment are run on a computer equipped with an 11th Gen Intel(R)Core(TM)i7-11800H processor and an NVIDIAGeForce RTX 3060 graphics card. All algorithm codes are implemented using Gurobi in Python3.12 environment. The results are obtained by Gurobi searching 500,000 nodes using the branch and bound method.

[0144] In the experiment, the ideal forward training time and ideal backward training time of all GPUs are the same, and the forward training time and backward training time of all Transformer layers are also the same. The number of micro-batches I is set to 4, the number of Transformer layers J is set to 8, and the number of GPUs K is set to 4. The 4 GPUs with non-interleaved pipeline parallelism store Transformer layers 1 and 2, Transformer layers 3 and 4, Transformer layers 5 and 6, and Transformer layers 7 and 8 respectively. The 4 GPUs with interleaved pipeline parallelism store Transformer layers 1 and 5, Transformer layers 2 and 6, Transformer layers 3 and 7, and Transformer layers 4 and 8 respectively. In order to control variables, the mixed integer programming model stipulates that each GPU can store at most two layers of Transformer models.

[0145] In the experiment, the text sets the ideal time for forward calculation to 24 and the ideal time for backward calculation to 48. Since the experiment does not involve data segmentation, the communication time will not change under the three pipeline parallel strategies, so the communication time and delay can be combined into the total communication time. This experiment sets up three groups of experiments with total communication time of 0, 27, and 54. When the total communication time is 0, it is called the ideal communication time; when the total communication time is 27, the ideal training time of traditional interleaved pipeline parallel and traditional non-interleaved pipeline parallel is the same, which is called the critical communication time; 54 is twice the critical communication time. Ten sub-experiments were conducted in each group. The training time of each layer of Transformer in each sub-experiment fluctuated randomly between [100%, 125%]. The fluctuation of training time of the three pipeline parallel strategies in the same sub-experiment group is consistent. The experimental results are as follows Figure 7-9 As shown, the solid line represents the experimental results, and the dotted line represents the ideal training time of the corresponding pipeline parallel strategy.

[0146] The results of the group experiment show that: under the ideal communication time, the training time of the traditional non-interleaved pipeline parallel is significantly higher than that of the traditional interleaved pipeline parallel. The ideal computing time of our optimization model is consistent with that of the traditional interleaved pipeline parallel, which shows that the mixed integer programming model of this patent is very consistent with the previous research results, confirming the correctness of the optimization model. At the same time, this also shows that without considering the influence of other factors, the traditional interleaved pipeline parallel is the optimal scheduling solution. When the training time fluctuates, the overall training time of the optimization model is the shortest, but the gap with the traditional interleaved pipeline parallel is not large.

[0147] Under critical communication time, the ideal training time of traditional interleaved pipeline parallelism and traditional non-interleaved pipeline parallelism is consistent, but both are significantly higher than the training time of the optimized model. This shows that under critical communication time, compared with the optimized model, the training efficiency of both interleaved pipeline parallelism and non-interleaved pipeline parallelism is not very ideal. In addition, since the training time is positively fluctuating, in actual training, the training efficiency of non-interleaved pipeline parallelism is not as good as that of interleaved pipeline parallelism.

[0148] Under double critical communication time, the ideal training time of traditional interleaved pipeline parallel is significantly higher than that of traditional interleaved pipeline parallel. The ideal training time of the optimization model is similar to that of traditional non-interleaved pipeline parallel. Secondly, the inter-group experimental results show that with the increase of communication time, the training time of traditional interleaved pipeline parallel increases greatly, while the training time of traditional non-interleaved pipeline parallel increases less. The efficient optimization model we proposed shows an optimization time similar to that of traditional interleaved pipeline parallel when the communication time is small, and shows an optimization time similar to that of traditional non-interleaved pipeline parallel when the communication time is large, and always achieves the best optimization results among the three parallel strategies. After further analysis of the experimental data: the forward calculation time increased by an average of 13.4%, the backward calculation time increased by an average of 12.0%, and the weighted average training time increased by 12.5%. Under the ideal communication time, the total training time of the three parallel strategies increased by more than 12.5%. This can explain to a certain extent that under the ideal communication time, the increase in training time of the training cluster caused by hardware failure is higher than the increase in training time of a single hardware, and as the total communication time increases, the impact of the training cluster gradually decreases.

[0149] The present embodiment also explores the actual combination scheme of optimization model and data parallel strategy, and carries out relevant numerical experiments. The experiment sets the number of micro-batch data I to 14, the number of Transformer layers J to 3, and the number of GPUs K to 8. In order to facilitate the grouping of GPUs, the time for training Transformer layers by different GPUs should be the same (which is also consistent with the actual situation). In addition, setting all forward calculation times and backward calculation times to be consistent is not very meaningful for this experiment. Meanwhile, in actual training: as the number of Transformer layers increases, the time for Transformer layers to complete training will also increase. Therefore, in this experiment, as the number of Transformer layers increases by one layer, the training time required for it is 25% higher than the first layer. In specific parameter settings, we set the time for forward calculation of Transformer layers 1, 2, and 3 to 24, 30, and 36 respectively, and the time for backward calculation to 48, 60, and 72 respectively. The total communication time is set to 24, and each GPU stores up to two layers of Transformer.

[0150] Under the data parallel strategy, there are 6 data parallel schemes: two groups of data parallelism: the number of GPUs is 2 and 6 (data parallel group 1), the number of GPUs is 3 and 5 (data parallel group 2), and the number of GPUs is 4 (data parallel group 3); three groups of data parallelism, the number of GPUs is 2, 2, 4 (data parallel group 4) and 2, 3, 3 (data parallel group 5), and a data parallel group of four groups with two GPUs in each group (data parallel group 6). We calculated the optimal training time of these six schemes respectively, and compared them with not using data parallelism and directly using the efficient optimization model. The experimental results are shown in the figure. Fig.10 As shown, all the results are from the experimental results. When the number of micro-batch data is 1-8, the training time of the efficient scheduling scheme is consistent with the optimal training time of data parallelism. When the number of micro-batch data is 9 and 10, the training time of the efficient scheduling scheme is slightly lower than the optimal training time of data parallelism. When the number of micro-batches is greater than 12, the efficient scheduling scheme is significantly lower than the optimal training time of data parallelism. However, according to the results of theoretical analysis, the scheduling scheme proposed in this paper should achieve consistent results with data parallelism when the amount of data increases. In the experiment, the results are obtained by searching only 500,000 nodes by the branch and bound method, indicating that directly using the high-profile optimization model cannot quickly solve the optimal scheduling strategy. In practical applications, combining the solution of the efficient optimization model with the data parallel strategy based on dynamic programming can more efficiently find the optimal scheduling solution.

[0151] The present invention takes training scheduling as the core of the problem, combines the specific characteristics of large model training, establishes a large model training scheduling model that comprehensively considers the communication overhead and training overhead in large model training, and the fluctuation of training time caused by uncertainty, and uses mixed integer programming to solve it. By introducing reliability constraints in the scheduling model, the present invention can ensure the stability and reliability of the system while improving the training efficiency, thereby reducing the unexpected downtime and additional training costs caused by failures. Specifically, the scheduling strategy of the present invention reduces the window period and delay phenomenon of the GPU by optimizing the resource utilization of the GPU, thereby significantly improving the overall training efficiency. Compared with the traditional single-machine single-card training method, through parallel computing, the present invention can achieve a faster and more stable training process for large models, effectively shortening the training time. At the same time, the optimized scheduling strategy also reduces the communication overhead during the training process, ensures the efficiency of data transmission, and further improves the overall performance of the system. In practical applications, the present invention applies the algorithm to the training of the GPT model to verify the feasibility and efficiency of the method. Experimental results show that the scheduling strategy of the present invention not only significantly improves the training speed, but also effectively reduces the training cost. In addition, by comprehensively considering communication and training overheads, the optimized scheduling strategy shows excellent results in resource utilization and performance improvement, providing new ideas for further improving the efficiency and reliability of large model training.

[0152] The embodiment of the present invention also provides a large model computing cluster service reliability evaluation and optimization device; the specific device may include:

[0153] The computing cluster modeling module is used to perform mathematical modeling on the large model computing cluster and define basic parameters and training tasks;

[0154] A training data modeling module, used to model the training time, wherein the training time includes the communication time caused by the GPU for data transmission and the delay caused by the network structure between the GPUs;

[0155] A reliability index definition module, used to define, based on the training time, a first reliability index of the large model computing cluster service as the probability that all micro-batch data complete the training task within the specified time, that is, the product of the probability that each micro-batch data completes a single task within the specified time;

[0156] The deterministic model is used to establish an objective function based on the first reliability index of the large model computing cluster service, and to model related constraints to construct a deterministic optimization model for parallel strategy scheduling of the large model computing cluster;

[0157] The initial solution generation module is used to obtain the initial solution of the scheduling strategy based on the deterministic optimization model using a mixed integer programming method.

[0158] The large model computing cluster service reliability evaluation and optimization device of this embodiment is used to implement the aforementioned large model computing cluster service reliability evaluation and optimization method. Therefore, the specific implementation method of the large model computing cluster service reliability evaluation and optimization device can be seen in the embodiment part of the large model computing cluster service reliability evaluation and optimization method in the previous text. For example, the computing cluster modeling module, the training data modeling module, the reliability index definition module, the deterministic model based module, and the initial solution generation module are respectively used to implement steps S101, S102, S103, S104 and S105 in the aforementioned large model computing cluster service reliability evaluation and optimization method. Therefore, its specific implementation method can refer to the description of the corresponding each part of the embodiment, which will not be repeated here.

[0159] A specific embodiment of the present invention also provides a large model computing cluster service reliability assessment and optimization device, including: a memory for storing computer programs; a processor for implementing the steps of the above-mentioned large model computing cluster service reliability assessment and optimization method when executing the computer program.

[0160] A specific embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned large model computing cluster service reliability evaluation and optimization method are implemented.

[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A large model computing cluster service reliability evaluation and optimization method, characterized in that: include: Mathematically model the large model computing cluster and define basic parameters and training tasks; Modeling training time, which includes communication time caused by data transfer between GPUs and latency caused by network structures between GPUs; Based on the training time, a first reliability index of the large model computing cluster service is defined as the probability that all micro-batch data complete the training task within the specified time, that is, the product of the probability that each micro-batch data completes a single task within the specified time; An objective function is established based on the first reliability index of the large model computing cluster service, and related constraints are modeled to construct a deterministic optimization model for parallel strategy scheduling of the large model computing cluster; Based on the deterministic optimization model, a mixed integer programming method is used to obtain an initial solution of the scheduling strategy.

2. The large model computing cluster service reliability evaluation and optimization method according to claim 1 is characterized in that: The mathematical modeling of the large model computing cluster and the definition of basic parameters and training tasks include: Build a GPT model containing only the decoder; Defining the basic parameters includes the initial micro-batch number, the number of Transformer layers, and the number of GPUs; The training task is defined to ensure that all micro-batch data complete a forward and backward calculation in order on each Transformer layer.

3. The large model computing cluster service reliability evaluation and optimization method according to claim 1 is characterized in that: The training time also includes random time fluctuations.

4. The large model computing cluster service reliability evaluation and optimization method according to claim 3 is characterized in that: After obtaining the initial solution of the scheduling strategy based on the deterministic optimization model by using a mixed integer programming method, the method further includes: generating a plurality of sets of actual training times based on an initial solution of the scheduling strategy; The initial solution is evaluated according to the second reliability index, and the initial solution with the highest reliability is output as the optimal solution of the scheduling strategy.

5. The large model computing cluster service reliability evaluation and optimization method according to claim 4 is characterized in that: The second reliability indicator is defined as the ratio of the time at which the forward calculation of the i-th micro-batch data at the first Transformer layer ends to the specified time at which the forward calculation of the i-th micro-batch data at the j-Transformer layer ends under a certain number of experiments.

6. The large model computing cluster service reliability evaluation and optimization method according to claim 1 is characterized in that: The communication time is determined by the amount of data in the training task and the computing rate of the GPU.

7. The large model computing cluster service reliability evaluation and optimization method according to claim 1 is characterized in that: The construction of a deterministic optimization model for parallel strategy scheduling of a large model computing cluster and the modeling of related constraints include: Define the data size of each group of micro-batch data to be the same; When defining the GPU to place the Transformer model, only the memory usage generated by storing the Transformer model is considered, and the memory usage generated by the Transformer model does not exceed the memory upper limit of the GPU; Defines that the GPU can perform forward or backward computation and data transfer simultaneously; Define that for the same micro-batch training task, the forward calculation and backward calculation must be performed on the same GPU; Define that the GPU can only perform one forward or backward computing task at a time; When the amount of data in each micro-batch is defined to be constant, the time required for forward and backward calculations is only related to the number of Transformer layers and GPU, and has nothing to do with the order of micro-batches; When defining micro-batch data to complete the training of different Transformer layers between different GPUs, data transmission will be performed, and the time consumption of data transmission is divided into communication time and latency; Define any training task. The forward calculation of the next layer of Transformer can be performed only after the forward calculation of the previous layer of Transformer is completed. The backward calculation can be performed only after all forward calculations are completed. The backward calculation of the previous layer of Transformer can be performed only after the backward calculation of the next layer of Transformer is completed.

8. A large model computing cluster service reliability evaluation and optimization device, characterized in that: include: The computing cluster modeling module is used to perform mathematical modeling on the large model computing cluster and define basic parameters and training tasks; A training data modeling module, used to model the training time, wherein the training time includes the communication time caused by the GPU for data transmission and the delay caused by the network structure between the GPUs; A reliability index definition module, used to define, based on the training time, a first reliability index of the large model computing cluster service as the probability that all micro-batch data complete the training task within the specified time, that is, the product of the probability that each micro-batch data completes a single task within the specified time; The deterministic model is used to establish an objective function based on the first reliability index of the large model computing cluster service, and to model related constraints to construct a deterministic optimization model for parallel strategy scheduling of the large model computing cluster; The initial solution generation module is used to obtain the initial solution of the scheduling strategy based on the deterministic optimization model using a mixed integer programming method.

9. A large model computing cluster service reliability evaluation and optimization device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of a large model computing cluster service reliability evaluation and optimization method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a large model computing cluster service reliability assessment and optimization method as described in any one of claims 1 to 7.

Images