A method and apparatus for reliability assessment and optimization of large-scale computing cluster services.

Abstract

This invention relates to the field of large model computing cluster technology, and in particular to a method, apparatus, device, and computer storage medium for evaluating and optimizing the reliability of large model computing cluster services based on parallel scheduling strategies. The method for evaluating and optimizing the reliability of large model computing cluster services described in this invention uses pipelined parallelism as its core, comprehensively considering the performance differences between different GPUs, data transfer time, and the impact of uncertainties on training time during the training process. It combines existing interleaved and non-interleaved pipelined parallel strategies to establish a model that fully considers the inconsistency of training time and integrates various pipelined parallel strategies. Furthermore, it provides specific constraints for the training process based on the specific characteristics of large model training, and solves the model using integer programming to achieve efficient and reliable training of large model computing clusters.

Description

Technical Field

[0001] This invention relates to the field of large model computing cluster technology, and in particular to a method, apparatus, equipment, and computer storage medium for evaluating and optimizing the reliability of large model computing cluster services based on parallel strategy scheduling. Background Technology

[0002] Since OpenAI launched ChatGPT, large language models have rapidly become a focus of attention. Large models refer to neural network models containing an extremely large number of parameters (typically exceeding one billion). Through specific large model training processes, they can analyze massive amounts of data and handle complex tasks. Among them, natural language processing models based on the Transformer architecture, through self-attention mechanisms, have achieved modeling of texts of arbitrary length, greatly improving language understanding capabilities. With the rapid development of Transformer-based natural language processing models, their parameter count and training scale have grown exponentially. The enormous model scale also places certain demands on the training process: large model training requires powerful computing resources; traditional single-machine, single-GPU training takes a long time to complete and can no longer meet the requirements of large model training. Parallel computing can achieve a faster and more stable training process for large models. Therefore, designing an excellent parallel scheduling strategy is crucial for large model training.

[0003] Parallelism is a crucial part of training large models. By rationally partitioning and allocating the training data and the model itself, multi-machine, multi-GPU training can be achieved, improving training efficiency and stability. Transformer-based natural language processing models consist of multiple encoding and decoding layers. Each encoding and decoding layer can be further divided into a feedforward neural network layer and an attention layer. Parallelism strategies can be categorized into data parallelism and model parallelism. Data parallelism involves partitioning the training data, with each training GPU retaining the complete training model, collectively completing one training iteration. Model parallelism involves partitioning the Transformer layers, and depending on the partitioning method, it can be further divided into pipeline parallelism and tensor parallelism. Pipeline parallelism distributes the encoding and decoding layers across different GPUs for training, while tensor parallelism distributes the weight matrices of the feedforward neural network layer and attention layer across different GPUs for training.

[0004] In actual training, each training data point undergoes forward computation sequentially across all Transformer layers, followed by backward computation in reverse order. Due to the training order constraints imposed by the parallel strategy, significant GPU downtime occurs during training, resulting in latency that substantially increases the overall training time. Even with the latest GPT-4 model, the actual GPU utilization during training is only 32%-36%.

[0005] Meanwhile, existing parallel strategies set the training time as a fixed value, without considering the performance differences between different training GPUs or the time required to transfer training data. This naive pipeline parallel strategy cannot cope with the uncertainties in the training process. Therefore, inspired by the modeling methods in industrial pipeline scheduling, this paper analyzes and combines different pipeline parallel strategies to optimize existing parallel strategies, achieving system service reliability assurance under uncertain environments. Existing methods for reliability assessment and optimization of large-scale model computing clusters based on parallel strategy scheduling still face the following challenges:

[0006] 1. Previous scheduling models modeled training time as one unit of time for forward computation and two units of time for backward computation due to recomputation. This is a relatively ideal condition and does not take into account the different training times that may occur when different GPUs train different Transformer layers. Randomness leads to fluctuations in training time and increases in training time due to hardware failure or aging.

[0007] 2. Existing models do not consider the time spent on data transfer. The training time on the GPU and the communication time between GPUs are on the same order of magnitude. Therefore, the impact of communication time on the total training time cannot be ignored when modeling.

[0008] 3. Although pipeline parallelism provides a scheduling scheme, it is a very conventional one: each micro-batch of data enters the training cluster sequentially according to its number to complete the training. Existing research has not provided an explanation for the rationality of this scheduling scheme.

[0009] 4. Previous pipeline parallel strategies only considered the case where the number of GPUs and Transformer layers was 1:1. However, in actual training, the number of GPUs is often much higher than the number of Transformer layers. This means that almost every Transformer layer will be assigned to multiple GPUs for training, and there is no corresponding optimization scheme for existing models.

[0010] In summary, existing technologies do not comprehensively consider the performance differences between different GPUs, data transfer time, and the impact of uncertainties on training time during the training process, resulting in inefficient and unreliable training of large model clusters. Summary of the Invention

[0011] Therefore, the technical problem to be solved by the present invention is to overcome the problems of inefficiency and poor reliability of large model cluster training in the prior art.

[0012] To address the aforementioned technical problems, this invention provides a method for reliability assessment and optimization of large-scale model computing cluster services, comprising:

[0013] Mathematical modeling is performed on large model computing clusters, and basic parameters and training tasks are defined;

[0014] The training time is modeled, including the communication time caused by data transfer between GPUs and the latency caused by the network structure between GPUs;

[0015] Based on the training time, the first reliability index of the large model computing cluster service is defined as the probability that all micro-batch data completes the training task within a specified time, that is, the product of the probabilities that each micro-batch data completes a single task within a specified time.

[0016] Based on the first reliability index of the large model computing cluster service, an objective function is established, and relevant constraints are modeled to construct a deterministic optimization model for parallel strategy scheduling of the large model computing cluster.

[0017] Based on the deterministic optimization model, the initial solution of the scheduling strategy is obtained using a mixed integer programming method.

[0018] Preferably, the step of mathematically modeling the large model computing cluster and defining basic parameters and training tasks includes:

[0019] Construct a GPT model containing only the decoder;

[0020] The basic parameters defined include the initial number of micro-batches, the number of Transformer layers, and the number of GPUs;

[0021] The training task is defined as ensuring that all micro-batch data completes one forward and one backward computation in sequence on each Transformer layer.

[0022] Preferably, the training time also includes random time fluctuations.

[0023] Preferably, after obtaining the initial solution of the scheduling strategy using the mixed-integer programming method based on the deterministic optimization model, the method further includes:

[0024] Multiple sets of actual training times are generated based on the initial solution of the scheduling strategy;

[0025] The initial solution is evaluated based on the second reliability index, and the initial solution with the highest reliability is output as the optimal solution for the scheduling strategy.

[0026] Preferably, the second reliability index is defined as the proportion of the time when the forward computation of the i-th micro-batch data in the first Transformer layer ends is less than the specified time when the forward computation of the i-th micro-batch data in the j-th Transformer layer ends, under a certain number of experiments.

[0027] Preferably, the communication time is determined by the amount of data in the training task and the computing speed of the GPU.

[0028] Preferably, the construction of a deterministic optimization model for parallel strategy scheduling of a large model computing cluster, and the modeling of relevant constraints, includes:

[0029] Define that the data volume of each micro-batch is the same;

[0030] When defining the GPU placement of a Transformer model, only the memory usage generated by storing the Transformer model is considered, and the memory usage generated by the Transformer model does not exceed the GPU's memory limit;

[0031] Define the GPU to perform forward or backward computation and data transfer simultaneously;

[0032] Define that for the same micro-batch training task, forward computation and backward computation must be performed on the same GPU;

[0033] The GPU is defined to perform only one forward or backward computation task at a time.

[0034] When the amount of data in each micro-batch is fixed, the time required for forward and backward computation depends only on the number of Transformer layers and the GPU, and is independent of the order of the micro-batches.

[0035] When micro-batch data is used to train different Transformer layers on different GPUs, data transmission will occur, and the time consumption of data transmission is divided into communication time and latency.

[0036] For any training task, the forward computation of the next Transformer can only be performed after the forward computation of the previous Transformer is completed, the backward computation can only be performed after all forward computations are completed, and the backward computation of the previous Transformer can only be performed after the backward computation of the next Transformer is completed.

[0037] This invention also provides a device for reliability assessment and optimization of large-scale model computing cluster services, comprising:

[0038] The computing cluster modeling module is used to perform mathematical modeling on large model computing clusters and define basic parameters and training tasks.

[0039] The training data modeling module is used to model the training time, which includes the communication time caused by data transfer between GPUs and the latency caused by the network structure between GPUs.

[0040] The reliability index definition module is used to define the first reliability index of the large model computing cluster service based on the training time as the probability that all micro-batch data completes the training task within a specified time, that is, the product of the probabilities of each micro-batch data completing a single task within a specified time.

[0041] The deterministic model, based on the module, is used to establish an objective function based on the first reliability index of the large model computing cluster service, and to model the relevant constraints, thereby constructing a deterministic optimization model for the parallel strategy scheduling of the large model computing cluster.

[0042] The initial solution generation module is used to obtain the initial solution of the scheduling strategy based on the deterministic optimization model using a mixed integer programming method.

[0043] This invention also provides a device for reliability assessment and optimization of large-scale model computing cluster services, comprising:

[0044] Memory, used to store computer programs;

[0045] A processor is used to implement the above-described method steps for reliability assessment and optimization of a large-scale computing cluster service when executing the computer program.

[0046] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for reliability assessment and optimization of a large-scale computing cluster service.

[0047] The technical solution of the present invention has the following advantages compared with the prior art:

[0048] In large-scale model training, training efficiency and cost are crucial considerations, while reliability risks require careful attention. The reliability of large-scale model computing clusters directly impacts their performance stability, security, and long-term operational efficiency. Traditional scheduling strategies struggle to handle dynamic changes during training and often overlook the overhead of inter-GPU communication. This invention aims to reduce GPU idle time and latency by optimizing parallel scheduling strategies, thereby improving actual GPU utilization, particularly addressing the low GPU utilization in current large-scale model training. Improved scheduling strategies significantly enhance training efficiency. By defining reliability metrics and incorporating communication overhead reliability constraints into the parallel training strategy, the expected performance standards can be ensured during training, while reducing unexpected downtime and training costs due to failures. Based on this, this invention proposes a mixed-integer programming optimization model for an efficient pipelined parallel training strategy, comprehensively considering communication and training overhead in large-scale model training, as well as training time fluctuations caused by hardware aging and uncertainty. The optimized parallel strategy not only improves training efficiency but also reduces additional costs due to GPU idle time and latency, lowering overall training costs through more efficient resource utilization. Attached Figure Description

[0049] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0050] Figure 1 This is a flowchart illustrating the implementation of a method for reliability assessment and optimization of large-scale computing cluster services provided by this invention.

[0051] Figure 2 Here is a diagram of the Transformer architecture;

[0052] Figure 3 This is a simple parallel process flow diagram for a pipeline.

[0053] Figure 4 This is a flowchart of the parallel process for a micro-batch pipeline.

[0054] Figure 5 This is a flowchart of an interleaved pipeline parallel process;

[0055] Figure 6 This is a flowchart illustrating the implementation of a large-scale computing cluster service reliability assessment and optimization method provided in an embodiment of the present invention.

[0056] Figure 7 Optimize experimental results for ideal communication time;

[0057] Figure 8 Optimize experimental results for critical communication time;

[0058] Figure 9 Optimize experimental results to double the critical communication time;

[0059] Figure 10 The results are from a data parallel experiment. Detailed Implementation

[0060] The core of this invention is to provide a method, apparatus, device, and computer storage medium for evaluating and optimizing the reliability of a large model computing cluster service, which effectively improves the efficiency and reliability of training large model clusters.

[0061] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Please refer to Figure 1 , Figure 1 This invention provides a flowchart for the implementation of a large-scale model computing cluster service reliability assessment and optimization method. The invention offers a method for optimizing the reliability of a large-scale model computing cluster service based on a mixed-integer programming model, generating a reliability optimization scheme for the large-scale model training cluster, thereby ensuring efficient training of the large-scale model computing cluster while minimizing training time. The specific operation steps are as follows:

[0063] S101: Perform mathematical modeling on large model computing clusters and define basic parameters and training tasks;

[0064] S102: Model the training time, which includes the communication time caused by data transfer between GPUs and the latency caused by the network structure between GPUs;

[0065] S103: Based on the training time, the first reliability index of the large model computing cluster service is defined as the probability that all micro-batch data completes the training task within a specified time, that is, the product of the probabilities that each micro-batch data completes a single task within a specified time.

[0066] S104: Based on the first reliability index of the large model computing cluster service, an objective function is established, and relevant constraints are modeled to construct a deterministic optimization model for parallel strategy scheduling of the large model computing cluster.

[0067] S105: Based on the deterministic optimization model, the initial solution of the scheduling strategy is obtained using the mixed integer programming method.

[0068] Based on the above embodiments, this embodiment will provide a detailed description of step S101:

[0069] • Construct a GPT model containing only the decoder;

[0070] The current mainstream large-scale models are natural language processing models based on Transformers. A Transformer can be divided into two parts: an encoding part and a decoding part. For example... Figure 2 In the Transformer shown, the left side represents the encoding part, and the right side represents the decoding part. The encoding part consists of multiple layers of encoders. The decoding part consists of multiple layers of decoders. Different encoders and decoders are structurally the same, but their weight matrix parameters differ. The encoder mainly consists of attention layers and a feed-forward neural network (FFNN).

[0071] In addition to these two layers, the decoder also inserts an Encoder-DecoderAttention layer between them. This layer receives information from the encoder, helping the decoder focus on the relevant parts of the input sentence. Furthermore, after each Attention layer and feedforward neural network layer in both the encoder and decoder, there is an Add & Normalize layer. Add represents residual connections to prevent network degradation, and Normalize represents layer normalization, used to normalize the activation values ​​of each layer.

[0072] In actual training, word vectors are encoded and then fed into the encoding section. After the word vectors are calculated in the first encoder layer, the output is used as the new input to the next encoder layer for calculation, and so on, calculating in all encoders to reinforce the output. After all encoders have completed their calculations, the encoding section passes the calculated key and value vectors to the encoder-decoder attention layer of each decoder. The final output is then used as the new input to the decoder's attention layer to calculate the query vector. After calculations by each decoder layer, the final output vector is obtained, where each element is a floating-point number. The linear layer is a typical fully connected neural network that maps the decoder's output vector into a longer vector called the logits vector, where each number represents a word's score. The softmax layer converts these scores into probabilities. Then, the word corresponding to the highest probability number is selected as the final predicted word for output.

[0073] With further research, the structure of the Transformer model has undergone some changes. Current Transformer models can be divided into three categories based on their structure: pure Encoder models containing only an Encoder, pure Decoder models containing only a Decoder, and traditional Encoder-Decoder models that contain both.

[0074] This invention focuses on the GPT model, which is a pure Decoder model, where training data is computed sequentially in both forward and backward directions on the Decoder. The fundamental objective of this invention is to ensure that each task completes one forward and one backward computation in sequence at each Transformer layer.

[0075] The basic parameters defined include the initial number of micro-batches, the number of Transformer layers, and the number of GPUs.

[0076] • The training task is defined as ensuring that all micro-batch data completes one forward and one backward computation in sequence on each Transformer layer.

[0077] Based on the above embodiments, this embodiment will provide a detailed description of step S102:

[0078] Pipeline parallelism involves partitioning the model into Transformer layers and then assigning one or more layers to a GPU. Data is then processed sequentially on different GPUs for forward and backward computations to complete a full training task.

[0079] Simple assembly line in parallel Figure 3 As shown, naive pipelined parallelism does not further partition the data. At the beginning and end of each Transformer layer's computation, the GPUs of other layers are idle, meaning only one GPU can perform model training at a time. In naive pipelined parallelism, as the number of parallel devices increases, the proportion of idle GPUs increases, and GPU utilization gradually decreases. Furthermore, due to communication overhead in data transfer between devices, multi-GPU parallel training is less efficient than direct training with a single GPU, provided memory allows.

[0080] To address the issue of low performance in naive pipelines, further processing of individual batches of data is performed. The completed training data is split into micro-batch training data, allowing simultaneous training on multiple GPUs, thereby improving GPU utilization. Figure 4In this architecture, a set of training data is divided into four micro-batches, where F and B represent forward and backward computations, respectively, and the numbers represent the micro-batch numbers. Under micro-batch pipeline parallelism, once micro-batch data 1 completes its forward computation in Transformer layer 1, Transformer layer 1 can simultaneously perform forward computation on micro-batch data 2 and pass the result of micro-batch data 1 to Transformer layer 2. Micro-batch data 1 can then immediately begin its forward computation in Transformer layer 2, unlike traditional pipeline parallelism which requires waiting for all forward computations in Transformer layer 1 to complete before starting its own forward computation. Micro-batch pipeline parallelism allows multiple GPUs to perform training tasks simultaneously, significantly improving training efficiency.

[0081] In this training mode, GPU utilization increases with the number of micro-batches. When the number of micro-batches approaches infinity, the utilization of each GPU approaches 1, and GPU idle time becomes negligible. However, increasing the number of micro-batches to improve the bubble score places high demands on memory because it requires storing a large number of intermediate computational variables. Therefore, in actual training, micro-batches are not cut indefinitely.

[0082] In addition, there is an interleaved 1F1B pipelined parallel model that can further improve GPU utilization. In interleaved 1F1B pipelined parallelism, the number of consecutive Transformer layers allocated to each GPU is reduced, but interleaved Transformer layers are allocated. Taking training a 16-layer Transformer on 4 GPUs as an example: In non-interleaved pipelined parallelism, Transformer layers 1, 2, 3, and 4 might be allocated to GPU1, Transformer layers 5, 6, 7, and 8 to GPU2, Transformer layers 9, 10, 11, and 12 to GPU3, and Transformer layers 13, 14, 15, and 16 to GPU4. In interleaved pipeline parallelism, during training, Transformer layers 1, 2, 9, and 10 are assigned to GPU1; Transformer layers 3, 4, 11, and 12 to GPU2; Transformer layers 5, 6, 13, and 14 to GPU3; and Transformer layers 7, 8, 15, and 16 to GPU4. Figure 5For example, each light-colored block represents the computation of the first 8 layers, and the dark-colored block represents the computation of the last 8 layers. In actual training, under non-interleaved training, data is trained on GPU1, GPU2, GPU3, and GPU4 sequentially for 16 layers. Under interleaved training, data is first trained on GPU1, GPU2, GPU3, and GPU4 for the first eight layers, and then intermediate variables are transferred from GPU4 to GPU1 to complete the last eight layers sequentially. Under non-interleaved training, a single micro-batch of data needs to complete 4 layers of computation continuously on each GPU, while under interleaved training, a single micro-batch of data only needs to complete 2 layers of computation continuously on each GPU. This makes the GPU idle time under interleaved training only 50% of that under non-interleaved training. However, at the same time, the communication frequency under interleaved scheduling doubles, and correspondingly, the communication time also doubles. Therefore, the interleaved scheduling strategy is actually a way to trade communication time for GPU utilization. In actual training, the impact of increased communication volume can be reduced by using high-speed network bandwidth in multi-GPU servers.

[0083] The communication time is determined by the amount of data in the training task and the computing speed of the GPU.

[0084] Existing research on scheduling strategies lacks modeling of training time. In actual training, training time consists of two parts: one is the communication time caused by data transfer by the GPU, which is determined by the amount of data in the training task and the computing speed of the GPU; the other is the latency caused by the network structure between GPUs, which is a fixed value and is related to the specific hardware performance.

[0085] The communication time for data transmission is positively correlated with the amount of data transmitted and inversely proportional to the transmission rate. Therefore, the model of this invention selects the micro-batch data size C and the bandwidth v between GPUs as relevant parameters for communication time, and denotes the latency as α to model the data transmission time. The total time required for data transmission is:

[0086]

[0087] Based on the above embodiments, this embodiment will provide a detailed description of step S103:

[0088] This invention defines the reliability of a large model computing cluster service as the probability that the computing cluster completes the training task within a specified time. This invention assigns the micro-batch data number i, the Transformer number j, and the GPU number k. The micro-batch data set is [I], the Transformer set is [J], and the GPU set is [K]. A complete training process involves all micro-batch data completing one forward and one backward computation on each Transformer layer. The time t at which the forward computation of micro-batch data i ends in Transformer layer j is defined as... ij The time τ when the micro-batch data i finishes its backward computation in the Transformer layer j. ij .

[0089] The specified time when the micro-batch data completes the forward computation of Transformer layer j is denoted as . The specified time for the micro-batch data to complete the backward computation in Transformer layer j is denoted as Therefore, the reliability of a single forward task is: The reliability of a single backward task is:

[0090] Let R be the overall reliability of the system's forward computation. f The overall reliability of the backward computation is R. b Considering the reliability of each task, the overall reliability of the training cluster can be obtained as follows:

[0091]

[0092] However, when setting time limits on a layer-by-layer basis and considering the reliability of all training tasks, releasing memory for backward computation tasks in advance is meaningless. The optimal strategy is to perform all forward computation tasks first, followed by all backward tasks. That is, the model will optimize the scheduling scheme according to the F-then-B (F-then-B) scheduling strategy. However, this invention does not intend for the model to necessarily optimize the scheduling scheme according to the F-then-B scheduling strategy. Therefore, considering the reliability of all training tasks under the premise of setting time limits on a layer-by-layer basis, setting a separate time limit for each training task is too cumbersome and meaningless. Therefore, the reliability index proposed in this invention only considers whether each training task can complete the training process within the specified time, and no longer focuses on whether the training task can be completed on time at each layer.

[0093]

[0094] Based on the above embodiments, this embodiment will provide a detailed description of step S104:

[0095] This invention focuses on the interleaving degree and scheduling strategy of pipeline parallelism to reduce the total training time of large models. The interleaving degree primarily affects the ratio of communication time to computation time, thus influencing the total training time. The scheduling strategy primarily affects the execution order of forward and backward computation tasks. During large model training, training data sequentially completes forward computation from the first Transformer layer to the last Transformer layer, and then sequentially completes backward computation from the last Transformer layer to the first Transformer layer. In actual training, the training data is equally divided into v micro-batches, reducing the bubbling time during training to [the original value]. This improves efficiency. Based on research into the training process of large models, the model assumptions of this invention are as follows:

[0096] (1) The amount of data in each micro-batch is the same.

[0097] (2) Placing the Transformer model on the GPU will occupy a certain amount of memory. The specific memory usage is related to the Transformer layer. Only the memory usage generated by storing the Transformer model is considered, and the memory usage generated by intermediate variables is not considered. The memory usage generated by storing the Transformer model cannot exceed the GPU's memory limit.

[0098] (3) GPUs can perform forward or backward computation and data transmission simultaneously.

[0099] (4) For the same micro-batch training task, forward computation and backward computation must be performed on the same GPU.

[0100] (5) A GPU can only perform one forward or backward computation task at a time.

[0101] (6) When the amount of data is fixed, the time required for forward and backward computation depends only on the number of Transformer layers and the GPU, and is not related to the order of micro-batch.

[0102] (7) When micro-batch data is trained on different Transformer layers between different GPUs, data transmission will occur. The time consumption of data transmission is divided into communication time and latency time. Communication time is related to the amount of micro-batch data and the communication rate between GPUs, while latency time is a fixed value.

[0103] (8) For any training task, the forward computation of the next layer of Transformer can only be performed after the forward computation of the previous layer Transformer is completed, the backward computation can only be performed after the forward computation is completed, and the backward computation of the previous layer Transformer can only be performed after the backward computation of the next layer Transformer is completed.

[0104] In one specific embodiment:

[0105] Define the time required for GPUk to complete the forward computation of Transformer layer j as f. jk The time required for GPUk to complete the backward computation of Transformer layer j is b. jk This is used as a parameter for modeling computation time; the memory capacity of the GPU is defined as G, and the memory space required by Transformerj is defined as g. j , which is used as a parameter for memory usage modeling.

[0106] To ensure the model satisfies its assumptions, a series of decision variables were set up for modeling. Decision variable z ijk The core decision variable, x, determines whether the computation of micro-batch data i in Transformer layer j is performed on GPU k. If the computation of micro-batch data i in Transformer layer j is performed on GPU k, the decision variable takes the value 1. jk The decision variable y is used to determine whether GPUk needs to store the model of Transformer layer j. If storage is required, the decision variable takes the value of 1. ij This is used to determine whether micro-batch data i needs to be transferred after computation in Transformer layer j. The value is 1 if data transfer is required. ij , τ ij These correspond to the time taken for micro-batch data i to complete forward and backward computations in Transformer layer j, respectively.

[0107] The optimization objective of this invention is to minimize the total training time. In the last Transformer layer, the end time of the backward computation of the last Transformer to finish computation across all micro-batch data is taken as the total training time of the model.

[0108]

[0109] Regarding the constraint (2) above, this invention defines a boolean variable x. jk It has been defined that when any task's computation in Transformer layer j needs to be performed on GPU k, it means that GPU k needs to store the model of Transformer layer j. In this case, x...jk The value is 1:

[0110]

[0111] On any single GPU, the memory usage for storing all Transformer models shall not exceed the GPU's memory capacity limit:

[0112]

[0113] For the constraint (4) above, each computational task is performed on a single GPU:

[0114]

[0115] Regarding the constraint (5) mentioned above, in actual training, a single GPU can only perform one training task at a time, F ijk B ijk This corresponds to the completion time of the computational task on a specific GPU k. ξ ijki′j′ , λ ijki′j′ Used to determine the order of two different tasks on the same GPU. When the decision variable is 1, it means that on GPU k, the computation task of data i' in Transformer layer j' precedes the computation task of data i in Transformer layer j. These decision variables are used to ensure that only one task is being computed on any GPU at any given time. ξ ijki′j It is responsible for determining the order of two forward computation tasks. Used to determine the order of two backward computation tasks, λ ijki′j′ This is to determine the forward computation task. ij and backward computation task i′j′ The order of priority.

[0116] Regarding the decision variable ξ ijki′j′ , λ ijki′j′ Since the order of the two tasks is mutually exclusive, the decision variable ξ for judging the two identical tasks is... ijki′j′ and ξ i′j′kij , and λ ijki′j and λ i′j′kij In any given pair, only one value can be 1, while the other is 0, that is:

[0117] ξ ijki′j′ +ξ i′j′kij ≤1

[0118]

[0119] λ ijki′j′ +λ i′j′kij ≤1

[0120] First, obtain the completion time of the computation task on the specific GPU:

[0121]

[0122] The specific mathematical modeling for having only one set of training data in computation at a time is as follows: First, determine whether the two computation tasks come from the same GPU. If the two computation tasks come from different GPUs, i.e., (2-z... ijk -z i′j′k When the expression is less than 1, the equation always holds true, and the model does not impose any time constraints on the two computational tasks. If the two computational tasks come from the same GPU, there must be a sequential relationship between them; the task that is computed first must finish before the task that is computed second must start. The constraints between two forward tasks on the same GPU are as follows:

[0123]

[0124] Suppose two different tasks, task A and task B, are on the same GPU. When task A is after task B, i.e., ξ ijki′j′ When the value is 1, the model performs the above judgment, indicating that the start time of task A is earlier than the end time of task B; when task A precedes task B, ξ ijki′j′ If the value is 1, the model will perform the following judgment: the end time of task A must be earlier than the start time of task B. The constraints for two backward tasks on the same GPU, and the corresponding constraints for any forward task and any backward task on the same GPU, are as follows:

[0125]

[0126]

[0127] Regarding the constraint (7) above, this invention defines a boolean variable y. ij When the computations of two adjacent layers of the same micro-batch data are not performed on the same GPU, a data transfer is required between these two computations. ij The value is 1:

[0128]

[0129] Regarding the constraint (8) above, for any training task, the forward computation task of the next Transformer layer can only begin after the forward computation task of the previous Transformer layer has been completed and the data has been transferred:

[0130]

[0131] For any training task, the backward computation task of the previous Transformer layer can only begin after the backward computation task of the next Transformer layer has been completed and the data has been passed.

[0132]

[0133] For any training task, the backward computation task can only begin after all forward computation tasks are completed.

[0134]

[0135] Based on the above embodiments, this embodiment will provide a detailed description of step S105:

[0136] • The training time also includes random time fluctuations;

[0137] In actual training, due to various software and hardware issues, training tasks often cannot be completed within the specified time. A timeout in one training task delays subsequent computations and the next task, creating gaps in the originally continuous training process and significantly increasing the total training time. This necessitates that our model consider training reliability under uncertain time conditions.

[0138] In reliability modeling, this invention changes the training time from a fixed value to a random variable. This random variable will fluctuate based on the original fixed value. The objective function remains the probability of the training task being completed within a specified time, but the reliability evaluation method is changed to the end time t under a certain number of experiments. i1 Less than The proportion.

[0139] In reliability optimization models, we can obtain feasible scheduling strategies through deterministic optimization models and other methods, and then further evaluate the reliability of the scheduling strategies through experiments. For example... Figure 6 The process of obtaining the initial solution for the scheduling strategy based on the deterministic optimization model using mixed-integer programming further includes:

[0140] Multiple sets of actual training times are generated based on the initial solution of the scheduling strategy;

[0141] The initial solution is evaluated based on the second reliability index, and the initial solution with the highest reliability is output as the optimal solution for the scheduling strategy.

[0142] The second reliability index is defined as the proportion of the time when the forward computation of the i-th micro-batch data in the first Transformer layer ends is less than the specified time when the forward computation of the i-th micro-batch data in the j-th Transformer layer, under a certain number of experiments.

[0143] Based on the above embodiments, this embodiment combines the training data of the GPT series large models to conduct experiments on the above mixed integer programming model. The experiments fully analyze the model's correctness under training time fluctuations and its efficiency compared to traditional scheduling strategies. All numerical experiments in this experiment were run on a computer equipped with an 11th Gen Intel(R) Core(TM) i7-11800H processor and an NVIDIA GeForce RTX 3060 graphics card. All algorithm code was implemented using Gurobi in a Python 3.12 environment. The results are obtained by searching 500,000 nodes using the branch and bound method in Gurobi.

[0144] In the experiment, the ideal forward training time and ideal backward training time were the same for all GPUs, and the forward training time and backward training time for all Transformer layers were also the same. The number of micro-batches I was set to 4, the number of Transformer layers J was set to 8, and the number of GPUs K was set to 4. The four GPUs in the non-interleaved pipeline parallelism stored Transformer layers 1 and 2, Transformer layers 3 and 4, Transformer layers 5 and 6, and Transformer layers 7 and 8, respectively. The four GPUs in the interleaved pipeline parallelism stored Transformer layers 1 and 5, Transformer layers 2 and 6, Transformer layers 3 and 7, and Transformer layers 4 and 8, respectively. To control variables, the mixed-integer programming model stipulated that each GPU could store a maximum of two Transformer layers.

[0145] In the experiment, the ideal time for forward computation was set to 24, and the ideal time for backward computation was set to 48. Since the experiment did not involve data partitioning, the communication time remained unchanged under the three pipelined parallel strategies. Therefore, the communication time and latency could be combined into the total communication time. This experiment set up three groups of experiments with total communication times of 0, 27, and 54. The total communication time of 0 was called the ideal communication time; the total communication time of 27 was the same for both traditional interleaved and non-interleaved pipelined parallelism, and was called the critical communication time; 54 was twice the critical communication time. Ten sub-experiments were conducted in each group. The training time of each Transformer layer in each sub-experiment fluctuated randomly between [100% and 125%]. The fluctuation of training time was consistent for the three pipelined parallel strategies within the same sub-experiment group. The experimental results are as follows: Figure 7-9 As shown, the solid line represents the experimental results, and the dashed line represents the ideal training time for the corresponding pipelined parallel strategy.

[0146] In-group experimental results show that, under ideal communication time, the training time of traditional non-interleaved pipelined parallelism is significantly longer than that of traditional interleaved pipelined parallelism. Our optimized model and the ideal computation time of traditional interleaved pipelined parallelism are consistent, indicating that the mixed-integer programming model of this patent has good consistency with previous research results, confirming the correctness of the optimized model. This also shows that, without considering other influencing factors, traditional interleaved pipelined parallelism is the optimal scheduling scheme. When training time fluctuates, the overall training time of the optimized model is the shortest, but the difference from traditional interleaved pipelined parallelism is not significant.

[0147] Under critical communication time, the ideal training time for both traditional interleaved and non-interleaved pipelined parallelism is the same, but both are significantly longer than the training time of the optimized model. This indicates that, under critical communication time, the training efficiency of both interleaved and non-interleaved pipelined parallelism is not ideal compared to the optimized model. Furthermore, since the training time fluctuates positively, in actual training, the training efficiency of non-interleaved pipelined parallelism is lower than that of interleaved pipelined parallelism.

[0148] Under double the critical communication time, the ideal training time of traditional interleaved pipeline parallelism is significantly higher than that of traditional interleaved pipeline parallelism. The ideal training time of the optimized model is similar to that of traditional non-interleaved pipeline parallelism. Secondly, the inter-group experimental results show that with increasing communication time, the training time of traditional interleaved pipeline parallelism increases more significantly, while the increase in training time of traditional non-interleaved pipeline parallelism is less. Our proposed efficient optimized model exhibits an optimization time similar to that of traditional interleaved pipeline parallelism when the communication time is short, and an optimization time similar to that of traditional non-interleaved pipeline parallelism when the communication time is long, consistently achieving the best optimization results among the three parallelism strategies. Further analysis of the experimental data revealed that the average forward computation time increased by 13.4%, the average backward computation time increased by 12.0%, and the weighted average training time increased by 12.5%. Under ideal communication time, the total training time for all three parallel strategies increased by more than 12.5%. This suggests that, under ideal communication time, the increase in training time caused by hardware failures in the training cluster is greater than the increase in training time caused by individual hardware failures. As the total communication time increases, the impact on the training cluster gradually decreases.

[0149] This embodiment also explores a practical combination of optimization model and data parallelism strategy, and conducts related numerical experiments. The experiment sets the number of micro-batch data I to 14, the number of Transformer layers J to 3, and the number of GPUs K to 8. To facilitate GPU grouping, the training time for different GPUs on the Transformer layers should be the same (which is consistent with reality). Furthermore, setting all forward and backward computation times to be identical is not very meaningful for this experiment. Meanwhile, in actual training: as the number of Transformer layers increases, the training time for each Transformer layer also increases. Therefore, in this experiment, with each additional Transformer layer, the required training time is 25% higher than the first layer. Specifically, we set the forward computation time for Transformer layers 1, 2, and 3 to 24, 30, and 36, respectively, and the backward computation time to 48, 60, and 72, respectively. The total communication time is set to 24, and each GPU stores a maximum of two Transformer layers.

[0150] Under the data parallelism strategy, there are six data parallelism schemes: two groups of data parallelism with 2 and 6 GPUs respectively (data parallelism group 1), 3 and 5 GPUs respectively (data parallelism group 2), and 4 GPUs each (data parallelism group 3); three groups of data parallelism with 2, 2, and 4 GPUs respectively (data parallelism group 4) and 2, 3, and 3 GPUs respectively (data parallelism group 5); and four groups of data parallelism with two GPUs each (data parallelism group 6). We calculated the optimal training time for each of these six schemes and compared it with not using data parallelism and directly using the efficient optimization model. The experimental results are as follows: Figure 10 As shown in the results, all experimental findings indicate that when the number of micro-batch data points is 1-8, the training time of the efficient scheduling scheme is consistent with the optimal training time of data parallelism. When the number of micro-batch data points is 9 and 10, the training time of the efficient scheduling scheme is slightly lower than the optimal training time of data parallelism. However, when the number of micro-batch data points is greater than 12, the training time of the efficient scheduling scheme is significantly lower than the optimal training time of data parallelism. However, according to theoretical analysis, the scheduling scheme proposed in this paper should achieve results consistent with data parallelism when the amount of data increases. In the experiment, the results were obtained by searching only 500,000 nodes using the branch-and-bound method, indicating that directly using the high-frequency optimization model cannot quickly solve for the optimal scheduling strategy. In practical applications, combining the efficient optimization model solution with a data parallel strategy based on dynamic programming can more efficiently find the optimal scheduling scheme.

[0151] This invention focuses on training scheduling as the core problem. Combining the specific characteristics of large-scale model training, it establishes a large-scale model training scheduling model that comprehensively considers communication overhead, training overhead, and training time fluctuations caused by uncertainty. The model is solved using mixed-integer programming. By introducing reliability constraints into the scheduling model, this invention can improve training efficiency while ensuring system stability and reliability, thereby reducing unexpected downtime and additional training costs caused by failures. Specifically, the scheduling strategy of this invention optimizes GPU resource utilization, reducing GPU idle periods and latency, thus significantly improving overall training efficiency. Compared to traditional single-machine, single-GPU training methods, this invention achieves a faster and more stable training process for large models through parallel computing, effectively shortening training time. Simultaneously, the optimized scheduling strategy also reduces communication overhead during training, ensuring efficient data transmission and further improving overall system performance. In practical applications, this invention applies the algorithm to the training of GPT models, verifying the feasibility and efficiency of the method. Experimental results show that the scheduling strategy of this invention not only significantly improves training speed but also effectively reduces training costs. Furthermore, by comprehensively considering communication and training overhead, the optimized scheduling strategy demonstrates excellent results in both resource utilization and performance improvement, providing a new approach to further enhance the efficiency and reliability of large model training.

[0152] This invention also provides a large-scale model computing cluster service reliability assessment and optimization device; the specific device may include:

[0153] The computing cluster modeling module is used to perform mathematical modeling on large model computing clusters and define basic parameters and training tasks.

[0154] The training data modeling module is used to model the training time, which includes the communication time caused by data transfer between GPUs and the latency caused by the network structure between GPUs.

[0155] The reliability index definition module is used to define the first reliability index of the large model computing cluster service based on the training time as the probability that all micro-batch data completes the training task within a specified time, that is, the product of the probabilities of each micro-batch data completing a single task within a specified time.

[0156] The deterministic model, based on the module, is used to establish an objective function based on the first reliability index of the large model computing cluster service, and to model the relevant constraints, thereby constructing a deterministic optimization model for the parallel strategy scheduling of the large model computing cluster.

[0157] The initial solution generation module is used to obtain the initial solution of the scheduling strategy based on the deterministic optimization model using a mixed integer programming method.

[0158] The large model computing cluster service reliability assessment and optimization device of this embodiment is used to implement the aforementioned large model computing cluster service reliability assessment and optimization method. Therefore, the specific implementation of the large model computing cluster service reliability assessment and optimization device can be found in the previous embodiment section of the large model computing cluster service reliability assessment and optimization method. For example, the computing cluster modeling module, training data modeling module, reliability index definition module, deterministic model basis module, and initial solution generation module are respectively used to implement steps S101, S102, S103, S104, and S105 in the above-mentioned large model computing cluster service reliability assessment and optimization method. Therefore, its specific implementation can be referred to the description of the corresponding embodiment, and will not be repeated here.

[0159] A specific embodiment of the present invention also provides a device for reliability assessment and optimization of large model computing cluster services, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-described method for reliability assessment and optimization of large model computing cluster services.

[0160] A specific embodiment of the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for reliability assessment and optimization of a large-scale computing cluster service.

[0161] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for reliability assessment and optimization of large-scale model computing cluster services, characterized in that, include: Mathematical modeling is performed on large model computing clusters, and basic parameters and training tasks are defined; The training time is modeled, including the communication time caused by data transfer between GPUs and the latency caused by the network structure between GPUs; Based on the training time, the first reliability index of the large model computing cluster service is defined as the probability that all micro-batch data completes the training task within a specified time, that is, the product of the probabilities that each micro-batch data completes a single task within a specified time. Based on the first reliability index of the large-scale computing cluster service, an objective function is established, and relevant constraints are modeled to construct a deterministic optimization model for the parallel strategy scheduling of the large-scale computing cluster: Define that the data volume of each micro-batch is the same; When defining the GPU placement of a Transformer model, only the memory usage generated by storing the Transformer model is considered, and the memory usage generated by the Transformer model does not exceed the GPU's memory limit; Define the GPU to perform forward or backward computation and data transfer simultaneously; Define that for the same micro-batch training task, forward computation and backward computation must be performed on the same GPU; The GPU is defined to perform only one forward or backward computation task at a time. When the amount of data in each micro-batch is fixed, the time required for forward and backward computation depends only on the number of Transformer layers and the GPU, and is independent of the order of the micro-batches. When micro-batch data is used to train different Transformer layers on different GPUs, data transmission will occur, and the time consumption of data transmission is divided into communication time and latency. Define any training task such that the forward computation of the next Transformer can only be performed after the forward computation of the previous Transformer is completed, the backward computation can only be performed after all forward computations are completed, and the backward computation of the previous Transformer can only be performed after the backward computation of the next Transformer is completed. Based on the deterministic optimization model, the initial solution of the scheduling strategy is obtained using a mixed integer programming method.

2. The method for reliability assessment and optimization of large-scale model computing cluster services according to claim 1, characterized in that, The mathematical modeling of the large model computing cluster, and the definition of basic parameters and training tasks, include: Construct a GPT model containing only the decoder; The basic parameters defined include the initial number of micro-batches, the number of Transformer layers, and the number of GPUs; The training task is defined as ensuring that all micro-batch data completes one forward and one backward computation in sequence on each Transformer layer.

3. The method for reliability assessment and optimization of large model computing cluster services according to claim 1, characterized in that, The training time also includes random time fluctuations.

4. The method for reliability assessment and optimization of large model computing cluster services according to claim 3, characterized in that, The process of obtaining the initial solution of the scheduling strategy based on the deterministic optimization model using a mixed-integer programming method further includes: Multiple sets of actual training times are generated based on the initial solution of the scheduling strategy; The initial solution is evaluated based on the second reliability index, and the initial solution with the highest reliability is output as the optimal solution for the scheduling strategy.

5. The method for reliability assessment and optimization of large model computing cluster services according to claim 4, characterized in that, The second reliability index is defined as the proportion of the time when the forward computation of the i-th micro-batch data in the first Transformer layer ends is less than the specified time when the forward computation of the i-th micro-batch data in the j-th Transformer layer, under a certain number of experiments.

6. The method for reliability assessment and optimization of large model computing cluster services according to claim 1, characterized in that, The communication time is determined by the amount of data in the training task and the computing speed of the GPU.

7. A device for reliability assessment and optimization of large-scale model computing cluster services, characterized in that, include: The computing cluster modeling module is used to perform mathematical modeling on large model computing clusters and define basic parameters and training tasks. The training data modeling module is used to model the training time, which includes the communication time caused by data transfer between GPUs and the latency caused by the network structure between GPUs. The reliability index definition module is used to define the first reliability index of the large model computing cluster service based on the training time as the probability that all micro-batch data completes the training task within a specified time, that is, the product of the probabilities of each micro-batch data completing a single task within a specified time. The deterministic model, based on the module, establishes an objective function based on the first reliability index of the large model computing cluster service, and models relevant constraints to construct a deterministic optimization model for parallel strategy scheduling of the large model computing cluster. Define that the data volume of each micro-batch is the same; When defining the GPU placement of a Transformer model, only the memory usage generated by storing the Transformer model is considered, and the memory usage generated by the Transformer model does not exceed the GPU's memory limit; Define the GPU to perform forward or backward computation and data transfer simultaneously; Define that for the same micro-batch training task, forward computation and backward computation must be performed on the same GPU; The GPU is defined to perform only one forward or backward computation task at a time. When the amount of data in each micro-batch is fixed, the time required for forward and backward computation depends only on the number of Transformer layers and the GPU, and is independent of the order of the micro-batches. When micro-batch data is used to train different Transformer layers on different GPUs, data transmission will occur, and the time consumption of data transmission is divided into communication time and latency. Define any training task such that the forward computation of the next Transformer can only be performed after the forward computation of the previous Transformer is completed, the backward computation can only be performed after all forward computations are completed, and the backward computation of the previous Transformer can only be performed after the backward computation of the next Transformer is completed. The initial solution generation module is used to obtain the initial solution of the scheduling strategy based on the deterministic optimization model using a mixed integer programming method.

8. A device for reliability assessment and optimization of large-scale model computing cluster services, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the method for reliability assessment and optimization of a large-scale computing cluster service as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for reliability assessment and optimization of a large-scale computing cluster service as described in any one of claims 1 to 6.

Images