Time consumption prediction method and device for parallel training of heterogeneous system sequences and program product

Abstract

The invention discloses a time consumption prediction method and device for sequence parallel training of a heterogeneous system and a program product, and is applied to the technical field of distributed training, and the method comprises the steps: obtaining parameter information reflecting a training condition during sequence parallel training; based on the parameter information, determining calculation time, communication time and self-attention execution time of the node before self-attention execution, and calculation time from the end of self-attention execution to the next converter block, and determining forward propagation time consumption of the node in a single iteration training process; based on the parameter information, determining back propagation time consumption of the node in a single iteration training process; and superposing the maximum value of the forward propagation time consumption of each node with the maximum value of the back propagation time consumption of each node, and taking an obtained result as the time consumption of the single iteration training process of the to-be-trained model. By applying the scheme of the invention, the time consumption prediction of the sequence parallel training of the heterogeneous system can be effectively realized.

Description

Technical Field

[0001] This application relates to the field of distributed training technology, and in particular to a method, device and program product for predicting the time consumption of parallel training of heterogeneous systems. Background Technology

[0002] Currently, distributed training has become one of the mainstream training methods. By splitting large models and deploying the split models on multiple computing nodes, distributed training of large models can be achieved. On the other hand, in recent years, the concept of multi-heterogeneous computing systems has been gradually proposed. In such systems, heterogeneous computing power with different computing performance, memory capacity, and communication capabilities can be deployed in the same distributed computing environment to collaboratively complete the distributed training task of the model.

[0003] Sequence parallel training is a distributed training technique proposed in recent years. Specifically, it is an architectural design that distributes the sequence dimension across multiple devices. It can break down long sequence tasks into multiple sub-sequences, allowing them to be processed on different devices, alleviating memory pressure on individual devices, and improving overall training efficiency through parallel computing. Currently, some research has proposed predicting the training time of distributed training, but most of these studies focus on traditional data parallelism, model parallelism, and pipelined parallelism. For sequence parallel training on heterogeneous systems, reasonable predictions of training time are still not possible.

[0004] In summary, how to effectively predict the time consumption of parallel training sequences in heterogeneous systems is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This application provides a method, device, and program product for predicting the time consumption of parallel training sequences in heterogeneous systems, so as to effectively predict the time consumption of parallel training sequences in heterogeneous systems.

[0006] This application provides a method for predicting the time consumption of sequential parallel training in heterogeneous systems, including: Obtain parameter information that reflects the training status of a heterogeneous system during sequence-parallel training of a model to be trained; Based on the parameter information, for any node in the heterogeneous system, the computation time of a single transformer block of the model to be trained before self-attention execution, the communication time before self-attention execution, the self-attention execution time, and the computation time from the completion of self-attention execution to the next transformer block are determined as the forward propagation time information of the node during a single iteration of training. Based on the forward propagation time information of the node, the forward propagation time of the node in a single iteration of training is determined; Based on the parameter information, for any node in the heterogeneous system, the backpropagation time of the node in a single iteration of training is determined; The maximum forward propagation time of each node is added together with the maximum backward propagation time of each node, and the result is used as the time consumption of a single iteration of the training process of the model to be trained.

[0007] This application also provides an electronic device, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the time-consuming prediction method for parallel training of heterogeneous system sequences as described above.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the time-consuming prediction method for parallel training of heterogeneous system sequences as described above.

[0009] To effectively predict the training time of a heterogeneous system's sequential parallel training using the technical solution of this application, firstly, it is necessary to obtain parameter information reflecting the training status of the heterogeneous system during sequential parallel training of the model to be trained. After obtaining the parameter information, for any node in the heterogeneous system, the backpropagation time and forward propagation time of the node in a single iteration of training can be determined based on the parameter information. Backpropagation requires waiting for each node to finish its forward propagation; therefore, the maximum forward propagation time of each node needs to be superimposed with the maximum backpropagation time of each node, and the result is used as the training time of the model in a single iteration. Furthermore, this application takes into account that for parallel training of heterogeneous systems, during a single iteration of training, data interaction between nodes is involved in the execution of each transformer block during forward propagation. Therefore, based on parameter information, it is necessary to determine the forward propagation time information of any node in the heterogeneous system for each node during a single iteration of training. This includes the computation time of a single transformer block of the model to be trained before self-attention execution, the communication time before self-attention execution, the self-attention execution time, and the computation time from the completion of self-attention execution to the next transformer block. These four stages are called the forward propagation time information of the node, so that the forward propagation time of the node can be accurately and effectively determined through the forward propagation time information of the node.

[0010] In summary, the proposed solution can effectively predict the time consumption of a single iteration of training for a model, thus enabling time prediction for parallel training of heterogeneous systems. Furthermore, it should be noted that predicting the time consumption of parallel training provides a quantitative basis for the planning, optimization, and management of parallel training, and is applicable in various scenarios. For example, based on the predicted time consumption, staff can plan training resources; for instance, if the time consumption is too long, the number of GPUs can be increased, the structure of the model to be trained can be simplified, the distributed training configuration can be adjusted, and the communication strategy between nodes can be adjusted so that the time consumption meets the staff's requirements. Moreover, during subsequent training execution, the actual time consumption can be compared with the predicted time consumption. If the difference is too large, it may be due to faulty nodes, incorrect parameter configuration, or other reasons, allowing staff to promptly identify and troubleshoot the problem. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 The flowchart illustrates the implementation of a method for predicting the time consumption of parallel training sequences in heterogeneous systems according to a specific embodiment of the present invention. Figure 2 This is a schematic diagram of the heterogeneous system. Figure 3 This is a schematic diagram of the structure of an electronic device provided in a specific embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer-readable storage medium according to the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating the implementation of a method for predicting the time consumption of heterogeneous system sequence parallel training according to a specific embodiment of the present invention. The method may include the following steps: Step S101: Obtain parameter information that reflects the training status of the heterogeneous system during sequential parallel training of the model to be trained.

[0016] Specifically, a heterogeneous system, also known as a heterogeneous computing system, includes multiple nodes, which can also be called heterogeneous computing power or heterogeneous computing nodes. The heterogeneous system described in this application refers to a system where nodes can be accelerator cards from different manufacturers with different computing performance, and communication between different nodes can be intra-server communication or inter-server communication. Figure 2 This is a schematic diagram of the heterogeneous system, showing heterogeneous computing power 1 to heterogeneous computing power 4. In this example, the heterogeneous system includes four nodes. Of course, in practical applications, the number of nodes in a heterogeneous system can be much greater. It should also be noted that in this application's scheme, the nodes in the heterogeneous system can also be configured with accelerator cards from the same manufacturer and with the same computing performance, and the time prediction for sequential parallel training can still be achieved according to the principles of this application.

[0017] Sequence parallelism (SP) is a distributed training technique used in the training of large models, such as LLMs (Large Language Models). SP "slices" a long input sequence, distributing it across multiple heterogeneous computing resources (e.g., GPUs) for parallel processing. In other words, it breaks down a long sequence into multiple sub-sequences along the sequence dimension, which are then processed by different nodes, effectively improving overall training efficiency. The forward propagation process in SP primarily involves the computational and communication overhead of the attention mechanism, making it computationally intensive and bandwidth-sensitive. By optimizing the sharding strategy and communication mode, training time can be significantly reduced and resource utilization improved.

[0018] The steps of sequence-parallel training can be divided into: a. Input sequence segmentation. b. Parallel forward propagation. c. Communication and exchange of intermediate results. d. Attention calculation. e. Parallel backpropagation. f. Parameter update and synchronization. Step a refers to uniformly or dynamically segmenting the input sequence along the sequence dimension into multiple parts. The resulting subsequences are assigned to each node, and the boundary information of the subsequences (such as token indices) is recorded for subsequent calculation and communication. Step b refers to each node independently performing forward propagation of the model on its assigned subsequence, including the attention mechanism (self-attention) and the feed-forward network. In the attention mechanism, heterogeneous computing power only calculates the query, key, and value matrices of the local subsequence, reducing computational complexity and memory requirements. Step c refers to the attention mechanism requiring global sequence information; therefore, nodes communicate to exchange key and value matrices (KV cache). Step d refers to each node using its local query matrix and the exchanged global key and value matrices to calculate the attention output. Step e refers to the process where, based on the output of forward propagation, each node can independently calculate the gradient for its subsequence and perform backpropagation. Backpropagation only needs to process the local subsequence, reducing computational and memory overhead. Step f refers to the process where the global gradient can be aggregated through an All-Reduce operation to update the model parameters, ensuring that the model parameters are consistent across all nodes and preparing for the next iteration. Steps b to f constitute a complete iterative process including forward and backpropagation, which is a single iteration of training. By repeating this iterative process, subsequent input batches can be processed until a specified number of rounds are completed or the model converges, at which point training can end.

[0019] To effectively predict training time, it's necessary to pre-acquire parameter information reflecting the training status of the heterogeneous system during sequence-parallel training of the model to be trained. In one specific implementation, this parameter information may include training task information and node information of the heterogeneous system. The training task information reflects the training status during sequence-parallel training and includes information about the model architecture and training data configuration. The node information reflects the performance of each node, such as its individual computing power and the communication bandwidth between nodes. By combining these information, the training status during sequence-parallel training can be comprehensively and accurately reflected. For example, in one implementation, the training task information includes the input sequence length, the hidden dimension of the model to be trained, the intermediate dimension of the feedforward neural network, the number of layers in the transformer block, the sequence parallelism, and the total number of parameters. The node information of the heterogeneous system includes the total number of nodes, the computing power of each node, the communication bandwidth between every two nodes, and the memory bandwidth of each node. Based on this information, the time consumption of each stage can be effectively calculated, ultimately yielding the predicted time consumption of the sequence-parallel training.

[0020] The length of the input sequence is denoted as l, which represents the length of the token sequence input into the model to be trained (e.g., 512 for BERT, 2048 for GPT). The length of the input sequence affects the computation and memory access scale. In practical applications, the value of l can be obtained from the training dataset configuration, from the preprocessing code, by checking the parameters of the data loader, or by manual input.

[0021] The hidden dimension of the model to be trained is denoted as h. Since the Transformer is a deep learning neural network architecture based on self-attention, and is the core infrastructure of current large language models (LLM) and other types of trainable models, the hidden dimension of the model to be trained described in this application is usually specifically the hidden dimension of the Transformer model, that is, the hidden state dimension of the Transformer. It is the length of the feature vector corresponding to each token in the model (e.g., 768 for BERT, 12288 for GPT-3), which determines the scale of matrix operations. In practical applications, the hidden dimension h of the model to be trained can be obtained from the model architecture configuration file, by checking the model code definition, or by manual input.

[0022] The intermediate dimension of the feedforward neural network of the model to be trained is denoted as d. ff This refers to the intermediate dimension of the FNN (Feed-Forward Network) layer (usually 4 times the hidden dimension h, such as 3072 in BERT), dff This will affect FFN computation and memory access. In practical applications, d can be obtained from the model architecture configuration file. ff Alternatively, you can check the FFN layer code definition to obtain it, or it can be manually entered.

[0023] The number of transformer blocks in the model being trained is denoted by L, which stands for Transformer Block Number. For example, BERT has 12 layers, and GPT-3 has 96 layers. L determines the repetition of computation and communication layers. In practical applications, L can be obtained from the model architecture configuration file or entered manually.

[0024] Sequence parallelism, denoted by D, represents the number of slices after the input sequence is partitioned. It determines the length of the subsequence processed by each node, affecting computation and memory allocation. In practical applications, the value of D can be obtained from the distributed training configuration, by checking the parameters in the training script or framework code, or by manual input.

[0025] The total number of parameters in the model to be trained is denoted as P, representing the total number of bytes of all weight parameters in the model, which affects global gradient synchronization communication. In practical applications, the model architecture can be determined based on the model architecture configuration file, and then P can be calculated according to the model architecture, or it can be manually input.

[0026] The total number of nodes, denoted as N, represents the total computing power (such as GPUs and TPUs) in the heterogeneous system, which determines the parallelism scale and communication overhead. This data can be collected directly, and an independent index can be set for each node.

[0027] The computing power of a node is typically measured in floating-point operations per second (FLOPs / s), which determines the time taken to complete a computation task. In practical applications, this can be obtained from hardware specifications or by running benchmark tests (such as matrix multiplication) on the node to directly measure its actual FLOPs / s. It is understood that each node has its own computing power; in this application, the computing power of the i-th node is denoted as C. i , where i is a positive integer and represents the i-th node in the heterogeneous system, and i can take values ​​from 1 to N.

[0028] The communication bandwidth between any two nodes reflects the data transfer rate between them (e.g., NVLink 600GB / s, InfiniBand 100Gb / s), and affects gradient synchronization time. In practical applications, this bandwidth can be obtained by consulting hardware specifications or by measuring the communication bandwidth between nodes using communication benchmarks. (B can be used...) ijThis represents the communication bandwidth between node i and node j. Furthermore, it should be noted that the average mutual communication bandwidth B between all nodes will be used in the following implementation. This value can be determined based on the communication bandwidth between any two nodes, and can be expressed as: Here, A represents the permutation operator.

[0029] The memory bandwidth of each node refers to the individual memory access rate (bytes / s) of each node, which determines the memory access time (such as loading gradients and weights). In practical applications, this can be obtained from hardware specifications or measured by running benchmark tests (such as STREAM benchmark or DeepSpeed ​​memory copy test) on the node. It is understood that each node has its own memory bandwidth. In this application, the memory bandwidth of the i-th node is denoted as M. bw，i , where i is a positive integer and represents the i-th node in the heterogeneous system, and i can take values ​​from 1 to N.

[0030] Step S102: Based on the parameter information, for any node in the heterogeneous system, determine the computation time of a single transformer block of the model to be trained before self-attention execution, the communication time before self-attention execution, the self-attention execution time, and the computation time from the completion of self-attention execution to the next transformer block during a single iteration of training. These are used as the forward propagation time information of the determined node.

[0031] After obtaining the node parameter information, based on the sequence parallel training process, the forward propagation time and backpropagation time can be determined using the parameter information. For forward propagation, this application considers that data interaction between nodes is involved in the execution of each transformer block. Therefore, based on the parameter information, for any node in the heterogeneous system, it is necessary to determine the computation time of a single transformer block of the model to be trained before self-attention execution, the communication time before self-attention execution, the self-attention execution time, and the computation time from the completion of self-attention execution to the next transformer block during a single iteration of training. Then, based on the times of these four stages, the forward propagation time of that node can be accurately and effectively determined. Of course, there are multiple ways to determine the specific implementation of these four stages; the key is to accurately obtain the times of these four stages.

[0032] For example, in one specific embodiment of the present invention, based on parameter information, determining the computation time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training for any node in a heterogeneous system can specifically include: Based on parameter information, for any node in a heterogeneous system, determine the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training, and use this value as the computation time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training.

[0033] This implementation takes into account that data computation and memory access can occur simultaneously during the computation phase before self-attention execution in each transformer block. Therefore, for any node, it is necessary to determine the data computation time and memory access time before self-attention execution, and then use the maximum of the two as the computation time of a single transformer block of the model to be trained in a single iteration of training for that node before self-attention execution. For the computation time of node i before self-attention execution, this application uses T. f，pre，i In this expression, i represents the i-th node, f represents forward propagation, and pre represents the computation time before the self-attention is executed, that is, the computation time of the transformer block before its own attention.

[0034] It should also be noted that the time consumption of a single iteration training described in this application, i.e., one step, is the time consumption of a single iteration training process of the model to be trained, i.e. the time consumption of a single step, including one complete forward propagation and backward propagation process.

[0035] Furthermore, in one specific embodiment of the present invention, based on parameter information, for any node in a heterogeneous system, the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training is determined, and used as the computation time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training. This can specifically include: Based on parameter information, for any node in a heterogeneous system, the computation time of a single transformer block of the model to be trained before self-attention execution is determined according to a preset second calculation formula during a single iteration of training. The second calculation formula is: .

[0036] Where i is a positive integer and represents the i-th node in the heterogeneous system, f represents forward propagation; T f，pre，i This represents the computation time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node; max indicates taking the maximum value; T pre，comp，i This represents the data computation time for a single transformer block of the model to be trained before self-attention is executed, for the i-th node; Tpre，mem，i This represents the memory access time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node.

[0037] The parameter represents the computation time correction parameter, μ represents the memory access time correction parameter, l represents the input sequence length, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and C represents the computation time correction parameter. i M represents the computing power of the i-th node. bw，i This represents the memory bandwidth of the i-th node.

[0038] In this implementation method, T is calculated according to the second formula. f，pre，i The calculation, where the first term in the formula is the time consumed by the data processing in this stage, i.e., T. pre，comp，i The computation time for residual connections and layer normalization (calculating mean / variance and normalization) after the previous transformer block ends is a pointwise operation. Therefore, the approximate complexity can be expressed as (l / D)×h. Dividing this value by the computational power C of the i-th node... i This allows us to estimate the data processing time for that stage, denoted as T. pre，comp，i The second term in the formula is the memory access time for this stage. Since the amount of active data output from the previous block can be expressed as (l / D)×h in this stage, this value can be divided by the memory bandwidth M of the i-th node. bw，i This gives the memory access time for that stage, denoted as T. pre，mem，i .

[0039] Since the data processing and memory access in this stage almost overlap, we can take T. pre，comp，i and T pre，mem，i The maximum value of these two values ​​reflects the time bottleneck of that stage. In other words, the maximum value of T represents the time consumed by the i-th node in that stage. f，pre，i .

[0040] Before its own attention, the transformer block needs to perform computation and communication prior to attention. In one specific embodiment of the present invention, based on parameter information, for any node in a heterogeneous system, determining the communication time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training can specifically include: Based on parameter information, for any node in a heterogeneous system, the communication time of a single transformer block of the model to be trained before self-attention execution is determined according to a preset third calculation formula during a single iteration of training. The third calculation formula is: .

[0041] Where i is a positive integer and represents the i-th node in the heterogeneous system, f represents forward propagation; T f，comm，i This represents the communication time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node. β represents the communication time correction parameter, l represents the length of the input sequence, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and B represents the average inter-node communication bandwidth.

[0042] This implementation takes into account that, at this stage, each transformer block needs to synchronously collect data from other parallel computing power before calculating attention. Since all-to-all communication is used, the average mutual communication bandwidth B is used for time-consuming calculation. The calculation method of B has been given above and will not be repeated.

[0043] This stage represents the communication time before attention is executed. Since the approximate size of the KV matrix can be represented by (l / D)×h, it reflects the amount of data that each node needs to transmit in this stage. D-1 is the number of transmissions. Multiplying this by (l / D)×h gives the amount of data that a single node needs to receive. Dividing this by the average mutual communication bandwidth B gives a value that effectively reflects the time consumption of this stage, denoted as T. f，comm，i .

[0044] In one specific embodiment of the present invention, based on parameter information, determining the self-attention execution time of a single transformer block of the model to be trained during a single iteration of training for any node in a heterogeneous system may specifically include: Based on parameter information, for any node in a heterogeneous system, determine the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained during self-attention execution in a single iteration of training, and use it as the self-attention execution time of a single transformer block of the model to be trained in a single iteration of training.

[0045] This implementation takes into account that data computation and memory access can occur simultaneously during attention execution. Therefore, for any node, it is necessary to determine the data computation time and memory access time during self-attention execution, and then use the maximum of the two as the self-attention execution time of a single transformer block of the model to be trained during a single iteration of training. For the self-attention execution time of node i, this application uses T. f，attn，i Let i represent the i-th node, f represent forward propagation, and attn represent the self-attention execution time.

[0046] Furthermore, in one specific embodiment of the present invention, based on parameter information, for any node in a heterogeneous system, the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained during self-attention execution in a single iteration of training is determined, and used as the self-attention execution time of a single transformer block of the model to be trained in a single iteration of training. This can specifically include: Based on parameter information, for any node in a heterogeneous system, the self-attention execution time of a single transformer block of the model to be trained during a single iteration of training is determined according to a preset fourth calculation formula. The fourth calculation formula is: .

[0047] Where i is a positive integer and represents the i-th node in the heterogeneous system, f represents forward propagation; T f，attn，i This represents the self-attention execution time of a single transformer block of the model to be trained for the i-th node; max indicates taking the maximum value; T attn，comp，i This represents the data computation time for a single transformer block of the model to be trained during self-attention execution, for the i-th node; T attn，mem，i This represents the memory access time of a single transformer block of the model to be trained during self-attention execution for the i-th node.

[0048] The parameter represents the computation time correction parameter, μ represents the memory access time correction parameter, l represents the input sequence length, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and C represents the computation time correction parameter. i M represents the computing power of the i-th node. bw，i This represents the memory bandwidth of the i-th node.

[0049] In this implementation method, it is according to The calculation method for T f，attn，i The calculation, where the first term in the formula is the time consumed by the data processing in this stage, i.e., T. attn，comp，i Furthermore, in this implementation, the approximate complexity of the attention matrix multiplication is expressed as (l / D)×h×l, and this value is divided by the computational power C of the i-th node. i This allows us to estimate the data processing time for that stage, denoted as T. attn，comp，i The second term in the formula represents the memory access time at this stage. Since the memory access at this stage specifically involves loading the global key-value pair (data size h×l) and the local key-value pair (l / D)×h, we can add (l / D)×h to (h×l) and then divide by the memory bandwidth M of the i-th node. bw，i This gives the memory access time for that stage, denoted as T. attn，mem，i .

[0050] Since the data processing and memory access in this stage almost overlap, we can take T. attn，comp，i and T attn，mem，i The maximum value of the two values ​​can reflect the time bottleneck of that stage and serve as the execution time of attention. In other words, the maximum value of the two values ​​serves as the attention execution time of the i-th node.

[0051] In one specific embodiment of the present invention, based on parameter information, for any node in a heterogeneous system, the computation time of a single transformer block of the model to be trained during a single iteration of training is determined, from the completion of self-attention to the start of the next transformer block, including: Based on parameter information, for any node in a heterogeneous system, determine the maximum value of data computation time and memory access time of a single transformer block of the model to be trained during a single iteration of training, from the completion of self-attention to the next transformer block. This value is then used as the computation time of a single transformer block of the model to be trained during a single iteration of training.

[0052] This implementation takes into account that, after attention and before the next transformer block, data computation and memory access can occur simultaneously. Therefore, for any node, it is necessary to determine the data computation time and memory access time of a single transformer block from the completion of self-attention to the next transformer block, and then use the maximum of the two as the time consumed by that node in that stage. This application uses T f，post，i In this expression, i represents the i-th node, f represents forward propagation, and post represents the time taken from the attention point to the next transformer block.

[0053] Furthermore, in a specific embodiment of the present invention, based on parameter information, for any node in a heterogeneous system, the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained during a single iteration of training is determined, from the completion of self-attention to the next transformer block. This value is then used as the computation time of a single transformer block of the model to be trained during a single iteration of training, from the completion of self-attention to the next transformer block. Specifically, this can include: Based on parameter information, for any node in a heterogeneous system, according to the preset fifth calculation formula, the computation time of a single transformer block of the model to be trained during a single iteration of training is determined from the completion of self-attention to the start of the next transformer block. The fifth calculation formula is: .

[0054] Where i is a positive integer and represents the i-th node in the heterogeneous system, f represents forward propagation; T f，post，i This represents the computation time for a single transformer block of the model to be trained, from the completion of self-attention to the start of the next transformer block, for the i-th node; max indicates taking the maximum value; T post，comp，i This represents the data computation time for a single transformer block of the model to be trained, from the completion of self-attention to the start of the next transformer block, for the i-th node; T post，mem，i This represents the memory access time for a single transformer block of the model to be trained from the completion of self-attention to the start of the next transformer block for the i-th node. The parameter represents the computation time correction parameter, μ represents the memory access time correction parameter, l represents the input sequence length, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and C represents the computation time correction parameter. i M represents the computing power of the i-th node. bw，i This represents the memory bandwidth of the i-th node, d ff This represents the intermediate dimension of the feedforward neural network of the model to be trained.

[0055] In this implementation method, it is according to The calculation method for T f，post，i The calculation, where the first term in the formula is the time consumed by the data processing in this stage, i.e., T. post，comp，i In this implementation, it is considered that the data operation at this stage mainly includes the calculation of the FFN part and the calculation of the normalization layer, wherein the computational amount of the FFN part is (l / D)×h×d. ff The computational cost of the normalization layer is (l / D)×h. Therefore, the sum of these two computational costs is divided by the computational capacity C of the i-th node. i This allows us to estimate the data processing time for that stage, denoted as T. post，comp，i The second term in the formula represents the memory access time for this stage, mainly including reading the corresponding model weights and activation data, with data volumes of (l / D)×d respectively. ff and h×d ff Therefore, the sum of these two computational costs is divided by the memory bandwidth M of the i-th node. bw，i This gives the memory access time for that stage, denoted as T. post，mem，i .

[0056] Since the data processing and memory access in this stage almost overlap, we can take T. post，comp，i and T post，mem，i The maximum value of this value can reflect the time bottleneck of this stage, and can be used as the time T between the attention and the next transformer block.f，post，i .

[0057] Step S103: Based on the forward propagation time information of the node, determine the forward propagation time of the node in a single iteration of training.

[0058] Based on parameter information, for any node in a heterogeneous system, determine the computation time of a single transformer block of the model to be trained before self-attention execution, the communication time before self-attention execution, the self-attention execution time, and the computation time from the completion of self-attention execution to the next transformer block during a single iteration of training. For ease of description, these four stages are called the forward propagation time information of the node, and as described above, they are denoted as T. f，pre，i T f，comm，i T f，attn，i And T f，post，i .

[0059] Then, it can be based on T f，pre，i T f，comm，i T f，attn，i And T f，post，i This determines the forward propagation time of the node during a single training iteration.

[0060] In one specific embodiment of the present invention, step S103 may specifically include: Based on the forward propagation time information of nodes, and following the principle that for any transformer block, a node only begins self-attention execution after all nodes in the heterogeneous system have completed cross-node data interaction, the forward propagation time of a node in a single iteration of training is determined.

[0061] This implementation takes into account that determining the forward propagation time of a node based on its forward propagation time information requires considering that for each transformer block, in addition to computation, pre-attention communication is also necessary before its own attention. Furthermore, during this communication phase, each transformer block needs to synchronously collect data from other parallel computing power before calculating the attention, thus requiring waiting for the slowest node to complete. Therefore, this implementation determines the forward propagation time of a node based on the principle that the node only begins self-attention execution after all nodes in the heterogeneous system have completed cross-node data interaction, enabling the proposed solution to accurately determine the forward propagation time of a node.

[0062] For example, in one specific embodiment of the present invention, based on the forward propagation time information of a node, and following the principle that during forward propagation, for any transformer block, the node only begins self-attention execution after all cross-node data interactions between the node and the other nodes in the heterogeneous system have been completed, the forward propagation time of a node in a single iteration of training can be determined, specifically including: Based on the forward propagation time information, the forward propagation time in a single training iteration is determined according to a preset first calculation formula. The first calculation formula is: .

[0063] Where i is a positive integer and represents the i-th node in the heterogeneous system, f represents forward propagation; T f，pre，i T represents the computation time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node; f，comm，i This represents the communication time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node; T f，attn，i T represents the self-attention execution time of a single transformer block of the model to be trained for the i-th node; f，post，i This represents the computation time for a single transformer block of the model to be trained, from the completion of self-attention to the start of the next transformer block, for the i-th node; max indicates taking the maximum value; L represents the number of layers in the transformer blocks of the model to be trained; T f，i This represents the forward propagation time of the i-th node during a single iteration of training.

[0064] In this implementation, the forward propagation time of a node is determined based on the principle that the node only begins self-attention execution after all cross-node data interactions with other nodes in the heterogeneous system have been completed. For ease of understanding, let's take a 3-layer transformer block as an example, where L is set to 3.

[0065] First is the T of the first transformer block. f，pre，i Stages and T f，comm，i This stage specifically represents the computation and communication time before the attention function of the first transformer block is executed, T. f，comm，i The phase needs to wait for the slowest node to complete, i.e., T. f，i In the calculation formula, max(T) f，pre，i +T f，comm，i This item represents the T of the first transformer block. f，pre，i Stages and T f，comm，i The time taken for each stage.

[0066] Next is the T of the first transformer block. f，attn，i Stages and T f，post，i Phase 1, followed by the T phase of the second transformer block. f，pre，i Stages and T f，comm，i Phase 1. T reaches the second transformer block. f，comm，i In this phase, we also need to wait for the slowest node to complete, therefore in T... f，i In the calculation formula, max(T) f，attn，i +T f，post，i +T f，pre，i +T f，comm，i This item has a maximum setting.

[0067] Then comes the T of the second transformer block. f，attn，i Stages and T f，post，i Phase 1, followed by the T phase of the third transformer block. f，pre，i Stages and T f，comm，i Phase 1. T reaches the 3rd transformer block. f，comm，i In this phase, we also need to wait for the slowest node to complete, therefore in T... f，i In the calculation formula, max(T) f，attn，i +T f，post，i +T f，pre，i +T f，comm，i This item has a maximum value. In this example, L=3, therefore... Specifically .

[0068] Finally, there's the T of the third transformer block. f，attn，i Stages and T f，post，i Stage, corresponding to T f，i The last two terms in the calculation formula, namely T f，attn，i +T f，post，i .

[0069] As can be seen from the above, in each T f，comm，i In each stage, the max function needs to be used to ensure that the slowest node can be waited for to complete. That is, max will include synchronous communication and all calculations before communication. Therefore, it can be seen that this implementation effectively satisfies the principle that "the node will start self-attention execution only after all nodes in the heterogeneous system have completed cross-node data interaction", which makes it possible to accurately determine the forward propagation time of the node.

[0070] Step S104: Based on the parameter information, determine the backpropagation time of any node in the heterogeneous system during a single iteration of training.

[0071] Based on parameter information, the backpropagation time can be determined for any node in a heterogeneous system. Typically, the computation during the backpropagation process and all-reduce communication can be considered to determine the backpropagation time.

[0072] In one specific embodiment of the present invention, step S104 may specifically include: Based on parameter information, for any node in a heterogeneous system, the computation time and communication time of backpropagation of the model to be trained during a single iteration of training are determined, and the maximum value of the computation time and communication time of backpropagation is taken as the backpropagation time of the node during a single iteration of training.

[0073] This implementation takes into account that, for backpropagation, in order to simplify the calculation, it is not necessary to perform fine-grained modeling with attention as the boundary as in the forward propagation above, and the calculation and communication can be regarded as being performed simultaneously. The backpropagation time obtained is usually not much of an error. Therefore, in this implementation, the maximum value of the calculation time and communication time of backpropagation is taken as the backpropagation time of the node in a single iteration training process.

[0074] For example, in one specific embodiment of the present invention, based on parameter information, determining the computation time and communication time of backpropagation of the model to be trained during a single iteration of training for any node in a heterogeneous system can specifically include: Based on parameter information, for any node in a heterogeneous system, the computation time for backpropagation of the model to be trained during a single iteration of training is determined according to a pre-defined sixth calculation formula. The sixth calculation formula is: .

[0075] Based on parameter information, for any node in the heterogeneous system, the communication time for backpropagation of the model to be trained during a single iteration of training is determined according to a preset seventh calculation formula. The seventh calculation formula is as follows: .

[0076] Where i is a positive integer and represents the i-th node in the heterogeneous system, b represents backpropagation; T b，comp，i This represents the computation time of backpropagation of the model to be trained during a single iteration of training for the i-th node; max indicates taking the maximum value; T b，comm，iThis represents the communication time of backpropagation of the model to be trained during a single iteration of training for the i-th node; N represents the total number of nodes; B represents the average mutual communication bandwidth between nodes; P represents the total number of parameters of the model to be trained; and β represents the communication time correction parameter.

[0077] T backward，comp，i This represents the time consumed by the backpropagation data computation of the model to be trained during a single iteration of training for the i-th node. , This represents the computation time correction parameter, l represents the input sequence length, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and C represents the sequence length. i This represents the computing power of the i-th node, d ff This represents the intermediate dimension of the feedforward neural network of the model to be trained.

[0078] T backward，mem，i This represents the memory access time of the backpropagation of the model to be trained during a single iteration of training for the i-th node. μ represents the memory access time correction parameter, M bw，i This represents the memory bandwidth of the i-th node.

[0079] In this implementation, the calculation time for backpropagation is T. b，comp，i Specifically, this includes backpropagation with a total of L layers. For each layer of backpropagation, the data retrieval and computation time is T. backwaed，comp，i and memory access time T backwaed，mem，i The maximum value of T. backwaed，comp，i The first term in the formula represents the computational complexity of the gradient for the attention mechanism, the second term represents the computational complexity of the FFN gradient, and the third term represents the computational complexity of layer normalization (mean / variance calculation, standardization) and residual connection gradients. Based on practical experience, backpropagation is approximately twice as computationally intensive as forward propagation, hence the multiplication by a factor of 2. T backwaed，mem，i The first to third terms in the calculation formula represent the global KV gradient, local activation, and FFN weights loaded during memory access, respectively.

[0080] The communication time for backpropagation is T. b，comm，i It is based on the transmission principle of allreduce, and considering that the amount of data transmitted by each node is (P×2×(N−1)) / N, the approximate communication time is estimated accordingly.

[0081] Step S105: The maximum forward propagation time of each node is added together with the maximum backward propagation time of each node, and the result is used as the time consumption of a single iteration of the training process of the model to be trained.

[0082] After obtaining the forward propagation time for each node, we need to take the maximum value. Similarly, after obtaining the backward propagation time for each node, we also need to take the maximum value. Then, we add the two together, and the result is the time taken for one step of the model to be trained.

[0083] Furthermore, it should be noted that in the process of parallel training of a sequence in one step, a long sequence is usually divided into several subsequences, and then distributed training is carried out by various nodes. The above calculations of related times are all based on this situation. However, if the input sequence includes K long sequences in one step, then for the T described above... f，pre，i T f，comm，i T f，attn，i T f，post，i , and T b，comp，i Simply multiply the calculation results by K. The number of long sequences does not affect the backpropagation communication time T. b，comm，i Therefore, T b，comm，i No need to multiply by K.

[0084] The above text in this application mentions T. f，pre，i T f，comm，i T f，attn，i T f，post，i T b，comp，i And T b，comm，i During the calculation, corresponding correction parameters are set in the calculation formula, specifically including the calculation time correction parameter. The memory access time correction parameter μ and the communication time correction parameter β, by default, can all be set to 1. The purpose of setting these correction parameters is to account for even if the solution in this application is targeted at T... f，pre，i T f，comm，i T f，attn，i T f，post，i T b，comp，i And T b，comm，i The calculation can usually reflect the actual time spent in the corresponding stage quite accurately. However, in practical applications, certain errors are unavoidable. The setting of correction parameters allows the actual time spent to be compared with the calculated value, and the corresponding correction parameters can be adjusted to further improve the accuracy of the solution in this application.

[0085] With T f，pre，i Taking calibration as an example, in one specific embodiment of the present invention, it may further include: During the training process of the model to be trained, for the computation time of a single transformer block of the model before the execution of self-attention, the actual time consumption of each node is detected, and the actual time consumption of each node is compared with the corresponding T. f，pre，i Compare the results and obtain the corresponding difference coefficients; The average value of each difference coefficient is used as the updated computation time correction parameter. .

[0086] In this implementation, due to T f，pre，i This represents the computational phase before self-attention execution. Therefore, the actual time consumed by each node in this phase is directly detected, and each detection result can be compared with the corresponding node's T. f，pre，i The differences are compared to obtain the corresponding coefficients, and then the average of these coefficients is taken. For example, the actual time taken by a node in this stage is T of that node. f，pre，i If the difference is 1.1 times, then the coefficient of variation is 1.1. For example, if the average coefficient of variation is 1.2, then 1.2 can be used as the correction parameter for the updated computation time. .

[0087] Furthermore, it is understandable that this is based on T. f，pre，i For example, for T f，pre，i The computation time correction parameter in In other implementations, T can be corrected. f，comm，i T f，attn，i T f，post，i T b，comp，i And T b，comm，i The computation time correction parameters involved The memory access time correction parameter μ and the communication time correction parameter β are both corrected.

[0088] Furthermore, it should be noted that, theoretically speaking, for T... f，pre，i T f，comm，i T f，attn，i T f，post，i T b，comp，i And T b，comm，i The calculations can use different correction parameters for each stage, but this application considers that although they correspond to different stages, the correction parameters used will be roughly the same in terms of computation time. Therefore, in order to simplify the correction process, T... f，pre，i T f，comm，i T f，attn，i T f，post，i T b，comp，i And T b，comm，i In the text, computational formulas related to operations use a unified computation time correction parameter. Similarly, for calculations related to memory access, a unified memory access time correction parameter μ can be used, and for calculations related to communication, a unified communication time correction parameter β can be used.

[0089] To effectively predict the time consumption of parallel training sequences in heterogeneous systems using the technical solution provided in this invention, the first step is to obtain parameter information reflecting the training status of the heterogeneous system during parallel training of the model to be trained. After obtaining the parameter information, for any node in the heterogeneous system, the backpropagation time and forward propagation time during a single iteration of training can be determined based on the parameter information. Since backpropagation requires waiting for each node to finish its forward propagation, the maximum forward propagation time of each node needs to be superimposed with the maximum backpropagation time of each node. The result is then used as the time consumption of a single iteration of training for the model to be trained. Furthermore, this application takes into account that for parallel training of heterogeneous systems, during a single iteration of training, data interaction between nodes is involved in the execution of each transformer block during forward propagation. Therefore, based on parameter information, it is necessary to determine the forward propagation time information of any node in the heterogeneous system for each node during a single iteration of training. This includes the computation time of a single transformer block of the model to be trained before self-attention execution, the communication time before self-attention execution, the self-attention execution time, and the computation time from the completion of self-attention execution to the next transformer block. These four stages are called the forward propagation time information of the node, so that the forward propagation time of the node can be accurately and effectively determined through the forward propagation time information of the node.

[0090] In summary, the proposed solution can effectively predict the time consumption of a single iteration of the training process of the model to be trained, that is, it can realize the time consumption prediction of parallel training of heterogeneous systems.

[0091] Furthermore, it's worth noting that predicting the time consumption of sequence-parallel training provides a quantitative basis for its planning, optimization, and management, and is applicable in various scenarios. For example, based on the predicted time consumption, staff can plan training resources; for instance, if the time consumption is too long, the number of GPUs can be increased, the structure of the model to be trained can be simplified, the distributed training configuration can be adjusted, and the communication strategies between nodes can be adjusted so that the time consumption meets the staff's requirements. In addition, during subsequent training execution, the actual time consumption can be compared with the predicted time consumption. If the difference is too large, it may be due to reasons such as partial node failure or incorrect parameter configuration, allowing staff to promptly identify and troubleshoot the problem.

[0092] Corresponding to the above method embodiments, the present invention also provides an electronic device, a computer-readable storage medium, and a computer program product, which can be referred to in conjunction with the above.

[0093] See Figure 3 As shown, the device may include: Memory 301 is used to store computer programs; Processor 302 is configured to execute a computer program to implement the steps of the time-consuming prediction method for parallel training of heterogeneous system sequences as described in any of the above embodiments.

[0094] The computer program product includes a computer program / instruction that, when executed by a processor, implements the steps of the time-consuming prediction method for parallel training of heterogeneous system sequences as described in any of the above embodiments.

[0095] See also Figure 4 The computer-readable storage medium 40 stores a computer program 41, which, when executed by a processor, implements the steps of the time-consuming prediction method for parallel training of heterogeneous system sequences as described in any of the above embodiments. The computer-readable storage medium 40 referred to herein includes RAM (Random Access Memory), main memory, ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), registers, hard disks, removable disks, or any other form of storage medium known in the art.

[0096] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] The above provides a detailed description of the time-consuming prediction method, device, and program product for parallel training of heterogeneous system sequences provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for predicting the time consumption of parallel training sequences in a heterogeneous system, characterized in that, include: Obtain parameter information that reflects the training status of a heterogeneous system during sequence-parallel training of a model to be trained; Based on the parameter information, for any node in the heterogeneous system, the computation time of a single transformer block of the model to be trained before self-attention execution, the communication time before self-attention execution, the self-attention execution time, and the computation time from the completion of self-attention execution to the next transformer block are determined as the forward propagation time information of the node during a single iteration of training. Based on the forward propagation time information of the node, the forward propagation time of the node in a single iteration of training is determined; Based on the parameter information, for any node in the heterogeneous system, the backpropagation time of the node in a single iteration of training is determined; The maximum forward propagation time of each node is added together with the maximum backward propagation time of each node, and the result is used as the time consumption of a single iteration of the training process of the model to be trained.

2. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 1, characterized in that, Based on the forward propagation time information of the node, the forward propagation time of the node in a single iteration of training is determined, including: Based on the forward propagation time information of the node, and following the principle that during forward propagation, for any transformer block, the node only starts self-attention execution after all nodes in the heterogeneous system have completed cross-node data interaction, the forward propagation time of the node in a single iteration training process is determined.

3. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 2, characterized in that, Based on the forward propagation time information of the node, and following the principle that during forward propagation, for any transformer block, the node only begins self-attention execution after all cross-node data interactions with other nodes in the heterogeneous system have been completed, the forward propagation time of the node in a single iteration of training is determined, including: Based on the forward propagation time information, the forward propagation time in a single iteration training process is determined according to the preset first calculation formula; The first calculation formula is: ； i is a positive integer representing the i-th node in the heterogeneous system, and f represents forward propagation; T f，pre，i T represents the computation time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node; f，comm，i T represents the communication time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node; f，attn，i T represents the self-attention execution time of a single transformer block of the model to be trained for the i-th node; f，post，i This represents the computation time for a single transformer block of the model to be trained, from the completion of self-attention to the start of the next transformer block, for the i-th node; max indicates taking the maximum value; L represents the number of layers in the transformer blocks of the model to be trained; T f，i This represents the forward propagation time of the i-th node during a single iteration of training.

4. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 1, characterized in that, Based on the parameter information, for any node in the heterogeneous system, determine the computation time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training, including: Based on the parameter information, for any node in the heterogeneous system, the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained before self-attention execution is determined during a single iteration of training, and is used as the computation time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training.

5. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 4, characterized in that, Based on the parameter information, for any node in the heterogeneous system, the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training is determined, and this value is used as the computation time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training, including: Based on the parameter information, for any node in the heterogeneous system, the computation time of a single transformer block of the model to be trained before self-attention is determined according to the preset second calculation formula during a single iteration of training. The second calculation formula is as follows: ; i is a positive integer representing the i-th node in the heterogeneous system, and f represents forward propagation; T f，pre，i This represents the computation time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node; max indicates taking the maximum value; T pre，comp，i This represents the data computation time for a single transformer block of the model to be trained before self-attention is executed, for the i-th node; T pre，mem，i This represents the memory access time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node; The parameter represents the computation time correction parameter, μ represents the memory access time correction parameter, l represents the input sequence length, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and C represents the computation time correction parameter. i M represents the computing power of the i-th node. bw，i This represents the memory bandwidth of the i-th node.

6. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 5, characterized in that, Also includes: During the training process of the model to be trained, for the computation time of a single transformer block of the model to be trained before the execution of self-attention, the actual time consumption of each node is detected, and the actual time consumption of each node is compared with the corresponding T. f，pre，i Compare the results and obtain the corresponding difference coefficients; The average value of each difference coefficient is used as the updated computation time correction parameter. .

7. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 1, characterized in that, Based on the parameter information, for any node in the heterogeneous system, determine the communication time of a single transformer block of the model to be trained before self-attention execution during a single iteration of training, including: Based on the parameter information, for any node in the heterogeneous system, the communication time of a single transformer block of the model to be trained before the execution of self-attention is determined according to the preset third calculation formula. The third calculation formula is as follows: ; i is a positive integer representing the i-th node in the heterogeneous system, and f represents forward propagation; T f，comm，i This represents the communication time of a single transformer block of the model to be trained before self-attention is executed, for the i-th node; β represents the communication time correction parameter, l represents the input sequence length, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and B represents the average mutual communication bandwidth of each node.

8. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 1, characterized in that, Based on the parameter information, for any node in the heterogeneous system, determine the self-attention execution time of a single transformer block of the model to be trained during a single iteration of training, including: Based on the parameter information, for any node in the heterogeneous system, the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained during self-attention execution in a single iteration of training is determined, and this value is used as the self-attention execution time of a single transformer block of the model to be trained in a single iteration of training.

9. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 8, characterized in that, Based on the parameter information, for any node in the heterogeneous system, the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained during self-attention execution in a single iteration of training is determined, and used as the self-attention execution time of a single transformer block of the model to be trained in a single iteration of training, including: Based on the parameter information, for any node in the heterogeneous system, the self-attention execution time of a single transformer block of the model to be trained is determined according to the preset fourth calculation formula. The fourth calculation formula is as follows: ； i is a positive integer representing the i-th node in the heterogeneous system, and f represents forward propagation; T f，attn，i This represents the self-attention execution time of a single transformer block of the model to be trained for the i-th node; max indicates taking the maximum value; T attn，comp，i This represents the data computation time for a single transformer block of the model to be trained during self-attention execution, for the i-th node; T attn，mem，i This represents the memory access time of a single transformer block of the model to be trained during self-attention execution for the i-th node; The parameter represents the computation time correction parameter, μ represents the memory access time correction parameter, l represents the input sequence length, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and C represents the computation time correction parameter. i M represents the computing power of the i-th node. bw，i This represents the memory bandwidth of the i-th node.

10. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 1, characterized in that, Based on the parameter information, for any node in the heterogeneous system, the computation time of a single transformer block of the model to be trained during a single iteration of training is determined, from the completion of self-attention to the start of the next transformer block, including: Based on the parameter information, for any node in the heterogeneous system, the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained from the completion of self-attention to the next transformer block is determined during a single iteration of training. This value is then used as the computation time of the single transformer block of the model to be trained during a single iteration of training.

11. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 10, characterized in that, Based on the parameter information, for any node in the heterogeneous system, the maximum value of the data computation time and memory access time of a single transformer block of the model to be trained during a single iteration of training is determined, and this value is used as the computation time of the single transformer block of the model to be trained during a single iteration of training, including: Based on the parameter information, for any node in the heterogeneous system, the computation time of a single transformer block of the model to be trained during a single iteration of training is determined according to the preset fifth calculation formula, from the completion of self-attention to the next transformer block. The fifth calculation formula is as follows: ； i is a positive integer representing the i-th node in the heterogeneous system, and f represents forward propagation; T f，post，i This represents the computation time for a single transformer block of the model to be trained, from the completion of self-attention to the start of the next transformer block, for the i-th node; max indicates taking the maximum value; T post，comp，i This represents the data computation time for a single transformer block of the model to be trained, from the completion of self-attention to the start of the next transformer block, for the i-th node; T post，mem，i This represents the memory access time for a single transformer block of the model to be trained, from the completion of self-attention to the start of the next transformer block, for the i-th node. The parameter represents the computation time correction parameter, μ represents the memory access time correction parameter, l represents the input sequence length, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and C represents the computation time correction parameter. i M represents the computing power of the i-th node. bw，i This represents the memory bandwidth of the i-th node, d ff This represents the intermediate dimension of the feedforward neural network of the model to be trained.

12. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to any one of claims 1 to 11, characterized in that, Based on the parameter information, for any node in the heterogeneous system, the backpropagation time of the node in a single iteration of training is determined, including: Based on the parameter information, for any node in the heterogeneous system, the computation time and communication time of the backpropagation of the model to be trained during a single iteration of training are determined, and the maximum value of the computation time and communication time of the backpropagation is taken as the backpropagation time of the node during a single iteration of training.

13. The method for predicting the time consumption of parallel training sequences in heterogeneous systems according to claim 12, characterized in that, Based on the parameter information, for any node in the heterogeneous system, the computation time and communication time of backpropagation of the model to be trained during a single iteration of training are determined, including: Based on the parameter information, for any node in the heterogeneous system, the calculation time of backpropagation of the model to be trained during a single iteration of training is determined according to the preset sixth calculation formula. The sixth calculation formula is as follows: ; Based on the parameter information, for any node in the heterogeneous system, the communication time of the backpropagation of the model to be trained during a single iteration of training is determined according to the preset seventh calculation formula. The seventh calculation formula is as follows: ; i is a positive integer representing the i-th node in the heterogeneous system, and b represents backpropagation; T b，comp，i This represents the computation time of backpropagation of the model to be trained during a single iteration of training for the i-th node; max indicates taking the maximum value; T b，comm，i This represents the communication time of backpropagation of the model to be trained during a single iteration of training for the i-th node; N represents the total number of nodes; B represents the average mutual communication bandwidth between nodes; P represents the total number of parameters of the model to be trained; β represents the communication time correction parameter; and L represents the number of layers in the transformer block of the model to be trained. T backward，comp，i This indicates the time consumed by the backpropagation data computation of the model to be trained during a single iteration of training for the i-th node. , This represents the computation time correction parameter, l represents the input sequence length, D represents the sequence parallelism, h represents the hidden dimension of the model to be trained, and C represents the sequence parallelism. i This represents the computing power of the i-th node, d ff This represents the intermediate dimension of the feedforward neural network of the model to be trained; T backward，mem，i This represents the memory access time of the backpropagation of the model to be trained during a single iteration of training for the i-th node. μ represents the memory access time correction parameter, M bw，i This represents the memory bandwidth of the i-th node.

14. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the time-consuming prediction method for parallel training of heterogeneous system sequences as described in any one of claims 1 to 13.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the time-consuming prediction method for parallel training of heterogeneous system sequences as described in any one of claims 1 to 13.

Images