Method, device, and system for heterogeneous cluster parallel training of large models

Abstract

The application relates to the technical field of artificial intelligence, and provides a method, device and system for heterogeneous cluster parallel training of a large model. The method comprises the following steps: acquiring model structure parameter information representing a large model to be trained and device parameter information representing a heterogeneous cluster for training the large model; determining a parameter search space according to the parameter information, and searching for a target parameter combination in the parameter search space according to a preset search rule; and performing parallel training on the large model by the heterogeneous cluster based on the target parameter combination, wherein the target parameter combination is a parameter combination in the parameter search space, which makes the large model consume the least time in one round of training. The application provides an automatic search scheme of a target parameter combination to replace a parameter search scheme in the related art, so that the time consumption of the target parameter combination can be reduced, and the model development cycle can be shortened.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a method, apparatus, and system for parallel training of heterogeneous clusters of large models. Background Technology

[0002] Large models refer to machine learning models with large-scale parameters and complex computational structures, and they have wide applications in many fields such as natural language processing, image recognition, and speech recognition. Summary of the Invention

[0003] When training large models, it is possible to utilize heterogeneous clusters for parallel training. Heterogeneous clusters typically include multiple types of processing units, such as different types of Graphics Processing Units (GPUs). Parallel training typically includes data parallelism, pipeline parallelism, and tensor parallelism.

[0004] The inventors of this application have noted that in related technologies, to achieve the highest training efficiency on heterogeneous clusters, developers often need to manually select multiple parameter combinations to train a large model. They also need to wait for the large model to complete training under all parameter combinations to obtain and record the training efficiency data for each combination, thereby selecting the parameter combination with the highest training efficiency. This parameter search process is extremely cumbersome and time-consuming. Furthermore, when the cluster topology or machine model changes, it is often necessary to manually search again for the optimal parameter combination for model training, resulting in a long model development cycle. Especially in heterogeneous cluster environments, the hardware performance and software development completeness of different processing units may vary, making the search for parameter combinations even more difficult.

[0005] Therefore, the purpose of this application is to provide a method, apparatus, system, computer-readable storage medium, and computer program product for parallel training of heterogeneous clusters of large models, in order to solve the technical problems in the related art.

[0006] In a first aspect, this application provides a method for parallel training of a heterogeneous cluster of large models, wherein the heterogeneous cluster includes at least multiple processing units of various types, and the parallelism includes data parallelism, pipeline parallelism, and tensor parallelism, wherein at least some pipeline stages of the pipeline parallelism are executed by different types of processing units, and the method includes:

[0007] Obtain first input information and second input information, wherein the first input information represents the model structure parameters of the large model to be trained, and the second input information represents the device parameters of the heterogeneous cluster for training the large model;

[0008] The parameter search space is determined based on the first input information and the second input information;

[0009] According to pre-set search rules, search for target parameter combinations in the parameter search space; and

[0010] The heterogeneous cluster performs parallel training on the large model based on the target parameter combination.

[0011] Wherein, the target parameter combination is the parameter combination that minimizes the training cost among multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform one round of training on the large model based on the corresponding parameter combination.

[0012] The total time is the sum of the time for one pipelined parallel training based on the corresponding parameter combination, the time for gradient data set communication, and the time for the optimizer of the large model to update the model parameters of the large model.

[0013] Secondly, this application also provides an apparatus for parallel training of a heterogeneous cluster of large models, wherein the heterogeneous cluster includes at least multiple processing units of various types, and the parallelism includes data parallelism, pipeline parallelism, and tensor parallelism, wherein at least some pipeline stages of the pipeline parallelism are executed by different types of processing units, and the apparatus includes:

[0014] Processor; and

[0015] A memory storing a computer program executable by the processor, which, when executed by the processor, causes the processor to perform the following operations:

[0016] Obtain first input information and second input information, wherein the first input information represents the model structure parameters of the large model to be trained, and the second input information represents the device parameters of the heterogeneous cluster for training the large model;

[0017] The parameter search space is determined based on the first input information and the second input information;

[0018] According to pre-set search rules, search for target parameter combinations in the parameter search space; and

[0019] The target parameter combination is sent to the heterogeneous cluster, enabling the heterogeneous cluster to perform parallel training on the large model based on the target parameter combination.

[0020] Wherein, the target parameter combination is the parameter combination that minimizes the training cost among multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform one round of training on the large model based on the corresponding parameter combination.

[0021] The total time is the sum of the time for one pipelined parallel training based on the corresponding parameter combination, the time for gradient data set communication, and the time for the optimizer of the large model to update the model parameters of the large model.

[0022] Thirdly, this application provides a system for parallel training of large models, comprising:

[0023] A heterogeneous cluster, comprising at least multiple types of processing units, wherein the parallelism includes data parallelism, pipelined parallelism, and tensor parallelism, and at least some pipeline stages of the pipelined parallelism are executed by different types of processing units; and

[0024] A device for parallel training of heterogeneous clusters of large models, the device comprising:

[0025] Processor; and

[0026] A memory storing a computer program executable by the processor, which, when executed by the processor, causes the processor to perform the following operations:

[0027] Obtain first input information and second input information, wherein the first input information represents the model structure parameters of the large model to be trained, and the second input information represents the device parameters of the heterogeneous cluster for training the large model;

[0028] The parameter search space is determined based on the first input information and the second input information;

[0029] According to pre-set search rules, search for target parameter combinations in the parameter search space; and

[0030] The target parameter combination is sent to the heterogeneous cluster, enabling the heterogeneous cluster to perform parallel training on the large model based on the target parameter combination.

[0031] Wherein, the target parameter combination is the parameter combination that minimizes the training cost among multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform one round of training on the large model based on the corresponding parameter combination.

[0032] The total time is the sum of the time for one pipelined parallel training based on the corresponding parameter combination, the time for gradient data set communication, and the time for the optimizer of the large model to update the model parameters of the large model.

[0033] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described according to embodiments of this application.

[0034] Fifthly, this application provides a computer program product including computer instructions that, when executed by a processor, implement the steps of the method described according to embodiments of this application.

[0035] In heterogeneous training scenarios, this application first determines the parameter search space based on the user-input model structure parameters and device parameters. Then, based on preset search rules, it uses the minimum total time required for the heterogeneous cluster to train a large model in one round based on the corresponding parameter combination as the search objective. Within the determined parameter search space, it automatically searches for the target parameter combination, i.e., it automatically searches for the optimal parameter combination. This replaces the method of searching for the optimal parameter combination in related technologies, effectively reducing the time spent obtaining the optimal parameter combination, improving the accuracy of the obtained optimal parameter combination, and effectively shortening the model development cycle. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a method for parallel training of large models on heterogeneous clusters, as provided in an embodiment of this application.

[0037] Figure 2 This is a timing diagram of a pipelined parallel training method provided in an embodiment of this application;

[0038] Figure 3 This is a schematic diagram of the structure of a system for parallel training of heterogeneous clusters of large models provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] This application provides a method for heterogeneous cluster parallel training of large models. The main steps of this method are as follows: Figure 1As shown. The heterogeneous cluster according to embodiments of this application includes at least multiple processing units of multiple types, such as multiple models of graphics processing units (GPUs) from different manufacturers. "Parallelism" in parallel training includes data parallelism, pipeline parallelism, and tensor parallelism. Data parallelism refers to dividing the training data into several groups, assigning each data group to a group of processing units, and having each group of processing units run the same processing program to process the assigned data group. Pipeline parallelism refers to dividing one or more adjacent layers of the model into a pipeline stage, thereby dividing the model into multiple pipeline stages. In this application, at least some pipeline stages of pipeline parallelism are executed by different types of processing units. For example, H pipeline stages out of p pipeline stages (p≥H, H≥2) are assigned to H types of GPUs, each type of GPU including, for example, m GPUs. Of course, the number of each type of GPU executing each pipeline stage can also vary according to actual needs. Tensor parallelism refers to dividing the tensor parameters of the model into multiple groups according to a specific dimension, with each group of tensor parameters assigned to a different processing unit.

[0041] like Figure 1 As shown, in step 101, the first input information and the second input information are obtained.

[0042] The first input information represents the model structure parameters of the large model to be trained, and the second input information represents the device parameters of the heterogeneous cluster used to train the large model.

[0043] Without loss of generality, model structure parameters may include the total number of operators in the large model (i.e., the total number of layers in the large model), the length of the training data sequence, and the number of hidden states in the model. Device parameters may include the number of categories corresponding to multiple processing units, the number of processing units for each category, the memory capacity of each type of processing unit, and the communication bandwidth. The communication bandwidth includes the bandwidth for communication within processing unit nodes, the bandwidth for communication between different processing unit nodes of the same category, and the bandwidth for communication between different processing unit nodes of different categories.

[0044] Next, proceed to step 102.

[0045] In step 102, the parameter search space is determined based on the first input information and the second input information.

[0046] The parameter search space can, for example, represent a set of optional parameter combinations for parallel training of a large model by a heterogeneous cluster.

[0047] In some embodiments, the parameter search space includes parallelism combination schemes, processing unit allocation schemes, pipeline partitioning schemes, and recomputation layer setting schemes.

[0048] Parallelism combination schemes can represent, for example, optional combinations of data parallelism, pipeline parallelism, and tensor parallelism. For instance, when DP represents data parallelism, PP represents pipeline parallelism, and TP represents tensor parallelism, the parallelism combination scheme can be represented as (DP, PP, TP). Processing unit allocation schemes can represent, for example, the optional types of processing units that execute each pipeline stage. For instance, each of p pipeline stages can be executed by any one or more of H types of processing units (e.g., GPUs). Pipeline partitioning schemes can represent, for instance, the number of model layers to be trained in each pipeline stage. For example, the i-th pipeline stage in p pipeline stages trains an ni-layer model. Recomputation layer setting schemes represent the number of model layers recomputed in each pipeline stage. For example, for training n... i In the i-th pipeline stage of the layer model, the number of model layers to be recomputed in this pipeline stage can be set from 0 to n. i .

[0049] Any of the above parallelism combination schemes (DP, PP, TP) satisfies the following condition:

[0050] DP×PP×TPM;

[0051] In this context, DP refers to data parallelism, TP refers to tensor parallelism, PP refers to pipeline parallelism, and M refers to the total number of processing units (e.g., GPUs) included in the heterogeneous cluster.

[0052] The above processing unit allocation scheme satisfies the following condition: the memory usage of the training task allocated to each processing unit (e.g., GPU) in the heterogeneous cluster is less than or equal to the memory capacity of that processing unit.

[0053] The conditions that the above pipeline segmentation scheme should meet can be expressed as:

[0054]

[0055] The total number of pipeline segmentation schemes is:

[0056]

[0057] Where N represents the total number of operators (total number of layers) of the large model, p represents the total number of pipeline stages, and ni represents the number of operators (model layers) to be trained in the i-th pipeline stage. The number of operators to be trained in any pipeline stage should be less than or equal to N.

[0058] The total number of schemes for recalculating the number of layers mentioned above can be expressed as:

[0059]

[0060] The total number of composite schemes formed by combining pipeline segmentation with recalculation of the number of layers is:

[0061]

[0062] Npipe refers to the total number of recomputation schemes in the case of stacked pipeline splitting schemes.

[0063] As can be seen from the above, by combining the above-mentioned schemes, the parameter search space can be reasonably narrowed (so that the parameter combinations in the parameter search space at least meet the above conditions) while adapting to the actual situation of the model structure and the maximum processing capacity of the heterogeneous cluster. This can limit the size of the parameter search space as much as possible while ensuring the accuracy of the final target parameter combination, thereby improving the search efficiency of the target parameter combination.

[0064] Next, proceed to step 103.

[0065] In step 103, the target parameter combination is searched in the parameter search space according to the pre-set search rules.

[0066] The target parameter combination can be the parameter combination that minimizes the training cost among multiple parameter combinations included in the parameter search space. When the parameter search space includes the parallelism combination scheme, processing unit allocation scheme, pipeline partitioning scheme, and recomputation layer setting scheme as described above, the parameter combinations in the parameter search space can include: combinations of data parallelism, pipeline parallelism, and tensor parallelism; the type of processing unit executing each pipeline stage; the number of model layers to be trained in each pipeline stage; and the number of model layers recomputed in each pipeline stage.

[0067] The training cost can be the total time required for a heterogeneous cluster to train a large model in one round based on parameter combinations in the parameter search space. In some embodiments, this total time may be the sum of the time for a pipelined parallel training operation based on parameter combinations in the parameter search space, the time for gradient data set communication, and the time for the large model's optimizer to update the model parameters. In this case, the training cost can be expressed based on the following formula:

[0068] Tcost=[TPP+Tall-reduce+Toptimizer];

[0069] In the above formula, Tcos tTpp represents the training cost, which is the total time required for the heterogeneous cluster to train the large model in one round based on the parameter combinations in the parameter search space. Tpp represents the time for one pipelined parallel training based on the corresponding parameter combinations. Tall_reduce represents the time for gradient data set communication. Toptimizer represents the time for the large model's optimizer to update the model parameters of the large model.

[0070] In heterogeneous training scenarios, this application first determines the parameter search space for the user's required parameter combination based on the model structure parameters and device parameters input by the user. Then, based on pre-set search rules, it takes minimizing the total time required for the heterogeneous cluster to train a large model for one round using the corresponding parameter combination as the search objective. Within the determined parameter search space, it automatically searches for the target parameter combination, i.e., it automatically searches for the optimal parameter combination. This replaces the method of searching for the optimal parameter combination in related technologies, effectively reducing the time spent obtaining the optimal parameter combination, improving the accuracy of the obtained optimal parameter combination, and effectively shortening the model development cycle.

[0071] By comprehensively considering the time required for a single pipelined parallel training operation based on corresponding parameter combinations in the parameter search space, the time for gradient data set communication, and the time for the optimizer to update the model parameters of the large model, the total time required for a heterogeneous cluster to train a large model for a given parameter combination is determined. This total time is then used as the training cost for that parameter combination, providing a more accurate representation of the training cost. Furthermore, searching for the optimal parameter combination (i.e., the target parameter combination) based on the total training time for a single round of model training using the corresponding parameter combination as the training cost ensures that the found optimal parameter combination has high accuracy and reliability.

[0072] It should be noted that large-scale communication between heterogeneous chips in a heterogeneous cluster can lead to a significant decrease in computational efficiency. Therefore, in this application, both data parallelism and tensor parallelism are set as homogeneous strategies, meaning that the types of processing units involved are consistent for both data parallelism and tensor parallelism. Heterogeneity is only set at the pipeline parallelism dimension, that is, different types of processing units are used to handle different pipeline stages. Therefore, the training time is the same in both data parallelism and tensor parallelism dimensions. Thus, when designing the training cost, only pipeline parallelism is considered, and the time for one pipeline parallel training is included in the training cost, thereby simplifying the calculation formula of the training cost and improving the computational efficiency of the training cost.

[0073] It should also be understood that T optimizer Based on the actual runtime of each processing unit in the heterogeneous cluster, performance data of the processing units can be collected in advance through pre-testing, without having to collect performance data across the entire heterogeneous cluster during each training session.

[0074] In some embodiments, a tree structure can be constructed based on the parameter search space determined in step 102, where each node in the tree structure represents a parameter combination in the parameter search space. A depth-first search or breadth-first search algorithm is used, with the goal of minimizing the total time required for the heterogeneous cluster to train the large model in one round based on the corresponding parameter combination, to search the tree structure and obtain the target parameter combination.

[0075] In some embodiments, the AllReduce algorithm can be used for gradient data set communication. AllReduce is a commonly used communication algorithm in distributed training of large models, used to synchronize parameters between processing unit nodes. During the training of a large model, model parameters need to be synchronized across nodes to ensure training consistency. AllReduce achieves parameter synchronization by summing the parameters of all nodes and then broadcasting the result to all nodes, ensuring that each node receives the updated parameter values. Compared to other communication algorithms, AllReduce better balances the communication overhead between nodes, improving training efficiency.

[0076] When using the AllReduce algorithm for gradient data set communication, for example, when using T... all_reduce When representing the time of gradient data set communication, T all_reduce For example, it can be determined based on the following formula:

[0077]

[0078] Where, `data` represents the amount of data that needs to be transmitted for gradient data set communication between parallel data groups, `bandwidth` represents the data transmission bandwidth for gradient data set communication, and `layers` represents the data transmission bandwidth. i This indicates the number of model layers to be executed in the i-th pipeline stage, micro_batch_size indicates the size of a set of input data for each pipeline stage, and gradient indicates the gradient.

[0079] In this method of calculating communication time, the time for gradient set data communication is determined by the ratio of the amount of data to be transmitted for gradient set data communication between parallel data groups to the data transmission bandwidth. Furthermore, the amount of data is defined as the number of model layers to be executed in the i-th pipeline stage, the size of a set of input data for each pipeline stage, and the gradient. This method makes the calculation of gradient data communication time more accurate and reliable.

[0080] Note that in a heterogeneous training environment, there may be multiple different bandwidths for different types of processing units in a heterogeneous cluster. The bandwidth should be understood as the minimum bandwidth among the aforementioned multiple different bandwidths.

[0081] Additionally, the aforementioned set of input data can be a micro-batch. Micro-batch data is obtained by further splitting the mini-batch data obtained by splitting the batch data of the training data.

[0082] For pipelined parallelism, various different pipelined parallelism strategies can be used. For example, in some embodiments, pipelined parallelism in the form of PipeDream 1F1B (One Forward pass followed by One Backward pass) can be used. The core idea of ​​the PipeDream 1F1B strategy is to alternate forward and backward computations on each computing device (processing unit). Specifically, when a computing device (processing unit) completes the forward propagation of a set of input data, it immediately begins the backward propagation of that set of input data without waiting for other computing devices (processing units) to complete their forward propagation. In this way, the activation values ​​on each computing device (processing unit) can be released after the backward propagation is completed, thereby greatly reducing memory usage.

[0083] However, calculating the training time for pipelined parallelism using the PipeDream 1F1B strategy can become complex. Therefore, in these embodiments, a reasonable time modeling approach is adopted to simplify the calculation of training time. For example, the pipelined parallel training period can be divided into two parts: an edge stage and a middle stage. Considering that the edge stage performs only forward or only backward computation, and the middle stage performs alternating forward and backward computation, the training time cost of the edge stage and the middle stage are calculated separately. The training time of the edge and middle stages is then used as the total time for one pipelined parallel training cycle. This allows for effective protection of the calculated pipelined parallel training time T while adapting to the PipeDream 1F1B strategy. pp The accuracy.

[0084] In this case, when using T pp When T represents the time required for one pipelined parallel training operation based on parameter combinations in the parameter search space, T pp Determined based on the following formula:

[0085] T pp =tedge +t middle ;

[0086]

[0087] In the above formula, t edge t represents the total time during pipelined parallel training when only forward computation and only backward computation are performed in all pipeline stages. middle t represents the time taken to perform forward and backward computations interleaved across all pipeline stages during parallel pipeline training. fi t represents the time t takes for the i-th pipeline stage to perform forward computation on a set of input data. bi S represents the time required for the i-th pipeline stage to perform reverse computation on a set of input data, and S represents the number of pipeline stages. max This represents the time taken to perform forward and backward computations on a set of input data across all pipeline stages, and the maximum pipeline number. num_mbs represents the number of sets of input data for each pipeline stage.

[0088] For ease of understanding, please refer to Figure 2 Explanation T pp The calculation method. Figure 2 The timing diagram of the pipelined parallel training shown represents a pipeline stage in one row. The width of the square marked with the number i indicates the time required to perform computation on the i-th set of input data, where dark gray represents the forward computation time and light gray represents the backward computation time. Figure 2 In the example, a single parallel training session consists of four pipeline stages performed on eight sets of input data, i.e., S = 4, num_mbs = 8. Figure 2 In the above-mentioned edge stage, there are two time periods: the left side, where only forward computation is performed in all pipeline stages, and the right side, where only backward computation is performed in all pipeline stages. The middle stage is a time period in which forward and backward computations are performed alternately.

[0089] Figure 2 The solid black lines in the diagram depict the squares corresponding to the calculation time for each time period. Figure 2In the example, each set of input data is the same size. Therefore, the time for each pipeline stage to perform forward computation (i.e., the width of each dark gray square in the same row) on each set of input data is the same, and the time for performing backward computation (i.e., the width of each light gray square in the same row) on each set of input data is also the same. Because forward and backward computations have different complexities, the time for each pipeline stage to perform forward computation on the same set of input data is different from the time to perform backward computation (e.g., the width of the dark gray square marked with the digit 0 in each row is different from the width of the light gray square marked with the digit 0). Because different pipeline stages use different types of processing units, the time for different pipeline stages to perform forward computation on the same set of input data is different (e.g., the dark gray square marked with the digit 0 has different widths in different rows), and the time for different pipeline stages to perform backward computation on the same set of input data is also different (e.g., the light gray square marked with the digit 0 has different widths in different rows).

[0090] In some embodiments, the pre-set search rules in step 103 may include: using a depth-first search algorithm to determine the parallelism combination scheme of the large model among all parallelism combinations based on the limitation of the parameter search space of the large model; using a dynamic programming algorithm to construct a state transition equation based on the training cost; and traversing the iterative parameters of the state transition equation to obtain the target parameter combination that minimizes the training cost.

[0091] Depth-First Search (DFS) is an algorithm for traversing or searching trees or graphs. It traverses the tree's nodes along its depth, searching the branches as deeply as possible. When all edges containing a node have been explored, or a node no longer meets the search criteria, the search backtracks to the starting node of the edge that found that node and tries other possible paths. This process is repeated until all nodes have been visited. Determining parallelism combinations using DFS should be done within the constraints of the parameter search space imposed by the large model. These constraints can be, for example, conditions that any possible combination of parallelism combinations (DP, PP, TP) must satisfy. For instance, a search tree can be built based on all parallelism combinations, with each node in the search tree corresponding to a specific parallelism combination. Using DFS, nodes satisfying these constraints are searched in the search tree, and the parallelism combinations corresponding to all nodes satisfying these constraints are taken as the parallelism combination scheme for the large model.

[0092] Next, dynamic programming can be used to construct the state transition equation based on the training cost described above. Dynamic programming is an algorithmic strategy for solving optimization problems by decomposing complex problems into smaller subproblems and storing the solutions to these subproblems to avoid redundant computation. The basic idea of ​​dynamic programming is to decompose a large problem into several smaller problems, which may overlap, meaning multiple smaller problems may contain the same subproblems. Dynamic programming avoids redundant computation by solving these smaller problems and storing the results for direct use when needed. Dynamic programming algorithms are typically used to solve problems with certain optimal properties. In such problems, there may be many feasible solutions. Each solution corresponds to a value, and the goal is to find the solution with the optimal value. In this application, the solution with the optimal value can be understood as the parameter combination that minimizes the aforementioned training cost. Therefore, the state equation can be constructed based on the training cost, and the target parameter combination that minimizes the training cost can be obtained by iterating through the parameters of the state transition equation. This effectively reduces the search difficulty of finding the optimal parameter combination in the parameter search space, thereby improving the search efficiency of the optimal parameter combination.

[0093] The substructure corresponding to the state transition equation can be States[stage, layer_index, mesh], where stage represents the index of the pipeline stage, layer_index represents the number of allocated model layers, and mesh represents the number of processing units used. layer_index can be used as a hyperparameter input, allowing users to flexibly select the allocation scheme of multiple operators in the model to adapt to actual application needs while reducing search difficulty.

[0094] The above state transition equation can be expressed as:

[0095]

[0096] Where [i, s, m] represents the index of the current state; e and prev_m represent the iteration parameters; This represents the time of the edge phase in pipelined parallel training. This represents the time of the middle stage in pipelined parallel training. t represents the communication time of the gradient data set. a This represents the result calculated up to the current state. The maximum value is the historical maximum value; t stage`pp_size` represents the computation time of the current pipeline stage, `pp_size` represents the current pipeline parallelism, and `num_mbs` represents the number of input data sets. By iterating over the iteration parameters `e` and `prev_m` within their respective ranges, the optimal solution to the state transition equation can be found, that is, the parameter combination that minimizes the aforementioned training cost.

[0097] In some embodiments, the pre-set search rules in step 103 may further include: performing at least one of heterogeneous communication pruning, load balancing pruning, and recompute pruning during the search process.

[0098] In heterogeneous communication pruning, if the time for point-to-point communication between different types of processing units exceeds the forward computation time of any pipeline stage in a pipeline partitioning scheme, the search branch corresponding to that pipeline partitioning scheme is pruned. This avoids unnecessary delays in pipelined parallel training caused by excessively long communication times in heterogeneous clusters.

[0099] In load balancing pruning, a threshold for the number of model layers to be trained in each pipeline stage is set based on the computational capacity of the processing units executing each pipeline stage. If, in a pipeline partitioning scheme, the number of model layers allocated to any pipeline stage is greater than or equal to the threshold, the search branch corresponding to that pipeline partitioning scheme is pruned. This avoids overloading each pipeline stage with computational resources and prevents computational time losses caused by some pipeline stages being allocated excessive computational loads.

[0100] In recomputation pruning, the target number of recomputation layers for each pipeline stage is calculated based on the number of model layers to be trained, the pipeline parallelism, and the GPU memory of the processing units. If a recomputation layer setting scheme does not meet the target number of recomputation layers, the search branch corresponding to that scheme is pruned. This allows for full utilization of the GPU memory of each processing unit, achieving shorter training time through a space-for-time tradeoff strategy.

[0101] By setting various pruning strategies such as heterogeneous communication pruning, load balancing pruning, and recomputation pruning during the parameter combination search process, parameter combination schemes that are overloaded, have excessively long heterogeneous communication times, or consume too much recomputation time can be quickly eliminated, reducing the number of parameter combinations to be searched and further improving the search efficiency of parameter combinations.

[0102] Next, proceed to step 104.

[0103] In step 104, the heterogeneous cluster performs parallel training on the large model based on the combination of target parameters.

[0104] After finding the target parameter combination in step 103, the heterogeneous cluster can perform parallel training on the large model based on the target parameter combination. Since the target parameter combination is the parameter combination that minimizes the training cost, i.e., the total time required for the heterogeneous cluster to perform one round of training on the large model based on the corresponding parameter combination, it can save training time and resources and improve training efficiency compared with other parameter combinations when the heterogeneous cluster performs parallel training on the large model based on the target parameter combination.

[0105] In general, the method for parallel training of large models on heterogeneous clusters based on the embodiments of this application can replace the method of searching for parameter combinations in related technologies. It can efficiently search for optimization schemes for specific parameter combinations (such as combinations of data parallelism, pipeline parallelism and tensor parallelism, the types of processing units to be executed in each pipeline stage, the number of model layers to be trained in each pipeline stage, and the number of model layers to be recomputed in each pipeline stage) for heterogeneous clusters, thereby saving training time and training resources and improving training efficiency.

[0106] like Figure 3 As shown, this application embodiment also provides a system 300 for parallel training of large models. System 300 includes: a heterogeneous cluster 301, which includes at least multiple processing units of various types, with parallelism including data parallelism, pipelined parallelism, and tensor parallelism, and at least some pipelined stages of pipelined parallelism being executed by different types of processing units; and a device 302 for parallel training of large models using the heterogeneous cluster. Device 302 includes: a processor 3021; ​​and a memory 3022 storing a computer program executable by the processor 3021. When the computer program is executed by the processor 3021, the processor 3021 implements a parameter search process according to the embodiments of this application (as described above, the process of searching for target parameter combinations), and after finding a target parameter combination, sends the target parameter combination to the heterogeneous cluster 301, enabling the heterogeneous cluster 301 to perform parallel training of the large model based on the target parameter combination.

[0107] It should be understood that, although Figure 3 Device 302, used for parallel training of large models on heterogeneous clusters, is shown as a device independent of heterogeneous cluster 301, but in some embodiments, some or all of the functionality of device 302 may be implemented on any computing device in heterogeneous cluster 301.

[0108] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for parallel training of heterogeneous clusters for large models according to embodiments of this application.

[0109] This application also provides a computer program product, including computer instructions that, when executed by a processor, cause the processor to implement a method for parallel training of heterogeneous clusters for large models according to embodiments of this application.

[0110] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0113] As used herein, the term "machine-readable medium" refers to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0115] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0116] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0117] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0118] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for parallel training of a heterogeneous cluster of large models, wherein the heterogeneous cluster comprises at least multiple processing units of multiple types, the parallelism includes data parallelism, pipeline parallelism, and tensor parallelism, at least some pipeline stages of the pipeline parallelism are executed by processing units of different types, and the data parallelism and the tensor parallelism are both configured with a homogeneous strategy, the method comprising: Obtain first input information and second input information, wherein the first input information represents the model structure parameters of the large model to be trained, and the second input information represents the device parameters of the heterogeneous cluster for training the large model; The parameter search space is determined based on the first input information and the second input information; According to pre-set search rules, search for target parameter combinations in the parameter search space; and The heterogeneous cluster performs parallel training on the large model based on the target parameter combination. Wherein, the target parameter combination is the parameter combination that minimizes the training cost among multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform one round of training on the large model based on the corresponding parameter combination. The total time is the sum of the time for one pipelined parallel training based on the corresponding parameter combination, the time for gradient data set communication, and the time for the optimizer of the large model to update the model parameters of the large model. The pipelined parallelism adopts a method of alternating forward and backward computation on each processing unit. The pipelined parallel training process is divided into an edge stage indicating the beginning and end and a middle stage indicating the middle part. The time for one pipelined parallel training is the sum of the training time of the edge stage and the training time of the middle stage. The edge stage performs only forward computation or only backward computation, and the middle stage performs forward and backward computation alternately.

2. The method according to claim 1, wherein, The parameter search space includes parallelism combination schemes, processing unit allocation schemes, pipeline partitioning schemes, and recomputation layer setting schemes. Here, the parallelism combination scheme represents the optional combination of data parallelism, pipeline parallelism, and tensor parallelism; the processing unit allocation scheme represents the optional types of processing units to execute each pipeline stage; the pipeline partitioning scheme represents the number of model layers to be trained in each pipeline stage; and the recomputation layer setting scheme represents the number of model layers to be recomputed in each pipeline stage. The parameter combination includes: a combination of data parallelism, pipeline parallelism, and tensor parallelism; the type of processing unit executing each pipeline stage; the number of model layers to be trained in each pipeline stage; and the number of model layers to be recomputed in each pipeline stage.

3. The method according to claim 1, wherein, When using T pp T represents the time required for one pipelined parallel training operation based on the corresponding parameter combinations. all_reduce When representing the time of gradient data set communication, T pp Determined based on the following formula: T pp =t edge +t middle ， Among them, t edge t represents the total time during pipelined parallel training when only forward computation and only backward computation are performed in all pipeline stages. middle t represents the time taken to perform forward and backward computations interleaved across all pipeline stages during parallel pipeline training. fi t represents the time t takes for the i-th pipeline stage to perform forward computation on a set of input data. bi S represents the time required for the i-th pipeline stage to perform reverse computation on a set of input data, and S represents the number of pipeline stages. max This represents the pipeline number with the maximum sum of time for forward and backward computation on a single set of input data across all pipeline stages, and num_mbs represents the number of sets of input data for each pipeline stage. T all_reduce Determined based on the following formula: ， Where, `data` represents the amount of data that needs to be transmitted for gradient data set communication between parallel data groups, `bandwidth` represents the data transmission bandwidth for gradient data set communication, and `layers` represents the data transmission bandwidth. i This indicates the number of model layers to be executed in the i-th pipeline stage, micro_batch_size indicates the size of a set of input data for each pipeline stage, and gradient indicates the gradient.

4. The method according to claim 2, wherein, The pre-set search rules include: Using a depth-first search algorithm, and based on the constraints of the large model on the parameter search space, the parallelism combination scheme is determined from all parallelism combinations. Using a dynamic programming algorithm, a state transition equation is constructed based on the training cost; The iterative parameters of the state transition equation are traversed to obtain the target parameter combination that minimizes the training cost.

5. The method according to claim 4, wherein, The pre-set search rules also include: During the search process, at least one of heterogeneous communication pruning, load balancing pruning, and recompute pruning is performed. In the heterogeneous communication pruning, if the time for point-to-point communication between different types of processing units exceeds the forward computation time of any pipeline stage in a pipeline partitioning scheme, the search branch corresponding to that pipeline partitioning scheme is pruned. In the load balancing pruning, a threshold for the number of model layers to be trained in each pipeline stage is set based on the computing power of the processing unit executing each pipeline stage. If, in a pipeline splitting scheme, the number of model layers allocated to any pipeline stage is greater than or equal to the threshold, the search branch corresponding to that pipeline splitting scheme is pruned. In the recomputation pruning, the target number of recomputation layers for the pipeline stage is calculated based on the number of model layers to be trained in the pipeline stage, the pipeline parallelism, and the GPU memory of the processing unit. If a recomputation layer setting scheme does not meet the target number of recomputation layers, the search branch corresponding to the recomputation layer setting scheme is pruned.

6. The method according to claim 3, wherein, The gradient data set communication uses the AllReduce algorithm, and The pipeline parallelism includes pipeline parallelism in the form of PipeDream 1F1B.

7. An apparatus for parallel training of a heterogeneous cluster of large models, the heterogeneous cluster comprising at least multiple types of processing units, the parallelism including data parallelism, pipelined parallelism, and tensor parallelism, wherein at least some pipeline stages of the pipelined parallelism are executed by different types of processing units, and the data parallelism and the tensor parallelism are both configured with a homogeneous strategy, the apparatus comprising: processor; as well as A memory storing a computer program executable by the processor, which, when executed by the processor, causes the processor to perform the following operations: Obtain first input information and second input information, wherein the first input information represents the model structure parameters of the large model to be trained, and the second input information represents the device parameters of the heterogeneous cluster for training the large model; The parameter search space is determined based on the first input information and the second input information; According to pre-set search rules, search for target parameter combinations in the parameter search space; and The target parameter combination is sent to the heterogeneous cluster, enabling the heterogeneous cluster to perform parallel training on the large model based on the target parameter combination. Wherein, the target parameter combination is the parameter combination that minimizes the training cost among multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform one round of training on the large model based on the corresponding parameter combination. The total time is the sum of the time for one pipelined parallel training based on the corresponding parameter combination, the time for gradient data set communication, and the time for the optimizer of the large model to update the model parameters of the large model. The pipelined parallelism adopts a method of alternating forward and backward computation on each processing unit. The pipelined parallel training process is divided into an edge stage indicating the beginning and end and a middle stage indicating the middle part. The time for one pipelined parallel training is the sum of the training time of the edge stage and the training time of the middle stage. The edge stage performs only forward computation or only backward computation, and the middle stage performs forward and backward computation alternately.

8. A system for parallel training of large models, comprising: The heterogeneous cluster includes at least multiple types of processing units, and the parallelism includes data parallelism, pipeline parallelism and tensor parallelism. At least some pipeline stages of the pipeline parallelism are executed by different types of processing units, and the data parallelism and the tensor parallelism are both set to a homogeneous strategy. as well as A device for parallel training of heterogeneous clusters of large models, the device comprising: Processor; and A memory storing a computer program executable by the processor, which, when executed by the processor, causes the processor to perform the following operations: Obtain first input information and second input information, wherein the first input information represents the model structure parameters of the large model to be trained, and the second input information represents the device parameters of the heterogeneous cluster for training the large model; The parameter search space is determined based on the first input information and the second input information; According to pre-set search rules, search for target parameter combinations in the parameter search space; and The target parameter combination is sent to the heterogeneous cluster, enabling the heterogeneous cluster to perform parallel training on the large model based on the target parameter combination. Wherein, the target parameter combination is the parameter combination that minimizes the training cost among multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform one round of training on the large model based on the corresponding parameter combination. The total time is the sum of the time for one pipelined parallel training based on the corresponding parameter combination, the time for gradient data set communication, and the time for the optimizer of the large model to update the model parameters of the large model. The pipelined parallelism adopts a method of alternating forward and backward computation on each processing unit. The pipelined parallel training process is divided into an edge stage indicating the beginning and end and a middle stage indicating the middle part. The time for one pipelined parallel training is the sum of the training time of the edge stage and the training time of the middle stage. The edge stage performs only forward computation or only backward computation, and the middle stage performs forward and backward computation alternately.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the method as described in any one of claims 1 to 6.

10. A computer program product comprising computer instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 6.

Images