Heterogeneous cluster parallel training method, device and system for large model

Abstract

The invention relates to the technical field of artificial intelligence, and provides a heterogeneous cluster parallel training method, device and system for a large model. The method comprises the steps of obtaining model structure parameter information representing a to-be-trained large model and equipment parameter information representing a heterogeneous cluster for training the large model; determining a parameter search space according to the parameter information, and searching a target parameter combination in the parameter search space according to a preset search rule; the heterogeneous cluster carries out parallel training on the large model based on the target parameter combination, and the target parameter combination is a parameter combination which enables the time consumption of one round of training of the large model to be minimum in the multiple parameter combinations included in the parameter search space. The invention provides an automatic search scheme of the target parameter combination to replace a parameter search scheme in related technologies, so that the time consumption for obtaining the target parameter combination can be reduced, and the model development period can be shortened.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a method, device, and system for heterogeneous cluster parallel training of large models. Background Art

[0002] A large model refers to a machine learning model with large-scale parameters and complex computing structure. It is widely used in many fields such as natural language processing, image recognition, and speech recognition. Summary of the invention

[0003] When training a large model, it is possible to use a heterogeneous cluster to train the large model in parallel. Heterogeneous clusters usually include multiple types of processing units, such as different types of graphics processing units (GPUs). Parallel training usually includes data parallelism, pipeline parallelism, and tensor parallelism.

[0004] The inventors of the present application have noticed that in the related art, in order to obtain the highest training efficiency on a heterogeneous cluster, developers often need to manually select multiple parameter combinations to train a large model, and need to wait for the large model to complete training under all parameter combinations to obtain and record the training efficiency data under each parameter combination, so as to select a parameter combination with higher training efficiency. Such a parameter search process is very cumbersome and time-consuming, and when the cluster topology or machine model changes, it is often necessary to manually search for the best parameter combination for model training again, resulting in a long model development cycle. Especially in the environment of heterogeneous clusters, the hardware performance and software development completeness of different processing units may differ, making the search for parameter combinations more difficult.

[0005] In view of this, the purpose of the present application is to provide a method, device, system, computer-readable storage medium and computer program product for heterogeneous cluster parallel training of large models to solve the technical problems in the related art.

[0006] In a first aspect, the present application provides a method for heterogeneous cluster parallel training of a large model, wherein the heterogeneous cluster includes at least a plurality of processing units of a plurality of types, wherein the parallelism includes data parallelism, pipeline parallelism, and tensor parallelism, and at least some pipeline stages of the pipeline parallelism are executed by processing units of different types, and the method includes:

[0007] Acquire first input information and second input information, wherein the first input information represents model structure parameters of the large model to be trained, and the second input information represents device parameters of the heterogeneous cluster for training the large model;

[0008] Determining a parameter search space according to the first input information and the second input information;

[0009] According to a preset search rule, searching for a target parameter combination in the parameter search space; and

[0010] The heterogeneous cluster performs parallel training on the large model based on the target parameter combination,

[0011] The target parameter combination is a parameter combination that minimizes the training cost among the multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform a round of training on the large model based on the corresponding parameter combination.

[0012] The total time is the sum of the time for performing a pipeline parallel training based on the corresponding parameter combination, the time for communicating the gradient data set, and the time for the optimizer of the large model to update the model parameters of the large model.

[0013] In a second aspect, the present application further provides a device for heterogeneous cluster parallel training of a large model, wherein the heterogeneous cluster includes at least a plurality of processing units of a plurality of types, wherein the parallelism includes data parallelism, pipeline parallelism, and tensor parallelism, and at least some pipeline stages of the pipeline parallelism are executed by processing units of different types, and the device includes:

[0014] Processor; and

[0015] A memory having a computer program executable by the processor stored therein, which, when executed by the processor, causes the processor to implement the following operations:

[0016] Acquire first input information and second input information, wherein the first input information represents model structure parameters of the large model to be trained, and the second input information represents device parameters of the heterogeneous cluster for training the large model;

[0017] Determining a parameter search space according to the first input information and the second input information;

[0018] According to a preset search rule, searching for a target parameter combination in the parameter search space; and

[0019] Sending the target parameter combination to the heterogeneous cluster so that the heterogeneous cluster performs parallel training on the large model based on the target parameter combination,

[0020] The target parameter combination is a parameter combination that minimizes the training cost among the multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform a round of training on the large model based on the corresponding parameter combination.

[0021] The total time is the sum of the time for performing a pipeline parallel training based on the corresponding parameter combination, the time for communicating the gradient data set, and the time for the optimizer of the large model to update the model parameters of the large model.

[0022] In a third aspect, the present application provides a system for parallel training of a large model, comprising:

[0023] A heterogeneous cluster, the heterogeneous cluster comprising at least a plurality of processing units of a plurality of types, the parallelism comprising data parallelism, pipeline parallelism and tensor parallelism, at least some pipeline stages of the pipeline parallelism being executed by processing units of different types; and

[0024] A device for heterogeneous cluster parallel training of a large model, the device comprising:

[0025] Processor; and

[0026] A memory having a computer program executable by the processor stored therein, which, when executed by the processor, causes the processor to implement the following operations:

[0027] Acquire first input information and second input information, wherein the first input information represents model structure parameters of the large model to be trained, and the second input information represents device parameters of the heterogeneous cluster for training the large model;

[0028] Determining a parameter search space according to the first input information and the second input information;

[0029] According to a preset search rule, searching for a target parameter combination in the parameter search space; and

[0030] Sending the target parameter combination to the heterogeneous cluster so that the heterogeneous cluster performs parallel training on the large model based on the target parameter combination,

[0031] The target parameter combination is a parameter combination that minimizes the training cost among the multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform a round of training on the large model based on the corresponding parameter combination.

[0032] The total time is the sum of the time for performing a pipeline parallel training based on the corresponding parameter combination, the time for communicating the gradient data set, and the time for the optimizer of the large model to update the model parameters of the large model.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the embodiments of the present application are implemented.

[0034] In a fifth aspect, the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the embodiments of the present application.

[0035] In the heterogeneous mixed training scenario, the present application first determines the parameter search space according to the model structure parameters and device parameters input by the user, and then based on the preset search rules, takes the total time required for the heterogeneous cluster to perform a round of training on the large model based on the corresponding parameter combination as the minimum as the search target, and automatically searches for the target parameter combination in the determined parameter search space, that is, automatically searches for the optimal parameter combination, thereby replacing the method of searching for the optimal parameter combination in the related art, which can effectively reduce the time consumption of obtaining the optimal parameter combination, improve the accuracy of the obtained optimal parameter combination, and effectively shorten the model development cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of a method for parallel training of heterogeneous clusters of large models provided in an embodiment of the present application;

[0037] Figure 2 It is a timing diagram of pipeline parallel training provided in an embodiment of the present application;

[0038] Figure 3 It is a structural diagram of a system for parallel training of heterogeneous clusters of large models provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] The present application embodiment provides a method for parallel training of heterogeneous clusters of large models. The main steps of the method are as follows: Figure 1As shown. According to the heterogeneous cluster of the embodiment of the present application, at least multiple processing units of multiple types are included, for example, multiple models of graphics processing units (GPUs) from different manufacturers. "Parallelism" in parallel training includes data parallelism, pipeline parallelism and tensor parallelism. Data parallelism refers to dividing the training data into several groups, assigning each data group to a group of processing units, and each group of processing units runs the same processing program to process the assigned data groups. Pipeline parallelism refers to dividing one or more adjacent layers of the model into a pipeline stage, thereby dividing the model into multiple pipeline stages. In the present application, at least some pipeline stages of pipeline parallelism are executed by processing units of different types. For example, H pipeline stages (p≥H, H≥2) in p pipeline stages are respectively assigned to H types of GPUs, and each GPU includes, for example, m GPUs. Of course, the number of each GPU executing each pipeline stage can also be different according to actual needs. Tensor parallelism refers to dividing the tensor parameters of the model into multiple groups according to a specific dimension, and each group of tensor parameters is assigned to different processing units.

[0041] like Figure 1 As shown, at step 101, first input information and second input information are obtained.

[0042] The first input information represents the model structure parameters of the large model to be trained, and the second input information represents the device parameters of the heterogeneous cluster for training the large model.

[0043] Without loss of generality, the model structure parameters may include the total number of operators of the large model, i.e., the total number of layers of the large model, the length of the training data sequence, and the number of hidden states in the model. The device parameters may include the number of types corresponding to the multiple processing units, the number of processing units of each type, the memory capacity of each type of processing unit, and the communication bandwidth, wherein the communication bandwidth includes the bandwidth for internal communication of the processing unit node, the bandwidth for communication between different processing unit nodes of the same type, and the bandwidth for communication between different processing unit nodes of different types.

[0044] Next, go to step 102 .

[0045] At step 102, a parameter search space is determined according to first input information and second input information.

[0046] The parameter search space may, for example, represent a set of optional parameter combinations of some parameters for parallel training of a large model by a heterogeneous cluster.

[0047] In some embodiments, the parameter search space includes a parallelism combination scheme, a processing unit allocation scheme, a pipeline segmentation scheme, and a recalculation layer number setting scheme.

[0048] The parallelism combination scheme can, for example, represent an optional combination of data parallelism, pipeline parallelism, and tensor parallelism. For example, when DP represents data parallelism, PP represents pipeline parallelism, and TP represents tensor parallelism, the parallelism combination scheme can be expressed as (DP, PP, TP). The processing unit allocation scheme can, for example, represent the optional types of processing units that execute each pipeline stage. For example, each of the p pipeline stages can be executed by any one or more of the H types of processing units (such as GPUs). The pipeline segmentation scheme can, for example, represent the number of model layers to be trained in each pipeline stage. For example, the i-th pipeline stage among the p pipeline stages is to train ni-layer models. The recalculation layer number setting scheme represents the number of model layers to be recalculated in each pipeline stage. For example, for training n i The i-th pipeline stage of the layer model can set the number of model layers to be recalculated in this pipeline stage to 0~n i .

[0049] Any optional combination (DP, PP, TP) in the above parallelism combination schemes meets the following conditions:

[0050] DP×PP×TPM;

[0051] Among them, DP refers to data parallelism, TP refers to tensor parallelism, PP refers to pipeline parallelism, and M refers to the total number of processing units (such as GPUs) included in the heterogeneous cluster.

[0052] The above processing unit allocation scheme satisfies the following condition: the video memory occupancy of the training task assigned to each processing unit (such as GPU) in the heterogeneous cluster is less than or equal to the video memory capacity of the processing unit.

[0053] The conditions that the above pipeline segmentation scheme should meet can be expressed as:

[0054]

[0055] The total number of pipeline segmentation solutions is:

[0056]

[0057] Among them, N refers to the total number of operators (total number of layers) of the large model, p refers to the total number of pipeline stages included in the pipeline, and ni refers to the number of operators (number of model layers) to be trained in the i-th pipeline stage. Among them, the number of operators to be trained in any pipeline stage should also be less than or equal to N.

[0058] The total number of solutions for the above recalculation layer settings can be expressed as:

[0059]

[0060] The total number of composite solutions formed by pipeline segmentation combined with the setting of the number of recalculation layers is:

[0061]

[0062] Here, Npipe refers to the total number of recalculation schemes in the case of superimposed pipeline segmentation schemes.

[0063] From the above, it can be seen that through the combination of the above-mentioned schemes, the parameter search space can be reasonably limited (so that the parameter combinations in the parameter search space at least meet the above-mentioned conditions) while adapting to the actual situation of the model structure and the maximum processing capacity of the heterogeneous cluster. This can limit the size of the parameter search space as much as possible while ensuring the accuracy of the target parameter combination in the final search, thereby improving the search efficiency of the target parameter combination.

[0064] Next, go to step 103 .

[0065] At step 103, a target parameter combination is searched in the parameter search space according to a preset search rule.

[0066] The target parameter combination may be a parameter combination that minimizes the training cost among the multiple parameter combinations included in the parameter search space. In the case where the parameter search space includes the parallelism combination scheme, the processing unit allocation scheme, the pipeline segmentation scheme, and the recalculation layer number setting scheme as described above, the parameter combination in the parameter search space may include: a combination of data parallelism, pipeline parallelism, and tensor parallelism, the type of processing unit executing each pipeline stage, the number of model layers to be trained in each pipeline stage, and the number of model layers to be recalculated in each pipeline stage.

[0067] The training cost may be the total time required for the heterogeneous cluster to perform one round of training on the large model based on the parameter combination in the parameter search space. In some embodiments, the total time may be, for example, the sum of the time for performing one pipeline parallel training based on the parameter combination in the parameter search space, the time for communicating the gradient data set, and the time for the optimizer of the large model to update the model parameters of the large model. In this case, the training cost may be expressed based on the following formula:

[0068] Tcost=[TPP+Tall-reduce+Toptimizer];

[0069] In the above formula, Tcos tIt represents the training cost, that is, the total time required for a heterogeneous cluster to perform a round of training on the large model based on the parameter combination in the parameter search space. Tpp represents the time for a pipeline parallel training based on the corresponding parameter combination. Tall_reduce represents the time for gradient data set communication. Toptimizer represents the time for the optimizer of the large model to update the model parameters of the large model.

[0070] In the heterogeneous mixed training scenario, the present application first determines the parameter search space of the parameter combination required by the user according to the model structure parameters and device parameters input by the user, and then based on the pre-set search rules, takes the minimum total time required for the heterogeneous cluster to perform a round of training on the large model based on the corresponding parameter combination as the search target, and automatically searches for the target parameter combination in the determined parameter search space, that is, automatically searches for the optimal parameter combination, thereby replacing the method of searching for the optimal parameter combination in the related art, which can effectively reduce the time consumption of obtaining the optimal parameter combination, improve the accuracy of the obtained optimal parameter combination, and effectively shorten the model development cycle.

[0071] By comprehensively considering the time for a pipeline parallel training based on the corresponding parameter combination in the parameter search space, the time for the gradient data set communication, and the time for the optimizer of the large model to update the model parameters of the large model, the total time required for the heterogeneous cluster to perform a round of training on the large model based on the corresponding parameter combination is determined, and the total time is used as the training cost under the corresponding parameter combination, which can more accurately characterize the training cost. In addition, the search for the optimal parameter combination (i.e., the target parameter combination) based on the total time for a round of model training based on the corresponding parameter combination as the training cost can make the searched optimal parameter combination have higher accuracy and reliability.

[0072] It should be noted that if large-scale communication is carried out between heterogeneous chips in a heterogeneous cluster, the computing efficiency will be severely attenuated. Therefore, in this application, data parallelism and tensor parallelism are both set as isomorphic strategies, that is, for data parallelism and tensor parallelism, the types of processing units involved are consistent. Heterogeneity is set only in the pipeline parallel dimension, that is, different types of processing units are used to process different pipeline stages. Therefore, in the dimensions of data parallelism and tensor parallelism, the training time is the same, so when designing the training cost, only pipeline parallelism is considered, and the time for a pipeline parallel training is included in the training cost, thereby simplifying the calculation formula of the training cost and improving the calculation efficiency of the training cost.

[0073] It should also be understood that T optimizer The actual running time of each processing unit in the heterogeneous cluster can be obtained, so the performance data of the processing unit can be collected in advance through pre-testing, without collecting performance data on the entire heterogeneous cluster during each training.

[0074] In some embodiments, a tree structure may be constructed based on the parameter search space determined in step 102, and each node in the tree structure represents a parameter combination in the parameter search space. The tree structure is searched using a depth-first algorithm or a breadth-first algorithm, with the total time required for a heterogeneous cluster to perform a round of training on a large model based on the corresponding parameter combination being minimized as the search target, thereby obtaining the target parameter combination.

[0075] In some embodiments, for example, the AllReduce algorithm can be used to communicate gradient data sets. The AllReduce algorithm is a communication algorithm commonly used in large model distributed training, which is used to synchronize parameters between processing unit nodes. During the training process of large models, model parameters need to be synchronized between nodes to ensure the consistency of training. The AllReduce algorithm adds the parameters of all nodes and then broadcasts the results to all nodes, so that each node can obtain the updated parameter value, thereby achieving parameter synchronization. Compared with other communication algorithms, the AllReduce algorithm can better balance the communication overhead between nodes and improve training efficiency.

[0076] When using the AllReduce algorithm to communicate gradient data sets, such as when using T all_reduce When T represents the time of gradient data set communication, all_reduce For example, it can be determined based on the following formula:

[0077]

[0078] Among them, data represents the amount of data that needs to be transmitted for gradient data set communication between data parallel groups, bandwidth represents the data transmission bandwidth of gradient data set communication, and layer s i It represents the number of model layers to be executed in the i-th pipeline stage, micro_batch_size represents the size of a set of input data for each pipeline stage, and gradient represents the gradient.

[0079] In this calculation method of communication time, the ratio of the amount of data required to be transmitted for gradient data set communication between data parallel groups to the data transmission bandwidth is determined as the gradient set data communication time, and the data amount is defined based on the number of model layers to be executed in the i-th pipeline stage, the size of a set of input data for each pipeline stage, and the gradient, which can make the calculation of the gradient data communication time more accurate and reliable.

[0080] Note that in a heterogeneous training environment, there may be multiple different bandwidths for different types of processing units in a heterogeneous cluster, and bandwidth should be understood as the minimum bandwidth among the aforementioned multiple different bandwidths.

[0081] In addition, the aforementioned set of input data may be a micro batch data. The micro batch data is obtained by further splitting the mini batch data obtained by splitting the batch data of the training data.

[0082] For pipeline parallelism, a variety of different pipeline parallel strategies can also be used. For example, in some embodiments, pipeline parallelism in the form of PipeDream 1F1B (One Forward pass followed by One Backward pass) can be used. The core idea of ​​the PipeDream 1F1B strategy is: on each computing device (processing unit), forward calculation and reverse calculation are performed alternately. Specifically, when a computing device (processing unit) completes the forward propagation of a group of input data, the reverse propagation of the group of input data is started immediately without waiting for other computing devices (processing units) to complete the forward propagation. In this way, the activation value (Activation) on each computing device (processing unit) can be released after the reverse propagation is completed, thereby greatly reducing memory usage.

[0083] However, the calculation of pipeline parallel training time using the PipeDream 1F1B strategy may become complicated. Based on this, in these embodiments, in order to simplify the calculation of training time, a reasonable time modeling method is adopted. For example, the pipeline parallel training period can be divided into the edge stage at the beginning and the end and the middle stage in the middle. In view of the characteristics that only forward calculation or only reverse calculation is performed in the edge stage, and forward calculation and reverse calculation are performed alternately in the middle stage, the training time overhead of the edge stage and the training time overhead of the middle stage are calculated respectively, and the training time of the edge stage and the middle stage are used as the total time for a pipeline parallel training. In this way, while adapting the PipeDream 1F1B strategy, the calculated pipeline parallel training time T can be effectively guaranteed. pp accuracy.

[0084] In this case, when using T pp T represents the time for a pipeline parallel training based on the parameter combination in the parameter search space. pp Determined based on the following formula:

[0085] T pp =tedge +t middle ;

[0086]

[0087] In the above formula, t edge represents the total time of executing only forward computation and only backward computation in all pipeline stages during pipeline parallel training, t middle represents the time to interleave forward and backward computations in all pipeline stages during pipeline parallel training, t fi represents the time for the i-th pipeline stage to perform forward calculation on a set of input data, t bi represents the time it takes for the i-th pipeline stage to perform the reverse calculation for a set of input data, S represents the number of pipeline stages, and S max It indicates the time and the maximum pipeline number for performing forward and reverse calculations on a set of input data in all pipeline stages. num_mbs indicates the number of groups of input data for each pipeline stage.

[0088] For ease of understanding, refer to Figure 2 Description pp The calculation method of . Figure 2 Each row in the timing diagram of pipeline parallel training shown represents a pipeline stage, and the width of the square marked with the number i represents the time required to perform the calculation for the i-th group of input data, where dark gray is the forward calculation time and light gray is the reverse calculation time. Figure 2 In the example of , one parallel training includes 4 pipeline stages executed for 8 sets of input data, that is, S=4, num_mbs=8. Figure 2 In the figure, the aforementioned edge stage includes a time period on the left side during which only forward calculation is performed in all pipeline stages and a time period on the right side during which only reverse calculation is performed in all pipeline stages. The middle stage in the middle is a time period during which forward calculation and reverse calculation are performed alternately.

[0089] Figure 2 The black solid line in the figure shows the corresponding squares of the calculation time for each time period. Figure 2In the example, the size of each set of input data is the same, so the time for each pipeline stage to perform forward calculation for each set of input data (i.e., the width of each dark gray square in the same row) is the same, and the time for each pipeline stage to perform reverse calculation for each set of input data (i.e., the width of each light gray square in the same row) is also the same. Due to the different complexities of forward calculation and reverse calculation, the time for each pipeline stage to perform forward calculation and reverse calculation for the same set of input data is different (for example, the width of the dark gray square marked with the number 0 in each row is different from the width of the light gray square marked with the number 0). Since different pipeline stages use different types of processing units, for the same set of input data, the time for different pipeline stages to perform forward calculation for the same set of input data is different (for example, the dark gray square marked with the number 0 has different widths in different rows), and the time for different pipeline stages to perform reverse calculation for the same set of input data is also different (for example, the light gray square marked with the number 0 has different widths in different rows).

[0090] In some embodiments, the pre-set search rules in step 103 may include: using a depth-first search algorithm to determine the parallelism combination scheme of the large model among all parallelism combinations based on the restrictions of the large model on the parameter search space; using a dynamic programming algorithm to construct a state transfer equation based on the training cost; and traversing the iterative parameters of the state transfer equation to obtain the target parameter combination that minimizes the training cost.

[0091] The depth-first search algorithm (DFS) is an algorithm for traversing or searching a tree or graph. The algorithm traverses the nodes of the tree along the depth of the tree and searches the branches of the tree as deeply as possible. When the edges where a node is located have been explored, or the node does not meet the search conditions, the search will trace back to the starting node of the edge where the node is found and try other possible paths. This process is repeated until all nodes are visited. The parallelism combination scheme determined by the depth-first search algorithm should be carried out under the restriction of the large model on the parameter search space. The restriction of the large model on the parameter search space can be, for example, the conditions that any optional combination (DP, PP, TP) in the parallelism combination scheme described above should meet. For example, a search tree can be established based on all parallelism combinations, and a node of the search tree corresponds to a parallelism combination. Using the depth-first search algorithm, search for nodes that meet the above restrictions in the search tree, and use the parallelism combinations corresponding to all nodes that meet the above restrictions as the parallelism combination scheme of the large model.

[0092] Next, a dynamic programming algorithm can be used to construct a state transfer equation based on the training cost as described above. The dynamic programming algorithm is an algorithm strategy for solving optimization problems by decomposing complex problems into smaller sub-problems and storing the solutions to the sub-problems to avoid repeated calculations. The basic idea of ​​dynamic programming is to decompose a large problem into several small problems, which may overlap with each other, that is, multiple small problems may contain the same sub-problems. Dynamic programming avoids repeated calculations by solving these small problems and storing the results so that they can be used directly when needed. Dynamic programming algorithms are usually used to solve problems with certain optimal properties. In such problems, there may be many feasible solutions. Each solution corresponds to a value, and the goal is to find a solution with an optimal value. In this application, a solution with an optimal value can be understood as a parameter combination that minimizes the aforementioned training cost. Therefore, a state equation can be constructed based on the training cost, and the target parameter combination that minimizes the training cost can be obtained by traversing the iterative parameters of the state transfer equation. As a result, the difficulty of searching for the optimal parameter combination in the parameter search space can be effectively reduced, thereby improving the search efficiency of the optimal parameter combination.

[0093] The substructure corresponding to the state transition equation can be States[stage,layer_index,mesh], where stage represents the index of the pipeline stage, layer_index represents the number of model layers allocated, and mesh represents the number of processing units used. For example, layer_index can be used as a hyperparameter input, allowing users to flexibly select the allocation scheme of multiple operators in the model to adapt to actual application needs while reducing the difficulty of search.

[0094] The above state transfer equation can be expressed as:

[0095]

[0096] Among them, [i, s, m] represents the index of the current state; e and prev_m represent iteration parameters; Indicates the time of edge stage in pipeline parallel training, Indicates the time of the middle stage in pipeline parallel training, represents the communication time of the gradient data set; t a Indicates the calculated value up to the current state The maximum value is the historical maximum value; t stageIndicates the calculation time of the current pipeline stage, pp_size indicates the current pipeline parallelism; num_mbs indicates the number of input data groups. By traversing the iteration parameters e and prev_m in their respective value ranges, the optimal solution of the state transition equation can be found, that is, the parameter combination that minimizes the aforementioned training cost.

[0097] In some embodiments, the pre-set search rules in step 103 may also include: performing at least one of heterogeneous communication pruning, load balancing pruning and recalculation pruning during the search process.

[0098] In heterogeneous communication pruning, when the time for point-to-point communication between different types of processing units is greater than the forward calculation time of any pipeline stage in a pipeline segmentation scheme, the search branch corresponding to the pipeline segmentation scheme is pruned. In this way, unnecessary delays in pipeline parallel training caused by excessive communication time in heterogeneous clusters can be avoided.

[0099] In load-balanced pruning, a threshold for the number of model layers to be trained in each pipeline stage is set according to the computing power of the processing unit executing each pipeline stage. If the number of model layers allocated to any pipeline stage in a pipeline segmentation scheme is greater than or equal to the threshold, the search branch corresponding to the pipeline segmentation scheme is pruned. In this way, each pipeline stage can be prevented from bearing an excessive computing load, and computing time loss caused by excessive computing loads being allocated to some pipeline stages can be avoided.

[0100] In recalculation pruning, the target number of recalculation layers for the pipeline stage is calculated based on the number of model layers to be trained in the pipeline stage, the pipeline parallelism, and the video memory of the processing unit. If a recalculation layer setting scheme does not meet the target number of recalculation layers, the search branch corresponding to the recalculation layer setting scheme is pruned. In this way, the video memory of each processing unit can be fully utilized, and a shorter training time can be achieved by trading space for time.

[0101] By setting various pruning strategies such as heterogeneous communication pruning, load balancing pruning and recomputation pruning during the parameter combination search process, parameter combinations that are overloaded, have too long heterogeneous communication time, or take too much time to recompute can be quickly eliminated, thereby reducing the number of parameter combinations to be searched and further improving the search efficiency of parameter combinations.

[0102] Next, go to step 104 .

[0103] At step 104, the large model is trained in parallel by the heterogeneous cluster based on the target parameter combination.

[0104] After searching for the target parameter combination in step 103, the heterogeneous cluster can perform parallel training on the large model based on the target parameter combination. Since the target parameter combination is the parameter combination that minimizes the training cost, i.e., the total time required for the heterogeneous cluster to perform one round of training on the large model based on the corresponding parameter combination, when the heterogeneous cluster performs parallel training on the large model based on the target parameter combination, it can save training time and training resources compared with other parameter combinations, thereby improving training efficiency.

[0105] In general, the method for parallel training of heterogeneous clusters of large models based on the embodiments of the present application can replace the method of searching parameter combinations in the related art, and efficiently and automatically search for optimization solutions for specific parameter combinations (such as the combination of data parallelism, pipeline parallelism and tensor parallelism, the type of processing units executing each pipeline stage, the number of model layers to be trained in each pipeline stage, and the number of model layers to be recalculated in each pipeline stage) for heterogeneous clusters, thereby saving training time and training resources and improving training efficiency.

[0106] like Figure 3 As shown, the embodiment of the present application also provides a system 300 for parallel training of a large model. The system 300 includes: a heterogeneous cluster 301, the heterogeneous cluster 301 includes at least a plurality of processing units of multiple types, the parallelism includes data parallelism, pipeline parallelism and tensor parallelism, and at least some pipeline stages of pipeline parallelism are executed by processing units of different types; and a device 302 for parallel training of heterogeneous clusters of large models. The device 302 includes: a processor 3021; ​​and a memory 3022, on which a computer program that can be executed by the processor 3021 is stored. When the computer program is executed by the processor 3021, the processor 3021 implements the parameter search process according to the embodiment of the present application (the process of searching for the target parameter combination as described above), and after searching for the target parameter combination, the target parameter combination is sent to the heterogeneous cluster 301, so that the heterogeneous cluster 301 performs parallel training on the large model based on the target parameter combination.

[0107] It should be understood that although Figure 3 The device 302 for heterogeneous cluster parallel training of large models is shown as a device independent of the heterogeneous cluster 301 , but in some embodiments, some or all functions of the device 302 can be implemented on any computing device in the heterogeneous cluster 301 .

[0108] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for parallel training of heterogeneous clusters of large models according to an embodiment of the present application.

[0109] An embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the processor implements the method for parallel training of heterogeneous clusters of large models according to an embodiment of the present application.

[0110] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0112] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] As used herein, the term "machine-readable medium" refers to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0117] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0119] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for heterogeneous cluster parallel training of a large model, wherein the heterogeneous cluster includes at least a plurality of processing units of a plurality of types, wherein the parallelism includes data parallelism, pipeline parallelism, and tensor parallelism, wherein at least some pipeline stages of the pipeline parallelism are executed by processing units of different types, and wherein the method comprises: Acquire first input information and second input information, wherein the first input information represents model structure parameters of the large model to be trained, and the second input information represents device parameters of the heterogeneous cluster for training the large model; Determining a parameter search space according to the first input information and the second input information; According to a preset search rule, searching for a target parameter combination in the parameter search space; and The heterogeneous cluster performs parallel training on the large model based on the target parameter combination, The target parameter combination is a parameter combination that minimizes the training cost among the multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform a round of training on the large model based on the corresponding parameter combination. The total time is the sum of the time for performing a pipeline parallel training based on the corresponding parameter combination, the time for communicating the gradient data set, and the time for the optimizer of the large model to update the model parameters of the large model.

2. The method according to claim 1, wherein: The parameter search space includes a parallelism combination scheme, a processing unit allocation scheme, a pipeline segmentation scheme, and a recalculation layer number setting scheme. Among them, the parallelism combination scheme represents the optional combination of data parallelism, pipeline parallelism and tensor parallelism, the processing unit allocation scheme represents the optional type of processing unit that executes each pipeline stage, the pipeline segmentation scheme represents the number of model layers to be trained in each pipeline stage, and the recalculation layer setting scheme represents the number of model layers to be recalculated in each pipeline stage, and Among them, the parameter combination includes: a combination of data parallelism, pipeline parallelism and tensor parallelism, the type of processing unit executing each pipeline stage, the number of model layers to be trained in each pipeline stage, and the number of model layers to be recalculated in each pipeline stage.

3. The method according to claim 1, wherein: When using T pp It represents the time of a pipeline parallel training based on the corresponding parameter combination, and T all_reduce When representing the time of gradient data set communication, T pp Determined based on the following formula: T pp =t edge +t middle Among them, t edge represents the total time of executing only forward computation and only backward computation in all pipeline stages during pipeline parallel training, t middle represents the time to interleave forward and backward computations in all pipeline stages during pipeline parallel training, t fi represents the time for the i-th pipeline stage to perform forward calculation on a set of input data, t bi represents the time it takes for the i-th pipeline stage to perform the reverse calculation for a set of input data, S represents the number of pipeline stages, and S max represents the time and maximum pipeline number of forward and reverse calculations for a set of input data in all pipeline stages, num_mbs represents the number of input data groups for each pipeline stage, and T all_reduce Determined based on the following formula: Among them, data represents the amount of data that needs to be transmitted for gradient data set communication between data parallel groups, bandwidth represents the data transmission bandwidth of gradient data set communication, and layer s i It represents the number of model layers to be executed in the i-th pipeline stage, micro_batch_size represents the size of a set of input data for each pipeline stage, and gradient represents the gradient.

4. The method according to claim 2, wherein: The preset search rules include: Determine the parallelism combination scheme among all parallelism combinations according to the restriction of the parameter search space by the large model by using a depth-first search algorithm; Using a dynamic programming algorithm, constructing a state transfer equation according to the training cost; The iterative parameters of the state transfer equation are traversed to obtain the target parameter combination that minimizes the training cost.

5. The method according to claim 4, wherein: The preset search rules also include: During the search, at least one of heterogeneous communication pruning, load balancing pruning and recomputation pruning is performed. In the heterogeneous communication pruning, when the time for point-to-point communication between different types of processing units is greater than the forward calculation time of any pipeline stage in a pipeline segmentation scheme, the search branch corresponding to the pipeline segmentation scheme is pruned. In the load balancing pruning, a threshold of the number of model layers to be trained in each pipeline stage is set according to the computing power of the processing unit executing each pipeline stage. When the number of model layers allocated to any pipeline stage in a pipeline segmentation scheme is greater than or equal to the threshold, the search branch corresponding to the pipeline segmentation scheme is pruned. In the recomputation pruning, the target number of recomputation layers for the pipeline stage is calculated based on the number of model layers to be trained in the pipeline stage, the pipeline parallelism, and the video memory of the processing unit. When a recomputation layer setting scheme does not meet the target number of recomputation layers, the search branch corresponding to the recomputation layer setting scheme is pruned.

6. The method according to claim 3, wherein: The gradient data set communication uses the AllReduce algorithm, and The pipeline parallelism includes pipeline parallelism in the form of PipeDream 1F1B.

7. A device for heterogeneous cluster parallel training of a large model, wherein the heterogeneous cluster includes at least a plurality of processing units of a plurality of types, wherein the parallelism includes data parallelism, pipeline parallelism, and tensor parallelism, wherein at least some pipeline stages of the pipeline parallelism are executed by processing units of different types, and wherein the device comprises: processor; as well as A memory having a computer program executable by the processor stored therein, which, when executed by the processor, causes the processor to implement the following operations: Acquire first input information and second input information, wherein the first input information represents model structure parameters of the large model to be trained, and the second input information represents device parameters of the heterogeneous cluster for training the large model; Determining a parameter search space according to the first input information and the second input information; According to a preset search rule, searching for a target parameter combination in the parameter search space; and Sending the target parameter combination to the heterogeneous cluster so that the heterogeneous cluster performs parallel training on the large model based on the target parameter combination, The target parameter combination is a parameter combination that minimizes the training cost among the multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform a round of training on the large model based on the corresponding parameter combination. The total time is the sum of the time for performing a pipeline parallel training based on the corresponding parameter combination, the time for communicating the gradient data set, and the time for the optimizer of the large model to update the model parameters of the large model.

8. A system for parallel training of a large model, comprising: A heterogeneous cluster, wherein the heterogeneous cluster includes at least a plurality of processing units of a plurality of types, wherein the parallelism includes data parallelism, pipeline parallelism and tensor parallelism, and at least some pipeline stages of the pipeline parallelism are executed by processing units of different types; as well as A device for heterogeneous cluster parallel training of a large model, the device comprising: Processor; and A memory having a computer program executable by the processor stored therein, which, when executed by the processor, causes the processor to implement the following operations: Acquire first input information and second input information, wherein the first input information represents model structure parameters of the large model to be trained, and the second input information represents device parameters of the heterogeneous cluster for training the large model; Determining a parameter search space according to the first input information and the second input information; According to a preset search rule, searching for a target parameter combination in the parameter search space; and Sending the target parameter combination to the heterogeneous cluster so that the heterogeneous cluster performs parallel training on the large model based on the target parameter combination, The target parameter combination is a parameter combination that minimizes the training cost among the multiple parameter combinations included in the parameter search space, and the training cost represents the total time required for the heterogeneous cluster to perform a round of training on the large model based on the corresponding parameter combination. The total time is the sum of the time for performing a pipeline parallel training based on the corresponding parameter combination, the time for communicating the gradient data set, and the time for the optimizer of the large model to update the model parameters of the large model.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 6.

10. A computer program product, comprising computer instructions, which, when executed by a processor, cause the processor to implement the method according to any one of claims 1 to 6.

Images