Parallel training method and device for transformer model
By constructing a parallel training strategy search space and decision tree, and combining it with a dynamic programming search algorithm to select the optimal combination of parallel training strategies, the problem of high computational and memory overhead in large-scale Transformer model training is solved, thus improving training efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PEKING UNIV
- Filing Date
- 2022-11-17
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies incur huge computational and memory overhead in training large-scale Transformer models, and existing automatic parallelization methods only support a limited parallel dimension, resulting in low training efficiency.
By constructing a parallel training strategy search space, using M preset parallel training strategies, constructing a decision tree, determining the set of parallel training strategies, and using a dynamic programming search algorithm to select the target parallel training strategy combination with the highest throughput for training.
It improves the training efficiency of the Transformer model, reduces the range of parallel training strategies, identifies the strategy combination with the highest throughput, and optimizes the training process.
Smart Images
Figure CN116128019B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a parallel training method and apparatus for a Transformer model. Background Technology
[0002] With the development of artificial intelligence technology, the Transformer model has been widely used in fields such as computer vision (CV), natural language processing (NLP), graph learning, and recommender systems.
[0003] In practical applications, increasing the size of model parameters can effectively improve model performance. For example, the second-generation generative pre-trained Transformer (GPT-2) model with 1.5 billion model parameters performs far better than the original Transformer with only one million model parameters. However, such a large number of model parameters results in huge computational and memory overhead for model training.
[0004] Therefore, how to efficiently train Transformers has become an important issue that the industry urgently needs to address. Summary of the Invention
[0005] To address the problems existing in the prior art, this invention provides a parallel training method and apparatus for Transformer models.
[0006] This invention provides a parallel training method for Transformer models, comprising:
[0007] Based on M preset parallel training strategies, the search space for parallel training strategies of the initial Transformer model is determined; the search space for parallel training strategies includes N1 combinations of parallel training strategies, and each combination of parallel training strategies includes N2 preset parallel training strategies; M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M.
[0008] Based on each of the preset parallel training strategies, at least one decision tree is constructed; each decision tree is used to determine the set of parallel training strategies for the initial Transformer model from the search space of the parallel training strategies; the set of parallel training strategies includes N3 combinations of parallel training strategies, where N3 is a positive integer greater than or equal to 1 and N3 is less than N1.
[0009] Based on the set of parallel training strategies, a target combination of parallel training strategies is determined; the target combination of parallel training strategies is the combination of parallel training strategies with the highest throughput in the set of parallel training strategies.
[0010] The initial Transformer model is trained using the target parallel training strategy combination to obtain the target Transformer model.
[0011] Optionally, the preset parallel training strategy includes at least one of the following:
[0012] Data-parallel training strategy;
[0013] Parallel training strategy using fragmented data;
[0014] Tensor parallel training strategy;
[0015] Streamlined parallel training strategy.
[0016] Optionally, constructing at least one decision tree based on each of the preset parallel training strategies includes:
[0017] Using the pipelined parallel training strategy, the initial Transformer model is divided into multiple Transformer sub-models; each Transformer sub-model corresponds to at least one device group, and the device group includes at least one graphics processor device.
[0018] For each Transformer sub-model, a decision tree corresponding to the Transformer sub-model is constructed based on the decision tree construction rules;
[0019] The decision tree construction rules include at least one of the following:
[0020] The root node of the decision tree is the data parallel training strategy;
[0021] Each layer of the decision tree represents any one of the data parallel training strategy, the sharded data parallel training strategy, and the tensor parallel training strategy.
[0022] The preset parallel training strategies differ between the layers of the decision tree;
[0023] The degree of the non-leaf nodes in the decision tree is a power of 2;
[0024] The total number of decision tree leaf nodes is equal to the number of graphics processor devices in each device group.
[0025] Optionally, the method further includes:
[0026] When the data parallel training strategy and the sharded data parallel training strategy exist in the same decision tree, the decision tree is pruned based on the data parallel training strategy and the sharded data parallel training strategy.
[0027] Optionally, determining the target parallel training strategy combination based on the training strategy set includes:
[0028] For each Transformer sub-model, a dynamic programming search algorithm is used to determine a first parallel training strategy combination for each Transformer sub-model from the set of parallel training strategies; the first parallel training strategy combination is the parallel training strategy with the minimum time cost.
[0029] Among each of the first parallel training strategy combinations, the parallel training strategy combination with the highest throughput is determined as the target parallel training strategy combination.
[0030] Optionally, the dynamic programming search algorithm is expressed by the following formula (1):
[0031]
[0032] Where L represents the number of layers in the Transformer sub-model; E represents the memory limit of the graphics processing unit; S represents the set of training strategies; S j The parallel training strategy used in the Lth layer of the Transformer sub-model is represented by C(L,E); the total execution time of the Lth layer of the Transformer sub-model under the memory constraints of the graphics processing unit is represented by O(L,S). j The statement indicates that the Lth layer in the Transformer sub-model uses a parallel training strategy S. j Memory overhead; c(L,S) j The statement indicates that the Lth layer in the Transformer sub-model uses a parallel training strategy S. j Execution time; R(L,S) i ,S j The statement indicates that the Lth layer in the Transformer sub-model uses a parallel training strategy S. j The L-1 layer uses a parallel training strategy S i The resulting conversion overhead.
[0033] The present invention also provides a parallel training apparatus for a Transformer model, comprising:
[0034] The first determining module is used to determine the parallel training strategy search space of the initial Transformer model based on M preset parallel training strategies; the parallel training strategy search space includes N1 combinations of parallel training strategies, and each combination of parallel training strategies includes N2 preset parallel training strategies; M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M.
[0035] The construction module is used to construct at least one decision tree based on each of the preset parallel training strategies; each decision tree is used to determine the set of parallel training strategies for the initial Transformer model from the search space of the parallel training strategies; the set of parallel training strategies includes N3 combinations of parallel training strategies, where N3 is a positive integer greater than or equal to 1 and N3 is less than N1.
[0036] The second determining module is used to determine a target parallel training strategy combination based on the set of parallel training strategies; the target parallel training strategy combination is the parallel training strategy combination with the highest throughput in the set of parallel training strategies.
[0037] The training module is used to train the initial Transformer model using the target parallel training strategy combination to obtain the target Transformer model.
[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a parallel training method for the Transformer model as described above.
[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a parallel training method for the Transformer model as described above.
[0040] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a parallel training method for the Transformer model as described above.
[0041] The parallel training method and apparatus for Transformer models provided by this invention expands the search space of parallel training strategies for the initial Transformer model by combining various preset parallel strategies, thereby providing more optional combinations of parallel training strategies for the initial Transformer model. Then, based on each preset combination of parallel training strategies, at least one decision tree is constructed to determine the set of parallel training strategies for the initial Transformer model from the search space. The search space is decomposed based on each decision tree, reducing the selection range of parallel training strategy combinations within the search space, thus determining the set of parallel training strategies for the initial Transformer model. Based on the set of training strategies, the target parallel training strategy combination with the highest throughput can be determined. The initial Transformer model is then trained using this target parallel training strategy combination with the highest throughput, improving the training efficiency of the Transformer model. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0043] Figure 1 This is one of the flowcharts illustrating the parallel training method for the Transformer model provided by this invention;
[0044] Figure 2 This is the second flowchart illustrating the parallel training method for the Transformer model provided by this invention.
[0045] Figure 3 This is a schematic diagram of the structure of the parallel training device for the Transformer model provided by the present invention;
[0046] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0048] To facilitate a clearer understanding of the various embodiments of this application, some relevant knowledge will be introduced as follows.
[0049] Because large-scale model parameters result in huge computational and memory overhead for model training, building and designing large-scale Transformer models places higher demands on system optimization, and how to efficiently train Transformers is becoming increasingly challenging.
[0050] Existing distributed deep learning systems typically use multiple Graphics Processing Unit (GPU) devices to perform distributed training using parallel training strategies; these parallel training strategies are as follows:
[0051] a) Data parallelism;
[0052] Specifically, the data-parallel training strategy divides a large amount of input data into multiple parts, with each GPU device handling only a portion of the data, and gradients need to be synchronized between GPU devices. Data parallelism requires each GPU device to store a complete copy of the model, thus it is easily limited by the size of large models;
[0053] b) Model parallelism;
[0054] Model parallelism divides the model into multiple parts, so that each GPU device stores a subset of model parameters. Depending on the partitioning dimension, there are two main model parallel training strategies: tensor parallelism and pipeline parallelism.
[0055] Tensor parallelism refers to vertically splitting the fully connected layers and attention layers in the Transformer model across multiple GPU devices and inserting communication operators (All-Reduce) to ensure computational consistency.
[0056] Pipeline parallelism refers to treating the Transformer model as a continuous layer and horizontally dividing the model into several pipeline stages containing continuous layers. Each GPU device is responsible for the computation of one pipeline stage, and the devices are organized into a pipeline to transmit intermediate computation results.
[0057] c) Automatic parallel training strategy;
[0058] Specifically, the automatic parallel training strategy refers to automatically finding parallel strategies by combining data parallelism and model parallelism for each operator in the Transformer model with fine granularity. The background of the automatic parallel training strategy is as follows.
[0059] When training large-scale Transformer models in a distributed manner, the choice of various parallel strategies, such as data parallelism, tensor parallelism, and pipelined parallelism, leads to different memory consumption, communication overhead, and execution efficiency. Since no single parallel technique can always achieve optimal training efficiency under different hardware conditions (e.g., different communication bandwidth, computational efficiency, GPU memory capacity) and different model configurations (e.g., different sequence lengths, number of layers, hidden layer widths, etc.), directly applying these single parallel techniques to large-scale Transformer model training faces significant challenges in terms of system efficiency and availability.
[0060] To address these issues, researchers have proposed more advanced automatic parallelization methods that combine data parallelism and model parallelism to search for better distributed training strategies. However, existing automatic parallelization methods only support a limited parallelism dimension (i.e., data parallelism and at most one model parallelism dimension). This limited decision space cannot provide sufficiently efficient parallelism strategies for many training scenarios, which is a limitation of existing automatic parallel training strategies.
[0061] In summary, to address the problems existing in the above-mentioned technologies and to efficiently train large-scale Transformer models and improve the training efficiency of Transformer models, this invention provides a parallel training method and apparatus for Transformer models.
[0062] The following is combined with Figures 1-2 The data transmission method provided by this invention will be described in detail. Figure 1 This is one of the flowcharts illustrating the parallel training method for the Transformer model provided by this invention. See [link / reference]. Figure 1 As shown, steps 101-104 are included, wherein:
[0063] Step 101: Based on M preset parallel training strategies, determine the parallel training strategy search space for the initial Transformer model; the parallel training strategy search space includes N1 combinations of parallel training strategies, and each combination of parallel training strategies includes N2 preset parallel training strategies; M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M.
[0064] First, it should be noted that the subject of this invention can be any electronic device capable of parallel training of the Transformer model, such as a smartphone, smartwatch, desktop computer, laptop, etc.
[0065] In order to efficiently train large-scale Transformer models and improve the training efficiency of Transformer models, in this embodiment, it is first necessary to determine the search space of parallel training strategies for the initial Transformer model based on at least one preset parallel training strategy.
[0066] Optionally, the preset parallel training strategy includes at least one of the following:
[0067] a) Data parallelism (DP) training strategy;
[0068] b) Sharded data parallelism (SDP) strategy;
[0069] c) Tensor parallelism (TP) training strategy;
[0070] d) Pipeline parallelism (PP) training strategy.
[0071] It should be noted that the parallel training strategy search space includes any combination of preset parallel training strategies. In scenarios with multiple GPU devices, the training strategy search space generated by such combinations is extremely large.
[0072] For example, in a dual-machine, four-card scenario, one feasible training strategy is to use 2-way TP within the machine and 2-way PP between the machines; another feasible training strategy is to use 2-way PP within the machine and 2-way DP between the machines.
[0073] When the number of GPU devices is expanded to 8 cards, there are hundreds of candidate training strategies for each layer of the Transformer model. For a given Transformer model, the size of its parallel training strategy search space grows exponentially with the number of layers. Therefore, the training strategy search space is very large in the scenario of multiple GPU devices.
[0074] In the above implementation, by combining the preset parallel strategies, the search space for parallel training strategies of the initial Transformer model can be expanded, thereby providing more optional combinations of parallel training strategies for the initial Transformer model.
[0075] Step 102: Based on each of the preset parallel training strategies, construct at least one decision tree; each decision tree is used to determine the set of parallel training strategies for the initial Transformer model from the search space of the parallel training strategies; the set of parallel training strategies includes N3 combinations of parallel training strategies, where N3 is a positive integer greater than or equal to 1 and N3 is less than N1.
[0076] In this embodiment, since the search space for parallel training strategies determined based on each preset parallel training strategy is very large, in order to efficiently generate the best combination of parallel training strategies for a given Transformer model and distributed environment within the complex and large search space for parallel training strategies, it is necessary to construct at least one decision tree based on each preset parallel training strategy to determine the set of parallel training strategies for the initial Transformer model from the search space for parallel training strategies. The decision tree is used to decompose the search space for parallel training strategies.
[0077] It should be noted that the following empirical guidelines are followed when constructing decision trees:
[0078] Guidance 1) PP tends to be placed across device islands
[0079] Specifically, a device island refers to a group of GPU devices with high internal bandwidth. Since PPs have significantly less communication traffic compared to other parallel processing methods, it is reasonable to place PPs among device islands with low communication bandwidth after horizontally partitioning the Transformer model.
[0080] Guideline 2) In scenarios with homogeneous devices, parallel training strategies tend to divide the GPU device evenly.
[0081] For example, for a 2-way DP with 4 GPUs, it is preferable to split the GPU devices into two groups of 2-card devices rather than one group of 1-card devices and one group of 3-card devices. Therefore, the optimal hybrid parallel training strategy within a device group is the same as the optimal parallel training strategy within other groups.
[0082] Guideline 3) Using SDP alone is better than using DP and SDP together.
[0083] Specifically, theoretical analysis shows that the communication overhead and memory overhead of N-way SDP are both better than the combination of N1-way DP and N2-way SDP, where N = N1 × N2.
[0084] Step 103: Based on the set of parallel training strategies, determine the target combination of parallel training strategies; the target combination of parallel training strategies is the combination of parallel training strategies with the highest throughput in the set of parallel training strategies.
[0085] In this embodiment, after determining the set of parallel training strategies for the initial Transformer model from the parallel training strategy search space based on each decision tree, it is necessary to determine the target parallel training strategy combination with the highest throughput from the set of parallel training strategies.
[0086] Step 104: Train the initial Transformer model using the target parallel training strategy combination to obtain the target Transformer model.
[0087] The parallel training method for Transformer models provided by this invention expands the search space for parallel training strategies of the initial Transformer model by combining various preset parallel strategies, thereby providing more optional combinations of parallel training strategies for the initial Transformer model. Then, based on the combinations of preset parallel training strategies, at least one decision tree is constructed to determine the set of parallel training strategies for the initial Transformer model from the search space. The search space is decomposed based on each decision tree, reducing the selection range of parallel training strategy combinations in the search space, thereby determining the set of parallel training strategies for the initial Transformer model. Based on the set of training strategies, the target parallel training strategy combination with the highest throughput can be determined. The initial Transformer model is trained using the target parallel training strategy combination with the highest throughput, improving the training efficiency of the Transformer model.
[0088] Optionally, the construction of at least one decision tree based on each of the preset parallel training strategies is specifically achieved through the following steps 1)-2):
[0089] Step 1) Using the pipelined parallel training strategy, the initial Transformer model is divided into multiple Transformer sub-models; each Transformer sub-model corresponds to at least one device group, and the device group includes at least one graphics processor device;
[0090] Step 2) For each Transformer sub-model, construct a decision tree corresponding to the Transformer sub-model based on the decision tree construction rules;
[0091] The decision tree construction rules include at least one of the following:
[0092] a) The root node of the decision tree is the data parallel training strategy;
[0093] b) Each layer of the decision tree represents any one of the data parallel training strategy, the sharded data parallel training strategy, and the tensor parallel training strategy; it should be noted that the height of the decision tree represents the number of available parallel training strategies.
[0094] c) The preset parallel training strategies are different between the layers of the decision tree; that is, the parallel training strategies cannot be reused between different layers of the decision tree.
[0095] d) The degree of the non-leaf nodes in the decision tree is a power of 2; for example, the degree of the non-leaf nodes should be selected from powers of 2 {2, 4, 8...}.
[0096] e) The total number of decision leaf nodes is the number of graphics processor devices in each of the device groups.
[0097] Specifically, for a given initial Transformer model, based on the guidance 1) and guidance 2) mentioned above, the initial Transformer model is first divided into multiple stages using PP to obtain multiple Transformer sub-models, that is, one stage corresponds to one Transformer sub-model.
[0098] Meanwhile, multiple GPU devices are evenly and continuously divided into multiple device groups. Each Transformer sub-model corresponds to at least one device group, and each device group includes at least one graphics processor device.
[0099] For example, in an 8-card scenario, the initial Transformer model is divided into 1 / 2 / 4 / 8-way PPs, which correspond to device group sizes of 8 / 4 / 2 / 1 respectively.
[0100] It should be noted that each PP splitting type corresponds to a decision tree and a sub-search space, and all sub-search spaces together constitute the parallel training policy search space.
[0101] After splitting the initial Transformer model into multiple Transformer sub-models, a decision tree corresponding to each Transformer sub-model is constructed based on the decision tree construction rules.
[0102] In the above implementation, the initial Transformer model is divided into multiple Transformer sub-models using a pipelined parallel training strategy. Then, for each Transformer sub-model, a decision tree corresponding to the Transformer sub-model is constructed based on decision tree construction rules. Each decision tree can represent any combination of the above parallel training strategies without overlap or omission. In other words, the search space for parallel training strategies is decomposed based on each decision tree, reducing the range of parallel training strategies to be selected in the search space, thereby determining the set of parallel training strategies for the initial Transformer model.
[0103] Optionally, as can be seen from the above-mentioned guidance 3), using SDP alone is better than using DP and SDP together. Therefore, when the data parallel training strategy and the sharded data parallel training strategy exist in the same decision tree, the decision tree is pruned based on the data parallel training strategy and the sharded data parallel training strategy.
[0104] In practical applications, for scenarios where an 8-GPU model is trained to a single layer, based on the above decision tree construction rules, 34 candidate parallel training strategies will be generated.
[0105] Then, using guideline 3) to prune the case where DP and SDP appear in the same decision tree, the number of 8 candidate parallel training strategies is reduced to 22.
[0106] In the above implementation, when the data parallel training strategy and the sharded data parallel training strategy exist in the same decision tree, pruning the decision tree based on the data parallel training strategy and the sharded data parallel training strategy can effectively avoid inefficient combinations of parallel training strategies, further reduce the selection range of parallel training strategies in the search space of parallel training strategies, and thus determine a more accurate set of parallel training strategies.
[0107] Optionally, after determining the set of parallel training strategies for the initial Transformer model from the parallel training strategy search space based on each decision tree, it is necessary to determine the target parallel training strategy with the highest throughput from the set of parallel training strategies. This can be achieved through the following steps [1]-[2]:
[0108] Step [1]: For each Transformer sub-model, use a dynamic programming search algorithm to determine the first parallel training strategy combination for each Transformer sub-model from the set of parallel training strategies; the first parallel training strategy combination is the parallel training strategy with the least time overhead.
[0109] Step [2]: Among each of the first parallel training strategy combinations, the parallel training strategy combination with the highest throughput is determined as the target parallel training strategy combination.
[0110] In this embodiment, given an L-layer Transformer sub-model M and N GPUs with a memory capacity of E, it is necessary to determine the optimal target parallel training strategy combination as the parallel training strategy combination with the highest throughput. Here, throughput is equal to the ratio of batch size to execution time. Since different parallel training strategies have different memory, computation, and communication overheads at different batch sizes, the target parallel training strategy combination with the highest throughput may not necessarily appear at the largest batch size; therefore, it is necessary to iterate through the batch sizes.
[0111] Specifically, after using PP to split the initial Transformer model and generate P stages (i.e. P Transformer sub-models), the GPU devices corresponding to the initial Transformer model are also split into P groups, with each group containing N / P GPU devices.
[0112] Then, for each Transformer sub-model, a decision tree corresponding to each Transformer sub-model is constructed based on the decision tree construction rules. Since the decision tree can represent any combination of DP, SDP, and TP without repetition or omission, the set of parallel training strategies S for the initial Transformer model can be determined from the parallel training strategy search space based on each decision tree.
[0113] After obtaining the set of parallel training strategies S, for each Transformer sub-model, a dynamic programming search algorithm is used to determine the first combination of parallel training strategies with the minimum time cost for each Transformer sub-model from the set of parallel training strategies.
[0114] Finally, among the various first parallel training strategy combinations, the one with the highest throughput is determined as the target parallel training strategy.
[0115] Optionally, the dynamic programming search algorithm is expressed by the following formula (1):
[0116]
[0117] Where L represents the number of layers in the Transformer sub-model; E represents the memory limit of the graphics processing unit; S represents the set of training strategies; S jThe parallel training strategy used in the Lth layer of the Transformer sub-model is represented by C(L,E); the total execution time of the Lth layer of the Transformer sub-model under the memory constraints of the graphics processing unit is represented by O(L,S). j The statement indicates that the Lth layer in the Transformer sub-model uses a parallel training strategy S. j Memory overhead; c(L,S) j The statement indicates that the Lth layer in the Transformer sub-model uses a parallel training strategy S. j Execution time; R(L,S) i ,S j The statement indicates that the Lth layer in the Transformer sub-model uses a parallel training strategy S. j The L-1 layer uses a parallel training strategy S i The resulting transition overhead; where C(L,E) is the overhead function, with initial values C(0,.)=0 and C(.,0)=∞; during state transitions, if the memory overhead exceeds the device memory limit E, the overhead function C returns to infinity.
[0118] It should be noted that the size of the sub-search space of each layer of the Transformer sub-model is very important for the search complexity of the overall parallel training strategy. The search space decomposition based on decision trees can significantly reduce the search space and control the search overhead within a reasonable range.
[0119] In one possible implementation of this invention, the Lth layer of the Transformer sub-model uses a parallel training strategy S. j The memory overhead can be determined based on the shape and data type of the tensor.
[0120] In the Transformer submodel, the Lth layer uses a parallel training strategy. j The execution time can be obtained in the following ways:
[0121] First, it is necessary to use the parallel training strategy S j The computation time and communication overhead are estimated.
[0122] 1) Regarding calculation time:
[0123] The total computation time needs to be simulated based on the product of batch size and per-sample computation time, where per-sample computation time can be measured by profiling on a single device; in addition, since the Transformer model is mainly composed of matrix multiplication, its backpropagation computation time is usually twice that of forward computation.
[0124] 2) Regarding communication overhead:
[0125] The estimated communication overhead is obtained by dividing the communication volume by the GPU device's communication bandwidth, where the communication volume is obtained by theoretical calculation and the communication bandwidth is obtained by profiling measurement.
[0126] Then, based on computation time and communication overhead, the parallel training strategy S for the Lth layer in the Transformer sub-model was determined. j Execution time c(L,S) j It consists of two steps: forward propagation and backward propagation.
[0127] For forward propagation, the estimated computation time and communication overhead (e.g., all-gather in SDP and all-reduce in TP) are added together.
[0128] For backpropagation, the overlap between computation time and communication overhead is taken into account, as well as the GPU performance degradation caused by this overlap. Backpropagation takes the maximum value of computation time and communication overhead, which makes the overhead estimation more accurate and the parallel optimization effect better.
[0129] Optionally, in one possible implementation of the present invention, the parallel training method for the Transformer model provided by the present invention can be applied to a layer-by-layer hybrid parallel large-scale Transformer model training framework, which allows the model to use different parallel training strategies for each layer.
[0130] It should be noted that, since the Transformer model training framework supports complex parallel training strategies, different parallel training strategies need to communicate within different GPU device groups, which makes the number of communication groups very large. Therefore, it is necessary to maintain a global communication group pool, and all communication groups are created before training and shared by all layers.
[0131] This Transformer model training framework can switch between parallel training strategies between layers, thus supporting different parallel training strategies for different layers.
[0132] In practical applications, when the previous layer uses parallel training strategy A and the current layer uses parallel training strategy B, it is necessary to ensure that the input activation values of the current layer are placed on the correct GPU device according to the requirements of parallel training strategy B.
[0133] Specifically, the output activation values of the previous layer need to be divided into several parts and redundant parts need to be removed according to the requirements of the parallel training strategy B; then all-gather communication is used to allow each GPU device to collect the activation values needed from other GPU devices.
[0134] In the above implementation, for each Transformer sub-model, a dynamic programming search algorithm is used to determine the first parallel training strategy combination with the minimum time cost for each Transformer sub-model from the set of parallel training strategies. Then, among the first parallel training strategy combinations, the parallel training strategy combination with the highest throughput is determined as the target parallel training strategy combination. Through the above method, based on the set of training strategies, the target parallel training strategy combination with the highest throughput in the set of parallel training strategies can be determined. The initial Transformer model is trained using the target parallel training strategy combination with the highest throughput, thereby improving the training efficiency of the Transformer model.
[0135] Figure 2 This is the second flowchart illustrating the parallel training method for the Transformer model provided by this invention. See also... Figure 2 As shown, steps 201-207 are included, wherein:
[0136] Step 201: Based on M preset parallel training strategies, determine the parallel training strategy search space of the initial Transformer model; wherein, the parallel training strategy search space includes N1 combinations of parallel training strategies, and each combination of parallel training strategies includes N2 preset parallel training strategies; M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M.
[0137] Specifically, the preset parallel training strategies include at least one of the following: a) data parallel training strategy; b) sharded data parallel training strategy; c) tensor parallel training strategy; d) pipelined parallel training strategy.
[0138] Step 202: Using a pipelined parallel training strategy, the initial Transformer model is split into multiple Transformer sub-models; each Transformer sub-model corresponds to at least one device group, wherein the device group includes at least one graphics processor device.
[0139] Step 203: For each Transformer sub-model, construct a decision tree corresponding to the Transformer sub-model based on the decision tree construction rules. Each decision tree is used to determine the set of parallel training strategies for the initial Transformer model from the parallel training strategy search space. The set of parallel training strategies includes N3 combinations of parallel training strategies, where N3 is a positive integer greater than or equal to 1 and less than N1.
[0140] Specifically, the decision tree construction rules include at least one of the following:
[0141] a) The root node of the decision tree is a data-parallel training strategy;
[0142] b) Each layer of the decision tree represents any one of the following strategies: parallel data training strategy, sharded data parallel training strategy, and tensor parallel training strategy.
[0143] c) The pre-set parallel training strategies differ between the layers of the decision tree;
[0144] d) The degree of a non-leaf node in a decision tree is a power of 2;
[0145] e) The total number of decision leaf nodes is the number of graphics processor devices in each device group.
[0146] Step 204: When the data parallel training strategy and the sharded data parallel training strategy exist in the same decision tree, perform pruning on the decision tree based on the data parallel training strategy and the sharded data parallel training strategy.
[0147] Step 205: For each Transformer sub-model, use a dynamic programming search algorithm to determine the first parallel training strategy combination for each Transformer sub-model from the set of parallel training strategies.
[0148] Specifically, the first parallel training strategy combination is the parallel training strategy combination with the minimum time cost.
[0149] Step 206: Among the first parallel training strategy combinations, the parallel training strategy combination with the highest throughput is determined as the target parallel training strategy combination.
[0150] Step 207: Train the initial Transformer model using a combination of target parallel training strategies to obtain the target Transformer model.
[0151] The parallel training method for Transformer models provided by this invention expands the search space of parallel training strategies for the initial Transformer model by combining various preset parallel strategies, thereby providing more optional combinations of parallel training strategies for the initial Transformer model. Then, based on these preset combinations of parallel training strategies, at least one decision tree is constructed to determine the set of parallel training strategies for the initial Transformer model from the search space. The search space is then decomposed based on these decision trees, reducing the range of choices for combinations of parallel training strategies within the search space, thereby determining the initial Transformer model. The set of parallel training strategies is provided. Furthermore, when both the data-parallel training strategy and the sharded data-parallel training strategy exist within the same decision tree, pruning the decision tree based on these strategies effectively avoids inefficient combinations of parallel training strategies, further reducing the selection range of parallel training strategy combinations in the search space, and thus determining a more accurate set of parallel training strategies. Based on this set of training strategies, the target parallel training strategy combination with the highest throughput can be identified. Using this target parallel training strategy combination with the highest throughput to train the initial Transformer model improves the training efficiency of the Transformer model.
[0152] The parallel training apparatus for the Transformer model provided by the present invention will be described below. The parallel training apparatus for the Transformer model described below can be referred to in correspondence with the parallel training method for the Transformer model described above. Figure 3 This is a schematic diagram of the structure of the parallel training device for the Transformer model provided by the present invention, as shown below. Figure 3 As shown, the parallel training device 300 for the Transformer model includes: a first determination module 301, a construction module 302, a second determination module 303, and a training module 304, wherein:
[0153] The first determining module 301 is used to determine the parallel training strategy search space of the initial Transformer model based on M preset parallel training strategies; the parallel training strategy search space includes N1 combinations of parallel training strategies, and each combination of parallel training strategies includes N2 preset parallel training strategies; M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M.
[0154] The construction module 302 is used to construct at least one decision tree based on each of the preset parallel training strategies; each decision tree is used to determine the set of parallel training strategies for the initial Transformer model from the search space of the parallel training strategies; the set of parallel training strategies includes N3 combinations of parallel training strategies, where N3 is a positive integer greater than or equal to 1 and N3 is less than N1.
[0155] The second determining module 303 is used to determine a target parallel training strategy combination based on the set of parallel training strategies; the target parallel training strategy combination is the parallel training strategy combination with the highest throughput in the set of parallel training strategies.
[0156] The training module 304 is used to train the initial Transformer model using the target parallel training strategy combination to obtain the target Transformer model.
[0157] The parallel training apparatus for the Transformer model provided by this invention expands the search space for parallel training strategies of the initial Transformer model by combining various preset parallel strategies, thereby providing more optional combinations of parallel training strategies for the initial Transformer model. Then, based on the combinations of preset parallel training strategies, at least one decision tree is constructed to determine the set of parallel training strategies for the initial Transformer model from the search space. The search space is decomposed based on each decision tree, reducing the selection range of parallel training strategy combinations in the search space, thereby determining the set of parallel training strategies for the initial Transformer model. Based on the set of training strategies, the target parallel training strategy combination with the highest throughput can be determined, and the initial Transformer model is trained using the target parallel training strategy combination with the highest throughput, thereby improving the training efficiency of the Transformer model.
[0158] Optionally, the preset parallel training strategy includes at least one of the following:
[0159] Data-parallel training strategy;
[0160] Parallel training strategy using fragmented data;
[0161] Tensor parallel training strategy;
[0162] Streamlined parallel training strategy.
[0163] Optionally, module 302 is further used for:
[0164] Using the pipelined parallel training strategy, the initial Transformer model is divided into multiple Transformer sub-models; each Transformer sub-model corresponds to at least one device group, and the device group includes at least one graphics processor device.
[0165] For each Transformer sub-model, a decision tree corresponding to the Transformer sub-model is constructed based on the decision tree construction rules;
[0166] The decision tree construction rules include at least one of the following:
[0167] The root node of the decision tree is the data parallel training strategy;
[0168] Each layer of the decision tree represents any one of the data parallel training strategy, the sharded data parallel training strategy, and the tensor parallel training strategy.
[0169] The preset parallel training strategies differ between the layers of the decision tree;
[0170] The degree of the non-leaf nodes in the decision tree is a power of 2;
[0171] The total number of decision tree leaf nodes is equal to the number of graphics processor devices in each device group.
[0172] Optionally, the device further includes:
[0173] The processing module is configured to perform pruning on the decision tree based on the data parallel training strategy and the sharded data parallel training strategy when the data parallel training strategy and the sharded data parallel training strategy exist in the same decision tree.
[0174] Optionally, the second determining module 303 is further configured to:
[0175] For each Transformer sub-model, a dynamic programming search algorithm is used to determine a first parallel training strategy combination for each Transformer sub-model from the set of parallel training strategies; the first parallel training strategy combination is the parallel training strategy combination with the minimum time cost; among each of the first parallel training strategy combinations, the parallel training strategy combination with the highest throughput is determined as the target parallel training strategy.
[0176] Optionally, the dynamic programming search algorithm is expressed by the following formula (1):
[0177]
[0178] Where L represents the number of layers in the Transformer sub-model; E represents the memory limit of the graphics processing unit; S represents the set of training strategies; S j The parallel training strategy used in the Lth layer of the Transformer sub-model is represented by C(L,E); the total execution time of the Lth layer of the Transformer sub-model under the memory constraints of the graphics processing unit is represented by O(L,S). j The statement indicates that the Lth layer in the Transformer sub-model uses a parallel training strategy S. j Memory overhead; c(L,S) j The statement indicates that the Lth layer in the Transformer sub-model uses a parallel training strategy S. j Execution time; R(L,S) i ,S j The statement indicates that the Lth layer in the Transformer sub-model uses a parallel training strategy S. j The L-1 layer uses a parallel training strategy S i The resulting conversion overhead.
[0179] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logic instructions in the memory 430 to execute a parallel training method for a Transformer model. This method includes: determining a parallel training strategy search space for an initial Transformer model based on M preset parallel training strategies; the parallel training strategy search space includes N1 combinations of parallel training strategies, each combination including N2 preset parallel training strategies; M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M; constructing at least one decision tree based on each preset parallel training strategy; each decision tree is used to determine a set of parallel training strategies for the initial Transformer model from the parallel training strategy search space; the set of parallel training strategies includes N3 combinations of parallel training strategies, N3 is a positive integer greater than or equal to 1, and N3 is less than N1; determining a target parallel training strategy combination based on the set of training strategies; the target parallel training strategy combination is the parallel training strategy combination with the highest throughput in the set of parallel training strategies; and training the initial Transformer model using the target parallel training strategy combination to obtain a target Transformer model.
[0180] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0181] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the parallel training method for the Transformer model provided by the above methods. The method includes: determining a parallel training strategy search space for an initial Transformer model based on M preset parallel training strategies; the parallel training strategy search space includes N1 combinations of parallel training strategies, and each combination of parallel training strategies includes N2 preset parallel training strategies; M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M. Integer; based on each of the preset parallel training strategies, construct at least one decision tree; each decision tree is used to determine the set of parallel training strategies for the initial Transformer model from the search space of the parallel training strategies; the set of parallel training strategies includes N3 combinations of parallel training strategies, where N3 is a positive integer greater than or equal to 1 and less than N1; based on the set of training strategies, determine the target parallel training strategy combination; the target parallel training strategy combination is the parallel training strategy combination with the highest throughput in the set of parallel training strategies; train the initial Transformer model using the target parallel training strategy combination to obtain the target Transformer model.
[0182] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a parallel training method for the Transformer model provided by the methods described above. This method includes: determining a parallel training strategy search space for an initial Transformer model based on M preset parallel training strategies; the parallel training strategy search space includes N1 combinations of parallel training strategies, each combination including N2 preset parallel training strategies; M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M; based on each of the preset parallel training strategies... The strategy involves constructing at least one decision tree; each decision tree is used to determine a set of parallel training strategies for the initial Transformer model from the parallel training strategy search space; the set of parallel training strategies includes N3 combinations of parallel training strategies, where N3 is a positive integer greater than or equal to 1 and less than N1; based on the set of training strategies, a target parallel training strategy combination is determined; the target parallel training strategy combination is the parallel training strategy combination with the highest throughput in the set of parallel training strategies; the initial Transformer model is trained using the target parallel training strategy combination to obtain the target Transformer model.
[0183] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0184] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A parallel training method for a Transformer model, characterized in that, include: Based on M preset parallel training strategies, the search space for parallel training strategies of the initial Transformer model is determined. The parallel training strategy search space includes N1 combinations of parallel training strategies, and each combination of parallel training strategies includes N2 preset parallel training strategies. M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M; The preset parallel training strategy includes at least one of the following: data parallel training strategy, sharded data parallel training strategy, tensor parallel training strategy, and pipelined parallel training strategy; Based on each of the preset parallel training strategies, at least one decision tree is constructed; each decision tree is used to determine the set of parallel training strategies for the initial Transformer model from the search space of the parallel training strategies; the set of parallel training strategies includes N3 combinations of parallel training strategies, where N3 is a positive integer greater than or equal to 1 and N3 is less than N1. Based on the set of parallel training strategies, a target combination of parallel training strategies is determined; the target combination of parallel training strategies is the combination of parallel training strategies with the highest throughput in the set of parallel training strategies. The initial Transformer model is trained using the target parallel training strategy combination to obtain the target Transformer model; The method further includes: When the data parallel training strategy and the sharded data parallel training strategy exist in the same decision tree, the decision tree is pruned based on the data parallel training strategy and the sharded data parallel training strategy. The step of determining the target parallel training strategy combination based on the set of parallel training strategies includes: For each Transformer sub-model, a dynamic programming search algorithm is used to determine the first parallel training strategy combination for each Transformer sub-model from the set of parallel training strategies; the first parallel training strategy combination is the parallel training strategy combination with the minimum time cost. Among each of the first parallel training strategy combinations, the parallel training strategy combination with the highest throughput is determined as the target parallel training strategy combination. The dynamic programming search algorithm is expressed by the following formula (1): （1） in, This indicates the number of layers in the Transformer sub-model; Indicates the memory limit of the graphics processing unit (GPU); This represents the set of training strategies; In the Transformer sub-model, the first... The parallel training strategy used in the layer; In the Transformer sub-model, the first... The total execution time of the layer within the memory limits of the graphics processor device; In the Transformer sub-model, the first... Layers use parallel training strategies Memory overhead; In the Transformer sub-model, the first... Layers use parallel training strategies Execution time; In the Transformer sub-model, the first... Layers use parallel training strategies , No. Layers use parallel training strategies The resulting conversion overhead.
2. The parallel training method for the Transformer model according to claim 1, characterized in that, The construction of at least one decision tree based on each of the preset parallel training strategies includes: Using the pipelined parallel training strategy, the initial Transformer model is divided into multiple Transformer sub-models; each Transformer sub-model corresponds to at least one device group, and the device group includes at least one graphics processor device. For each Transformer sub-model, a decision tree corresponding to the Transformer sub-model is constructed based on the decision tree construction rules; The decision tree construction rules include at least one of the following: The root node of the decision tree is the data parallel training strategy; Each layer of the decision tree represents any one of the data parallel training strategy, the sharded data parallel training strategy, and the tensor parallel training strategy. The preset parallel training strategies differ between the layers of the decision tree; The degree of the non-leaf nodes in the decision tree is a power of 2; The total number of decision tree leaf nodes is equal to the number of graphics processor devices in each device group.
3. A parallel training device for a Transformer model, characterized in that, include: The first determining module is used to determine the search space of parallel training strategies for the initial Transformer model based on M preset parallel training strategies. The parallel training strategy search space includes N1 combinations of parallel training strategies, and each combination of parallel training strategies includes N2 preset parallel training strategies. M and N1 are positive integers greater than 1, and N2 is a positive integer greater than or equal to 1 and less than or equal to M; The preset parallel training strategy includes at least one of the following: data parallel training strategy, sharded data parallel training strategy, tensor parallel training strategy, and pipelined parallel training strategy; The construction module is used to construct at least one decision tree based on each of the preset parallel training strategies; each decision tree is used to determine the set of parallel training strategies for the initial Transformer model from the search space of the parallel training strategies; the set of parallel training strategies includes N3 combinations of parallel training strategies, where N3 is a positive integer greater than or equal to 1 and N3 is less than N1. The second determining module is used to determine the target parallel training strategy combination based on the set of parallel training strategies. The target parallel training strategy combination is the parallel training strategy combination with the highest throughput in the set of parallel training strategies. The training module is used to train the initial Transformer model using the target parallel training strategy combination to obtain the target Transformer model; The device further includes: The processing module is configured to perform pruning on the decision tree based on the data parallel training strategy and the sharded data parallel training strategy when the data parallel training strategy and the sharded data parallel training strategy exist in the same decision tree. The second determining module is specifically used for: For each Transformer sub-model, a dynamic programming search algorithm is used to determine the first parallel training strategy combination for each Transformer sub-model from the set of parallel training strategies; the first parallel training strategy combination is the parallel training strategy combination with the minimum time cost. Among each of the first parallel training strategy combinations, the parallel training strategy combination with the highest throughput is determined as the target parallel training strategy combination. The dynamic programming search algorithm is expressed by the following formula (1): （1） in, This indicates the number of layers in the Transformer sub-model; Indicates the memory limit of the graphics processing unit (GPU); This represents the set of training strategies; In the Transformer sub-model, the first... The parallel training strategy used in the layer; In the Transformer sub-model, the first... The total execution time of the layer within the memory limits of the graphics processor device; In the Transformer sub-model, the first... Layers use parallel training strategies Memory overhead; In the Transformer sub-model, the first... Layers use parallel training strategies Execution time; In the Transformer sub-model, the first... Layers use parallel training strategies , No. Layers use parallel training strategies The resulting conversion overhead.
4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the parallel training method for the Transformer model as described in claim 1 or 2.
5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the parallel training method for the Transformer model as described in claim 1 or 2.
6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the parallel training method for the Transformer model as described in claim 1 or 2.