A pipelined parallel training method, system, terminal, and storage medium for a natural language processing model.
By dividing the natural language processing model into a self-attention module and a feedforward neural network module, and performing pre-training and simulation calculations, the optimal partitioning scheme is selected for pipelined parallel training, which solves the problem of low efficiency in existing methods and achieves a more efficient training process.
Patent Information
- Application Number
- CN202410856962.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-06-28
AI Technical Summary
Existing pipelined parallel methods are inefficient when training large natural language processing models and struggle to effectively utilize the computing power of multiple computing devices.
The natural language processing model is divided into a self-attention module and a feedforward neural network module, and pre-trained to obtain computation time. The model is divided into multiple parts according to different network layer partitioning methods. The model partitioning scheme with the minimum iteration time is selected through simulation calculation and distributed to multiple computing devices for pipelined parallel training.
It improves the efficiency of pipelined parallel training of natural language processing models and reduces training computation time.
Smart Images

Figure CN118503764B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and in particular to a pipelined parallel training method, system, terminal and storage medium for a natural language processing model. Background Technology
[0002] Natural Language Processing (NLP) models are deep neural network models in the field of natural language processing, primarily used for functions such as language translation and question answering. Training an NLP model requires batch input of data for computation, followed by updating the model's parameters based on the results. This process is iterative until the model achieves the desired performance on the actual task, such as more accurate language translation or more reasonable question answers. Larger models achieve better performance on real-world tasks by increasing the number of parameters and the amount of data used during training. However, the large number of parameters and the massive amount of data required for training large models make their training a very time-consuming process.
[0003] Currently, using distributed training methods to train large models has become a consensus. Distributed training methods can utilize computing devices on multiple machines to train large models, meeting the computational and storage requirements of model training. Pipeline parallelism is one of the commonly used distributed training methods; however, for scenarios where large natural language processing models are trained using computing devices on multiple machines, the acceleration effect of existing pipeline parallelism methods is generally limited.
[0004] Therefore, how to improve the pipelined parallel training efficiency of natural language processing models is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] To address the aforementioned technical problems, this application provides a pipelined parallel training method for natural language processing models, which can improve the efficiency of pipelined parallel training of natural language processing models. This application also provides a pipelined parallel training system, terminal, and storage medium for natural language processing models, achieving the same technical effects.
[0006] The first objective of this application is to provide a pipelined parallel training method for natural language processing models.
[0007] The aforementioned objective of this application is achieved through the following technical solution: A pipelined parallel training method for a natural language processing model includes: Each single-layer model in the natural language processing model is divided into a self-attention module and a feedforward neural network module, and the natural language processing model is pre-trained to obtain the computation time of each self-attention module and the computation time of each feedforward neural network module. According to different network layer partitioning methods, the multiple network layers in the natural language processing model are divided into N network layers to obtain multiple model partitioning schemes for the natural language processing model. Based on the multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module, simulation calculations are performed to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme. The model partitioning scheme with the smallest iteration time of the pipelined parallel method is selected from the multiple model partitioning schemes and used as the target model partitioning scheme. The N network layers defined by the target model partitioning scheme are distributed to N computing devices, and the natural language processing model is iteratively trained using the N computing devices in a pipelined parallel manner to obtain a trained natural language processing model.
[0008] Preferably, in the pipelined parallel training method of the natural language processing model, the self-attention module includes a Layer Norm layer, a MultiHead Self-Attention layer, a Dropout layer, and an Add layer connected in sequence; the feedforward neural network module includes a Layer Norm layer, a Feed Forward layer, a Dropout layer, and an Add layer connected in sequence.
[0009] Preferably, in the pipelined parallel training method of the natural language processing model, the step of dividing the multiple network layers in the natural language processing model into N network layers according to different network layer partitioning methods to obtain multiple model partitioning schemes of the natural language processing model includes: The multiple network layers in the natural language processing model are divided into N network layers to obtain a basic model partitioning scheme. Based on the basic model partitioning scheme, the pipeline level where the critical path of the pipeline is located is obtained; Adjust the number of network layers in the pipeline stage where the critical path of the pipeline is located to obtain the adjusted model partitioning scheme; Based on the adjusted model partitioning scheme, the pipeline level where the critical path of the pipeline is located is re-obtained, and it is determined whether the pipeline level where the critical path of the pipeline is located remains unchanged or moves to a lower pipeline level compared to the pipeline level where the critical path of the pipeline is located before the previous adjustment. If so, the adjusted model partitioning scheme is output, and then the process of adjusting the number of network layers on the pipeline level where the critical path is located is re-executed to obtain the adjusted model partitioning scheme is completed.
[0010] Preferably, in the pipelined parallel training method for the natural language processing model, the step of performing simulation calculations based on multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme includes: According to each of the model partitioning schemes, the forward and backward computations of each micro-batch on each pipeline stage corresponding to each of the model partitioning schemes are regarded as a basic block, and each basic block is numbered to obtain the numbered basic blocks corresponding to each of the model partitioning schemes. Based on the basic blocks numbered according to each model partitioning scheme, the forward and backward computation times of each micro-batch in each pipeline stage corresponding to each model partitioning scheme, the computation time of each self-attention module, and the computation time of each feedforward neural network module, the iteration time of the pipeline is simulated using a dynamic programming algorithm to obtain the iteration time of the pipeline parallel method corresponding to each model partitioning scheme.
[0011] Preferably, in the pipelined parallel training method for the natural language processing model, after selecting the model partitioning scheme corresponding to the minimum iteration time of the pipelined parallel method from multiple model partitioning schemes as the target model partitioning scheme, the method further includes: Starting from 0 and incrementing by 1, the number of micro-batches in the pipeline startup phase is continuously increased. After each increase in the number of micro-batches, an un-segmented micro-batch is placed in the last pipeline stage. Based on the target model partitioning scheme, the computation time of each self-attention module, the computation time of each feedforward neural network module, and the end time of the first micro-batch in the last pipeline stage, the latest start time of the un-segmented micro-batch in the first pipeline stage is deduced in reverse order from the last pipeline stage to the first pipeline stage. It is then determined whether the latest start time is greater than or equal to the end time of the last micro-batch in the current first pipeline stage. When the latest start time is greater than or equal to the end time of the last micro-batch in the current first pipeline stage, the process of increasing the number of micro-batches is stopped, resulting in the number S of micro-batches to be segmented.
[0012] Preferably, in the pipelined parallel training method of the natural language processing model, the step of iteratively training the natural language processing model using N computing devices in a pipelined parallel manner to obtain a trained natural language processing model includes: The training data required for the current iteration training process is obtained, and the training data is divided into M micro-batches of training data. The M micro-batches of training data are processed by N computing devices in a pipelined parallel manner for forward computation, backward computation and communication to complete the current round of iterative training of the natural language processing model. The above iterative training steps are repeated until the preset iteration termination condition is met to obtain the trained natural language processing model. In each iteration of training, the M micro-batches of training data are sequentially input into the computing device corresponding to the first pipeline stage. For the first S micro-batches of training data, each micro-batch is divided into two parts based on its batch size and then sequentially input into the computing device corresponding to the first pipeline stage. After completing the calculation of the first part of the training data, the computing device corresponding to the first pipeline stage communicates the intermediate result to the computing device corresponding to the next pipeline stage before starting the calculation of the second part of the training data. After completing the calculation of the second part of the training data, the intermediate result is communicated to the computing device corresponding to the next pipeline stage, and so on, until the first S micro-batches of training data are processed.
[0013] Preferably, in the pipelined parallel training method of the natural language processing model, during the pipelined processing of the first S micro-batches of training data, when the computing device corresponding to each pipeline receives the intermediate result transmitted by the computing device corresponding to the previous pipeline, if the pipeline to which the current computing device is located has already performed backward computation, the intermediate result is stored. When the computing device receives the two intermediate results corresponding to the two parts of training data after splitting the training data of a certain micro-batch in the first S micro-batches of training data, the computation process begins again.
[0014] The second objective of this application is to provide a pipelined parallel training system for natural language processing models.
[0015] The second objective of this application is achieved through the following technical solution: A pipelined parallel training system for a natural language processing model includes: The training unit is used to divide each single-layer model in the natural language processing model into a self-attention module and a feedforward neural network module, and to pre-train the natural language processing model to obtain the computation time of each self-attention module and the computation time of each feedforward neural network module. A partitioning unit is used to divide multiple network layers in the natural language processing model into N network layers according to different network layer partitioning methods, so as to obtain multiple model partitioning schemes of the natural language processing model. The simulation unit is used to perform simulation calculations based on the multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module, to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme. The selection unit is used to select the model partitioning scheme with the smallest iteration time of the pipelined parallel method from the multiple model partitioning schemes, and use it as the target model partitioning scheme. The training unit is further configured to distribute the N network layers divided by the target model partitioning scheme to N computing devices, and to iteratively train the natural language processing model using the N computing devices in a pipelined parallel manner to obtain a trained natural language processing model.
[0016] The third objective of this application is to provide a pipelined parallel training terminal for natural language processing models.
[0017] The aforementioned objective three of this application is achieved through the following technical solution: A pipelined parallel training terminal for a natural language processing model includes: a storage medium and a processor; The storage medium stores computer-executed instructions. The processor executes computer execution instructions stored in the storage medium to implement the pipelined parallel training method for the natural language processing model as described above.
[0018] The fourth objective of this application is to provide a computer-readable storage medium.
[0019] The fourth objective of this application is achieved through the following technical solution: A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a pipelined parallel training method for a natural language processing model as described above.
[0020] The above technical solution divides each single-layer model in the natural language processing (NLP) model into self-attention modules and feedforward neural network modules, and pre-trains the NLP model to obtain the computation time of each self-attention module and each feedforward neural network module. This refines the granularity of model partitioning from the layer level to the sub-layer level, which reduces the computation time differences between pipeline stages and improves the efficiency of subsequent pipeline partitioning. Furthermore, according to different network layer partitioning methods, the multiple network layers in the NLP model are divided into N network layers to obtain multiple model partitioning schemes for the NLP model. Based on these multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module... The computation time of the neural network module is simulated to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme. The model partitioning scheme with the smallest pipelined parallel method iteration time is selected from multiple model partitioning schemes as the target model partitioning scheme. This selection of the optimal model partitioning scheme from multiple schemes is beneficial to improving the computational efficiency of subsequent pipelined parallel methods. Finally, the N network layers partitioned by the target model partitioning scheme are distributed across N computing devices, and the natural language processing model is iteratively trained using a pipelined parallel approach on these N computing devices to obtain a trained natural language processing model, thereby reducing the computation time for natural language processing model training. In summary, the above technical solution can improve the pipelined parallel training efficiency of natural language processing models. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a flowchart illustrating a pipelined parallel training method for a natural language processing model provided in an embodiment of this application. Figure 2 This is a schematic diagram of the structure after the single-layer model is divided according to the embodiments of this application; Figure 3 This is a schematic diagram of the micro-batch segmentation process of the six-stage pipeline provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of a pipelined parallel training system for a natural language processing model provided in an embodiment of this application. Figure 5This is a schematic diagram of the structure of a pipelined parallel training terminal for a natural language processing model provided in an embodiment of this application. Detailed Implementation
[0023] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0024] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.
[0025] It should be understood that the use of terms such as "system," "device," "unit," and / or "module" in this application is merely one method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.
[0026] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.
[0027] If a flowchart is used in this application, it is used to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.
[0028] It should also be noted that, in this document, terms such as “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the article or device that includes the aforementioned element.
[0029] The embodiments in this application are written in a progressive manner.
[0030] like Figure 1 As shown, this application provides a pipelined parallel training method for a natural language processing model, including: S101. Divide each single-layer model in the natural language processing model into a self-attention module and a feedforward neural network module, and pre-train the natural language processing model to obtain the computation time of each self-attention module and the computation time of each feedforward neural network module. In S101, specifically regarding model partitioning in the pipelined parallel method, a coarse model partitioning granularity reduces the practicality of the pipelined parallel method. Large-scale natural language processing (NLP) models have a unified model structure, typically composed of stacked single layers. Based on this characteristic, structural analysis is performed on the single-layer models of large-scale NLP models. By obtaining the NLP model to be trained, each single-layer model in the NLP model is divided into interconnected self-attention modules and feedforward neural network modules, thus refining the granularity of model partitioning from the layer level to the sub-layer level. Then, several batches of training data can be obtained as pre-training data. The NLP model is pre-trained using the pre-training data to obtain the computation time of each self-attention module and each feedforward neural network module. The training data specifically consists of natural language sentences, and the specific method of obtaining them does not affect the implementation of this embodiment. In this step, by refining the granularity of model partitioning from the layer level to the sub-layer level, this granularity allows for smaller differences in computation time between different pipeline stages, improving the efficiency of subsequent pipeline partitioning.
[0031] In a specific embodiment, a single-layer model in a natural language processing model can be divided into a self-attention module and a feedforward neural network module, such as... Figure 2 As shown; the self-attention module includes a Layer Norm layer, a MultiHead Self-Attention layer, a Dropout layer, and an Add layer connected in sequence; the feedforward neural network module includes a Layer Norm layer, a Feed Forward layer, a Dropout layer, and an Add layer connected in sequence; In this system, the input data for the self-attention module and the feedforward neural network module can be fed into their respective Layer Norm layers or Add layers. The Add layer of the self-attention module is connected to the Layer Norm layer of the feedforward neural network module. LayerNorm can standardize individual data samples along the feature dimension. A key advantage is that it is independent of the batch size, making it suitable for small-batch or even single-sample scenarios. This batch-size independence makes LayerNorm an ideal choice for handling variable-length sequences and small-batch data. MultiHead Self-Attention refers to using a self-attention mechanism to compute the contextual representation of each word in a sentence. This process maps the original representation of each word into multiple subspaces, computes attention weights in each subspace, and finally sums the attention weights of these subspaces to obtain the contextual representation of each word. This method can capture complex dependencies between words and is interpretable. The core idea of Dropout is to randomly delete a portion of neurons during training, so that the model has a different structure in each iteration. This can prevent the model from overfitting and improve the model's generalization ability. Add stands for Residual Connection, a connection technique used in deep neural networks to address the vanishing gradient and representation bottleneck problems during deep neural network training. Specifically, residual connections allow the output of a layer to skip one or more layers and become the input to subsequent layers. This skip-layer connection makes gradients easier to backpropagate, thus aiding in the training of deeper networks. The Feed Forward layer is a fully connected neural network, typically containing two linear transformations and an activation function. It extracts and refines features, enabling the model to better understand and process input data, increasing its expressive power and improving training efficiency.
[0032] S102. According to different network layer partitioning methods, the multiple network layers in the natural language processing model are divided into N network layers to obtain multiple model partitioning schemes for the natural language processing model. In S102, specifically, according to different network layer partitioning methods, the multiple network layers in the natural language processing model are divided into N network layers. Each network layer partitioning method corresponds to a model partitioning scheme. The composition of the N network layers divided by each network layer partitioning method is not the same, thus obtaining multiple model partitioning schemes for the natural language processing model. Here, the value of N is the total number of computing devices.
[0033] In some embodiments, one implementation of this step specifically includes: S1021. Divide the multiple network layers in the natural language processing model into N network layers to obtain a basic model partitioning scheme; In S1021, specifically, existing network layer partitioning methods can be used to divide multiple network layers in a natural language processing model into N network layers, resulting in a basic model partitioning scheme. For example, taking a natural language processing model containing M network layers as an example, dividing it into N network layers can be achieved using a dynamic programming algorithm to obtain a basic model partitioning scheme. Specifically, for the first i network layers of the natural language processing model, it is divided into j parts, where i starts from 1 and increments by 1 each time until N, and j starts from 1 and increments by 1 each time until the smaller of i and N. Then, k is set as the split point between the last part and the second to last part in the j parts, and k starts from 0 and increments by 1 each time until i-1. In this way, the problem of dividing the first i network layers into j parts can be transformed into two subproblems: dividing the last part and dividing the first k network layers into j-1 parts. The maximum computation time of the two subproblems is taken as the cost of the current partitioning scheme, and the partitioning scheme with the minimum cost is taken as the basic model partitioning scheme.
[0034] S1022. Based on the basic model partitioning scheme, obtain the pipeline level where the critical path of the pipeline is located; In S1022, specifically, after obtaining the basic model partitioning scheme, the pipeline level where the critical path of the current pipeline is located can be obtained based on the basic model partitioning scheme. For example, in the process of solving the iteration time of the pipeline parallel method corresponding to the current model partitioning scheme through the pipeline dependencies, the pipeline execution process is simulated according to the pipeline dependencies. During this process, all intermediate results are recorded. When the final iteration time of the current model partitioning scheme is obtained, the critical path of the pipeline can be deduced from the pipeline dependencies and the final iteration time to obtain the pipeline level where the critical path of the current pipeline is located. It can be understood that the process of deducing the critical path of the pipeline is equivalent to the reverse process of simulating the iteration time using the dynamic programming algorithm, which is common knowledge in this field. Each pipeline level can be numbered in ascending order from the first pipeline level to the last pipeline level.
[0035] S1023. Adjust the number of network layers on the pipeline stage where the critical path of the pipeline is located to obtain the adjusted model partitioning scheme; In S1023, specifically, some network layers of the pipeline stage containing the critical path of the pipeline are moved to adjacent pipeline stages (such as the previous or next pipeline stage) to adjust the number of network layers in the pipeline stage containing the critical path of the pipeline, thereby obtaining the adjusted model partitioning scheme; wherein, the number of network layers to be moved can be preset based on actual needs, and this application does not impose specific restrictions on this.
[0036] S1024. Based on the adjusted model partitioning scheme, re-obtain the pipeline level where the critical path of the pipeline is located, and determine whether the pipeline level where the critical path of the pipeline is located remains unchanged or moves to a lower pipeline level compared to the pipeline level where the critical path of the pipeline was located before the previous adjustment. If so, output the adjusted model partitioning scheme, and then return to re-execute S1023.
[0037] In S1024, specifically, when the critical path of the pipeline moves to a lower pipeline stage (a smaller pipeline stage) or the length of the critical path in the same pipeline stage is longer, the iteration time of the pipeline parallelization method decreases. Furthermore, the range of change in the critical path is positively correlated with the number of pipeline stages, and generally, the range of change is not large. Based on this, by continuously adjusting the number of network layers in the pipeline stage where the critical path is located, an adjusted model partitioning scheme is obtained. Then, based on the adjusted model partitioning scheme, the pipeline stage where the critical path is located is re-acquired. When the re-acquired pipeline stage where the critical path is located is compared to the pipeline stage before the previous adjustment... For the pipeline level where the critical path of the pipeline resides, if it remains unchanged or moves to a lower pipeline level, then the adjusted network layer partitioning is considered the optimal possible partitioning. The adjusted model partitioning scheme can then be output as an option. It should be noted that moving to a lower pipeline level specifically refers to moving to a pipeline level with a lower number than the one where the critical path of the pipeline resides before the previous adjustment. Then, step S1023 is re-executed to readjust the number of network layers at the pipeline level where the critical path of the pipeline resides, and the corresponding judgment in S1024 is performed again, thereby obtaining multiple model partitioning schemes for the natural language processing model. It should be noted that in some embodiments, when it is determined that the pipeline level where the critical path of the newly acquired pipeline resides has not remained unchanged and has not moved to a lower pipeline level compared to the pipeline level where the critical path of the pipeline resides before the previous adjustment, or when the pipeline level where the critical path of the newly acquired pipeline resides is the first pipeline level, model partitioning can be stopped.
[0038] In this embodiment, the above implementation steps can effectively reduce the search space of the pipeline partitioning method, improve the efficiency of pipeline partitioning, and help improve the pipeline parallel training efficiency of natural language processing models.
[0039] S103. Based on multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module, perform simulation calculations to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme. In S103, specifically, the iteration time of the pipeline can be simulated based on multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module. This allows us to obtain the iteration time of the pipeline parallel method corresponding to each model partitioning scheme, thereby judging the merits of the model partitioning scheme.
[0040] In some embodiments, one implementation of this step specifically includes: S1031. Based on each model partitioning scheme, the forward and backward calculations of each micro-batch on each pipeline stage corresponding to each model partitioning scheme are regarded as a basic block, and each basic block is numbered to obtain the numbered basic blocks corresponding to each model partitioning scheme. In S1031, specifically, based on the various model partitioning schemes, the order in which the forward and backward computations of each micro-batch are executed in each stage of the pipeline can be determined, i.e., the dependencies of the micro-batch computations. However, in this process, the forward and backward computations are associated with the micro-batch number and cannot be formally represented to express the pipeline dependencies. Therefore, the forward and backward computations of micro-batches at each pipeline stage are considered as a basic block. By numbering each basic block, the numbered basic blocks corresponding to each model partitioning scheme are obtained, thus formally expressing the pipeline dependencies. Specifically, the pipeline can be divided into three stages: startup, stabilization, and cooling. During the startup stage, no numbering is performed. During the stabilization stage, adjacent forward and backward operations are considered as a basic block, and these basic blocks are numbered starting from 0. The forward and backward operations within each block are assigned the same number. For example, if the first pipeline stage has 12 operations—6 forward operations and 6 backward operations—arranged in forward and backward order, these 12 operations can be divided into 6 basic blocks, numbered from 0 to 5. Each block contains one forward operation and one backward operation, assigned the same number as the block. For example, the first forward and backward operation is numbered 0. During the cooling stage, each backward operation is considered as a basic block and numbered in reverse order. For example, the cooling stage of the first pipeline stage has 3 backward operations, which are numbered from left to right as 2, 1, and 0. The above numbering method is merely an example, and this application is not limited to it.
[0041] S1032. Based on the basic blocks numbered according to each model partitioning scheme, the forward and backward computation times of micro-batches at each pipeline stage corresponding to each model partitioning scheme, the computation time of each self-attention module, and the computation time of each feedforward neural network module, the iteration time of the pipeline is simulated using a dynamic programming algorithm to obtain the iteration time of the pipeline parallel method corresponding to each model partitioning scheme.
[0042] In S1032, specifically, based on the numbered basic blocks corresponding to each model partitioning scheme obtained from this formal expression, as well as the forward and backward computation times of each micro-batch, the computation time of each self-attention module, and the computation time of each feedforward neural network module, the iteration time of the pipeline is simulated using a dynamic programming algorithm to obtain the iteration time of the pipeline parallel method corresponding to each model partitioning scheme. Specifically, the pipeline startup phase is not simulated because the computation of the first micro-batch is not blocked. For the pipeline stabilization phase, dependency analysis shows that the start time of the forward computation of the i-th pipeline stage (j) depends on the end time of the (j-1)-th backward computation of the current pipeline stage and the end time of the (j-1)-th forward computation of the (i-1)-th pipeline stage. Conversely, the start time of the backward computation of the i-th pipeline stage (j) depends on the end time of the forward computation of the current pipeline stage (j) and the end time of the backward computation of the (i+1)-th pipeline stage (j). For the pipeline cooling phase, dependency analysis shows that the start time of the backward computation of the i-th pipeline stage (j) depends on the end time of the backward computation of the current pipeline stage (j+1) and the end time of the backward computation of the (i+1)-th pipeline stage (j). Based on these dependencies and the number of micro-batches in the pipeline, the iteration time of the pipeline parallelization method under the current model partitioning scheme can be simulated.
[0043] In this embodiment, the above implementation steps can effectively simulate and calculate the iteration time of the pipeline parallel method corresponding to each model partitioning scheme, which is beneficial for subsequent judgment of the merits of each model partitioning scheme and finding the optimal model partitioning scheme.
[0044] S104. Select the model partitioning scheme with the smallest iteration time of the pipelined parallel method from multiple model partitioning schemes, and use it as the target model partitioning scheme. In S104, specifically, the model partitioning scheme with the smallest iteration time of the pipelined parallel method is selected from multiple model partitioning schemes as the target model partitioning scheme. In this way, the optimal model partitioning scheme is selected from multiple model partitioning schemes, which is beneficial to improving the computational efficiency of the subsequent pipelined parallel method.
[0045] S105. The N network layers divided by the target model partitioning scheme are distributed to N computing devices, and the natural language processing model is iteratively trained using a pipelined parallel approach on the N computing devices to obtain the trained natural language processing model.
[0046] In S105, specifically, by acquiring the natural language processing model to be trained, training data, and computing node configuration (total number of computing devices N, network bandwidth, etc.), the target model partitioning scheme is obtained through the aforementioned steps. Then, the N network layers partitioned by the target model partitioning scheme are allocated to N computing devices respectively. For example, the first network layer is allocated to the first computing device, the second network layer is allocated to the second computing device, and so on. Each computing device acts as a primary pipeline, and N computing devices form a parallel pipeline. The natural language processing (NLP) model is trained using this pipelined parallel approach across N computing devices. After partitioning the NLP model and computing devices as described above, the training computation step begins. This step is similar to traditional training processes, employing an iterative training method. Each iteration uses a portion of the entire dataset's training sample data (batch) for forward, backward, and communication computations of the model, until a preset iteration termination condition is met (such as reaching the target accuracy or target number of iterations). The trained NLP model is then obtained, allowing the execution of corresponding data processing tasks, such as language translation or question answering.
[0047] Pipeline parallelism is one of the commonly used distributed training methods. However, for scenarios where large natural language processing models are trained using computing devices on multiple machines, the speedup effect of existing pipeline parallelism methods is generally limited.
[0048] The above embodiment divides each single-layer model in the natural language processing model into self-attention modules and feedforward neural network modules, and pre-trains the natural language processing model to obtain the computation time of each self-attention module and each feedforward neural network module. This refines the granularity of model partitioning from the layer level to the sub-layer level. This granularity allows for smaller differences in computation time between different pipeline stages, improving the efficiency of subsequent pipeline partitioning. Furthermore, according to different network layer partitioning methods, the multiple network layers in the natural language processing model are divided into N network layers to obtain multiple model partitioning schemes for the natural language processing model. Based on these multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module... The computation time of the neural network module is simulated to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme. The model partitioning scheme with the smallest pipelined parallel method iteration time is selected from multiple model partitioning schemes as the target model partitioning scheme. This selection of the optimal model partitioning scheme from multiple schemes is beneficial to improving the computational efficiency of subsequent pipelined parallel methods. Finally, the N network layers partitioned by the target model partitioning scheme are distributed across N computing devices, and the natural language processing model is iteratively trained using a pipelined parallel approach on the N computing devices to obtain a trained natural language processing model, thereby reducing the computational time for natural language processing model training. In summary, the above embodiments can improve the pipelined parallel training efficiency of natural language processing models.
[0049] Startup overhead is an inherent cost of the startup phase of pipelined parallel methods, which affects the efficiency of these methods to some extent. This impact is more significant when using deeper pipelines. To reduce the inherent startup overhead of pipelined parallel methods, in other embodiments of this application, after selecting the model partitioning scheme corresponding to the minimum iteration time of the pipelined parallel method from multiple model partitioning schemes as the target model partitioning scheme, the method further includes: S201. Starting from 0 and incrementing by 1, continuously increase the number of micro-batches in the pipeline startup phase. After each increase in the number of micro-batches, place an un-segmented micro-batch in the last pipeline stage. Based on the target model partitioning scheme, the computation time of each self-attention module and each feedforward neural network module, and the end time of the first micro-batch in the last pipeline stage, reverse the order from the last pipeline stage to the first pipeline stage to deduce the latest start time of the un-segmented micro-batch in the first pipeline stage. Determine whether the latest start time is greater than or equal to the end time of the last micro-batch in the current first pipeline stage. When the latest start time is greater than or equal to the end time of the last micro-batch in the current first pipeline stage, stop increasing the number of micro-batches and obtain the number of micro-batches S to be segmented. In S201, specifically, the execution process of the first micro-batch after splitting will not be blocked, so the end time of the first micro-batch in the last pipeline stage can be obtained. Based on this end time, starting from 0 and incrementing by 1, the number of split micro-batches in the pipeline startup phase is continuously increased. After each increase in the number of split micro-batches, an unsplit micro-batch is placed in the last pipeline stage, and the latest start time of the unsplit micro-batch in the first pipeline stage is deduced in the order from the last pipeline stage to the first pipeline stage. If the latest start time is greater than or equal to the end time of the last micro-batch in the current first pipeline stage, it is considered that the pipeline startup overhead no longer decreases and there is no need to split the micro-batch. Otherwise, the micro-batch is split, and it is assumed that the next micro-batch in the last pipeline stage will not be split. The above process is repeated until the first micro-batch that does not need to be split is found, and the process of increasing the number of split micro-batches is stopped, thus obtaining the number S of micro-batches that need to be split.
[0050] In a specific embodiment, with Figure 3 Taking the process of micro-batch segmentation in a six-stage pipeline as an example, Figure 3 (a) shows the workflow of the first three micro-batches in the pipeline before micro-batch splitting; Figure 3 (b) shows that after the first micro-batch is split, the second micro-batch is not split and is placed at the end of the last pipeline stage. When simulating from the last pipeline stage to the first pipeline stage, it is found that the latest start time of the second micro-batch is earlier than the end time of the current first pipeline stage. Therefore, the second micro-batch also needs to be split. Figure 3 Figure (c) shows that after splitting into two micro-batches, a third micro-batch is placed at the end of the last pipeline stage. A second simulation reveals that the latest start time of the third micro-batch in the first pipeline stage is later than the end time of the current first pipeline stage. Therefore, splitting into two micro-batches reduces pipeline startup overhead. It should be noted that... Figure 3 The green blocks represent the forward computation of the micro-batch, and the blue blocks represent the backward computation of the micro-batch. Figure 3 The red block in (b) indicates that the calculation results show that the latest start time of the second micro-batch is less than the end time of the current pipeline stage, so there will be an overlap in time period; it should be noted that each time the last pipeline stage is increased, it is the time of the forward calculation of a micro-batch plus the time of the backward calculation.
[0051] Accordingly, one implementation of the steps to iteratively train a natural language processing model using N computing devices in a pipelined parallel manner to obtain a trained natural language processing model includes: S202. Obtain the training data required for the current iteration training process, and divide the training data into M micro-batches. Use N computing devices to perform forward computation, backward computation, and communication on the M micro-batches of training data in a pipelined parallel manner to complete the current iteration training of the natural language processing model. Repeat the above iterative training steps until the preset iteration termination condition is met to obtain the trained natural language processing model. In each iteration training process, the obtained M micro-batches of training data are sequentially input into the computing device corresponding to the first pipeline stage, and for each M micro-batch... The training data of the first S micro-batches is divided into two parts according to the batch size dimension. These parts are then input into the computing device corresponding to the first pipeline stage. After the computing device of the first pipeline stage completes the calculation of the first part of the training data, it communicates the intermediate result to the computing device corresponding to the next pipeline stage, and then starts the calculation of the second part of the training data. After the calculation of the second part of the training data is completed, the intermediate result is communicated to the computing device corresponding to the next pipeline stage, and so on, until the first S micro-batches of training data are processed.
[0052] In S202, specifically, the total training data can be divided into multiple batches of training data. In each iteration of training, one batch of training data is obtained as the training data required for the current iteration. Then, the training data required for the current iteration is divided into M micro-batches. For example, assuming that one batch of training data includes 128 natural language sentences, then each micro-batch of training data has 128 / M natural language sentences. Then, all the obtained micro-batches of training data are sequentially input into the computing device corresponding to the first pipeline stage to start the calculation. The natural language processing model is trained in the current round using a pipelined parallel approach through N computing devices. The above iterative training process is repeated until the preset iteration termination condition is met, and the trained natural language processing model is obtained. In this process, the training data for the first S micro-batches is divided into two parts based on the batch size. These parts are then sequentially input into the computing device corresponding to the first pipeline stage. After the computing device in the first pipeline stage completes the computation of the first part of the training data, it communicates the intermediate result to the computing device corresponding to the next pipeline stage (the computing device in the next pipeline stage can start the computation process immediately after receiving the intermediate result). Then, the second part of the training data is computed. After the computation is completed, the device communicates with the computing device corresponding to the next pipeline stage again until the training data for the first S micro-batches is processed. This completes the splitting and processing of the training data for the first S micro-batches during the pipeline startup phase, which can effectively reduce the pipeline startup overhead.
[0053] In this embodiment, the number S of micro-batches to be split is first calculated. Then, the training data of the first S micro-batches in the pipeline startup phase is split, which effectively reduces the pipeline startup overhead. This method is simple and effective, with little additional overhead, and can effectively improve the efficiency of pipeline parallel methods, which is beneficial to improving the pipeline parallel training efficiency of natural language processing models.
[0054] In practical applications, the split micro-batches may encounter blockages during subsequent pipeline processing. To reduce the occurrence of blockages, in other embodiments of this application, during the pipeline processing of the training data of the first S micro-batches, when the computing device corresponding to each pipeline receives the intermediate result transmitted by the computing device corresponding to the previous pipeline, if the pipeline to which the current computing device is located has already performed backward computation, the intermediate result is stored. When the two intermediate results corresponding to the two parts of training data after splitting the training data of a certain micro-batch in the training data of the first S micro-batches are received, the computation process is started again.
[0055] Specifically, when the computing device corresponding to each pipeline stage receives the intermediate result transmitted by the computing device corresponding to the previous pipeline stage, if the pipeline stage to which the current computing device belongs has already performed backward computation, it will store the received intermediate result until it receives the two intermediate results corresponding to the two parts of the training data after splitting the training data of a certain micro-batch from the training data of the previous S micro-batch, and then performs computation. This effectively avoids blocking and helps to improve the efficiency of pipeline parallel methods.
[0056] like Figure 4 As shown, in another embodiment of this application, a pipelined parallel training system for a natural language processing model is also provided, comprising: Training unit 10 is used to divide each single-layer model in the natural language processing model into a self-attention module and a feedforward neural network module, and to pre-train the natural language processing model to obtain the computation time of each self-attention module and the computation time of each feedforward neural network module. The partitioning unit 11 is used to divide multiple network layers in the natural language processing model into N network layers according to different network layer partitioning methods, so as to obtain multiple model partitioning schemes of the natural language processing model. The simulation unit 12 is used to perform simulation calculations based on multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module, to obtain the iteration time of the pipeline parallel method corresponding to each model partitioning scheme. Selecting unit 13 is used to select the model partitioning scheme corresponding to the minimum iteration time of the pipelined parallel method from multiple model partitioning schemes, and use it as the target model partitioning scheme. Training unit 10 is also used to distribute the N network layers divided by the target model partitioning scheme to N computing devices, and to iteratively train the natural language processing model using a pipelined parallel approach on the N computing devices to obtain a trained natural language processing model.
[0057] In other embodiments of this application, in the pipelined parallel training system of the above-described natural language processing model, the self-attention module includes a Layer Norm layer, a MultiHead Self-Attention layer, a Dropout layer, and an Add layer connected in sequence; the feedforward neural network module includes a Layer Norm layer, a Feed Forward layer, a Dropout layer, and an Add layer connected in sequence.
[0058] In other embodiments of this application, in the above-described pipelined parallel training system for the natural language processing model, the partitioning unit 11, when performing the operation of dividing multiple network layers in the natural language processing model into N network layers according to different network layer partitioning methods to obtain multiple model partitioning schemes for the natural language processing model, is specifically used for: Divide the multiple network layers in the natural language processing model into N network layers to obtain a basic model partitioning scheme; Based on the basic model partitioning scheme, obtain the pipeline level where the critical path of the pipeline is located; Adjust the number of network layers in the pipeline stage where the critical path of the pipeline is located to obtain the adjusted model partitioning scheme; Based on the adjusted model partitioning scheme, the pipeline level where the critical path of the pipeline is located is re-acquired, and it is determined whether the pipeline level where the critical path of the pipeline is located remains unchanged or moves to a lower pipeline level compared to the pipeline level where the critical path of the pipeline is located before the previous adjustment. If so, the adjusted model partitioning scheme is output, and then the process is returned to re-execute the action of adjusting the number of network layers in the pipeline level where the critical path is located in order to obtain the adjusted model partitioning scheme.
[0059] In other embodiments of this application, in the above-described pipelined parallel training system for natural language processing models, the simulation unit 12, when performing simulation calculations based on multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme, is specifically used for: Based on each model partitioning scheme, the forward and backward computations of each micro-batch on each pipeline stage corresponding to each model partitioning scheme are regarded as a basic block, and each basic block is numbered to obtain the numbered basic blocks corresponding to each model partitioning scheme. Based on the basic blocks numbered according to each model partitioning scheme, the forward and backward computation times of each micro-batch in each pipeline stage corresponding to each model partitioning scheme, the computation time of each self-attention module, and the computation time of each feedforward neural network module, the iteration time of the pipeline is simulated using a dynamic programming algorithm to obtain the iteration time of the pipeline parallel method corresponding to each model partitioning scheme.
[0060] In other embodiments of this application, in the pipelined parallel training system of the above-described natural language processing model, the training unit 10 is further configured to: Starting from 0 and incrementing by 1, the number of micro-batches in the pipeline startup phase is continuously increased. After each increase in the number of micro-batches, an un-segmented micro-batch is placed in the last pipeline stage. Based on the target model partitioning scheme, the computation time of each self-attention module, the computation time of each feedforward neural network module, and the end time of the first micro-batch in the last pipeline stage, the latest start time of the un-segmented micro-batch in the first pipeline stage is deduced in reverse order from the last pipeline stage to the first pipeline stage. It is then determined whether the latest start time is greater than or equal to the end time of the last micro-batch in the current first pipeline stage. When the latest start time is greater than or equal to the end time of the last micro-batch in the current first pipeline stage, the process of increasing the number of micro-batches is stopped, resulting in the number S of micro-batches to be segmented.
[0061] In other embodiments of this application, in the above-described pipelined parallel training system for the natural language processing model, the training unit 10, when iteratively training the natural language processing model using N computing devices in a pipelined parallel manner to obtain a trained natural language processing model, is specifically used for: The training data required for the current iteration training process is obtained and divided into M micro-batches. The M micro-batches of training data are then processed by N computing devices in a pipelined parallel manner for forward computation, backward computation, and communication to complete the current iteration training of the natural language processing model. The above iterative training process is repeated until the preset iteration termination condition is met, and the trained natural language processing model is obtained. In each iteration of training, the obtained M micro-batches of training data are sequentially input into the computing device corresponding to the first pipeline stage. For the first S micro-batches of training data, each micro-batch of training data is divided into two parts according to the micro-batch size dimension, and then sequentially input into the computing device corresponding to the first pipeline stage. After the computing device corresponding to the first pipeline stage completes the calculation of the first part of the training data, it communicates the intermediate result to the computing device corresponding to the next pipeline stage, and then begins the calculation of the second part of the training data. After the calculation of the second part of the training data is completed, the intermediate result is communicated to the computing device corresponding to the next pipeline stage, until the first S micro-batches of training data are processed.
[0062] In other embodiments of this application, in the pipelined parallel training system of the above-mentioned natural language processing model, during the pipelined processing of the training data of the first S micro-batches, when the computing device corresponding to each pipeline receives the intermediate result transmitted by the computing device corresponding to the previous pipeline, if the pipeline to which the current computing device is located has already performed backward computation, the intermediate result is stored. When the two intermediate results corresponding to the two parts of training data after splitting the training data of a certain micro-batch in the first S micro-batches of training data are received, the computation process is started again.
[0063] like Figure 5 As shown, in another embodiment of this application, a pipelined parallel training terminal for a natural language processing model is also provided, including: a storage medium 20 and a processor 21; Storage medium 20 stores computer-executed instructions; The processor 21 executes computer execution instructions stored in the storage medium 20 to implement the pipelined parallel training method of the natural language processing model as described above.
[0064] The processor 21 may include one or more processing cores. The processor 21 executes instructions, programs, code sets, or instruction sets stored in the storage medium 20, and calls data stored in the storage medium 20 to perform various functions and process data as described in this application. The processor 21 may be at least one of a specific application-specific integrated circuit, digital signal processor, digital signal processing device, programmable logic device, field-programmable gate array, central processing unit, controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the functions of the processor 21 may also be other types.
[0065] The storage medium 20 can be used to store instructions, programs, code, code sets, or instruction sets. The storage medium 20 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing a pipelined parallel training method for any of the aforementioned natural language processing models. The data storage area may store data involved in the pipelined parallel training method of any of the aforementioned natural language processing models.
[0066] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions that, when executed by a processor, are used to implement the pipelined parallel training method of the natural language processing model as described above.
[0067] The computer-readable storage medium can be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory, random access memory, or optical disc.
[0068] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A pipelined parallel training method for a natural language processing model, characterized in that, include: Each single-layer model in the natural language processing model is divided into a self-attention module and a feedforward neural network module, and the natural language processing model is pre-trained to obtain the computation time of each self-attention module and the computation time of each feedforward neural network module. According to different network layer partitioning methods, the multiple network layers in the natural language processing model are divided into N network layers to obtain multiple model partitioning schemes for the natural language processing model. Based on the multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module, simulation calculations are performed to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme. The model partitioning scheme with the smallest iteration time of the pipelined parallel method is selected from the multiple model partitioning schemes and used as the target model partitioning scheme. The N network layers divided by the target model partitioning scheme are allocated to N computing devices, and the natural language processing model is iteratively trained using the N computing devices in a pipelined parallel manner to obtain a trained natural language processing model. The step of dividing the multiple network layers in the natural language processing model into N network layers according to different network layer partitioning methods to obtain multiple model partitioning schemes for the natural language processing model includes: The multiple network layers in the natural language processing model are divided into N network layers to obtain a basic model partitioning scheme. Based on the basic model partitioning scheme, the pipeline level where the critical path of the pipeline is located is obtained; Adjust the number of network layers in the pipeline stage where the critical path of the pipeline is located to obtain the adjusted model partitioning scheme; Based on the adjusted model partitioning scheme, the pipeline level where the critical path of the pipeline is located is re-obtained, and it is determined whether the pipeline level where the critical path of the pipeline is located remains unchanged or moves to a lower pipeline level compared to the pipeline level where the critical path of the pipeline is located before the previous adjustment. If so, the adjusted model partitioning scheme is output, and then the process is returned to re-execute the step of adjusting the number of network layers on the pipeline level where the critical path is located to obtain the adjusted model partitioning scheme.
2. The method as described in claim 1, characterized in that, The self-attention module includes a LayerNorm layer, a MultiHead Self-Attention layer, a Dropout layer, and an Add layer connected in sequence; the feedforward neural network module includes a LayerNorm layer, a FeedForward layer, a Dropout layer, and an Add layer connected in sequence.
3. The method as described in claim 1, characterized in that, The step involves performing simulation calculations based on multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme, including: According to each of the model partitioning schemes, the forward and backward computations of each micro-batch on each pipeline stage corresponding to each of the model partitioning schemes are regarded as a basic block, and each basic block is numbered to obtain the numbered basic blocks corresponding to each of the model partitioning schemes. Based on the basic blocks numbered according to each model partitioning scheme, the forward and backward computation times of each micro-batch in each pipeline stage corresponding to each model partitioning scheme, the computation time of each self-attention module, and the computation time of each feedforward neural network module, the iteration time of the pipeline is simulated using a dynamic programming algorithm to obtain the iteration time of the pipeline parallel method corresponding to each model partitioning scheme.
4. The method as described in claim 1, characterized in that, After selecting the model partitioning scheme with the smallest iteration time of the pipelined parallel method from among the multiple model partitioning schemes as the target model partitioning scheme, the method further includes: Starting from 0 and incrementing by 1, the number of micro-batches in the pipeline startup phase is continuously increased. After each increase in the number of micro-batches, an un-segmented micro-batch is placed in the last pipeline stage. Based on the target model partitioning scheme, the computation time of each self-attention module, the computation time of each feedforward neural network module, and the end time of the first micro-batch in the last pipeline stage, the latest start time of the un-segmented micro-batch in the first pipeline stage is deduced in reverse order from the last pipeline stage to the first pipeline stage. It is then determined whether the latest start time is greater than or equal to the end time of the last micro-batch in the current first pipeline stage. When the latest start time is greater than or equal to the end time of the last micro-batch in the current first pipeline stage, the process of increasing the number of micro-batches is stopped, resulting in the number S of micro-batches to be segmented.
5. The method as described in claim 4, characterized in that, The step of iteratively training the natural language processing model using N computing devices in a pipelined parallel manner to obtain a trained natural language processing model includes: The training data required for the current iteration training process is obtained, and the training data is divided into M micro-batches of training data. The M micro-batches of training data are processed by N computing devices in a pipelined parallel manner for forward computation, backward computation and communication to complete the current round of iterative training of the natural language processing model. The above iterative training steps are repeated until the preset iteration termination condition is met to obtain the trained natural language processing model. In each iteration of training, the M micro-batches of training data are sequentially input into the computing device corresponding to the first pipeline stage. For the first S micro-batches of training data, each micro-batch is divided into two parts based on its batch size and then sequentially input into the computing device corresponding to the first pipeline stage. After completing the calculation of the first part of the training data, the computing device corresponding to the first pipeline stage communicates the intermediate result to the computing device corresponding to the next pipeline stage before starting the calculation of the second part of the training data. After completing the calculation of the second part of the training data, the intermediate result is communicated to the computing device corresponding to the next pipeline stage, and so on, until the first S micro-batches of training data are processed.
6. The method as described in claim 5, characterized in that, During the pipelined processing of the training data in the first S micro-batches, when the computing device corresponding to each pipeline receives the intermediate result transmitted by the computing device corresponding to the previous pipeline, if the pipeline to which the current computing device is located has already performed backward computation, the intermediate result is stored. When the computing device receives the two intermediate results corresponding to the two parts of the training data after splitting the training data of a certain micro-batch in the first S micro-batches of training data, the computation process begins again.
7. A pipelined parallel training system for a natural language processing model, characterized in that, include: The training unit is used to divide each single-layer model in the natural language processing model into a self-attention module and a feedforward neural network module, and to pre-train the natural language processing model to obtain the computation time of each self-attention module and the computation time of each feedforward neural network module. A partitioning unit is used to divide multiple network layers in the natural language processing model into N network layers according to different network layer partitioning methods, so as to obtain multiple model partitioning schemes of the natural language processing model. The simulation unit is used to perform simulation calculations based on the multiple model partitioning schemes, the computation time of each self-attention module, and the computation time of each feedforward neural network module, to obtain the iteration time of the pipelined parallel method corresponding to each model partitioning scheme. The selection unit is used to select the model partitioning scheme with the smallest iteration time of the pipelined parallel method from the multiple model partitioning schemes, and use it as the target model partitioning scheme. The training unit is also used to allocate the N network layers divided by the target model partitioning scheme to N computing devices, and to iteratively train the natural language processing model using the N computing devices in a pipelined parallel manner to obtain a trained natural language processing model. Specifically, when the partitioning unit performs the process of dividing multiple network layers in the natural language processing model into N network layers according to different network layer partitioning methods to obtain multiple model partitioning schemes for the natural language processing model, it is used for: The multiple network layers in the natural language processing model are divided into N network layers to obtain a basic model partitioning scheme. Based on the basic model partitioning scheme, the pipeline level where the critical path of the pipeline is located is obtained; Adjust the number of network layers in the pipeline stage where the critical path of the pipeline is located to obtain the adjusted model partitioning scheme; Based on the adjusted model partitioning scheme, the pipeline level where the critical path of the pipeline is located is re-acquired, and it is determined whether the pipeline level where the critical path of the pipeline is located remains unchanged or moves to a lower pipeline level compared to the pipeline level where the critical path of the pipeline is located before the previous adjustment. If so, the adjusted model partitioning scheme is output, and then the process is returned to re-execute the action of adjusting the number of network layers on the pipeline level where the critical path is located to obtain the adjusted model partitioning scheme.
8. A pipelined parallel training terminal for a natural language processing model, characterized in that, include: Storage media and processor; The storage medium stores computer-executed instructions. The processor executes computer execution instructions stored in the storage medium to implement the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Parallel computing method and device for natural language processing model, equipment and medium
CN114356578A
Large model hybrid parallel training acceleration method and system
CN116755876A
