A neural network pipeline parallel training method and system based on bidirectional segmentation

Abstract

The application discloses a neural network pipeline parallel training method and system based on bidirectional segmentation, and the method comprises the following steps: performing real machine testing on a neural network model to obtain relevant parameters of the neural network model; based on the relevant parameters of the neural network model and hardware constraints of each node in a distributed mode, a bidirectional model segmentation algorithm is used to find an optimal segmentation scheme for minimizing pipeline execution time, and the range of forward propagation and back propagation that need to be calculated on each node is obtained as the result of model segmentation; according to the result of model segmentation, the corresponding part of the neural network model is deployed on each node, and iterative training is performed according to the results of forward propagation and back propagation segmentation until the neural network model converges. The application improves the training efficiency of the pipeline model parallel training system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed computing, and particularly relates to a neural network pipeline parallel training method and system. BACKGROUND

[0002] Artificial intelligence technology has become an important research field in computer science after decades of development. Deep neural networks, driven by the vast amounts of data in the big data era, have promoted great progress in a series of application fields, including image recognition, speech recognition, and machine translation, as an important achievement in the field of artificial intelligence research.

[0003] With the development of deep neural network research, researchers have been constantly proposing deep neural network models with higher precision and stronger expression ability. The trend of the development of these deep neural network models is deeper depth and larger parameter size. For example, in the field of natural language processing, the parameter size of the best-performing model in recent years has grown rapidly from 94 million parameters in 2018 (ELMo), to 1.5 billion parameters in 2019 (GPT-2), to 175 billion parameters in 2020 (GPT-3). As can be seen, the parameter size of deep neural networks has been growing rapidly year by year.

[0004] Although neural networks with increasingly large parameter sizes have brought better performance, they have also made the computational cost of training them higher. Currently, hardware devices such as GPUs (Graphic Processing Unit) are widely used in the model training process of neural networks because they can support fast parallel computing of neural networks. When training a neural network model, the entire model and data are usually loaded into the GPU memory for computation. However, even the latest NVIDIA-A100-80GB graphics card with a memory capacity of 80GB cannot load the 175 billion parameters of GPT-3 with a single graphics card. If the host memory is used to assist in storage for model training, the trained model and data will be frequently exchanged between the host memory and the graphics card memory, which will bring a huge time overhead (theoretically, a single NVIDIA-A100-80GB graphics card needs 288 years to train GPT-3).

[0005] Multi-device collaborative model training can break the limitation of insufficient computing and storage resources of single machine, and collect the computing and storage resources of all devices in a computing cluster, so as to shorten the training time of large-scale neural networks. For example, if the computing capacity of 1000 NVIDIA-A100-80GB can be fully utilized, the training time of GPT-3 will theoretically be reduced to about one month. However, the efficiency of the multi-device collaborative model training system is also limited by the communication resources between devices in the computing cluster, scheduling strategies and other factors, so that the resources of each device cannot be fully utilized. In summary, how to design a distributed machine learning parallel system to schedule the resources of each device so that the system can efficiently train large-scale neural networks is an important research direction.

[0006] Pipeline model parallel training is a paradigm of current distributed machine learning parallel training systems. In the pipeline model parallel training mode, the neural network model is divided by layer into each machine of the training system, each machine has a part of the neural network, and each neural network part forms a logical dependency relationship according to the data flow. The training data is input from the machine that has the input layer of the neural network, and the intermediate result is sent to the device that has the logically next layer after the training task. The training and transmission of intermediate results are performed in a pipeline manner, and the model converges. The advantages of pipeline model parallel are: 1) support large-scale network training: the neural network model is divided into each device, and each device only has part of the model parameters, which reduces the storage requirement; 2) low communication load: only a small amount of intermediate results need to be sent between devices, so the total communication volume of the whole system is independent of the model size. The disadvantage of pipeline model parallel is that it cannot fully utilize the computing resources between devices, because each device only has part of the model, and there is a dependency relationship in computing, so the computing parallelism is poor. In particular, the traditional layer-by-layer segmentation scheme has the inherent defect that the computing load allocated to each distributed node is uneven. Without a good pipeline scheduling strategy, most devices will be in an idle state. SUMMARY

[0007] The purpose of the application is to improve the training efficiency of the neural network model in the pipeline model parallel training system and solve the inherent defects of the traditional layer-by-layer segmentation scheme. The application provides a neural network pipeline parallel training method and system based on bidirectional segmentation, which can obtain an optimal model segmentation result in the model segmentation stage, thereby improving the training efficiency of the pipeline model parallel training system.

[0008] Technical scheme: In order to achieve the above application purpose, the technical scheme of the application is as follows:

[0009] A neural network pipeline parallel training method based on bidirectional segmentation, comprising the following steps:

[0010] The neural network model is tested on a real machine to obtain related parameters of the neural network model;

[0011] Based on the related parameters of the neural network model and the hardware constraints of each node in the distribution, a bidirectional model segmentation algorithm is used to find an optimal segmentation scheme that minimizes the pipeline execution time, and the range of forward propagation and back propagation that needs to be calculated on each node is obtained as the result of model segmentation;

[0012] According to the result of model segmentation, the corresponding part of the neural network model is deployed on each node, and iterative training is performed according to the results of forward propagation and back propagation segmentation until the neural network model converges.

[0013] Wherein, the bidirectional model segmentation algorithm represents the execution time of the forward propagation range from the i1th layer to the j1th layer and the back propagation range from the i2th layer to the j2th layer on m nodes, wherein 1≤i1, j1, i2, j2≤L, L is the maximum value of the number of layers of the neural network, and the problem A(i1, j1, i2, j2, m) is decomposed into two sub-problems A(i1, s1, i2, s2, m-m') and A(s1+1, j1, s2+1, j2, m') by the idea of dynamic programming, wherein the triple (s1, s2, m') is used to segment the mother problem, i1≤s1≤j1, i2≤s2≤j2, 1≤m'≤m, s1 segments the forward calculation range [i1, j1], s2 segments the back calculation range [i2, j2], and m' segments the device quantity.

[0014] The solving formula of the problem A(i1, j1, i2, j2, m) is represented as:

[0015]

[0016] Wherein a s1 represents the size of the activation value output by the forward calculation at s1, and a s2 represents the size of the activation value output by the back calculation at s2.

[0017] Further, the bidirectional model segmentation algorithm comprises:

[0018] When m=1, all legal A(i1, j1, i2, j2, 1) is obtained by directly calculating the calculation time of the layers contained in the forward propagation and back propagation of the device;

[0019] From m = 2, all legal A(i1, j1, i2, j2, m) are calculated one by one, and all legal triplets (s1, s2, m') are traversed, and by the idea of dynamic programming, A(i1, j1, i2, j2, m) is divided into two sub-problems A(i1, s1, i2, s2, m-m') and A(s1+1, j1, s2+1, j2, m') by using the triplet (s1, s2, m'), all triplets (s1, s2, m') are exhausted to find the one that makes the pipeline execution time represented by A(i1, j1, i2, j2, m) shortest, and then the value of the optimal A(i1, j1, i2, j2, m) is recorded as the value of the subsequent sub-problems of the remaining A(i1, j1, i2, j2, m), and the triplet (s1, s2, m') corresponding to the optimal value represents the segmentation point of this time;

[0020] Finally, after A(1, L, 1, L, m) is obtained, the time is the minimum pipeline parallel execution time solved by dynamic programming, and by recursive method, the triplet (s1, s2, m') corresponding to the value of the five-tuple (1, L, 1, L, m) is backtracked, and the value of the triplet corresponding to each value is obtained, that is, the result of the model segmentation scheme.

[0021] A neural network pipeline parallel training system based on bidirectional segmentation, comprising:

[0022] A model evaluation module is configured to perform real machine testing on the neural network model and obtain related parameters of the neural network model.

[0023] A bidirectional model segmentation module is configured to obtain the range of forward propagation and back propagation required to be calculated on each node by a bidirectional model segmentation algorithm based on the related parameters of the neural network model and the hardware constraints of each node in the distributed system, so as to find an optimal segmentation scheme that minimizes the pipeline execution time, and the range of forward propagation and back propagation is taken as the result of model segmentation.

[0024] A pipeline parallel module is configured to deploy the corresponding part of the neural network model on each node according to the result of model segmentation, and perform iterative training according to the results of forward propagation and back propagation segmentation until the neural network model converges.

[0025] Wherein, the bidirectional model segmentation algorithm is represented by A(i1, j1, i2, j2, m) as the execution time of the forward propagation range from the i1th layer to the j1th layer and the backward propagation range from the i2th layer to the j2th layer on m nodes before performing neural network training, wherein 1≤i1, j1, i2, j2≤L, L is the maximum value of the number of neural network layers, and the solving problem A(i1, j1, i2, j2, m) is decomposed into solving two sub-problems A(i1, s1, i2, s2, m-m') and A(s1+1, j1, s2+1, j2, m') by the idea of dynamic programming, wherein the mother problem is segmented by using a triple (s1, s2, m') and i1≤s1≤j1, i2≤s2≤j2, 1≤m'≤m, s1 segments the forward calculation range [i1, j1], s2 segments the backward calculation range [i2, j2], and m' segments the device quantity.

[0026] The solving formula of the problem A(i1, j1, i2, j2, m) is represented as:

[0027]

[0028] Wherein a s1 represents the size of the activation value output by the segmentation at s1 for the forward calculation, a s2 represents the size of the activation value output by the segmentation at s2 for the backward calculation.

[0029] Further, the bidirectional model segmentation module comprises:

[0030] An initialization unit, configured to, when m=1, obtain all legal A(i1, j1, i2, j2, 1) by directly calculating the calculation time of the layers contained in the forward propagation and the backward propagation of the device;

[0031] An iterative segmentation unit, configured to, starting from m=2, calculate all legal A(i1, j1, i2, j2, m) one by one, and traverse all legal triples (s1, s2, m'), and segment A(i1, j1, i2, j2, m) by using the triple (s1, s2, m') to obtain two sub-problems A(i1, s1, i2, s2, m-m') and A(s1+1, j1, s2+1, j2, m') by the idea of dynamic programming, find the triple that makes the pipeline execution time represented by A(i1, j1, i2, j2, m) shortest among all triples (s1, s2, m'), and then record the value of the optimal A(i1, j1, i2, j2, m) as the value of the subsequent solving of the sub-problems of the remaining A(i1, j1, i2, j2, m), and record the triple (s1, s2, m') corresponding to the optimal value to represent the segmentation point this time;

[0032] Backtracking calculation unit, for after getting A(1, L, 1, L, m), the time is the minimum time of pipeline parallel execution time after solving by dynamic programming, through the way of recursion, the five-tuple (1, L, 1, L, m) corresponding to the value is backtracked, the value of the three-tuple (s1, s2, m') corresponding to each value is obtained, that is, the result of the model segmentation scheme.

[0033] Compared with the prior art, the present application has the following advantages and beneficial effects: the present application breaks the model segmentation scheme of layer segmentation in traditional pipeline parallel, and designs an algorithm based on bidirectional model segmentation according to the characteristics of neural network training being divided into forward propagation and backward propagation two kinds of calculation. The algorithm divides forward propagation and backward propagation respectively. Finally, the optimal bidirectional segmentation scheme of the target neural network is found in polynomial time using the idea of dynamic programming, so that the load of the calculation task segmented by each node is more balanced, to achieve the purpose of realizing pipeline system training acceleration. The final acceleration effect can be accelerated by nearly 1.9 times compared with the traditional way. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a bidirectional model segmentation system schematic diagram according to the present application;

[0035] Figure 2 is a bidirectional model segmentation example according to the embodiment of the present application;

[0036] Figure 3 is a traditional layer model segmentation 1F1B pipeline parallel schematic diagram in the embodiment of the present application;

[0037] Figure 4 is a bidirectional model segmentation 1F1B pipeline parallel schematic diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical scheme of the present application will be further described below in combination with the drawings.

[0039] Figure 1As shown in the schematic diagram of the bidirectional model segmentation system of the application, the system comprises three modules, (1) a model evaluation module: the evaluation module will perform real machine testing on the target neural network, and obtain relevant parameters for model segmentation by the model segmentation algorithm, such as the calculation time of each layer, the parameter size and the activation value size; (2) a bidirectional model segmentation module: according to the relevant data of the target neural network obtained by the evaluation module, and the hardware constraints of the distributed nodes, such as the bandwidth size, the storage capacity constraints, through the bidirectional model segmentation algorithm, the forward propagation and the backward propagation range required to be calculated on each node are finally obtained as the result of model segmentation; (3) a pipeline parallel module: using the model segmentation result of the bidirectional model segmentation module, deploying the corresponding part of the model on each node, and then starting to train the entire neural network using the 1F1B strategy according to the training order, and because the model segmentation is bidirectional, the forward propagation and the backward propagation calculation of a layer of the neural network are allowed not to be on the same device, according to the respectively segmented results of the forward propagation and the backward propagation, iterative training is performed, and finally the model converges.

[0040] Correspondingly, the neural network pipeline parallel training method based on bidirectional segmentation of the application comprises the following steps:

[0041] S1, performing real machine testing on the target neural network, and obtaining relevant parameters for model segmentation by the model segmentation algorithm, such as the calculation time of each layer, the parameter size and the activation value size;

[0042] S2, based on the relevant parameters of the target neural network, and the hardware constraints of the distributed nodes, such as the bandwidth size, the storage capacity constraints, through the bidirectional model segmentation algorithm which takes finding an optimal segmentation scheme to minimize the pipeline execution time as the target, the forward propagation and the backward propagation range required to be calculated on each node are obtained as the result of model segmentation;

[0043] S3, according to the bidirectional model segmentation result, deploying the corresponding part of the model on each node, and then training the entire neural network using the 1F1B strategy according to the training order, according to the respectively segmented results of the forward propagation and the backward propagation, iterative training is performed, and finally the model converges.

[0044] The real machine testing refers to using a dataset to train the neural network on a device to collect the relevant parameters of the neural network.

[0045] According to an embodiment of the present invention, bidirectional model segmentation separately divides the forward propagation and backward propagation calculations. Thus, the final result obtained by the bidirectional model segmentation algorithm consists of the forward propagation range and the backward propagation range at each node, and these ranges represent consecutive layers in the neural network. Assuming a deep neural network model consists of L layers, and the forward propagation calculation for each layer is represented by F... l This indicates that the backpropagation calculation for each layer uses B. l Let D be the set of nodes, consisting of several devices, each device D. i After the bidirectional model segmentation algorithm, the forward propagation range will be obtained from F. i1 To F j1 The reverse propagation range is from B i2 To B j2 These two ranges represent D in the subsequent pipeline parallel modules. i The node should perform forward and backward propagation computations. Furthermore, for any node D... i Neither the forward propagation range nor the backward propagation range can overlap with other nodes, and both must satisfy the following constraints:

[0046]

[0047]

[0048] in and Representing the previous node D respectively i-1 The right endpoints of the forward and backward propagation ranges, and This indicates the next node D. i+1 The left endpoints of the forward and backward propagation ranges are defined. Clearly, when all nodes satisfy i1 = i2 and j1 = j2, meaning the forward and backward propagation computation ranges on each node are consistent, the result of the bidirectional model partitioning method is equivalent to the traditional layer-by-layer partitioning method. Therefore, the traditional layer-by-layer method can also be considered a special case of the bidirectional model partitioning method. Intuitively, the bidirectional model partitioning method allows for the exploration of more possible partitioning results, thus enabling the discovery of a partitioning scheme that improves the parallel efficiency of the pipeline model.

[0049] According to an embodiment of the present invention, the goal of the bidirectional model segmentation algorithm is to find an optimal segmentation scheme that minimizes pipeline execution time. Let A(i1, j1, i2, j2, m) represent the forward propagation range of training a deep neural network (DNN) on m nodes, with a range of F. i1 To F j1 The back propagation range is B. i2 To B j2the execution time of the forward propagation of the target DNN, where 1≤i1, j1, i2, j2≤L (L is the maximum number of layers of the target DNN), [i1, j1] represents the range of layers for which forward propagation is performed, from the i1th layer to the j1th layer, and [i2, j2] represents the range of layers for which backward propagation is performed, from the i2th layer to the j2th layer, then we have:

[0050]

[0051] By the idea of dynamic programming, the solution of the mother problem A(i1, j1, i2, j2, m) is decomposed into two sub-problems A(i1, s1, i2, s2, m-m') and A(s1+1, j1, s2+1, j2, m'), where the triplet (s1, s2, m') is used to partition the mother problem, where i1≤s1≤j1, i2≤s2≤j2, 1≤m'≤m, s1 partitions the forward calculation range [i1, j1], s2 partitions the backward calculation range [i2, j2], and m' partitions the number of devices, then the state transition of the dynamic programming equation can be obtained from the above formula, where the max term is obtained by taking the maximum of the optimal solution of the two sub-problems and an intermediate transmission time (the intermediate transmission time is obtained from (transmission amount / bandwidth), and for forward calculation, the size of the output activation value is a s1 , and similarly, the size of the output activation value of the backward propagation is a s2 ), and the maximum is taken.

[0052] The specific process of the bidirectional model partitioning algorithm is as follows:

[0053] 1) Input: For each layer l of the target neural network, the forward propagation calculation time T , the backward propagation calculation time T l b , the model parameter size w l , and the activation output size a l need to be obtained through the evaluation module. At the same time, the bandwidth B between nodes, the storage capacity C of the nodes, and the number of nodes M also need to be measured.

[0054] 2) Initialization: When m=1, i.e., when the number of devices is 1, all the legal A(i1, j1, i2, j2, 1) can be obtained by directly calculating the calculation time of the layers involved in the forward propagation and the backward propagation.

[0055] 3) from m = 2, calculate all legal A(i1, j1, i2, j2, m) one by one (legal means that i1> j1 and i2> j2 cannot appear, that is, the left end point of the interval exceeds the right end point) and traverse all legal triplets (s1, s2, m') (legal means that i1≤s1≤j1, i2≤s2≤j2, 1≤m'≤m are satisfied), by the idea of dynamic programming, using triplet (s1, s2, m') to divide A(i1, j1, i2, j2, m) to obtain two subproblems A(i1, s1, i2, s2, m-m') and A(s1+1, j1, s2+1, j2, m'), all triplets (s1, s2, m') are exhausted to find the triplet that makes the pipeline execution time represented by A(i1, j1, i2, j2, m) shortest, then record the value of the optimal A(i1, j1, i2, j2, m) as the value of the subsequent solving of the subproblem of the remaining A(i1, j1, i2, j2, m), and record the triplet (s1, s2, m') corresponding to the optimal value to represent the segmentation point for finally obtaining the segmentation result of each node.

[0056] 4) Finally, after A(1, L, 1, L, m) is obtained, the time is the minimum time of the pipeline parallel execution time solved by dynamic programming, and the value corresponding to the five-tuple (1, L, 1, L, m) is backtracked in a recursive manner, and since each A(i1, j1, i2, j2, m) records a unique triplet (s1, s2, m') when solving the minimum value, representing two subproblems A(i1, s1, i2, s2, m-m') and A(s1+1, j1, s2+1, j2, m'), the two subproblems continue to be segmented using the corresponding triplets, until the fifth dimension of the five-tuple A is 1. It is equivalent to a binary tree that is continuously split, and finally all leaf nodes represent the range of forward calculation and reverse calculation possessed by a single device, which is the result of the final model segmentation scheme.

[0057] According to the facility mode of the application, the 1F1B pipeline parallel strategy is used in the segmented model training, the node quantity batch data is input at the same time, the node in the pipeline performs a reverse calculation immediately after a forward calculation, and the batch data of the two calculations is different. Through the 1F1B node scheduling strategy, in the stable stage, the calculation of each device in the pipeline can reach full load, that is, there is no idle device.

[0058] Correspondingly, the bidirectional model segmentation module comprises:

[0059] An input unit is configured to input parameters, including the forward propagation calculation time T of each layer l of the target neural network lf Backpropagation computation time T l b Model parameter size w l And activate output size a l And the bandwidth between nodes B, the storage capacity of nodes C, and the number of nodes M;

[0060] An initialization unit is used to obtain all valid A(i1, j1, i2, j2, 1) by directly calculating the computation time of the layers involved in the forward and backward propagation of the device when m = 1.

[0061] The iterative partitioning unit, starting from m=2, calculates all valid A(i1, j1, i2, j2, m) one by one, and traverses all valid triples (s1, s2, m′). Using dynamic programming, it partitions A(i1, j1, i2, j2, m) using the triples (s1, s2, m′) to obtain two subproblems A(i1, s1, i2, s2, mm′) and A(s1+1, j1, s2+1, j2, ...). Enumerate all triples (s1, s2, m′) to find the triple that represents A(i1, j1, i2, j2, m) with the shortest pipeline execution time. Then record the value of the optimal A(i1, j1, i2, j2, m) as the value for solving the subproblems of the remaining A(i1, j1, i2, j2, m) in the subsequent calculation. Record the triple (s1, s2, m′) corresponding to the optimal value as the split point of this step.

[0062] The backtracking computation unit is used to backtrack after obtaining A(1, L, 1, L, m), which is the minimum time for pipeline parallel execution after dynamic programming. By recursively backtracking the quintuple (1, L, 1, L, m) corresponding to this value, the value of the triple (s1, s2, m′) corresponding to each value is obtained, which is the result of the model partitioning scheme.

[0063] like Figure 2 As shown in this embodiment of the invention, a neural network with four layers is trained on three machines. First, the forward propagation and backpropagation computation times for each layer of the neural network can be obtained through single-machine testing. Here, we assume the forward propagation computation time for each layer is F = {1, 3, 2, 3}. For generality, the backpropagation computation time for a typical layer in a neural network is twice the forward propagation time, so the backpropagation computation time here is B = {2, 6, 4, 6}.

[0064] Using the traditional layer-wise segmentation algorithm, the tasks assigned to the same machine are in units of layers, and when executing, the forward propagation and back propagation calculations represented by the layer need to be performed, so the optimal segmentation scheme obtained by the algorithm is that device 1 has 1-2 layers, device 2 has 3 layers, and device 3 has 4 layers. When the final execution pipeline parallel is performed through this segmentation scheme, it is obvious that the training load on each machine can be calculated as {12, 6, 9}. Figure 3 It can be seen that the calculation task on each machine at each moment during the pipeline parallel training of this segmentation scheme. It can be seen that the execution time of the pipeline training is limited by the load 12 of the most heavily loaded machine device 1. The other machine devices 2 and 3 will appear gray blocks-idle time in the stable stage of pipeline parallel, indicating that the device is idle and not fully utilizing its computing resources.

[0065] Using the bidirectional model segmentation algorithm, with the same example, the optimal segmentation scheme finally obtained is that device 1 has {F1} and {B1, B2}, device 2 has {F2, F3} and {B3}, and device 3 has {F4} and {B4}. The calculated load on each machine is {9, 9, 9}, obviously the calculation load of each machine in the pipeline is completely uniform, Figure 4 That is, the calculation graph of the pipeline parallel training using the segmentation result can be seen. In the stable stage, each machine is full, that is, there is no idle time to waste computing resources.

[0066] Finally, through this example, it can also be seen that the advantage of the bidirectional model segmentation algorithm is that it can balance the calculation load of each node in the pipeline, thereby achieving the purpose of fully utilizing the computing resources and improving the training efficiency. Figure 4 The part of the model that each machine has after using the bidirectional model segmentation algorithm in the previous example is also described. It is worth noting that device 1 only performs the back propagation calculation of layer 2, and device 2 only performs the forward propagation calculation of layer 2, so layer 2 is the overlapping layer. Using this overlapping layer, the forward propagation and back propagation calculations of the overlapping layer can be performed on different machines, thereby making the result of the entire model segmentation more uniform in load.

Claims

1. A method for training a neural network pipeline in parallel based on bidirectional segmentation, characterized in that, The method comprises the following steps: The neural network model is tested on a real machine to obtain related parameters of the neural network model; Based on the related parameters of the neural network model and the hardware constraints of the distributed nodes, a bidirectional model segmentation algorithm is used to find an optimal segmentation scheme that minimizes the pipeline execution time, and the range of forward propagation and back propagation that need to be calculated on each node is obtained as the result of model segmentation; the bidirectional model segmentation algorithm uses A(i1,j1,i2,j2,m) to represent the execution time of the forward propagation range from the i1th layer to the j1th layer and the back propagation range from the i2th layer to the j2th layer on m nodes, wherein 1≤i1,j1,i2,j2≤L, L is the maximum value of the number of neural network layers, and the problem A(i1,j1,i2,j2,m) is decomposed into two sub-problems A(i1,s1,i2,s2,m-m') and A(s1+1,j1,s2+1,j2,m') by using the dynamic programming idea, wherein the mother problem is segmented by using a triple (s1,s2,m') to segment the forward calculation range [i1,j1], the back calculation range [i2,j2], and the number of devices m', wherein i1≤s1≤j1, i2≤s2≤j 2, 1≤m′≤m, s1 segments the forward calculation range [i1,j1], s2 segments the back calculation range [i2,j2], and m' segments the number of devices. According to the model segmentation result, the corresponding part of the neural network model is deployed on each node, and iterative training is performed according to the forward propagation and backward propagation respectively segmented results until the neural network model converges.

2. The method of claim 1, wherein, The solution formula of the problem A(i1, j1, i2, j2, m) is expressed as: where a s1 represents the size of the activation value split output at s1 for forward computation, a s2 represents the size of the activation value split output at s2 for backward computation.

3. The method of claim 2, wherein, The bidirectional model segmentation algorithm comprises: When m = 1, all legal A(i1, j1, i2, j2, 1) are obtained by directly calculating the calculation time of the layers contained in the forward propagation and backward propagation of the computing device; From m = 2, all legal A(i1, j1, i2, j2, m) are calculated one by one, and all legal triplets (s1, s2, m') are traversed, and the idea of dynamic programming is used to divide A(i1, j1, i2, j2, m) into two sub-problems A(i1, s1, i2, s2, m-m') and A(s1+1, j1, s2+1, j2, m') by using the triplet (s1, s2, m'), all triplets (s1, s2, m') are exhausted to find the triplet that makes the pipeline execution time represented by A(i1, j1, i2, j2, m) shortest, and then the value of the optimal A(i1, j1, i2, j2, m) is recorded as the value of the subsequent solution of the remaining A(i1, j1, i2, j2, m) sub-problems, and the triplet (s1, s2, m') corresponding to the optimal value represents the segmentation point; Finally, after A(1, L, 1, L, m) is obtained, the time is the minimum time of the pipeline parallel execution time solved by dynamic programming, and the value of the five-tuple (1, L, 1, L, m) corresponding to the value is backtracked by using the recursive method, and the value of the triplet corresponding to each value is obtained, that is, the result of the model segmentation scheme.

4. The method of claim 1, wherein, The neural network model is trained using the 1F1B strategy, and the batch data of the input nodes are input at the same time. The nodes in the pipeline perform forward calculation and reverse calculation immediately after one forward calculation. 5.A neural network pipeline parallel training system based on bidirectional segmentation, characterized in that, It comprises: The model evaluation module is configured to test the neural network model on a real machine to obtain related parameters of the neural network model; The bidirectional model partitioning module is configured to derive the forward propagation and the backward propagation range to be calculated on each node based on the related parameters of the neural network model and the hardware constraints of the distributed nodes by using a bidirectional model partitioning algorithm which aims to find an optimal partitioning scheme that minimizes the pipeline execution time, as a result of the model partitioning; the bidirectional model partitioning algorithm uses A(i1, j1, i2, j2, m) to represent the execution time of the forward propagation range from the i1th layer to the j1th layer and the backward propagation range from the i2th layer to the j2th layer on m nodes for performing the neural network training, where 1≤i1, j1, i2, j2≤L, L is the maximum value of the number of layers of the neural network, and the solving problem A(i1, j1, i2, j2, m) is decomposed into solving two sub-problems A(i1, s1, i2, s2, m-m') and A(s1+1, j1, s2+1, j2, m') by using the dynamic programming idea, where the triplet (s1, s2, m') is used to partition the parent problem, i1≤s1≤j1, i2≤s2≤j 2, 1≤m'≤m, s1 partitions the forward calculation range [i1, j1], s2 partitions the backward calculation range [i2, j2], and m' partitions the number of devices The pipeline parallel module is configured to deploy the corresponding part of the neural network model on each node according to the model segmentation result, and perform iterative training according to the forward propagation and backward propagation respectively segmented results until the neural network model converges.

6. The system of claim 5, wherein, The solution formula of the problem A(i1, j1, i2, j2, m) is expressed as: where a s1 represents the size of the activation value split output at s1 for forward computation, a s2 represents the size of the activation value split output at s2 for backward computation.

7. The system of claim 6, wherein, The bidirectional model segmentation module comprises: The initialization unit is configured to, when m = 1, obtain all legal A(i1, j1, i2, j2, 1) by directly calculating the calculation time of the layers contained in the forward propagation and backward propagation of the computing device; The iterative segmentation unit is configured to start from m=2, calculate all legal A(i1, j1, i2, j2, m) one by one, and traverse all legal triplets (s1, s2, m′), and by using the triplet (s1, s2, m′) to segment A(i1, j1, i2, j2, m) to obtain two sub-problems A(i1, s1, i2, s2, m-m′) and A(s1+1, j1, s2+1, j2, m′), all triplets (s1, s2, m′) are exhausted to find one triplet that makes the pipeline execution time represented by A(i1, j1, i2, j2, m) shortest, and then the value of the optimal A(i1, j1, i2, j2, m) is recorded as the value of the subsequent solving of the remaining A(i1, j1, i2, j2, m) sub-problems, and the triplet (s1, s2, m′) corresponding to the optimal value represents the segmentation point of this time; The backtracking calculation unit is configured to, after obtaining A(1, L, 1, L, m), the time is the minimum time of the pipeline parallel execution time solved by dynamic programming, and by using the recursive manner, the value of the triplet corresponding to the five-tuple (1, L, 1, L, m) is obtained, that is, the result of the model segmentation scheme.

8. The system of claim 5, wherein, The neural network model is trained by using the 1F1B strategy, and the number of input nodes is batch data, and the nodes in the pipeline perform a reverse calculation immediately after a forward calculation is completed, and the batch data of the two calculations is different.

Images