A dynamic model-level scheduling method for synchronous pipeline model parallelism

Abstract

The present invention discloses a dynamic model layer scheduling method for synchronous pipeline model parallelism, including in the forward calculation stage, model training information is transmitted forward to the end node of the pipeline; the model training information includes model layer information, current time slot model partitioning, and node status information; in the backward calculation stage, the end node of the pipeline predicts and updates the global node status information based on a dynamic prediction algorithm; the end node of the pipeline takes minimizing the training time of the synchronous pipeline parallel system as the goal, establishes an optimization problem model, calculates the optimal partitioning and transmits the result backward to all upstream nodes; the nodes participating in the layer migration call the layer manager to complete the migration preparation work, and migrate the model layer after the parameter update is completed, and the forward calculation stage of the next batch is started when the layer scheduling is completed. The present invention avoids the additional overhead of monitoring by designing a scheduling method, provides dynamic prediction capabilities, and makes it better adapt to the dynamic changes of resources.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a dynamic model layer scheduling method for synchronous pipeline model parallelism. Background Art

[0002] Distributed training is the mainstream way to train large-scale deep learning models. It divides large data sets and complex models into multiple nodes or GPUs, and uses multi-machine computing resources to accelerate the model training process. Distributed training architecture is divided into data parallelism and model parallelism. Data parallelism divides large data sets into multiple devices for parallel processing to accelerate the training process, but requires each node to store a complete copy of the model, which is not suitable for ultra-large-scale model training. In contrast, model parallelism can break through the model capacity limit of single-machine training and achieve better scalability. It is an effective method for training ultra-large-scale models. Model parallelism divides complex models into multiple partitions and assigns them to different nodes. Different nodes are only responsible for the training and parameter updates of local model subsets. Model parallelism is divided into pipeline parallelism and tensor parallelism, among which pipeline parallelism is further divided into synchronous pipeline parallelism and asynchronous pipeline parallelism.

[0003] The synchronous pipeline parallel system divides the input batch into smaller micro-batches and injects the micro-batches into the pipeline for training in sequence. The training process of a micro-batch includes forward calculation, backward calculation and parameter update. In order for the synchronous pipeline parallel system to achieve the best training efficiency, the model needs to be divided reasonably. However, in practical applications, due to the heterogeneity and time-varying nature of the available resources in the distributed cluster, the GPU computing power, bandwidth, memory and other resources of each node are usually different and change over time. Therefore, the optimal model partitioning needs to be adjusted dynamically. At this time, the model partition can be changed through model layer scheduling to achieve load balancing between nodes.

[0004] Existing model layer scheduling load balancing methods are difficult to adapt to dynamic training environments where bandwidth / computing power / memory frequently changes, and there are problems such as insufficient dynamic prediction capabilities and excessive layer scheduling overhead. Taking the Vpipe architecture as an example, Vpipe's layer migration strategy requires hardware to monitor GPU utilization, bandwidth, and memory in real time. Frequent monitoring will bring additional processing overhead to the CPU and GPU; at the same time, Vpipe uses a naive method to estimate the future value of computing power and bandwidth. In a specific training environment, task preemption and network traffic changes are usually trendy, seasonal, and cyclical. The naive method estimation cannot adapt to the dynamic changes of available resources; in addition, Vpipe's layer migration events are triggered in units of a single micro-batch execution, and layer migration is performed during the forward / backward calculation execution process. Although this real-time layer migration strategy can respond quickly to resource changes, it needs to migrate model layer parameters and forward / backward calculation intermediate results at the same time, and the latter is usually much larger than the former, which will lead to excessive communication overhead and even cause pipeline stagnation. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, a dynamic model layer scheduling method for synchronous pipeline model parallelism is adopted to solve the problems raised in the above background technology.

[0006] A dynamic model layer scheduling method for synchronous pipeline model parallelism, the scheduling method includes a forward calculation stage and a backward calculation stage, and includes the following steps:

[0007] Step S1, in the forward calculation stage, the model training information is transmitted forward to the end node of the pipeline; the model training information includes model layer information, current time slot model division, and node status information;

[0008] Step S2: In the backward calculation stage, the end node of the pipeline predicts and updates the global node state information based on the dynamic prediction algorithm;

[0009] Step S3, the pipeline end node takes minimizing the training time of the synchronous pipeline parallel system as the goal, establishes an optimization problem model, calculates the optimal partition and passes the result backward to all upstream nodes;

[0010] Step S4: The nodes participating in the layer migration call the layer manager to complete the migration preparation work, and migrate the model layer after the parameter update is completed. When the layer scheduling is completed, the forward calculation phase of the next batch begins.

[0011] As a further solution of the present invention: the specific steps in step S1 include:

[0012] Step S11, define model layer information:

[0013] Set the total number of layers of the training model to The total number of nodes is , the number of samples in a batch is , the number of micro-batch samples injected into the pipeline is The model layer information consists of three parts, including layer calculation load, adjacent layer communication load and layer parameter size, respectively. , express Time The forward and backward computation loads of the layer are calculated using Indicates The model parameter size of the layer, where ; and Expressed as the number of floating point operations; express Time Layer and Communication load between layers, where ;

[0014] when At that time, The total forward computation load, the total backward computation load, and the Layer and The total communication load between layers is is proportional to;

[0015] Step S12, defining the model division of time slot t;

[0016] The training process of a synchronous pipeline for an input batch is one time slot, including forward calculation, backward calculation, and parameter update;

[0017] According to the model partitioning scheme of different time slots, use the partition point array represents the model partitioning of time slot t, where Representation Node The largest layer number among the model layers stored locally at time slot t; the model partitioning at time slot t is obtained by the best partitioning result at time slot t-1;

[0018] Step S13, each node samples node status information in micro batches in the forward phase of each time slot;

[0019] Node status information includes the node's memory capacity, computing power, and communication capabilities; , Respectively represent nodes memory capacity and computing power, including ;use Representation Node With Node The communication capability between ;

[0020] Each node will sample node status information in micro batches in the forward phase of each time slot; Train the micro-batch at time slot t ,in is the sequence number of the micro-batch in all micro-batches, ; After completing the forward training of micro-batch k, node Estimation by sampling , and The sampling values ​​are , and ;

[0021] in, Obtained through hardware query, and The formula is obtained by using the following equation:

[0022] ;

[0023] ;

[0024] in, ;

[0025] In the formula, Defined as a pair of nodes The sum of some model layer information in the model layer stored in time slot t, Calculates forward load for the layer or backward calculation load , Adjacent layer communication load and layer parameter size Any one of; , They respectively represent the time it takes for a node to complete the forward computation and communication of the current micro-batch;

[0026] Step S14: except for the node at the end of the pipeline In addition, after each micro-batch forward calculation is completed, the remaining nodes integrate the model training information of this node and all upstream nodes, and send it to the next node together with the intermediate activation value of the forward calculation; get the node Model training information, which includes time slot t nodes Model division position , local model layer information, and the node status information sampling value obtained after completing the current micro-batch;

[0027] When the forward calculation phase of a whole batch is completed, the end node of the pipeline further transforms the obtained node state information sampling value to obtain the node Node status information during the entire training time slot t , and , the transformation formula is:

[0028] ;

[0029] ;

[0030] ;

[0031] In the formula, and The values ​​reflect the entire time slot Midpoint Computing power and communication capabilities The average value of Reflects the entire time slot The middle node Memory capacity Lower limit;

[0032] The end nodes are integrated to obtain the global model training information, which includes the model partitioning of time slot t , model layer information of all layers, and node status information of all nodes in time slot t.

[0033] As a further solution of the present invention: the specific steps in step S2 include:

[0034] Step S21: The end node always maintains a historical sequence and predicted value for the computing power and communication power of each node; the end node is defined as the node state before the end of the forward phase of time slot t The maintained history sequence is:

[0035] ;

[0036] in, yes or Any one of It is a historical sequence Length, Reserved forward The historical value of the time slot; the end node is also Maintain a forecast value ; Calculated at time slot t-1, used to predict The value of

[0037] When time slot t=1, the pipeline system is training the first input batch. At this time, there is no historical sequence of node status information, and no maintenance is performed on The predicted value of

[0038] Step S22: After the forward phase of time slot t ends, the state of any node , the terminal node updates its history sequence length based on the prediction loss , and update the historical sequence synchronously ; The value of is obtained according to the following equation:

[0039] , ;

[0040] ;

[0041] In the formula, is the rounding operation. In addition, when t=1, ; The updated historical sequence is:

[0042] ;

[0043] Step S23: factorize the historical sequence based on the additive mode factorization method Decomposition into trend series , seasonal series , and the residual sequence sum;

[0044] ;

[0045] Step S24: End node trend sequence and seasonal series The autoregressive moving average model is applied respectively, the order of the ARIMA model is determined according to the minimum Bayesian information criterion, and the trend forecast value is estimated using the optimal forecast model and seasonal forecasts ; For node status The predicted value of is obtained from the following equation:

[0046] ;

[0047] in, and Node status The upper and lower bounds of .

[0048] As a further solution of the present invention: the specific steps in step S3 include:

[0049] Step S31, establishing an optimization problem model with the goal of minimizing the completion time of the synchronous pipeline parallel system;

[0050] After obtaining the model training information in step S1 and dynamically predicting the node status information in step S2, the total time for the synchronous pipeline model parallel architecture to complete a complete batch training is expressed as:

[0051] ;

[0052] in,

[0053]

[0054]

[0055]

[0056] In the formula, , and Respectively represent nodes The forward computation time, backward computation time, and node +1 single communication time; node status information in time slot t 、 The completion time of a synchronous pipeline system depends only on the model partitioning. ;

[0057] The optimization problem of minimizing completion time is defined as follows:

[0058] ;

[0059]

[0060]

[0061] The first constraint is the partitioning position constraint, which requires that L model layers be allocated to N nodes and each node be allocated at least one layer. The second constraint is the memory constraint, which ensures that each node does not have a memory overflow error during training.

[0062] Step S32: Solve the original optimization problem and use the improved KL community discovery algorithm to solve the optimal model partitioning strategy to obtain the optimal partitioning strategy. ;

[0063] Step S33: Starting from the end node of the pipeline, the optimal partitioning result obtained in step S32 is passed backward to each upstream node together with the gradient result of the backward calculation.

[0064] As a further solution of the present invention: the specific steps in step S4 include:

[0065] Step S41, calculate the optimal division of time slot t The current division with time slot t The partition difference array of , , whose partition difference array is given by the following equation:

[0066] ;

[0067] in, , ;

[0068] Step S42, each node generates a layer scheduling strategy;

[0069] like , then node i+1 sends to node i Layers;

[0070] like , then node i sends to node i+1 Layers;

[0071] like , then there is no need to migrate the model layer between node i and node i+1;

[0072] All nodes are divided into different arrays according to Generate the layer scheduling strategy of this node; if this node wants to receive the model layer, execute step S43; if it wants to send the model layer, execute step S44; a node can execute step S43 and step S44 at the same time; if no operation is required, jump to step S45;

[0073] Step S43: The node that needs to receive the layer calls the layer manager to initialize the model layer, allocates memory space for the new layer, and waits for the migration request of the source node; after receiving the request, it returns the request confirmation and establishes a connection with the source node, and stores the received model layer parameters in the specified memory; after the layer reception is completed, it returns the layer migration completion signal, disconnects the connection, and jumps to step S45;

[0074] Step S44: The node that needs to send the layer sends a layer migration request after the parameters are updated and waits for confirmation; after receiving the confirmation, it establishes a connection with the layer receiving node and calls the layer manager to transfer the model layer parameters; after the model layer parameters are sent and the layer migration completion signal is received, it disconnects, releases the memory of the invalid layer, and jumps to step S45;

[0075] Step S45, after the layer scheduling of all nodes is completed, the model division at this time is the optimal division of time slot t, and the synchronous pipeline system starts the forward calculation of time slot t+1.

[0076] Compared with the prior art, the present invention has the following technical effects:

[0077] The above technical solution is adopted to divide the model training into forward and backward stages. In the forward stage, the training information is transmitted to the end node; in the backward stage, the end node predicts and updates the global state, optimizes the training time of the parallel system, and transmits the results backward. After the nodes participating in the layer migration complete the preparation, the model layer is migrated and the next batch of forward calculations is started. In this way, load balancing of synchronous pipeline parallel model training is achieved with low scheduling overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0079] Figure 1A schematic diagram of the steps of the scheduling method of the embodiment disclosed in this application;

[0080] Figure 2 This is a diagram of the synchronous pipeline parallel model training architecture of the embodiment disclosed in this application;

[0081] Figure 3 A schematic diagram of a layer scheduling process for dividing a synchronous pipeline optimization model according to an embodiment disclosed in the present application;

[0082] Figure 4 A schematic diagram of the migration process of adjacent node layers of a synchronous pipeline according to an embodiment disclosed in the present application. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] Please refer to Figure 1 In an embodiment of the present invention, a dynamic model layer scheduling method for synchronous pipeline model parallelism is provided, wherein the scheduling method includes a forward calculation phase and a backward calculation phase, and includes the following steps:

[0085] In this embodiment, the basic architecture of deep learning synchronous pipeline parallel model training is defined as follows:

[0086] Synchronous pipeline parallel splits the model by layer, dividing the deep learning model into multiple partitions and assigning them to different nodes. Each partition is a set of continuous layers. The pipeline parallel system divides each batch input to the head-end node into smaller micro-batches and injects the micro-batches into the pipeline sequentially. The training process of a synchronous pipeline for an input batch includes three stages: forward calculation, backward calculation, and parameter update. The whole process is also called a time slot.

[0087] The initialization process of model training information is as follows: In the pre-training phase, the model is manually divided into N nodes, and then a benchmark phase is started to set And perform forward and backward calculation processes for dozens of micro-batches. Record the number of floating-point operations in the forward / backward stages of each layer, the number of model parameters, and the size of the intermediate calculation results of each layer to obtain the model layer information; record the maximum layer number of the model layer contained in each node in the current time slot to obtain the model division of the current time slot; record the calculation time and communication time of each node to calculate the node computing power and communication power, and query the system hardware information on the device to obtain the memory capacity of the current time slot.

[0088] Step S1: In the forward calculation stage, the model training information is transmitted forward to the end node of the pipeline; the model training information includes model layer information, current time slot model division, and node status information. The specific steps include:

[0089] Step S11, define model layer information:

[0090] Set the total number of layers of the training model to The total number of nodes is , the number of samples in a batch is , and the number of micro-batch samples injected into the pipeline is , model layers are numbered from 1, and nodes are numbered from 0. The model layer information consists of three parts, including layer calculation load, adjacent layer communication load and layer parameter size, respectively. , express Time The forward and backward computation loads of the layer are calculated using Indicates The model parameter size of the layer, where ; and Expressed as the number of floating point operations; express Time Layer and Communication load between layers, where , and is proportional to;

[0091] When time slot t=1, the pipeline system is training the first input batch. At this time, there is no historical sequence of node status information, and no maintenance is performed on The predicted value of

[0092] Step S12, defining the model division of time slot t;

[0093] The training process of a synchronous pipeline for an input batch is one time slot, including forward calculation, backward calculation, and parameter update;

[0094] According to the model partitioning scheme of different time slots, use the partition point array represents the model partitioning of time slot t, where Representation Node The largest layer number among the model layers stored locally at time slot t; the model partitioning at time slot t is obtained by the best partitioning result at time slot t-1;

[0095] Step S13, each node samples node status information in micro batches in the forward phase of each time slot;

[0096] Node status information includes the node's memory capacity, computing power, and communication capabilities; , Respectively represent nodes memory capacity and computing power, including ;use Representation Node With Node The communication capability between ;

[0097] Each node will sample node status information in micro batches in the forward phase of each time slot; Train the micro-batch at time slot t ,in is the sequence number of the micro-batch in all micro-batches, ; After completing the forward training of micro-batch k, node Estimation by sampling , and The sampling values ​​are , and ;

[0098] in, Obtained through hardware query, and The formula is obtained by using the following equation:

[0099] ;

[0100] ;

[0101] in, ;

[0102] In the formula, Defined as a pair of nodes The sum of some model layer information in the model layer stored in time slot t, Calculates forward load for the layer or backward calculation load , Adjacent layer communication load and layer parameter size Any one of; , They respectively represent the time it takes for a node to complete the forward computation and communication of the current micro-batch;

[0103] Step S14: except for the node at the end of the pipeline In addition, after each micro-batch forward calculation is completed, the remaining nodes integrate the model training information of this node and all upstream nodes, and send it to the next node together with the intermediate activation value of the forward calculation; get the node Model training information, which mainly includes time slot t node Model division position , local model layer information, and the node status information sampling value obtained after completing the current micro-batch;

[0104] like Figure 2 As shown in the figure, in the forward calculation phase, if a node is neither the head node nor the end node, it always loops through the following four steps:

[0105] a. Receive the intermediate activation value and model training information of the micro-batch forward calculation output by the previous node. The received model training information contains the model training information of all upstream nodes.

[0106] b. Perform forward computation on the same micro-batch based on the received forward computation intermediate activation value

[0107] c. After the current micro-batch forward calculation is completed, the sampling node status information

[0108] All nodes maintain local model layer information. After the model training information is initialized, the model layer information is static and will not change during the training process. is also fixed and is updated only in the next time slot. Therefore, when a node completes any micro-batch After the forward calculation, only the node status information of the current micro-batch forward calculation process needs to be sampled. , and ,in It can be obtained through hardware query. By recording the computation time of each round of forward calculation of micro batches and communication time , can be updated by calculating and ;

[0109] , and Only reflect nodes The node status during the k-th micro-batch forward training at time slot t cannot reflect the node During the entire training time slot t, the node The node status.

[0110] d. Integrate the model training information of this node and all upstream nodes, and send it to the next node together with the forward calculated intermediate activation value.

[0111] If it is the head node, step b in step ad will not be executed; if it is the end node, step d in step ad will not be executed.

[0112] When the forward calculation phase of a whole batch is completed, the end node of the pipeline further transforms the obtained node state information sampling value to obtain the node Node status information during the entire training time slot t , and , the transformation formula is:

[0113] ;

[0114] ;

[0115] ;

[0116] In the formula, and The values ​​reflect the entire time slot Midpoint Computing power and communication capabilities The average value of Reflects the entire time slot The middle node Memory capacity Lower limit;

[0117] Next, the terminal node further integrates the global model training information to obtain Figure 2 The global model training information shown includes the model partitioning of time slot t , model layer information of all layers, and node status information of all nodes in time slot t.

[0118] Step S2: In the backward calculation stage, the end node of the pipeline predicts and updates the global node state information based on the dynamic prediction algorithm. The specific steps include:

[0119] In a specific dynamic environment, task preemption and network traffic changes have trends, seasonality and periodicity, and the computing power and communication capabilities of each node change dynamically according to certain rules. Therefore, it is necessary to analyze the change pattern and trend from the historical node state information sequence to predict the possible value of the node state information in the next time slot. This embodiment is based on the setting of the dynamic prediction algorithm to better track the dynamics of the training environment. The dynamic prediction algorithm is specifically:

[0120] Step S21: The terminal node always maintains a historical sequence and predicted value for the computing power and communication power of each node. Define the terminal node as the node state before the end of the forward phase of time slot t The historical sequence maintained is

[0121] , ;

[0122] in, yes or Any one of It is a historical sequence Length, Reserved forward In addition, the end node also Maintain a forecast value . Calculated at time slot t-1, used to predict The value of .

[0123] If time slot t=1, the pipeline system is training the first input batch. At this time, there is no historical sequence of node status information, and no maintenance is performed on The predicted value of .

[0124] Step S22: After the forward phase of time slot t ends, the state of any node , the terminal node updates its history sequence length based on the prediction loss , and update the historical sequence synchronously .

[0125] The predicted value of node status information is based on historical sequences, but historical sequences are outdated. Recent sequence values ​​are often more valuable for prediction; however, when historical sequence values ​​continue to accumulate, outdated sequence values ​​cannot reflect the dynamic changes of node status information, but will interfere with the prediction model and reduce prediction accuracy.

[0126] To solve the obsolescence problem, a penalty term is added to the historical sequence length update equation, and the sequence is screened based on the prediction loss, so that obsolete values ​​are removed in time to ensure the quality of the prediction model. The update equation is as follows:

[0127] ,

[0128]

[0129] In the formula is the rounding operation. In addition, when t=1, . is the last time slot t-1 pair The predicted value at the current time slot t, It is the true value calculated in the current time slot. is the relative prediction error, which is used to measure the size of the prediction loss. When t=1, .

[0130] Furthermore, the updated historical sequence can be obtained as follows:

[0131] ;

[0132] Step S23: Based on the addition mode factor decomposition method, Decomposition into trend series , Seasonality , and the residual sum;

[0133]

[0134] Step S24: End node trend sequence and seasonal series The autoregressive moving average model is applied respectively, the order of the ARIMA model is determined according to the minimum Bayesian information criterion, and the trend forecast value is estimated using the optimal forecast model and seasonal forecasts ; Further, the node status The predicted value of is obtained from the following equation:

[0135]

[0136] in, and Node status The upper and lower bounds of It can be the bandwidth capacity of the communication link or the nominal computing power of the GPU. In actual applications, it can be set according to different training scenarios. , , to limit the node status prediction value to a reasonable range.

[0137] Step S3: The pipeline end node takes minimizing the training time of the synchronous pipeline parallel system as the goal, establishes an optimization problem model, calculates the optimal partition and passes the result backward to all upstream nodes. The specific steps include:

[0138] Step S31, establishing an optimization problem model with the goal of minimizing the completion time of the synchronous pipeline parallel system;

[0139] In this embodiment, taking the most classic synchronous pipeline model parallel architecture Gpipe as an example, after obtaining the model training information after dynamic prediction in step S2, the node The forward computation time for any micro-batch is:

[0140]

[0141] node The backward computation time for any micro-batch is:

[0142]

[0143] When executing any micro-batch forward / backward calculation, the node With Node The communication time is:

[0144]

[0145] Therefore, the time when node 0 completes micro-batch 1 in the forward computation phase is , node 1 receives the intermediate result of node 0 immediately after node 0 completes micro-batch 1, and starts the forward calculation of micro-batch 1 after receiving it. The time when node 1 completes micro-batch 1 is .

[0146] However, the time when node 1 starts to receive micro-batch 2 is uncertain, which depends on the order of the following two events: G1: node 0 completes the forward calculation of micro-batch 2. G2: node 1 has received the intermediate results of micro-batch 1.

[0147] So the time when node 1 finishes receiving micro-batch 2 is . Further, the time when node 1 starts the forward calculation of micro-batch 2 depends on the order of the completion of the following two events: G3: Node 1 completes receiving the intermediate results of micro-batch 2. G4: Node 1 completes the forward calculation of micro-batch 1. It can be obtained that the time when node 1 completes the forward calculation of micro-batch 2 is:

[0148]

[0149] in, represents the maximum load item (including computational load and communication load) on the forward path from node 0 to node 1. In fact, the completion time of a micro-batch in the forward computation phase of each node is always determined by the maximum load item on the path. From the derivation process, we can get the time consumption of a complete batch forward calculation of the synchronous pipeline parallel system as:

[0150]

[0151] After the terminal node completes the forward calculation of the last micro-batch, the backward phase will begin. The calculation and communication process of the backward phase is similar to that of the forward phase, except that the amount of calculation is different. Similarly, it can be deduced that the execution time of the entire backward phase of the synchronous pipeline parallel system is:

[0152]

[0153] After obtaining the model training information in step S1 and dynamically predicting the node status information in step S2, the total time for the synchronous pipeline model parallel architecture to complete a complete batch training is expressed as:

[0154] ;

[0155] in,

[0156]

[0157]

[0158]

[0159] In the formula, , and Respectively represent nodes The forward computation time, backward computation time, and node +1 single communication time. Node status information in time slot t 、 The completion time of a synchronous pipeline system depends only on the model partitioning. ;

[0160] The optimization problem of minimizing completion time is defined as follows:

[0161] ;

[0162]

[0163]

[0164] The first constraint is the partitioning position constraint, which requires that L model layers be allocated to N nodes and each node be allocated at least one layer. The second constraint is the memory constraint, which ensures that each node does not have a memory overflow error during training.

[0165] Step S32: Solve the original optimization problem and use the improved KL (Kernighan-Lin) community discovery algorithm to solve the optimal model partitioning strategy and obtain the optimal partitioning strategy. ;

[0166] The first step is to temporarily divide As the starting point, the computational load difference between all adjacent nodes is calculated. The temporary partition is the intermediate partition result of the optimal partition algorithm execution process. The optimal partition algorithm updates the temporary partition through multiple loop searches and iterations. To get the optimal score. At the beginning of the optimal partitioning algorithm, the temporary partition Initialized to the partition of the current time slot t The calculated load difference is given by the following equation:

[0167] , ;

[0168] Step 2: Press The size of all Arrange in descending order and get sequence S:

[0169]

[0170] in, Is a collection The more forward the element in the sequence S is, the greater the difference in computing load between the node and its downstream node is. In this case, migrating the model layer between the two nodes is more conducive to balancing the computing load.

[0171] The third step is to take the first element from S, denoted as If satisfied or and If they are equal, execute step 4, otherwise execute step 5. Removed from S.

[0172] Step 4: From step 3, we know that the communication load of a node in node j or node j+1 is the bottleneck of the pipeline. Let’s assume that the bottleneck node is node j, and assume that the model is temporarily divided into , in the temporary division Based on , node j tries to increase or decrease the local model layer to reduce the completion time of the synchronization pipeline. The new partition obtained after increasing or decreasing the model layer is:

[0173]

[0174] Division Indicates that in the temporary division Based on the above, node j migrates with node j-1 and node j+1 respectively. and model layers. , if it is non-negative, the migration represents sending; otherwise, the migration represents receiving. In addition, The following constraints need to be met:

[0175]

[0176]

[0177]

[0178]

[0179]

[0180] The first constraint ensures that The second constraint requires that the number of adjacent node migration layers between two adjacent temporary partitions does not exceed k layers. This constraint will find the partition The search space is limited to a reasonable range, which reduces the time consumption of the optimal partitioning algorithm; the third to fifth constraints require that the partitioning of the current time slot t , all attempts In the , the number of layers migrated between any adjacent nodes cannot exceed Z layers. This constraint can limit the overhead of layer scheduling in each time slot to a reasonable range. In addition, The memory constraints given in the optimization problem should also be met.

[0181] Among all possible attempts, select the one that makes the pipeline system complete in time The smallest division As a new temporary division, At this point, if there are no elements left in the sequence S, jump to step 6; otherwise, jump to step 3.

[0182] Step 5: From step 3, we know that the computational load between node j and node j+1 is unbalanced. In the temporary division Based on this, the model layer is migrated between node j and node j+1, and the new partition after migration is recorded as

[0183] Division Indicates that in the temporary division Based on, node j migrates to node j+1 Model layer. , at this time there is , migration represents a sending operation; if , at this time there is , migration represents a receive operation. In addition, The following constraints need to be met:

[0184]

[0185]

[0186]

[0187]

[0188] The first constraint ensures that The correctness of the division position; the second constraint requires that the number of adjacent node migration layers between two adjacent temporary divisions does not exceed k layers; the third and fourth constraints require that the division relative to the current time slot t , all attempts There cannot be any migration between adjacent nodes that exceeds Z layers. In addition, The memory constraints given in the optimization problem should also be met.

[0189] Among all possible attempts, select the one that makes the pipeline system complete in time The smallest division As a new temporary division, At this point, if there are no elements left in the sequence S, jump to step 6; otherwise, jump to step 3.

[0190] Step 6: If the temporary partition q has not been updated since the first step of this cycle, the current temporary partition q is the optimal partition, and the optimal partition result is output. ; Otherwise, jump to the first step to start a new loop, and continue to solve the optimal partition with the current temporary partition q as the starting point.

[0191] Step S33: Starting from the end node of the pipeline, the optimal partitioning result obtained in step S32 is It is passed backward to each upstream node together with the gradient result of the backward calculation.

[0192] like Figure 2 and Figure 3 As shown, at the beginning of the backward phase, the terminal node will pass the global model training information and the current partition after dynamic prediction. Input into the improved KL algorithm to generate the optimal division of the current time slot Afterwards, the optimal partition is passed back to the upstream node together with the intermediate gradient value of the backward calculation. The overhead of the optimal partitioning algorithm is very small and hardly affects the backward calculation process of the terminal node.

[0193] Step S4: The nodes participating in the layer migration call the layer manager to complete the migration preparation work, and migrate the model layer after the parameter update is completed. When the layer scheduling is completed, the forward calculation phase of the next batch begins. The specific steps include:

[0194] like Figure 3 and Figure 4 The figure shows the process and results of layer scheduling. After the optimal model partitioning strategy is solved in step S32 and the optimal partitioning is transferred backward in step S33, the final layer migration phase will be entered. The specific layer migration is divided into the following 7 steps (1)--(7), corresponding to Figure 4 1-7 of ;

[0195] Step S41, calculate the optimal division of time slot t The current division with time slot t The partition difference array. , , whose partition difference array is given by the following equation:

[0196]

[0197] in, , ;

[0198] Step S42, (1) Each node generates a layer scheduling strategy.

[0199] like , then node i+1 sends to node i Layers;

[0200] like , then node i sends to node i+1 Layers;

[0201] like , then there is no need to migrate the model layer between node i and node i+1.

[0202] All nodes are divided into different arrays according to Generate the layer scheduling strategy of this node;

[0203] If the node wants to receive the model layer, execute step S43; if the node wants to send the model layer, execute step S44; a node can execute step S43 and step S44 at the same time; if no operation is required, jump to step S45;

[0204] Step S43, (2) Allocate memory for the new layer. The node that needs to receive the layer calls the layer manager to initialize the model layer, allocate memory space for the new layer, and wait for the migration request from the source node;

[0205] (4) Request confirmation and establish connection. After receiving the request, return the request confirmation and establish a connection with the source node, and store the received model layer parameters in the specified memory; (6) After the layer reception is completed, return the layer migration completion signal, disconnect the connection, and jump to step S45;

[0206] Step S44, (3) Layer migration request. The node that needs to send the layer sends a layer migration request after the parameters are updated and waits for confirmation; (5) Send model layer parameters. After receiving the confirmation, establish a connection with the layer receiving node and call the layer manager to pass the model layer parameters; (7) Release memory. After the model layer parameters are sent and the layer migration completion signal is received, disconnect the connection, release the memory of the invalid layer, and jump to step S45;

[0207] Step S45, after the layer scheduling of all nodes is completed, the model division at this time is the optimal division of time slot t, and the synchronous pipeline system starts the forward calculation of time slot t+1.

[0208] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included in the scope of protection of the present invention.

Claims

1. A dynamic model layer scheduling method for synchronous pipeline model parallelism, characterized in that: The scheduling method includes a forward calculation phase and a backward calculation phase, and includes the following steps: Step S1, in the forward calculation stage, the model training information is transmitted forward to the end node of the pipeline; the model training information includes model layer information, current time slot model division, and node status information; Step S2: In the backward calculation stage, the end node of the pipeline predicts and updates the global node state information based on the dynamic prediction algorithm; Step S3: The pipeline end node takes minimizing the training time of the synchronous pipeline parallel system as the goal, establishes an optimization problem model, calculates the optimal partition and passes the result backward to all upstream nodes. The specific steps include: Step S31, establishing an optimization problem model with the goal of minimizing the completion time of the synchronous pipeline parallel system; After obtaining the model training information in step S1 and dynamically predicting the node status information in step S2, the total time for the synchronous pipeline model parallel architecture to complete a complete batch training is expressed as: ； in, In the formula, , and Respectively represent nodes The forward computation time, backward computation time, and node +1 single communication time; node status information in time slot t 、 The completion time of a synchronous pipeline system depends only on the model partitioning. ; The optimization problem of minimizing completion time is defined as follows: ； The first constraint is the partitioning position constraint, which requires that L model layers be allocated to N nodes and each node be allocated at least one layer. The second constraint is the memory constraint, which ensures that each node does not have a memory overflow error during training. Step S32: Solve the original optimization problem and use the improved KL community discovery algorithm to solve the optimal model partitioning strategy to obtain the optimal partitioning strategy. ; Step S33: Starting from the end node of the pipeline, the optimal partitioning result obtained in step S32 is transmitted backward to each upstream node together with the gradient result of the backward calculation; Step S4: The nodes participating in the layer migration call the layer manager to complete the migration preparation work, and migrate the model layer after the parameter update is completed. When the layer scheduling is completed, the forward calculation phase of the next batch begins.

2. According to claim 1, a dynamic model layer scheduling method for synchronous pipeline model parallelism is characterized in that: The specific steps in step S1 include: Step S11, define model layer information: Set the total number of layers of the training model to The total number of nodes is , the number of samples in a batch is , the number of micro-batch samples injected into the pipeline is The model layer information consists of three parts, including layer calculation load, adjacent layer communication load and layer parameter size, respectively. , express Time The forward and backward computation loads of the layer are calculated using Indicates The model parameter size of the layer, where ; and Expressed as the number of floating point operations; express Time Layer and Communication load between layers, where ; when At that time, The total forward computation load, the total backward computation load, and the Layer and The total communication load between layers is is proportional to; Step S12, defining the model division of time slot t; The training process of a synchronous pipeline for an input batch is one time slot, including forward calculation, backward calculation, and parameter update; According to the model partitioning scheme of different time slots, use the partition point array represents the model partitioning of time slot t, where Representation Node The largest layer number among the model layers stored locally at time slot t; the model partitioning at time slot t is obtained by the best partitioning result at time slot t-1; Step S13, each node samples node status information in micro batches in the forward phase of each time slot; Node status information includes the node's memory capacity, computing power, and communication capabilities; , Respectively represent nodes memory capacity and computing power, including ;use Representation Node With Node The communication capability between ; Each node will sample node status information in micro batches in the forward phase of each time slot; Train the micro-batch at time slot t ,in is the sequence number of the micro-batch in all micro-batches, ; After completing the forward training of micro-batch k, node Estimation by sampling , and The sampling values ​​are , and ; in, Obtained through hardware query, and The formula is obtained by using the following equation: ； ； in, ; In the formula, Defined as a pair of nodes The sum of some model layer information in the model layer stored in time slot t, Calculates forward load for the layer or backward calculation load , Adjacent layer communication load and layer parameter size Any one of; , They respectively represent the time it takes for a node to complete the forward computation and communication of the current micro-batch; Step S14: except for the node at the end of the pipeline In addition, after each micro-batch forward calculation is completed, the remaining nodes integrate the model training information of this node and all upstream nodes, and send it to the next node together with the intermediate activation value of the forward calculation; get the node Model training information, which includes time slot t nodes Model division position , local model layer information, and the node status information sampling value obtained after completing the current micro-batch; When the forward calculation phase of a whole batch is completed, the end node of the pipeline further transforms the obtained node state information sampling value to obtain the node Node status information during the entire training time slot t , and , the transformation formula is: ； ； ； In the formula, and The values ​​reflect the entire time slot Midpoint Computing power and communication capabilities The average value of Reflects the entire time slot The middle node Memory capacity Lower limit; The end nodes are integrated to obtain the global model training information, which includes the model partitioning of time slot t , model layer information of all layers, and node status information of all nodes in time slot t.

3. According to claim 1, a dynamic model layer scheduling method for synchronous pipeline model parallelism is characterized in that: The specific steps in step S2 include: Step S21: The end node always maintains a historical sequence and predicted value for the computing power and communication power of each node; the end node is defined as the node state before the end of the forward phase of time slot t The maintained history sequence is: ； in, yes or Any one of It is a historical sequence Length, Reserved forward The historical value of the time slot; the end node is also Maintain a forecast value ; Calculated at time slot t-1, used to predict The value of When time slot t=1, the pipeline system is training the first input batch. At this time, there is no historical sequence of node status information, and no maintenance is performed on The predicted value of Step S22: After the forward phase of time slot t ends, the state of any node , the terminal node updates its history sequence length based on the prediction loss , and update the historical sequence synchronously ; The value of is obtained according to the following equation: ， ； ； In the formula, is the rounding up operation. In addition, when t=1, ; The updated historical sequence is: ； Step S23: factorize the historical sequence based on the additive mode factorization method Decomposition into trend series , seasonal series , and the residual sequence sum; ； Step S24: End node trend sequence and seasonal series The autoregressive moving average model is applied respectively, the order of the ARIMA model is determined according to the minimum Bayesian information criterion, and the trend forecast value is estimated using the optimal forecast model and seasonal forecasts ; For node status The predicted value of is obtained from the following equation: ； in, and Node status The upper and lower bounds of .

4. According to claim 1, a dynamic model layer scheduling method for synchronous pipeline model parallelism is characterized in that: The specific steps in step S4 include: Step S41, calculate the optimal division of time slot t The current division with time slot t The partition difference array of , , whose partition difference array is given by the following equation: ； in, , ; Step S42, each node generates a layer scheduling strategy; like , then node i+1 sends to node i Layers; like , then node i sends to node i+1 Layers; like , then there is no need to migrate the model layer between node i and node i+1; All nodes are divided into different arrays according to Generate the layer scheduling strategy of this node; if this node wants to receive the model layer, execute step S43; if it wants to send the model layer, execute step S44; a node can execute step S43 and step S44 at the same time; if no operation is required, jump to step S45; Step S43: The node that needs to receive the layer calls the layer manager to initialize the model layer, allocates memory space for the new layer, and waits for the migration request of the source node; after receiving the request, it returns the request confirmation and establishes a connection with the source node, and stores the received model layer parameters in the specified memory; after the layer reception is completed, it returns the layer migration completion signal, disconnects the connection, and jumps to step S45; Step S44: The node that needs to send the layer sends a layer migration request after the parameters are updated and waits for confirmation; after receiving the confirmation, it establishes a connection with the layer receiving node and calls the layer manager to transfer the model layer parameters; after the model layer parameters are sent and the layer migration completion signal is received, it disconnects, releases the memory of the invalid layer, and jumps to step S45; Step S45, after the layer scheduling of all nodes is completed, the model division at this time is the optimal division of time slot t, and the synchronous pipeline system starts the forward calculation of time slot t+1.

Images