Method for parallel training of neural network model by multiple computing clusters, computing system and device, and medium

Abstract

The invention relates to a method, a computing system, a device and a medium for parallel training of a neural network model by a plurality of computing clusters, and the method comprises the following steps: enabling first to Nth computing nodes to execute forward computation of each micro-batch in a plurality of micro-batches by using deployed first to Nth model slices in sequence, enabling the Nth to the first computing nodes to sequentially execute reverse computing of each micro-batch, and enabling the Nth computing node to sequentially execute forward computing and reverse computing of each micro-batch; when it is set that the (i + 1) th computing node continuously executes forward computation of x micro-batches and then enters a stable stage of alternately executing reverse computation and forward computation, the ith computing node continuously executes forward computation of x + k micro-batches and then enters the stable stage, and when the ith computing node and the (i + 1) th computing node are located in the same computing cluster, k is equal to 1, 2,..., k, and k is equal to 1; and when the ith computing node and the (i + 1) th computing node are located in two different computing clusters, k = 2 or k = 3.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a method, a computing system, a device, and a medium for parallel training of neural network models using multiple computing clusters. Background Art

[0002] The training of large-scale neural network models (sometimes referred to as "models" in this article) is usually carried out in a computing cluster (sometimes referred to as "clusters" in this article) including multiple computing nodes. During the training process, the computing nodes need to communicate with each other to transfer data. In the same cluster including multiple computing nodes of the same type (such as the same model) or multiple computing nodes located in the same location (such as the same computer room), the communication between the computing nodes can use, for example, an IB (InfiniBand) network. Communication between computing nodes located in different clusters must rely on a dedicated line (such as a dedicated optical fiber) between the two clusters. The communication performance of the dedicated line is usually lower than that of the IB network, thus becoming a bottleneck in the training process.

[0003] When assigning different layers of a neural network model to multiple compute nodes for pipelined parallel training, a compute node must wait for data communication from the compute node assigned to the previous layer to complete before it can perform calculations. When compute nodes are located in different clusters, communication efficiency between compute nodes in different clusters is low, as the performance of dedicated lines between clusters is typically lower than that of the IB network within a cluster, affecting the training efficiency of the entire model. Summary of the Invention

[0004] In view of this, the present disclosure proposes a method, computing system, device, and medium for parallel training of neural network models in multiple computing clusters, which can mask the less efficient communication process between computing nodes located in different clusters as much as possible by the computing process, that is, when the computing nodes located in different clusters are communicating, they are also performing calculations, thereby achieving the purpose of improving the training efficiency of the entire model and accelerating the inference process.

[0005] According to one aspect of the present disclosure, a method for parallel training of neural network models in multiple computing clusters is provided, wherein each computing cluster in the multiple computing clusters includes multiple computing nodes, the multiple computing clusters include N computing nodes in total, and the communication time between the computing nodes of each computing cluster is shorter than the communication time between the computing nodes of the computing cluster and the computing nodes of other computing clusters. The method includes: splitting the neural network model to be trained into N model slices, so that each model slice includes one or more continuous layers of the neural network model; deploying the N model slices to the N computing nodes respectively, so that each computing node is deployed with a model slice, and the multiple computing nodes included in each computing cluster are deployed with multiple continuous model slices; obtaining training data including multiple micro-batches, so that the 1st to Nth computing nodes sequentially use the deployed 1st to Nth model slices to train the neural network model; N model slices execute forward computation for each of the multiple micro-batches, causing the Nth to 1st computing nodes to sequentially execute reverse computation for each micro-batch using the deployed Nth to 1st model slices, and causing the Nth computing node deployed with the Nth model slice to sequentially execute forward computation and reverse computation for each micro-batch, wherein, for the i-th computing node when 1≤i≤N-1, when it is set that the i+1th computing node enters a stable phase of alternating reverse computation and forward computation after continuously executing forward computation for x micro-batches, the method further includes: causing the i-th computing node to enter the stable phase after continuously executing forward computation for x+k micro-batches, wherein when the i-th computing node and the i+1th computing node are located in the same computing cluster, k=1, and when the i-th computing node and the i+1th computing node are located in two different computing clusters, k=2 or k=3.

[0006] In one possible implementation, the method further includes: when the i-th computing node and the i+1-th computing node are located in two different computing clusters, determining the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform forward calculations of a micro-batch.

[0007] In one possible implementation, determining the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform the forward computation of one micro-batch includes: determining k=2 in response to the communication time between the i-th computing node and the i+1-th computing node being no longer than the time required for either the i-th computing node or the i+1-th computing node to perform the forward computation of one micro-batch.

[0008] In one possible implementation, determining the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform the forward computation of one micro-batch includes: determining k=3 in response to the communication time between the i-th computing node and the i+1-th computing node being longer than the time required for either the i-th computing node or the i+1-th computing node to perform the forward computation of one micro-batch.

[0009] In one possible implementation, when the communication time between the i-th computing node and the i+1-th computing node is no longer than the total time required for either the i-th computing node and the i+1-th computing node to perform forward calculation and reverse calculation of one micro-batch, after the i-th computing node and the i+1-th computing node both enter the stable phase, when the result data of the forward calculation or reverse calculation is transmitted between the i-th computing node and the i+1-th computing node, the i-th computing node and the i+1-th computing node both perform forward calculation or reverse calculation.

[0010] In one possible implementation, after the i+1th computing node enters the stable phase, when the i-th computing node transmits the result data of the forward calculation of a micro-batch to the i+1th computing node, the i+1th computing node is performing the reverse calculation, and after the i-th computing node enters the stable phase, when the i+1th computing node transmits the result data of the reverse calculation of a micro-batch to the i-th computing node, the i-th computing node is performing the forward calculation.

[0011] In one possible implementation, when the communication time between the i-th computing node and the i+1-th computing node is no longer than the total time required for either the i-th computing node and the i+1-th computing node to perform the forward calculation and the reverse calculation of one micro-batch, after the i+1-th computing node enters the stable phase, when the i-th computing node transmits the result data of the forward calculation of one micro-batch to the i+1-th computing node, the i+1-th computing node is performing or has completed the forward calculation of the previous micro-batch, and after the i-th computing node enters the stable phase, when the i+1-th computing node transmits the result data of the reverse calculation of one micro-batch to the i-th computing node, the i-th computing node is performing or has completed the reverse calculation of the previous micro-batch.

[0012] According to another aspect of the present disclosure, a computing system for parallel training of a neural network model is provided, the computing system comprising a plurality of computing clusters, each of the plurality of computing clusters comprising a plurality of computing nodes, the plurality of computing clusters comprising a total of N computing nodes, and the communication time between the computing nodes of each computing cluster being shorter than the communication time between the computing nodes of the computing cluster and the computing nodes of other computing clusters, the computing system being configured to: split the neural network model to be trained into N model slices, so that each model slice comprises one or more continuous layers of the neural network model; deploy the N model slices to the N computing nodes respectively, so that each computing node is deployed with a model slice, and the plurality of computing nodes included in each computing cluster are deployed with a plurality of continuous model slices; obtain training data comprising a plurality of micro-batches, and enable the 1st to Nth computing nodes to sequentially utilize the deployed The computing system further comprises: configuring the i-th computing node to execute forward calculations of each micro-batch in the plurality of micro-batches using the deployed 1st to N-th model slices, enabling the N-th to 1st computing nodes to sequentially use the deployed N-th to 1st model slices to execute reverse calculations of each micro-batch, and enabling the N-th computing node deployed with the N-th model slice to sequentially execute forward calculations and reverse calculations of each micro-batch, wherein, for the i-th computing node when 1≤i≤N-1, when it is set that the i+1-th computing node enters a stable phase of alternating reverse calculations and forward calculations after continuously executing forward calculations of x micro-batches, the computing system is further configured to: enable the i-th computing node to enter the stable phase after continuously executing forward calculations of x+k micro-batches, wherein when the i-th computing node and the i+1-th computing node are located in the same computing cluster, k=1, and when the i-th computing node and the i+1-th computing node are located in two different computing clusters, k=2 or k=3.

[0013] In one possible implementation, the computing system is further configured to: when the i-th computing node and the i+1-th computing node are located in two different computing clusters, determine the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform forward calculations of a micro-batch.

[0014] According to another aspect of the present disclosure, a computing device for parallel training of a neural network model is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0015] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0017] According to various aspects of the present disclosure, for two computing nodes deployed with adjacent model slices (one or more consecutive layers of a model) located in different clusters, by making the computing node deployed with the earlier model slice (the model slice that is calculated first during forward calculation) perform one or two more micro-batches of forward calculations on the basis of performing one more micro-batch of forward calculations in advance according to the traditional pipeline parallel method relative to the computing node deployed with the later model slice (the model slice that is calculated later during forward calculation), it is possible to make the computing nodes located in different clusters perform calculations while communicating during the stable stage of pipeline parallel training (each computing node alternately performs reverse calculations and forward calculations), and try to eliminate the idle time of each computing node not calculating but only waiting for data to arrive, so that the computing process and the communication process overlap in time, thereby improving the training efficiency of the neural network model trained in parallel using multiple computing clusters and accelerating the training process.

[0018] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0020] Figure 1 A schematic diagram showing the parallel training process when the existing 1F1B pipeline parallel strategy is applied to a cluster.

[0021] Figure 2a and Figure 2b A schematic diagram showing the parallel training process when the existing 1F1B pipeline parallel strategy is applied to two clusters.

[0022] Figure 3 A flowchart of a method for parallel training of neural network models using multiple computing clusters according to an embodiment of the present disclosure is shown.

[0023] Figure 4a A schematic diagram illustrating a parallel training process in which two clusters perform 8 mini-batch training according to an embodiment of the present disclosure is shown.

[0024] Figure 4b A schematic diagram illustrating another parallel training process in which two clusters perform 8 mini-batch training according to an embodiment of the present disclosure is shown.

[0025] Figure 5 A block diagram of a computing device 1900 for parallel training of a neural network model according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0026] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0027] As used herein, the terms "comprises," "comprising," "having," or variations thereof are open ended and include one or more stated features, integers, elements, steps, parts, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, parts, functions, or groups thereof.

[0028] When an element is referred to as being "connected," "coupled," "responsive" or variations thereof to another element, it can be directly connected, coupled or responsive to the other element or intervening elements may be present.

[0029] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Therefore, without departing from the teachings of the present invention, the first element / operation in some embodiments may be referred to as the second element / operation in other embodiments.

[0030] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0031] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0032] To facilitate understanding, we first introduce pipeline parallel training of large-scale neural network models (large models). Due to the large number of model layers and the large number of parameters, a single computing node is usually unable to complete model training alone. Therefore, consider splitting the model into multiple parts and deploying them on multiple computing nodes in a computing cluster, so that multiple computing nodes can train the model in parallel. Assume that during the training process, the entire model is split into N model slices (stages), and each model slice includes one or more consecutive layers of the model. For example, the 1st to 5th layers of the model from the input side are regarded as the first model slice, the 6th to 8th layers are regarded as the second model slice, the 9th to 13th layers are regarded as the third model slice, and so on. These N model slices can be deployed to N computing nodes in a computing cluster. In this disclosure, a computing node refers to one or more computing units (such as one or more GPUs, etc.) that can deploy a model slice of a neural network model. From another perspective, one or more computing units deployed with a model slice can actually be regarded as a computing node. In the present disclosure, the computing node on which the nth model slice is deployed is referred to as the nth computing node. Pipeline parallel training refers to the process of training a set of training data (such as a mini-batch / micro-batch) on the 1st to Nth model slices deployed on the 1st to Nth computing nodes in sequence. A typical pipeline parallel training strategy is the 1F1B pipeline parallel strategy, the main idea of which is to make each computing node enter a stable stage of alternating reverse calculation and forward calculation after executing the forward calculation of stage_size-pp_rank micro-batches of training data, where stage_size is the total number of model slices, and pp_rank is the serial number of the model slice assigned to each computing node (starting from 0). For example, for the example of deploying N model slices to N computing nodes, stage_size = N, and the pp_rank of the i-th computing node deployed with the i-th (1≤i≤N) model slice from the input side of the model is i-1.

[0033] Figure 1 Figure 2 shows the parallel training process when the existing 1F1B pipeline parallel strategy is applied to a cluster. Figure 1As shown, the cluster includes four computing nodes, node 1 to node 4. The horizontal axis indicates the time t for the model slice deployed on each computing node to perform forward and reverse calculations. The blue squares in each row represent the forward calculation "F", and the orange squares "B" represent the reverse calculation. F1 to F8, B1 to B8 represent a total of 8 micro-batches of forward and reverse calculation processes. Assume that nodes 1 to 4 are respectively deployed with the 1st to 4th model slices of the model. In this example, stage_size = 4, and for nodes 1 to 4, pp_rank is 0 to 3 respectively. According to the 1F1B pipeline parallel strategy, node 1 continuously executes forward calculations of 4 micro-batches (i.e. Figure 1 The four blue blocks F1 to F4 of the first row of node 1 in the figure) enter the stable stage of alternating reverse and forward calculations (i.e. Figure 1 Similarly, node 2 continuously executes the forward computation of 3 micro-batches (i.e. Figure 1 The three blue blocks F1 to F3 of the second row node 2) enter the stable stage (i.e. Figure 1 The orange and blue alternating blocks B1, F5, B2, F6, ... B6 of the second row of node 2), node 3 continuously performs the forward calculation of 2 micro-batches (i.e. Figure 1 The two blue blocks F1 to F2 of the third row node 3) enter the stable stage (i.e. Figure 1 The third row of nodes 3 (B1, F5, B2, F6, ... B7, which are orange and blue alternating blocks), node 4 continuously performs the forward calculation of one micro-batch (i.e. Figure 1 The blue square F1 of the fourth row node 4) enters the stable stage (i.e. Figure 1 The phase of continuously executing forward computations of stage_size-pp_rank micro-batches can be understood as a warm-up phase, which is equivalent to a preparation phase for entering the stable phase of alternating backward and forward computations. Figure 1 In the example, the consecutive blue blocks before the first orange block of each node are the warm-up phase.

[0034] for Figure 1In the example above, since nodes 1 to 4 are in the same cluster, the communication efficiency of the IB network used for communication between computing nodes within the cluster is relatively high, so the communication delay between nodes 1 to 4 can be ignored. However, when multiple slices of the model are deployed to multiple computing nodes in different clusters (for example, clusters in different computer rooms or clusters with computing nodes of different models), the communication efficiency between computing nodes across clusters may be significantly lower than the communication efficiency between computing nodes within the same cluster. As a result, during cross-cluster data transmission during training, the downstream computing node may have completed the calculation for the previous micro-batch of training data, but the data from the upstream computing node in a different cluster has not yet arrived. Therefore, for the entire training process, there may be many such "idle" times when only communication is carried out and no calculation is performed, which reduces training efficiency.

[0035] For example, Figure 2a and Figure 2b This figure illustrates the parallel training process using the 1F1B pipeline parallelization strategy when nodes 1 through 4 are located in Cluster A and Cluster B, respectively. Similar to the previous example, nodes 1 through 4 each deploy four slices of the model. However, unlike the previous example, nodes 1 and 2 are located in Cluster A, while nodes 3 and 4 are located in Cluster B. It is assumed that all compute nodes in both clusters require the same or similar time to perform a forward / backward computation, and that the communication performance of the dedicated line between Cluster A and Cluster B is lower than that of their respective internal IB networks, resulting in non-negligible communication latency between Cluster A and Cluster B. Figure 2a This is a scenario where the communication delay between cluster A and cluster B is less than or equal to the duration of one forward computation. Figure 2b This is a scenario where the communication delay between cluster A and cluster B is greater than the duration of a forward calculation and less than the total duration of a forward and reverse calculation. Figure 2a and Figure 2b In the figure, the green squares represent the communication delay of cross-cluster forward communication (communication during forward calculation of training data), the gray squares represent the communication delay of cross-cluster reverse communication (communication during reverse calculation of training data), and the black squares represent the communication delay of cross-cluster bidirectional communication (i.e., forward communication and reverse communication at the same time). Figure 2a and Figure 2bAs shown in the figure, when the four nodes in the two clusters have continuously executed the corresponding number of micro-batches of forward calculations and entered the stable stage, the communication delay caused by the cross-cluster data transmission between node 2 and node 3 is large, and is significantly longer than the communication time between node 1 and node 2 or the communication time between node 2 and node 3. As a result, after entering the stable stage, each node has multiple idle periods waiting for data and cannot perform calculations (i.e., the white squares in the figure). In addition, the node with greater cross-cluster communication delay Figure 2b The idle time of the computing nodes in Figure 2a Such idle time will significantly increase the training time of the entire model and reduce training efficiency.

[0036] In view of this, the present disclosure proposes a method for parallel training of neural network models in multiple computing clusters. In this method, for two computing nodes deployed with adjacent model slices in different clusters, by making the computing node deployed with the earlier model slice (the model slice that is calculated first during forward calculation) perform one or two more micro-batches of forward calculations on the basis of performing one more micro-batch of forward calculations in advance according to the traditional 1F1B pipeline parallel method relative to the computing node deployed with the later model slice (the model slice that is calculated later during forward calculation), it can make the computing nodes located in different clusters perform calculations while communicating in the stable stage of pipeline parallel training, and try to eliminate the idle time of each computing node not calculating but only waiting for data to arrive, so that the calculation process and the communication process overlap in time, thereby improving the training efficiency of the neural network model trained in parallel using multiple computing clusters and accelerating the training process.

[0037] It should be noted that the method of the embodiment of the present disclosure is applied to a scenario in which multiple computing clusters train a neural network model in parallel, wherein each of the multiple computing clusters includes multiple computing nodes, and a computing node refers to one or more computing units (such as GPUs, etc.) that can deploy a model slice of the neural network model, and a model slice is one or more consecutive layers of the neural network model.

[0038] Figure 3 A flowchart of a method for parallel training of neural network models in multiple computing clusters according to an embodiment of the present disclosure is shown. The multiple computing clusters include N computing nodes in total, and because the hardware architectures of the computing nodes of different computing clusters are different (for example, different models), or because the computing nodes of different computing clusters are located in different locations (for example, located in different computer rooms), the communication time between the computing nodes of each computing cluster is shorter than the communication time between the computing nodes of the computing cluster and the computing nodes of other computing clusters. Figure 3 As shown, the method includes:

[0039] Step S11, splitting the neural network model to be trained into N model slices, such that each model slice includes one or more consecutive layers of the neural network model;

[0040] Step S12: deploy the N model slices to N computing nodes respectively, so that each computing node is deployed with one model slice, and each computing cluster includes multiple computing nodes deployed with multiple consecutive model slices;

[0041] Step S13: obtaining training data including multiple micro-batches, causing the 1st to Nth computing nodes to sequentially use the deployed 1st to Nth model slices to perform forward computation of each of the multiple micro-batches, causing the Nth to 1st computing nodes to sequentially use the deployed Nth to 1st model slices to perform reverse computation of each micro-batch, and causing the Nth computing node deployed with the Nth model slice to sequentially perform forward computation and reverse computation of each micro-batch;

[0042] Wherein, for the i-th computing node when 1≤i≤N-1, when the i+1-th computing node continuously executes x micro-batches of forward computation and then enters a stable phase of alternating backward computation and forward computation, the method further includes:

[0043] Step S14: The i-th computing node continuously executes forward computations of x+k micro-batches and then enters a stable phase, wherein when the i-th computing node and the i+1-th computing node are located in the same computing cluster, k=1, and when the i-th computing node and the i+1-th computing node are located in two different computing clusters, k=2 or k=3.

[0044] In step S11, for example, the entire neural network model can be split into N model slices based on the model splitting principles known in the art and based on the number N of computing nodes included in multiple computing clusters, so that each model slice includes one or more continuous layers of the neural network model. Of course, other customized splitting principles can also be used, and the embodiments of the present disclosure are not limited to this. Furthermore, in step S12, the N split model slices can be deployed to N computing nodes in multiple computing clusters respectively, with each computing node deployed with one model slice and multiple computing nodes in each computing cluster deployed with multiple continuous model slices. For example, a computing nodes of cluster A are deployed with a continuous a model slice, and b computing nodes of cluster B are deployed with b continuous model slices, and these a model slices and b model slices can also be continuous. The embodiments of the present disclosure do not limit the model deployment method, as long as the model slices can be deployed to each computing node.

[0045] In step S13, it can be seen that in pipeline parallel training, a large set of training data is usually divided into multiple micro-batches. For each micro-batch in the multiple micro-batches, in the forward calculation pipeline of the training process, the first computing node first receives the micro-batch, uses the first model slice deployed by it to perform the forward calculation of the micro-batch, and the calculation result is forward propagated along the increasing direction of the model slice number, and reaches the 2nd to Nth computing nodes in sequence, and the 2nd to Nth model slices deployed by the 2nd to Nth computing nodes are sequentially made to perform the forward calculation of the micro-batch. For each computing node except the Nth computing node, after performing the forward calculation of a micro-batch, the forward calculation of the next micro-batch can be continued. At the Nth computing node, after completing the forward calculation of a micro-batch, the reverse calculation of the micro-batch (such as gradient calculation) is immediately performed, and the calculation result is passed to the N-1th computing node. By analogy, the N-1th to 1st computing nodes are sequentially made to perform the reverse calculation of each micro-batch using the deployed model slices.

[0046] In order to mask the cross-cluster communication process during pipeline parallel training using the computational process and reduce the idle time caused by cross-cluster communication, in step S14, for the i-th compute node when 1≤i≤N-1, when the i+1-th compute node is set to enter the stable phase of alternating backward and forward computations after executing x micro-batches of forward computations, the i-th compute node is set to enter the stable phase (i.e., alternating backward and forward computations) after executing x+k micro-batches of forward computations. The value of k depends on whether the i-th compute node and the i+1-th compute node are located in the same compute cluster. When the i-th compute node and the i+1-th compute node are located in the same compute cluster, k = 1, i.e., the i-th compute node enters the stable phase after executing x+1 micro-batches of forward computations. This is similar to the computational logic in the traditional 1F1B pipeline parallel training strategy, in which the preceding compute node enters the stable phase after performing one more forward computation than the succeeding compute node. When the i-th compute node and the i+1-th compute node are located in two different compute clusters, k = 2 or k = 3, meaning the i-th compute node enters the stable phase after continuously executing x+2 or x+3 micro-batches of forward computations. This is equivalent to performing one or two additional forward computations on top of the traditional 1F1B pipeline parallel training strategy. This design assumes that when i+1 = N, x = 1, meaning the last compute node still follows the computational logic of the traditional 1F1B pipeline parallel training strategy, alternating between forward and backward computations for each micro-batch. Starting from the last compute cluster, for each cluster upstream, for upstream and downstream compute nodes located in different clusters, the upstream compute node performs one or two additional forward computations (x+1+1 or x+1+2, i.e., x+2 or x+3) on top of the one additional forward computation (x+1) that the downstream compute node should have performed.

[0047] By changing the difference in the number of forward calculations between two computing nodes across clusters, the upstream computing node is made to perform 2 or 3 more micro-batches of forward calculations than the downstream computing node in the other cluster before entering the stable phase. This allows the computing process and the communication process to overlap in time after each computing node enters the stable phase, minimizing idle time when no computing is done but only waiting for data to arrive.

[0048] In practical applications, when the i-th computing node and the i+1-th computing node are located in two different computing clusters, the value of k can be determined based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform a micro-batch of forward computations. The communication time between the i-th computing node and the i+1-th computing node can be determined based on the communication bandwidth of the dedicated line between the two computing clusters and the amount of data transmitted across the clusters. For example, assuming the amount of data to be transmitted is 500MB = 4,194,304,000 bits, and the dedicated line communication bandwidth is 40Gbps = 40×10 9 bits / s, the theoretical communication time is 4,194,304,000 bits / 40×10 9 bits / s≈105ms. The time required for a single computing node to perform the forward computation of a micro-batch can be obtained through experimental measurement, and this embodiment of the present disclosure does not impose any limitation on this.

[0049] Among them, the above-mentioned determination of the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform the forward calculation of a micro-batch may include: in response to the communication time between the i-th computing node and the i+1-th computing node being no longer than the time required for either the i-th computing node or the i+1-th computing node to perform the forward calculation of a micro-batch, determining k=2; or, in response to the communication time between the i-th computing node and the i+1-th computing node being longer than the time required for either the i-th computing node or the i+1-th computing node to perform the forward calculation of a micro-batch, determining k=3.

[0050] For example, Figure 4a Figure 2 shows the parallel training process of two clusters using 8 mini-batches of training data. Figure 4a In the example, the communication time between Node 2 and Node 3 across the cluster (a green square or a gray square) is no longer than the time required for either Node 2 or Node 3 to perform the forward computation of a micro-batch (either blue square). Figure 4aAs shown, node 4 still adopts the 1F1B strategy, alternating forward and backward calculations for each micro-batch with x=1. For node 3, since node 3 and node 4 are in the same cluster, k=1, and since node 4's x=1, node 3 continuously executes forward calculations of x+k=1+1=2 micro-batches (i.e., F1 to F2) before entering the stable phase. For node 2, since node 2 and node 3 are in different clusters and the communication time between node 2 and node 3 across clusters is no longer than the time required for either node 2 or node 3 to execute a forward calculation of a micro-batch, k=2, and since node 3's x=2, node 2 continuously executes forward calculations of x+k=2+2=4 micro-batches (i.e., F1 to F4) before entering the stable phase. For node 1, since node 1 and node 2 are in the same cluster, k=1, and since node 2's x=4, node 1 continuously executes forward calculations of x+k=4+1=5 micro-batches (i.e., F1 to F5) before entering the stable phase.

[0051] Figure 4b Another parallel training process is shown with two clusters performing 8 mini-batches. Figure 4b In the example, the communication time between Node 2 and Node 3 across the cluster (three green squares or three gray squares) is longer than the time required for either Node 2 or Node 3 to perform the forward computation of one micro-batch (any blue square). Figure 4b As shown, nodes 4 and 3 are Figure 4a The situation of nodes 4 and 3 in the example is the same, that is, node 4 continuously executes forward computations for x=1 and node 3 continuously executes forward computations for x=2 micro-batches (i.e., F1 to F2) before entering the stable phase. However, for node 2, since nodes 2 and 3 are in different clusters and the cross-cluster communication time between nodes 2 and 3 is longer than the time required for either node 2 or node 3 to execute the forward computation of one micro-batch, k=3, and since x=2 for node 3, node 2 continuously executes forward computations for x+k=2+3=5 micro-batches (i.e., F1 to F5) before entering the stable phase. For node 1, since nodes 1 and 2 are in the same cluster, k=1, and since x=5 for node 2, node 1 continuously executes forward computations for x+k=5+1=6 micro-batches (i.e., F1 to F6) before entering the stable phase.

[0052] from Figure 4a and Figure 4b It can be seen that compared with Figure 2a and Figure 2b In the case where the existing 1F1B pipeline parallel strategy is directly applied to two clusters, the overall training time spent on pipeline parallel training using the aforementioned method for parallel training of neural network models on multiple computing clusters provided by the present disclosure is shorter, the idle time is less, and the training efficiency is higher.

[0053] According to the pipeline parallel training strategy in the method for parallel training of neural network models in multiple computing clusters provided by the present disclosure, when the i+1th computing node is set to enter a stable stage of alternating reverse and forward calculations after continuously executing x micro-batches of forward calculations, the i-th computing node is set to enter a stable stage after continuously executing x+k micro-batches of forward calculations. The value of k depends on whether the i-th computing node and the i+1th computing node are located in the same computing cluster. When the i-th computing node and the i+1th computing node are located in the same computing cluster, k=1 is determined, and when the i-th computing node and the i+1th computing node are located in two different computing clusters, k=2 or k=3 is determined. Therefore, by changing the difference in the number of forward calculations between the two computing nodes across the cluster, the upstream computing node is made to perform 2 or 3 more micro-batches of forward calculations relative to the downstream computing node located in the other cluster (equivalent to the situation where the upstream computing node performs 1 more micro-batch of forward calculations when the upstream computing node and the downstream computing node are located in the same computing cluster, and then the upstream computing node performs 1 or 2 more micro-batches of forward calculations) before entering the stable stage. This allows the computing process and the communication process to overlap in time after each computing node enters the stable stage, thereby minimizing the idle time when no computing is done but only waiting for data to arrive.

[0054] Furthermore, when the i-th computing node and the i+1-th computing node are located in two different computing clusters, whether k=2 or k=3 is determined based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform forward computation for one micro-batch. Determining the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform forward computation for one micro-batch includes: determining k=2 in response to the communication time between the i-th computing node and the i+1-th computing node being no longer than the time required for either the i-th computing node or the i+1-th computing node to perform forward computation for one micro-batch; or determining k=3 in response to the communication time between the i-th computing node and the i+1-th computing node being longer than the time required for either the i-th computing node or the i+1-th computing node to perform forward computation for one micro-batch. Therefore, when the cross-cluster communication efficiency is relatively not that low, the upstream computing node is made to perform one more micro-batch of forward calculations before entering the stable stage. And when the cross-cluster communication efficiency is relatively lower, the upstream computing node is made to perform two more micro-batches of forward calculations before entering the stable stage. This can ensure training efficiency while making only minor changes to the original computing logic according to actual needs (cross-cluster communication efficiency), thereby reducing the cost of changing the original computing logic.

[0055] Therefore, when the i-th computing node and the i+1-th computing node are located in different clusters and the communication time between the i-th computing node and the i+1-th computing node is no longer than the total time required for either the i-th computing node and the i+1-th computing node to perform the forward calculation and the reverse calculation of one micro-batch, after the i-th computing node and the i+1-th computing node both enter the stable phase, when the result data of the forward calculation or the reverse calculation is transmitted between the i-th computing node and the i+1-th computing node, the i-th computing node and the i+1-th computing node both perform the forward calculation or the reverse calculation.

[0056] For example, Figure 4a As shown, when node 2 transmits the result data of the forward calculation F5 of the 5th micro-batch to node 3 (the 5th green square from the left), node 2 is performing the reverse calculation B2 of the 2nd micro-batch and node 3 is performing the reverse calculation B3 of the 3rd micro-batch; when node 3 transmits the result data of the reverse calculation B3 of the 3rd micro-batch to node 2 (the 3rd gray square from the left), node 2 is performing the forward calculation F6 of the 6th micro-batch and node 3 is performing the forward calculation F5 of the 5th micro-batch. In addition, as Figure 4b As shown, when node 2 transmits the result data of the forward calculation F6 of the 6th micro-batch to node 3 and at the same time node 3 transmits the result data of the reverse calculation B3 of the 3rd micro-batch to node 2 (the 7th to 9th black squares from the left), node 2 is performing the reverse calculation B2 of the 2nd micro-batch and the forward calculation F7 of the 7th micro-batch, and node 3 is performing the forward calculation F5 of the 5th micro-batch and the reverse calculation B4 of the 4th micro-batch.

[0057] In addition, when the i-th computing node and the i+1-th computing node are located in different clusters, and the communication time between the i-th computing node and the i+1-th computing node is no longer than the time required for either the i-th computing node and the i+1-th computing node to perform forward calculation of one micro-batch, after the i+1-th computing node enters the stable phase, when the i-th computing node transmits result data of forward calculation of one micro-batch to the i+1-th computing node, the i+1-th computing node is performing reverse calculation, and after the i-th computing node enters the stable phase, when the i+1-th computing node transmits result data of reverse calculation of one micro-batch to the i-th computing node, the i-th computing node is performing forward calculation.

[0058] For example, in Figure 4aIn the example, after node 3 enters the stable phase, when node 2 transmits the result data of the forward computation F5 of the fifth micro-batch to node 3, node 3 is performing the backward computation B3 of the third micro-batch. Moreover, after node 2 enters the stable phase, when node 3 transmits the result data of the backward computation B3 of the third micro-batch to node 2, node 2 is performing the forward computation F6 of the sixth micro-batch.

[0059] In addition, when the i-th computing node and the i+1-th computing node are located in different clusters, if the communication time between the i-th computing node and the i+1-th computing node is longer than the time required for either the i-th computing node and the i+1-th computing node to perform the forward calculation of one micro-batch but is not longer than the total time required for either the i-th computing node and the i+1-th computing node to perform the forward calculation and reverse calculation of one micro-batch, after the i+1-th computing node enters the stable phase, when the i-th computing node transmits the result data of the forward calculation of one micro-batch to the i+1-th computing node, the i+1-th computing node is performing or has completed the forward calculation of the previous micro-batch, and after the i-th computing node enters the stable phase, when the i+1-th computing node transmits the result data of the reverse calculation of one micro-batch to the i-th computing node, the i-th computing node is performing or has completed the reverse calculation of the previous micro-batch.

[0060] For example, in Figure 4b In the example, after node 3 enters the stable phase, when node 2 transmits the result data of the forward computation F6 of the 6th micro-batch to node 3, node 3 is executing or has completed the forward computation F5 of the 5th micro-batch. Moreover, after node 2 enters the stable phase, when node 3 transmits the result data of the backward computation B3 of the 3rd micro-batch to node 2, node 2 is executing or has completed the backward computation B2 of the 2nd micro-batch.

[0061] In practical applications, when the i-th computing node and the i+1-th computing node are located in two different computing clusters, it is possible that the communication time between the i-th computing node and the i+1-th computing node is longer than the total time required for either the i-th computing node or the i+1-th computing node to perform the forward and reverse calculations of a micro-batch. In this case, although it is impossible to completely mask the cross-cluster communication delay by the calculation process (the computing node still needs to wait for data without calculating), it is still better than the original pipeline parallel training solution (such as Figure 2a or Figure 2b shown).

[0062] It should be understood that the above Figure 4a and Figure 4bThis is a simplified explanation for ease of understanding. In practice, the number of computing clusters can be greater, and the number of computing nodes in each computing cluster can also be greater. However, regardless of the number of computing clusters and computing nodes, the above-described method of the disclosed embodiment can be used to mask the cross-cluster communication process through the computing process, reducing the idle time caused by cross-cluster communication, thereby improving the training efficiency of neural network models trained in parallel using multiple computing clusters and accelerating the training process.

[0063] According to the method of the embodiment of the present disclosure, for two computing nodes deployed with adjacent model slices (one or more continuous layers of a model) located in different clusters, by making the computing node deployed with the earlier model slice (the model slice that is calculated first during forward calculation) perform one or two more micro-batches of forward calculation on the basis of performing one more micro-batch of forward calculation in advance according to the traditional pipeline parallel method relative to the computing node deployed with the later model slice (the model slice that is calculated later during forward calculation), it is possible to make the computing nodes located in different clusters perform calculations while communicating during the stable stage of pipeline parallel training (each computing node alternately performs reverse calculations and forward calculations), and try to eliminate the idle time of each computing node not calculating but only waiting for data to arrive, so that the computing process and the communication process overlap in time, thereby improving the training efficiency of the neural network model trained in parallel using multiple computing clusters and accelerating the training process.

[0064] Based on the above method, an embodiment of the present disclosure further provides a computing system for parallel training of a neural network model, the computing system comprising multiple computing clusters, each of the multiple computing clusters comprising multiple computing nodes, the multiple computing clusters comprising a total of N computing nodes, and the communication time between the computing nodes of each computing cluster is shorter than the communication time between the computing nodes of the computing cluster and the computing nodes of other computing clusters, the computing system being configured as follows:

[0065] Splitting the neural network model to be trained into N model slices, such that each model slice includes one or more consecutive layers of the neural network model;

[0066] Deploying the N model slices to the N computing nodes respectively, such that each computing node is deployed with one model slice, and each computing cluster includes the plurality of computing nodes deployed with a plurality of consecutive model slices;

[0067] Obtain training data comprising a plurality of micro-batches, cause the 1st to Nth compute nodes to sequentially perform forward computation of each of the plurality of micro-batches using the 1st to Nth model slices deployed, cause the Nth to 1st compute nodes to sequentially perform reverse computation of each micro-batch using the Nth to 1st model slices deployed, and cause the Nth compute node with the Nth model slice deployed to sequentially perform forward computation and reverse computation of each micro-batch;

[0068] Wherein, for the i-th computing node when 1≤i≤N-1, when the i+1-th computing node continuously executes x micro-batches of forward computation and then enters a stable phase of alternating backward computation and forward computation, the computing system is further configured as follows:

[0069] The ith computing node is caused to continuously execute forward computations of x+k micro-batches before entering the stable phase, wherein when the ith computing node and the (i+1)th computing node are located in the same computing cluster, k=1, and when the ith computing node and the (i+1)th computing node are located in two different computing clusters, k=2 or k=3.

[0070] In one possible implementation, the computing system is further configured to: when the i-th computing node and the i+1-th computing node are located in two different computing clusters, determine the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform forward calculations of a micro-batch.

[0071] In one possible implementation, determining the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform the forward computation of one micro-batch includes: determining k=2 in response to the communication time between the i-th computing node and the i+1-th computing node being no longer than the time required for either the i-th computing node or the i+1-th computing node to perform the forward computation of one micro-batch.

[0072] In one possible implementation, determining the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform the forward computation of one micro-batch includes: determining k=3 in response to the communication time between the i-th computing node and the i+1-th computing node being longer than the time required for either the i-th computing node or the i+1-th computing node to perform the forward computation of one micro-batch.

[0073] In one possible implementation, when the communication time between the i-th computing node and the i+1-th computing node is no longer than the total time required for either the i-th computing node and the i+1-th computing node to perform forward calculation and reverse calculation of one micro-batch, after the i-th computing node and the i+1-th computing node both enter the stable phase, when the result data of the forward calculation or reverse calculation is transmitted between the i-th computing node and the i+1-th computing node, the i-th computing node and the i+1-th computing node both perform forward calculation or reverse calculation.

[0074] In one possible implementation, after the i+1th computing node enters the stable phase, when the i-th computing node transmits the result data of the forward calculation of a micro-batch to the i+1th computing node, the i+1th computing node is performing the reverse calculation, and after the i-th computing node enters the stable phase, when the i+1th computing node transmits the result data of the reverse calculation of a micro-batch to the i-th computing node, the i-th computing node is performing the forward calculation.

[0075] In one possible implementation, when the communication time between the i-th computing node and the i+1-th computing node is no longer than the total time required for either the i-th computing node and the i+1-th computing node to perform the forward calculation and the reverse calculation of one micro-batch, after the i+1-th computing node enters the stable phase, when the i-th computing node transmits the result data of the forward calculation of one micro-batch to the i+1-th computing node, the i+1-th computing node is performing or has completed the forward calculation of the previous micro-batch, and after the i-th computing node enters the stable phase, when the i+1-th computing node transmits the result data of the reverse calculation of one micro-batch to the i-th computing node, the i-th computing node is performing or has completed the reverse calculation of the previous micro-batch.

[0076] In some embodiments, the functions or modules included in the system provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. Its specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0077] An embodiment of the present disclosure also provides a computing device for parallel training of a neural network model, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0078] An embodiment of the present disclosure further provides a non-volatile computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0079] An embodiment of the present disclosure further provides a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0080] Figure 5 FIG1 shows a block diagram of a computing device 1900 for parallel training of a neural network model according to an embodiment of the present disclosure. For example, the device 1900 can be provided as a server or a terminal device. Figure 5 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0081] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0082] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the apparatus 1900 to perform the above-described method.

[0083] A computer-readable storage medium can be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0084] The computer programs (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0085] The computer program (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by utilizing state information of computer-readable program instructions to personalize and customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0086] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0088] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0089] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0090] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for parallel training of neural network models on multiple computing clusters, characterized in that: Each computing cluster in the plurality of computing clusters includes a plurality of computing nodes, the plurality of computing clusters include a total of N computing nodes, and a communication time between computing nodes in each computing cluster is shorter than a communication time between computing nodes in the computing cluster and computing nodes in other computing clusters, the method comprising: Splitting the neural network model to be trained into N model slices, such that each model slice includes one or more consecutive layers of the neural network model; Deploying the N model slices to the N computing nodes respectively, such that each computing node is deployed with one model slice, and each computing cluster includes the plurality of computing nodes deployed with a plurality of consecutive model slices; Obtain training data comprising a plurality of micro-batches, cause the 1st to Nth compute nodes to sequentially perform forward computation of each of the plurality of micro-batches using the 1st to Nth model slices deployed, cause the Nth to 1st compute nodes to sequentially perform reverse computation of each micro-batch using the Nth to 1st model slices deployed, and cause the Nth compute node with the Nth model slice deployed to sequentially perform forward computation and reverse computation of each micro-batch, Wherein, for the i-th computing node when 1≤i≤N-1, when the i+1-th computing node continuously executes x micro-batches of forward computation and then enters a stable phase of alternating backward computation and forward computation, the method further includes: The i-th computing node continuously executes forward calculations of x+k micro-batches and then enters the stable phase. When the i-th computing node and the i+1-th computing node are located in the same computing cluster, k=1; and when the i-th computing node and the i+1-th computing node are located in two different computing clusters, k=2 or k=3.

2. The method according to claim 1, characterized in that The method further comprises: When the i-th computing node and the i+1-th computing node are located in two different computing clusters, the value of k is determined based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform the forward calculation of a micro-batch.

3. The method according to claim 2, characterized in that Determining the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform a forward calculation of a micro-batch includes: In response to a communication time between the i-th computing node and the (i+1)-th computing node being no longer than a time required for either the i-th computing node and the (i+1)-th computing node to perform forward computation of one micro-batch, k=2 is determined.

4. The method according to claim 2, characterized in that Determining the value of k based on the relationship between the communication time between the i-th computing node and the i+1-th computing node and the time required for the i-th computing node and the i+1-th computing node to perform a forward calculation of a micro-batch includes: In response to a communication time between the i-th computing node and the (i+1)-th computing node being longer than a time required for either the i-th computing node and the (i+1)-th computing node to perform forward computation of one micro-batch, k=3 is determined.

5. The method according to claim 3 or 4, characterized in that When the communication time between the i-th computing node and the i+1-th computing node is no longer than the total time required for either the i-th computing node and the i+1-th computing node to perform the forward calculation and the reverse calculation of one micro-batch, after the i-th computing node and the i+1-th computing node both enter the stable phase, when the result data of the forward calculation or the reverse calculation is transmitted between the i-th computing node and the i+1-th computing node, the i-th computing node and the i+1-th computing node both perform the forward calculation or the reverse calculation.

6. The method according to claim 3, characterized in that After the i+1th computing node enters the stable phase, when the i-th computing node transmits the result data of the forward calculation of a micro-batch to the i+1th computing node, the i+1th computing node is performing the reverse calculation, and After the i-th computing node enters the stable phase, when the i+1-th computing node transmits result data of a micro-batch of reverse calculation to the i-th computing node, the i-th computing node is performing forward calculation.

7. The method according to claim 4, characterized in that If the communication time between the i-th computing node and the i+1-th computing node is no longer than the total time required for either the i-th computing node or the i+1-th computing node to perform the forward and backward computations of one micro-batch, After the i+1th computing node enters the stable phase, when the i-th computing node transmits the result data of the forward calculation of a micro-batch to the i+1th computing node, the i+1th computing node is executing or has completed the forward calculation of the previous micro-batch, and After the i-th computing node enters the stable phase, when the i+1-th computing node transmits the result data of the reverse calculation of a micro-batch to the i-th computing node, the i-th computing node is executing or has completed the reverse calculation of the previous micro-batch.

8. A computing system for parallel training of neural network models, characterized in that: The computing system includes multiple computing clusters, each of the multiple computing clusters includes multiple computing nodes, the multiple computing clusters include a total of N computing nodes, and communication time between computing nodes in each computing cluster is shorter than communication time between computing nodes in the computing cluster and computing nodes in other computing clusters. The computing system is configured to: Splitting the neural network model to be trained into N model slices, such that each model slice includes one or more consecutive layers of the neural network model; Deploying the N model slices to the N computing nodes respectively, such that each computing node is deployed with one model slice, and each computing cluster includes the plurality of computing nodes deployed with a plurality of consecutive model slices; Obtain training data comprising a plurality of micro-batches, cause the 1st to Nth compute nodes to sequentially perform forward computation of each of the plurality of micro-batches using the 1st to Nth model slices deployed, cause the Nth to 1st compute nodes to sequentially perform reverse computation of each micro-batch using the Nth to 1st model slices deployed, and cause the Nth compute node with the Nth model slice deployed to sequentially perform forward computation and reverse computation of each micro-batch, Wherein, for the i-th computing node when 1≤i≤N-1, when the i+1-th computing node continuously executes x micro-batches of forward computation and then enters a stable phase of alternating backward computation and forward computation, the computing system is further configured as follows: The i-th computing node continuously executes forward calculations of x+k micro-batches and then enters the stable phase. When the i-th computing node and the i+1-th computing node are located in the same computing cluster, k=1; and when the i-th computing node and the i+1-th computing node are located in two different computing clusters, k=2 or k=3.

9. A computing device for parallel training of a neural network model, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images