Load balancing method and device, electronic equipment and storage medium

Abstract

The invention relates to a load balancing method and device, electronic equipment and a storage medium, and the method comprises the steps: sequentially inputting N pieces of micro-batch data into a first calculation node, and after the first calculation node completes the forward calculation of the N pieces of micro-batch data on an assembly line corresponding to the N pieces of micro-batch data, outputting the N pieces of micro-batch data; the first computing node re-computes an intermediate result which is released by the second computing node and is generated when forward computation on the flow line corresponding to the M1 pieces of micro-batch data is carried out; and after the plurality of computing nodes complete forward computing on the assembly line corresponding to the N pieces of micro-batch data and when the second computing node performs reverse computing on the assembly line corresponding to the M1 pieces of micro-batch data, the second computing node obtains an intermediate result obtained by re-computing the second computing node by the first computing node. According to the method and the device, the computing resources in the distributed computing cluster are fully utilized, so that the load of each computing node is more balanced.

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and in particular to a load balancing method, device, electronic device, and storage medium. Background Art

[0002] An important model parallelism method in distributed training is pipeline parallelism. In pipeline parallelism, there is data dependency between the forward and reverse calculations between devices. When there are many devices participating in pipeline parallelism, the device that first performs forward calculations will be idle for a long time after the end of the forward calculation and before the start of the reverse calculation, while the device that performs forward calculations later is still busy performing calculations when the first device is idle, resulting in an unbalanced computing load on each device. Summary of the invention

[0003] In view of this, the present disclosure proposes a load balancing method, device, electronic device, storage medium and computer program product.

[0004] According to one aspect of the present disclosure, a load balancing method is provided, which is applied to a distributed computing cluster, wherein a plurality of computing nodes in the distributed computing cluster train a preset model in a pipeline parallel manner; wherein each computing node is deployed with a sub-model of the preset model split according to the network layer, and different computing nodes deploy different sub-models;

[0005] The method comprises:

[0006] Inputting N micro-batch data into a first computing node in sequence, the multiple computing nodes sequentially performing forward calculations on pipelines corresponding to each micro-batch data in the N micro-batch data, wherein when performing forward calculations on pipelines corresponding to each micro-batch data, an output result of the forward calculation of a previous computing node is used as input data of the forward calculation of a next computing node, and each computing node releases an intermediate result generated by the forward calculation; N is a positive integer, and the first computing node is the computing node that performs forward calculations earliest among the multiple computing nodes;

[0007] After the first computing node completes the forward calculation on the pipeline corresponding to the N micro-batches, the first computing node recalculates the intermediate result released by the second computing node when performing the forward calculation on the pipeline corresponding to the M1 micro-batches; wherein the second computing node is a computing node in a preset queue, and the preset queue includes at least one computing node arranged in the order of performing reverse calculation; M1 is an integer not greater than N;

[0008] After the multiple computing nodes complete the forward calculation on the pipelines corresponding to the N micro-batches, the multiple computing nodes sequentially perform reverse calculation on the pipelines corresponding to each micro-batch of the N micro-batches, wherein when performing reverse calculation on the pipeline corresponding to each micro-batch, the output result of the reverse calculation of the previous computing node is used as input data for the reverse calculation of the next computing node; when the second computing node performs reverse calculation on the pipelines corresponding to the M1 micro-batches, the second computing node obtains the intermediate result obtained by the first computing node recalculating the second computing node.

[0009] In a possible implementation, the method further includes:

[0010] If the second computing node is the computing node that performs reverse computing earliest among the multiple computing nodes, then after the second computing node completes the reverse computing on the pipeline corresponding to the N micro-batches of data, the second computing node recalculates the intermediate results released by the first computing node when performing forward computing on the pipeline corresponding to M2 micro-batches of data; M2 is an integer not greater than N;

[0011] When the first computing node performs reverse calculation on the pipeline corresponding to the M2 micro-batch data, the first computing node obtains an intermediate result obtained by the recalculation performed by the second computing node.

[0012] In a possible implementation, the first computing node recalculates the intermediate result released by the second computing node when performing forward calculation on the pipeline corresponding to the M1 micro-batch data, including:

[0013] The first computing node obtains the sub-model deployed by the second computing node and the input data of the second computing node for performing forward calculation on the pipeline corresponding to the M1 micro-batch data;

[0014] The first computing node processes the input data of the forward calculation on the pipeline corresponding to the M1 micro-batch data of the second computing node based on the sub-model deployed by the second computing node, and obtains the intermediate result generated when the forward calculation on the pipeline corresponding to the M1 micro-batch data is performed.

[0015] In a possible implementation, before the first computing node obtains the sub-model deployed by the second computing node and the second computing node performs input data for forward calculation on the pipeline corresponding to the M1 micro-batch data, the method further includes:

[0016] The first computing node transfers the sub-model deployed by the first computing node and the output result of the forward calculation performed by the first computing node on the pipeline corresponding to the N micro-batch data to the first storage node.

[0017] In a possible implementation, the first computing node transfers the sub-model deployed by the first computing node and the output result of the forward calculation on the pipeline corresponding to the N micro-batch data performed by the first computing node to the first storage node, including:

[0018] After the first computing node completes the forward calculation on the pipeline corresponding to the i-th micro-batch data among the N micro-batch data, the first computing node transfers the output result of the forward calculation on the pipeline corresponding to the i-th micro-batch data to the first storage node; wherein i is an integer less than N;

[0019] After the first computing node completes forward calculations on pipelines corresponding to all micro-batches of the N micro-batches, the sub-model deployed by the first computing node is transferred to the first storage node.

[0020] In a possible implementation, the second computing node recalculates the intermediate result released by the first computing node when performing forward calculation on the pipeline corresponding to the M2 micro-batch data, including:

[0021] The second computing node obtains the sub-model deployed by the first computing node and the M2 micro-batch data;

[0022] The second computing node processes the M2 micro-batch data based on the sub-model deployed by the first computing node to obtain intermediate results generated when performing forward calculations on the pipeline corresponding to the M2 micro-batch data.

[0023] In a possible implementation manner, before the second computing node obtains the sub-model deployed by the first computing node and the M2 micro-batch data, the method further includes:

[0024] The second computing node transfers the sub-model deployed by the second computing node to the second storage node.

[0025] In a possible implementation, when there is more than one computing node in the preset queue, the first computing node recalculates the intermediate result released by the second computing node when performing the forward calculation on the pipeline corresponding to the M1 micro-batch data, including:

[0026] The first computing node recalculates the intermediate results released by each computing node in the preset queue and generated when performing forward calculations on one or more pipelines corresponding to micro-batch data according to the arrangement order of each computing node in the preset queue.

[0027] In a possible implementation manner, the number of the multiple computing nodes is N.

[0028] In a possible implementation manner, the number of computing nodes included in the preset queue does not exceed a specific threshold, and the specific threshold is positively correlated with the number of the plurality of computing nodes.

[0029] In a possible implementation, any computing node among the multiple computing nodes includes multiple sub-nodes, wherein the multiple sub-nodes run the sub-models deployed on the corresponding computing nodes in a tensor parallel manner.

[0030] According to another aspect of the present disclosure, a load balancing device is provided, which is applied to a distributed computing cluster, wherein a plurality of computing nodes in the distributed computing cluster train a preset model in a pipeline parallel manner; wherein each computing node is deployed with a sub-model of the preset model split according to the network layer, and different computing nodes deploy different sub-models;

[0031] The device comprises:

[0032] A forward computing module, used to sequentially input N micro-batch data into a first computing node, wherein the plurality of computing nodes sequentially perform forward computing on a pipeline corresponding to each micro-batch data in the N micro-batch data, wherein when performing forward computing on a pipeline corresponding to each micro-batch data, an output result of the forward computing of a previous computing node is used as input data of the forward computing of a subsequent computing node, and each computing node releases an intermediate result generated by the forward computing; N is a positive integer, and the first computing node is the computing node that performs forward computing earliest among the plurality of computing nodes;

[0033] a recalculation module, configured to, after the first computing node completes the forward calculation on the pipeline corresponding to the N micro-batches of data, cause the first computing node to recalculate the intermediate result released by the second computing node when performing the forward calculation on the pipeline corresponding to the M1 micro-batches of data; wherein the second computing node is a computing node in a preset queue, and the preset queue includes at least one computing node arranged in the order of performing reverse calculation; and M1 is an integer not greater than N;

[0034] A reverse calculation module is used for, after the multiple computing nodes have completed the forward calculation on the pipelines corresponding to the N micro-batches of data, the multiple computing nodes sequentially perform reverse calculation on the pipelines corresponding to each micro-batch of data in the N micro-batches of data, wherein, when performing reverse calculation on the pipeline corresponding to each micro-batch of data, the output result of the reverse calculation of the previous computing node is used as the input data of the reverse calculation of the next computing node; when the second computing node performs reverse calculation on the pipelines corresponding to the M1 micro-batches of data, the second computing node obtains the intermediate result obtained by the first computing node recalculating the second computing node.

[0035] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0036] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0037] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device performs the above method.

[0038] Through various aspects of the present disclosure, in the process of multiple computing nodes in a distributed computing cluster training a preset model in a pipeline parallel manner, N micro-batch data are sequentially input into a first computing node, and the multiple computing nodes sequentially perform forward calculations on the pipeline corresponding to each micro-batch data in the N micro-batch data, and each computing node releases the intermediate results generated by the forward calculation; wherein, the intermediate results generated by the forward calculation of each computing node do not have data dependence, that is, the intermediate results generated by the forward calculation of each computing node can be recalculated through the sub-model deployed by the computing node and the input data of the forward calculation, without the need for data from other computing nodes. When performing forward calculation on the pipeline corresponding to each micro-batch data, the output result of the forward calculation of the previous calculation node is used as the input data of the forward calculation of the next calculation node. In this way, for the calculation node that completes the forward calculation on the pipeline corresponding to N micro-batches of data first, it is necessary to wait for all calculation nodes to complete the forward calculation on the pipeline corresponding to N micro-batches of data before each calculation node can perform the reverse calculation on the pipeline corresponding to each micro-batch of data in the N micro-batches of data in turn. Since the order in which each calculation node performs the calculation tasks in the forward calculation and reverse calculation on the same pipeline is opposite, and when performing each micro-batch of data During the reverse calculation on the corresponding pipeline, the output result of the reverse calculation of the previous computing node is used as the input data of the reverse calculation of the next computing node. Therefore, the computing node that performs forward calculation earlier completes the forward calculation earlier and performs reverse calculation later. Accordingly, the computing node that performs forward calculation earlier has a longer idle time after completing the forward calculation on the corresponding pipeline of N micro-batches of data; conversely, the computing node that performs forward calculation later completes the forward calculation later and performs reverse calculation earlier. Accordingly, the computing node that performs forward calculation later has a shorter idle time after completing the forward calculation on the corresponding pipeline of N micro-batches of data. After the first computing node that performs forward calculation earliest completes the forward calculation on the pipeline corresponding to the N micro-batch data, it will enter an idle state. Using the longer idle time, the first computing node recalculates the intermediate results released by the second computing node in the preset queue when performing the forward calculation on the pipeline corresponding to M1 micro-batch data, wherein the preset queue includes at least one computing node arranged in the order of reverse calculation; thereby, the computing resources of the first computing node can be fully utilized to share the recalculation tasks of the second computing node that do not have data dependencies, and the time spent by the second computing node for recalculation is also saved.Furthermore, after the multiple computing nodes have completed the forward calculations on the pipelines corresponding to the N micro-batches, the multiple computing nodes sequentially perform reverse calculations on the pipelines corresponding to each micro-batch of the N micro-batches, and when the second computing node performs reverse calculations on the pipelines corresponding to the M1 micro-batches, the second computing node obtains the intermediate results obtained by the first computing node for recalculating the second computing node, and the second computing node can directly use these intermediate results without recalculation, thereby saving the time overhead of the reverse calculation of the second computing node. In this way, by taking advantage of the weak data dependency of recalculation, the first computing node with more idle time is used to help the second computing node at the back of the pipeline share part or all of the recalculation work, thereby making full use of the computing resources in the distributed computing cluster, making the load of each computing node more balanced, improving the computing efficiency of distributed training, shortening the time consumption of parallel end-to-end pipelines, and making the overall training more efficient and time-saving.

[0039] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0041] Figure 1 A schematic diagram showing the structure of a distributed computing cluster according to an embodiment of the present disclosure is shown;

[0042] Figure 2 A schematic diagram showing the partitioning of model parameters when applying pipeline parallelism according to an embodiment of the present disclosure is shown.

[0043] Figure 3 A schematic diagram showing model parameter partitioning when pipeline parallelism and tensor parallelism are simultaneously applied according to an embodiment of the present disclosure is shown.

[0044] Figure 4 A schematic diagram showing pipeline parallelism combined with recalculation according to an embodiment of the present disclosure is shown.

[0045] Figure 5 A flow chart of a load balancing method according to an embodiment of the present disclosure is shown.

[0046] Figure 6 A schematic diagram showing a load balancing method according to an embodiment of the present disclosure is shown.

[0047] Figure 7 A schematic diagram showing a load balancing method according to an embodiment of the present disclosure is shown.

[0048] Figure 8 A flow chart of a load balancing method according to an embodiment of the present disclosure is shown.

[0049] Fig. 9 A schematic diagram showing a load balancing method according to an embodiment of the present disclosure is shown.

[0050] Fig.10 A flow chart of another load balancing method according to an embodiment of the present disclosure is shown.

[0051] Fig.11 A structural diagram of a load balancing device according to an embodiment of the present disclosure is shown.

[0052] Fig.12 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0053] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0054] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "exemplary", "in one embodiment", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0055] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0056] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0057] Figure 1 A schematic diagram of the structure of a distributed computing cluster according to an embodiment of the present disclosure is shown. The distributed computing cluster can be used for distributed training of models in the field of machine learning, wherein the model can be a deep learning neural network model (Deep Neural Network, DNN), a convolutional neural network model (Convolutional Neural Network, CNN), a large language model or other neural network models, etc.

[0058] like Figure 1 As shown, the distributed computing cluster may include: a computing node set 101. The computing node set 101 may include multiple computing nodes (also referred to as computing devices), which are responsible for performing actual computing tasks in the distributed training process; illustratively, they may be responsible for computing operations such as forward computing and reverse computing in the model training process. Figure 1 The computing node set 101 shown in includes T computing nodes, namely computing node 1, computing node 2... computing node T, T is a positive integer; wherein each computing node can communicate with each other. Exemplarily, each computing node can be a physical entity device with data processing capability, wherein the physical entity can be an electronic device with data processing capability such as a terminal or a server, or can be a part of these electronic devices, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc.; wherein one server can serve as a computing node. The terminal can be a smart phone, a tablet computer, a laptop, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, and a vehicle-mounted terminal, etc.; wherein one terminal can serve as a computing node.

[0059] As an example, one or more processors may be configured in an electronic device such as a terminal or server. The processor may be a circuit with the ability to read and run instructions, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a neural-network processing unit (NPU), a tensor processing unit (TPU), etc.; when an electronic device is configured with one processor, the processor may be used as a computing node; when an electronic device is configured with multiple processors, all or part of the multiple processors may be used as a computing node; for example, if an electronic device is configured with multiple GPUs, each GPU may be used as a computing node, or a number of GPUs may be used as a computing node.

[0060] Exemplarily, the distributed computing cluster may further include a management device 102: responsible for allocating computing tasks to each computing node, and monitoring resource usage and task progress of each computing node during model training, etc. Exemplarily, the management device 102 may be software, hardware, or a combination of software and hardware, which is not limited.

[0061] In one scenario, in the process of distributed training of the preset model through the above-mentioned distributed computing cluster, the preset model can be trained in a pipeline parallelism (PP) manner. Pipeline parallelism is a type of model parallelism. The idea is to divide the preset model into different sub-models according to the network layer, and deploy each sub-model on a different computing node to solve the problem that the video memory resources of a single computing node cannot accommodate the entire preset model. Exemplarily, when the preset model is trained in a pipeline parallel manner, the original training data can be divided into multiple micro-batches (mini-batches) and input into the distributed computing cluster, wherein one micro-batch corresponds to one pipeline, and multiple computing nodes in the distributed cluster process the forward calculation and reverse calculation on the pipeline corresponding to each micro-batch in a parallel manner.

[0062] Figure 2 FIG. 2 is a schematic diagram showing the partitioning of model parameters when applying pipeline parallelism according to an embodiment of the present disclosure. Figure 2As shown in FIG. 1 , assuming that there are four computing nodes, the preset model can be split into four sub-models in the order of network layers. For example, it can be split into four sub-models of equal size, where each sub-model contains 1 / 4 model parameters of the preset model. The four sub-models are deployed in sequence on four computing nodes (i.e. Figure 2 In the computing node 1-computing node 4), each computing node stores 1 / 4 of the total model parameters in the preset model. Since the sub-models deployed in each computing node are the preset model split by network layer, there is a data dependency feature when each computing node performs forward calculation (that is, the output result of the previous computing node is the input data of the next computing node). In this way, computing node 2 needs to wait for computing node 1 to complete the forward calculation before it can perform forward calculation, computing node 3 needs to wait for computing node 2 to complete the forward calculation before it can perform forward calculation, and so on; similarly, when each computing node performs reverse calculation, there is also a data dependency feature. In this way, computing node 2 needs to wait for computing node 3 to complete the reverse calculation before it can perform reverse calculation, computing node 4 needs to wait for computing node 3 to complete the reverse calculation before it can perform reverse calculation, and so on.

[0063] In another scenario, during the distributed training of the preset model through the above-mentioned distributed computing cluster, the preset model can be trained by pipeline parallelism and tensor parallelism (TP) at the same time. The idea of ​​tensor parallelism is to use tensors as the basic unit of calculation for parallel processing, split the model parameters into multiple tensors, and assign each tensor to different computing resources for calculation. There is no mutual influence between pipeline parallelism and tensor parallelism, and they are fully compatible.

[0064] Figure 3 FIG. 2 is a schematic diagram showing a model parameter partitioning when pipeline parallelism and tensor parallelism are applied simultaneously according to an embodiment of the present disclosure. Figure 3 As shown, Figure 3 Computing nodes 1 to 4 are as above Figure 2 There are four computing nodes in the computing node, each of which may include four sub-nodes; for a sub-model deployed in any computing node, it can be further divided into four tensors, for example, it can be divided into four tensors of equal size, and the four tensors are deployed on the four sub-nodes of the computing node, wherein each sub-node stores 1 / 16 of the total model parameters in the preset model; these four sub-nodes are called a pipeline parallel communication group, such as Figure 3 As shown in the red box, the four sub-nodes in computing node 2 are a pipeline parallel communication group; and for different computing nodes, such as Figure 3As shown in the middle blue box, each of computing nodes 1 to 4 has a subnode, where the output result of the previous subnode is the input data of the next subnode, and these four subnodes are called a tensor parallel communication group. When performing pipeline parallelism, the computing operations that need to be performed by the subnodes in each pipeline parallel communication group are the same. After each subnode completes the computing operation, the output result of the subnode is transmitted to the next subnode in the same tensor parallel communication group.

[0065] Since pipeline parallelism is usually applied to scenarios with limited video memory resources, and the intermediate results generated during the training process (such as the activation value of the model) occupy a large amount of video memory, when multiple computing nodes in the above-mentioned distributed computing cluster use pipeline parallelism or simultaneously use pipeline parallelism and tensor parallelism to train the model, they can be combined with recalculation technology to further optimize the use of video memory resources: that is, when the computing node performs forward calculations on the pipeline corresponding to each micro-batch of data, it can only save the input data for the forward calculation or only save the input data and some intermediate results generated during the forward calculation process, and the other intermediate results will be recalculated when the computing node performs reverse calculations.

[0066] For example, Figure 4 A schematic diagram of a pipeline parallel combined with recalculation according to an embodiment of the present disclosure is shown, as shown in FIG. Figure 4 As shown, Figure 4 There are 4 computing nodes in the figure, namely device 1, device 2, device 3, and device 4; at the same time, there are 4 mini-batches, where the square and the long block represent the forward and reverse calculations of a mini-batch respectively. Because of data dependency, the forward and reverse calculations on the pipeline corresponding to the same mini-batch are in a step-by-step manner. Taking the training model as the most common large language model in distributed training scenarios as an example, Figure 4 The time consumption of forward calculation and reverse calculation is estimated, and the time consumption of reverse calculation is about twice that of forward calculation; due to the combination of recalculation technology, the computing node only saves the input data or the input data and part of the intermediate results when performing forward calculation on the pipeline corresponding to each micro-batch data, and recalculates the other required intermediate results according to the saved data when performing reverse calculation on the pipeline corresponding to each micro-batch data. The corresponding recalculation time overhead is usually considered to be the same as the forward calculation; assuming that the time consumption of a device for performing forward calculation on the pipeline corresponding to a micro-batch data is 1 unit time, then the time spent on reverse calculation and recalculation on the pipeline corresponding to a micro-batch data by a device is 3 units of time in total, so, Figure 4The four computing nodes perform forward and reverse calculations on the pipelines corresponding to the four micro-batches in parallel. It takes 28 units of time from the start of forward calculation on the first computing node to the completion of reverse calculation on all computing nodes.

[0067] In the above Figure 4 In the scenario of pipeline parallelism combined with heavy calculation shown in the figure, due to the data dependency between the computing nodes; for example, when device 1-device 4 performs forward calculation on the pipeline corresponding to a micro-batch data, device 1 needs to complete the forward calculation and pass the output result of the forward calculation to device 2 before device 2 can start the forward calculation, and so on; in this way, device 1, which performs the forward calculation first, has a long idle time after the forward calculation is completed and before the reverse calculation is started, while device 4, which performs the forward calculation last, is still performing relatively busy calculation work when device 1 is idle, which makes the whole pipeline parallel have more bubbles (i.e., the time when the computing nodes are idle), resulting in unbalanced load of each computing node and low end-to-end computing efficiency. In the related art, bubbles are reduced by increasing the number of micro-batch data, but this method has limited effect on improving load imbalance.

[0068] In order to solve the above technical problems, a load balancing method is provided in an embodiment of the present disclosure (see below for specific description), which utilizes the feature that recalculation has weak data dependence (that is, the intermediate results released by a computing node during forward calculation can be recalculated through the model parameters saved by the computing node and the input data of the forward calculation without the need for data from other computing nodes). In a distributed training scenario where pipeline parallelism and recalculation technology are used simultaneously (or pipeline parallelism, tensor parallelism, and recalculation technology are used simultaneously), the first computing node with more idle time among each computing node can be used to help one or more computing nodes at the back of the pipeline share the recalculation work that does not have data dependence, thereby making full use of the computing resources in the distributed computing cluster, making the load of each computing node more balanced, improving the computing efficiency of distributed training, and shortening the time required for pipeline parallelism.

[0069] It should be noted that the pipeline parallel training scenario and the scenario of simultaneous application of pipeline parallelism and tensor parallelism in the field of machine learning described in the embodiments of the present disclosure are for the purpose of more clearly illustrating the technical solution of the embodiments of the present disclosure, and do not constitute a limitation on the application scenario of the technical solution provided by the embodiments of the present disclosure. A person of ordinary skill in the art can know that the technical solution provided by the embodiments of the present disclosure is also applicable to similar technical problems in response to the emergence of other similar or new scenarios.

[0070] The load balancing method provided by the embodiment of the present disclosure is described in detail below.

[0071] Figure 5FIG. 1 is a flow chart showing a load balancing method according to an embodiment of the present disclosure. Figure 1 The distributed computing cluster or some components in the distributed computing cluster are executed, such as Figure 5 As shown, the method may include the following steps:

[0072] Step 501: input N micro-batch data into a first computing node in sequence, and the multiple computing nodes sequentially perform forward calculations on pipelines corresponding to each micro-batch data in the N micro-batch data, wherein when performing forward calculations on pipelines corresponding to each micro-batch data, an output result of the forward calculation of a previous computing node is used as input data of the forward calculation of a next computing node, and each computing node releases an intermediate result generated by the forward calculation; N is a positive integer, and the first computing node is the computing node that performs forward calculations earliest among the multiple computing nodes.

[0073] The N micro-batches constitute the training data for training the preset model; illustratively, the multiple computing nodes in the distributed computing cluster can form a corresponding number of pipelines based on the number of micro-batches, and one micro-batch corresponds to one pipeline; each pipeline includes the multiple computing nodes in the distributed computing cluster, and the computing nodes on each pipeline are arranged in the same order. Figure 4 In the pipeline parallel scenario shown, there are four micro-batch data, and four devices form four pipelines, wherein each micro-batch data corresponds to a pipeline, and each pipeline is arranged in the order of device 1, device 2, device 3, and device 4.

[0074] Exemplarily, the number of multiple computing nodes and the number of micro-batch data can be set according to demand, and there is no limitation on this. As an example, the number of the multiple computing nodes is N, that is, the number of computing nodes in the distributed computing cluster for training the preset model is the same as the number of micro-batch data input into the distributed cluster.

[0075] Among them, multiple computing nodes train the preset model in a pipeline parallel manner; wherein each computing node is deployed with a sub-model of the preset model split according to the network layer, and different computing nodes deploy different sub-models. In addition, multiple computing nodes can also be used simultaneously with other distributed training methods on the basis of pipeline parallelism, and the strategy compatibility is good; as an example, multiple computing nodes can simultaneously train the preset model in pipeline parallelism and tensor parallelism. Exemplarily, any computing node among the multiple computing nodes includes multiple sub-nodes, wherein the multiple sub-nodes run the sub-models deployed on the corresponding computing nodes in a tensor parallel manner; in this way, the multiple sub-nodes in the same computing node are a pipeline parallel communication group, and each computing node trains the preset model in the form of a pipeline parallel communication group.

[0076] Exemplarily, for any computing node, during the forward calculation of the computing node, the input data is processed based on the deployed sub-model and the output result is obtained. Since the sub-model deployed by each computing node may include one or more network layers, that is, the sub-model deployed by the computing node includes at least one network layer of the multiple network layers that constitute the preset model. In the process of the computing node processing the input data based on the deployed sub-model, the input data will pass through each network layer in the sub-model in turn, and after passing through each network layer, the activation value of the network layer is obtained, which is the intermediate result. Since only the output result of the last network layer in the sub-model needs to be passed to the next computing node, and the intermediate results will occupy the video memory resources of the computing node, these intermediate results can be released, releasing more video memory resources so that the computing node can process other data.

[0077] Exemplarily, the original training data can be split into N micro-batches of data, and the N micro-batches of data are sequentially input into different pipelines for forward calculation, wherein the forward calculations on different pipelines do not interfere with each other. The first computing node in each pipeline is the same computing node, namely the first computing node. In this way, the N micro-batches of data are sequentially input into the first computing node. After the first computing node receives a micro-batch of data, the first computing node processes the micro-batch of data to obtain the output result of the forward calculation, and passes the output result to the next computing node on the pipeline corresponding to the micro-batch of data, and so on, until the last computing node on the pipeline completes the forward calculation, thereby completing the forward calculation on the pipeline corresponding to the micro-batch of data; at the same time, after the first computing node passes the above-mentioned output result to the next computing node on the pipeline corresponding to the micro-batch of data, the first computing node receives Another micro-batch data, then continue to process the other micro-batch data, obtain a new output result of the forward calculation, and pass the new output result to the next computing node on the pipeline corresponding to the other micro-batch data, and so on, until the last computing node on the pipeline completes the forward calculation, thereby completing the forward calculation on the pipeline corresponding to the other micro-batch data; in this way, the forward calculation on the pipeline corresponding to N micro-batch data can be completed, wherein when each computing node performs the forward calculation on the pipeline corresponding to each micro-batch data, the intermediate result generated by the forward calculation will be released; illustratively, part or all of the intermediate results generated by the forward calculation can be released. For example, the above Figure 4 In the pipeline parallel scenario shown, the micro-batch data represented by the red square is input into device 1, and devices 1, 2, 3, and 4 sequentially perform forward calculations on the pipeline corresponding to the red squares, wherein the output result of device 1 is the input data of device 2, the output result of device 2 is the input data of device 3, and the output result of device 3 is the input data of device 4, until device 4 completes the forward calculation; in the process of processing their respective input data, devices 1, 2, 3, and 4 will release the intermediate results generated instead of saving them locally.

[0078] Step 502: After the first computing node completes the forward calculation on the pipeline corresponding to the N micro-batches of data, the first computing node recalculates the intermediate result released by the second computing node when performing the forward calculation on the pipeline corresponding to M1 micro-batches of data; wherein the second computing node is a computing node in a preset queue, and the preset queue includes at least one computing node arranged in the order of performing reverse calculation; M1 is an integer not greater than N.

[0079] Among them, when M1 is equal to N, it indicates that the first computing node recalculates the intermediate results generated by the forward calculation on the pipeline corresponding to all N micro-batch data released by the second computing node; when M1 is not equal to N, it indicates that the first computing node recalculates the intermediate results generated by the forward calculation on the pipeline corresponding to part of the micro-batch data released by the second computing node.

[0080] The training of the preset model by multiple computing nodes in a pipeline parallel manner can be divided into a forward calculation stage and a reverse calculation stage. In the forward calculation stage, each computing node performs the corresponding forward calculation in the order of the pipeline. For any computing node, after the computing node completes the forward calculation on the pipeline corresponding to N micro-batches, it is necessary to wait for all computing nodes to complete the forward calculation on all N micro-batches. The forward calculation stage ends and enters the reverse calculation stage. In the reverse calculation stage, each computing node sequentially performs the reverse calculation on the pipeline corresponding to each micro-batch of the N micro-batches; since the order in which each computing node performs the computing tasks in the forward propagation stage and the reverse propagation stage is opposite, that is, the computing node that performs the forward calculation earlier performs the reverse calculation later, and accordingly, the idle time of the computing node after completing the forward calculation on the pipeline corresponding to N micro-batches is longer. On the contrary, the later the computing node performs forward calculation, the earlier it performs reverse calculation. Accordingly, the shorter the idle time after the computing node finishes the forward calculation on the pipeline corresponding to N micro-batches of data. Therefore, after the first computing node completes the forward calculation on the pipeline corresponding to N micro-batches of data, there will be a long idle time. During this idle time, the first computing node recalculates the intermediate results released by the second computing node when performing the forward calculation on the pipeline corresponding to M1 micro-batches of data, thereby making full use of the computing resources of the first computing node and saving the time spent by the second computing node on recalculation.

[0081] It is understandable that the time overhead for the first computing node to recalculate each computing node in the preset queue needs not exceed the idle time of the first computing node, which is the time from the first computing node completing the forward calculation on the corresponding pipeline of the last micro-batch data in N micro-batches to the first computing node starting the reverse calculation on the corresponding pipeline of the first micro-batch data in N micro-batch data. As an example, the number of computing nodes included in the preset queue does not exceed a specific threshold, and the specific threshold is positively correlated with the number of the multiple computing nodes. Exemplarily, the specific threshold can be 4(n-1) / n+1, where n is the number of multiple computing nodes (i.e., the total number of computing nodes trained on the preset model). Correspondingly, the relationship between the number x of computing nodes included in the preset queue and the number n of multiple computing nodes satisfies the following formula:

[0082]

[0083] In a possible implementation, when there is more than one computing node in the preset queue, the first computing node recalculates the intermediate results released by the second computing node when performing forward calculations on pipelines corresponding to M1 micro-batches, including: the first computing node recalculates the intermediate results released by each computing node in the preset queue in turn according to the arrangement order of each computing node in the preset queue when performing forward calculations on one or more pipelines corresponding to micro-batches. Considering that the preset queue includes at least one computing node arranged in the order of reverse calculation (i.e., in the order of the latest completion of forward calculation) among multiple computing nodes, since the computing node will use the intermediate results released in the forward calculation stage when performing reverse calculation, the first computing node can give priority to recalculating the computing node that starts reverse calculation earlier, so that the computing node can use the intermediate results recalculated by the first computing node when performing reverse calculation.

[0084] In a possible implementation, the first computing node recalculates the intermediate result released by the second computing node when performing forward calculations on the pipeline corresponding to M1 micro-batch data, including: the first computing node obtains the sub-model deployed by the second computing node and the input data of the second computing node performing forward calculations on the pipeline corresponding to the M1 micro-batch data; the first computing node processes the input data of the second computing node performing forward calculations on the pipeline corresponding to the M1 micro-batch data based on the sub-model deployed by the second computing node, and obtains the intermediate result generated when performing forward calculations on the pipeline corresponding to the M1 micro-batch data.

[0085] Exemplarily, the submodel deployed by the second computing node and the input data of the second computing node for forward calculation on the pipeline corresponding to the M1 micro-batch data are transmitted to the first computing node via a higher bandwidth node-to-node (Device-to-Device, D2D) communication link.

[0086] Exemplarily, after completing the recalculation for any second computing node, the first computing node can release the data previously obtained from the second computing node, such as the sub-model deployed by the second computing node, the input data of the forward calculation on the pipeline corresponding to M1 micro-batch data performed by the second computing node, etc. Thus, the limited computing resources of the first computing node can be used to continue to perform other computing tasks.

[0087] In a possible implementation, before the first computing node obtains the sub-model deployed by the second computing node and the input data of the second computing node performing forward calculations on the pipeline corresponding to the M1 micro-batch data, it also includes: the first computing node transfers the output results of the sub-model deployed by the first computing node and the forward calculations performed by the first computing node on the pipeline corresponding to the N micro-batch data to the first storage node.

[0088] Exemplarily, the first storage node may be a node with a data storage function, for example, it may be the CPU memory of the first computing node; in this way, the sub-model deployed by the first computing node and the output result of the forward calculation on the pipeline corresponding to the N micro-batch data performed by the first computing node are transferred to the first storage node, so that the first computing node releases sufficient video memory resources for performing recalculation work for other computing nodes.

[0089] In a possible implementation, the first computing node transfers the output result of the forward calculation on the pipeline corresponding to the N micro-batch data and the first computing node to the first storage node, including: after the first computing node completes the forward calculation on the pipeline corresponding to the i-th micro-batch data in the N micro-batch data, the first computing node transfers the output result of the forward calculation on the pipeline corresponding to the i-th micro-batch data to the first storage node; wherein i is an integer less than N; after the first computing node completes the forward calculation on the pipeline corresponding to all micro-batch data in the N micro-batch data, the sub-model deployed by the first computing node is transferred to the first storage node. In this way, the output result of the forward calculation can be transferred after the first computing node completes the forward calculation corresponding to each micro-batch data, thereby releasing the video memory space of the first computing node in time. At the same time, after the first computing node completes the forward calculation on all N micro-batch data corresponding pipelines, the sub-model deployed by the first computing node is transferred, thereby ensuring that the first computing node can stably complete the forward calculation task.

[0090] Step 503: After the plurality of computing nodes complete the forward calculation on the pipelines corresponding to the N micro-batches, the plurality of computing nodes sequentially perform reverse calculation on the pipelines corresponding to each micro-batch of the N micro-batches, wherein when performing reverse calculation on the pipeline corresponding to each micro-batch, the output result of the reverse calculation of the previous computing node is used as input data for the reverse calculation of the next computing node; when the second computing node performs reverse calculation on the pipelines corresponding to the M1 micro-batches, the second computing node obtains the intermediate result obtained by the first computing node recalculating the second computing node.

[0091] When multiple computing nodes train the preset model in a pipeline parallel manner, multiple computing nodes complete the forward calculation on the pipeline corresponding to N micro-batches of data, then the forward calculation stage ends and enters the reverse calculation stage, and multiple computing nodes sequentially perform reverse calculation on the pipeline corresponding to each micro-batch of data in the N micro-batches of data; in the reverse calculation stage, the order in which each computing node starts reverse calculation is opposite to that in the forward calculation stage. Exemplarily, for any computing node, during the reverse calculation process of the computing node, the input data is processed based on the deployed sub-model and the output result is obtained.

[0092] In a possible implementation, when the second computing node performs reverse calculation on the pipeline corresponding to the M1 micro-batches, the second computing node obtains the intermediate result obtained by the first computing node recalculating the second computing node, including: when the second computing node performs reverse calculation on the pipeline corresponding to the i-th micro-batch among the N micro-batches, the first computing node obtains the intermediate result corresponding to the i-th micro-batch obtained by recalculating the second computing node, that is, the intermediate result released when the second computing node performs forward calculation on the pipeline corresponding to the i-th micro-batch. Exemplarily, the second computing node can receive the intermediate result sent by the first computing node through a D2D communication link.

[0093] In one possible implementation, before the first computing node performs reverse calculation on the pipeline corresponding to each micro-batch of the N micro-batches of data, the first computing node will obtain from the first storage node the sub-model deployed by the first computing node and the output results of the forward calculation performed by the first computing node on the pipeline corresponding to the N micro-batches of data; so that the first computing node can perform reverse calculation.

[0094] In an embodiment of the present disclosure, in a process in which multiple computing nodes in a distributed computing cluster train a preset model in a pipeline parallel manner, N micro-batch data are sequentially input into a first computing node, and the multiple computing nodes sequentially perform forward calculations on the pipeline corresponding to each micro-batch data in the N micro-batch data, and each computing node releases an intermediate result generated by the forward calculation; wherein, the intermediate result generated by the forward calculation of each computing node has no data dependency, that is, the intermediate result generated by the forward calculation of each computing node can be recalculated through the sub-model deployed by the computing node and the input data of the forward calculation, without the need for data from other computing nodes. When performing forward calculation on the pipeline corresponding to each micro-batch data, the output result of the forward calculation of the previous calculation node is used as the input data of the forward calculation of the next calculation node. In this way, for the calculation node that completes the forward calculation on the pipeline corresponding to N micro-batches of data first, it is necessary to wait for all calculation nodes to complete the forward calculation on the pipeline corresponding to N micro-batches of data before each calculation node can perform the reverse calculation on the pipeline corresponding to each micro-batch of data in the N micro-batches of data in turn. Since the order in which each calculation node performs the calculation tasks in the forward calculation and reverse calculation on the same pipeline is opposite, and when performing each micro-batch of data During the reverse calculation on the corresponding pipeline, the output result of the reverse calculation of the previous computing node is used as the input data of the reverse calculation of the next computing node. Therefore, the computing node that performs forward calculation earlier completes the forward calculation earlier and performs reverse calculation later. Accordingly, the computing node that performs forward calculation earlier has a longer idle time after completing the forward calculation on the corresponding pipeline of N micro-batches of data; conversely, the computing node that performs forward calculation later completes the forward calculation later and performs reverse calculation earlier. Accordingly, the computing node that performs forward calculation later has a shorter idle time after completing the forward calculation on the corresponding pipeline of N micro-batches of data. After the first computing node that performs forward calculation earliest completes the forward calculation on the pipeline corresponding to the N micro-batch data, it will enter an idle state. Using the longer idle time, the first computing node recalculates the intermediate results released by the second computing node in the preset queue when performing the forward calculation on the pipeline corresponding to M1 micro-batch data, wherein the preset queue includes at least one computing node arranged in the order of reverse calculation; thereby, the computing resources of the first computing node can be fully utilized to share the recalculation tasks of the second computing node that do not have data dependencies, and the time spent by the second computing node for recalculation is also saved.Furthermore, after the multiple computing nodes have completed the forward calculations on the pipelines corresponding to the N micro-batches, the multiple computing nodes sequentially perform reverse calculations on the pipelines corresponding to each micro-batch of the N micro-batches, and when the second computing node performs reverse calculations on the pipelines corresponding to the M1 micro-batches, the second computing node obtains the intermediate results obtained by the first computing node for recalculating the second computing node, and the second computing node can directly use these intermediate results without recalculation, thereby saving the time overhead of the reverse calculation of the second computing node. In this way, by taking advantage of the weak data dependency of recalculation, the first computing node with more idle time is used to help the second computing node at the back of the pipeline share part or all of the recalculation work, thereby making full use of the computing resources in the distributed computing cluster, making the load of each computing node more balanced, improving the computing efficiency of distributed training, shortening the time consumption of parallel end-to-end pipelines, and making the overall training more efficient and time-saving.

[0095] For example, as an example, Figure 6 A schematic diagram showing a load balancing method according to an embodiment of the present disclosure is shown. Figure 6 The above Figure 4 The four devices and four micro-batches are shown in FIG. 1 ; device 1 is the first device among the four devices to start forward calculation, and device 4 is the last device among the four devices to start forward calculation. After device 1 completes the forward calculation on the pipeline corresponding to the four micro-batches, it starts to recalculate the intermediate results released by device 4 when performing the forward calculation on the pipeline corresponding to the four micro-batches. Figure 6 As shown in the figure, the numbers on the geometric blocks (including squares and rectangles) represent the corresponding micro-batch data, and the red squares (with red numbers) represent that the recalculation of the corresponding micro-batch data originally performed on device 4 has been transferred to device 1. Before device 1 performs recalculation for device 4, the model weights saved on device 4 need to be copied to device 1, and the output results of the forward calculations performed by device 3 on the corresponding pipelines of each micro-batch data are transmitted to device 1. Since the video memory of each device is limited, before the model weights saved on device 4 are copied to device 1, the model weights originally saved on device 1 and the output results of the forward calculations performed on the four corresponding pipelines of micro-batch data can be temporarily transferred to the CPU of device 1, wherein the output results can be transferred after device 1 completes the forward calculations on the corresponding pipelines of each micro-batch data, and the model weights need to be transferred after device 1 completes the forward calculations on all corresponding pipelines of micro-batch data; the intermediate results generated by device 1 when performing the forward calculations on the four corresponding pipelines of micro-batch data can also be released. Figure 6The black square (with red numbers) in the middle indicates the source of input data required by device 1 to recalculate the data released by device 4 (i.e., the intermediate results generated when device 4 performs forward calculations on the corresponding micro-batch data pipeline). The required input data is the output result of device 3 performing forward calculations on the corresponding pipelines of each micro-batch data. It can be seen that there is no waiting due to data dependency for each device under the current execution order. After device 1 helps device 4 complete the recalculation, device 1 can release the model weights of device 4 and retrieve its own model weights from the CPU of device 1 before starting the reverse calculation. Figure 6 The red rectangle (with red numbers) indicates that device 4 performs reverse calculations on the corresponding micro-batch data. After the above load balancing operation, device 1 helps device 4 complete the recalculation work, so that the reverse calculation of device 4 no longer includes recalculation, and the time consumption is reduced from 3 units of time to 2 units of time. The time consumption of the entire pipeline in parallel is reduced by 1 unit of time compared to before the load balancing is performed.

[0096] As another example, Figure 7 A schematic diagram of a load balancing method according to an embodiment of the present disclosure is shown. Figure 7 As shown, Figure 7 The above Figure 4 The four devices and four micro-batches are shown in FIG. 1 ; device 1 is the first device to start forward calculation among the four devices, device 3 is the second to last device to start forward calculation among the four devices, and device 4 is the last device to start forward calculation among the four devices. After device 1 completes the forward calculation on the pipeline corresponding to the four micro-batches, it starts to recalculate the data released by device 4 and device 3 (i.e., the intermediate results generated by device 4 and device 3 when performing forward calculation on the pipeline corresponding to the four micro-batches). Figure 7 As shown in the figure, the numbers on the geometric blocks (including squares and rectangles) represent the corresponding micro-batch data. The red squares (with red numbers) indicate that the recalculation of the corresponding micro-batch data originally performed on device 4 has been transferred to device 1, and the blue squares (with blue numbers) indicate that the recalculation of the corresponding micro-batch data originally performed on device 3 has been transferred to device 1. Device 1 first helps device 4 to complete the recalculation work, and then helps device 3 to complete the recalculation work; before device 1 recalculates for device 4, the model weights originally saved on device 1 and the output results of the forward calculation on the four micro-batch data corresponding pipelines are temporarily transferred to the CPU of device 1, and then the model weights saved on device 4 are copied to device 1, and the output results of the forward calculation on the pipeline corresponding to each micro-batch data performed by device 3 are transmitted to device 1. Figure 7The black square (with red numbers) in the middle indicates the source of input data required when device 1 recalculates the data released by device 4 (i.e., the intermediate results released by device 4 when performing forward calculations on the corresponding micro-batch data pipeline). The required input data is the output result of device 3 performing forward calculations on the corresponding pipelines of each micro-batch data. After device 1 helps device 4 complete the recalculation, device 1 can release the model weights of device 4, copy the model weights saved on device 3 to device 1, and transmit the output results of device 2 performing forward calculations on the corresponding pipelines of each micro-batch data to device 1. Figure 7 The black square (with blue numbers) in the middle indicates the source of input data required when device 1 recalculates the data released by device 3 (i.e., the intermediate results released by device 3 when performing forward calculations on the corresponding micro-batch data pipeline). The required input data is the output result of device 2 performing forward calculations on the corresponding pipelines of each micro-batch data. It can be seen that there is no waiting due to data dependency for each device under the current execution order. After device 1 helps device 3 complete the recalculation, device 1 can release the model weights of device 3 and retrieve its own model weights from the CPU of device 1 before starting the reverse calculation. Figure 7 The red rectangle (with red numbers) indicates that device 4 performs reverse calculations on the corresponding micro-batch data, and the blue rectangle (with blue numbers) indicates that device 3 performs reverse calculations on the corresponding micro-batch data. After the above load balancing operation, device 1 helps devices 4 and 3 complete the recalculation work, so that the reverse calculations of devices 4 and 3 no longer include recalculations. The time consumption for reverse calculations on the pipeline corresponding to a micro-batch of data is reduced from 3 units of time to 2 units of time, and the time consumption for parallel operation of the entire pipeline is reduced by 2 units of time compared to before the load balancing is performed.

[0097] Furthermore, considering the scenario of pipeline parallelism combined with recalculation, when each computing node performs reverse calculation on a pipeline of a micro-batch of data, there is also data dependency between the computing nodes, that is, the output result of the reverse calculation needs to be passed to the next computing node after the previous computing node completes the reverse calculation, and the next computing node can start the reverse calculation. In this way, the computing node that first performs reverse calculation will be in an idle state after the reverse calculation is completed, while at this time, the computing node that performs reverse calculation later (such as the first computing node) is still performing relatively busy calculation work. Therefore, when each computing node performs reverse calculation on the pipeline of each micro-batch of data, the computing node that completes the reverse calculation earliest can be used to help the subsequent computing nodes perform recalculation work.

[0098] Figure 8 FIG. 1 is a flow chart showing a load balancing method according to an embodiment of the present disclosure. Figure 1In the distributed computing cluster 10 or some components in the distributed computing cluster 10, such as Figure 8 As shown, the method may include the following steps:

[0099] Step 801: input N micro-batch data into a first computing node in sequence, and the multiple computing nodes sequentially perform forward calculations on pipelines corresponding to each micro-batch data in the N micro-batch data, wherein when performing forward calculations on pipelines corresponding to each micro-batch data, an output result of the forward calculation of a previous computing node is used as input data of the forward calculation of a next computing node, and each computing node releases an intermediate result generated by the forward calculation; N is a positive integer, and the first computing node is the computing node that performs forward calculations earliest among the multiple computing nodes.

[0100] Step 802: After the first computing node completes the forward calculation on the pipeline corresponding to the N micro-batches of data, the first computing node recalculates the intermediate result released by the second computing node when performing the forward calculation on the pipeline corresponding to M1 micro-batches of data; wherein the second computing node is a computing node in a preset queue, and the preset queue includes at least one computing node arranged in the order of performing reverse calculation; M1 is an integer not greater than N.

[0101] Step 803: After the multiple computing nodes complete the forward calculation on the pipelines corresponding to the N micro-batches, the multiple computing nodes sequentially perform reverse calculation on the pipelines corresponding to each micro-batch of the N micro-batches, wherein when performing reverse calculation on the pipeline corresponding to each micro-batch, the output result of the reverse calculation of the previous computing node is used as input data for the reverse calculation of the next computing node; when the second computing node performs reverse calculation on the pipelines corresponding to the M1 micro-batches, the second computing node obtains the intermediate result obtained by the first computing node recalculating the second computing node.

[0102] Among them, steps 801-803 are the same as above Figure 5 Steps 501-503 are the same.

[0103] Step 804: If the second computing node is the computing node that performs reverse computing earliest among the multiple computing nodes, then after the second computing node completes the reverse computing on the pipeline corresponding to the N micro-batches of data, the second computing node recalculates the intermediate results generated by the forward computing on the pipeline corresponding to M2 micro-batches of data released by the first computing node; M2 is an integer not greater than N.

[0104] Since the first computing node releases the intermediate results generated by the forward calculation when performing forward calculation on the pipeline corresponding to each micro-batch data, that is, the first computing node will release the intermediate results generated by the forward calculation on the pipeline corresponding to N micro-batches of data, therefore, in this step, the second computing node can recalculate the intermediate results generated by the forward calculation on the pipeline corresponding to all micro-batches of the N micro-batches of data released by the first computing node (that is, M2=N); it can also recalculate the intermediate results generated by the forward calculation on the pipeline corresponding to some micro-batches of the N micro-batches of data released by the first computing node (that is, M2<N).

[0105] Exemplarily, the time when the second computing node recalculates the intermediate result generated by the forward calculation on the pipeline corresponding to a micro-batch data released by the first computing node is no later than the time when the first computing node starts the reverse calculation of the pipeline corresponding to the micro-batch data.

[0106] In a possible implementation, the second computing node recalculates the intermediate result generated by the forward calculation on the pipeline corresponding to the M2 micro-batch data released by the first computing node, including: the second computing node obtains the sub-model deployed by the first computing node and the M2 micro-batch data; the second computing node processes the M2 micro-batch data based on the sub-model deployed by the first computing node to obtain the intermediate result generated by the forward calculation on the pipeline corresponding to the M2 micro-batch data.

[0107] Exemplarily, the sub-model deployed by the first computing node is transmitted to the second computing node via a higher bandwidth node-to-node (Device-to-Device, D2D) communication link. M2 micro-batches of data can be selected from the original input N micro-batches of data of the entire distributed computing cluster.

[0108] In a possible implementation, before the second computing node obtains the sub-model deployed by the first computing node and the M2 micro-batch data, it also includes: the second computing node transfers the sub-model deployed by the second computing node to a second storage node.

[0109] Exemplarily, the second storage node can be a node with data storage function, for example, it can be the CPU memory of the second computing node; in this way, the sub-model deployed by the second computing node is transferred to the second storage node so that the second computing node releases sufficient video memory resources to help the first computing node complete the recalculation work.

[0110] Exemplarily, after the second computing node completes the reverse calculation on the pipeline corresponding to N micro-batches of data, the data generated during the reverse calculation process can be released so that the second computing node releases sufficient video memory resources to help the first computing node complete the recalculation work.

[0111] Step 805: When the first computing node performs reverse calculation on the pipeline corresponding to the M2 micro-batch data, the first computing node obtains the intermediate result obtained by the second computing node through recalculation.

[0112] Exemplarily, when the first computing node performs reverse calculation on the pipeline corresponding to the i-th micro-batch data in the M2 micro-batches, the first computing node obtains the intermediate result corresponding to the i-th micro-batch data obtained by the second computing node recalculating the first computing node (i.e., the intermediate result released when the first computing node performs forward calculation on the pipeline corresponding to the i-th micro-batch), so as to perform reverse calculation on the pipeline corresponding to the i-th micro-batch. Exemplarily, the first computing node can receive the intermediate result sent by the second computing node through a D2D communication link.

[0113] In the disclosed embodiment, in the process of multiple computing nodes training a preset model in a pipeline parallel manner, the first computing node that first performs forward computing is used to help computing nodes that are later in the forward computing stage share part or all of the recomputation work by utilizing the characteristic that recomputation has weak data dependence. At the same time, the second computing node that first performs reverse computing is used to help computing nodes that are later in the reverse computing stage share part or all of the recomputation work, thereby making full use of the computing resources in the distributed computing cluster, making the load of each computing node more balanced, improving the computing efficiency of distributed training, shortening the time consuming of end-to-end pipeline parallelism, and making the overall training more efficient and time-saving. Compared with not adopting this load balancing method, the entire training can save a total of x+1 units of time, where x represents the number of computing nodes included in the preset queue.

[0114] For example, Fig. 9 A schematic diagram showing a load balancing method according to an embodiment of the present disclosure is shown. Fig. 9 The above Figure 4The four devices and four micro-batches are shown; device 1 is the first device to start forward calculation among the four devices, device 3 is the second to last device to start forward calculation among the four devices, and device 4 is the last device to start forward calculation among the four devices. After device 1 completes the forward calculation on the pipeline corresponding to the four micro-batches, it starts to recalculate the data released by device 4 and device 3 (i.e., the intermediate results generated by device 4 and device 3 when performing forward calculation on the pipeline corresponding to the four micro-batches); after device 4 completes the reverse calculation on the pipeline corresponding to the four micro-batches, it starts to recalculate the data released by device 1 during forward calculation (i.e., the intermediate results generated by device 1 when performing forward calculation on the pipeline corresponding to some micro-batches). Fig. 9 As shown in the figure, the numbers on the geometric blocks (including squares and rectangles) represent the corresponding micro-batch data. The red squares (with red numbers) indicate that the recalculation of the corresponding micro-batch data originally performed on device 4 has been transferred to device 1. The blue squares (with blue numbers) indicate that the recalculation of the corresponding micro-batch data originally performed on device 3 has been transferred to device 1. The yellow squares (with yellow numbers) indicate that the recalculation of the corresponding micro-batch data originally performed on device 1 has been transferred to device 4. Device 1 first helps device 4 to complete the recalculation work, and then helps device 3 to complete the recalculation work; before device 1 recalculates for device 4, the model weights originally saved on device 1 and the output results of the forward calculation on the four micro-batch data corresponding pipelines are temporarily transferred to the CPU of device 1, and then the model weights saved on device 4 are copied to device 1, and the output results of the forward calculation on the pipeline corresponding to each micro-batch data performed by device 3 are transmitted to device 1. Fig. 9 The black square (with red numbers) in the middle indicates the source of input data required by device 1 for recalculating the data released by device 4 (i.e., the intermediate results released by device 4 when performing forward calculations on the corresponding micro-batch data pipeline). The required input data is the output result of device 3 performing forward calculations on the corresponding pipelines of each micro-batch data. After device 1 helps device 4 complete the recalculation, device 1 can release the model weights of device 4, copy the model weights saved on device 3 to device 1, and transmit the output results of device 2 performing forward calculations on the corresponding pipelines of each micro-batch data to device 1. Fig. 9The black square (with blue numbers) in the middle indicates the source of input data required by device 1 to recalculate the data released by device 3 (i.e., the intermediate results released by device 3 when performing forward calculations on the corresponding micro-batch data pipeline). The required input data is the output result of device 2 when performing forward calculations on the corresponding pipelines of each micro-batch data. After device 1 helps device 3 complete the recalculation, device 1 can release the model weights of device 3 and retrieve its own model weights from the CPU of device 1 before starting the reverse calculation. Then, after device 4 completes the reverse calculations on the corresponding pipelines of these four micro-batches, it helps device 1 complete the recalculation work. Since device 1 starts the reverse calculation earlier than device 4 completes the reverse calculation, the model weights saved on device 1 can be copied to device 4. Device 4 recalculates the intermediate results corresponding to the three micro-batches after device 1 completes the calculation. Fig. 9 The yellow square (with yellow numbers) indicates that the recalculation of the corresponding micro-batch data originally performed on device 1 has been transferred to device 4; Fig. 9 The yellow rectangle (with yellow numbers) indicates that device 1 performs reverse calculations for the corresponding micro-batch data. After the above load balancing operation, device 1 helps device 4 and device 3 complete all recalculation work, so that the reverse calculations of device 4 and device 3 no longer include recalculation. At the same time, device 4 helps device 1 complete part of the recalculation work, so that the reverse calculation of device 1 only needs to be recalculated when performing the reverse calculation on the pipeline corresponding to the first micro-batch data. It can be seen that under the current execution order, there is no waiting caused by data dependence on each device. In this way, the end-to-end parallel time of the pipeline can be reduced to 25 units of time.

[0115] Fig.10 A flow chart of another load balancing method according to an embodiment of the present disclosure is shown. Fig.10As shown, it is possible to calculate in advance how many of these devices device 1 (i.e., the first device to start forward calculation) can do recalculation work for according to the total number of devices participating in the pipeline parallel, and insert the devices that need device 1 to help recalculate into the device queue from back to front in the order of starting forward calculation. This device queue is the preset queue. Then, when device 1 performs forward calculation of the model on the current device (i.e., the sub-model deployed by device 1), first determine whether the current micro-batch data (micro-batch) has been executed. If not, continue to execute. If the current micro-batch data has been executed, determine whether the forward calculation work of device 1 is being executed, that is, whether the forward calculation of the pipeline corresponding to each micro-batch data is being executed based on the sub-model deployed by device 1; if the forward calculation work of device 1 is not being executed, it means that device 1 is helping other devices to perform recalculation work, and the intermediate results of the recalculation need to be saved; if the forward calculation work of device 1 is being executed, the output result (activation value) of the forward calculation is transferred to the CPU of device 1. and further determine whether it is the last micro-batch data; if it is not the last micro-batch data, select the next micro-batch data and repeat the above operation; if it is the last micro-batch data, further determine whether the device queue is empty. If it is not empty, device 1 performs three communication operations: 1. Transfer the model of device 1 to the CPU of device 1, 2. Copy the model of the current head device in the device queue to device 1, 3. Set the source of the forward calculation input data of device 1 to the output result of the corresponding micro-batch data in the previous device of the current head device in the device queue; after the above communication operations are completed, the current head device is removed from the device queue, and device 1 continues to perform forward calculations on the model on device 1. During the forward calculation process of the model on the current device performed by the above-mentioned device 1, each device in the device queue performs forward calculation of the model on the corresponding device, and uses the intermediate results recalculated for each device saved by device 1 during reverse calculation; if it is the last device in the device queue, that is, the device that performs forward calculation last, then after the device completes the reverse calculation task, three communication operations are performed simultaneously: 1. Transfer the model on this device to the CPU of the model, 2. Copy the model of device 1 to this device, 3. Set the source of the forward calculation input data to the micro-batch data input by device 1; then, the device helps device 1 to perform recalculation and saves the recalculated intermediate results; when the above-mentioned device 1 completes the forward calculation task and continues to perform reverse calculation, the intermediate results recalculated for device 1 saved by the device are used.

[0116] Based on the same inventive concept of the above method embodiment, the embodiment of the present disclosure also provides a load balancing device, which can be used to execute the technical solution described in the above method embodiment. Figure 3-Figure 10The steps of the load balancing method shown in .

[0117] Fig.11 A structural diagram of a load balancing device according to an embodiment of the present disclosure is shown, and the device is applied to a distributed computing cluster, and multiple computing nodes in the distributed computing cluster train a preset model in a pipeline parallel manner; wherein each computing node is deployed with a sub-model of the preset model split according to the network layer, and different computing nodes deploy different sub-models; Fig.11 As shown, the device includes: a forward calculation module 1101, which is used to input N micro-batch data into the first computing node in sequence, and the multiple computing nodes sequentially perform forward calculations on the pipelines corresponding to each micro-batch data in the N micro-batch data, wherein, when performing forward calculations on the pipelines corresponding to each micro-batch data, the output result of the forward calculation of the previous computing node is used as the input data of the forward calculation of the next computing node, and each computing node releases the intermediate result generated by the forward calculation; N is a positive integer, and the first computing node is the computing node that performs forward calculations earliest among the multiple computing nodes; a recalculation module 1102, which is used to, after the first computing node completes the forward calculations on the pipelines corresponding to the N micro-batch data, the first computing node recalculates the intermediate result released by the second computing node when performing forward calculations on the pipelines corresponding to M1 micro-batch data intermediate result; wherein, the second computing node is a computing node in a preset queue, and the preset queue includes at least one computing node arranged in the order of reverse calculation; M1 is an integer not greater than N; a reverse calculation module 1103, which is used for, after the multiple computing nodes have completed the forward calculation on the pipeline corresponding to the N micro-batch data, the multiple computing nodes sequentially perform reverse calculation on the pipeline corresponding to each micro-batch data in the N micro-batch data, wherein, when performing reverse calculation on the pipeline corresponding to each micro-batch data, the output result of the reverse calculation of the previous computing node is used as the input data of the reverse calculation of the next computing node; when the second computing node performs reverse calculation on the pipeline corresponding to the M1 micro-batch data, the second computing node obtains the intermediate result obtained by the first computing node recalculating the second computing node.

[0118] In one possible implementation, the recalculation module 1102 is further used for: if the second computing node is the earliest computing node to perform reverse calculation among the multiple computing nodes, then after the second computing node completes the reverse calculation on the pipeline corresponding to the N micro-batches of data, the second computing node recalculates the intermediate result generated by the forward calculation on the pipeline corresponding to M2 micro-batches of data released by the first computing node; M2 is an integer not greater than N; when the first computing node performs reverse calculation on the pipeline corresponding to the M2 micro-batches of data, the first computing node obtains the intermediate result obtained by the recalculation performed by the second computing node.

[0119] In one possible implementation, the recalculation module 1102 is further used for: the first computing node obtains the sub-model deployed by the second computing node and the input data of the second computing node performing forward calculations on the pipeline corresponding to the M1 micro-batch data; the first computing node processes the input data of the second computing node performing forward calculations on the pipeline corresponding to the M1 micro-batch data based on the sub-model deployed by the second computing node, and obtains the intermediate results generated when performing forward calculations on the pipeline corresponding to the M1 micro-batch data.

[0120] In a possible implementation, the recalculation module 1102 is also used for: the first computing node transfers the sub-model deployed by the first computing node and the output result of the forward calculation performed by the first computing node on the pipeline corresponding to the N micro-batch data to the first storage node.

[0121] In a possible implementation, the recalculation module 1102 is further used for: after the first computing node completes the forward calculation on the pipeline corresponding to the i-th micro-batch data among the N micro-batch data, the first computing node transfers the output result of the forward calculation on the pipeline corresponding to the i-th micro-batch data to the first storage node; wherein i is an integer less than N; after the first computing node completes the forward calculation on the pipeline corresponding to all micro-batch data among the N micro-batch data, the sub-model deployed by the first computing node is transferred to the first storage node.

[0122] In one possible implementation, the recalculation module 1102 is further used for: the second computing node obtains the sub-model deployed by the first computing node and the M2 micro-batch data; the second computing node processes the M2 micro-batch data based on the sub-model deployed by the first computing node to obtain the intermediate results generated when performing forward calculations on the pipeline corresponding to the M2 micro-batch data.

[0123] In a possible implementation, the recalculation module 1102 is further configured to: the second computing node transfers the sub-model deployed by the second computing node to the second storage node.

[0124] In a possible implementation, when there is more than one computing node in the preset queue, the recalculation module 1102 is further used for: the first computing node recalculates the intermediate results released by each computing node in the preset queue in the forward calculation on the pipeline corresponding to one or more micro-batch data in sequence according to the arrangement order of the computing nodes in the preset queue.

[0125] In a possible implementation manner, the number of the multiple computing nodes is N.

[0126] In a possible implementation manner, the number of computing nodes included in the preset queue does not exceed a specific threshold, and the specific threshold is positively correlated with the number of the plurality of computing nodes.

[0127] In a possible implementation, any computing node among the multiple computing nodes includes multiple sub-nodes, wherein the multiple sub-nodes run the sub-models deployed on the corresponding computing nodes in a tensor parallel manner.

[0128] Above Fig.11 The technical effects and specific descriptions of the load balancing device shown and its various possible implementations can be found in the above-mentioned load balancing method, which will not be repeated here.

[0129] It should be understood that the division of the modules in the above device is only a division of logical functions, and in actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the modules in the device can be implemented in the form of a processor calling software; for example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the modules of the device, wherein the processor is, for example, a general-purpose processor, such as a CPU or a microprocessor, and the memory is a memory inside the device or a memory outside the device. Alternatively, the modules in the device can be implemented in the form of hardware circuits, and the functions of some or all modules can be realized by designing the hardware circuits, and the hardware circuits can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all modules above are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be realized by a programmable logic device (PLD), taking a field programmable gate array (FPGA) as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all modules above. All modules of the above device can be realized in the form of a processor calling software, or in the form of a hardware circuit, or in part by a processor calling software, and the rest by a hardware circuit.

[0130] In the embodiments of the present disclosure, the processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a neural-network processing unit (NPU), a tensor processing unit (TPU), etc.; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules.

[0131] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the above embodiment method, such as: CPU, GPU, NPU, TPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms. In addition, each module in the above device can be fully or partially integrated together, or can be implemented independently, which is not limited.

[0132] The present disclosure also provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the method of the above embodiment when executing the instructions. Figure 3-Figure 10 The steps of the load balancing method shown in .

[0133] Fig.12 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.12 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0134] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0135] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0136] The embodiments of the present disclosure provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method in the above embodiments is implemented. Figure 3-Figure 10The steps of the load balancing method shown in .

[0137] The embodiments of the present disclosure provide a computer program product, which may include, for example, a computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code; when the computer program product is run on a computer, the computer executes the method in the above embodiment. Figure 3-Figure 10 The steps of the load balancing method shown in .

[0138] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0139] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction-performing device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber-optic cable), or an electrical signal transmitted through a wire.

[0140] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0141] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be performed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may perform computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0142] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0143] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0144] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions performed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0145] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more instructions that can be performed to realize the specified logical function. In some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be performed substantially in parallel, and they can sometimes be performed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special-purpose hardware and computer instructions.

[0146] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A load balancing method, characterized in that: Applied to a distributed computing cluster, multiple computing nodes in the distributed computing cluster train the preset model in a pipeline parallel manner; wherein each computing node is deployed with a sub-model of the preset model split according to the network layer, and different computing nodes deploy different sub-models; The method comprises: Inputting N micro-batch data into a first computing node in sequence, the multiple computing nodes sequentially performing forward calculations on pipelines corresponding to each micro-batch data in the N micro-batch data, wherein when performing forward calculations on pipelines corresponding to each micro-batch data, an output result of the forward calculation of a previous computing node is used as input data of the forward calculation of a next computing node, and each computing node releases an intermediate result generated by the forward calculation; N is a positive integer, and the first computing node is the computing node that performs forward calculations earliest among the multiple computing nodes; After the first computing node completes the forward calculation on the pipeline corresponding to the N micro-batches, the first computing node recalculates the intermediate result released by the second computing node when performing the forward calculation on the pipeline corresponding to the M1 micro-batches; wherein the second computing node is a computing node in a preset queue, and the preset queue includes at least one computing node arranged in the order of performing reverse calculation; M1 is an integer not greater than N; After the multiple computing nodes complete the forward calculation on the pipelines corresponding to the N micro-batches, the multiple computing nodes sequentially perform reverse calculation on the pipelines corresponding to each micro-batch of the N micro-batches, wherein when performing reverse calculation on the pipeline corresponding to each micro-batch, the output result of the reverse calculation of the previous computing node is used as input data for the reverse calculation of the next computing node; when the second computing node performs reverse calculation on the pipelines corresponding to the M1 micro-batches, the second computing node obtains the intermediate result obtained by the first computing node recalculating the second computing node.

2. The method according to claim 1, characterized in that The method further comprises: If the second computing node is the computing node that performs reverse computing earliest among the multiple computing nodes, then after the second computing node completes the reverse computing on the pipeline corresponding to the N micro-batches of data, the second computing node recalculates the intermediate results released by the first computing node when performing forward computing on the pipeline corresponding to M2 micro-batches of data; M2 is an integer not greater than N; When the first computing node performs reverse calculation on the pipeline corresponding to the M2 micro-batch data, the first computing node obtains an intermediate result obtained by the recalculation performed by the second computing node.

3. The method according to claim 1, characterized in that The first computing node recalculates the intermediate results released by the second computing node when performing forward calculations on the pipeline corresponding to the M1 micro-batch data, including: The first computing node obtains the sub-model deployed by the second computing node and the input data of the second computing node for performing forward calculation on the pipeline corresponding to the M1 micro-batch data; The first computing node processes the input data of the forward calculation on the pipeline corresponding to the M1 micro-batch data of the second computing node based on the sub-model deployed by the second computing node, and obtains the intermediate result generated when the forward calculation on the pipeline corresponding to the M1 micro-batch data is performed.

4. The method according to claim 3, characterized in that Before the first computing node obtains the sub-model deployed by the second computing node and the second computing node performs the input data of the forward calculation on the pipeline corresponding to the M1 micro-batch data, the method further includes: The first computing node transfers the sub-model deployed by the first computing node and the output result of the forward calculation performed by the first computing node on the pipeline corresponding to the N micro-batch data to the first storage node.

5. The method according to claim 4, characterized in that The first computing node transfers the sub-model deployed by the first computing node and the output result of the forward calculation performed by the first computing node on the pipeline corresponding to the N micro-batch data to the first storage node, including: After the first computing node completes the forward calculation on the pipeline corresponding to the i-th micro-batch data among the N micro-batch data, the first computing node transfers the output result of the forward calculation on the pipeline corresponding to the i-th micro-batch data to the first storage node; wherein i is an integer less than N; After the first computing node completes forward calculations on pipelines corresponding to all micro-batches of the N micro-batches, the sub-model deployed by the first computing node is transferred to the first storage node.

6. The method according to claim 2, characterized in that The second computing node recalculates the intermediate results generated when the first computing node performs forward calculations on the pipeline corresponding to the M2 micro-batch data, including: The second computing node obtains the sub-model deployed by the first computing node and the M2 micro-batch data; The second computing node processes the M2 micro-batch data based on the sub-model deployed by the first computing node to obtain intermediate results generated when performing forward calculations on the pipeline corresponding to the M2 micro-batch data.

7. The method according to claim 6, characterized in that Before the second computing node obtains the sub-model deployed by the first computing node and the M2 micro-batch data, the method further includes: The second computing node transfers the sub-model deployed by the second computing node to the second storage node.

8. The method according to claim 1, characterized in that In the case where there is more than one computing node in the preset queue, the first computing node recalculates the intermediate result released by the second computing node when performing the forward calculation on the pipeline corresponding to the M1 micro-batch data, including: The first computing node recalculates the intermediate results released by each computing node in the preset queue in sequence when performing forward calculations on one or more pipelines corresponding to micro-batch data.

9. The method according to claim 1, characterized in that: The number of the multiple computing nodes is N.

10. The method according to claim 9, characterized in that The number of computing nodes included in the preset queue does not exceed a specific threshold, and the specific threshold is positively correlated with the number of the plurality of computing nodes.

11. The method according to any one of claims 1 to 10, characterized in that Any computing node among the multiple computing nodes includes multiple sub-nodes, wherein the multiple sub-nodes run the sub-models deployed on the corresponding computing nodes in a tensor parallel manner.

12. A load balancing device, characterized in that: Applied to a distributed computing cluster, multiple computing nodes in the distributed computing cluster train the preset model in a pipeline parallel manner; wherein each computing node is deployed with a sub-model of the preset model split according to the network layer, and different computing nodes deploy different sub-models; The device comprises: A forward computing module, used to sequentially input N micro-batch data into a first computing node, wherein the plurality of computing nodes sequentially perform forward computing on a pipeline corresponding to each micro-batch data in the N micro-batch data, wherein when performing forward computing on a pipeline corresponding to each micro-batch data, an output result of the forward computing of a previous computing node is used as input data of the forward computing of a subsequent computing node, and each computing node releases an intermediate result generated by the forward computing; N is a positive integer, and the first computing node is the computing node that performs forward computing earliest among the plurality of computing nodes; a recalculation module, configured to, after the first computing node completes the forward calculation on the pipeline corresponding to the N micro-batches of data, cause the first computing node to recalculate the intermediate result released by the second computing node when performing the forward calculation on the pipeline corresponding to the M1 micro-batches of data; wherein the second computing node is a computing node in a preset queue, and the preset queue includes at least one computing node arranged in the order of performing reverse calculation; and M1 is an integer not greater than N; A reverse calculation module is used for, after the multiple computing nodes have completed the forward calculation on the pipelines corresponding to the N micro-batches of data, the multiple computing nodes sequentially perform reverse calculation on the pipelines corresponding to each micro-batch of data in the N micro-batches of data, wherein, when performing reverse calculation on the pipeline corresponding to each micro-batch of data, the output result of the reverse calculation of the previous computing node is used as the input data of the reverse calculation of the next computing node; when the second computing node performs reverse calculation on the pipelines corresponding to the M1 micro-batches of data, the second computing node obtains the intermediate result obtained by the first computing node recalculating the second computing node.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by a processor; Wherein, the processor is configured to implement the method described in any one of claims 1 to 11 when executing the instructions stored in the memory.

14. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images