Model training method and apparatus

Abstract

The present application provides a model training method and an apparatus. The method comprises: deploying a first part of a target model into a first cluster, and deploying a second part into a second cluster; instructing a first node in the first cluster to execute first computation of a first model block, so as to obtain a first result, the first node being used for sending the first result to the second cluster by means of a link between the first node and the second cluster; and, when the duration of the first node sending the first result to the second cluster is longer than the duration of the first node executing the first computation of the first model block, instructing k nodes in the first cluster to alternately execute the first computation of the first model block, wherein each of the k nodes is used for sending, by means of a link between the node and the second cluster, to the second cluster a computation result of executing the first computation of the first model block by the node, k being an integer greater than 1. The method can eliminate the restriction of cross-region communication on model training efficiency.

Description

A model training method and device

[0001] This application claims priority to the Chinese Patent Application No. 202410726984.5, filed on June 5, 2024, entitled “A Model Training Method and Device”, and the Chinese Patent Application No. 202411295419.4, filed on September 14, 2024, entitled “A Model Training Method and Device”, both of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a model training method and device. BACKGROUND

[0003] With the development of AI technology, large-scale models have become a trend. For example, the parameter quantity of a large language model (LLM) has currently reached several billion, and the parameter quantity of the LLM is still growing exponentially. Due to the huge parameter quantity, the training of a large-scale model requires a large number of hardware devices. Limited by power, cooling capacity, and networking capability, the hardware devices in a single region are already difficult to meet the training needs of a large-scale model. Therefore, it is necessary to deploy the training task of a large-scale model across regions, that is, to train a large-scale model using hardware devices in two or more regions.

[0004] The cross-region deployment of a training task requires cross-region communication. Compared with the communication capability between hardware devices in the same region, the communication capability between hardware devices in different regions is weaker. Therefore, the communication capability between hardware devices in different regions is often a bottleneck that restricts the training efficiency. SUMMARY

[0005] The present application provides a model training method and device, which can eliminate the restriction of cross-region communication on the efficiency of model training.

[0006] In a first aspect, a model training method is provided. The method is applied to a management device in a training system. The training system further includes a first cluster and a second cluster. The first cluster includes a plurality of nodes. Different nodes in the plurality of nodes are connected to the second cluster through different links. The method includes: deploying a first part of a target model to the first cluster and a second part to the second cluster. The first part includes a first model block of the target model. A calculation result of a first calculation of the first model block is an input of the second part. A first node in the first cluster is instructed to perform the first calculation of the first model block to obtain a first result. The first node is configured to send the first result to the second cluster through a link between the first node and the second cluster. In a case where a time length of sending the first result from the first node to the second cluster is greater than a time length of performing the first calculation of the first model block by the first node, k nodes in the first cluster are instructed to alternately perform the first calculation of the first model block. Each of the k nodes is configured to send a calculation result of performing the first calculation of the first model block by the node to the second cluster through a link between the node and the second cluster. k is an integer greater than 1. Exemplarily, a communication capability of the link between the nodes in the first cluster and the second cluster is less than a communication capability of the links between different nodes in the first cluster. The first cluster and the second cluster can be arranged in different regions, for example, different cities or regions.

[0007] In the method, the k nodes alternately perform the first calculation of the first model block. Each node sends a calculation result of performing the first calculation of the first model block by the node to the second cluster through a link between the node and the second cluster. Moreover, the links between the k nodes and the second cluster are independent of each other. In this way, the calculation result of the first calculation of the first model block is sent to the second cluster through the k links, which can significantly shorten the sending time length. When a batch of data sets is calculated, the node can send the calculation result in time, avoiding the consumption of the calculation result on the node memory and ensuring that the calculation result can be sent to the second cluster in time, so that the nodes in the second cluster can obtain data in time and perform calculation. Therefore, the method can eliminate the restriction of cross-cluster communication on the model training efficiency and improve the training efficiency of the target model and the utilization rate of the computing resources of the nodes.

[0008] In a possible implementation, the first part includes k model blocks of the target model. The k model blocks are adjacent in a data transmission direction of the target model. In the data transmission direction, the first model block is the last model block in the k model blocks. The method of instructing the k nodes in the first cluster to alternately perform the first calculation of the first model block includes: merging the k model blocks to obtain a sub-model; and instructing the k nodes to alternately perform a first calculation of the sub-model.

[0009] Compared with the k nodes only performing the first model block, the k nodes alternately perform the sub-model composed of the k model blocks, which can ensure that the calculation result is sent in time and can avoid waste of the calculation resources of the k nodes, and improve the utilization of the calculation resources of the k nodes.

[0010] In a possible implementation, the method further includes: in a case where a time length of the second node in the k nodes sending the second result to the second cluster is greater than a time length of the second node performing the first calculation of the sub-model, instructing the k+m nodes in the first cluster to alternately perform the first calculation of the sub-model; wherein each node in the k+m nodes is configured to send a calculation result of the node performing the first calculation of the sub-model to the second cluster through a link between the node and the second cluster; wherein the second result is the calculation result of the second node performing the first calculation of the sub-model; and m is an integer greater than or equal to 1.

[0011] The calculation result of the first calculation of the sub-model is also the calculation result of the first calculation of the first model block. In a case where the time length of sending the calculation result of the first calculation of the first model block to the second cluster through the k+m links is still greater than the time length of the node performing the sub-model, the node performing the sub-model or the first model block can be increased, so as to increase the number of links for sending the calculation result of the first calculation of the first model block to the second cluster, so as to further shorten the sending time length of the calculation result of the first calculation of the first model block.

[0012] In a possible implementation, the k nodes are nodes with the largest memory in the first cluster.

[0013] The nodes in the k nodes may need to perform the calculation of the sub-model composed of the k model blocks, and therefore need to load the parameters of the k model blocks, which consumes more memory. In this implementation, the node with the largest memory in the cluster is used to perform the calculation of the sub-model, which ensures the smooth execution of the calculation of the sub-model.

[0014] In a possible implementation, the first cluster includes system memory, and the system memory is used for each node in the k nodes to perform the first calculation of the first model block.

[0015] Each node in the k nodes needs to perform the calculation of the sub-model composed of the k model blocks, and therefore needs to load the parameters of the k model blocks, which consumes more memory. The system memory in the cluster is used to perform the calculation of the sub-model, which ensures the smooth execution of the calculation of the sub-model.

[0016] In a possible implementation, the time length of the first node sending the first result to the second cluster is greater than the time length of the first node performing the first calculation of the first model block, including: the communication time length between the first node and the second cluster is greater than the calculation time length of the first model block; wherein the calculation time length is the sum of the time length of the first node performing the first calculation of the first model block and the time length of the first node performing the second calculation of the first model block; the communication time length is the sum of the time length of the first node sending the first result to the second cluster and the time length of the first node receiving the calculation result of the second calculation of the second part from the second cluster; the first calculation is one of the forward calculation and the backward calculation, and the second calculation is the other of the forward calculation and the backward calculation.

[0017] This implementation can ensure the timely sending of the calculation result of the forward calculation, and can also ensure the timely sending of the calculation result of the backward calculation, thereby ensuring the efficiency of the forward propagation and the efficiency of the backward propagation at the same time, and improving the model training efficiency.

[0018] In a second aspect, a management device is provided. A training system in which the management device is located further includes a first cluster and a second cluster. The first cluster includes a plurality of nodes, and different nodes in the plurality of nodes are connected to the second cluster through different links. The management device includes: a deployment module configured to deploy a first part of a target model to the first cluster and a second part to the second cluster; the first part includes a first model block of the target model, and a calculation result of a first calculation of the first model block is an input of the second part; a first indication module configured to instruct a first node in the first cluster to perform the first calculation of the first model block to obtain a first result; the first node is configured to send the first result to the second cluster through a link between the first node and the second cluster; and a second indication module configured to instruct k nodes in the first cluster to alternately perform the first calculation of the first model block in a case where a time length of the first node sending the first result to the second cluster is greater than a time length of the first node performing the first calculation of the first model block; each node of the k nodes is configured to send a calculation result of the first node performing the first calculation of the first model block to the second cluster through a link between the node and the second cluster; and k is an integer greater than 1.

[0019] In a possible implementation, the first part includes k model blocks of the target model, the k model blocks are sequentially adjacent in a data transmission direction of the target model, and in the data transmission direction, the first model block is the last model block in the k model blocks; and the second indication module is configured to: combine the k model blocks to obtain a sub-model; and instruct the k nodes to alternately perform a first calculation of the sub-model.

[0020] In a possible implementation, the second indication module is further configured to instruct the k+m nodes in the first cluster to alternately perform the first calculation of the sub-model in a case where a time length for the second node in the k nodes to perform the first calculation of the sub-model is greater than a time length for the second node to send the second result to the second cluster; each of the k+m nodes is configured to send a calculation result of performing the first calculation of the sub-model by the node to the second cluster through a link between the node and the second cluster; the second result is the calculation result of performing the first calculation of the sub-model by the second node; and m is an integer greater than or equal to 1.

[0021] In a possible implementation, the k nodes are nodes with the largest memory in the first cluster; or the first cluster includes system memory, and the system memory is used for each of the k nodes to perform the first calculation of the first model block.

[0022] In a possible implementation, the time length for the first node to send the first result to the second cluster is greater than the time length for the first node to perform the first calculation of the first model block, including that a communication time length between the first node and the second cluster is greater than a calculation time length of the first model block; the calculation time length is a sum of the time length for the first node to perform the first calculation of the first model block and a time length for the first node to perform the second calculation of the first model block; the communication time length is a sum of a time length for the first node to send the first result to the second cluster and a time length for the first node to receive the calculation result of the second part of the second calculation from the second cluster; the first calculation is one of a forward calculation and a backward calculation, and the second calculation is the other of the forward calculation and the backward calculation.

[0023] In a possible implementation, a communication capability of a link between a node in the first cluster and the second cluster is less than a communication capability of a link between different nodes in the first cluster.

[0024] In a third aspect, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method provided in the first aspect.

[0025] In a fourth aspect, a computer-readable storage medium is provided, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0026] In a fifth aspect, a computer program product including instructions is provided, when the instructions are run by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0027] The beneficial effects of the second aspect to the fifth aspect can refer to the beneficial effects of the first aspect described above, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0028] FIG. 1 is a schematic diagram of a training system according to an embodiment of the present application;

[0029] FIG. 2 is a schematic diagram of a cluster according to an embodiment of the present application;

[0030] FIG. 3 is a schematic diagram of a cluster and inter-cluster connection mode according to an embodiment of the present application;

[0031] FIG. 4 is a flowchart of a model training method according to an embodiment of the present application;

[0032] FIG. 5 is a flowchart of a model training method according to an embodiment of the present application;

[0033] FIG. 6 is a schematic diagram of a model training method according to an embodiment of the present application;

[0034] FIG. 7 is a schematic diagram of a node workflow according to an embodiment of the present application;

[0035] FIG. 8 is a schematic diagram of a node workflow according to an embodiment of the present application;

[0036] FIG. 9 is a schematic diagram of a node workflow according to an embodiment of the present application;

[0037] FIG. 10 is a flowchart of a node selection method according to an embodiment of the present application;

[0038] FIG. 11 is a schematic diagram of data exchange between a node and system memory according to an embodiment of the present application;

[0039] FIG. 12 is a schematic diagram of a model training method according to an embodiment of the present application;

[0040] FIG. 13 is a structural schematic diagram of a management device according to an embodiment of the present application;

[0041] FIG. 14 is a structural schematic diagram of a computing device according to an embodiment of the present application;

[0042] FIG. 15 is a structural schematic diagram of a computing device cluster according to an embodiment of the present application;

[0043] FIG. 16 is a structural schematic diagram of a computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The solutions provided by the embodiments of the present application will be described below with reference to the drawings. In the embodiments of the present application, "multiple" refers to two or more, and "multiple types" refers to two or more types. "First", "second", and the like are only used to distinguish similar objects and do not necessarily indicate a specific order or number of objects.

[0045] To facilitate understanding of the solutions provided by the embodiments of the present application, the technical terms that may be involved in the embodiments of the present application are first introduced.

[0046] Model: composed of multiple neural network layers. Generally, the multiple neural network layers are arranged in series, and the output of one neural network layer is the input of the next neural network layer. A neural network layer can be simply referred to as a network layer or a layer.

[0047] Model block: a neural network block obtained by inter-layer segmentation of a model. One model block can include one or more neural network layers.

[0048] Pipeline parallelism (PP): a model distributed training technique, which refers to different nodes running different layers of a model, and belongs to inter-layer parallelism. That is, in pipeline parallelism, the model is serialized and segmented in layer granularity to obtain multiple model blocks connected in sequence. During model training, related calculations of different model blocks are performed by different nodes. Among them, one model block can be referred to as one stage in the model training process, and N model blocks are N stages in the model training process.

[0049] Stage: a model block in pipeline parallelism.

[0050] Model training: a process of updating parameters of a model. Generally, the parameters of the model are preset or randomly generated, and the parameters of the model are updated to enable the model to learn the ability to process corresponding tasks, such as image recognition, speech recognition, natural language translation, computer game playing, content generation, and the like. Model training needs to go through multiple iterations until convergence or reach a predetermined number of training times. Each iteration needs to input a certain amount of data set. The model performs forward calculation on the input data set to obtain a loss function value. The model performs backward calculation on the loss function to obtain a gradient value. Then, the gradient value is used to update the model parameters.

[0051] Forward propagation: also referred to as forward propagation, which refers to starting from data input, calculating and passing data through each layer of the model to obtain the output result of the model. The purpose of forward propagation is to calculate the output result of the model.

[0052] Forward computation: refers to the computation of a layer or a model block in a model for forward propagation, which is used to obtain the loss function value of the model.

[0053] Back propagation: also referred to as backward propagation, refers to calculating gradient information in reverse according to the difference between the output result of a model and a label, and passing the calculated gradient information from the last layer (usually the output layer) of the model to each layer of the model to update the parameters of the model. Through back propagation, the gradient information of each parameter on the loss function can be obtained, which is used to update the parameters.

[0054] Backward computation: refers to the computation of a layer or a model block in a model for backward propagation. Backward computation is used to obtain gradient values based on the loss function value obtained by forward computation, and the gradient values are used to update the parameters of the layer or the model block.

[0055] Output of a model block: refers to the calculation result of a model block, specifically the calculation result of a node executing the related computation of the model block. The node executing the related computation of the model block refers to the node performing computation on the calculation object of the model block using the parameters in the model block. The related computation of the model block can be forward computation or backward computation.

[0056] Input of a model block: refers to the calculation object of a model block. The node can perform computation on the input of the model block using the parameters of the model block to obtain the output of the model block, i.e., the calculation result.

[0057] Micro-batch: a fine-grained data set. In pipeline parallelism, if the data set input to a model block each time is large (e.g., mini-batch), the computation of the model block on the data set requires a long time, resulting in low concurrency of pipeline parallelism. In order to improve the concurrency of pipeline parallelism, the large data set (e.g., mini-batch) can be further split to obtain a data set with finer granularity. The data set with finer granularity is referred to as micro-batch.

[0058] Region: a deployment unit of a computing node in an infrastructure, which is a deployment unit divided in terms of geographical location and network latency. The computing nodes in the same region share common services such as elastic computing, block storage, object storage, VPC network, elastic public IP, and image.

[0059] Availability zone (AZ): A computing node deployment unit in the infrastructure, an AZ is a collection of one or more physical data centers with independent wind, fire, water, and electricity. An AZ includes one data center or multiple data centers in close geographical proximity. A region can include multiple AZs.

[0060] Node: Also referred to as a training node or a worker, refers to a computing node used to perform training of a model or a model block. A node can include a processing component and a storage component. The processing component can be a processor or a computing core. The storage component can be a memory, such as an on-chip memory or an on-board memory. Common training nodes include graphics processing units (GPUs) and the like.

[0061] Cluster: Also referred to as a training cluster, a cluster composed of multiple nodes. Training nodes that make up the same cluster are in the same area, such as in the same region or AZ.

[0062] In pipeline parallelism, different model blocks of a model to be trained are deployed to different nodes, so that related computations of different model blocks are performed by different nodes to eliminate the impact of insufficient memory of a single node on model training. Whenever a node completes related computations of a model block for a batch of data (e.g., a micro-batch), if the computation results cannot be sent out in time, the computation results occupy the memory of the node, affecting the computation of the node on the next batch of data. Moreover, if the computation results of a model block cannot be sent out in time, the next model block of the model block cannot obtain data and cannot perform related computations. In short, if the computation results of the computation of a model block performed by a node cannot be sent out from the node in time, the training efficiency of the model is low, and the utilization rate of computing resources is also low.

[0063] Generally, nodes in the same cluster are close in geographical location, and signal attenuation is small or almost nonexistent; and communication resources between nodes in the same cluster are sufficient. Therefore, the communication capability between nodes in the same cluster is strong, and there is usually no case where computation time cannot mask communication time. Wherein, computation time masking communication time refers to computation time being greater than or equal to communication time.

[0064] The geographical locations of nodes in different clusters are far apart, and signal attenuation is large; and communication resources between nodes in different clusters are limited. Therefore, relative to the communication capability between nodes in the same cluster, the communication capability between nodes in different clusters is weak. When data needs to be transmitted between nodes in different clusters, it is easy to have a case where computation time cannot mask communication time. In this case, cross-cluster transmission of data is a bottleneck of model training, and the efficiency of model training and the utilization rate of computing resources of nodes are low.

[0065] Therefore, the embodiment of the present application provides a model training method. In the method, the target model can be divided into multiple model blocks, and at least two model blocks of the multiple model blocks are deployed to nodes of a cluster, and the remaining at least one model block is deployed to nodes of another cluster, wherein different model blocks are deployed to different nodes. The calculation result of a certain model block of the at least two model blocks is the input of the at least one model block, that is, the node where the model block is located needs to send the calculation result of the model block to the node in the other cluster. When the time length of the node where the model block is located for sending the calculation result of the model block to the node in the other cluster is greater than the time length of the node for performing the calculation of the model block, the multiple nodes can be instructed to alternately perform the calculation of the model block, and each node of the multiple nodes can send the calculation result obtained by performing the calculation of the model block to the other cluster through the link between the respective node and the other cluster. That is, the calculation result of the model block is transmitted through multiple links, so that the transmission bandwidth of the calculation result is improved, the communication time consumption is reduced, so that the calculation time consumption can cover the communication time consumption, and the utilization of the computing resources and the training efficiency of the model are improved,

[0066] Next, the model training method provided by the embodiment of the present application is specifically introduced.

[0067] FIG. 1 shows a training system that can be used to implement the method. The training system includes a management device 100, a cluster 200 and a cluster 300. Wherein, the cluster 200 and the cluster 300 can respectively include multiple nodes. For example, the cluster 200 can include nodes 210, 220, 230 and the like. The cluster 300 can include nodes 310 and the like.

[0068] In the following, when the cluster 200 and the cluster 300 are not specially distinguished, they can be simply referred to as clusters.

[0069] In some embodiments, the nodes in the cluster can be GPUs.

[0070] In some embodiments, as shown in FIG. 2, the cluster, for example, the cluster 200, also includes a system memory. For example, the system memory can be one or more memory bars. For example, the cluster 200 also includes a central processing unit (CPU), and the system memory is the memory corresponding to the CPU, and the nodes can store data in the system memory through the CPU.

[0071] In some embodiments, as shown in FIG. 3, different nodes in the cluster 200 are connected through different links and the cluster 300, or in other words, different nodes in the cluster 200 are connected through different links and nodes in the cluster 300. That is, the links between the nodes in the cluster 200 and the cluster 300 are independent and not shared with other nodes.

[0072] In some embodiments, as shown in FIG. 3, there is a link between any two nodes in the cluster 200. Among them, the nodes in the cluster 200 can adopt a full mesh structure, and any two nodes are directly connected.

[0073] In some embodiments, the communication capability of the link between different nodes in the cluster 200 is greater than the communication capability of the link between the nodes in the cluster 200 and the cluster 300. The link between the nodes in the cluster 200 and the cluster 300 is specifically the link between the nodes in the cluster 200 and the nodes in the cluster 300.

[0074] The management device 100 can be any device, software, equipment, platform or cluster with data processing function. In some embodiments, the management device 100 can also be software deployed in one or more nodes in the cluster 200 or the cluster 300.

[0075] The management device 100 is used to deploy model training tasks in the cluster 200 and the cluster 300. The management device 100 can split the target model to be trained into multiple model blocks, then deploy a part of the model blocks to the cluster 200 and another part of the model blocks to the cluster 300, so as to train the target model by using the cluster 200 and the cluster 300, thereby training the target model by multiple clusters in the case that a single cluster cannot carry the target model training.

[0076] In some embodiments, the management apparatus 100 can perform the steps shown in FIG. 4. In step 401, the management apparatus 100 can obtain the source code of the target model. In step 402, the management apparatus 100 can compile the source code of the target model, compile the target model represented in a high-level language into a target model represented in a machine language, and obtain the parameters of each network layer of the target model. In step 403, the management apparatus 100 can obtain the node information of the available cluster. The available cluster refers to a cluster that has idle nodes and can be used to train the target model. The nodes in the cluster that can be used to train the target model can be referred to as available nodes. The node information can include the number of available nodes, the computing power of the available nodes (such as the processor specifications, memory specifications, etc. of the nodes), link information between the nodes, and the like. In step 404, the target model is split based on the node information of the available cluster. For example, if the number of available nodes is 10, the target model can be split into 10 model blocks. In step 405, the model blocks of the target model are deployed to the cluster. One model block is deployed to one node in the cluster. In step 406, the nodes on which the model blocks are deployed are instructed to perform the computation of the model blocks.

[0077] Different model blocks are deployed on different nodes, and the nodes on which the model blocks are located are used to perform the computation of the model blocks, such as forward computation and backward computation. When the output of a model block needs to be transmitted across the cluster, if the time length for the node on which the model block is located to send the computation result of the model block is greater than the time length for the node to perform the computation of the model block, that is, the computation time of the model block cannot mask the communication time of the model block, the management apparatus 100 can instruct more nodes to alternately perform the computation of the model block, so that the computation result of the model block can be sent to other clusters through more nodes. The links between different nodes and other clusters are relevant and independent, and sending the computation result of the model block to other clusters through more nodes can reduce the communication time for sending the computation result of the model block to the cluster, avoid the limitation of data transmission across the cluster on the model training efficiency, and improve the training efficiency of the model.

[0078] The above examples introduce the training system provided by the embodiments of the present application. Next, taking implementation in the training system as an example, the model training method provided by the embodiments of the present application is introduced. The method is specifically performed by the management apparatus 100. As shown in FIG. 5, the method includes the following steps.

[0079] In step 501a, the management apparatus 100 can deploy the A1 part of the target model to the cluster 200. In step 501b, the management apparatus 100 can deploy the A2 part of the target model to the cluster 300. The A1 part is a part of the target model, and the A2 part is another part of the target model.

[0080] The management apparatus 100 splits the target model into a plurality of model blocks, i.e., model block 1, model block 2, …, model block i-2, model block i-1, model block i, model block i+1, etc. Each of the plurality of model blocks includes one or more network layers in the target model. As shown in FIG. 6, the plurality of model blocks are sequentially adjacent in a data transmission direction of the target model, i.e., model block 1, model block 2, …, model block i-2, model block i-1, model block i, model block i+1, etc. are sequentially adjacent. The output of a model block is the input of a model block that is adjacent to the model block and after the model block, e.g., the output of model block i-2 is the input of model block i-1, the output of model block i-1 is the input of model block i, and the output of model block i is the input of model block i+1. The data transmission direction can be a forward propagation direction or a backward propagation direction. When the data transmission direction is the forward propagation direction, the output of a model block is specifically the calculation result of forward calculation of the model block. When the data transmission direction is the backward propagation direction, the output of a model block is specifically the calculation result of backward calculation of the model block.

[0081] In some embodiments, the management apparatus 100 can split the target model into a plurality of model blocks based on an equal split strategy to equally split the network layers of the target model into the plurality of model blocks. As shown in FIG. 6, different model blocks in the plurality of model blocks obtained based on the equal split strategy include the same number of network layers, or the difference between the numbers is less than a set threshold.

[0082] The A1 part can include at least two model blocks in the plurality of model blocks, e.g., model block 1, model block 2, …, model block i-2, model block i-1, model block i. The A2 part includes at least one model block in the plurality of model blocks, e.g., model block i+1, etc. The at least one model block can be a model block in the plurality of model blocks other than the at least two model blocks.

[0083] Specifically, the management apparatus 100 deploys the model blocks included in the A1 part to at least two nodes in the cluster 200, and deploys the model blocks included in the A2 part to at least one node in the cluster 300. One model block is deployed to one node, e.g., model block i is deployed to node 210, model block i-1 is deployed to node 220, model block i-2 is deployed to node 230, and model block i+1 is deployed to node 310.

[0084] The deployment of a model block to a node can mean that the parameters of the model block are stored in a storage space accessible to the node, and the node is instructed to load the parameters of the model block from the storage space to perform the calculation of the model block. That is, the node where the model block is deployed is used to perform the calculation of the model block. The calculation of the model block includes forward calculation and backward calculation.

[0085] For convenience of description, the node on which the model block is deployed can be referred to as the node corresponding to the model block.

[0086] After the deployment of the model block is completed, the management apparatus 100 performs step 502, instructing the node corresponding to the model block to perform the computation B1 of the model block. For example, the node (i.e., the node 210) corresponding to the model block i is instructed to perform the computation B1 of the model block. Wherein, the node corresponding to the model block is configured to send the computation result C1 obtained by the node performing the computation B1 of the model block to the node corresponding to the next model block of the model block. For example, the node 210 sends the computation result C1 obtained by the node 210 performing the computation B1 of the model block i to the node (i.e., the node 310) corresponding to the model block i+1.

[0087] The computation B1 is a forward computation or a backward computation. The next model block of the model block refers to the model block in the data transmission direction of the model block. The data transmission direction is the transmission mode corresponding to the computation B1. When the computation B1 is a forward computation, the data transmission direction is forward propagation. When the computation B2 is a backward computation, the data transmission direction is backward propagation.

[0088] Wherein, the model block i+1 deployed in the cluster 300 is the next model block of the model block i deployed in the cluster 200. The node (i.e., the node 210) corresponding to the model block i needs to send the computation result C1 obtained by the node 210 performing the computation B1 of the model block i to the cluster 300. The node 210 specifically sends the computation result C1 to the cluster 300 through the link between the node 210 and the cluster 300. Wherein, the link is specifically the link between the node 210 and the node 310, and the computation result C1 is specifically sent to the node 310 of the cluster 300.

[0089] Step 503, confirming that the time length of the node 210 sending the computation result C1 to the cluster 300 is greater than the time length of the node 210 performing the computation B1 of the model block i.

[0090] In some embodiments, as mentioned above, the data set is input to the target model in batches, for example, the data set is input to the target model in the granularity of micro-batch. Therefore, the model block processes the output of the previous model block in batches, and the previous model block also outputs in batches. For model block i, the node (node 220) corresponding to the previous model block (i.e., model block i-1) outputs the calculation result of the calculation B1 of model block i-1 to the node (i.e., node 210) corresponding to model block i in batches. Node 210 receives the calculation result in batches, and accordingly, node 210 performs the calculation B1 of model block i on the calculation result of the calculation B1 of model block i-1 in batches. When the calculation B1 of model block i performed by node 210 on the next batch of calculation results is completed, and the calculation result of the calculation B1 of model block i performed by node 210 on the current batch of calculation results has not been completely sent, it is confirmed that the time length for node 210 to send the calculation result C1 to cluster 300 is greater than the time length for node 210 to perform the calculation B1 of model block i. Wherein, the current batch of calculation results is the result obtained by node 220 performing the calculation B1 of model block i-1, and the next batch of calculation results is the result obtained by node 220 performing the calculation B1 of model block i-1 immediately after outputting the current batch of results.

[0091] In some embodiments, management device 100 can count the calculation time length required for node 210 to perform the calculation B1 of model block i on a certain data set. Wherein, node 210 performs the calculation B1 of model block i on a certain data set to obtain the calculation result C1. Management device 100 can also count the communication time length required for node 210 to send the calculation result C1 to cluster 300. If the communication time length is greater than the calculation time length, it is confirmed that the time length for node 210 to send the calculation result C1 to cluster 300 is greater than the time length for node 210 to perform the calculation B1 of model block i.

[0092] If the time length for node 210 to send the calculation result C1 to cluster 300 is greater than the time length for node 210 to perform the calculation B1 of model block i, the calculation result of the calculation B1 of model block i performed by node 210 cannot be sent in time, the calculation result consumes the memory of node 210, affects the calculation efficiency of node 210, and the calculation result cannot be transmitted to the node corresponding to model block i+1 in time. These will all reduce the training efficiency.

[0093] Figure 7 shows the case where the time length for node 210 to send the calculation result C1 to cluster 300 is greater than the time length for node 210 to perform the calculation B1 of model block i, wherein D1, D2, and D3 represent different data sets, C D1 , C D2 , C D3The respective nodes 210 respectively perform the computation B1 of the model i on the data sets D1, D2, D3 respectively. The node 210 can be configured to perform the computation B1 of the model i for each batch of data sets for a unit time duration, and the node 210 can be configured to send the computation result C1 of the computation B1 of the model i for the data sets to the cluster 300 for a time duration of two unit time durations. When the computation B1 of the model i performed by the node 210 on the data set D1 is completed, the node 210 starts to send the computation result C D1 to the cluster 300, and starts to perform the computation B1 of the model i on the data set D2. When the computation B1 of the model i performed by the node 210 on the data set D2 is completed, the sending of the computation result C D1 is not completed, and thus the computation result C D2 cannot be sent out, and the memory of the node 210 is consumed. In addition, the node 310 can be configured to perform the computation B1 of the model i+1 for each batch of data sets for a unit time duration, and the node 310 can be configured to receive the data sets (i.e. the computation result C1) every two unit time durations, and thus the node 310 has no data set to process in one unit time duration of every two unit time durations, and thus the utilization rate of the computing resources of the node 310 is low.

[0094] In some embodiments, at step 503, it is confirmed that the communication time duration between the node 210 and the cluster 300 is greater than the computation time duration of the model i. The computation time duration of the model i is the sum of the time duration of the node 210 performing the computation B1 of the model i and the time duration of the node 210 performing the computation B2 of the model i. The computation B1 is one of the forward computation and the backward computation, and the computation B2 is the other of the forward computation and the backward computation. That is, if the computation B1 is the forward computation, the computation B2 is the backward computation. If the computation B1 is the backward computation, the computation B2 is the forward computation. The communication time duration between the node 210 and the cluster 300 is the sum of the time duration of the node 210 sending the computation result C1 to the cluster 300 and the time duration of the node 210 receiving the computation result of the computation B2 of the A2 part from the cluster 300. The computation result of the computation B2 of the A2 part is specifically the computation result of the node 310 performing the computation B2 of the model i+1.

[0095] In the case that the time duration of the node 210 sending the computation result C1 to the cluster 300 is greater than the time duration of the node 210 performing the computation B1 of the model i, the management device 100 can perform step 504 to instruct the k nodes in the cluster 200 to alternately perform the computation B1 of the model i. Each of the k nodes can send the computation result of the computation B1 of the model i performed by the node to the cluster 300 through the link between the node and the cluster 300.

[0096] where k is an integer greater than 1. In some embodiments, k is 2.

[0097] Each of the k nodes performs the calculation B1 of the model i, and obtains the calculation result corresponding to the node. The node sends the calculation result corresponding to the node to the cluster 300 through the link between the node and the cluster 300. In this way, the calculation result of the calculation B1 of the model i is sent to the cluster 300 through the k links between the k nodes and the cluster 300. As described above, the links between the different nodes in the cluster 200 and the cluster 300 are independent of each other, so the k links between the k nodes and the cluster 300 are k independent links that can be transmitted in parallel. By sending the calculation result of the calculation B1 of the model i to the cluster 300 through the k links, the transmission time can be significantly shortened, so that the node can send the calculation result in time after performing the calculation of each batch of data sets, avoiding the consumption of the calculation result on the node memory, and ensuring that the calculation result can be sent to the cluster 300 in time, so that the nodes in the cluster 300 can obtain the data in time and perform the calculation, improving the utilization rate of the computing resources of the nodes in the cluster 300.

[0098] In some embodiments, the k nodes include the node 210. The k nodes perform the calculation of the model i, and the model i needs to be deployed in each of the k nodes. If the k nodes include the node 210, the model i does not need to be deployed in the node 210 repeatedly. In this way, the workload of deploying the model i can be reduced.

[0099] In some embodiments, the k nodes can be set as the node 210 and the node 220. Referring to FIG. 8, when the node 210 completes the calculation B1 of the model i for the data set D1, the node 210 can immediately send the obtained calculation result C D1 to the cluster 300 through the link between the node 210 and the cluster 300. Next, the node 200 performs the calculation B1 of the model i for the data set D2, and when the calculation is completed, the node 220 can immediately send the obtained calculation result C D2 to the cluster 300 through the link between the node 220 and the cluster 300. In this way, the node 210 and the node 220 alternately perform the calculation B1 of the model i and send the calculation result, avoiding the consumption of the calculation result on the node memory, and ensuring that the node 310 obtains the data set in time, thereby ensuring the training efficiency of the model and the utilization rate of the computing resources of the node.

[0100] In some embodiments, the A1 part at least includes k model blocks of the target model, which are model block i-k+1, model block i-k+2, …, and model block i, respectively. The k model blocks are sequentially adjacent in the data transmission direction corresponding to the calculation of B1, and model block i is the last model block in the k model blocks in the data transmission direction corresponding to the calculation of B1. That is, the output of model block i-k+1 is the input of model block i-k+2, …, and the output of model block i-1 is the input of model block i.

[0101] In step 504, the k model blocks can be combined to obtain a sub-model, and the k nodes are instructed to alternately perform the calculation of B1 of the sub-model. In this way, the utilization rate of the computing resources of the k nodes can be improved.

[0102] It can be understood that if the k nodes alternately perform the calculation of model block i, the amount of data that each node needs to process is k times less than the amount of data that node 210 needs to process when performing the calculation of model block i alone. In this way, the computing resources of the nodes cannot be fully utilized. For example, as shown in FIG. 8, there is an idle time (one unit of time in every two units of time is an idle time) for each node. In the present embodiment, the k nodes perform the calculation of the sub-model composed of the k model blocks, so that the computing performance of the nodes can be fully utilized, and the training efficiency is not affected.

[0103] In addition, the data sent by the nodes in cluster 200 to cluster 300 is the data relied on by the model blocks in cluster 300, that is, the input of model block i+1, which is also the output of model block i. Whether the nodes perform the calculation of B1 of the sub-model or only perform the calculation of B1 of model block i, the data sent by the nodes to cluster 300 is the calculation result of the calculation of B1 of model block i. That is, the data transmitted between the nodes and the nodes or the cluster is irrelevant to the granularity of the model blocks performed by the nodes.

[0104] In one example, k can be set to 2, as shown in FIG. 9. The k nodes perform the calculation of the sub-model for two unit times, and the nodes send the calculation results to cluster 300 for two unit times. In this way, the timely sending of the calculation results can be ensured, and the computing resources of the nodes can be fully utilized.

[0105] In one example of the present embodiment, the k model blocks and the k nodes are in one-to-one correspondence. That is, in step 501a, the management apparatus 100 deploys the k model blocks to the k nodes, wherein one node deploys one model block.

[0106] In another example of this embodiment, the management apparatus 100 can determine whether the time length for any node (e.g., the node 220) in the k nodes to send the calculation result C2 to the cluster 300 is greater than the time length for the node to perform the calculation Bl of the sub-model. The calculation result C2 is the calculation result of the node performing the calculation Bl of the sub-model. The determination manner can refer to the description of step 503 above, and will not be repeated here.

[0107] If the time length for the node 220 to send the calculation result C2 of the node 200 performing the calculation Bl of the sub-model to the cluster 200 is greater than the time length for the node 220 to perform the calculation Bl of the sub-model, the management apparatus 100 instructs the k+m nodes in the cluster 200 to alternately perform the calculation Bl of the sub-model. Each node in the k+m nodes sends the calculation result of the node performing the calculation Bl of the sub-model to the cluster 300 (specifically, the node 310) through the link between the node and the cluster 300 (specifically, the node 310). The m is an integer greater than or equal to 1. In this way, more nodes can be used to send the calculation result of the calculation Bl of the sub-model to the cluster 300 (or the node 310).

[0108] The calculation result of the calculation Bl of the sub-model is the calculation result of the calculation Bl of the model i.

[0109] In addition, a node is originally used to perform a model block, that is, the memory of a node is sufficient for the parameters of a model block. If a node performs a sub-model composed of k model blocks, the memory of the node may not be sufficient for the parameters of the sub-model, thereby affecting the node to perform the calculation of the sub-model. For this case, the embodiments of the present application provide the following solutions.

[0110] One solution is to select the node with the largest memory in the cluster 200 to perform the calculation of the sub-model, that is, the k nodes are the nodes with the largest memory in the cluster 200. The specific operation can be as shown in FIG. 10. In step 1001, the nodes in the cluster 200 are sorted in descending order of memory. In step 1002, the k nodes with the largest memory are selected from the sorting result. In step 1003a, the sub-model is deployed to the k nodes. In step 1003b, the original model blocks in the k nodes are deployed to other nodes in the cluster 200,

[0111] Another solution is to perform the calculation of the sub-model using the system memory of the cluster 200. That is, the parameters of the sub-model are stored in the system memory of the cluster 200. When performing the calculation of the sub-model, the node can obtain the parameters of at least one model block in the sub-model from the system memory, and perform the calculation using the obtained parameters. After completing the calculation of the one model block, the parameters of the at least one model block are stored in the system memory, and the parameters of another at least one model block are obtained from the system memory and calculated. By analogy, until the calculation of the sub-model is completed. In one example, as shown in FIG. 11, a data flow planning unit can be provided to support data exchange between the node and the system memory.

[0112] In summary, the model training method provided by the embodiments of the present application can use multiple nodes to alternately perform the calculation of the model block and send the calculation result when the time length of the node sending the calculation result of the model block is greater than the time length of the node performing the calculation of the model block, thereby avoiding the consumption of the calculation result to the node memory, and enabling the downstream node to obtain the data set in time, thereby ensuring the training efficiency of the model and the utilization rate of the computing resources of the node.

[0113] Next, a possible implementation manner of the model training method provided by the embodiments of the present application is introduced. As shown in FIG. 12, the method includes the following steps.

[0114] Step 1201, splitting the target model. The target model can be split into multiple model blocks, at least two of which are deployed to the cluster 200, and the remaining model blocks are deployed to the cluster 300, and one model block is deployed to one node.

[0115] Step 1202, identifying the model blocks that need to be communicated across clusters. The model blocks that need to be communicated across clusters refer to two adjacent model blocks in the data transmission direction of the target model, which are deployed to different clusters. The model blocks that need to be communicated across clusters can be set as model i and model i+1.

[0116] Step 1203, obtaining the forward communication time length Fc1 and the backward communication time length Bc1 of the model i. Wherein, in the forward propagation direction, the model i+1 is after the model i, then the forward communication time length Fc1 is the time length for the node corresponding to the model i to send the calculation result of the forward calculation of the model i to the node corresponding to the model i+1, and the backward communication time length Bc1 is the time length for the node corresponding to the model i to receive the calculation result of the backward calculation of the model i+1 from the node corresponding to the model i+1.

[0117] Step 1203, setting k=2, that is, the initial value of k is 2.

[0118] Step 1205, merging the k model blocks into a sub-model. Wherein, the k model blocks are sequentially adjacent in the above data transmission direction, and in the above data transmission direction, the model block i is the last model block in the k model blocks.

[0119] Step 1206, obtaining the forward calculation time length Fc2 and the backward calculation time length Bc2 of the sub-model.

[0120] In one example, the sum of the forward calculation time lengths of the k model blocks can be taken as the forward calculation time length Fc2 of the sub-model, and the sum of the backward calculation time lengths of the k model blocks can be taken as the backward calculation time length Bc2 of the sub-model.

[0121] In one example, the k nodes alternately perform the forward calculation and the backward calculation of the sub-model, wherein the time length of a node performing the forward calculation of the sub-model can be taken as the forward calculation time length Fc2 of the sub-model, and the time length of the node performing the backward calculation of the sub-model can be taken as the backward calculation time length Bc2 of the sub-model.

[0122] Step 1207, judging whether the sum of Fc1 and Bc1 is greater than the sum of Fc2 and Bc2.

[0123] If the sum of Fc1 and Bc1 is greater than the sum of Fc2 and Bc2, the value of k can be updated, wherein k is added by 1 to obtain the updated k. Then, step 1205 and the subsequent steps of step 1205 are executed again.

[0124] If the sum of Fc1 and Bc1 is less than or equal to the sum of Fc2 and Bc2, the sub-model does not need to be adjusted. That is, the k nodes alternately perform the calculation of the sub-model until the target model training is completed.

[0125] In some embodiments, then in step 1208, the k nodes with the largest memory in the cluster 200 are obtained. Then, in step 1209, the sub-model is deployed to the k nodes with the largest memory. In step 1210, the k nodes with the largest memory alternately perform the calculation of the sub-model. Wherein, the k nodes take turns to receive the calculation results of the model block i-k and perform the calculation of the sub-model. In step 1211, each node of the k nodes sends the calculation results to the node 310.

[0126] In this way, when the time length of the node sending the calculation results of the model block is greater than the time length of the node performing the calculation of the model block, the calculation of the model block and the sending of the calculation results can be alternately performed by multiple nodes, avoiding the consumption of the calculation results to the node memory, and the downstream nodes can obtain the data set in time, thereby guaranteeing the training efficiency of the model and the utilization rate of the calculation resources of the nodes.

[0127] Based on the introduction of the method embodiments above, the embodiments of the present application further provide a management device 1300. The training system where the management device 1300 is located further includes a first cluster and a second cluster, wherein the first cluster includes a plurality of nodes, and different nodes in the plurality of nodes are connected with the second cluster through different links. As shown in FIG. 13, the management device 1300 includes:

[0128] a deployment module 1310, configured to deploy a first part of a target model into the first cluster and a second part into the second cluster; the first part includes a first model block of the target model, and a calculation result of a first calculation of the first model block is an input of the second part;

[0129] a first indication module 1320, configured to instruct a first node in the first cluster to perform the first calculation of the first model block to obtain a first result; and the first node is configured to send the first result to the second cluster through a link between the first node and the second cluster;

[0130] a second indication module 1330, configured to instruct k nodes in the first cluster to alternately perform the first calculation of the first model block in a case that a time length during which the first node sends the first result to the second cluster is greater than a time length during which the first node performs the first calculation of the first model block; each node in the k nodes is configured to send a calculation result of the first calculation of the first model block performed by the node to the second cluster through a link between the node and the second cluster; and k is an integer greater than 1.

[0131] In some embodiments, the first part includes k model blocks of the target model, the k model blocks are sequentially adjacent in a data transmission direction of the target model, and in the data transmission direction, the first model block is a last model block in the k model blocks; and the second indication module 1330 is configured to: combine the k model blocks to obtain a sub-model; and instruct the k nodes to alternately perform a first calculation of the sub-model.

[0132] In one example of this embodiment, the second indication module 1330 is further configured to: instruct k+m nodes in the first cluster to alternately perform the first calculation of the sub-model in a case that a time length during which a second node in the k nodes sends a second result to the second cluster is greater than a time length during which the second node performs the first calculation of the sub-model; each node in the k+m nodes is configured to send a calculation result of the first calculation of the sub-model performed by the node to the second cluster through a link between the node and the second cluster; the second result is the calculation result of the first calculation of the sub-model performed by the second node; and m is an integer greater than or equal to 1.

[0133] In another example of this embodiment, the k nodes are the largest memory nodes in the first cluster.

[0134] In yet another example of this embodiment, the first cluster includes system memory for each of the k nodes to perform the first computation of the first model block.

[0135] In some embodiments, the duration of the first node sending the first result to the second cluster is greater than the duration of the first node performing the first computation of the first model block, including: the duration of communication between the first node and the second cluster is greater than the duration of the computation of the first model block; wherein the duration of the computation is the sum of the duration of the first node performing the first computation of the first model block and the duration of the first node performing the second computation of the first model block; the duration of the communication is the sum of the duration of the first node sending the first result to the second cluster and the duration of the first node receiving the second part of the second computation result from the second cluster; the first computation is one of the forward computation and the backward computation, and the second computation is the other of the forward computation and the backward computation.

[0136] In some embodiments, the communication capability of the link between the nodes in the first cluster and the second cluster is less than the communication capability of the link between different nodes in the first cluster.

[0137] Wherein, the deployment module 1310, the first indication module 1320 and the second indication module 1330 can be implemented by software or can be implemented by hardware. For example, the implementation of the deployment module 1310 is introduced as follows. Similarly, the implementation of the first indication module 1320 and the second indication module 1330 can refer to the implementation of the deployment module 1310.

[0138] As an example of a software functional unit, the deployment module 1310 can include code running on a compute instance. The compute instance can include at least one of a physical host (computing device), a virtual machine, a container. Further, the compute instance can be one or more. For example, the deployment module 1310 can include code running on multiple hosts / virtual machines / containers. It is noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Typically, a region can include multiple AZs.

[0139] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same VPC, or in multiple VPCs. Typically, a VPC is set up within a region, and communication between two VPCs in the same region, or between VPCs in different regions, requires a communication gateway in each VPC to enable interconnection between VPCs.

[0140] As an example of a hardware functional unit, the deployment module 1310 can include at least one computing device, such as a server, etc. Alternatively, the deployment module 1310 can be a device implemented using an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), such as a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0141] The multiple computing devices included in the deployment module 1310 can be distributed in the same region or in different regions. The multiple computing devices included in the deployment module 1310 can be distributed in the same AZ or in different AZs. Likewise, the multiple computing devices included in the deployment module 1310 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0142] It should be noted that, in other embodiments, the deployment module 1310 can be configured to perform any of the steps in the method shown in FIG. 5 or any of the operations performed by the management apparatus 100, the first indication module 1320 can be configured to perform any of the steps in the method shown in FIG. 5 or any of the operations performed by the management apparatus 100, and the second indication module 1330 can be configured to perform any of the steps in the method shown in FIG. 5 or any of the operations performed by the management apparatus 100. The steps or operations implemented by the deployment module 1310, the first indication module 1320, and the second indication module 1330 can be specified as needed, and the entire function of the management apparatus 1300 can be implemented by the deployment module 1310, the first indication module 1320, and the second indication module 1330 implementing different steps in the method shown in FIG. 5 or performing different operations by the management apparatus 100, respectively.

[0143] The present application also provides a computing device 1400. As shown in FIG. 14, the computing device 1400 includes a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate with each other through the bus 1402. The computing device 1400 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 1400 is not limited by the present application.

[0144] The bus 1402 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 14, but it does not mean that there is only one bus or only one type of bus. The bus 1402 can include a path for transmitting information between various components (e.g., the memory 1406, the processor 1404, the communication interface 1408) of the computing device 1400.

[0145] The processor 1404 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), among other processors.

[0146] The memory 1406 can include volatile memory, such as random access memory (RAM), and non-volatile memory, such as read-only memory (ROM), floppy disks, mechanical hard disks, or solid state hard disks, among others.

[0147] The memory 1406 stores executable program code that the processor 1404 executes to implement the functionality of the aforementioned deployment module 1310, the first indication module 1320, and the second indication module 1330, respectively, to implement the operations of the method of FIG. 5 or the operations performed by the management apparatus 100. That is, the memory 1406 stores instructions for performing the method of FIG. 5 or the operations performed by the management apparatus 100.

[0148] The communication interface 1408 enables communication among the computing device 1400, other devices, or communication networks using, for example and without limitation, a network interface card, a transceiver, or a set of transceivers, among other components.

[0149] Embodiments of the present disclosure also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a notebook computer, or a terminal device such as a smartphone.

[0150] As shown in FIG. 15, the computing device cluster includes at least one computing device 1400. The memory 1406 in one or more of the computing devices 1400 in the computing device cluster can store the same instructions for performing the method of FIG. 5 or the operations performed by the management apparatus 100.

[0151] In some possible implementation, the memory 1406 of one or more of the computing devices 1400 in the cluster of computing devices can also respectively store instructions for performing part of the instructions of the method shown in FIG. 5 or part of the operations performed by the management apparatus 100. In other words, the combination of one or more computing devices 1400 can collectively perform the instructions for performing the method shown in FIG. 5 or the operations performed by the management apparatus 100.

[0152] It should be noted that the memories 1406 in different computing devices 1400 in the cluster of computing devices can store different instructions respectively for performing part of the functions of the management apparatus 1300. That is, the memories 1406 in different computing devices 1400 can store instructions for implementing the functions of one or more of the deployment module 1310, the first indication module 1320 and the second indication module 1330.

[0153] In some possible implementation, one or more of the computing devices in the cluster of computing devices can be connected through a network. The network can be a wide area network, a local area network or the like. FIG. 16 shows one possible implementation. As shown in FIG. 16, two computing devices 1400A and 1400B are connected through a network. Specifically, the computing devices are connected to the network through the communication interfaces in the computing devices. In this type of possible implementation, the memory 1406 in the computing device 1400A stores instructions for performing the functions of the deployment module 1310. Meanwhile, the memory 1406 in the computing device 1400B stores instructions for performing the functions of the first indication module 1320 and the second indication module 1330.

[0154] It should be understood that the functions of the computing device 1400A shown in FIG. 16 can also be completed by multiple computing devices 1400. Similarly, the functions of the computing device 1400B can also be completed by multiple computing devices 1400.

[0155] Embodiments of the present application also provide another cluster of computing devices. The connection relationship between the computing devices in the cluster of computing devices can be similar to the connection manners of the cluster of computing devices described with reference to FIG. 15 and FIG. 16. The difference is that the memories 1406 in one or more of the computing devices 1400 in the cluster of computing devices can store the same instructions for performing the method shown in FIG. 5 or the operations performed by the management apparatus 100.

[0156] In some possible implementation manners, the memory 1406 of one or more of the computing devices 1400 in the computing device cluster can also respectively store instructions for performing part of the method shown in FIG. 5 or part of the operations performed by the management apparatus 100. In other words, the combination of one or more computing devices 1400 can collectively perform the instructions for performing the method shown in FIG. 5 or the operations performed by the management apparatus 100.

[0157] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to perform the method shown in FIG. 5 or the operations performed by the management apparatus 100.

[0158] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a host migration device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to perform the method shown in FIG. 5 or the operations performed by the management apparatus 100.

[0159] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that, A management device applied to a training system, the training system further comprising a first cluster and a second cluster, wherein the first cluster comprises a plurality of nodes, different nodes in the plurality of nodes are connected with the second cluster through different links; the method comprises: deploying a first part of a target model into the first cluster and a second part into the second cluster; the first part comprises a first model block of the target model, and a calculation result of a first calculation of the first model block is an input of the second part; instructing a first node in the first cluster to perform the first calculation of the first model block to obtain a first result; the first node is configured to send the first result to the second cluster through a link between the first node and the second cluster; in a case where a time length of the first node sending the first result to the second cluster is greater than a time length of the first node performing the first calculation of the first model block, instructing k nodes in the first cluster to alternately perform the first calculation of the first model block; wherein each node in the k nodes is configured to send a calculation result of the first node performing the first calculation of the first model block to the second cluster through a link between the node and the second cluster; k is an integer greater than 1.

2. The method of claim 1, wherein, the first part comprises k model blocks of the target model, the k model blocks are sequentially adjacent in a data transmission direction of the target model, and in the data transmission direction, the first model block is a last model block in the k model blocks; the instructing the k nodes in the first cluster to alternately perform the first calculation of the first model block comprises: merging the k model blocks to obtain a sub-model; instructing the k nodes to alternately perform a first calculation of the sub-model.

3. The method of claim 2, wherein, The method further comprises: in a case where a time length of a second node in the k nodes sending a second result to the second cluster is greater than a time length of the second node performing the first calculation of the sub-model, instructing k+m nodes in the first cluster to alternately perform the first calculation of the sub-model; wherein each node in the k+m nodes is configured to send a calculation result of the node performing the first calculation of the sub-model to the second cluster through a link between the node and the second cluster; wherein the second result is the calculation result of the second node performing the first calculation of the sub-model; m is an integer greater than or equal to 1.

4. The method of claim 2 or 3, wherein: the k nodes are nodes with the largest memory in the first cluster; or, the first cluster comprises a system memory, and the system memory is used for each node in the k nodes to perform the first calculation of the first model block.

5. The method according to any one of claims 1-4, characterized in that, the time length of the first node sending the first result to the second cluster is greater than the time length of the first node performing the first calculation of the first model block, comprising that a communication time length between the first node and the second cluster is greater than a calculation time length of the first model block; The calculation duration is a sum of a duration of the first calculation of the first model block performed by the first node and a duration of the second calculation of the first model block performed by the first node. The communication duration is a sum of a duration of the first node sending the first result to the second cluster and a duration of the first node receiving a calculation result of the second calculation of the second part from the second cluster. The first calculation is one of forward calculation and backward calculation, and the second calculation is the other of forward calculation and backward calculation.

6. The method according to any one of claims 1-5, characterized in that, A communication capability of a link between a node in the first cluster and the second cluster is less than a communication capability of a link between different nodes in the first cluster.

7. A management device characterized by comprising: The training system where the management device is located further includes a first cluster and a second cluster, wherein the first cluster includes a plurality of nodes, different nodes in the plurality of nodes are connected to the second cluster through different links; and the management device includes: A deployment module configured to deploy a first part of a target model to the first cluster and a second part to the second cluster; the first part includes a first model block of the target model, and a calculation result of a first calculation of the first model block is an input of the second part; A first indication module configured to instruct a first node in the first cluster to perform the first calculation of the first model block to obtain a first result; and the first node is configured to send the first result to the second cluster through a link between the first node and the second cluster. A second indication module configured to instruct k nodes in the first cluster to alternately perform the first calculation of the first model block in a case where a duration of the first node sending the first result to the second cluster is greater than a duration of the first node performing the first calculation of the first model block; each node in the k nodes is configured to send a calculation result of the first node performing the first calculation of the first model block to the second cluster through a link between the node and the second cluster; and k is an integer greater than 1.

8. The apparatus of claim 7, wherein, The first part includes k model blocks of the target model, the k model blocks are sequentially adjacent in a data transmission direction of the target model, and the first model block is a last model block in the k model blocks in the data transmission direction; The second indication module is configured to: merge the k model blocks to obtain a sub-model; and instruct the k nodes to alternately perform a first calculation of the sub-model.

9. The apparatus of claim 8, wherein, The second indication module is further configured to: instruct k+m nodes in the first cluster to alternately perform the first calculation of the sub-model in a case where a duration of a second node in the k nodes sending a second result to the second cluster is greater than a duration of the second node performing the first calculation of the sub-model; and each node in the k+m nodes is configured to send a calculation result of the node performing the first calculation of the sub-model to the second cluster through a link between the node and the second cluster. The second result is a calculation result of the first calculation of the sub-model performed by the second node; and m is an integer greater than or equal to 1.

10. The apparatus of claim 8 or 9, wherein, the k nodes are nodes with the largest memory in the first cluster; or, the first cluster includes system memory used for each of the k nodes to perform the first calculation of the first model block.

11. The apparatus of any one of claims 7-10, wherein, The first node sends the first result to the second cluster for a time duration greater than a time duration for the first node to perform the first calculation of the first model block, including a communication time duration between the first node and the second cluster being greater than a calculation time duration of the first model block. The calculation time duration is a sum of the time duration for the first node to perform the first calculation of the first model block and a time duration for the first node to perform the second calculation of the first model block. The communication time duration is a sum of a time duration for the first node to send the first result to the second cluster and a time duration for the first node to receive the calculation result of the second calculation of the second part from the second cluster. The first calculation is one of a forward calculation and a backward calculation, and the second calculation is the other of the forward calculation and the backward calculation.

12. The apparatus of any one of claims 7-11, wherein, A communication capability of a link between a node in the first cluster and the second cluster is less than a communication capability of a link between different nodes in the first cluster.

13. A cluster of computing devices, characterized in that, The at least one computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The at least one computing device includes a processor and a memory; 15. A computer program product comprising instructions, characterized in that, The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1-6. The at least one computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1-6.

Images