Model training method and apparatus

Abstract

The present application provides a model training method and apparatus. The method comprises: dividing a training set of a target model into a plurality of training subsets, the plurality of training subsets corresponding one-to-one to a plurality of nodes; sending each training subset of the plurality of training subsets to a node corresponding to the training subset, so that the node corresponding to the training subset executes calculation of the target model for the training subset so as to obtain a calculation result of the node corresponding to the training subset; when a first node in a first cluster is used for acquiring calculation results of several nodes and sending the acquired calculation results to at least one node, instructing the first node to send to the at least one node 1 / N of recently acquired calculation results of the several nodes each time 1 / N of the calculation results of the several nodes are acquired, N being an integer greater than 1. The method can reduce the time consumed for sharing calculation results between nodes in different clusters, thereby improving model training efficiency.

Description

Model training method and device

[0001] The present application claims priority to the Chinese patent application No. 202411106032.X, filed on August 13, 2024, and entitled "A model training method and device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI), and in particular to a model training method and device. BACKGROUND

[0003] At present, large language models (LLMs) and other large-scale models have emerged. The parameter quantity of a large-scale model has reached tens of billions, or even hundreds of billions. Such a huge amount of parameters makes it difficult to train a large-scale model through a single device. The industry uses distributed training technology to train a large-scale model. Due to the limitations of power, cooling capacity, and networking capability, a single cluster often cannot meet the training needs of a large-scale model. Therefore, two or more clusters are needed to train the same large-scale model.

[0004] A common distributed training method is data parallelism (DP). In data parallelism, multiple training nodes respectively perform model calculation on different data sets to improve the calculation speed of the model. Since each training node calculates a part of the model training set, the gradient calculated by each training node is also a part of the gradient of the model. The update of the model parameters requires all the gradients, which requires sharing the gradients between different nodes.

[0005] The time-consuming of sharing the gradients directly affects the training efficiency of the model. Generally, the communication ability between training nodes in the same cluster is strong, and the gradient sharing can be quickly completed. The communication between training nodes in different clusters is weak, and the time-consuming of gradient sharing is long, which affects the efficiency of model training. SUMMARY

[0006] The present application provides a model training method and device, which can reduce the time-consuming of sharing the calculation results (such as the gradient) between nodes in different clusters, thereby improving the training efficiency of the model.

[0007] In a first aspect, a model training method is provided, which is applied to a management device in a training system. The training system further includes a plurality of nodes, each of which is deployed with a target model. A plurality of nodes are located in a first cluster, and at least one node other than the plurality of nodes is located in a second cluster. The method includes: dividing a training set of the target model into a plurality of training subsets; wherein the plurality of training subsets and the plurality of nodes correspond one by one; sending each training subset in the plurality of training subsets to the node corresponding to the training subset, so that the node corresponding to the training subset performs calculation of the target model for the training subset to obtain a calculation result of the node corresponding to the training subset; in the case that a first node in the first cluster is used to obtain the calculation results of the plurality of nodes and send the obtained calculation results to the at least one node, the first node is instructed to send the most recently obtained one of the N parts of the calculation results of the plurality of nodes to the at least one node each time when the one of the N parts of the calculation results of the plurality of nodes is obtained; N is an integer greater than 1; wherein the at least one node is used to update the parameters of the target model deployed in the at least one node by using the calculation results of the plurality of nodes and the calculation result of the at least one node.

[0008] The calculation result can be a gradient. For example, the first node is used to obtain the one of the N parts of the calculation results of the plurality of nodes through a first network, and the first node is used to send the most recently obtained one of the N parts of the calculation results of the plurality of nodes to the at least one node through a second network. The network performance of the first network is greater than that of the second network. For example, the first node is a node in the plurality of nodes.

[0009] During the acquisition of the calculation results of the nodes in the first cluster by the first node, each time when the first node obtains the one of the N parts of the calculation results of the nodes in the first cluster, the first node sends the one of the N parts of the calculation results to the nodes in the second cluster. Thus, during the acquisition of the remaining calculation results of the nodes in the first cluster by the first node, the one of the N parts of the calculation results is transmitted from the first cluster to the second cluster, so that the inter-cluster transmission and intra-cluster transmission in the sharing of the calculation results by different clusters are parallel, the time consumption of the sharing of the calculation results is reduced, and the training efficiency of the model is improved.

[0010] In a possible implementation, the one of the N parts of the calculation results of the plurality of nodes includes the one of the N parts of the calculation results of each node in the plurality of nodes; the at least one node is used to update the target calculation result of the at least one node by using the most recently received one of the N parts of the calculation results of the plurality of nodes each time when the one of the N parts of the calculation results of the plurality of nodes is received; and after the calculation results of the plurality of nodes and the calculation result of the at least one node are updated to the target calculation result, the target calculation result is used to update the parameters of the target model deployed in the at least one node.

[0011] In this implementation, the computing result of the other node can be updated into the target computing result of the node each time the computing result of the other node is received, so that the computing result of the other node does not need to be stored specially, and storage resources are saved. In addition, the computing result is updated into the target computing result in a way of receiving and updating, so that the node can obtain the target computing result containing the computing result of all nodes when receiving the computing result of all nodes, and the parameters of the model can be updated by using the target computing result.

[0012] In a possible implementation, the sending of the Nth part of the computing result of the plurality of nodes recently acquired by the first node to the at least one node includes: sending, by the first node, the Nth part of the computing result of the plurality of nodes recently acquired to a second node in the second cluster; and sending, by the second node, the Nth part of the computing result of the plurality of nodes recently received to the at least one node each time the second node receives the Nth part of the computing result of the plurality of nodes.

[0013] In this implementation, the second node sends the Nth part of the computing result of the node in the first cluster to the node in the second cluster each time the second node receives the Nth part of the computing result of the node in the first cluster, so that the parallel of inter-cluster transmission and intra-cluster transmission in the sharing of the computing result of the plurality of nodes is realized, the time consumption of sharing of the computing result of the plurality of nodes is shortened, and the training efficiency of the model is improved.

[0014] In a possible implementation, the method further includes: setting the value of N in a manner of reducing the time length required for the transmission of the computing result of the plurality of nodes from the plurality of nodes to the at least one node.

[0015] The time length required for the transmission of the computing result of the plurality of nodes from the plurality of nodes to the at least one node is the time consumption of sharing of the computing result of the plurality of nodes, and the size of N affects the time consumption. In this implementation, the value of N is set in a manner of reducing the time consumption, so as to obtain the value of N that can reduce the time consumption to the minimum. In this way, the time consumption can be reduced to the maximum extent when the value of N is used for sharing of the computing result, and the training efficiency of the model is improved.

[0016] In a possible implementation, the value of N is set in a manner of reducing the time length required for the transmission of the computing result of the plurality of nodes from the plurality of nodes to the at least one node, including: calculating the value of N according to formula (I). Minimize: F(x) = T RS (N) + T AG (N) + Max(N*T AR (N), (N-1)*(T RS (N) + T AG (N))) (I)

[0017] F(x) represents a time length required for the calculation results of the plurality of nodes to be transmitted to the at least one node, T RS (N) represents a time length required for the first node to obtain the Nth part of the calculation results of the plurality of nodes, T AR (N) represents a time length required for the first node to send the Nth part of the calculation results of the plurality of nodes to the second node in the second cluster, T AG (N) represents a time length required for the second node to send the Nth part of the calculation results of the plurality of nodes to the at least one node.

[0018] In a second aspect, a management device is provided. The training system in which the device is located further includes a plurality of nodes, each of the plurality of nodes is deployed with a target model, a plurality of nodes in the plurality of nodes are located in a first cluster, and at least one node in the plurality of nodes other than the plurality of nodes is located in a second cluster. The device includes: a division module configured to divide a training set of the target model into a plurality of training subsets; wherein the plurality of training subsets and the plurality of nodes are in one-to-one correspondence; a sending module configured to send each training subset in the plurality of training subsets to a node corresponding to the training subset, so that the node corresponding to the training subset performs calculation of the target model for the training subset to obtain a calculation result of the node corresponding to the training subset; and an indication module configured to, in a case where a first node in the first cluster is used to obtain the calculation results of the plurality of nodes and send the obtained calculation results to the at least one node, instruct the first node to send, to the at least one node, the Nth part of the calculation results of the plurality of nodes that is obtained most recently each time the Nth part of the calculation results of the plurality of nodes is obtained; N is an integer greater than 1; wherein the at least one node is configured to update parameters of the target model deployed in the at least one node by using the calculation results of the plurality of nodes and a calculation result of the at least one node.

[0019] In a possible implementation, the Nth part of the calculation results of the plurality of nodes includes the Nth part of the calculation result of each of the plurality of nodes; the at least one node is configured to update the target calculation result of the at least one node by using the Nth part of the calculation results of the plurality of nodes that is received most recently each time the Nth part of the calculation results of the plurality of nodes is received; and after the calculation results of the plurality of nodes and the calculation result of the at least one node are updated to the target calculation result, the target calculation result is used to update the parameters of the target model deployed in the at least one node.

[0020] In a possible implementation, the sending, by the at least one node, of the Nth part of the computation result of the plurality of nodes recently acquired by the first node comprises: the first node sending the Nth part of the computation result of the plurality of nodes recently acquired to a second node in the second cluster; and the second node sending, to the at least one node, the Nth part of the computation result of the plurality of nodes recently received each time the second node receives the Nth part of the computation result of the plurality of nodes.

[0021] In a possible implementation, the indication module is further configured to set the value of N in a manner that reduces a time length required for transmission of the computation result of the plurality of nodes from the plurality of nodes to the at least one node.

[0022] In a possible implementation, the indication module is further configured to calculate the value of N according to formula (I). RS (N) + T AG (N) + Max(N*T AR (N), (N-1)*(T RS (N) + T AG (N))) (I)

[0023] In formula (I), F(x) represents the time length required for transmission of the computation result of the plurality of nodes from the plurality of nodes to the at least one node, T RS (N) represents the time length required for the first node to acquire the Nth part of the computation result of the plurality of nodes, T AR (N) represents the time length required for the first node to send the Nth part of the computation result of the plurality of nodes to the second node in the second cluster, T AG (N) represents the time length required for the second node to send the Nth part of the computation result of the plurality of nodes to the at least one node.

[0024] In a possible implementation, the first node is configured to acquire the Nth part of the computation result of the plurality of nodes through a first network, and the first node is configured to send the Nth part of the computation result of the plurality of nodes recently acquired to the at least one node through a second network, wherein a network performance of the first network is greater than a network performance of the second network; and / or the first node is a node in the plurality of nodes.

[0025] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method provided in the first aspect.

[0026] In a fourth aspect, a computer-readable storage medium is provided, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0027] In a fifth aspect, a computer program product including instructions is provided, when the instructions are run by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0028] The advantages of the second aspect to the fifth aspect can refer to the advantages of the first aspect described above, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0029] FIG. 1 is a schematic diagram of an inter-cluster gradient sharing scheme;

[0030] FIG. 2 is a schematic diagram of an inter-cluster computing structure sharing scheme provided by an embodiment of the present application;

[0031] FIG. 3 is a schematic diagram of a training system provided by an embodiment of the present application;

[0032] FIG. 4 is a schematic diagram of a training system provided by an embodiment of the present application;

[0033] FIG. 5 is a schematic diagram of a training system provided by an embodiment of the present application;

[0034] FIG. 6 is a flowchart of a model training method provided by an embodiment of the present application;

[0035] FIG. 7 is a structural schematic diagram of a management apparatus provided by an embodiment of the present application;

[0036] FIG. 8 is a structural schematic diagram of a computing device provided by an embodiment of the present application;

[0037] FIG. 9 is a structural schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0038] FIG. 10 is a structural schematic diagram of a computing device cluster provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] The schemes provided by the embodiments of the present application will be described below with reference to the accompanying drawings. In the embodiments of the present application, “multiple” refers to two or more. “First”, “second”, and the like are only used to distinguish similar objects, and do not necessarily describe a specific order or number of objects.

[0040] In order to facilitate understanding of the schemes provided by the embodiments of the present application, the technical terms that may be involved in the embodiments of the present application will be introduced first.

[0041] Model: composed of multiple neural network layers. Usually, the multiple neural network layers are arranged in series, and the output of one neural network layer is the input of the next neural network layer. A neural network layer can be referred to as a network layer or a layer for short.

[0042] Model training: refers to the process of updating the parameters of the model. Generally, the parameters of the model are preset or randomly generated. By updating the parameters of the model, the model learns to handle corresponding tasks, such as image recognition, speech recognition, natural language translation, computer game playing, content generation, etc. Model training needs to go through multiple iterations until convergence or reach a predetermined number of training times. Each iteration requires inputting a certain amount of data set. The model performs forward calculation on the input data set to obtain the loss function. The model then performs backward calculation on the loss function to obtain the gradient. Then, the gradient is used to update the model parameters.

[0043] Forward propagation: also known as forward propagation, refers to starting from data input, calculating and passing data through each layer of the model to obtain the output result of the model. The purpose of forward propagation is to calculate the output result of the model.

[0044] Model calculation: refers to the calculation performed by the model on the relevant data set using the model parameters. Model calculation can be divided into forward calculation and backward calculation.

[0045] Calculation result: refers to the result obtained by calculating the relevant data set using the model parameters. The calculation result of forward calculation includes the loss function, and the calculation result of backward calculation includes the gradient.

[0046] Target calculation result: refers to the aggregated calculation result, which is the data aggregated from the calculation results of one or more nodes participating in model training. When the calculation result is the gradient, the target calculation result is the calculation result used to update the parameters of the target model. Aggregation can be summation or averaging.

[0047] Forward calculation: refers to the calculation used by the layers or model blocks in the model for forward propagation, which is used to obtain the loss function of the model.

[0048] Backward propagation: also known as backward propagation, refers to obtaining the loss function according to the difference between the output result of the model and the label, and then using the loss function to calculate the gradient layer by layer in reverse through the chain rule. Through backward propagation, the gradient of each parameter on the loss function can be calculated, and the gradient of the parameter is used to update the parameter.

[0049] Backward computation: refers to the computation of the layers or model blocks in the model for backward propagation. Backward computation is used to obtain the gradient based on the loss function obtained by forward computation, and the gradient is used to update the parameters of the model.

[0050] Gradient: refers to the direction in which the loss function value decreases most quickly. The gradient is obtained by backward computation and is a result of backward computation. The gradient is actually a vector representing a direction, and the elements of each position in the gradient represent the rate of change of the corresponding position elements of the loss function value with respect to the weight. The gradient is used to update the parameters of the model. Specifically, the gradient can be multiplied by the learning step size to obtain a product, and then the parameters of the model are added (or subtracted) to the product to obtain the updated parameters.

[0051] Gradient synchronization: Gradient synchronization is a technique used in distributed training of a model (such as data parallel training) to ensure that the model parameters in each training node remain consistent after each training iteration. In data parallel training, each training node calculates the gradient of a small piece of data, and then through gradient synchronization, the gradients calculated by all training nodes are aggregated (such as averaging or accumulating), and then the aggregated gradient is used to update the model parameters.

[0052] Data parallelism (DP): a distributed training technique for a model that achieves distributed training of the model through parallel computing paradigm. In data parallelism, each training node in multiple training nodes deploys a model, and different training nodes perform calculations on different data sets to split the model calculation task into multiple smaller, independent, and parallel processing subtasks. In data parallelism, the gradient calculated by each training node is a partial gradient of the model, and multiple training nodes perform gradient synchronization to aggregate the gradients calculated by multiple training nodes. The aggregated gradient is used to update the model parameters to ensure consistency of the model parameters in each training node.

[0053] Pipeline parallelism (PP): a distributed training technique for a model, which refers to different nodes running different layers of the model and belongs to inter-layer parallelism. That is, in pipeline parallelism, the model is serialized and divided into multiple model blocks connected in sequence. During model training, related calculations of different model blocks are performed by different nodes. A model block can be referred to as a stage in the model training process, and N model blocks are N stages in the model training process.

[0054] Region: A computing node deployment unit in the infrastructure, which is a deployment unit divided from the dimensions of geographical location and network latency. The isolation level between regions is high. Among them, the computing nodes in the same region share public services such as elastic computing, block storage, object storage, VPC network, elastic public IP, and image.

[0055] Availability zone (AZ): A computing node deployment unit in the infrastructure, and an AZ is a collection of one or more physical data centers with independent wind, fire, water, and electricity. An AZ includes one data center or multiple data centers in close geographical locations. A region can include multiple AZs.

[0056] Training node: It can be referred to as a node, which refers to a computing node used for performing model training or model calculation. The node can include a processing component and a storage component. Among them, the processing component can be a processor or a computing core. The storage component can be a memory, such as an on-chip memory or an on-board memory. Common training nodes include hosts, graphics processing units (GPUs), tensor processing units (TPUs), and the like. Among them, the host can be a server or other computing device. A host can have at least one processing unit and a network card, wherein the processing unit is used to perform model calculation to obtain a calculation result.

[0057] Inter-chip interconnect (ICI): A high-speed communication channel, which is usually used for connection between nodes in the same cluster. The inter-chip interconnect channel combines the processes and protocols used by chips in a multi-node system, allowing resources to be shared between chips in a multi-node system.

[0058] Data center network (DCN): A connection network between clusters, which is used to connect nodes in multiple clusters together. The DCN has scalability and high efficiency, and can connect tens or even hundreds of thousands of nodes to meet the growing connection needs of cloud computing.

[0059] Parameter plane network: A high-speed network specially designed for distributed training. Nodes in the same cluster can be connected through the parameter plane network. Among them, when the node is a host, the processing units of two hosts in the same cluster can communicate through the parameter plane network without going through the network card of the host. Generally speaking, the communication rate of the parameter plane network can reach 200GE.

[0060] Cluster: also referred to as a training cluster, a cluster composed of multiple nodes. The nodes constituting the same cluster are located in proximity, for example, in the same region or AZ. The communication capability between the nodes in the same cluster is strong, that is, the communication capability between the nodes in the same cluster is greater than the communication capability between the nodes in the cluster and the nodes outside the cluster. For example, the network between the nodes in the same cluster is a parameter plane network or a quasi-parameter plane network, and the network between the nodes in the cluster and the training nodes outside the cluster is a DCN or a quasi-DCN network. In addition, in the embodiments of the present application, the nodes in the same region or AZ are not necessarily in the same cluster. This is because some nodes in the same region or AZ can not be connected with other nodes in the region or AZ using a parameter plane network or a quasi-parameter plane network, resulting in poor communication capability between the nodes and other nodes in the region or AZ, and the nodes and other nodes in the region or AZ can be considered to be in different clusters.

[0061] Tensor processing unit (TPU): an application-specific integrated circuit for accelerating machine learning workloads. TPU uses hardware designed specifically to perform large matrix operations common in machine learning algorithms, enabling more efficient model training.

[0062] In the related art, when training a model in a data parallel manner using two or more clusters, a scheme of serializing intra-cluster communication and inter-cluster communication is adopted to perform gradient sharing. Specifically, a scheme of training a model using cluster A1 and cluster A2, and sharing gradients from cluster A1 to cluster A2 can be as shown in FIG. 1. The sending node in cluster A1 obtains the gradients calculated by all nodes in cluster A1 through inter-cluster communication. Then, the sending node sends the obtained gradients to the receiving node in cluster A2 through inter-cluster communication. After that, the receiving node in cluster A2 sends the received gradients from cluster A1 to each node in cluster A2 through inter-cluster communication. The method of sharing gradients from cluster A2 to cluster A1 is similar to the scheme shown in FIG. 1. This scheme takes a long time to share gradients, affecting the training efficiency of the model.

[0063] The embodiment of the present application provides a model training method. In the method, two clusters participating in target model training can share calculation results in parallel through intra-cluster communication and inter-cluster communication. The calculation results can be gradients. Specifically, the clusters participating in target model training can include cluster 100 and cluster 200, a plurality of nodes in the cluster are used for training the target model, and at least one node in the cluster 200 is used for training the target model. When the node T in the cluster 100 receives one Nth of the calculation results of the plurality of nodes in the cluster 100, the node T sends the one Nth of the calculation results of the plurality of nodes to the at least one node in the cluster 200, and continues to obtain another one Nth of the calculation results of the plurality of nodes in the cluster 100, and subsequently sends the other one Nth of the calculation results of the plurality of nodes to the at least one node in the cluster 200.

[0064] Wherein, N is an integer greater than 1. The one Nth of the calculation results of the plurality of nodes is the calculation result transmission granularity of the cluster 100, that is, the granularity of the inter-cluster transmission of the calculation results when sharing the calculation results, and is the maximum data amount of a single transmission between clusters.

[0065] Referring to FIG. 2, the node T sends the one Nth of the calculation results of the plurality of nodes to the node R in the cluster 200 every time the node T obtains the one Nth of the calculation results of the plurality of nodes. The node R sends the one Nth of the calculation results of the plurality of nodes to the node for training the target model in the cluster 200 every time the node R receives the one Nth of the calculation results of the plurality of nodes from the node T. The node T is also used for training the target model, that is, the node T is a node in the plurality of nodes. The node R is also used for training the target model, that is, the node R is a node in the at least one node.

[0066] As shown in FIG. 2, the method realizes parallel intra-cluster communication and inter-cluster communication, reduces the time consumption of inter-cluster sharing of calculation results, and improves the training efficiency of the model.

[0067] Next, the model training method provided by the embodiment of the present application is described.

[0068] FIG. 3 shows a training system that can be used to implement the method. The training system includes a management device 300 and a plurality of clusters, such as cluster 100 and cluster 200. Each cluster can include a plurality of nodes, such as cluster 100 which can include node 110, node 120, etc., and cluster 200 which can include node 210, node 220, etc.

[0069] In some embodiments, the plurality of clusters are divided according to the communication capability between nodes, the communication capability between nodes in the same cluster is greater than the communication capability between nodes in different clusters, for example, nodes in the same cluster communicate with each other using a parameter plane network or a similar parameter plane network, and nodes in different clusters communicate with each other using a DCN network or a similar DCN network.

[0070] The management device 300 can be any device, software, equipment, platform or cluster with data processing function. In some embodiments, the management device 300 can also be software deployed in one or more nodes in any cluster in the plurality of clusters.

[0071] As shown in FIG. 4, the management device 300 is configured to receive a training task of a target model submitted by a user, and select nodes in the plurality of clusters to perform the training task, i.e., select nodes in the plurality of clusters to train the target model.

[0072] The management device 300 can select a plurality of nodes in the plurality of clusters, and the selected plurality of nodes are used to train the target model. Wherein, the management device 300 can deploy the target model in each node in the plurality of nodes to train the target model in a data parallel manner. Deploying the target model to a node can mean storing parameters of the target model in a storage space accessible to the node, so that the node can load the parameters of the target model and then perform the calculation of the target model based on the parameters of the target model. Wherein, the calculation of the target model includes forward calculation and backward calculation.

[0073] Wherein, the plurality of nodes for training the target model are distributed in different clusters, for example, several nodes in the plurality of nodes are located in the cluster 100, and at least one node in the plurality of nodes other than the several nodes is located in the cluster 200.

[0074] In some embodiments, the management device 300 can also indicate the calculation result sending granularity of the cluster to which the target model is related to. The cluster to which the target model is related to refers to the cluster where the nodes for training the target model are located.

[0075] In addition, since the number of nodes for training the target model in different clusters can be different, the calculation result sending granularity of different clusters can also be different.

[0076] The sending granularity of the calculation result influences the time consumption of sharing the calculation result among clusters. If the sending granularity of the calculation result is too large, the parallelism of inter-cluster communication and intra-cluster communication is low, resulting in large time consumption of sharing the calculation result among clusters. If the sending granularity of the calculation result is too small, the calculation overhead required for sending the calculation result is increased, and the bandwidth cannot be fully utilized, resulting in low bandwidth utilization and also large time consumption of sharing the calculation result among clusters. Therefore, the sending granularity of the calculation result needs to be set to minimize the time consumption of sharing the calculation result among clusters.

[0077] The management apparatus 300 can set the sending granularity of the calculation result of the cluster involved in the target model. Next, taking the cluster 100 and the cluster 200 as examples of the cluster involved in the target model, the setting method of the sending granularity of the calculation result is exemplarily introduced.

[0078] A plurality of nodes in the cluster 100 can be set to train the target model, and at least one node in the cluster 200 can be set to train the target model. The management apparatus 300 sets the sending granularity of the calculation result of the cluster 100 with the aim of reducing the time length required for sharing the calculation result of the plurality of nodes to the at least one node. The sharing of the calculation result of the plurality of nodes to the at least one node refers to the transmission of the calculation result of the plurality of nodes from the plurality of nodes to the at least one node. As described above, the sending granularity of the calculation result of the cluster 100 is N-th of the calculation result. Therefore, setting the sending granularity of the calculation result of the cluster 100 can also be referred to as setting the value of N.

[0079] The transmission of the calculation result of the plurality of nodes from the plurality of nodes to the at least one node includes: the node T obtaining the calculation result of the plurality of nodes from the plurality of nodes, the node T sending the calculation result of the plurality of nodes to the node R in the cluster 200, and the node R sending the calculation result of the plurality of nodes to the at least one node. Therefore, the time length required for the transmission of the calculation result of the plurality of nodes from the plurality of nodes to the at least one node depends on the time length required for the node T to obtain the calculation result of the plurality of nodes and the time length required for the node T to send the calculation result of the plurality of nodes to the node R. The higher the parallelism between the node T obtaining the calculation result of the plurality of nodes and the node T sending the calculation result of the plurality of nodes to the node R, the shorter the time length required for the transmission of the calculation result of the plurality of nodes from the plurality of nodes to the at least one node. Therefore, the management apparatus 300 can set the value of N with the aim of improving the parallelism between the node T obtaining the calculation result of the plurality of nodes and the node T sending the calculation result of the plurality of nodes to the node R.

[0080] In some embodiments, the value of N can be set by formula (I). Minimize: F(x) = T RS (N) + T AG(N) + Max(N*T AR (N) + (N-1) * (T RS (N) + T AG (N) + (N-1) * (T

[0081] wherein, T RS (N) represents the time length required for the node T to obtain the Nth part of the calculation results of the several nodes. T AR (N) represents the time length required for the node T to send the Nth part of the calculation results of the several nodes to the node R. T AG (N) represents the time length required for the node R to send the Nth part of the calculation results of the several nodes received from the node T to the at least one node.

[0082] F(x) represents the time length required for the calculation results of the several nodes to be transmitted from the several nodes to the at least one node, and Minimize represents the minimum value. Max represents the maximum value.

[0083] In one example, the node can be a host, which has at least one processing unit and a network card. The processing units of the hosts in the same cluster can directly communicate with each other, for example, through a parameter plane network. The calculation result of the node is specifically the calculation result of the processing unit of the node. That is, the processing unit of the node T can obtain the calculation result from the processing units of the several nodes. The nodes of different clusters communicate through the network card. Therefore, the calculation results of the several nodes are transmitted from the several nodes to the at least one node, which further includes the transmission of the calculation results of the several nodes from the processing unit of the node T to the network card of the node T, and the transmission of the calculation results of the several nodes from the network card of the node R to the network card of the node R. In this case, the value of N can be set through formula (II). Minimize: F(x) = T RS (N) + T AG (N) + N * (H2D(N) + D2H(N)) + Max(N*T AR (N) + (N-1) * (T RS (N) + T AG (N) + (N-1) * (T

[0084] wherein, H2D(N) represents the time length required for the Nth part of the calculation results of the several nodes to be transmitted from the processing unit of the node T to the network card of the node T, and D2H(N) represents the time length required for the Nth part of the calculation results of the several nodes to be transmitted from the network card of the node R to the processing unit of the node R.

[0085] The setting of the value of N is converted into a nonlinear programming solution task by formula (I) or formula (II). Gradient descent, stochastic gradient descent, momentum gradient descent, or conjugate gradient method can be used to find the optimal solution of formula (I) or formula (II), thereby obtaining the value of N.

[0086] In addition, the setting of the value of N does not need to consider the calculation time of the calculation performed by the plurality of nodes, because:

[0087] The calculation time is much smaller than the communication transmission time.

[0088] If the total data amount of the calculation result to be shared is unchanged, the calculation time is actually a fixed constant value.

[0089] Therefore, the influence of the calculation overhead on the overall time can be ignored.

[0090] The value of N can be calculated by T RS (N), T AG (N), H2D(N), D2H(N), T AR (N) based on the information of the target model and the topology information of the cluster. That is, the value of N is set based on the information of the target model and the topology information of the cluster by using the above formula. As shown in FIG. 5, the management device 300 can obtain the information of the target model from a user, obtain the topology information of the cluster 100 from the cluster 100, and obtain the topology information of the cluster 200 from the cluster 200.

[0091] The information of the target model is used by the management device 300 to calculate the data amount of the calculation result of a single node.

[0092] In some embodiments, the user can directly input the data amount of the calculation result of a single node as the information of the target model to the management device 300.

[0093] In some embodiments, the information of the target model can include: a model type (e.g., GPT, T5, Llama, etc.), an input length (sequence length), a feature length (embedding size), a hidden layer length (hidden size), a number of model layers (layer num), a gradient data type (weight type) (e.g., bfloat, float16, float32, etc.), a cross-cluster synchronization operator (e.g., reduce scatter, all reduce, all gather), whether to use a distributed optimizer (use distributed optimizer), a number of data parallel (data parallel), a number of pipeline parallel (pipeline parallel), a number of context parallel, a number of tensor parallel, etc. Through these information, the management device 300 can calculate the data amount of the calculation result of a single node.

[0094] The topology information of the cluster can include the number of nodes in the cluster for training the target model, the bandwidth between the nodes in the same cluster, the bandwidth between the nodes of different clusters, the data transmission protocol between the nodes, the buffer size for transmitting data, and the related underlying transmission overhead, etc. Among them, the bandwidth can be the upper limit of the physical bandwidth. The data transmission protocol between the nodes includes the delay characteristics, the congestion control algorithm, etc. When the node is a host, the bandwidth between the nodes in the same cluster refers to the bandwidth between the processing units of different nodes, and the bandwidth between the nodes of different clusters includes the bandwidth between the processing unit and the network card of the node where the processing unit is located, and the bandwidth between the network cards of different hosts.

[0095] Based on the information of the target model and the topology information of the cluster, the T RS (N), T AG (N) can be calculated by using formula (III).

[0096] Size represents the data size of the computation results of the plurality of nodes, DP represents the number of nodes in the cluster 100 for training the target model (i.e., the number of nodes in the plurality of nodes) and the number of nodes in the cluster 200 for training the target model (i.e., the number of nodes in the at least one node), N is the number of partitions, Band represents the interconnection physical bandwidth between the processing units of the nodes in the same cluster, Latency represents the transmission delay between the processing units of the nodes in the same cluster and the transmission delay between the processing units of the same node and the network card, and Base represents the transmission basic overhead between the processing units of the nodes in the same cluster and the transmission delay between the processing units of the same node and the network card.

[0097] T theoretical (Size, DP, N, Band) represents the theoretical time consumption of the node T to obtain the computation results of the plurality of nodes and the theoretical time consumption of the node R to send the computation results of the plurality of nodes to the at least one node. In one example, if the node T obtains the computation results from the plurality of nodes through the Ring Reduce Scatter algorithm, the node R sends the computation results of the plurality of nodes to the at least one node through the All Gather algorithm.

[0098] represents the effective utilization rate of the physical bandwidth, represents the additional overhead caused by the congestion algorithm, is the additional overhead caused by the transmission buffer, and all of them are in a non-linear relationship with the size of the data size of the computation results of the plurality of nodes. The relationship with the data size can be obtained by pure theoretical calculation, or a function can be obtained by experiment fitting. Different communication protocols need to consider different influence factors, and the above function can be customized according to the characteristics of the communication protocol.

[0099] In some embodiments, T AR (N) can be obtained from the data size of the computation results of the plurality of nodes, the inter-cluster bandwidth between the cluster 100 and the cluster 200, and the inter-node bandwidth between the node T and the node R. Specifically, T AR (N) is positively correlated with the data size of the computation results of the plurality of nodes, T AR (N) is negatively correlated with the inter-cluster bandwidth, T AR (N) is negatively correlated with the inter-node bandwidth.

[0100] In addition, H2D(N), D2H(N) is determined by the bandwidth between the network card and the processing unit in the related node. Generally, the network card and the processing unit in the same node communicate through a peripheral component interconnect express (PCIe) bus, and then H2D(N), D2H is determined by the PCIe bus bandwidth between the network card and the processing unit of the related node.

[0101] In this way, by the above manner, the value of N that can minimize the time consumption of sharing the calculation result between clusters can be calculated.

[0102] The above example introduces the system architecture provided by the embodiments of the present application and the related functions of the management device 300. Next, taking the management device 300 as an execution subject in the implementation of the system architecture as an example, the flow of the model training method provided by the embodiments of the present application is introduced in combination with FIG. 6.

[0103] In step 601, the management device 300 can divide the training set of the target model into a plurality of training subsets. The plurality of training subsets and the plurality of nodes used for training the target model are in one-to-one correspondence.

[0104] The training of the target model adopts data parallelism, and the management device 300 divides the training set according to the number of nodes used for training the target model to obtain a plurality of training subsets. One node corresponds to one training subset.

[0105] In step 602, the management device 300 sends each training subset in the plurality of training subsets to the node corresponding to the training subset, so that the node corresponding to the training subset performs calculation of the target model for the training subset to obtain the calculation result of the node corresponding to the training subset.

[0106] Step 602 includes step 602a and step 602b. As described above, several nodes in the cluster 100 are used for training the target model, and at least one node in the cluster 100 is used for training the target model. In step 602a, the management device 300 sends several training subsets corresponding to the several nodes to the cluster 100 to send the several training subsets to the corresponding nodes. In step 602b, the management device 300 sends at least one training subset corresponding to the at least one node to the cluster 200 to send the at least one training subset to the corresponding node. As described above, the several nodes are the nodes in the cluster 100 used for training the target model, and the at least one node is the node in the cluster 200 used for training the target model.

[0107] For convenience of description, as shown in FIG. 6, the plurality of nodes can include node 110, node 120, etc., and the at least one node can include node 210, node 220, etc.

[0108] Each node in the plurality of nodes for training the target model can perform step 701, perform calculation of the target model to obtain a calculation result. In the forward propagation, when the node receives a data set (i.e., a training subset corresponding to the node), the node can perform forward calculation of the target model for the training subset to obtain an activation value or a loss function. Then, in the backward propagation, the node performs backward calculation of the target model for the loss function or the activation value calculated by another node to obtain a calculation result of the node. The calculation result of the node can be a gradient.

[0109] In some embodiments, the node can be a host including a plurality of processing units. The node can train the target model in a pipelined parallel manner by using the plurality of processing units. Specifically, the target model deployed in the host can be divided into a plurality of model blocks, and the plurality of model blocks and the plurality of processing units correspond one by one. Each model block in the plurality of model blocks is deployed into the processing unit corresponding to the model block. The node performs calculation of the target model for the training subset, specifically, the processing unit in the node performs calculation of the model block corresponding to the processing unit for the training subset.

[0110] In step 603, the management device 300 instructs the node T in the cluster 100 to obtain calculation results of the plurality of nodes, and in the case where the node T sends the obtained calculation results to the at least one node, the node T sends the most recently obtained Nth part of the calculation results of the plurality of nodes to the at least one node each time the node T obtains the Nth part of the calculation results of the plurality of nodes.

[0111] Each node in the plurality of nodes can send the calculation result to the node T, so that the node T can obtain the calculation result of each node in the plurality of nodes. The node T can be any node in the cluster 100, and the node T can communicate with the nodes (e.g., the at least one node) in the cluster 200. In some embodiments, the node T is also used for training the target model, i.e., the node T is any node in the plurality of nodes.

[0112] In some embodiments, each of the several nodes described above can simultaneously serve as a node T, where different nodes T are used to obtain different data in the calculation result of the nodes. For example, the several nodes can be specifically M nodes, where M is an integer greater than 1. Each of the M nodes serves as a node T, i.e., there are M nodes T. Each of the M nodes divides the calculation result calculated by itself into M data blocks of the same size. Different data blocks of the M data blocks correspond to different ranks, and the ranks of the M data blocks correspond to the M nodes T one by one. Each of the M nodes sends the corresponding data block to a node T in the M nodes T according to the ranks of the data blocks. That is, each of the M nodes T is used to obtain a data block of the same rank in the calculation result of the M nodes, and the rank of the data block obtained by each node T corresponds to the node T. In this way, each of the M nodes T is used to obtain a part of the calculation results of all nodes in the M nodes, and the part of the calculation results of all nodes is the calculation results of several nodes to be obtained by the node T. In step 603, the node T obtains Nth of the calculation results of the several nodes (i.e., the M nodes), and the Nth of the calculation results of the several nodes to be obtained by the node T is Nth of the calculation results of the several nodes.

[0113] In addition, in the following description, the operations performed by one of the M nodes T1 are described as an example, and the operations performed by other nodes T1 can be implemented by reference, and will not be described one by one.

[0114] In some embodiments, the cluster 100 has at least two nodes T, and the calculation results of the several nodes can be divided into at least two sub-results, and the at least two sub-results correspond to the at least two nodes T one by one. The node T corresponding to the sub-result is used to obtain the sub-result of each of the several nodes. That is, each of the several nodes can divide the calculation result of the node into at least two sub-results, and send each of the at least two sub-results to the node T corresponding to the sub-result. In addition, the node T corresponding to the sub-result can obtain Nth of the sub-result each time, so that the at least two nodes T can obtain Nth of the calculation results of the several nodes.

[0115] In some embodiments, the node T can use the Reduce Scatter algorithm to receive the calculation results of the several nodes.

[0116] During the process that node T acquires the calculation results of the plurality of nodes, node T sends the Nth part of the calculation results of the plurality of nodes that is acquired most recently to the at least one node in cluster 200 for training the target model, each time node T acquires the Nth part of the calculation results of the plurality of nodes. During the process that node T sends the Nth part of the calculation results of the plurality of nodes to the at least one node, node T does not stop acquiring the calculation results of the plurality of nodes, i.e., node T continues to receive the remaining part of the calculation results of the plurality of nodes. For example, as shown in FIG. 6, node T acquires the Nth part of the calculation results at step 702, node T performs step 703a to send the Nth part of the calculation results to the at least one node, and performs step 703b to acquire another Nth part of the calculation results. When node T acquires another Nth part of the calculation results at step 703b, node T performs step 704 to send the Nth part of the calculation results acquired at step 703b to the at least one node. During the process that node T sends the Nth part of the calculation results of the plurality of nodes acquired at step 703b to the at least one node, node T continues to acquire the remaining part of the calculation results of the plurality of nodes, and the process is repeated until node T acquires all the calculation results of the plurality of nodes.

[0117] In some embodiments, the Nth part of the calculation results of the plurality of nodes acquired by node T includes the Nth part of the calculation results of each node in the plurality of nodes. After node T aggregates the Nth part of the calculation results of all the nodes in the plurality of nodes, node T sends the aggregated calculation results to the at least one node. That is, the Nth part of the calculation results of the plurality of nodes sent by node T to the at least one node is the aggregated calculation results. The aggregation can be summation or averaging.

[0118] In some embodiments, node T can use the All Reduce algorithm to send the Nth part of the calculation results of the plurality of nodes to the at least one node.

[0119] In some embodiments, node T and the plurality of nodes are connected through network B1, and node T acquires the calculation results of the plurality of nodes through network B1. Node T and the at least one node are connected through network B2, and node T sends the Nth part of the calculation results of the plurality of nodes acquired most recently to the at least one node through network B2. The network performance of network B1 is greater than that of network B2. For example, network B1 can be a parameter plane network (e.g., an inter-card fast interconnection channel), and network B2 can be a DCN.

[0120] In some embodiments, as shown in FIG. 6, node T sends the Nth part of the calculation results of the plurality of nodes that node T has recently obtained to node R in cluster 200. For example, in step 703a, node T sends the Nth part of the calculation results obtained in step 702 to node R. For another example, in step 704b, node T sends the Nth part of the calculation results obtained in step 703b to node R.

[0121] Node R can be any node in cluster 200. In some embodiments, node R can be a node in the at least one node described above, i.e., node R is also used to train the target model. For example, when the at least one node described above is at least two nodes, the at least two nodes can simultaneously serve as node R. In some embodiments, node R can be a node other than the at least one node described above, i.e., node R is not used to train the target model.

[0122] Node R can send the calculation results of the nodes to the at least one node described above each time node R receives the calculation results of the nodes. That is, node R sends the Nth part of the calculation results of the plurality of nodes that node T has recently obtained to the at least one node each time node R receives the Nth part of the calculation results of the plurality of nodes that node T has recently obtained. For example, as shown in FIG. 6, when node R receives the Nth part of the calculation results sent by node T in step 703a, node R can perform step 704a to send the Nth part of the calculation results of the plurality of nodes to the at least one node. When node R receives the Nth part of the calculation results of the plurality of nodes sent by node T in step 704b, node R can perform step 705b to send the Nth part of the calculation results of the plurality of nodes to the at least one node.

[0123] When node R is a node in the at least one node described above, node R sends the Nth part of the calculation results of the plurality of nodes to a node (e.g., node 220) in the at least one node other than node R.

[0124] When node R is a node other than the at least one node described above, node R sends the Nth part of the calculation results of the plurality of nodes to all nodes in the at least one node.

[0125] In some embodiments, node R includes a processing unit and a network card, and the at least one node includes a processing unit. Node R receives the calculation results sent by node T through the network card of node 210, then the network card of node R sends the received calculation results to the processing unit of node R, and then the processing unit of node R sends the calculation results to the processing units of the at least one node, thereby realizing the transmission of the calculation results from node R to the at least one node.

[0126] In some embodiments, the node R can employ an All Gather algorithm to send the computation result received from the node T to the at least one node.

[0127] In some embodiments, as described above, the cluster 100 has at least two nodes T, and the computation results of the plurality of nodes can be divided into at least two sub-results, the at least two sub-results corresponding to the at least two nodes T one by one. The node T corresponding to the sub-result is used to sequentially obtain the sub-result of each node in the plurality of nodes. Accordingly, the cluster 200 has at least two nodes R, the at least two nodes R and the at least two nodes T corresponding one by one, and each node R is used to receive the sub-result sent by the node T corresponding to the node R, and send the received sub-result to the at least one node.

[0128] In this way, the at least one node can receive the computation results of the plurality of nodes with the granularity of Nth of the computation results of the plurality of nodes.

[0129] The at least one node can update the parameters of the target model deployed in the at least one node by using the computation results of the plurality of nodes and the computation result of the at least one node. The computation result of the at least one node is obtained by the at least one node performing computation of the target model on the training subset corresponding to the at least one node.

[0130] When the at least one node is at least two nodes, each node in the at least two nodes receives the computation results of the plurality of nodes. Moreover, the nodes in the at least two nodes can obtain the computation results of each other through the network (e.g., parameter plane network) within the cluster. That is, each node in the at least two nodes can obtain the computation results of other nodes in the at least two nodes through the network within the cluster. In this way, each node in the at least two nodes can obtain the computation results of the at least two nodes.

[0131] Each node in the at least two nodes can update the parameters of the target model deployed in the node by using the computation results of the plurality of nodes and the computation results of the at least two nodes.

[0132] In some embodiments, the Nth of the computation results of the plurality of nodes includes the Nth of the computation result of each node in the plurality of nodes. That is, the Nth of the computation result of each node in the plurality of nodes constitutes the Nth of the computation results of the plurality of nodes. More specifically, the Nth of the computation results of the plurality of nodes obtained in step 702 is the Nth of the computation result of each node in the plurality of nodes.

[0133] In this embodiment, the at least one node updates the target computation result of the at least one node with the most recently received Nth of the computation results of the plurality of nodes each time the at least one node receives the Nth of the computation results of the plurality of nodes. The target computation result, also referred to as the aggregated computation result, is a computation result used to update the parameters of the target model. The node for training the target model updates the computation result calculated by itself and the computation results calculated by other nodes for training the target model into the target computation result of the node. For example, the updating can be summation or averaging. When the computation results calculated by all the nodes for training the target model are updated into the target computation result of the node, the target computation result is used to update the parameters of the target model deployed in the node. That is, when the computation results of the plurality of nodes and the computation result of the at least one node are updated into the target computation result of the at least one node, the at least one node can update the parameters of the target model deployed in the at least one node with the target computation result of the at least one node.

[0134] For example, when the node 210 receives the Nth of the computation results of the plurality of nodes sent by the node R through the step 704a, the node 210 can perform the step 705a to update the Nth of the computation results of the plurality of nodes sent by the node R through the step 704a into the target computation result of the node 210. Similarly, when the node 220 receives the Nth of the computation results of the plurality of nodes sent by the node R through the step 704a, the node 220 can perform the step 705a to update the Nth of the computation results of the plurality of nodes sent by the node R through the step 704a into the target computation result of the node 220.

[0135] Next, when the node 210 receives the Nth of the computation results of the plurality of nodes sent by the node R through the step 705b, the node 210 can perform the step 706 to update the Nth of the computation results of the plurality of nodes sent by the node R through the step 705b into the target computation result of the node 210. Similarly, when the node 220 receives the Nth of the computation results of the plurality of nodes sent by the node R through the step 705b, the node 220 can perform the step 706 to update the Nth of the computation results of the plurality of nodes sent by the node R through the step 705b into the target computation result of the node 220.

[0136] In this way, the node 220 can update the target computation result of the target model each time the node 220 receives the Nth of the computation results of the plurality of nodes, and can delete the received computation result after updating the received computation result into the target computation result, thereby saving the storage resource of the node 220.

[0137] When the calculation result of the plurality of nodes and the calculation result of the at least one node are both updated to the target calculation result of the node 210, the node 210 can perform step 707 to update the parameters of the target model deployed in the node 210 by using the target calculation result. Similarly, when the calculation result of the plurality of nodes and the calculation result of the at least one node are both updated to the target calculation result of the node 220, the node 220 can perform step 707 to update the parameters of the target model deployed in the node 220 by using the target calculation result.

[0138] In some embodiments, the management device 300 can set the value of N before performing step 603. The value of N can be set to reduce the time required for the calculation result of the plurality of nodes to be transmitted from the plurality of nodes to the at least one node. For details, reference can be made to the description of the embodiment shown in FIG. 5, which will not be repeated here.

[0139] In summary, the model training method provided by the embodiments of the present application can share the calculation results in a parallel manner of inter-cluster communication and intra-cluster communication, thereby reducing the time consumption of sharing the calculation results.

[0140] The time consumption of sharing the calculation results that can be saved by the embodiments of the present application can be calculated by the following formula.

[0141] The time consumption of sharing the calculation results of the existing scheme is as follows: RS+AR+H2D+D2H+AG+3*T (IV)

[0142] Wherein, RS represents the time required for node T to obtain the calculation result of the plurality of nodes, AR represents the time required for node T to send the calculation result of the plurality of nodes to node R, AG represents the time required for node R to send the calculation result of the plurality of nodes to the at least one node, and T represents the basic time overhead of the related operator of the calculation result sharing.

[0143] The time consumption of sharing the calculation results in the model training method provided by the embodiments of the present application is as follows:

[0144] The time consumption of sharing the calculation results that can be saved by the model training method provided by the embodiments of the present application can be represented by formula (VI).

[0145] For example, when When RS=AG=500 ms, H2D=D2H=70 ms, AR=1500 ms, N=40, and T=1 ms, the calculation result sharing time of the existing scheme is 2643 ms, the calculation result sharing time of the model training method provided in the embodiment of the present application is 1785 ms, the specific time saved is 858 ms, and the time saving rate is 32.5%.

[0146] When When RS=AG=500 ms, H2D=D2H=70 ms, AR=1500 ms, N=40, and T=1 ms, the calculation result sharing time of the existing scheme is 2643 ms, the calculation result sharing time of the model training method provided in the embodiment of the present application is 1785 ms, the specific time saved is 858 ms, and the time saving rate is 32.5%.

[0147] The embodiment of the present application also provides a management device 700. The training system in which the device 700 is located also includes a plurality of nodes, each node of the plurality of nodes is deployed with a target model, and a plurality of nodes of the plurality of nodes are located in a first cluster, and at least one node of the plurality of nodes other than the plurality of nodes is located in a second cluster. As shown in FIG. 7, the device 700 includes:

[0148] The division module 710 is configured to divide a training set of the target model into a plurality of training subsets; wherein the plurality of training subsets and the plurality of nodes correspond to each other in one-to-one correspondence.

[0149] The sending module 720 is configured to send each training subset in the plurality of training subsets to the node corresponding to the training subset, so that the node corresponding to the training subset performs calculation of the target model for the training subset to obtain the calculation result of the node corresponding to the training subset.

[0150] The indication module 730 is configured to, in a case where the first node in the first cluster is used to acquire the calculation results of the plurality of nodes and send the acquired calculation results to the at least one node, instruct the first node to send, to the at least one node, the Nth part of the calculation results of the plurality of nodes acquired most recently each time the Nth part of the calculation results of the plurality of nodes is acquired; N is an integer greater than 1.

[0151] The at least one node is configured to update the parameters of the target model deployed in the at least one node by using the calculation results of the plurality of nodes and the calculation result of the at least one node.

[0152] In some embodiments, the N-th part of the computation result of the plurality of nodes comprises an N-th part of the computation result of each of the plurality of nodes; the at least one node is configured to update the target computation result of the at least one node by using the most recently received N-th part of the computation result of the plurality of nodes each time the at least one node receives an N-th part of the computation result of the plurality of nodes; and the target computation result is used to update parameters of a target model deployed in the at least one node after the computation result of the plurality of nodes and the computation result of the at least one node are updated to the target computation result.

[0153] In some embodiments, the sending, by the first node, the N-th part of the computation result of the plurality of nodes that the first node has most recently acquired to the at least one node comprises: sending, by the first node, the N-th part of the computation result of the plurality of nodes that the first node has most recently acquired to a second node in the second cluster; and sending, by the second node, the most recently received N-th part of the computation result of the plurality of nodes to the at least one node each time the second node receives an N-th part of the computation result of the plurality of nodes.

[0154] In some embodiments, the indication module 730 is further configured to set a value of N in order to reduce a time length required for transmission of the computation result of the plurality of nodes from the plurality of nodes to the at least one node.

[0155] In one example of this embodiment, the indication module 730 is further configured to calculate the value of N according to formula (I). RS Minimize: F(x) = T AG (N) + T AR (N) + Max(N*T RS (N) + T AG (N) + T RS (N) + T AR (N) + T AG (N) + T (Ⅰ)

[0156] wherein F(x) represents the time length required for transmission of the computation result of the plurality of nodes from the plurality of nodes to the at least one node, T RS (N) represents a time length required for acquisition, by the first node, of the N-th part of the computation result of the plurality of nodes, T AR (N) represents a time length required for sending, by the first node, the N-th part of the computation result of the plurality of nodes to a second node in the second cluster, T AG (N) represents a time length required for sending, by the second node, the N-th part of the computation result of the plurality of nodes to the at least one node.

[0157] In some embodiments, the first node is configured to obtain, through a first network, one Nth of the computation results of the plurality of nodes, and the first node is configured to send, through a second network, the most recently obtained one Nth of the computation results of the plurality of nodes to the at least one node, wherein a network performance of the first network is greater than a network performance of the second network.

[0158] In some embodiments, the first node is one of the plurality of nodes.

[0159] The dividing module 710, the sending module 720, and the indicating module 730 can be implemented by software or by hardware. For example, the dividing module 710 is taken as an example to introduce the implementation of the dividing module 710. Similarly, the implementation of the sending module 720 and the indicating module 730 can refer to the implementation of the dividing module 710.

[0160] As an example of the software function unit, the dividing module 710 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the dividing module 710 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code can be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers for running the code can be distributed in the same AZ or in different AZs, and each AZ includes one data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0161] Similarly, the multiple hosts / virtual machines / containers for running the code can be distributed in the same VPC or in multiple VPCs. Generally, one VPC is set in one region, and a communication gateway needs to be set in each VPC for cross-zone communication between two VPCs in the same region or between VPCs in different regions, and the interconnection between VPCs is realized through the communication gateway.

[0162] As an example of a hardware functional unit, the division module 710 can include at least one computing device, such as a server or the like. Alternatively, the division module 710 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the like. The PLD can be implemented by a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0163] The plurality of computing devices included in the division module 710 can be distributed in the same region or in different regions. The plurality of computing devices included in the division module 710 can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the division module 710 can be distributed in the same VPC or in multiple VPCs. The plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0164] It should be noted that in other embodiments, the division module 710 can be configured to perform any operation performed by the management apparatus 300 in the method shown in FIG. 6, the sending module 720 can be configured to perform any operation performed by the management apparatus 300 in the method shown in FIG. 6, and the indication module 730 can be configured to perform any operation performed by the management apparatus 300 in the method shown in FIG. 6. The steps or operations performed by the division module 710, the sending module 720, and the indication module 730 can be specified as needed, and the entire function of the management apparatus 700 can be implemented by the division module 710, the sending module 720, and the indication module 730 performing different operations of the management apparatus 300 in the method shown in FIG. 6.

[0165] The present application also provides a computing device 800. As shown in FIG. 8, the computing device 800 includes a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other through the bus 802. The computing device 800 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0166] The bus 802 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, or the like. For ease of representation, only one line is represented in FIG. 8, but it does not mean that there is only one bus or only one type of bus. The bus 802 can include a path for transmitting information between various components (for example, the memory 806, the processor 804, the communication interface 808) of the computing device 800.

[0167] The processor 804 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), or the like.

[0168] The memory 806 can include a volatile memory (for example, a random access memory (RAM)), and can also include a non-volatile memory (for example, a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0169] The memory 806 stores executable program code, and the processor 804 executes the executable program code to respectively implement the functions of the aforementioned division module 710, the sending module 720, and the indication module 730, thereby implementing the operations performed by the management device 300 in the method shown in FIG. 6. That is, the memory 806 stores instructions for executing the operations performed by the management device 300 in the method shown in FIG. 6.

[0170] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, or the like, to implement communication between the computing device 800 and other devices or communication networks.

[0171] The embodiments of the present application also provide a computing device cluster. The computing device cluster comprises at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0172] As shown in FIG. 9, the computing device cluster comprises at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster can store the same instructions for performing the operations performed by the management apparatus 300 in the method shown in FIG. 6.

[0173] In some possible implementations, the memory 806 in one or more computing devices 800 in the computing device cluster can also respectively store instructions for performing part of the operations performed by the management apparatus 300 in the method shown in FIG. 6. In other words, the combination of one or more computing devices 1400 can collectively perform the instructions for performing the operations performed by the management apparatus 300 in the method shown in FIG. 6.

[0174] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster can store different instructions, respectively, for performing part of the functions of the management apparatus 700. That is, the instructions stored in the memory 806 in different computing devices 800 can implement the functions of one or more of the dividing module 710, the sending module 720, and the indicating module 730.

[0175] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 10 shows one possible implementation. As shown in FIG. 10, two computing devices 800A and 800B are connected through a network. Specifically, the computing devices are connected to the network through the communication interfaces in the computing devices. In this type of possible implementation, the memory 806 in the computing device 800A stores instructions for performing the functions of the dividing module 710. Meanwhile, the memory 806 in the computing device 800B stores instructions for performing the functions of the sending module 720 and the indicating module 730.

[0176] It should be understood that the functions of the computing device 800A shown in FIG. 10 can also be completed by multiple computing devices 800. Similarly, the functions of the computing device 800B can also be completed by multiple computing devices 800.

[0177] The embodiments of the present application further provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manner of the computing device cluster described with reference to FIG. 9 and FIG. 10. The difference is that the same instruction for performing the operation performed by the management apparatus 300 in the method shown in FIG. 6 can be stored in the memory 806 of one or more computing devices 800 in the computing device cluster.

[0178] In some possible implementation manners, the instruction for performing the part of the operation performed by the management apparatus 300 in the method shown in FIG. 6 can also be stored in the memory 806 of one or more computing devices 800 in the computing device cluster respectively. In other words, the combination of one or more computing devices 800 can collectively perform the instruction for performing the operation performed by the management apparatus 300 in the method shown in FIG. 6.

[0179] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the operation performed by the management apparatus 300 in the method shown in FIG. 6.

[0180] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium that can be stored by a computing device or a host migration device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk) and the like. The computer readable storage medium contains instructions, which instruct the computing device to perform the operation performed by the management apparatus 300 in the method shown in FIG. 6.

[0181] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that, The application discloses a management device applied to a training system, wherein the training system further comprises a plurality of nodes, each of the plurality of nodes is deployed with a target model, a plurality of nodes in the plurality of nodes are located in a first cluster, and at least one node other than the plurality of nodes in the plurality of nodes is located in a second cluster; the method comprises the following steps: dividing a training set of the target model into a plurality of training subsets; wherein the plurality of training subsets and the plurality of nodes are in one-to-one correspondence; sending each training subset in the plurality of training subsets to a node corresponding to the training subset, so that the node corresponding to the training subset performs calculation of the target model for the training subset to obtain a calculation result of the node corresponding to the training subset; in the case that a first node in the first cluster is used to acquire calculation results of the plurality of nodes and send the acquired calculation results to the at least one node, instructing the first node to send, to the at least one node, one of the calculation results of the plurality of nodes acquired most recently each time when one of the calculation results of the plurality of nodes is acquired; N is an integer greater than 1; wherein the at least one node is used to update parameters of the target model deployed in the at least one node by using the calculation results of the plurality of nodes and the calculation result of the at least one node.

2. The method of claim 1, wherein, one of the calculation results of the plurality of nodes comprises one of the calculation results of each node in the plurality of nodes; the at least one node is used to update the target calculation result of the at least one node by using one of the calculation results of the plurality of nodes received most recently each time when one of the calculation results of the plurality of nodes is received; after the calculation results of the plurality of nodes and the calculation result of the at least one node are updated to the target calculation result, the target calculation result is used to update the parameters of the target model deployed in the at least one node.

3. The method according to claim 1 or 2, characterized in that, the sending of one of the calculation results of the plurality of nodes acquired most recently by the first node to the at least one node comprises: the first node sends one of the calculation results of the plurality of nodes acquired most recently to a second node in the second cluster; the second node sends one of the calculation results of the plurality of nodes received most recently to the at least one node each time when one of the calculation results of the plurality of nodes is received.

4. The method according to any one of claims 1-3, characterized in that, the method further comprises: setting the value of N aiming at reducing the time length required for transmission of the calculation results of the plurality of nodes from the plurality of nodes to the at least one node.

5. The method of claim 4, wherein, the setting of the value of N aiming at reducing the time length required for transmission of the calculation results of the plurality of nodes from the plurality of nodes to the at least one node comprises: calculating the value of N according to formula (I); Minimize: F(x) = T RS (N) + T AG (N) + Max(N*T AR (N), (N-1)*(T RS (N) + T AG (N) ) (I) wherein F(x) represents a time length required for transmitting the calculation results of the plurality of nodes to the at least one node, T RS (N) represents a time length required for the first node to obtain one Nth of the calculation results of the plurality of nodes, T AR (N) represents a time length required for the first node to send one Nth of the calculation results of the plurality of nodes to a second node in the second cluster, T AG (N) represents a time length required for the second node to send one Nth of the calculation results of the plurality of nodes to the at least one node.

6. The method according to any one of claims 1-5, characterized in that, The first node is configured to obtain N-th of the calculation results of the plurality of nodes through a first network, and the first node is configured to send the N-th of the calculation results of the plurality of nodes that is obtained most recently to the at least one node through a second network, wherein a network performance of the first network is greater than a network performance of the second network; and / or, The first node is one of the plurality of nodes.

7. A management device characterized by comprising: The training system in which the device is located further includes a plurality of nodes, each node of the plurality of nodes is deployed with a target model, a plurality of nodes of the plurality of nodes are located in a first cluster, and at least one node of the plurality of nodes other than the plurality of nodes is located in a second cluster; the device comprises: a division module configured to divide a training set of the target model into a plurality of training subsets; wherein the plurality of training subsets and the plurality of nodes correspond to each other one by one; a sending module configured to send each training subset of the plurality of training subsets to a node corresponding to the training subset, so that the node corresponding to the training subset performs calculation of the target model for the training subset to obtain a calculation result of the node corresponding to the training subset; an indication module configured to, in a case where a first node in the first cluster is configured to obtain the calculation results of the plurality of nodes and send the obtained calculation results to the at least one node, instruct the first node to send N-th of the calculation results of the plurality of nodes that is obtained most recently to the at least one node each time N-th of the calculation results of the plurality of nodes is obtained; N is an integer greater than 1; wherein the at least one node is configured to update parameters of the target model deployed in the at least one node by using the calculation results of the plurality of nodes and the calculation result of the at least one node.

8. The apparatus of claim 7, wherein, N-th of the calculation results of the plurality of nodes includes N-th of the calculation result of each node of the plurality of nodes; The at least one node is configured to update the target calculation result of the at least one node by using N-th of the calculation results of the plurality of nodes that is received most recently each time N-th of the calculation results of the plurality of nodes is received; After the calculation results of the plurality of nodes and the calculation result of the at least one node are updated to the target calculation result, the target calculation result is configured to update the parameters of the target model deployed in the at least one node.

9. The apparatus of claim 7 or 8, wherein, The sending of N-th of the calculation results of the plurality of nodes that is obtained most recently by the first node to the at least one node comprises: The first node sends N-th of the calculation results of the plurality of nodes that is obtained most recently to a second node in the second cluster; The second node sends N-th of the calculation results of the plurality of nodes that is received most recently to the at least one node each time N-th of the calculation results of the plurality of nodes is received.

10. The apparatus of any one of claims 7-9, wherein, The indication module is further configured to: set a value of N in order to reduce a time length required for transmission of the calculation results of the plurality of nodes from the plurality of nodes to the at least one node.

11. The apparatus of claim 10, wherein, The indication module is further configured to: calculate the value of N according to formula (I); Minimize: F(x) = T RS (N) + T AG (N) + Max(N*T AR (N), (N-1)*(T RS (N) + T AG (N) ) ) (I) wherein F(x) represents a time length required for transmitting the calculation results of the plurality of nodes to the at least one node, T RS (N) represents a time length required for the first node to obtain one Nth of the calculation results of the plurality of nodes, T AR (N) represents a time length required for the first node to send one Nth of the calculation results of the plurality of nodes to a second node in the second cluster, T AG (N) represents a time length required for the second node to send one Nth of the calculation results of the plurality of nodes to the at least one node. 12.The apparatus of any one of claims 7-11, wherein, the first node is configured to obtain, through a first network, one N-th of the computation results of the plurality of nodes, and the first node is configured to send, through a second network, the most recently obtained one N-th of the computation results of the plurality of nodes to the at least one node, the first network having a network performance greater than a network performance of the second network; and / or, the first node is one of the plurality of nodes.

13. A cluster of computing devices, characterized in that, at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-6.

15. A computer program product comprising instructions, characterized in that, computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-6.

Images