Model training method and device

Abstract

The invention provides a model training method and device. The method comprises the following steps: dividing a training set of a target model into a plurality of training subsets; wherein the plurality of training subsets and the plurality of nodes are in one-to-one correspondence; sending each training subset in the plurality of training subsets to a node corresponding to the training subset, so that the node corresponding to the training subset executes calculation of the target model for the training subset to obtain a gradient of the node corresponding to the training subset; instructing a first node in the first cluster to send the gradient of the first at least one node to the second at least one node by using inter-cluster communication resources of at least two nodes in the first cluster when acquiring the gradient of the first at least one node; wherein the second at least one node is used for updating the parameters of the target model deployed in the second at least one node by using the gradient of the first at least one node and the gradient of the second at least one node. According to the method, the time consumption of gradient sharing among the nodes in different clusters can be reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model training method and apparatus. Background Technology

[0002] As the intelligence and functionality of models increase, the number of model parameters is growing exponentially. Currently, the latest models have hundreds of billions, even trillions, of parameters. This massive amount of parameters makes it difficult to train large-scale models using a single device. The industry employs distributed training techniques to train large-scale models. A single cluster is often insufficient to meet the training needs of large-scale models. Therefore, two or more clusters are required to train the same large-scale model.

[0003] A common distributed training technique is data parallelism (DP). In data parallelism, multiple training nodes perform model computation on different datasets to improve computation speed. Since each training node computes a subset of the model's training set, the gradient calculated by each node is only a partial gradient of the model. However, updating the model parameters requires all gradients, necessitating gradient sharing among different nodes.

[0004] The time required for gradient sharing directly impacts model training efficiency. Generally, training nodes within the same cluster have strong communication capabilities, enabling rapid gradient sharing. However, communication between training nodes in different clusters is weaker, resulting in longer gradient sharing times, which affects model training efficiency. Summary of the Invention

[0005] This application provides a model training method and apparatus that can reduce the time consumption of gradient sharing between nodes in different clusters, thereby improving the training efficiency of the model.

[0006] A first aspect provides a model training method applied to a management device in a training system. The training system further includes multiple nodes, each of which deploys a target model. At least one first node is located within a first cluster, and at least one second node is located within a second cluster. Nodes in the first cluster have inter-cluster communication resources for communicating with nodes in the second cluster. The method includes: dividing a training set of the target model into multiple training subsets; wherein each training subset corresponds one-to-one with a node; sending each training subset to the node corresponding to the training subset, causing the node corresponding to the training subset to perform calculations of the target model for the training subset to obtain the gradient of the node corresponding to the training subset; instructing the first node in the first cluster, upon obtaining the gradient of the first at least one node, to send the gradient of the first at least one node to the second at least one node using the inter-cluster communication resources of at least two nodes in the first cluster; wherein the inter-cluster communication resources of different nodes among the at least two nodes are used to send different data from the gradient of the first at least one node to the second at least one node; and wherein the second at least one node uses the gradient of the first at least one node and the gradient of the second at least one node to update the parameters of the target model deployed in the second at least one node. The first node can be a node among at least one of the first nodes.

[0007] This method allows nodes in the first cluster to send gradients to nodes in the second cluster by utilizing the inter-cluster communication resources of two or more nodes in the first cluster. This enables the use of more inter-cluster communication resources for gradient transmission across clusters, improving the efficiency of gradient transmission and reducing the time required for gradient sharing across clusters, thereby enhancing the training efficiency of the model.

[0008] In one possible implementation, at least two nodes include a first node and a second node; using the inter-cluster communication resources of at least two nodes within the first cluster, sending the gradient of the first at least one node to the second at least one node includes: the first node sending a portion of the gradient data of the first at least one node to the second node, such that the second node sends a portion of the data to the second at least one node through the inter-cluster communication resources of the second node; wherein the first node is used to send the data other than the portion of the gradient data of the first at least one node to the second at least one node through the inter-cluster communication resources of the first node.

[0009] In this implementation, the first node also has inter-cluster communication resources. The first node can not only use the inter-cluster communication resources of other nodes in the first cluster to send gradients to nodes in the second cluster, but also use its own inter-cluster communication resources to send gradients to nodes in the second cluster. In this way, the intra-cluster data transmission overhead in the first cluster can be saved, and the utilization rate of the first node's inter-cluster communication resources can be improved.

[0010] In one possible implementation, the first node and the second node each include a processing unit and a network interface card (NIC). The first node sends a portion of the gradient data from the first at least one node to the second node, including: the processing unit in the first node sending a portion of the data to the processing unit of the second node, such that the processing unit of the second node sends a portion of the data to the NIC of the second node unit; or, the processing unit in the first node sending a portion of the data to the NIC of the second node unit; the second node sends a portion of the data to the second at least one node through the inter-cluster communication resources of the second node, including: the NIC of the second node sending a portion of the data to the second at least one node. The processing unit of the node is used to perform the calculation of the target model to obtain the gradient.

[0011] Within the same cluster, nodes can connect via parametric plane networks or parametric-like networks, which offer higher network performance. Furthermore, nodes within the same cluster can communicate with the network interface cards (NICs) of other nodes using technologies such as remote direct memory access (RDMA). A node's NIC serves as an inter-cluster communication resource, used to send gradients to nodes within other clusters.

[0012] In this implementation, the first node can send the gradient to the processing unit of the second node through its processing unit, and the second node's processing unit will then forward the gradient to its network interface card (NIC). This allows for efficient gradient transmission using parametric surface networks or parametric-like networks. Alternatively, the first node can directly send the gradient to the second node's NIC using techniques such as RDMA, thus saving computational overhead.

[0013] In one possible implementation, the target model includes a first model block and a second model block in sequence, the gradient of the first at least one node is obtained by the first at least one node performing the computation of the first model block, and at least two nodes include nodes for performing the computation of the second model block.

[0014] In this implementation, the target model is trained using a combination of data parallelism and pipelined parallelism. Different nodes within the same cluster can execute different model blocks of the target model. The gradient sharing time for different model blocks is different; in other words, the time spent by the node containing the different model blocks to send gradients to other clusters is different. When the node containing the first model block needs to share the gradient of the first model block, it can call the inter-cluster communication resources of the node containing the second model block. Thus, without configuring additional nodes for the target model, the node containing the first model block can call the inter-cluster communication resources of two or more nodes for gradient cross-cluster transmission. This improves the efficiency of gradient cross-cluster transmission while also increasing the utilization rate of the inter-cluster communication resources of the node containing the second model block.

[0015] In one possible implementation, the method further includes: instructing a first node to receive gradients from a second at least one node using inter-cluster communication resources of at least two nodes; wherein the inter-cluster communication resources of different nodes among the at least two nodes are used to receive different data in the gradients of the second at least one node; the first node is used to send the gradients of the second at least one node to the first at least one node, such that the first at least one node uses the gradients of the second at least one node and the gradients of the first at least one node to update the parameters of the target model deployed in the first at least one node.

[0016] In this implementation, the first node can utilize the inter-cluster communication resources of two or more nodes to receive gradients from nodes in the second cluster, which improves the efficiency of the first node in receiving gradients from nodes in the second cluster and further reduces the time consumption of gradient sharing across clusters, thereby further improving the training efficiency of the target model.

[0017] In one possible implementation, the method further includes: determining the number of nodes in at least two nodes based on at least one of the following: the amount of gradient data of the first at least one node, the availability of inter-cluster communication resources of nodes within the first cluster, and the expected time for gradient transmission. Specifically, the number of nodes in the at least two nodes is positively correlated with the amount of gradient data to be transmitted, negatively correlated with the expected time for gradient transmission, and negatively correlated with the availability of cluster communication resources of nodes within the cluster.

[0018] In this implementation, appropriate nodes and an appropriate number of nodes can be selected based on at least one of the following: the amount of gradient data of at least one node, the amount of inter-cluster communication resources available for nodes in the first cluster, and the expected time for gradient transmission. This allows for cross-cluster gradient sharing by utilizing the inter-cluster communication resources of the selected nodes. This approach ensures the efficiency of cross-cluster gradient sharing while maintaining the performance stability of the training system and minimizing waste of inter-cluster communication resources.

[0019] In one possible implementation, the method further includes: determining the amount of data to be sent by each of the at least two nodes based on the amount of data in the gradient of the first at least one node and the amount of inter-cluster communication resources available for each of the at least two nodes; and instructing the first node to use the inter-cluster communication resources of each of the at least two nodes to send data to the second at least one node that corresponds to the amount of data to be sent by the node.

[0020] The amount of data a node sends is the amount of data that needs to be sent through the node's inter-cluster communication resources. The amount of data a node sends is positively correlated with the amount of gradient data to be sent, and also positively correlated with the availability of the node's inter-cluster communication resources. Thus, this implementation can use available inter-cluster communication resources to send more gradients and use available resources to send fewer gradients, achieving optimized resource utilization.

[0021] Secondly, a management device is provided. The training system in which the management device is located further includes multiple nodes. Each of the multiple nodes deploys a target model. At least one first node among the multiple nodes is located in a first cluster, and at least one second node among the multiple nodes is located in a second cluster. Nodes in the first cluster have inter-cluster communication resources for communicating with nodes in the second cluster. The device includes: a partitioning module for partitioning the training set of the target model into multiple training subsets; wherein the multiple training subsets correspond one-to-one with the multiple nodes; and a sending module for sending each training subset in the multiple training subsets to the node corresponding to the training subset, such that the node corresponding to the training subset is targeted... For the training subset, the target model is computed to obtain the gradient of the node corresponding to the training subset; the instruction module is used to instruct the first node in the first cluster to send the gradient of the first at least one node to the second at least one node when it obtains the gradient of the first at least one node, using the inter-cluster communication resources of at least two nodes in the first cluster; wherein, the inter-cluster communication resources of different nodes among the at least two nodes are used to send different data in the gradient of the first at least one node to the second at least one node; wherein, the second at least one node is used to update the parameters of the target model deployed in the second at least one node using the gradient of the first at least one node and the gradient of the second at least one node.

[0022] In one possible implementation, at least two nodes include a first node and a second node; using the inter-cluster communication resources of at least two nodes within the first cluster, sending the gradient of the first at least one node to the second at least one node includes: the first node sending a portion of the gradient data of the first at least one node to the second node, such that the second node sends a portion of the data to the second at least one node through the inter-cluster communication resources of the second node; wherein the first node is used to send the data other than the portion of the gradient data of the first at least one node to the second at least one node through the inter-cluster communication resources of the first node.

[0023] In one possible implementation, the first node and the second node each include a processing unit and a network interface card (NIC); the first node sends a portion of the gradient data from the first at least one node to the second node, including: the processing unit in the first node sends a portion of the data to the processing unit of the second node, such that the processing unit of the second node sends a portion of the data to the NIC of the second node unit; or, the processing unit in the first node sends a portion of the data to the NIC of the second node unit; the second node sends a portion of the data to the second at least one node through the inter-cluster communication resources of the second node, including: the NIC of the second node sends a portion of the data to the second at least one node.

[0024] In one possible implementation, the target model includes a first model block and a second model block in sequence, the gradient of the first at least one node is obtained by the first at least one node performing the computation of the first model block, and at least two nodes include nodes for performing the computation of the second model block.

[0025] In one possible implementation, the instruction module is further configured to: instruct the first node to receive the gradient of the second at least one node using the inter-cluster communication resources of at least two nodes; wherein the inter-cluster communication resources of different nodes among the at least two nodes are used to receive different data in the gradient of the second at least one node; the first node is configured to send the gradient of the second at least one node to the first at least one node, such that the first at least one node uses the gradient of the second at least one node and the gradient of the first at least one node to update the parameters of the target model deployed in the first at least one node.

[0026] In one possible implementation, the instruction module is further configured to: determine the number of nodes in at least two nodes based on at least one of the following: the amount of gradient data of the first at least one node, the amount of inter-cluster communication resources available for nodes in the first cluster, and the expected time of gradient transmission.

[0027] In one possible implementation, the instruction module is further configured to: determine the amount of data to be sent by each of the at least two nodes based on the amount of data in the gradient of the first at least one node and the amount of inter-cluster communication resources available for each of the at least two nodes; and instruct the first node to use the inter-cluster communication resources of each of the at least two nodes to send data to the second at least one node that corresponds to the amount of data to be sent by the node.

[0028] Fourthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a cluster of computing devices, execute the method provided in the first aspect.

[0029] Fifthly, a computer program product containing instructions is provided, which, when executed by a cluster of computer devices, causes the cluster of computer devices to perform the method provided in the first aspect.

[0030] The beneficial effects of the second to fifth aspects can be referred to the introduction of the beneficial effects of the first aspect above, and will not be repeated here. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of gradient sharing between clusters;

[0032] Figure 2 This is a schematic diagram of a training system provided in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of a training system provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of the structure of a management device provided in an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of gradient allocation provided in an embodiment of this application;

[0036] Figure 6 This is a flowchart of a model training method provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of a gradient transmission method provided in an embodiment of this application;

[0038] Figure 8 This is a schematic diagram of gradient sharing provided in an embodiment of this application;

[0039] Figure 9 This is a schematic diagram of the structure of a management device provided in an embodiment of this application;

[0040] Figure 10This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0041] Figure 11 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0042] Figure 12 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0043] The solutions provided in the embodiments of this application will now be described with reference to the accompanying drawings. In the embodiments of this application, "multiple" refers to two or more. Terms such as "first," "second," etc., are merely used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.

[0044] To facilitate understanding of the solutions provided in the embodiments of this application, the technical terms that may be involved in the embodiments of this application will be introduced first.

[0045] Model: Composed of multiple artificial neural network layers. Typically, these layers are arranged sequentially, with the output of one layer serving as the input to the next. A neural network layer can be simply referred to as a network layer or layer.

[0046] Model training refers to the process of updating the parameters of a model. Typically, model parameters are preset or randomly generated. By updating these parameters, the model learns the ability to handle corresponding tasks, such as image recognition, speech recognition, natural language translation, computer game playing, and content generation. Model training requires multiple iterations until convergence or a predetermined number of training iterations are reached. Each iteration requires a certain amount of data as input. The model performs forward computation on the input dataset to obtain a loss function. Then, the model performs backward computation on the loss function to obtain the gradient. Finally, the gradient is used to update the model parameters.

[0047] Forward propagation, also known as forward pass, refers to the process of calculating and passing data layer by layer through the model, starting from the data input, to obtain the model's output. The purpose of forward propagation is to calculate the model's output.

[0048] Model computation refers to the calculations performed on a relevant dataset using the model's parameters. Model computation can be divided into forward computation and backward computation.

[0049] Calculation results refer to the results obtained by using the model's parameters on a relevant dataset. The results of the forward computation include the loss function, while the results of the backward computation include the gradient.

[0050] Forward computation refers to the computation performed on layers or blocks in the model during forward propagation, which is used to obtain the model's loss function.

[0051] Back propagation, also known as reverse propagation, refers to the process of obtaining a loss function based on the difference between the model's output and the labels. Then, using the loss function, the gradient is calculated layer by layer in reverse using the chain rule. Through back propagation, the gradient of each parameter with respect to the loss function can be calculated, and these gradients are used to update the parameters.

[0052] Backward computation refers to the computation performed on layers or blocks of the model during backward propagation. Backward computation is used to obtain gradients based on the loss function obtained from forward computation, and these gradients are then used to update the model parameters.

[0053] Gradient: The gradient refers to the direction in which the loss function value decreases the fastest. It is obtained through backward computation and is a result of that computation. The gradient is actually a vector representing the direction; each element in the gradient represents the rate of change of the loss function value relative to the weights at that position. The gradient is used to update the model's parameters. Specifically, the gradient is multiplied by the learning step size to obtain a product, which is then added to (or subtracted from) the model's parameters to obtain the updated parameters.

[0054] The gradient of a node refers to the gradient obtained by a node when performing calculations on the target model with respect to the corresponding dataset.

[0055] Activation values ​​refer to the outputs of network layers or model blocks. The activation values ​​of a network layer or model block are used for computational calculations in the next network layer or model block, generating its activation values. Furthermore, during backpropagation, the activation values ​​of a network layer or model block are used for computational calculations in the next network layer or model block, generating its activation values ​​and gradients.

[0056] Gradient synchronization is a technique used in distributed model training (such as data-parallel training) to ensure that the model parameters in each training node remain consistent after each training iteration. In data-parallel training, each training node computes the gradient of a small piece of data. Then, gradient synchronization aggregates the gradients computed by all training nodes (e.g., by averaging or summing), and finally, the aggregated gradients are used to update the model parameters.

[0057] Data parallelism (DP) is a distributed model training technique that achieves distributed model training through a parallel computing paradigm. In data parallelism, each of multiple training nodes deploys a model, and different training nodes perform model computations on different datasets, thus breaking down the model computation task into multiple smaller, independent, and parallel subtasks. In data parallelism, the gradient calculated by each training node is a partial gradient of the model. Gradient synchronization is performed across multiple training nodes to aggregate the gradients calculated by all nodes. The aggregated gradients are used to update the model parameters, thereby ensuring the consistency of model parameters across all training nodes.

[0058] Pipeline parallelism (PP) is a distributed model training technique where different nodes run different layers of the model, representing inter-layer parallelism. In other words, pipeline parallelism sequentially divides the model at the layer level, resulting in multiple interconnected model blocks. During model training, the computations related to different model blocks are performed by different nodes. A single model block can be considered a stage in the model training process, and N model blocks represent N stages in the model training process.

[0059] A model block is a neural network block obtained by splitting the model between layers. A model block can include one or more neural network layers.

[0060] Stage: A model block in pipelined parallel technology.

[0061] Region: A unit of deployment for compute nodes in infrastructure, defined by geographical location and network latency. Regions have a high level of isolation. Within the same region, compute nodes share common services such as elastic computing, block storage, object storage, VPC networks, elastic public IPs, and images.

[0062] Availability Zone (AZ): A unit of deployment for computing nodes in an infrastructure. An AZ is a collection of one or more physical data centers with its own independent power, cooling, and water supply. An AZ includes one or more geographically proximate data centers. A region can include multiple AZs.

[0063] Training node: Often simply called a node, this refers to a computing node used to perform model training or computation. A node can include at least one processing unit and a network interface card (NIC). The processing unit can be a neural processing unit (NPU), graphics processing unit (GPU), tensor processing unit (TPU), etc. The NIC is used for communication between the node and other nodes in the cluster. Common node types include hosts and virtual machines (VMs). Hosts can be physical computing devices such as servers.

[0064] Data center network (DCN) is a network connecting clusters, used to link nodes within multiple clusters together. DCNs are scalable and efficient, capable of connecting tens or even hundreds of thousands of nodes to meet the ever-growing connectivity demands of cloud computing.

[0065] Parametric plane networks (PPNs) are high-speed networks specifically designed for distributed training. Nodes within the same cluster can be connected via PPNs. Specifically, when a node is a host, the processing units of two hosts within the same cluster can communicate via PPNs without going through the host's network interface card (NIC). Typically, PPNs can achieve communication speeds of up to 200 GE.

[0066] A cluster, also known as a computing cluster or training cluster, is a group of multiple nodes. Nodes within the same cluster are located in close proximity, such as within the same region or Availability Zone (AZ). Communication between nodes within the same cluster is strong; that is, the communication capability between nodes within the same cluster is greater than the communication capability between nodes within the cluster and nodes outside the cluster. For example, the network between nodes within the same cluster is a parametric plane network or a parametric-plane-like network, while the network between nodes within the cluster and training nodes outside the cluster is a DCN or a DCN-like network. However, in this embodiment, nodes within the same region or AZ are not necessarily in the same cluster. This is because some nodes within the same region or AZ may not use a parametric plane network or a parametric-plane-like network, resulting in poor communication capability between these nodes and other nodes in the same region or AZ. Therefore, these nodes can be considered not to be in the same cluster as other nodes in the same region or AZ, but rather to be in different clusters.

[0067] Inter-cluster communication resources refer to the resources a node uses to communicate with nodes within another cluster. These resources include network interface cards (NICs), CPUs, and memory. Specifically, with the cooperation of the CPU and memory, the NIC sends data from one node to nodes in another cluster and receives data from nodes in another cluster.

[0068] A neural processing unit (NPU) is a hardware processor specifically designed to accelerate neural network computations. Compared to a central processing unit (CPU) and a GPU, the NPU optimizes computations such as matrix multiplication and convolution operations in neural networks, resulting in higher computational efficiency and energy efficiency. NPUs can be applied to artificial intelligence and deep learning tasks, such as image recognition, natural language processing, and speech recognition.

[0069] All Reduce is an algorithm used in data parallelism for sharing gradients between clusters, allowing the gradients of one cluster to be shared with another. "All" refers to all nodes involved in model training, and "Reduce" refers to the aggregation operation.

[0070] Scattering is an algorithm in data parallelism that shares gradients within a cluster, used to send the gradients of one node to another within the same cluster. Specifically, a cluster of N nodes participates in model training. Each of these N nodes divides its computed gradient into N equal-sized data blocks (buffers). These N data blocks correspond to different rank indices, and each of the N data blocks corresponds one-to-one with one of the N nodes. Each of the N nodes sends its corresponding data blocks to the other N nodes according to their rank indices. In other words, each of the N nodes receives the same data blocks as the N nodes, and the rank indices of the data blocks received by each node correspond to the rank indices of that node.

[0071] Gathering is a gradient aggregation method used in data parallelism. The gathering process involves a node participating in model training acquiring the gradients of all other nodes participating in model training, and then aggregating these gradients with its own gradient, for example, by summing or averaging.

[0072] As the number of model parameters continues to increase, the size of the training set also expands, with hundreds of terabytes (TB) or even petabytes (PB) of training sets becoming commonplace. This places enormous demands on computing resources, power, and cooling capabilities. Clusters located in a single geographical location are no longer sufficient to meet these needs. The industry is seeking to utilize clusters distributed across different geographical locations for model training. Related technologies include... Figure 1 As shown, after node A1 in cluster A obtains the gradients of the nodes participating in model training in cluster A, node A1 uses its inter-cluster communication resources to send the gradients of the nodes participating in model training in cluster A to node B1 in cluster B. Due to distance limitations, high-speed networks such as high-speed Ethernet and parametric surface networks are difficult to use to connect nodes in different clusters. The inter-cluster communication resources of nodes are weak, and gradients are difficult to transmit efficiently between different clusters. This makes gradient sharing between clusters a bottleneck for model training efficiency.

[0073] This application provides a model training method. In this method, when node T1 within the local cluster, which is used to send gradients to the target cluster, obtains the gradients of the nodes in its local cluster participating in the training of the target model, it can utilize the inter-cluster communication resources of at least two nodes within its local cluster to send the gradients of the nodes in its local cluster participating in the training of the target model to the nodes in the target cluster. In this way, node T1 can utilize more inter-cluster communication resources to send gradients to the nodes in the target cluster, improving the parallelism of gradient transmission between clusters, thereby enhancing the gradient transmission capability between clusters. This allows gradients to be efficiently transmitted from the local cluster to the target cluster, shortening the time spent sharing gradients between clusters, and ultimately improving the training efficiency of the target model.

[0074] Next, the model training method provided in the embodiments of this application will be described in detail.

[0075] Figure 2 A training system for implementing this method is illustrated. The training system includes a management device 100 and multiple clusters, such as cluster 200, cluster 300, etc. The management device 100 can be any device, software, equipment, platform, or cluster with data processing capabilities. In some embodiments, the management device 100 can also be software deployed on one or more nodes within any of the multiple clusters. Each of the multiple clusters can include multiple nodes; for example, cluster 200 includes nodes 210, 220, etc., and cluster 300 includes nodes 310, 320, etc.

[0076] Each node in cluster 200 has inter-cluster communication resources to communicate with nodes in cluster 300, and each node in cluster 300 has inter-cluster communication resources to communicate with nodes in cluster 200.

[0077] like Figure 2 As shown, a node may include at least one processing unit and a network interface card (NIC). Processing units of different nodes within the same cluster are interconnected, allowing direct communication between them. In some embodiments, processing units of different nodes within the same cluster can be connected via a parametric plane network or a parametric-like plane network. The NIC of a node is used to connect to NICs of nodes in other clusters, enabling communication between nodes in different clusters. In some embodiments, such as... Figure 3 As shown, the network cards of nodes in different clusters can be connected using DCN networks or DCN-like networks.

[0078] The management device 100 can receive a training task for a target model, which is used to train the target model in a data-parallel manner. The management device 100 can select nodes within the multiple clusters for training the target model. It can be configured that the management device 100 selects multiple nodes within the multiple clusters for training the model, with at least one node located in cluster 200 and at least another node located in cluster 300. In this case, gradient sharing is required between cluster 200 and cluster 300.

[0079] The management device 100 can control gradient sharing between cluster 200 and cluster 300 by executing the model training method provided in the embodiments of this application, so as to improve the efficiency of gradient sharing between cluster 200 and cluster 300.

[0080] In some embodiments, in order to execute the model training method provided in the embodiments of this application, such as Figure 4 As shown, the management device 100 may include a decision module 110, an allocation module 120, and a control module 130.

[0081] The decision module 110 can obtain the amount of gradient data to be sent within the cluster, the expected time for gradient transmission, and the available cluster communication resources of the nodes within the cluster. The expected time can be set by the user. Based on the amount of gradient data to be sent within the cluster, the expected time for gradient transmission, and the available cluster communication resources of the nodes within the cluster, it determines the number of nodes used for sending gradients and which nodes within the cluster are used for sending gradients.

[0082] The number of nodes used to send gradients is positively correlated with the amount of gradient data to be sent, the number of nodes used to send gradients is negatively correlated with the expected time taken to send gradients, and the number of nodes used to send gradients is negatively correlated with the availability of cluster communication resources for nodes within the cluster.

[0083] When the amount of gradient data to be sent is large, or the expected time for gradient transmission is short, or the available amount of cluster communication resources for each node in the cluster is small, the decision module 110 can determine to use a larger number of nodes to send gradients, thereby ensuring efficient gradient transmission and avoiding excessive load on inter-cluster communication resources of nodes, thus ensuring the performance stability of the training system.

[0084] When the amount of gradient data to be sent is small, or the expected time for gradient transmission is long, or the available cluster communication resources of each node in the cluster are large, the decision module 110 can determine to use a smaller number of nodes to send gradients in order to avoid investing too much inter-cluster communication resources.

[0085] When a node used to send gradients fails or becomes overloaded, the decision module 110 can adjust the node used to send gradients or add a node used to send gradients to ensure timely transmission of gradients.

[0086] The allocation module 120 allocates a transmission data amount to a node based on the amount of gradient data to be transmitted and the availability of inter-cluster communication resources for transmitting the gradient. When the amount of gradient data to be transmitted is fixed, the availability of inter-cluster communication resources for a node is positively correlated with the amount of transmission data allocated to that node. For example, such as... Figure 5 As shown, the inter-cluster communication resources of nodes 210 and 220 within cluster 200 can be configured to send a gradient of 100MB to nodes within cluster 300. The available amount of inter-cluster communication resources can be specifically configured as the available bandwidth of the network interface card (NIC), where the available bandwidth of the NIC for node 210 is 8Gbps and the available bandwidth of the NIC for node 220 is 2Gbps. Therefore, the allocation module 120 allocates 80MB of data to node 210 and 20MB of data to node 220. In other words, 80MB of the gradient is transmitted using the inter-cluster communication resources of node 210, and 20MB is transmitted using the inter-cluster communication resources of node 220, thereby optimizing resource utilization.

[0087] The control module 130 can be used to ensure the correct order and integrity of data during gradient transmission. After receiving the gradient sent by its own cluster, the nodes in the target cluster summarize the received gradient and the gradients of the nodes in the target cluster (e.g., sum or average). To ensure the correct summation or averaging of gradients from nodes in different clusters, the correct order of gradient data must be guaranteed. Integrity means that the gradient is successfully sent to the nodes in the target cluster without data loss. In some embodiments, after confirming that the gradients of its own cluster and the target cluster have been scattered, the control module 130 controls the nodes in its own cluster to send gradients to the nodes in the target cluster, so that the nodes in the target cluster can summarize the gradients of their own cluster and the target cluster after receiving the gradients of their own cluster.

[0088] The control module 130 can also be used to manage transmission paths between nodes in different clusters. Typically, there may be multiple transmission paths between nodes in different clusters. Through transmission path management, the control module 130 can select a superior path from these multiple paths to use as the path for nodes in its own cluster to send gradients to nodes in a target cluster. The superior path may be the path with the fewest hops, or an idle path, etc.

[0089] The control module 130 can also be used to handle gradient transmission anomalies. For example, when data loss occurs during gradient transmission between clusters, the control module 130 can instruct the retransmission of the lost data. As another example, when the gradient transmission latency is relatively high (e.g., greater than a preset threshold), the control module 130 can adjust the gradient transmission path or the node sending the gradient to reduce the gradient transmission latency.

[0090] In this way, the control module 130 can ensure the accurate and efficient transmission of gradients between clusters, thereby ensuring the smooth training of the target model.

[0091] The foregoing example illustrates the functions of the training system and management device 100 provided in this application embodiment. Next, based on the foregoing description and taking implementation within this training system as an example, the flow of the model training method provided in this application embodiment will be described.

[0092] Multiple nodes in the training system can be configured to train the target model in a data-parallel manner, and the target model can be deployed to each of these nodes. In some embodiments, the target model can be trained using both data-parallel and pipelined parallel methods. The target model can be divided into multiple model blocks, such as model block C1, model block C2, etc. The multiple nodes are used to train model block C1, and model block C1 is deployed to each of these nodes. In some embodiments, pipelined parallel training of the target model may not be used; instead, each of the multiple nodes may deploy the complete target model.

[0093] Deploying a model or model block to a node can refer to storing the parameters of the model or model block in the node or its accessible storage space, enabling the node to load the parameters and then perform calculations based on those parameters. These calculations include forward and backward computations.

[0094] At least one of the multiple nodes used to train the target model or model block is deployed in cluster 200, and at least another node is deployed in cluster 300. For ease of description, the at least one node deployed in cluster 200 is referred to as the first at least one node, and the at least one node deployed in cluster 300 is referred to as the second at least one node. For example, the first at least one node may include nodes 210 and 220, and the second at least one node may include nodes 310 and 320.

[0095] like Figure 6 As shown, the method includes the following steps.

[0096] In step 601, the management device 100 can divide the training set of the target model into multiple training subsets. Each of these multiple training subsets corresponds one-to-one with the multiple nodes used to train the target model.

[0097] The aforementioned multiple nodes are used to train the target model in a data-parallel manner. The management device 100 divides the training set into multiple training subsets according to the number of nodes among these multiple nodes. Each node corresponds to one training subset, meaning one training subset corresponds to one node, and different training subsets correspond to different nodes.

[0098] In step 602, the management device 100 sends each training subset in the plurality of training subsets to the node corresponding to the training subset, so that the node corresponding to the training subset performs the calculation of the target model for the training subset to obtain the gradient D1 of the node corresponding to the training subset.

[0099] Step 602 includes steps 602a and 602b. As described above, a first at least one node among the plurality of nodes used to train the target model is deployed within cluster 200. In step 602b, the management device 100 sends at least one training subset corresponding to the first at least one node to cluster 200, thereby sending the at least one training subset to the first at least one node. A second at least one node among the plurality of nodes used to train the target model is deployed within cluster 200. In step 602b, the management device 100 sends at least one training subset corresponding to the second at least one node to cluster 200, thereby sending the at least one training subset to the second at least one node.

[0100] Nodes receiving the corresponding training subset can execute step 701, performing target model computation on the received training subset to obtain gradient D1. As described above, the nodes used to train the target model are equipped with the target model. When a node receives a dataset (i.e., the training subset corresponding to that node), it can perform forward computation of the target model on that training subset to obtain a loss function or activation values. Then, the node performs backward computation of the target model on the loss function or activation values ​​calculated by other nodes to obtain the gradient D1 of that node.

[0101] In step 603, when node T1 in cluster 200 obtains the gradient D1 of the first at least one node, it is instructed to use the inter-cluster communication resources of at least two nodes T in cluster 200 to send the gradient D1 of the first at least one node to the second at least one node.

[0102] like Figure 6 As shown, each of the first at least one nodes can send the gradient D1 calculated in step 701 to node T1 through step 702, so that node T1 can obtain the gradient D1 of each of the first at least one nodes.

[0103] Node T1 can be any node in cluster 200, and this node can communicate with other nodes in cluster 200 (e.g., the first at least one node). In some embodiments, node T1 is also used to train the target model, i.e., node T1 is any node among the first at least one nodes.

[0104] In some embodiments, each node in the first at least one node can simultaneously act as node T1, where different nodes T1 are used to obtain different data in the gradient D1 of the node. For example, the plurality of nodes can be specifically set to M nodes, where M is an integer greater than 1. Each of the M nodes is a node T1, i.e., there are M nodes T1. Each of the M nodes divides its calculated gradient D1 into M data blocks of the same size. Different data blocks in the M data blocks correspond to different sequences, and the sequences of the M data blocks correspond one-to-one with the M nodes T1. Each of the M nodes sends the corresponding data blocks to the nodes T1 in the M nodes T1 according to the sequence of the data blocks. That is, each node T1 in the M nodes T1 is used to obtain data blocks with the same sequence in the gradient D1 of the M nodes, and the sequence of data blocks used by each node T1 to obtain corresponds to that node T1. Thus, each of the M nodes T1 is used to obtain a portion of the gradient D1 of all the nodes in the M nodes. This portion of the gradient D1 of all the nodes is the gradient D1 of several nodes that node T1 wants to obtain.

[0105] Furthermore, in the following description, the operation performed by one of the M nodes T1 is used as an example. The operations performed by other nodes T1 can be implemented by referring to this example, and will not be described in detail here.

[0106] In some embodiments, node T1 may employ the Reduce Scatter algorithm to obtain the gradient D1 of at least one first node.

[0107] When node T1 obtains the gradient D1 of the first at least one node, it can use the inter-cluster communication resources of at least two nodes in cluster 200 to send the gradient D1 obtained by node T1 of the first at least one node to the second at least one node.

[0108] In some embodiments, node T1 can summarize the gradient D1 obtained from the first at least one node, and then send the summarized gradient to the second at least one node through the inter-cluster communication resources of the at least two nodes T. The summarization can be either a summation or an averaging.

[0109] In some embodiments, the management device 100 can further determine the number of nodes in the at least two nodes T based on at least one of the following: the amount of data of the first at least one node's gradient D1 obtained by node T1, the availability of inter-cluster communication resources of nodes within the cluster 200, and the expected time for gradient D1 transmission. For example, the management device 100 can also determine the specific nodes in the at least two nodes T based on at least one of the following: the amount of data of the first at least one node's gradient D1 obtained by node T1, the availability of inter-cluster communication resources of nodes within the cluster 200, and the expected time for gradient D1 transmission. Specific implementation details can be found in the above description of the decision module 110, and will not be repeated here.

[0110] The inter-cluster communication resources of the nodes in at least two nodes T are respectively used by the second at least one node to send different data in the gradient D1 of the first at least one node. For example, such as Figure 6 As shown, the at least two nodes T may include node T1 and node T2. Node T2's inter-cluster communication resources are used to send data D11 from the gradient D1 of the first at least one node to the second at least one node. Node T1's inter-cluster communication resources are used to send data D12 from the gradient D1 of the first at least one node to the second at least one node. Data D11 is a portion of the gradient D1 from the first at least one node obtained by node T1, and data D12 is another portion of the gradient D1 from the first at least one node obtained by node T1.

[0111] In some embodiments, such as Figure 6 As shown, in step 703, node T1 can send data D11 to node T2. Then, in step 704, node T2 uses its inter-cluster communication resources to send data D11 to each of the second at least one node. In step 704, node T1 uses its inter-cluster communication resources to send data D12 to each of the second at least one node.

[0112] As described above, the nodes within cluster 200 have processing units and network interface cards (NICs); that is, node T1 and node T2 each include a processing unit and a NIC. The NICs of a node are part of the inter-cluster communication resources.

[0113] In one example, in step 703, the processing unit of node T1 can send data D11 to the processing unit of node T2. As described above, the processing units of different nodes within cluster 200 are connected via a parametric plane network or a parametric-like plane network, resulting in high communication performance. In step 703, transmitting data D11 through the network between the processing units of node T1 and node T2 improves the transmission efficiency of data D11. Furthermore, after receiving data D11 from the processing unit of node T1, the processing unit of node T2 can send data D11 to its network interface card (NIC). Then, the NIC of node T2 sends data D11 to at least one second node.

[0114] In another example, the processing unit of node T1 can send data D11 to the network interface card (NIC) of node T2; that is, the processing unit of node T1 directly sends data D11 to the NIC of node T2. In one example, the processing unit of node T1 can use RDMA technology to directly send data D11 to the NIC of node T2. In another example, the processing unit of node T1 and the NIC of node T2 can be interconnected via a UB (unified bus), and the processing unit of node T1 directly sends data D11 to the NIC of node T2 via the UB bus.

[0115] In some embodiments, node T1 can be a virtual machine, and correspondingly, the network interface card (NIC) of node T1 is a virtual NIC. More network resources, such as bandwidth, can be allocated to the NIC of node T1 to improve its communication capabilities, thereby improving the efficiency of node T1 in sending gradient D1 to at least one second node through the NIC.

[0116] In some embodiments, the management device 100 can determine the amount of data to be sent by each of the at least two nodes T based on the amount of data in gradient D1 of the first at least one node obtained by node T1 and the availability of inter-cluster communication resources for each of the at least two nodes T. Here, the amount of data to be sent by a node refers to the amount of data that the node needs to send. The amount of data to be sent by a node is positively correlated with the availability of inter-cluster communication resources for that node. Then, the management device 100 instructs node T1 to use the inter-cluster communication resources of each of the at least two nodes T to send data corresponding to the amount of data to be sent by the node, wherein the data is the data in gradient D1 of the first at least one node obtained by node T1. For details, please refer to the above description of the allocation module 120, which will not be repeated here.

[0117] In some embodiments, in addition to training the target model using data parallelism, a pipelined parallelism approach can also be used to train the target model. In pipelined parallelism, the target model is divided into multiple model blocks, and different model blocks are trained by different nodes. Since these multiple model blocks are trained serially, the gradient sharing time for different model blocks varies. Therefore, when gradient sharing is required for the current model block, the inter-cluster communication resources of nodes in other model blocks can be used to send the gradient of the current model block. This achieves inter-cluster communication resource sharing among nodes of different model blocks, improving the efficiency of gradient sharing and the utilization rate of inter-cluster communication resources.

[0118] For example, as described above, the target model can be divided into model block C1 and model block C2. A first at least one node is used to train model block C1, and another at least one node within cluster 200 is used to train model block C2. Here, gradient D1 is the gradient used to update the parameters of model block C1, and can be simply referred to as the gradient of model block C1. Alternatively, for ease of description, the at least one node within cluster 200 used to train model block D2 can be referred to as the third at least one node. The third at least one node is used to perform the calculation of model block C2, obtaining gradient D2. Gradient D2 is the gradient used to update the parameters of model block C2, and can be simply referred to as the gradient of model block C2.

[0119] The at least two nodes T may include at least a third node. This not only improves the efficiency of gradient sharing but also enhances the utilization of inter-cluster communication resources of the at least third node.

[0120] Specifically, it can be as follows: Figure 7 As shown. We can define the first at least one node as node T1, and the third at least one node as node T2. We can also define that, in the forward propagation direction of the target model, model block C2 precedes model block C1. Correspondingly, in the backward propagation direction of the target model, model block C1 precedes model block C2. That is, in the same training iteration, the gradient of model block C1 (i.e., gradient D1) begins to be shared first, and then the gradient of model block C2 (i.e., gradient D2) begins to be shared. More specifically, as... Figure 7 As shown, during forward propagation, node T2 performs the forward computation of model block C2 and passes the resulting activation value to node T1. Node T1 then performs the forward computation of model block C1 based on this activation value. During backward propagation, node T1 performs the backward computation of model block C1, obtaining the gradient D1 and the activation value. Node T1 then passes this activation value to node T2. Node T2 then performs the backward computation of model block C2 based on this activation value, obtaining the gradient D2.

[0121] Node T1 can be configured to complete the backward computation of model block C1 at time t0 and begin sharing gradient D1. Node T2 can complete the backward computation of model block C2 at time t1 and begin sharing gradient D2. During the time interval between time t0 and time t1, the inter-cluster communication resources of node T2 are idle. During this time interval, node T1 can use node T2's inter-cluster communication resources to send gradient D12 to at least a second node, and node T1 can use its own inter-cluster communication resources to send gradient D11 to at least a second node, thus efficiently sending gradient D1 to at least a second node.

[0122] When gradient D2 sharing begins, data D11 has already been sent, and node T1's inter-cluster communication resources are idle. Node T2 can use node T1's inter-cluster communication resources to send data D21 to at least a second node, and node T2 can use its own inter-cluster communication resources to send data D22 to at least a second node. Here, data D21 and data D22 are different data from gradient D2. In this way, gradient D2 can be efficiently sent to at least a second node.

[0123] In some embodiments, the target model may not be trained in parallel using streaming, and the complete target model may be deployed in the first at least one node and the second at least one node. In this case, node T2 is a node other than the first at least one node, and gradient D1 is used to update the parameters of the target model.

[0124] In this way, the gradient D1 of the first at least one node can be efficiently sent to the second at least one node.

[0125] Continue reading Figure 6 In step 705, each node in the second at least one node can update the parameters of the target model deployed in the second at least one node using the gradient D1 of the first at least one node and the gradient D1 of the second at least one node. The gradient D1 of the second at least one node is the gradient obtained by the second at least one node performing the calculation of the target model for the corresponding training subset.

[0126] In some embodiments, when gradient D1 is the gradient of model block C1, in step 705, the parameters of the model block deployed in the second at least one node are updated using the gradient D1 of the first at least one node and the gradient D1 of the second at least one node.

[0127] When the second at least one node consists of two or more nodes, each of these two or more nodes receives the gradient D1 from the first at least one node. Furthermore, these two or more nodes can obtain each other's gradient D1 through the network within the cluster (e.g., a parametric surface network). In other words, each of these two or more nodes can obtain the gradient D1 from the other nodes within the cluster through the network within the cluster. Thus, each of these two or more nodes can obtain the gradient D1 from the other two or more nodes.

[0128] Referring to the method of sending the gradient D1 of at least one first node to at least one second node, the gradient D1 of at least one second node can also be sent to at least one first node. The management device 100 can instruct node T1 to utilize the inter-cluster communication resources of the at least two nodes T to receive the gradient D1 of the at least one second node. The inter-cluster communication resources of different nodes in the at least two nodes T are used to receive different data in the gradient D1 of the at least one second node. This allows node T1 to efficiently receive the gradient D1 of the at least one second node.

[0129] When node T1 receives gradient D1 from at least a second node, it can send gradient D1 from at least a second node to at least a first node. At least a first node can then use the gradient D1 from both the second and first nodes to update the parameters of the target model deployed within it. When there are two or more nodes, node T1 sends the gradient D1 from the second node to each of the two or more nodes. These nodes can obtain each other's gradient D1 through a network within the cluster (e.g., a parametric network). In other words, each of the two or more nodes can obtain the gradient D1 from the other nodes within the cluster through the network. Thus, each of the two or more nodes can obtain the gradient D1 from the other two or more nodes. Then, each of the two or more nodes can use the gradient D1 from both the second and first nodes to update the parameters of the target model deployed within that node.

[0130] The overall process of the model training method provided in the embodiments of this application has been described above. Next, in a specific embodiment, the gradient transfer process under the influence of this method is illustrated.

[0131] See Figure 8You can configure nodes in clusters 200, 300, and 400 to be used for training the target model. You can configure the gradients calculated by the nodes in cluster 200 as "1" and "2", the gradients calculated by the nodes in cluster 300 as "3" and "4", and the gradients calculated by the nodes in cluster 400 as "5" and "6".

[0132] First, node T1 in cluster 200 can obtain "1" and "2", node T3 in cluster 300 can obtain "3" and "4", and node T5 in cluster 400 can obtain "5" and "6". Here, "1", "3", and "5" correspond, and "2", "4", and "6" correspond. Nodes T1, T3, and T5 are used to train the target model, while nodes T2, T4, and T6 are not.

[0133] Next, node T1 sends "2" to node T2 in cluster 200, node T3 sends "4" to node T4 in cluster 300, and node T5 sends "6" to node T6 in cluster 400.

[0134] Next, node T1 can send "1" to nodes T3 and T5, node T2 can send "2" to nodes T4 and T6, node T3 can send "3" to nodes T1 and T5, node T4 can send "4" to nodes T2 and T6, node T5 can send "5" to nodes T1 and T3, and node T6 can send "6" to nodes T2 and T4. Furthermore, each node adds the received gradient to its own recorded gradient. The sum of "1", "3", and "5" is "9". The sum of "2", "4", and "6" is "12".

[0135] Then, node T2 sends "12" to node T1, enabling node T1 to update the parameters of the target model deployed in node T1 based on "12" and "9". Node T4 sends "12" to node T3, enabling node T3 to update the parameters of the target model deployed in node T1 based on "12" and "9". Node T6 sends "12" to node T5, enabling node T5 to update the parameters of the target model deployed in node T1 based on "12" and "9".

[0136] In this way, gradients can be transmitted between clusters through inter-cluster communication resources of two or more nodes, improving the efficiency of gradient sharing and thus improving the training efficiency of the model.

[0137] This application embodiment also provides a management device 900. The training system in which the management device 900 is located further includes multiple nodes, each of which deploys a target model. At least one first node among the multiple nodes is located within a first cluster, and at least one second node among the multiple nodes is located within a second cluster. Nodes in the first cluster have inter-cluster communication resources for communicating with nodes in the second cluster. Figure 9 As shown, the management device 900 includes:

[0138] The partitioning module 910 is used to divide the training set of the target model into multiple training subsets; wherein, the multiple training subsets correspond one-to-one with the multiple nodes;

[0139] The sending module 920 is used to send each training subset in the plurality of training subsets to the node corresponding to the training subset, so that the node corresponding to the training subset performs the calculation of the target model for the training subset to obtain the gradient of the node corresponding to the training subset.

[0140] The instruction module 930 is used to instruct the first node in the first cluster to send the gradient of the first at least one node to the second at least one node when it obtains the gradient of the first at least one node; wherein, the inter-cluster communication resources of different nodes among the at least two nodes are used to send different data in the gradient of the first at least one node to the second at least one node.

[0141] The second at least one node is used to update the parameters of the target model deployed in the second at least one node by utilizing the gradients of the first at least one node and the second at least one node.

[0142] In some embodiments, the at least two nodes include the first node and the second node; the step of using the inter-cluster communication resources of the at least two nodes in the first cluster to send the gradient of the first at least one node to the second at least one node includes: the first node sending a portion of the gradient data of the first at least one node to the second node, such that the second node sends the portion of the data to the second at least one node through the inter-cluster communication resources of the second node; wherein, the first node is used to send data other than the portion of the gradient data of the first at least one node to the second at least one node through the inter-cluster communication resources of the first node.

[0143] In one example of this embodiment, the first node and the second node each include a processing unit and a network interface card (NIC); the first node sends a portion of the gradient data from the first at least one node to the second node, including: the processing unit in the first node sends the portion of the data to the processing unit of the second node, such that the processing unit of the second node sends the portion of the data to the NIC of the second node unit; or, the processing unit in the first node sends the portion of the data to the NIC of the second node unit; the second node sends the portion of the data to the second at least one node through the inter-cluster communication resources of the second node, including: the NIC of the second node sends the portion of the data to the second at least one node.

[0144] In some embodiments, the target model includes a first model block and a second model block in sequence, the gradient of the first at least one node is obtained by the first at least one node performing the calculation of the first model block, and the at least two nodes include a node for performing the calculation of the second model block.

[0145] In some embodiments, the instruction module 930 is further configured to: instruct the first node to receive the gradient of the second at least one node using the inter-cluster communication resources of the at least two nodes; wherein the inter-cluster communication resources of different nodes among the at least two nodes are used to receive different data in the gradient of the second at least one node; the first node is configured to send the gradient of the second at least one node to the first at least one node, so that the first at least one node updates the parameters of the target model deployed in the first at least one node using the gradient of the second at least one node and the gradient of the first at least one node.

[0146] In some embodiments, the indicating module 930 is further configured to: determine the number of nodes among the at least two nodes based on at least one of the following: the amount of gradient data of the first at least one node, the amount of inter-cluster communication resources available for nodes in the first cluster, and the expected time of gradient transmission.

[0147] In some embodiments, the instruction module 930 is further configured to: determine the amount of data to be sent by each of the at least two nodes based on the amount of gradient data of the first at least one node and the amount of inter-cluster communication resources available for each of the at least two nodes; and instruct the first node to use the inter-cluster communication resources of each of the at least two nodes to send data to the second at least one node that matches the amount of data to be sent by the node.

[0148] The partitioning module 910, the sending module 920, and the indication module 930 can all be implemented in software or in hardware. For example, the implementation of the partitioning module 910 will be described below. Similarly, the implementation of the sending module 920 and the indication module 930 can refer to the implementation of the partitioning module 910.

[0149] As an example of a software functional unit, module 910 may include code running on a computing instance. A computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0150] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0151] As an example of a hardware functional unit, module 910 may include at least one computing device, such as a server. Alternatively, module 910 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0152] The partitioning module 910 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the partitioning module 910 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the partitioning module 910 includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0153] It should be noted that, in other embodiments, the partitioning module 910 can be used to perform... Figure 6 In the method shown, the sending module 920 can perform any operation executed by the management device 100. Figure 6 In the method shown, the instruction module 930 can be used to perform any operation executed by the management device 100. Figure 6 The method shown describes any operation performed by the management device 100. The steps implemented by the partitioning module 910, the sending module 920, and the indication module 930 can be specified as needed and implemented separately by the partitioning module 910, the sending module 920, and the indication module 930. Figure 6 The different operations performed by the management device 100 in the method shown achieve all the functions of the management device 900.

[0154] This application also provides a computing device 1000. For example... Figure 10 As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0155] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1002 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1002 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0156] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0157] The memory 1006 may include volatile memory, such as random access memory (RAM). The memory 1006 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0158] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the aforementioned partitioning module 910, sending module 920, and indicating module 930, thereby achieving... Figure 6 The operation performed by the method or management device 100 shown. That is, the memory 1006 stores the operations for performing the operation. Figure 6 The instructions for the operation executed by the management device 100 in the method shown.

[0159] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0160] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0161] like Figure 11 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same memory for executing... Figure 6 The instructions for the operation executed by the management device 100 in the method shown.

[0162] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store memory for execution. Figure 6 The method shown refers to the instructions for some operations performed by the management device 100. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for performing... Figure 6 The instructions for the operation executed by the management device 100 in the method shown.

[0163] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the management device 900. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules among the partitioning module 910, the sending module 920, and the instruction module 930.

[0164] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 One possible implementation is shown. For example... Figure 12 As shown, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1006 in computing device 1000A stores instructions for executing the functions of the partitioning module 910. Simultaneously, the memory 1006 in computing device 1000B stores instructions for executing the functions of the sending module 920 and the instruction module 930.

[0165] It should be understood that Figure 12 The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0166] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 11 and Figure 12 The connection method of the computing device cluster. The difference is that the memory 1006 in one or more computing devices 1000 within this computing device cluster can store the same data for execution. Figure 6 The instructions for the operation executed by the management device 100 in the method shown.

[0167] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store memory for execution. Figure 6The instructions shown are partial instructions for the operation performed by the management device 100. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for performing... Figure 6 The instructions for the operation executed by the management device 100 in the method shown.

[0168] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform... Figure 6 The operation performed by the management device 100 in the method shown.

[0169] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a host migration device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute... Figure 6 The operation performed by the management device 100 in the method shown.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, A management device for use in a training system, the training system further comprising multiple nodes, each of the multiple nodes deploying a target model, at least one first node of the multiple nodes located within a first cluster, and at least one second node of the multiple nodes located within a second cluster, wherein the nodes in the first cluster have inter-cluster communication resources for communicating with the nodes in the second cluster; the method includes: The training set of the target model is divided into multiple training subsets; wherein, each of the multiple training subsets corresponds one-to-one with the multiple nodes; Each training subset in the plurality of training subsets is sent to the node corresponding to the training subset, so that the node corresponding to the training subset performs the calculation of the target model for the training subset to obtain the gradient of the node corresponding to the training subset; When a first node in the first cluster obtains the gradient of the first at least one node, it is instructed to use the inter-cluster communication resources of at least two nodes in the first cluster to send the gradient of the first at least one node to the second at least one node; wherein, the inter-cluster communication resources of different nodes among the at least two nodes are used to send different data in the gradient of the first at least one node to the second at least one node. The second at least one node is used to update the parameters of the target model deployed in the second at least one node by utilizing the gradients of the first at least one node and the second at least one node.

2. The method according to claim 1, characterized in that, The at least two nodes include the first node and the second node; The step of using the inter-cluster communication resources of at least two nodes within the first cluster to send the gradient of the first at least one node to the second at least one node includes: The first node sends a portion of the gradient data from the first at least one node to the second node, so that the second node sends the portion of the data to the second at least one node through the inter-cluster communication resources of the second node; The first node is used to send data other than the aforementioned partial data from the gradient of the first at least one node to the second at least one node through the inter-cluster communication resources of the first node.

3. The method according to claim 2, characterized in that, The first node and the second node each include a processing unit and a network interface card (NIC); The first node sends a portion of the gradient data from the first at least one node to the second node, including: The processing unit in the first node sends the partial data to the processing unit of the second node, so that the processing unit of the second node sends the partial data to the network card of the second node unit; or, The processing unit in the first node sends the portion of the data to the network card of the second node unit; The second node sends the partial data to the second at least one node through the inter-cluster communication resources of the second node, including: the network card of the second node sending the partial data to the second at least one node.

4. The method according to any one of claims 1-3, characterized in that, The target model includes a first model block and a second model block in sequence. The gradient of the first at least one node is obtained by the first at least one node performing the calculation of the first model block. The at least two nodes include a node for performing the calculation of the second model block.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: The first node is instructed to use the inter-cluster communication resources of the at least two nodes to receive the gradient of the second at least one node; wherein the inter-cluster communication resources of different nodes among the at least two nodes are used to receive different data in the gradient of the second at least one node; The first node is used to send the gradient of the second at least one node to the first at least one node, so that the first at least one node uses the gradient of the second at least one node and the gradient of the first at least one node to update the parameters of the target model deployed in the first at least one node.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The number of nodes in the at least two nodes is determined based on at least one of the following: the amount of gradient data of the first at least one node, the amount of inter-cluster communication resources available for nodes in the first cluster, and the expected time of gradient transmission.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: The amount of data to be sent by each of the at least two nodes is determined based on the amount of gradient data of the first at least one node and the amount of inter-cluster communication resources available for each of the at least two nodes. The first node is instructed to use the inter-cluster communication resources of each of the at least two nodes to send data to the second at least one node that matches the amount of data sent by the node.

8. A management device, characterized in that, The training system containing the management device further includes multiple nodes, each of which deploys a target model. At least one first node among the multiple nodes is located within a first cluster, and at least one second node among the multiple nodes is located within a second cluster. Nodes in the first cluster have inter-cluster communication resources for communicating with nodes in the second cluster. The device includes: A partitioning module is used to divide the training set of the target model into multiple training subsets; wherein, the multiple training subsets correspond one-to-one with the multiple nodes; The sending module is used to send each training subset in the plurality of training subsets to the node corresponding to the training subset, so that the node corresponding to the training subset performs the calculation of the target model for the training subset to obtain the gradient of the node corresponding to the training subset. An instruction module is used to instruct a first node in the first cluster to send the gradient of the first at least one node to a second at least one node when the first node obtains the gradient of the first at least one node; wherein, the inter-cluster communication resources of different nodes among the at least two nodes are used to send different data in the gradient of the first at least one node to the second at least one node. The second at least one node is used to update the parameters of the target model deployed in the second at least one node by utilizing the gradients of the first at least one node and the second at least one node.

9. The apparatus according to claim 8, characterized in that, The at least two nodes include the first node and the second node; The step of using the inter-cluster communication resources of at least two nodes within the first cluster to send the gradient of the first at least one node to the second at least one node includes: The first node sends a portion of the gradient data from the first at least one node to the second node, so that the second node sends the portion of the data to the second at least one node through the inter-cluster communication resources of the second node; The first node is used to send data other than the aforementioned partial data from the gradient of the first at least one node to the second at least one node through the inter-cluster communication resources of the first node.

10. The apparatus according to claim 9, characterized in that, The first node and the second node each include a processing unit and a network interface card (NIC); The first node sends a portion of the gradient data from the first at least one node to the second node, including: The processing unit in the first node sends the partial data to the processing unit of the second node, so that the processing unit of the second node sends the partial data to the network card of the second node unit; or, The processing unit in the first node sends the portion of the data to the network card of the second node unit; The second node sends the partial data to the second at least one node through the inter-cluster communication resources of the second node, including: the network card of the second node sending the partial data to the second at least one node.

11. The apparatus according to any one of claims 8-10, characterized in that, The target model includes a first model block and a second model block in sequence. The gradient of the first at least one node is obtained by the first at least one node performing the calculation of the first model block. The at least two nodes include a node for performing the calculation of the second model block.

12. The apparatus according to any one of claims 8-11, characterized in that, The indicator module is also used for: The first node is instructed to use the inter-cluster communication resources of the at least two nodes to receive the gradient of the second at least one node; wherein the inter-cluster communication resources of different nodes among the at least two nodes are used to receive different data in the gradient of the second at least one node; The first node is used to send the gradient of the second at least one node to the first at least one node, so that the first at least one node uses the gradient of the second at least one node and the gradient of the first at least one node to update the parameters of the target model deployed in the first at least one node.

13. The apparatus according to any one of claims 8-12, characterized in that, The indicator module is also used for: The number of nodes in the at least two nodes is determined based on at least one of the following: the amount of gradient data of the first at least one node, the amount of inter-cluster communication resources available for nodes in the first cluster, and the expected time of gradient transmission.

14. The apparatus according to any one of claims 8-13, characterized in that, The indicator module is also used for: The amount of data to be sent by each of the at least two nodes is determined based on the amount of gradient data of the first at least one node and the amount of inter-cluster communication resources available for each of the at least two nodes. The first node is instructed to use the inter-cluster communication resources of each of the at least two nodes to send data to the second at least one node that matches the amount of data sent by the node.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-7.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by a cluster of computer devices, the cluster of computer devices performs the method as described in any one of claims 1-7.

Images