Model training method and apparatus

Abstract

The present application provides a model training method and apparatus. The method comprises: a first node receives, from a second node, a first data set output by a second model block; the first node executes, for the first data set, a first task among training tasks of a first model block; the first node receives, from the second node, a second data set output by the second model block; in the case that the first node has completed the first task for the first data set, the first node executes, for the first data set, a second task among the training tasks of the first model block; and meanwhile, the first node executes the first task for the second data set, wherein resource requirements of the first task and the second task are different. The method can reduce the training time of a model and improve the resource utilization rate of a node.

Description

Model training method and device

[0001] The present application claims priority to the Chinese patent application No. 202411110276.5, filed on August 13, 2024, and entitled "A model training method and device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI), and in particular, to a model training method and device. BACKGROUND

[0003] With the pursuit of intelligence of models, the parameter quantity of models is getting larger and larger. Currently, the parameters of some models have reached hundreds of billions, or even thousands of billions. Such a huge amount of parameters makes it difficult for a model to be trained by a single device. The industry adopts a distributed training strategy to train a model in a distributed manner using a large number of training nodes. Among them, pipeline parallelism (PP) only needs to transmit a small amount of activation value between different training nodes, and has a lower requirement on the communication performance between training nodes. Therefore, pipeline parallelism has become a commonly used distributed training strategy.

[0004] In pipeline parallelism, a model is split into multiple serial model blocks, and different model blocks are deployed on different training nodes, which are used to perform training tasks of the model blocks for a data set of the model. In related technologies, a node serially performs training tasks of a model block for different data sets. That is, after completing the training task of a model block for one data set, the training task of the model block is performed for the next data set. This results in a long training time of the model. SUMMARY

[0005] The present application provides a model training method and device, which can reduce the training time of a model and improve the resource utilization of a node.

[0006] In a first aspect, a model training method is provided, which is applied to a first node in a training system, the training system further comprising a second node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the method comprises: the first node receiving, from the second node, a first data set output by the second model block; the first node performing, for the first data set, a first task in a training task of the first model block; the first node receiving, from the second node, a second data set output by the second model block; in a case where the first node completes the first task for the first data set, the first node performs, for the first data set, a second task in the training task of the first model block; meanwhile, the first node performs the first task for the second data set; wherein the resource requirements of the first task and the second task are different.

[0007] Through the method, the node can simultaneously perform two or more tasks in the training task of the same model block for two or more data sets respectively, thereby improving the concurrency of the model block training, and correspondingly, improving the concurrency of the model training. Moreover, the resource requirements of different tasks in the simultaneously performed two or more tasks are different, and there is no resource competition or other conflicts between the simultaneously performed tasks, so that performance degradation and other problems can be avoided. In short, the method improves the concurrency of the model training while avoiding performance degradation and other problems, so that the method can save the time consumption of the model training and improve the efficiency of the model training.

[0008] In addition, the node simultaneously performs tasks with different resource requirements, thereby avoiding performance degradation while reducing the idle time of multiple resources of the node and improving the resource utilization of the multiple resources.

[0009] In a possible implementation manner, the first node comprises a first network card and a first processing unit; wherein the first node receives, from the second node, the first data set output by the second model block comprises: receiving the first data set from the second node by using the first network card; performing the first task in the training task of the first model block comprises: performing the first task by using the first processing unit; wherein the first task is used for copying the first data set from the first network card to the first processing unit; performing the second task in the training task of the first model block comprises: performing the second task by using the first processing unit; wherein the second task is used for performing the calculation of the first model block for the first data set.

[0010] In the implementation manner, while the processing unit performs the model block calculation for the data set, the processing unit can also copy a new data set from the network card to the processing unit for subsequent calculation, so as to realize the mutual masking of the calculation task and the copying task, save the training time consumption of the model block, and improve the training efficiency of the model block.

[0011] In a possible implementation, the first task is one of a compute-intensive task, a memory-intensive task, and a collective communication task, and the second task is another of the compute-intensive task, the memory-intensive task, and the collective communication task.

[0012] The resource requirements of the compute-intensive task, the memory-intensive task, and the collective communication task are different, where the compute-intensive task requires more computing resources, the memory-intensive task requires more memory resources, and the collective communication task requires more communication resources. Meanwhile, different tasks in the compute-intensive task, the memory-intensive task, and the collective communication task are respectively executed for different data sets, which avoids performance degradation and the like while improving the concurrency of model block training, thereby reducing the training time consumption of the model block and improving the training efficiency of the model block.

[0013] In a possible implementation, the second node includes a second network card and a second processing unit, where the second node is configured to: for a third data set, perform, by using the second processing unit, computation of a second model block to obtain a first data set; the second node is further configured to: copy, by using the second processing unit, the first data set from the second processing unit to the second network card; meanwhile, for a fourth data set, perform, by using the second processing unit, computation of the second model block to obtain a second data set; and the second node is further configured to: send, by using the second network card, the first data set to the first node.

[0014] In this implementation, the processing unit performs computation of the model block to generate and output a new data set while copying the data set output by the model block to the network card, which realizes mutual masking of the copying task and the computation task, saves the training time consumption of the model block, and improves the training efficiency of the model block.

[0015] In a possible implementation, the first node and the second node are located in different clusters.

[0016] Since the first node and the second node are located in different clusters and communicate with each other through the network card, the training task of the model block includes data copying tasks, computation tasks, and other tasks with different resource requirements between the processing unit and the network card. Through this method, mutual masking of the copying task and the computation task is realized, and the training time consumption of the model block is saved.

[0017] In addition, network fluctuations can exist between the first node and the second node, and the data set transmitted between the first node and the second node does not completely arrive on time. The training task of the model block can be divided into multiple tasks according to different resource requirements. When the first node has not completed the training task for the current data set, but the first node receives a new data set, the first node can simultaneously perform different tasks in the multiple tasks for the current data set and the new data set, respectively, so that the task for the new data set can be performed in time, and the training efficiency of the model block is improved.

[0018] In a second aspect, a model training method is provided, applied to a second node in a training system, the training system further comprising a first node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node comprises a second network card and a second processing unit; the method comprises: the second node uses the second processing unit to perform calculation of the second model block for third data to obtain a first data set; the second node uses the second processing unit to copy the first data set from the second processing unit to the second network card; at the same time, the second node uses the second processing unit to perform calculation of the second model block for fourth data to obtain a second data set; and the second node uses the second network card to send the first data set to the first node.

[0019] Through the method, the processing unit performs calculation of the model block to generate and output a new data set while copying the data set output by the model block to the network card. In this way, the copying task and the calculation task are mutually masked, the training time of the model block is saved, and the training efficiency of the model block is improved.

[0020] In a third aspect, a model training apparatus is provided, applied to a first node in a training system, the training system further comprising a second node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the apparatus comprises: a receiving module, configured to receive a first data set output by the second model block from the second node; an execution module, configured to perform a first task in a training task of the first model block for the first data set; the receiving module is further configured to receive a second data set output by the second model block from the second node; and the execution module is further configured to, in a case where the first task is completed for the first data set, perform a second task in the training task of the first model block for the first data set, and simultaneously perform the first task for the second data set; wherein the resource requirements of the first task and the second task are different.

[0021] In a possible implementation, the first node comprises a first network card and a first processing unit; the receiving module is configured to receive the first data set from the second node by using the first network card; the execution module is configured to execute the first task by using the first processing unit; the first task comprises copying the first data set from the first network card to the first processing unit; the execution module is configured to execute the second task by using the first processing unit; the second task comprises performing the calculation of the first model block on the first data set.

[0022] In a possible implementation, the first task is one of a compute-intensive task, a memory-intensive task, and a collective communication task, and the second task is another one of the compute-intensive task, the memory-intensive task, and the collective communication task.

[0023] In a possible implementation, the second node comprises a second network card and a second processing unit; the second node is configured to perform the calculation of the second model block on the third data set by using the second processing unit to obtain the first data set; the second node is further configured to copy the first data set from the second processing unit to the second network card by using the second processing unit, and perform the calculation of the second model block on the fourth data set by using the second processing unit to obtain the second data set; the second node is further configured to send the first data set to the first node by using the second network card.

[0024] In a possible implementation, the first node and the second node are located in different clusters.

[0025] In a fourth aspect, a model training apparatus is provided, which is applied to a second node in a training system, and the training system further comprises a first node; the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node comprises a second network card and a second processing unit; the apparatus comprises an execution module and a sending module; the execution module is configured to perform the calculation of the second model block on the third data by using the second processing unit to obtain a first data set; the execution module is further configured to copy the first data set from the second processing unit to the second network card by using the second processing unit, and perform the calculation of the second model block on the fourth data by using the second processing unit to obtain a second data set; and the sending module is configured to send the first data set to the first node by using the second network card.

[0026] In a fifth aspect, a computing device cluster is provided, comprising at least one computing device, and each computing device comprises a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method provided in the first aspect or the method provided in the second aspect.

[0027] In a sixth aspect, a computer-readable storage medium is provided, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the first aspect or the method provided in the second aspect.

[0028] In a seventh aspect, a computer program product including instructions is provided, when the instructions are run by a computing device cluster, the computing device cluster executes the method provided in the first aspect or the method provided in the second aspect.

[0029] The beneficial effects of the third aspect to the seventh aspect can be referred to the beneficial effects of the first aspect or the second aspect described above, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0030] FIG. 1A is a schematic diagram of a model training scheme;

[0031] FIG. 1B is a schematic diagram of a model training scheme;

[0032] FIG. 2 is a schematic diagram of a training system provided by an embodiment of the present application;

[0033] FIG. 3 is a schematic diagram of a node provided by an embodiment of the present application;

[0034] FIG. 4 is a schematic diagram of a training apparatus provided by an embodiment of the present application;

[0035] FIG. 5 is a schematic diagram of a training apparatus provided by an embodiment of the present application;

[0036] FIG. 6 is a flowchart of a model training method provided by an embodiment of the present application;

[0037] FIG. 7 is a schematic diagram of a model training method provided by an embodiment of the present application;

[0038] FIG. 8 is a flowchart of a model training method provided by an embodiment of the present application;

[0039] FIG. 9 is a schematic diagram of a model training method provided by an embodiment of the present application;

[0040] FIG. 10 is a structural schematic diagram of a training apparatus provided by an embodiment of the present application;

[0041] FIG. 11 is a structural schematic diagram of a training apparatus provided by an embodiment of the present application;

[0042] FIG. 12 is a structural schematic diagram of a computing device provided by an embodiment of the present application;

[0043] FIG. 13 is a structural schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0044] FIG. 14 is a structural diagram of a computing device cluster according to an embodiment of the present application;

[0045] FIG. 15 is a structural diagram of a computing device according to an embodiment of the present application;

[0046] FIG. 16 is a structural diagram of a computing device cluster according to an embodiment of the present application;

[0047] FIG. 17 is a structural diagram of a computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] The schemes provided by the embodiments of the present application will be described below with reference to the drawings. In the embodiments of the present application, "multiple" refers to two or more, and "multiple types" refers to two or more types. "First", "second", and the like are only used to distinguish similar objects, and do not necessarily indicate a specific order or number of objects.

[0049] To facilitate understanding of the schemes provided by the embodiments of the present application, the technical terms that may be involved in the embodiments of the present application are first introduced.

[0050] Model: composed of multiple artificial neural network layers. Usually, the multiple neural network layers are arranged in series, and the output of one neural network layer is the input of the next neural network layer. The neural network layer can be simply referred to as a network layer or a layer.

[0051] Model training: refers to the process of updating the parameters of the model. Generally, the parameters of the model are preset or randomly generated. By updating the parameters of the model, the model learns the ability to process corresponding tasks, such as image recognition, speech recognition, natural language translation, computer game playing, content generation, etc. The model training needs to go through multiple iterations until convergence or reach a predetermined number of training times. Each iteration needs to input a certain amount of data set. The model performs forward calculation on the input data set to obtain a loss function. The model then performs backward calculation on the loss function to obtain a gradient. Then, the gradient is used to update the model parameters.

[0052] Model calculation: refers to the calculation performed by the model on the relevant data set using the parameters of the model. The model calculation can be divided into forward calculation and backward calculation.

[0053] Calculation result: refers to the result obtained by calculating the relevant data set using the parameters of the model. The calculation results of forward calculation and backward calculation both include activation values. In addition, the calculation result of the last layer of the model can include a loss function. The calculation result of the backward calculation also includes a gradient.

[0054] Forward propagation: also known as forward pass, refers to the calculation and transmission of data through each layer of the model from the data input to obtain the output result of the model. The purpose of forward propagation is to calculate the output result of the model. The output result of the model includes the loss function of the model.

[0055] Forward calculation: refers to the calculation of the layer or model block in the model for forward propagation, which is used to obtain the activation value of the model layer or model block. The forward calculation of the last layer or the last model block in the model is used to obtain the loss function of the model.

[0056] Back propagation: also known as backward pass, refers to obtaining the loss function according to the difference between the output result of the model and the label, and then using the loss function to calculate the gradient in reverse through the chain rule. Specifically, in back propagation, the activation value calculated by the model block is transmitted, and the activation value is used to calculate the gradient and the activation value of the next model block. More specifically, the current model block transmits the activation value calculated by the current model block to the next model block of the current model block, so that the next model block of the current model block calculates the gradient and the activation value, and so on. In this way, through back propagation, each model block can obtain the gradient of the loss function, which is used to update the parameters of the model block. In the back propagation, the previous model block of the current model block refers to the previous model block in the data transmission direction of the back propagation, and the next model block of the current model block refers to the next model block in the data transmission direction of the back propagation.

[0057] Backward calculation: refers to the calculation of the layer or model block in the model for back propagation. Backward calculation is used to obtain the gradient based on the loss function obtained by forward calculation, and to update the parameters of the model using the gradient.

[0058] Gradient: refers to the direction in which the loss function value decreases most quickly. The gradient is obtained by backward calculation and is the result of backward calculation. The gradient is actually a vector representing a direction, and each element of the gradient at a position represents the rate of change of the corresponding element of the loss function value with respect to the weight. The gradient is used to update the parameters of the model. Specifically, the gradient can be multiplied by a learning step to obtain a product, and then the parameters of the model are added (or subtracted) to the product to obtain the updated parameters.

[0059] Gradient of a node: refers to the gradient obtained by the node for the corresponding data set by executing the calculation of the model or the model block.

[0060] Activation: refers to an intermediate result generated in the model training process, which is the output of each network layer or model block of the model. The activation of a network layer or model block is used for the relevant calculation of the next network layer or model block of the network layer or model block, and generates the activation of the next network layer or model block. In addition, in back propagation, the activation of a network layer or model block is used for the relevant calculation of the next network layer or model block of the network layer or model block, and generates the activation and gradient of the next network layer or model block.

[0061] Pipeline parallelism (PP): a model distributed training technique, which refers to different nodes running different layers of the model, belonging to inter-layer parallelism. That is, in pipeline parallelism, the model is serialized and divided by layer as the granularity, obtaining multiple model blocks connected in sequence. In model training, the relevant calculations of different model blocks are performed by different nodes. Among them, a model block can be referred to as a stage in the model training process, and N model blocks are N stages in the model training process.

[0062] Model block: a neural network block obtained by inter-layer division of the model. A model block can include one or more neural network layers.

[0063] Stage: a model block in pipeline parallelism technology.

[0064] Tensor parallelism (TP): another model distributed training technique, which refers to splitting the tensors in the model to obtain multiple tensor sets of the model, and then deploying different tensor sets in multiple tensor sets to different training nodes for training. Each tensor set includes at least one tensor.

[0065] Training task of a model block: the tasks related to a model block during the training process of the model can be collectively referred to as the training task of the model block, which can include tasks such as obtaining a data set of the model block, performing relevant calculations (such as forward calculation and backward calculation) of the model block on the data set, and issuing a new data set obtained by calculation. The resources required by different stages of the training task can be different, and the training task can be divided into different tasks based on the requirement for resources. For example, the node where the model block is located receives or sends a data set through a network card, performs calculation on the data set through a processing unit, and the model block obtains and issues the data set, which involves data copying between the network card and the processing unit, and requires more communication resources between the network and the processing unit. The calculation of the processing unit on the data set requires more computing resources of the processing unit. Therefore, the training task of the model block can be divided into a copying task and a calculation task. Among them, the copying task is used for data copying between the network card and the processing unit, and the calculation task is used for the processing unit to perform calculation of the model block on the data set. For another example, some calculations (such as multiplication calculation of tensors) in the relevant calculation of the model block on the data set require more computing resources, some calculations (such as addition calculation of tensors) require more memory resources, and some calculations (such as parameter exchange) require collective communication resources. Therefore, the training task of the model block can be divided into: computation-intensive task, memory-intensive task, and collective communication task.

[0066] Collective communication: communication required for model training using both pipelining parallelism and tensor parallelism. Specifically, a model block in pipelining parallelism is split into multiple tensor sets of the model block in tensor granularity, wherein each tensor set includes at least one tensor. Then, different tensor sets in the multiple tensor sets are deployed to different processing units in a node for training. That is, the tensors in the same model block are distributed in multiple processing units for training. The multiple processing units form a processing unit set of the model block. During the training of the model block or during the execution of the calculation task of the model block, the multiple processing units need to exchange parameters. The collective communication is the communication involved in exchanging parameters, which requires more communication resources between the processing units.

[0067] Data set: the training set of the model, the intermediate calculation result (such as activation value) of the model, the input and output of the model or the model block can be collectively referred to as the data set.

[0068] Micro-batch: a kind of fine-grained data set. In pipeline parallelism, if the data set input to the model block each time is large (for example, mini-batch), the time required for the calculation of the model block for the data set is also long, resulting in low concurrency of pipeline parallelism. In order to improve the concurrency of pipeline parallelism, the large data set (for example, mini-batch) can be further split to obtain a fine-grained data set. The fine-grained data set is called micro-batch.

[0069] Region: a kind of computing node deployment unit in infrastructure, which is a deployment unit divided from the dimensions of geographical location and network latency. The isolation level between regions is high. Among them, the computing nodes in the same region share common services such as elastic computing, block storage, object storage, VPC network, elastic public network IP, and image.

[0070] Availability zone (AZ): a computing node deployment unit in infrastructure, and an AZ is a collection of one or more physical data centers with independent wind, fire, water, and electricity. An AZ includes one data center or multiple data centers in close geographical locations. A region can include multiple AZs.

[0071] Training node: which can be referred to as node, refers to a computing node used for performing model training or model calculation. The node can include at least one processing unit and a network card. Among them, the processing unit can be a neural processing unit (NPU), a graphics processing unit (GPU), a tensor processing unit (TPU), etc. The network card of the node is used for communication between the node and other nodes in the cluster. Common nodes include hosts, virtual machines (VMs). Among them, the host can be a server or other physical computing device.

[0072] Data center network (DCN): a kind of inter-cluster connection network, used to connect nodes in multiple clusters together. The DCN has scalability and high efficiency, and can connect tens or even hundreds of thousands of nodes to meet the growing connection needs of cloud computing.

[0073] Parameter plane network: a high-speed network specially designed for distributed training. Nodes in the same cluster can be connected through the parameter plane network. Among them, when the node is a host, the processing units of two hosts in the same cluster can communicate through the parameter plane network without going through the network card of the host. Generally speaking, the communication rate of the parameter plane network can reach 200GE.

[0074] Cluster: also known as computing cluster or training cluster, a cluster composed of multiple nodes. The nodes in the same cluster are in close proximity, for example, in the same region or AZ. The communication capability between nodes in the same cluster is strong, that is, the communication capability between nodes in the same cluster is greater than that between nodes in the cluster and nodes outside the cluster. For example, the network between nodes in the same cluster is a parameter plane network or a parameter plane-like network, and the network between nodes in the cluster and training nodes outside the cluster is a DCN or a DCN-like network. In addition, in the embodiments of the present application, nodes in the same region or AZ are not necessarily in the same cluster. This is because some nodes in the same region or AZ may not use the parameter plane network or the parameter plane-like network, resulting in poor communication capability between the node and other nodes in the region or AZ. It can be considered that the node and other nodes in the region or AZ are not in the same cluster, but are in different clusters.

[0075] Neural processing unit (NPU): a hardware processor specially designed to accelerate neural network computation. Compared with central processing unit (CPU) and GPU, NPU optimizes matrix multiplication, convolution operation and other calculations in neural network, has higher computing efficiency and energy efficiency. NPU can be applied to artificial intelligence and deep learning tasks, such as image recognition, natural language processing and speech recognition, etc.

[0076] Stream: a task stream composed of a series of asynchronous tasks in a node, which are executed on the processing unit according to the calling order of the CPU side. The tasks in the same stream are executed serially, and the asynchronous tasks on different streams can be executed in parallel.

[0077] The parameter quantity of a model is constantly expanding, and the parameter quantity of some models has reached hundreds of billions or even trillions. Moreover, the size of the training set of the model is constantly expanding, and training sets of hundreds of terabytes (TB) or even petabytes (PB) are not uncommon, which makes the demand for computing resources, power, heat dissipation capacity, etc. of model training huge. A single geographical location cluster is difficult to meet this demand. The industry seeks to use clusters scattered in different geographical locations to train models. Due to distance restrictions, it is difficult to deploy high-performance networks between different clusters, making the communication performance between clusters weaker than the communication performance within a cluster. Since pipeline parallelism only needs to transmit a small amount of activation value between different nodes, the communication performance between nodes is less demanding. Therefore, when training a model using multiple clusters, pipeline parallelism has become a commonly used distributed training strategy.

[0078] In pipeline parallelism, a model is split into multiple serial model blocks, where a model block is deployed to a node or a set of nodes. The node where the first model block in the multiple model blocks is located sequentially acquires the training set in micro-batch granularity, and for each micro-batch, performs the training task of the first model block, generates a data set (intermediate result of the model) corresponding to the micro-batch, and sends the data set to the node where the second model block is located. The node where the second model block is located performs the training task of the second model block for the data set, generates a new data set corresponding to the micro-batch, and sends the new data set to the node where the next model block is located, and so on until the node where the last model block is located outputs the final data set.

[0079] In related technologies, as shown in FIG. 1A, each node uses one stream to perform the task of the model block for each micro-batch data set. That is, the node performs the training task of the model block for the next data set only after completing the training task of the model block for one data set. More specifically, the training task of the model block includes a copy task of copying the data set from the network card to the processing unit and a calculation task of performing the calculation of the model block for the data set. In this technology, as shown in FIG. 1A, the node sequentially performs the copy task and the calculation task for the next data task only after completing the copy task and the calculation task for one data set. This results in a long training time for the model and low resource utilization of the node.

[0080] In addition, the network between the nodes where the adjacent model blocks are located can fluctuate, causing the node to not complete the training task of the model block performed by the node for the previous data set when the node receives the current data set. This causes the node to wait until the training task of the node for the previous data set is completed before performing the training task of the model block for the current data set. Specifically, it can be set that the model blocks deployed by node A1 and node A2 are adjacent model blocks. Influenced by the network fluctuation between node A1 and node A2, node A2 receives the current data set at time t0, but node A2 completes the training task for the previous data set at time t1. In this technology, node A2 cannot start performing the training task for the current data set until time t1. This also causes the training of the model to take a long time and the resource utilization of the node to be low.

[0081] The embodiment of the present application provides a model training method. The method can be applied to pipeline parallelism, and can improve the model training efficiency of pipeline parallelism and the resource utilization of the node. In the method, the training task of the model block can be divided based on the required resources to obtain a plurality of tasks, wherein different tasks in the plurality of tasks require different resources. The node can perform one task in the plurality of tasks for one data set, and simultaneously, the node performs another task in the plurality of tasks for another data set. In this way, the node can perform tasks for different data sets in parallel, and different tasks require different resources, and there is no conflict between different tasks of the node, and simultaneously performing two or more tasks does not cause the node to have performance degradation, so that the training time of the model can be saved, and the training efficiency of the model can be improved. In addition, different tasks require different resources, and the node simultaneously performs two or more tasks, so that the plurality of resources of the node can be fully utilized, and the resource utilization of the node can be improved.

[0082] Next, the model training method provided by the embodiment of the present application is introduced.

[0083] FIG. 2 shows a training system 100 that can be used to implement the method. The training system 100 includes a plurality of nodes 110, 120, etc. The plurality of nodes are used to train a target model in a pipeline parallel manner. That is, the target model can be divided into a plurality of model blocks in series, the plurality of model blocks and the plurality of nodes are one-to-one corresponding, and the model block is deployed in the node corresponding to the model block. The node can perform the training task of the model block deployed in the node for the data set of the target model.

[0084] For convenience of description, the model blocks deployed in the node 110 can be referred to as model blocks 210, and the model blocks deployed in the node 120 can be referred to as model blocks 220. Among them, the model blocks 210 and the model blocks 220 are adjacent in the target model. That is, in one of the forward propagation and the backward propagation, the data set (for example, the activation value) of the output of the model block 110 is the input of the model block 120. In the other of the forward propagation and the backward propagation, the data set (for example, the activation value) of the output of the model block 120 is the input of the model block 110. This makes the node 110 and the node 120 need to interact the data set.

[0085] In some embodiments, as shown in FIG. 2, the node 110 and the node 120 can be located in different clusters, for example, the node 110 is located in the cluster B1, and the node 120 is located in the cluster B2. Exemplarily, the node 110 and the node 120 can be connected through a DCN network. Exemplarily, the node 110 and the node 120 interact the data set through the transmission control protocol (TCP).

[0086] In some embodiments, as shown in FIG. 3, the node 110 and the node 120 can each include at least one processing unit and a network card. Among them, the processing unit is used to perform the training task of the model block, and the network card is used for sending or receiving the data set. Taking the node 120 sending the data set to the node 110 as an example, the processing unit in the node 120 performs the training task of the model block 220 for the data set obtained by the node 120, and obtains the data set output by the model block 220. The network card in the node 120 is used to send the data set output by the model block 220 to the network card of the node 110. The processing unit of the node 110 can perform the training task of the model block 210 for the data set output by the model block 220, and obtain the data set output by the model block 210.

[0087] In some embodiments, the processing unit in the node 110 and / or the node 120 can include a computing module and a data transmission module. Among them, the computing module can perform a computing task, for example, the computing task of the model block. The data transmission module can perform a data transmission task, for example, a data copying task, etc. The computing module and the data transmission module can work simultaneously, and each perform its own task.

[0088] In the following description, when the node 110 and the node 120 are not specially distinguished, they can be simply referred to as nodes.

[0089] As shown in FIG. 3, the node can include a training device. In some embodiments, the training device in the node can be a software device running in the CPU or processing unit of the node. Under the action of the training device, the node can perform the training task of the model block for two and more data sets at the same time. As shown in FIG. 4, the training device can include a task profiling module and a task scheduling module.

[0090] The task profiling module is used to analyze the tasks (i.e., training tasks) in the training process of the target model and classify the training tasks. As shown in FIG. 4, the task types that can be identified by the task profiling module include copy tasks, computing tasks. In some embodiments, as shown in FIG. 5, the computing tasks can be further divided into computing-intensive tasks, memory-intensive tasks and collective communication tasks. The task profiling module can extract the features of the tasks under each task type, including the predicted execution time, resource requirements and data dependency between tasks. Based on this information, the task profiling module generates a label for the tasks under each task type. The task profiling module can also assign priorities according to the importance of the tasks. For example, the importance of the tasks is determined by the data set of the tasks. Specifically, the importance of the tasks performed for the previous data set is greater than the importance of the tasks performed for the current data set. If the tasks to be processed include tasks performed for the previous data set and tasks performed for the current data set, and the two tasks conflict (for example, the same resource is required), the tasks performed for the previous data set are executed first. The task profiling module can also collect historical execution data of the tasks under each task type. The historical execution data can be used to improve future task classification and feature extraction. The foregoing information can be passed to the task scheduling module to provide a basis for task scheduling of the task scheduling module.

[0091] The task scheduling module can instruct the processing unit to execute the tasks using multiple streams. The task scheduling module can manage the multiple streams and assign tasks to the multiple streams. The task scheduling module can use the information obtained from the task profiling module to assign the tasks to be executed to the corresponding streams, for example, assign tasks with data dependencies to the same stream to ensure the correct execution order of the tasks. Assign tasks without data dependencies to different streams to achieve concurrent execution of the tasks. For example, as shown in FIG. 4, the copy tasks of data set D1 are assigned to stream C1, and the computing tasks of data set D2 are assigned to stream C2.

[0092] The task scheduling module can identify tasks that have conflicts based on resource requirements of the tasks. When the resource requirements of two tasks are the same, it can be considered that the two tasks have conflicts. When the resource requirements of two tasks are different, it can be considered that the two tasks do not have conflicts. When there are multiple tasks to be executed and the multiple tasks have conflicts, the task scheduling module can control the execution order of the tasks based on the priorities of the tasks. When the priority of a task is high, the task is executed first. When there are multiple tasks to be executed and the multiple tasks do not have conflicts, the multiple tasks can be executed simultaneously. For example, as shown in FIG. 4, the copy task of the data set D1 and the calculation task of the data set D2 do not have conflicts and can be executed simultaneously. For another example, as shown in FIG. 5, the calculation task E1 of the data set D1 and the calculation task E2 of the data set D2 do not have conflicts and can be executed simultaneously. The calculation task E1 is one of a calculation-intensive task, a memory-intensive task, and a collective communication task, and the calculation task E2 is another of the calculation-intensive task, the memory-intensive task, and the collective communication task.

[0093] In some embodiments, the task scheduling module can control the number of tasks allocated to each stream to balance the load of each stream and provide overall execution efficiency of the tasks.

[0094] In this way, the task scheduling module can control concurrent execution of tasks of different data sets and adjust the execution order of the tasks in the model training process, reduce training time consumption, improve resource utilization, ensure that tasks of a preceding data set are executed in priority, and ensure that tasks having a data dependency relationship are executed in order.

[0095] The above examples introduce the training system 100 provided by the embodiments of the present application. Next, taking implementation in the training system 100 as an example, the flow of the model training method provided by the embodiments of the present application is introduced. As shown in FIG. 6, the method includes the following steps.

[0096] In step 601, the node 110 receives the data set D1 output by the model block 220 from the node 120.

[0097] The data set D1 is obtained by the node 120 performing a calculation task of the model block 220 on the data set D1’. Specifically, the node 120 deploys the model block 220, the node 120 inputs the data set D1’ into the model block 220, the model block 220 calls related resources of the node 120 to perform calculation, and obtains and outputs the data set D1.

[0098] If the model block 220 is a previous model block of the model block 210 in the forward propagation, the data set D1 is obtained by the forward calculation of the model block 220. If the model block 220 is a previous model block of the model block 210 in the backward propagation, the data set D1 is obtained by the backward calculation of the model block 220.

[0099] In some embodiments, the model block 220 can be the first model block in a plurality of model blocks obtained by dividing the target model, and the data set D1' is a micro-batch of the target model, i.e., a training set of a micro-batch granularity. In some embodiments, the model block 220 is not the first model block in a plurality of model blocks obtained by dividing the target model, and the data set D1' is a data set output by a previous model block of the model block 220.

[0100] In some embodiments, as shown in FIG. 6, the node 120 can include a processing unit 121 and a network card 122. The node 120 performs step 501 by using the processing unit 121 to perform the calculation of the model block 220 (i.e., the calculation task in the training task of the model block 220) for the data set D1', and obtain the data set D1. The calculation is the forward calculation or the backward calculation. Then, the node 120 performs step 502a by using the processing unit 121 to copy the data set D1 from the processing unit 121 to the network card 122. In other words, in step 502a, the processing unit 121 performs a copy task in the training task of the model block 220 for the data set D1. The copy task is used to copy the data set D1 from the processing unit 121 to the network card 122. After the completion of the copy task, the network card 122 is used to send the data set D1 to the node 110.

[0101] The copy task can also be referred to as an input / output (I / O) task. In one example of the embodiment, the processing unit 121 can specifically copy the data set D1 to the system memory of the node 120, and the network card 122 can obtain the data set D1 from the system memory.

[0102] If the processing unit 121 obtains the data set D2' before or during the execution of the copy task for the data set D1 by the processing unit 121, the node 120 can perform step 502b by using the processing unit 121 to perform the calculation of the model block 220 (i.e., the calculation task in the training task of the model block 220) for the data set D2', and obtain the data set D2.

[0103] Dataset D2’ is the next dataset of dataset D1’. For example, if dataset D1’ and dataset D2’ are respectively one micro-batch of the target model, dataset D2’ is the next micro-batch of dataset D1’ in this micro-batch. For another example, if dataset D1’ and dataset D2’ are respectively datasets outputted by a previous model block of model block 220, dataset D2’ is the next dataset outputted by the previous model block of model block 220 after outputting dataset D1’.

[0104] After processing unit 121 finishes step 502b, i.e., finishes the computation task of model block 220 for dataset D2’, node 120 can use processing unit 121 to perform step 503, copying dataset D2 from processing unit 121 to network card 122. That is, in step 503, for dataset D2, a copy task in the training task of model block 220 is performed to copy dataset D2 to network card 122. At the same time, processing unit 121 can also perform the computation of model block 220 for the next dataset of dataset D2’. This is also true for the subsequent datasets.

[0105] The computation task of the model block requires computing resources, and the copy task requires communication resources. The required resources of these two tasks are different, so processing unit 121 can simultaneously perform the computation task for one dataset and the copy task for another dataset, which improves the degree of concurrency of task execution without causing performance degradation.

[0106] In some embodiments, processing unit can use different streams to concurrently perform the computation task for different datasets and the copy task.

[0107] Compared with the node serially performing the computation task for a dataset and the copy task for the next dataset of the dataset, the node concurrently performing the computation task for a dataset and the copy task for the next dataset of the dataset can make the time consumption of the computation task and the time consumption of the copy task mask each other, thereby shortening the training time consumption of the model.

[0108] Referring to FIG. 7, it can be set that the target model is split into the model block 220 and the model block 210, and on the forward propagation, the model block 220 is before the model block 210. It can also be set that the training set of the target model is split into 3 micro-batches, the time consumption of a single forward calculation is 8 unit time lengths, the time consumption of a single backward calculation is 16 unit time lengths, the time consumption of a single copy task is 1 unit time length, and the time consumption of transmitting a data set corresponding to a micro-batch between the node 120 and the node 110 is 2 unit time lengths. In the case of using the Gpipe pipeline parallel algorithm, the time consumption of a single iteration of training is 84 unit time lengths. In the case of using the model training method provided in the embodiments of the present application (i.e., the node parallelly executes the calculation task for a data set and the copy task for the next data set of the data set), the time consumption of a single iteration of training is 80 unit time lengths. Thus, 4 unit time lengths can be saved, i.e., T1 in FIG. 7 is 4 unit time lengths.

[0109] In addition, the saved time consumption and the number of micro-batches are positively correlated. With the increase of the number of micro-batches, the time consumption of model training is shorter.

[0110] Continuing to refer to FIG. 6, the node 110 can execute step 602 and step 603. In step 602, the node 110 executes task F1 in the training task of the model block 210 for data set D1. In step 603, the node 110 receives data set D2 output by the model block 220 from the node 120. The data set D2 is obtained by the node 120 executing the calculation task of the model block 220 after outputting data set D1 by the model block 220. For details, refer to the foregoing description, which will not be repeated here.

[0111] Then, the node 110 can execute step 604. In the case that the node 110 completes task F1 for data set D1, the node 110 executes task F2 in the training task of the model block 210 for data set D1, and simultaneously, the node 110 executes task F1 of the model block 210 for data set D2. The resource requirements of task F1 and task F2 are different.

[0112] In some embodiments, as shown in FIG. 8, the node 110 includes a processing unit 111 and a network card 112. The model block 210 is deployed in the processing unit 111, and the node 110 executes the calculation task of the model block 210 by using the processing unit 111. The calculation task of the model block 210 is forward calculation or backward calculation. The node 110 receives the data set from the node 120 by using the network card 112. In this way, the training task of the model block 210 includes the copy task of copying the data set from the network card 112 to the processing unit 111 and the calculation task of the model block 210.

[0113] In this embodiment, task F1 can be a copy task of copying a data set from network card 112 to processing unit 111, and task F2 can be a computation task of executing model block 210 for the data set. That is, after completing the copy task of copying data set D1 from network card 112 to processing unit 111, node 110 can execute the computation task of model block 210 for data set D1 using processing unit 111, while node 110 can execute the copy task of copying data set D2 from network card 112 to processing unit 111 using processing unit 111. Thus, node 110 can concurrently execute related tasks for two or more data sets, saving the training time of the target model and improving the resource utilization of node 110.

[0114] In some embodiments, as described above, the training tasks of a model block can be divided into a computation-intensive task, a memory-intensive task, and a collective communication task. The computation-intensive task, the memory-intensive task, and the collective communication task require different resources, where the computation-intensive task requires more computing resources, the memory-intensive task requires more memory resources, and the collective communication task requires more communication resources between processing units. Task F1 can be one of the computation-intensive task, the memory-intensive task, and the collective communication task, and task F2 can be another of the computation-intensive task, the memory-intensive task, and the collective communication task, that is, task F1 and task F2 are different tasks of the computation-intensive task, the memory-intensive task, and the collective communication task.

[0115] In this embodiment, when the network between node 110 and node 120 fluctuates and the data set cannot reach node 120 on time or even on demand, node 110 does not need to wait for the completion of the training task for the preceding data set before executing the training task for the subsequent data set. For example, node 110 obtains data set D2 before completing the training task for data set D1, in which case node 110 can execute task F1 for data set D1 and task F2 for data set D2 at the same time. For example, as shown in FIG. 9, at time t2, node 110 executes the memory-intensive task (task F1) for data set D1 and executes the computation-intensive task (task F2) for data set D2.

[0116] Thus, as shown in FIG. 9, in each iteration, the present embodiment can save the time consumption of T2 relative to the scheme shown in FIG. IB, thereby saving the training time of the model and improving the resource utilization of node 110.

[0117] In some embodiments, the node 110 includes a processing unit 111 and a network card 112. The node 110 can perform the computing task of the model block 210 by using the processing unit 111. The node 110 can receive the data set from the node 120 by using the network card 112. The training task of the model block 210 includes the copying task of copying the data set from the network card 112 to the processing unit 111 and the computing task of the model block 210. The computing-intensive task, the memory-intensive task, and the collective communication task all belong to the computing task in the training task of the model block. The task F1 can be the copying task for copying the data set from the network card 112 to the processing unit 111, and the task F2 can be any one of the computing-intensive task, the memory-intensive task, and the collective communication task.

[0118] In summary, by using the model training method provided in the embodiments of the present application, the node can perform the tasks for different data sets in parallel, and the resource requirements of two or more tasks performed at the same time are different and do not conflict with each other, and the performance of the node is not degraded by performing two or more tasks at the same time. Therefore, the training time of the model can be saved, and the training efficiency of the model can be improved. In addition, the resources required by different tasks are different, and the simultaneous execution of two or more tasks by the node can make full use of various resources of the node, thereby improving the resource utilization rate of the node.

[0119] The embodiments of the present application provide a model training device 1000. The device 1000 is applied to a first node in a training system, and the training system further includes a second node. The first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model. In the target model, the first model block and the second model block are adjacent. As shown in FIG. 10, the device 1000 includes:

[0120] The receiving module 1010 is configured to receive a first data set output by the second model block from the second node;

[0121] The execution module 1020 is configured to perform a first task in the training task of the first model block for the first data set;

[0122] The receiving module 1010 is further configured to receive a second data set output by the second model block from the second node;

[0123] The execution module 1020 is further configured to, in a case where the first task for the first data set is completed, perform a second task in the training task of the first model block for the first data set, and simultaneously perform the first task for the second data set. The resource requirements of the first task and the second task are different.

[0124] In some embodiments, the first node comprises a first network card and a first processing unit; wherein the receiving module 1010 is configured to receive the first data set from the second node by using the first network card; the performing module 1020 is configured to perform the first task by using the first processing unit; wherein the first task comprises copying the first data set from the first network card to the first processing unit; the performing module 1020 is configured to perform the second task by using the first processing unit; wherein the second task comprises performing the computation of the first model block on the first data set.

[0125] In some embodiments, the first task is one of a compute-intensive task, a memory-intensive task, and a collective communication task, and the second task is another one of the compute-intensive task, the memory-intensive task, and the collective communication task.

[0126] In some embodiments, the second node comprises a second network card and a second processing unit, wherein the second node is configured to perform the computation of the second model block on a third data set by using the second processing unit to obtain the first data set; the second node is further configured to copy the first data set from the second processing unit to the second network card by using the second processing unit, and simultaneously, perform the computation of the second model block on a fourth data set by using the second processing unit to obtain the second data set; the second node is further configured to send the first data set to the first node by using the second network card.

[0127] In some embodiments, the first node and the second node are located in different clusters.

[0128] The receiving module 1010 and the performing module 1020 can be implemented by software or by hardware. For example, the implementation of the receiving module 1010 is described below. The implementation of the performing module 1020 can be similar to the implementation of the receiving module 1010.

[0129] As an example of a software functional unit, the receiving module 1010 can include code running on a compute instance. The compute instance can include at least one of a physical host (computing device), a virtual machine, a container. Further, the compute instance can be one or more. For example, the receiving module 1010 can include code running on multiple hosts / virtual machines / containers. It is noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Typically, a region can include multiple AZs.

[0130] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Typically, a VPC is set up within a region, and communication between two VPCs in the same region, or between VPCs in different regions, requires a communication gateway in each VPC to enable interconnection between the VPCs.

[0131] As an example of a hardware functional unit, the receiving module 1010 can include at least one computing device, such as a server, etc. Alternatively, the receiving module 1010 can also be a device implemented using an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), etc. The PLD can be implemented as a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0132] The multiple computing devices included in the receiving module 1010 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 1010 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the receiving module 1010 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0133] It should be noted that, in other embodiments, the receiving module 1010 can be configured to perform the operations performed by the node 110 in the method shown in FIG. 6, and the execution module 1020 can be configured to perform the operations performed by the node 110 in the method shown in FIG. 6. The operations that the receiving module 1010 and the execution module 1020 are responsible for implementing can be specified as needed, and the entire function of the apparatus 1000 can be implemented by the receiving module 1010 and the execution module 1020 respectively implementing different operations performed by the node 110 in the method shown in FIG. 6.

[0134] The embodiments of the present application also provide a model training apparatus 1100. The apparatus 1100 is applied to a second node in a training system, the training system further comprising a first node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node comprises a second network card and a second processing unit. As shown in FIG. 11, the apparatus 1100 comprises an execution module 1110 and a sending module 1120; wherein,

[0135] The execution module 1110 is configured to perform, by using the second processing unit, calculation of the second model block for third data to obtain a first data set;

[0136] The execution module 1110 is further configured to copy, by using the second processing unit, the first data set from the second processing unit to the second network card; and perform, by using the second processing unit, calculation of the second model block for fourth data to obtain a second data set;

[0137] The sending module 1120 is configured to send, by using the second network card, the first data set to the first node.

[0138] The execution module 1110 and the sending module 1120 can be implemented by software or by hardware. The implementation of the execution module 1110 and the sending module 1120 can refer to the implementation of the acquisition module 1010, and details are described above.

[0139] It should be noted that in other embodiments, the execution module 1110 can be configured to perform operations performed by the node 120 in the method shown in FIG. 6, and the sending module 1120 can be configured to perform operations performed by the node 120 in the method shown in FIG. 6. The steps implemented by the execution module 1110 and the sending module 1120 can be specified as needed, and the entire function of the apparatus 1100 can be implemented by the execution module 1110 and the sending module 1120 performing different operations of the node 120 in the method shown in FIG. 6.

[0140] The present application also provides a computing device 1200. As shown in FIG. 12, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other through the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1200.

[0141] The bus 1202 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 12, but it does not mean that there is only one bus or only one type of bus. The bus 1202 can include a path for transmitting information between various components (e.g., the memory 1206, the processor 1204, the communication interface 1208) of the computing device 1200.

[0142] The processor 1204 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0143] The memory 1206 can include volatile memory, such as random access memory (RAM), and non-volatile memory, such as read-only memory (ROM), floppy disk drive, optical disk drive, compact disk drive, hard disk drive, or solid-state drive.

[0144] The executable program code stored in the memory 1206 is executed by the processor 1204 to implement the functions of the aforementioned receiving module 1010 and performing module 1020, respectively, so as to implement the operations performed by the node 110 in the method shown in FIG. 6. That is, the memory 1206 stores instructions for performing the operations performed by the node 110 in the method shown in FIG. 6.

[0145] The communication interface 1208 uses a transceiver module such as, but not limited to, a network interface card or a transceiver, to implement the communication between the computing device 1200 and other devices or communication networks.

[0146] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0147] As shown in FIG. 13, the computing device cluster includes at least one computing device 1200. The memory 1206 in one or more computing devices 1200 in the computing device cluster can store the same instructions for performing the operations performed by the node 110 in the method shown in FIG. 6.

[0148] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster can also respectively store partial instructions for performing the operations performed by the node 110 in the method shown in FIG. 6. In other words, the combination of one or more computing devices 1200 can collectively execute the instructions for performing the operations performed by the node 110 in the method shown in FIG. 6.

[0149] It should be noted that the memories 1206 in different computing devices 1200 in the computing device cluster can store different instructions for performing part of the functions of the apparatus 1000, respectively. That is, the instructions stored in the memories 1206 in different computing devices 1200 can implement the functions of one or more of the receiving module 1010 and the performing module 1020.

[0150] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 14 shows one possible implementation manner. As shown in FIG. 14, two computing devices 1200A and 1200B are connected through a network. Specifically, the computing devices are connected to the network through the communication interfaces in the computing devices. In this kind of possible implementation manner, the memory 1206 in the computing device 1200A stores instructions for performing the functions of the receiving module 1010. Meanwhile, the memory 1206 in the computing device 1200B stores instructions for performing the functions of the performing module 1020.

[0151] It should be understood that the functions of the computing device 1200A shown in FIG. 14 can also be completed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be completed by multiple computing devices 1200.

[0152] Embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manners of the computing device clusters described with reference to FIG. 13 and FIG. 14. The difference is that the memories 1206 in one or more computing devices 1200 in the computing device cluster can store the same instructions for performing the operations performed by the nodes 110 in the method shown in FIG. 6.

[0153] In some possible implementation manners, the memories 1206 in one or more computing devices 1200 in the computing device cluster can also respectively store part of the instructions for performing the operations performed by the nodes 110 in the method shown in FIG. 6. In other words, the combination of one or more computing devices 1200 can collectively execute the instructions for performing the operations performed by the nodes 110 in the method shown in FIG. 6.

[0154] The present application also provides a computing device 1500. As shown in FIG. 15, the computing device 1500 includes a bus 1502, a processor 1504, a memory 1506 and a communication interface 1508. The processor 1504, the memory 1506 and the communication interface 1508 communicate through the bus 1502. The computing device 1500 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1500.

[0155] The implementation manners of the bus 1502, the processor 1504, the memory 1506, and the communication interface 1508 can refer to the implementation manners of the bus 1202, the processor 1204, the memory 1206, and the communication interface 1208 respectively.

[0156] The memory 1506 stores executable program code, and the processor 1504 executes the executable program code to realize the functions of the foregoing execution module 1110 and the sending module 1120 respectively, so as to realize the operations performed by the node 120 in the method shown in FIG. 6. That is, the memory 1506 stores instructions for performing the operations performed by the node 120 in the method shown in FIG. 6.

[0157] The communication interface 1508 uses a transceiver module such as but not limited to a network interface card and a transceiver to realize the communication between the computing device 1500 and other devices or communication networks.

[0158] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0159] As shown in FIG. 16, the computing device cluster includes at least one computing device 1500. The memory 1506 in one or more computing devices 1500 in the computing device cluster can store the same instructions for performing the operations performed by the node 120 in the method shown in FIG. 6.

[0160] In some possible implementation manners, the memory 1506 of one or more computing devices 1500 in the computing device cluster can also respectively store partial instructions for performing the operations performed by the node 120 in the method shown in FIG. 6. In other words, the combination of one or more computing devices 1500 can collectively execute the instructions for performing the operations performed by the node 120 in the method shown in FIG. 6.

[0161] It should be noted that the memory 1506 in different computing devices 1500 in the computing device cluster can store different instructions respectively for performing part of the functions of the apparatus 1100. That is, the instructions stored in the memory 1506 in different computing devices 1500 can realize the functions of one or more of the execution module 1110 and the sending module 1120.

[0162] In some possible implementations, one or more of the computing devices in the cluster of computing devices can be connected through a network. Among others, the network can be a wide area network or a local area network, etc. FIG. 17 illustrates one possible implementation. As shown in FIG. 17, two computing devices 1500A and 1500B are connected through a network. Specifically, the computing devices are connected to the network through a communication interface in each of the computing devices. In this type of possible implementation, the memory 1506 in the computing device 1500A stores instructions for performing the functions of the execution module 1110. Meanwhile, the memory 1506 in the computing device 1500B stores instructions for performing the functions of the sending module 1120.

[0163] It should be understood that the functions of the computing device 1500A shown in FIG. 17 can also be performed by multiple computing devices 1500. Similarly, the functions of the computing device 1500B can also be performed by multiple computing devices 1500.

[0164] The embodiments of the present application also provide another cluster of computing devices. The connection relationship between the computing devices in the cluster of computing devices can be similar to the connection manners of the cluster of computing devices described with reference to FIG. 16 and FIG. 17. The difference is that the memory 1506 in one or more of the computing devices 1500 in the cluster of computing devices can store the same instructions for performing the operations performed by the node 120 in the method shown in FIG. 6.

[0165] In some possible implementations, the memory 1506 in one or more of the computing devices 1500 in the cluster of computing devices can also respectively store partial instructions for performing the operations performed by the node 120 in the method shown in FIG. 6. In other words, the combination of one or more of the computing devices 1500 can collectively execute the instructions for performing the operations performed by the node 120 in the method shown in FIG. 6.

[0166] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device performs the operations performed by the node 110 in the method shown in FIG. 6.

[0167] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device performs the operations performed by the node 120 in the method shown in FIG. 6.

[0168] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be used to store the desired information and that can be accessed by a computing device. The computer readable storage medium can be a magnetic medium, (e.g., a floppy diskette, a hard disk drive, a magnetic tape), an optical medium, (e.g., a Compact Disc (CD), a DVD, an optical tape), solid state medium, (e.g., a flash memory, a solid state drive (SSD) or a cache, etc.) or any suitable combination of the foregoing. The computer readable storage medium includes instructions for causing a computing device to perform the operations described in the method shown in FIG. 6.

[0169] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be used to store the desired information and that can be accessed by a computing device. The computer readable storage medium can be a magnetic medium, (e.g., a floppy diskette, a hard disk drive, a magnetic tape), an optical medium, (e.g., a Compact Disc (CD), a DVD, an optical tape), solid state medium, (e.g., a flash memory, a solid state drive (SSD) or a cache, etc.) or any suitable combination of the foregoing. The computer readable storage medium includes instructions for causing a computing device to perform the operations described in the method shown in FIG. 6.

[0170] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that, A first node applied to a training system, the training system further comprising a second node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the method comprises: The first node receives a first data set output by the second model block from the second node; The first node executes a first task in a training task of the first model block for the first data set; The first node receives a second data set output by the second model block from the second node; In the case that the first node completes the first task for the first data set, the first node executes a second task in the training task of the first model block for the first data set; at the same time, the first node executes the first task for the second data set; wherein the resource requirements of the first task and the second task are different.

2. The method of claim 1, wherein, The first node comprises a first network card and a first processing unit; wherein, The first node receives the first data set output by the second model block from the second node, comprising: receiving the first data set from the second node by using the first network card; The first node executes the first task in the training task of the first model block, comprising: executing the first task by using the first processing unit; wherein the first task is used to copy the first data set from the first network card to the first processing unit; The first node executes the second task in the training task of the first model block, comprising: executing the second task by using the first processing unit; wherein the second task is used to perform the calculation of the first model block for the first data set.

3. The method of claim 1, wherein, The first task is one of a compute-intensive task, a memory-intensive task, and a collective communication task, and the second task is another one of the compute-intensive task, the memory-intensive task, and the collective communication task.

4. The method according to any one of claims 1 to 3, characterized in that, The second node comprises a second network card and a second processing unit, wherein, The second node is configured to: perform the calculation of the second model block for a third data set by using the second processing unit to obtain the first data set; The second node is further configured to: copy the first data set from the second processing unit to the second network card by using the second processing unit; and perform the calculation of the second model block for a fourth data set by using the second processing unit to obtain the second data set; The second node is further configured to: send the first data set to the first node by using the second network card.

5. The method according to any one of claims 1-4, characterized in that, The first node and the second node are located in different clusters.

6. A model training method, comprising: A second node applied to a training system, the training system further comprising a first node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node comprises a second network card and a second processing unit; the method comprises: The second node performs, by using the second processing unit, a calculation of the second model block for third data to obtain a first data set; The second node copies, by using the second processing unit, the first data set from the second processing unit to the second network card; meanwhile, the second node performs, by using the second processing unit, a calculation of the second model block for fourth data to obtain a second data set; The second node transmits, by using the second network card, the first data set to the first node.

7. A model training apparatus characterized by comprising: The first node is applied to a training system, and the training system further includes a second node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the device comprises: a receiving module configured to receive, from the second node, a first data set output by the second model block; an executing module configured to perform, for the first data set, a first task in a training task of the first model block; The receiving module is further configured to receive, from the second node, a second data set output by the second model block; The executing module is further configured to, in a case where the first task is completed for the first data set, perform, for the first data set, a second task in the training task of the first model block; meanwhile, perform the first task for the second data set; wherein the first task and the second task have different resource requirements.

8. The apparatus of claim 7, wherein, The first node comprises a first network card and a first processing unit; wherein The receiving module is configured to receive, from the second node, the first data set by using the first network card; The executing module is configured to perform the first task by using the first processing unit; wherein the first task comprises copying the first data set from the first network card to the first processing unit; The executing module is configured to perform the second task by using the first processing unit; wherein the second task comprises performing a calculation of the first model block for the first data set.

9. The apparatus of claim 7, wherein, The first task is one of a compute-intensive task, a memory-intensive task and a collective communication task, and the second task is another one of the compute-intensive task, the memory-intensive task and the collective communication task.

10. The apparatus of any one of claims 7-9, wherein, The second node comprises a second network card and a second processing unit, wherein The second node is configured to perform, by using the second processing unit, a calculation of the second model block for a third data set to obtain the first data set; The second node is further configured to copy, by using the second processing unit, the first data set from the second processing unit to the second network card; meanwhile, perform, by using the second processing unit, a calculation of the second model block for a fourth data set to obtain the second data set; The second node is further configured to transmit, by using the second network card, the first data set to the first node.

11. The apparatus of any of claims 7-19, wherein, The first node and the second node are located in different clusters.

12. A model training apparatus, comprising: A second node applied to a training system, the training system further comprising a first node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node comprises a second network card and a second processing unit; the apparatus comprises an execution module and a sending module; wherein The execution module is configured to: for third data, perform calculation of the second model block by using the second processing unit to obtain a first data set; The execution module is further configured to: copy the first data set from the second processing unit to the second network card by using the second processing unit; and at the same time, for fourth data, perform calculation of the second model block by using the second processing unit to obtain a second data set; The sending module is configured to: send the first data set to the first node by using the second network card.

13. A cluster of computing devices, characterized in that, comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1-5 or the method of claim 6.

14. A computer-readable storage medium, characterized in that, comprising computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-5 or the method of claim 6.

15. A computer program product comprising instructions, characterized in that, The instructions, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-5 or the method of claim 6.

Images