Model training method and device

Abstract

The invention provides a model training method and device. The method comprises the steps that a first node receives a first data set output by a second model block from a second node; the first node executes a first task in the training tasks of the first model block for the first data set; the first node receives a second data set output by the second model block from the second node; under the condition that the first node completes the first task for the first data set, the first node executes a second task in the training tasks of the first model block for the first data set; meanwhile, the first node executes the first task for the second data set; wherein the resource requirements of the first task and the second task are different. The method can reduce the training time consumption of the model and improve the resource utilization rate of the node.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence (AI), and in particular to a model training method and device. BACKGROUND

[0002] With the pursuit of intelligence of the model by human beings, the parameter quantity of the model is getting larger and larger. Currently, the parameters of some models have reached hundreds of billions, or even thousands of billions. Such a huge amount of parameters makes it difficult for the model to be trained by a single device. The industry adopts a distributed training strategy to train the model in a distributed manner using a large number of training nodes. Among them, pipe line parallelism (PP) only needs to transmit a small amount of activation value between different training nodes, and has a relatively low requirement on the communication performance between the training nodes. Therefore, pipe line parallelism has become a commonly used distributed training strategy.

[0003] In pipe line parallelism, the model is split into multiple serial model blocks, wherein different model blocks are deployed on different training nodes, and the training nodes are used to perform the training task of the model block for a data set of the model. In related technologies, the node serially performs the training task of the model block for different data sets. That is, after the training task of the model block is completed for one data set, the training task of the model block is performed for the next data set. This results in a long training time of the model. SUMMARY

[0004] The present application provides a model training method and device, which can reduce the training time of the model and improve the resource utilization of the node.

[0005] In a first aspect, a model training method is provided, applied to a first node in a training system, the training system further comprising a second node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the method comprises: the first node receiving a first data set output by the second model block from the second node; the first node performing a first task in the training task of the first model block for the first data set; the first node receiving a second data set output by the second model block from the second node; in the case that the first node completes the first task for the first data set, the first node performs a second task in the training task of the first model block for the first data set; at the same time, the first node performs the first task for the second data set; wherein the resource requirements of the first task and the second task are different.

[0006] Through the method, the node can simultaneously perform two or more tasks in the training task of the same model block for two or more data sets respectively, improving the concurrency of the model block training, and accordingly, improving the concurrency of the model training. Moreover, the resource requirements of different tasks in the simultaneously performed two or more tasks are different, and there is no resource competition or other conflicts between the simultaneously performed tasks, which can avoid performance degradation and other problems. In short, the method improves the concurrency of the model training while avoiding performance degradation and other problems, and therefore, the method can save the time consumption of the model training and improve the efficiency of the model training.

[0007] In addition, the node simultaneously performs tasks with different resource requirements, avoids performance degradation, reduces the idle time of multiple resources of the node, and improves the resource utilization of the multiple resources.

[0008] In a possible implementation, the first node includes a first network card and a first processing unit; the first node receives a first data set output by the second model block from the second node, including: receiving the first data set from the second node by using the first network card; performing a first task in the training task of the first model block, including: performing the first task by using the first processing unit; the first task is used for copying the first data set from the first network card to the first processing unit; performing a second task in the training task of the first model block, including: performing the second task by using the first processing unit; the second task is used for performing the calculation of the first model block on the first data set.

[0009] In this implementation, while the processing unit performs the model block calculation on the data set, the processing unit can also copy a new data set from the network card to the processing unit for subsequent calculation, thereby realizing mutual masking of the calculation task and the copying task, saving the training time consumption of the model block, and improving the training efficiency of the model block.

[0010] In a possible implementation, the first task is one of a compute-intensive task, a memory-intensive task, and a collective communication task, and the second task is another one of the compute-intensive task, the memory-intensive task, and the collective communication task.

[0011] The compute-intensive task, the memory-intensive task, and the collective communication task have different resource requirements, where the compute-intensive task requires more computing resources, the memory-intensive task requires more memory resources, and the collective communication task requires more communication resources. Simultaneously performing different tasks in the compute-intensive task, the memory-intensive task, and the collective communication task on different data sets improves the concurrency of the model block training while avoiding performance degradation and other problems, thereby reducing the training time consumption of the model block and improving the training efficiency of the model block.

[0012] In a possible implementation, the second node comprises a second network card and a second processing unit, and the second node is configured to: for the third data set, perform, by using the second processing unit, the calculation of the second model block to obtain the first data set; the second node is further configured to: copy, by using the second processing unit, the first data set from the second processing unit to the second network card; meanwhile, for the fourth data set, perform, by using the second processing unit, the calculation of the second model block to obtain the second data set; and the second node is further configured to: send, by using the second network card, the first data set to the first node.

[0013] In this implementation, the processing unit performs the calculation of the model block to generate and output a new data set while copying the data set output by the model block to the network card. In this way, the copying task and the calculation task are mutually masked, the training time of the model block is saved, and the training efficiency of the model block is improved.

[0014] In a possible implementation, the first node and the second node are located in different clusters.

[0015] Since the first node and the second node are located in different clusters and communicate through the network card, the training task of the model block includes a plurality of tasks with different resource requirements such as the data copying task and the calculation task between the processing unit and the network card. Through this method, the copying task and the calculation task are mutually masked, and the training time of the model block is saved.

[0016] In addition, there can be network fluctuations between the first node and the second node, and the data sets transmitted between the first node and the second node do not completely arrive on time. The training task of the model block can be divided into a plurality of tasks according to different resource requirements. When the first node has not completed the training task for the current data set but receives a new data set, the first node can simultaneously perform different tasks in the plurality of tasks for the current data set and the new data set, so that the task for the new data set can be performed in time, and the training efficiency of the model block is improved.

[0017] In a second aspect, a model training method is provided, which is applied to a second node in a training system, the training system further comprising a first node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node comprises a second network card and a second processing unit; the method comprises: the second node performing, by using the second processing unit, calculation of the second model block for third data to obtain a first data set; the second node copying, by using the second processing unit, the first data set from the second processing unit to the second network card; meanwhile, performing, by using the second processing unit, calculation of the second model block for fourth data to obtain a second data set; and the second node sending, by using the second network card, the first data set to the first node.

[0018] By this method, the processing unit performs calculation of the model block to generate and output a new data set while copying the data set output by the model block to the network card. In this way, the copying task and the calculation task are mutually masked, the training time consumption of the model block is saved, and the training efficiency of the model block is improved.

[0019] In a third aspect, a model training apparatus is provided, which is applied to a first node in a training system, the training system further comprising a second node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the apparatus comprises: a receiving module configured to receive, from the second node, a first data set output by the second model block; an executing module configured to perform, for the first data set, a first task in a training task of the first model block; the receiving module is further configured to receive, from the second node, a second data set output by the second model block; and the executing module is further configured to, in a case where the first task is completed for the first data set, perform, for the first data set, a second task in the training task of the first model block; and meanwhile, perform the first task for the second data set; wherein the first task and the second task have different resource requirements.

[0020] In a possible implementation, the first node comprises a first network card and a first processing unit; wherein the receiving module is configured to receive, from the second node, the first data set by using the first network card; and the executing module is configured to perform the first task by using the first processing unit; wherein the first task comprises copying the first data set from the first network card to the first processing unit; and the executing module is configured to perform the second task by using the first processing unit; wherein the second task comprises performing calculation of the first model block for the first data set.

[0021] In a possible implementation, the first task is one of a compute-intensive task, a memory-intensive task, and a collective communication task, and the second task is another one of the compute-intensive task, the memory-intensive task, and the collective communication task.

[0022] In a possible implementation, the second node comprises a second network card and a second processing unit, and the second node is configured to: perform, by using the second processing unit, calculation of the second model block for the third data set to obtain a first data set; the second node is further configured to: copy, by using the second processing unit, the first data set from the second processing unit to the second network card; meanwhile, perform, by using the second processing unit, calculation of the second model block for a fourth data set to obtain a second data set; and the second node is further configured to: send, by using the second network card, the first data set to the first node.

[0023] In a possible implementation, the first node and the second node are located in different clusters.

[0024] In a fourth aspect, a model training apparatus is provided, which is applied to a second node in a training system, and the training system further comprises a first node; the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node comprises a second network card and a second processing unit; the apparatus comprises an execution module and a sending module; the execution module is configured to: perform, by using the second processing unit, calculation of the second model block for a third data to obtain a first data set; the execution module is further configured to: copy, by using the second processing unit, the first data set from the second processing unit to the second network card; meanwhile, perform, by using the second processing unit, calculation of the second model block for a fourth data to obtain a second data set; and the sending module is configured to: send, by using the second network card, the first data set to the first node.

[0025] In a fifth aspect, a computing device cluster is provided, which comprises at least one computing device, and each computing device comprises a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method provided in the first aspect or the method provided in the second aspect.

[0026] In a sixth aspect, a computer readable storage medium is provided, which comprises computer program instructions; when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the first aspect or the method provided in the second aspect.

[0027] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed by a computing device cluster, causes the computing device cluster to execute the method provided in the first aspect or the method provided in the second aspect.

[0028] The beneficial effects of the third aspect to the seventh aspect can be referred to the beneficial effects of the first aspect or the second aspect described above, and will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1A is a schematic diagram of a model training scheme;

[0030] Figure 1A is a schematic diagram of a model training scheme;

[0031] Figure 2 is a schematic diagram of a training system provided by an embodiment of the present application;

[0032] Figure 3 is a schematic diagram of a node provided by an embodiment of the present application;

[0033] Figure 4 is a schematic diagram of a training device provided by an embodiment of the present application;

[0034] Figure 5 is a schematic diagram of a training device provided by an embodiment of the present application;

[0035] Figure 6 is a flowchart of a model training method provided by an embodiment of the present application;

[0036] Figure 7 is a schematic diagram of a model training method provided by an embodiment of the present application;

[0037] Figure 8 is a flowchart of a model training method provided by an embodiment of the present application;

[0038] Figure 9 is a schematic diagram of a model training method provided by an embodiment of the present application;

[0039] Figure 10 is a structural schematic diagram of a training device provided by an embodiment of the present application;

[0040] Figure 11 is a structural schematic diagram of a training device provided by an embodiment of the present application;

[0041] Figure 12 is a structural schematic diagram of a computing device provided by an embodiment of the present application;

[0042] Figure 13 is a structural schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0043] Figure 14 is a structural schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0044] Figure 15 is a structural schematic diagram of a computing device provided by an embodiment of the present application;

[0045] Figure 16 A structural diagram of a computing device cluster provided by an embodiment of the present application is shown in FIG. 1.

[0046] Figure 17 A structural diagram of a computing device cluster provided by an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0047] The schemes provided by the embodiments of the present application will be described below with reference to the drawings. In the embodiments of the present application, “multiple” refers to two or more, and “multiple types” refers to two or more types. “First”, “second”, and the like are only used to distinguish similar objects, and do not necessarily indicate a specific order or number of objects.

[0048] To facilitate understanding of the schemes provided by the embodiments of the present application, the technical terms that may be involved in the embodiments of the present application are first introduced.

[0049] Model: composed of multiple artificial neural network layers. Usually, the multiple neural network layers are arranged in series, and the output of one neural network layer is the input of the next neural network layer. The neural network layer can be simply referred to as a network layer or a layer.

[0050] Model training: refers to the process of updating the parameters of the model. Generally, the parameters of the model are preset or randomly generated. By updating the parameters of the model, the model learns the ability to process corresponding tasks, such as image recognition, speech recognition, natural language translation, computer game playing, content generation, etc. The model training needs to go through multiple iterations until convergence or reach a predetermined number of training times. Each iteration needs to input a certain amount of data set. The model performs forward calculation on the input data set to obtain the loss function. The model then performs backward calculation on the loss function to obtain the gradient. Then, the gradient is used to update the model parameters.

[0051] Model calculation: refers to the calculation performed by the model on the relevant data set using the parameters of the model. The model calculation can be divided into forward calculation and backward calculation.

[0052] Calculation result: refers to the result obtained by calculating the relevant data set using the parameters of the model. The calculation results of forward calculation and backward calculation both include activation values. In addition, the calculation result of the last layer of the model can include the loss function. The calculation result of the backward calculation also includes the gradient.

[0053] Forward propagation: also known as forward propagation, refers to starting from data input, calculating and passing data through each layer of the model to obtain the output result of the model. The purpose of forward propagation is to calculate the output result of the model. The output result of the model includes the loss function of the model.

[0054] Forward computation refers to the computation performed on layers or blocks in the model during forward propagation. This computation is used to obtain the activation values ​​of the model layers or blocks. Specifically, the forward computation of the last layer or the last block in the model is used to obtain the model's loss function.

[0055] Backpropagation, also known as reverse propagation, refers to the process of obtaining a loss function based on the difference between the model's output and the labels. Then, using this loss function, the gradient is calculated layer by layer in reverse using the chain rule. Specifically, in backpropagation, the activation values ​​calculated backward from the current model block are passed to the next model block. These activation values ​​are used to calculate the gradient and activation values ​​for the next model block. More specifically, the current model block passes its backward-calculated activation values ​​to the next model block, allowing that next model block to calculate its gradient and activation values, and so on. Thus, through backpropagation, each model block obtains the gradient of the loss function, which is used to update the block's parameters. In backpropagation, the previous model block refers to the model block preceding the current model block in the data transmission direction of the backpropagation, and the next model block refers to the model block following the current model block in the data transmission direction of the backpropagation.

[0056] Backward computation refers to the computation performed on layers or blocks of the model during backward propagation. Backward computation is used to obtain gradients based on the loss function obtained from forward computation, and these gradients are then used to update the model parameters.

[0057] Gradient: The gradient refers to the direction in which the loss function value decreases the fastest. It is obtained through backward computation and is a result of that computation. The gradient is actually a vector representing the direction; each element in the gradient represents the rate of change of the loss function value relative to the weights at that position. The gradient is used to update the model's parameters. Specifically, the gradient is multiplied by the learning step size to obtain a product, which is then added to (or subtracted from) the model's parameters to obtain the updated parameters.

[0058] The gradient of a node refers to the gradient obtained by a node when performing calculations on the corresponding dataset, model, or model block.

[0059] Activation values: These are intermediate results generated during model training, representing the outputs of each network layer or model block. The activation values ​​of a network layer or model block are used for computational calculations in the next network layer or model block, generating its activation values. Furthermore, during backpropagation, the activation values ​​of a network layer or model block are used for computational calculations in the next network layer or model block, generating its activation values ​​and gradients.

[0060] Pipeline parallel (PP): is a model distributed training technique, which means different nodes run different layers of the model, and belongs to inter-layer parallelism. That is, in pipeline parallel, the model is serialized and divided by layer as the granularity, and a plurality of model blocks connected in sequence are obtained. During model training, the related calculations of different model blocks are performed by different nodes. Among them, a model block can be referred to as a stage in the model training process, and N model blocks are N stages in the model training process.

[0061] Model block: is a neural network block obtained by inter-layer division of the model. A model block can include one or more neural network layers.

[0062] Stage: is a model block in the pipeline parallel technique.

[0063] Tensor parallel (TP): is another model distributed training technique, which means that the tensors in the model are split to obtain a plurality of tensor sets of the model, and then different tensor sets in the plurality of tensor sets are deployed to different training nodes for training. Among them, each tensor set includes at least one tensor.

[0064] Training task of a model block: the tasks related to the model block in the model training process can be collectively referred to as the training task of the model block, which can include tasks such as obtaining the dataset of the model block, performing related calculations (such as forward calculation and backward calculation) of the model block on the dataset, and issuing the new dataset obtained by calculation. The resources required by different stages of the training task can be different, and the training task can be divided into different tasks based on the requirement for resources. For example, the node where the model block is located receives or sends the dataset through the network card, performs calculations on the dataset through the processing unit, and the model block obtains and issues the dataset, which involves data copying between the network card and the processing unit, and requires more communication resources between the network and the processing unit. However, the processing unit requires more computing resources for calculations on the dataset. Therefore, the training task of the model block can be divided into copying tasks and calculation tasks. Among them, the copying task is used for data copying between the network card and the processing unit, and the calculation task is used for the processing unit to perform calculations of the model block on the dataset. For another example, some calculations (such as multiplication calculation of tensors) in the related calculations of the model block on the dataset require more computing resources, some calculations (such as addition calculation of tensors) require more memory resources, and some calculations (such as parameter exchange) require collective communication resources. Therefore, the training task of the model block can be divided into: computation-intensive task, memory-intensive task, and collective communication task.

[0065] Collection communication: is the communication required for model training by using both pipeline parallelism and tensor parallelism. Specifically, a tensor is taken as a granularity, and a same model block in pipeline parallelism is split to obtain multiple tensor collections of the model block, where each tensor collection includes at least one tensor. Then, different tensor collections in the multiple tensor collections are deployed to different processing units in a node for training. That is, the tensors in the same model block are distributed in multiple processing units for training. The multiple processing units form a processing unit collection of the model block. During training of the model block or during execution of a computing task of the model block, parameters need to be exchanged between the multiple processing units. The collection communication is specifically the communication involved in exchanging the parameters, and requires more communication resources between the processing units.

[0066] Dataset: is a collective name of a training set of a model and an intermediate computing result (such as an activation value) of the model. The input and output of the model or the model block can be collectively referred to as a dataset.

[0067] Micro-batch: is a fine-grained dataset. In pipeline parallelism, if a dataset input to a model block each time is large (such as a mini-batch), the time required for the calculation of the model block on the dataset is also long, resulting in low concurrency of pipeline parallelism. In order to improve the concurrency of pipeline parallelism, the large dataset (such as a mini-batch) can be further split to obtain a dataset with finer granularity. The dataset with finer granularity is referred to as a micro-batch.

[0068] Region: is a computing node deployment unit in an infrastructure, which is a deployment unit divided from the dimensions of geographical location and network latency. The isolation level between regions is high. The computing nodes in a same region share common services such as elastic computing, block storage, object storage, VPC network, elastic public network IP, and image.

[0069] Availability zone (AZ): is a computing node deployment unit in an infrastructure, and an AZ is a collection of one or more physical data centers with independent wind, fire, water, and electricity. An AZ includes a data center or multiple data centers in close geographical locations. A region can include multiple AZs.

[0070] Training node: can be referred to as node, refers to a computing node used for performing model training or model calculation. The node can include at least one processing unit and a network card. The processing unit can be a neural processing unit (NPU), a graphics processing unit (GPU), a tensor processing unit (TPU), etc. The network card of the node is used for communication between the node and other nodes in the cluster. Common nodes include hosts, virtual machines (VMs). The host can be a server or other physical computing device.

[0071] Data center network (DCN): a connection network between clusters, used to connect nodes in multiple clusters together. The DCN has scalability and high efficiency, and can connect tens or even hundreds of thousands of nodes to meet the growing connection needs of cloud computing.

[0072] Parameter plane network: a high-speed network specially designed for distributed training. Nodes in the same cluster can be connected through the parameter plane network. When the node is a host, the processing units of two hosts in the same cluster can communicate through the parameter plane network without going through the network card of the host. Generally, the communication rate of the parameter plane network can reach 200GE.

[0073] Cluster: also known as computing cluster or training cluster, a cluster composed of multiple nodes. Nodes in the same cluster are located in a similar location, such as in the same region or AZ. The communication capability between nodes in the same cluster is strong, that is, the communication capability between nodes in the same cluster is greater than the communication capability between nodes in the cluster and nodes outside the cluster. For example, the network between nodes in the same cluster is a parameter plane network or a parameter plane-like network, and the network between nodes in the cluster and training nodes outside the cluster is a DCN or a DCN-like network. In addition, in the embodiments of the present application, nodes in the same region or AZ are not necessarily in the same cluster. This is because some nodes in the same region or AZ may not use the parameter plane network or the parameter plane-like network, resulting in poor communication capability between the node and other nodes in the region or AZ. It can be considered that the node and the other nodes in the region or AZ are not in the same cluster, but are in different clusters.

[0074] Neural Processing Unit (NPU): A hardware processor specifically designed to accelerate neural network computations. Compared with Central Processing Unit (CPU) and GPU, NPU optimizes matrix multiplication, convolution operations and other calculations in neural networks, with higher computational efficiency and energy efficiency. NPU can be applied to artificial intelligence and deep learning tasks such as image recognition, natural language processing and speech recognition.

[0075] Stream: A task stream composed of a series of asynchronous tasks in a node, which are executed on the processing unit according to the calling order on the CPU side. The tasks in the same stream are executed serially, and the asynchronous tasks on different streams can be executed in parallel.

[0076] The number of parameters of a model is constantly expanding, and the number of parameters of some models has reached hundreds of billions or even trillions. Moreover, the size of the training set of the model is also expanding, and training sets of hundreds of terabytes (TB) or even petabytes (PB) are not uncommon, which makes the model training have huge demands for computing resources, power, heat dissipation capacity, etc. A single geographical location cluster is difficult to meet this demand. The industry seeks to use clusters scattered in different geographical locations to train models. Due to distance restrictions, it is difficult to deploy high-performance networks between different clusters, making the communication performance between clusters weaker than the communication performance within clusters. Since pipeline parallelism only needs to transmit a small amount of activation value between different nodes, the communication performance between nodes is less demanding. Therefore, when training a model using multiple clusters, pipeline parallelism has become a commonly used distributed training strategy.

[0077] In pipeline parallelism, the model is split into multiple serial model blocks, where a model block is deployed to a node or a set of nodes. The node where the first model block in the multiple model blocks is located obtains the training set in turn with micro-batch as the granularity, and for each micro-batch, performs the training task of the first model block, generates a data set (intermediate result of the model) corresponding to the micro-batch, and sends the data set to the node where the second model block is located. The node where the second model block is located performs the training task of the second model block for the data set, generates a new data set corresponding to the micro-batch, and sends the new data set to the node where the next model block is located, and so on, until the last model block in the node outputs the final data set.

[0078] In related technologies, such as Figure 1AAs shown, each node adopts one stream, and performs the task of a model block for each micro-batch of data set. That is, the node performs the training task of a model block for a next data set only after completing the training task of the model block for a data set. More specifically, the training task of a model block includes a copy task of copying a data set from a network card to a processing unit and a calculation task of performing calculation of a model block for the data set. In this technology, as shown, the node performs the copy task and the calculation task for a next data set in sequence only after completing the copy task and the calculation task for a data set. This results in long training time of the model and low resource utilization of the node. Figure 1A As shown, the node performs the copy task and the calculation task for a next data set in sequence only after completing the copy task and the calculation task for a data set. This results in long training time of the model and low resource utilization of the node.

[0079] In addition, the network between the nodes where the adjacent model blocks are located can fluctuate, causing the node to receive a current data set while the training task of a model block performed by the node for a previous data set has not been completed. This has to wait for the training task of the node for the previous data set to be completed before performing the training task of the model block for the current data set. Specifically, it can be set that the model blocks deployed by the node A1 and the node A2 are adjacent model blocks. Influenced by the network fluctuation between the node A1 and the node A2, the node A2 receives a current data set at time t0, but the node A2 completes the training task for a previous data set at time t1. In this technology, the node A2 has to wait until time t1 before starting to perform the training task for the current data set. This also results in long training time of the model and low resource utilization of the node.

[0080] Embodiments of the present application provide a model training method. The method can be applied to pipeline parallelism and can improve the model training efficiency of pipeline parallelism and the resource utilization of a node. In the method, the training task of a model block can be divided based on the required resources to obtain a plurality of tasks, wherein different tasks in the plurality of tasks require different resources. A node can perform one task in the plurality of tasks for a data set, and at the same time, the node performs another task in the plurality of tasks for another data set. In this way, the node can perform tasks for different data sets in parallel, and different tasks require different resources, there is no conflict between different tasks of the node, and the simultaneous execution of two or more tasks does not cause performance degradation of the node, so the training time of the model can be saved and the training efficiency of the model can be improved. In addition, the simultaneous execution of two or more tasks by the node can make full use of the various resources of the node and improve the resource utilization of the node.

[0081] Next, the model training method provided by the embodiments of the present application is introduced.

[0082] Figure 2A training system 100 that can be used to implement the method is shown. The training system 100 includes a plurality of nodes 110, 120, etc. The plurality of nodes are used to train a target model in a pipelined parallel manner. That is, the target model can be divided into a plurality of model blocks in series, the plurality of model blocks and the plurality of nodes correspond one-to-one, and the model blocks are deployed in the nodes corresponding to the model blocks. The nodes can execute the training tasks of the model blocks deployed in the nodes for the data set of the target model.

[0083] For ease of description, the model block deployed in the node 110 can be referred to as model block 210, and the model block deployed in the node 120 can be referred to as model block 220. Among them, the model block 210 and the model block 220 are adjacent in the target model. That is, in one of the forward propagation and the backward propagation, the data set (such as the activation value) of the output of the model block 110 is the input of the model block 120. In the other of the forward propagation and the backward propagation, the data set (such as the activation value) of the output of the model block 120 is the input of the model block 110. This makes the node 110 and the node 120 need to interact the data set.

[0084] In some embodiments, as shown in Figure 2 The node 110 and the node 120 can be located in different clusters, for example, the node 110 is located in the cluster B1, and the node 120 is located in the cluster B2. The node 110 and the node 120 can be connected through a DCN network, for example. The node 110 and the node 120 interact data through a transmission control protocol (TCP), for example.

[0085] In some embodiments, as shown in Figure 3 The node 110 and the node 120 can each include at least one processing unit and a network card. Among them, the processing unit is used to execute the training task of the model block, and the network card is used for sending or receiving the data set. Taking the node 120 sending the data set to the node 110 as an example, the processing unit in the node 120 executes the training task of the model block 220 for the data set obtained by the node 120, and obtains the data set output by the model block 220. The network card in the node 120 is used to send the data set output by the model block 220 to the network card of the node 110. The processing unit of the node 110 can execute the training task of the model block 210 for the data set output by the model block 220, and obtain the data set output by the model block 210.

[0086] In some embodiments, the processing units in node 110 and / or node 120 may include a computing module and a data transmission module. The computing module can perform computing tasks, such as computing model blocks. The data transmission module can perform data transmission tasks, such as data copying. The computing module and the data transmission module can operate simultaneously, each performing its own task.

[0087] In the following description, when no special distinction is made between node 110 and node 120, they may be referred to simply as nodes.

[0088] like Figure 3 As shown, a node may include a training device. In some embodiments, the training device in a node may be a software device running on the node's CPU or processing unit. Under the action of the training device, the node can simultaneously perform training tasks for model blocks on two or more datasets. For example, Figure 4 As shown, the training device may include a task profiling module and a task scheduling module.

[0089] The task profiling module is used to analyze the tasks (i.e., training tasks) during the training process of the target model and to classify these training tasks. For example... Figure 4 As shown, the task profiling module can identify task types including copy tasks and computation tasks. In some embodiments, such as Figure 5 As shown, computational tasks can be further categorized into computationally intensive tasks, memory-intensive tasks, and collective communication tasks. The task profiling module can extract features of tasks under each task type, including estimated execution time, resource requirements, and data dependencies between tasks. Based on this information, the task profiling module generates labels for tasks under each task type. The task profiling module can also assign priorities based on task importance. For example, task importance is determined by the task's dataset. Specifically, tasks executed on the previous dataset are more important than tasks executed on the current dataset. If the tasks to be processed include tasks executed on the previous dataset and tasks executed on the current dataset, and these two tasks conflict (e.g., require the same resources), then the task executed on the previous dataset is executed first. The task profiling module can also collect historical execution data for tasks under each task type. Historical execution data can be used to improve future task classification and feature extraction. The aforementioned information can be passed to the task scheduling module to provide a basis for task scheduling.

[0090] The task scheduling module can instruct the processing unit to execute tasks using multiple streams. The task scheduling module can manage the multiple streams and assign tasks to the multiple streams. The task scheduling module can assign tasks to be executed to corresponding streams using information obtained from the task profiling module, for example, assign tasks with data dependency to the same stream to ensure correct execution order of the tasks. Assign tasks without data dependency to different streams to enable concurrent execution of the tasks. Here, tasks with data dependency refer to tasks corresponding to the same data set, and tasks without data dependency refer to tasks corresponding to different data sets. For example, as shown in FIG. 2, the copy task of data set D1 is assigned to stream C1, and the compute task of data set D2 is assigned to stream C2. Figure 4

[0091] The task scheduling module can identify tasks in conflict based on resource requirements of the tasks. When the resource requirements of two tasks are the same, it can be considered that the two tasks are in conflict. When the resource requirements of two tasks are different, it can be considered that the two tasks are not in conflict. When there are multiple tasks to be executed and multiple tasks are in conflict, the task scheduling module can control the execution order of the tasks based on the priority of the tasks. Here, tasks with high priority are executed first. When there are multiple tasks to be executed and multiple tasks are not in conflict, the multiple tasks can be controlled to be executed simultaneously. For example, as shown in FIG. 2, the copy task of data set D1 and the compute task of data set D2 are not in conflict and can be executed simultaneously. For another example, as shown in FIG. 3, the compute task E1 of data set D1 and the compute task E2 of data set D2 are not in conflict and can be executed simultaneously. Here, compute task E1 is one of a compute-intensive task, a memory-intensive task, and a collective communication task, and compute task E2 is another of a compute-intensive task, a memory-intensive task, and a collective communication task. Figure 4 Figure 5

[0092] In some embodiments, the task scheduling module can control the number of tasks assigned to each stream to balance the load of each stream and provide overall execution efficiency of the tasks.

[0093] In this way, the task scheduling module can control concurrent execution of tasks of different data sets and adjust the execution order of the tasks during model training, reduce training time consumption, improve resource utilization, ensure that tasks of previous data sets are executed first, and ensure that tasks with data dependency are executed in order.

[0094] ​​​The above example introduces the training system 100 provided by the embodiments of the present application. Next, taking implementation in the training system 100 as an example, the example introduces the flow of the model training method provided by the embodiments of the present application. As shown in Figure 6 The method includes the following steps.

[0095] In step 601, the node 110 receives the data set D1 output by the model block 220 from the node 120.

[0096] The data set D1 is obtained by the node 120 performing the calculation task of the model block 220 on the data set D1'. Specifically, the node 120 deploys the model block 220, the node 120 inputs the data set D1' into the model block 220, the model block 220 invokes the related resources of the node 120 to perform calculation, and obtains and outputs the data set D1.

[0097] If the model block 220 is the previous model block of the model block 210 in the forward propagation, the data set D1 is obtained by the forward calculation of the model block 220. If the model block 220 is the previous model block of the model block 210 in the backward propagation, the data set D1 is obtained by the backward calculation of the model block 220.

[0098] In some embodiments, the model block 220 can be the first model block in the plurality of model blocks obtained by dividing the target model, and the data set D1' is a micro-batch of the target model, that is, a training set of micro-batch granularity. In some embodiments, the model block 220 is not the first model block in the plurality of model blocks obtained by dividing the target model, and the data set D1' is the data set output by the previous model block of the model block 220.

[0099] In some embodiments, as shown in Figure 6 The node 120 can include a processing unit 121 and a network card 122. The node 120 performs step 501 by using the processing unit 121 to perform the calculation of the model block 220 on the data set D1', that is, the calculation task in the training task of the model block 220, and obtains the data set D1. The calculation is forward calculation or backward calculation. Next, the node 120 performs step 502a by using the processing unit 121 to copy the data set D1 from the processing unit 121 to the network card 122. In other words, in step 502a, the processing unit 121 performs the copy task in the training task of the model block 220 on the data set D1. The copy task is used to copy the data set D1 from the processing unit 121 to the network card 122. After the copy task is completed, the network card 122 is used to send the data set D1 to the node 110.

[0100] The copy task can also be referred to as an input / output (I / O) task. In one example of the embodiment, the processing unit 121 can specifically copy the data set D1 to the system memory of the node 120, and the network card 122 can obtain the data set D1 from the system memory.

[0101] If the processing unit 121 obtains the data set D2’ before or during the execution of the copy task for the data set D1, the node 120 can execute step 502b to perform the computation of the model block 220 (i.e., the computation task in the training task of the model block 220) for the data set D2’ using the processing unit 121, to obtain the data set D2.

[0102] The data set D2’ is the next data set of the data set D1’. For example, if the data set D1’ and the data set D2’ are respectively a micro-batch of a target model, the data set D2’ is the next micro-batch of the micro-batch of the data set D1’. For another example, if the data set D1’ and the data set D2’ are respectively data sets output by a previous model block of the model block 220, the data set D2’ is the next data set output by the previous model block of the model block 220 after the output of the data set D1’.

[0103] After the execution of step 502b (i.e., the execution of the computation task of the model block 220 for the data set D2’), the node 120 can execute step 503 to copy the data set D2 from the processing unit 121 to the network card 122 using the processing unit 121. That is, in step 503, the copy task in the training task of the model block 220 is performed for the data set D2 to copy the data set D2 to the network card 122. At the same time, the processing unit 121 can also perform the computation of the model block 220 for the next data set of the data set D2’. The same applies to the subsequent data sets.

[0104] The computation task of the model block requires computing resources, and the copy task requires communication resources. The two tasks require different resources, and thus the processing unit 121 can concurrently execute the computation task for one data set and the copy task for another data set, to improve the concurrency of the task execution without causing performance degradation.

[0105] In some embodiments, the processing unit can concurrently execute the computation task and the copy task for different data sets using different streams.

[0106] Compared to nodes sequentially executing computation tasks for a dataset and copying tasks for the next dataset, nodes executing computation tasks for a dataset and copying tasks for the next dataset in parallel allows the time spent on computation and copying tasks to mask each other, thereby shortening the training time of the model.

[0107] See Figure 7 The target model can be configured to be split into model block 220 and model block 210, with model block 220 preceding model block 210 in the forward propagation. The training set of the target model can also be configured to be split into three micro-batches, with a single forward computation taking 8 units of time, a single backward computation taking 16 units of time, a single copy task taking 1 unit of time, and the time for transferring the dataset corresponding to one micro-batch between nodes 120 and 110 taking 2 units of time. Using the Gpipe pipelined parallel algorithm, the training iteration takes 84 units of time. Using the model training method provided in this embodiment (i.e., nodes execute computation tasks for the dataset and copy tasks for the next dataset in parallel), the training iteration takes 80 units of time. This saves 4 units of time, i.e. Figure 7 T1 in the text represents 4 units of time.

[0108] Furthermore, the time saved is positively correlated with the number of micro-batches. As the number of micro-batches increases, the model training time decreases further.

[0109] Continue reading Figure 6 Node 110 can execute steps 602 and 603. In step 602, node 110 executes task F1 of the training task of model block 210 for dataset D1. In step 603, node 110 receives dataset D2 output by model block 220 from node 120. Dataset D2 is obtained by node 120 after it outputs dataset D1, by executing the computation task of model block 220. See the above description for details, which will not be repeated here.

[0110] Next, node 110 can execute step 604. After completing task F1 for dataset D1, node 110 executes task F2 in the training task of model block 210 for dataset D1. At the same time, node 110 executes task F1 of model block 210 for dataset D2. The resource requirements of tasks F1 and F2 are different.

[0111] In some embodiments, such as Figure 8As shown, the node 110 comprises a processing unit 111 and a network card 112. The model block 210 is deployed in the processing unit 111, and the node 110 performs the computing task of the model block 210 by using the processing unit 111. The computing task of the model block 210 is the forward computation or the backward computation. The node 110 receives the data set from the node 120 by using the network card 112. Thus, the training task of the model block 210 comprises the copying task of copying the data set from the network card 112 to the processing unit 111 and the computing task of the model block 210.

[0112] In this embodiment, the task F1 can be the copying task of copying the data set from the network card 112 to the processing unit 111, and the task F2 can be the computing task of the model block 210 for the data set. That is, after the copying task of copying the data set D1 from the network card 112 to the processing unit 111 is completed, the node 110 can perform the computing task of the model block 210 for the data set D1 by using the processing unit 111, while the node 110 can perform the copying task of copying the data set D2 from the network card 112 to the processing unit 111 by using the processing unit 111. Thus, the node 110 can concurrently perform the related tasks for two or more data sets, saving the training time of the target model and improving the resource utilization of the node 110.

[0113] In some embodiments, as described above, the training task of the model block can be divided into the computation-intensive task, the memory-intensive task, and the collective communication task. The computation-intensive task, the memory-intensive task, and the collective communication task require different resources, in which the computation-intensive task requires more computing resources, the memory-intensive task requires more memory resources, and the collective communication task requires more communication resources between the processing units. The task F1 can be one of the computation-intensive task, the memory-intensive task, and the collective communication task, and the task F2 can be another of the computation-intensive task, the memory-intensive task, and the collective communication task, that is, the task F1 and the task F2 are different tasks of the computation-intensive task, the memory-intensive task, and the collective communication task.

[0114] In this embodiment, when the network between the node 110 and the node 120 fluctuates, and the data set cannot reach the node 120 on time or even cannot reach the node 120 as required, the node 110 does not need to wait for the training task for the preceding data set to be completed before performing the training task for the subsequent data set. For example, the node 110 obtains the data set D2 before the training task for the data set D1 is completed, in which case the node 110 can simultaneously perform the task F1 for the data set D1 and the task F2 for the data set D2. For example, Figure 9As shown, at time t2, the node 110 performs a memory-intensive task (task F1) for the data set D1, and performs a compute-intensive task (task F2) for the data set D2.

[0115] Thus, as Figure 9 shown, in each iteration, relative to Figure 1B the scheme shown, the embodiment can save a time length of T2, thereby saving the training time of the model and improving the resource utilization of the node 110.

[0116] In some embodiments, the node 110 includes a processing unit 111 and a network card 112. The node 110 can use the processing unit 111 to perform the compute task of the model block 210. The node 110 can use the network card 112 to receive the data set from the node 120. The training task of the model block 210 includes the copy task of copying the data set from the network card 112 to the processing unit 111 and the compute task of the model block 210. The compute-intensive task, the memory-intensive task, and the collective communication task all belong to the compute task in the training task of the model block. The task F1 can be the copy task for copying the data set from the network card 112 to the processing unit 111, and the task F2 can be any one of the compute-intensive task, the memory-intensive task, and the collective communication task.

[0117] In summary, through the model training method provided by the embodiments of the present application, the node can perform tasks for different data sets in parallel, and the resource requirements of two or more tasks performed at the same time are different and do not conflict with each other, and the simultaneous execution of two or more tasks does not cause performance degradation of the node, so that the training time of the model can be saved and the training efficiency of the model can be improved. In addition, since the resource requirements of different tasks are different, the simultaneous execution of two or more tasks by the node can make full use of various resources of the node, thereby improving the resource utilization of the node.

[0118] The embodiments of the present application provide a model training device 1000. The device 1000 is applied to a first node in a training system, and the training system further includes a second node; wherein the first node is deployed with a first model block of a target model, and the second node is deployed with a second model block of the target model; in the target model, the first model block and the second model block are adjacent. As Figure 10 shown, the device 1000 includes:

[0119] The receiving module 1010 is configured to receive a first data set output by the second model block from the second node;

[0120] The execution module 1020 is configured to perform a first task in the training task of the first model block for the first data set;

[0121] The receiving module 1010 is further configured to receive, from the second node, a second data set output by the second model block;

[0122] The execution module 1020 is further configured to, in a case where the first task is completed for the first data set, perform a second task in the training task of the first model block for the first data set, while performing the first task for the second data set; wherein the resource requirements of the first task and the second task are different.

[0123] In some embodiments, the first node comprises a first network card and a first processing unit; wherein the receiving module 1010 is configured to receive the first data set from the second node by using the first network card; the execution module 1020 is configured to perform the first task by using the first processing unit; wherein the first task comprises copying the first data set from the first network card to the first processing unit; the execution module 1020 is configured to perform the second task by using the first processing unit; wherein the second task comprises performing the calculation of the first model block for the first data set.

[0124] In some embodiments, the first task is one of a compute-intensive task, a memory-intensive task, and a collective communication task, and the second task is another one of the compute-intensive task, the memory-intensive task, and the collective communication task.

[0125] In some embodiments, the second node comprises a second network card and a second processing unit, wherein the second node is configured to perform, for a third data set, the calculation of the second model block by using the second processing unit to obtain the first data set; the second node is further configured to copy, by using the second processing unit, the first data set from the second processing unit to the second network card; while performing, for a fourth data set, the calculation of the second model block by using the second processing unit to obtain the second data set; the second node is further configured to send, by using the second network card, the first data set to the first node.

[0126] In some embodiments, the first node and the second node are located in different clusters.

[0127] The receiving module 1010 and the execution module 1020 can be implemented by software or by hardware. For example, the implementation of the receiving module 1010 is described below. Similarly, the implementation of the execution module 1020 can refer to the implementation of the receiving module 1010.

[0128] As an example of a software functional unit, the receiving module 1010 can include code running on a compute instance. The compute instance can include at least one of a physical host (computing device), a virtual machine, a container. Further, the compute instance can be one or more. For example, the receiving module 1010 can include code running on multiple hosts / virtual machines / containers. It is noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Typically, a region can include multiple AZs.

[0129] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Typically, a VPC is set up within a region, and communication between two VPCs in the same region, or between VPCs in different regions, requires a communication gateway in each VPC to enable interconnection between the VPCs.

[0130] As an example of a hardware functional unit, the receiving module 1010 can include at least one computing device, such as a server, etc. Alternatively, the receiving module 1010 can also be a device implemented using an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), etc. The PLD can be implemented as a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0131] The multiple computing devices included in the receiving module 1010 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the receiving module 1010 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the receiving module 1010 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0132] It should be noted that, in other embodiments, the receiving module 1010 can be used to perform... Figure 6 The operation performed by node 110 in the method shown can be executed by execution module 1020. Figure 6 The operation performed by node 110 in the method shown. The operations implemented by the receiving module 1010 and the execution module 1020 can be specified as needed, and are implemented by the receiving module 1010 and the execution module 1020 respectively. Figure 6 The different operations performed by node 110 in the method shown achieve all the functions of device 1000.

[0133] This application also provides a model training device 1100. The device 1100 is applied to a second node in a training system, which further includes a first node; wherein the first node deploys a first model block of the target model, and the second node deploys a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node includes a second network interface card (NIC) and a second processing unit. Figure 11 As shown, the device 1100 includes: an execution module 1110 and a transmission module 1120; wherein,

[0134] The execution module 1110 is used to: for the third data, use the second processing unit to perform the calculation of the second model block to obtain the first dataset;

[0135] The execution module 1110 is further configured to: use the second processing unit to copy the first dataset from the second processing unit to the second network card; and simultaneously, for the fourth data, use the second processing unit to perform the calculation of the second model block to obtain the second dataset;

[0136] The sending module 1120 is used to: send the first dataset to the first node using the second network card.

[0137] Both the execution module 1110 and the sending module 1120 can be implemented in software or in hardware. The implementation methods of the execution module 1110 and the sending module 1120 can be referenced from the implementation method of the acquisition module 1010, as described above, and will not be repeated here.

[0138] It should be noted that, in other embodiments, the execution module 1110 can be used to execute... Figure 6 The operations performed by node 120 in the method shown can be executed by the sending module 1120. Figure 6 The operation performed by node 120 in the method shown. The steps implemented by execution module 1110 and sending module 1120 can be specified as needed, and are implemented by execution module 1110 and sending module 1120 respectively. Figure 6 The different operations performed by node 120 in the method shown achieve all the functions of device 1100.

[0139] This application also provides a computing device 1200. For example... Figure 12 As shown, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0140] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1202 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1202 may include a path for transmitting information between various components of the computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0141] Processor 1204 may include a central processing unit (CPU) and a graphics processing unit (GPU).

[0142] Processing unit (GPU), microprocessor (MP), or digital signal processor (DSP) are any one or more of these processors.

[0143] The memory 1206 may include volatile memory, such as random access memory (RAM). The memory 1206 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD).

[0144] The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the functions of the aforementioned receiving module 1010 and execution module 1020, thereby achieving... Figure 6 The operation performed by node 110 in the method shown. That is, the memory 1206 stores the operations used for execution. Figure 6 The instructions for the operation executed by node 110 in the method shown.

[0145] The communication interface 1208 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0146] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0147] like Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same memory for executing... Figure 6 The instructions for the operation executed by node 110 in the method shown.

[0148] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store memory for execution. Figure 6The instructions for the operations performed by the node 110 in the method shown. Figure 6 The instructions for the operations performed by the node 110 in the method shown.

[0149] It should be noted that the memories 1206 in different computing devices 1200 in the computing device cluster can store different instructions for performing part of the functions of the apparatus 1000 respectively. That is, the instructions stored in the memories 1206 in different computing devices 1200 can implement the functions of one or more of the receiving module 1010 and the performing module 1020.

[0150] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network, a local area network, or the like. Figure 14 A possible implementation manner is shown. As shown in Figure 14 As shown, two computing devices 1200A and 1200B are connected through a network. Specifically, the computing devices are connected to the network through the communication interfaces in the computing devices. In this type of possible implementation manner, the memory 1206 in the computing device 1200A stores instructions for performing the functions of the receiving module 1010. Meanwhile, the memory 1206 in the computing device 1200B stores instructions for performing the functions of the performing module 1020.

[0151] It should be understood that Figure 14 The functions of the computing device 1200A shown in the foregoing can also be completed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be completed by multiple computing devices 1200.

[0152] Embodiments of the present application further provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection relationship between the computing devices in the computing device cluster shown in Figure 13 and Figure 14 The connection manner of the computing device cluster. The difference is that the memory 1206 in one or more computing devices 1200 in the computing device cluster can store the same instructions for performing Figure 6 The instructions for the operations performed by the node 110 in the method shown.

[0153] In some possible implementation manners, the memory 1206 in one or more computing devices 1200 in the computing device cluster can also respectively store instructions for performing part of the operations performed by the node 110 in the method shown. That is, a combination of one or more computing devices 1200 can collectively execute instructions for performing the operations performed by the node 110 in the method shown. Figure 6 The instructions for the operations performed by the node 110 in the method shown. Figure 6 The instructions for the operations performed by the node 110 in the method shown.

[0154] The present application also provides a computing device 1500. As shown in Figure 15 The computing device 1500 includes a bus 1502, a processor 1504, a memory 1506 and a communication interface 1508. The processor 1504, the memory 1506 and the communication interface 1508 communicate with each other through the bus 1502. The computing device 1500 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 1500 is not limited.

[0155] The implementation of the bus 1502, the processor 1504, the memory 1506 and the communication interface 1508 can refer to the implementation of the bus 1202, the processor 1204, the memory 1206 and the communication interface 1208 respectively.

[0156] The memory 1506 stores executable program codes, and the processor 1504 executes the executable program codes to realize the functions of the aforementioned execution module 1110 and the sending module 1120 respectively, thereby realizing the operations performed by the node 120 in the method shown in Figure 6 The memory 1506 stores executable program codes, and the processor 1504 executes the executable program codes to realize the functions of the aforementioned execution module 1110 and the sending module 1120 respectively, thereby realizing the operations performed by the node 120 in the method shown in Figure 6 The memory 1506 stores executable program codes, and the processor 1504 executes the executable program codes to realize the functions of the aforementioned execution module 1110 and the sending module 1120 respectively, thereby realizing the operations performed by the node 120 in the method shown in

[0157] The communication interface 1508 uses a transceiver module such as but not limited to a network interface card and a transceiver to realize the communication between the computing device 1500 and other devices or communication networks.

[0158] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer or a smart phone.

[0159] As shown in Figure 16 The computing device cluster includes at least one computing device 1500. The memory 1506 in one or more computing devices 1500 in the computing device cluster can store the same instructions for performing the operations performed by the node 120 in the method shown in Figure 6 The computing device cluster includes at least one computing device 1500. The memory 1506 in one or more computing devices 1500 in the computing device cluster can store the same instructions for performing the operations performed by the node 120 in the method shown in

[0160] In some possible implementations, the memory 1506 of one or more computing devices 1500 in the computing device cluster can also respectively store part of the instructions for performing the operations performed by the node 120 in the method shown in Figure 6 In some possible implementations, the memory 1506 of one or more computing devices 1500 in the computing device cluster can also respectively store part of the instructions for performing the operations performed by the node 120 in the method shown in Figure 6instructions for the operations performed by the node 120 in the method shown.

[0161] It should be noted that the memories 1506 in different computing devices 1500 in the computing device cluster can store different instructions for performing part of the functions of the apparatus 1100, respectively. That is, the memories 1506 in different computing devices 1500 can store instructions for implementing the functions of one or more of the execution module 1110 and the sending module 1120.

[0162] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network, a local area network, or the like. Figure 17 A possible implementation manner is shown. As shown in Figure 17 As shown, two computing devices 1500A and 1500B are connected through a network. Specifically, the communication interfaces in the computing devices are connected to the network. In this type of possible implementation manner, the memory 1506 in the computing device 1500A stores instructions for performing the functions of the execution module 1110. Meanwhile, the memory 1506 in the computing device 1500B stores instructions for performing the functions of the sending module 1120.

[0163] It should be understood that Figure 17 The functions of the computing device 1500A shown in

[0164] Embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manner of the computing device cluster shown in Figure 16 and Figure 17 The connection manner of the computing device cluster. The difference is that the memory 1506 in one or more computing devices 1500 in the computing device cluster can store the same instructions for performing Figure 6 the operations performed by the node 120 in the method shown.

[0165] In some possible implementation manners, the memory 1506 of one or more computing devices 1500 in the computing device cluster can also respectively store part of the instructions for performing Figure 6 the operations performed by the node 120 in the method shown. In other words, the combination of one or more computing devices 1500 can collectively perform the instructions for performing Figure 6 the operations performed by the node 120 in the method shown.

[0166] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be software or program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to execute Figure 6 the operations performed by node 110 in the method shown.

[0167] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be software or program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to execute Figure 6 the operations performed by node 120 in the method shown.

[0168] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be accessed by a computing device, or a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to execute Figure 6 the operations performed by node 110 in the method shown.

[0169] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be accessed by a computing device, or a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to execute Figure 6 Figure 6 the operations performed by node 120 in the method shown.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that, The method includes: a first node applied to a training system, the training system further including a second node; wherein the first node deploys a first model block of the target model, and the second node deploys a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the method includes: The first node receives the first dataset output by the second model block from the second node; The first node performs the first task in the training task of the first model block for the first dataset; The first node receives the second dataset output by the second model block from the second node; When the first node completes the first task for the first dataset, the first node executes the second task in the training task of the first model block for the first dataset; at the same time, the first node executes the first task for the second dataset; wherein the resource requirements of the first task and the second task are different.

2. The method according to claim 1, characterized in that, The first node includes a first network interface card (NIC) and a first processing unit; wherein, The first node receives the first dataset output by the second model block from the second node, including: receiving the first dataset from the second node using the first network interface card; The first task in the training task of the first model block includes: executing the first task using the first processing unit; wherein the first task is used to copy the first dataset from the first network interface card to the first processing unit; The second task in the training task of the first model block includes: using the first processing unit to execute the second task; wherein the second task is used to compute the first model block for the first dataset.

3. The method according to claim 1, characterized in that, The first task is one of a computationally intensive task, a memory-intensive task, and a collective communication task, and the second task is another of a computationally intensive task, a memory-intensive task, and a collective communication task.

4. The method according to any one of claims 1-3, characterized in that, The second node includes a second network interface card (NIC) and a second processing unit, wherein, The second node is used to: perform the calculation of the second model block using the second processing unit for the third dataset, so as to obtain the first dataset; The second node is also configured to: use the second processing unit to copy the first dataset from the second processing unit to the second network interface card; and simultaneously, for the fourth dataset, use the second processing unit to perform the calculation of the second model block to obtain the second dataset; The second node is also used to: send the first dataset to the first node using the second network interface card.

5. The method according to any one of claims 1-4, characterized in that, The first node and the second node are located in different clusters.

6. A model training method, characterized in that, A second node is applied to a training system, the training system further comprising a first node; wherein the first node deploys a first model block of the target model, and the second node deploys a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node includes a second network interface card and a second processing unit; the method includes: The second node, in response to the third data, uses the second processing unit to perform calculations on the second model block to obtain the first dataset; The second node uses the second processing unit to copy the first dataset from the second processing unit to the second network card; at the same time, for the fourth data, the second processing unit performs the calculation of the second model block to obtain the second dataset; The second node uses the second network interface card to send the first dataset to the first node.

7. A model training device, characterized in that, A first node is applied to a training system, the training system further comprising a second node; wherein the first node deploys a first model block of the target model, and the second node deploys a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the device comprises: The receiving module is used to receive the first dataset output by the second model block from the second node; An execution module is configured to perform the first task in the training task of the first model block for the first dataset; The receiving module is further configured to: receive a second dataset output by the second model block from the second node; The execution module is further configured to: upon completing the first task for the first dataset, execute the second task in the training task of the first model block for the first dataset; simultaneously, execute the first task for the second dataset; wherein the resource requirements of the first task and the second task are different.

8. The apparatus according to claim 7, characterized in that, The first node includes a first network interface card (NIC) and a first processing unit; wherein, The receiving module is used to: receive the first dataset from the second node using the first network interface card; The execution module is configured to: execute the first task using the first processing unit; wherein the first task includes: copying the first dataset from the first network interface card to the first processing unit; The execution module is used to: execute the second task using the first processing unit; wherein the second task includes: calculating the first model block for the first dataset.

9. The apparatus according to claim 7, characterized in that, The first task is one of a computationally intensive task, a memory-intensive task, and a collective communication task, and the second task is another of a computationally intensive task, a memory-intensive task, and a collective communication task.

10. The apparatus according to any one of claims 7-9, characterized in that, The second node includes a second network interface card (NIC) and a second processing unit, wherein, The second node is used to: perform the calculation of the second model block using the second processing unit for the third dataset, so as to obtain the first dataset; The second node is also configured to: use the second processing unit to copy the first dataset from the second processing unit to the second network interface card; and simultaneously, for the fourth dataset, use the second processing unit to perform the calculation of the second model block to obtain the second dataset; The second node is also used to: send the first dataset to the first node using the second network interface card.

11. The apparatus according to any one of claims 7-19, characterized in that, The first node and the second node are located in different clusters.

12. A model training device, characterized in that, A second node is applied to a training system, which further includes a first node; wherein the first node deploys a first model block of the target model, and the second node deploys a second model block of the target model; in the target model, the first model block and the second model block are adjacent; the second node includes a second network interface card and a second processing unit; the device includes an execution module and a transmission module; wherein... The execution module is used to: for the third data, use the second processing unit to perform the calculation of the second model block to obtain the first dataset; The execution module is further configured to: use the second processing unit to copy the first dataset from the second processing unit to the second network card; and simultaneously, for the fourth data, use the second processing unit to perform the calculation of the second model block to obtain the second dataset; The sending module is used to send the first dataset to the first node using the second network card.

13. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-5 or the method as described in claim 6.

14. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-5 or the method as described in claim 6.

15. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1-5 or the method as described in claim 6.

Images