Model training method and apparatus

Abstract

A model training method and apparatus. The method is applied to an accelerator cluster, the cluster comprises a plurality of accelerators, and a training model is deployed on each accelerator. Taking a first accelerator among the plurality of accelerators as an example, the first accelerator is configured to perform a first round of training on the training model to obtain a first parameter set, the parameter set comprising a first-type parameter and a second-type parameter; and the first accelerator is further configured to send the first-type parameter to a second accelerator before the start of a second round of training, and send the second-type parameter to the second accelerator after the start of the second round of training. In this way, an amount of data that needs to be transmitted between the two rounds of training is reduced; that is, only the first-type parameter needs to be transmitted before the start of the second round of training. Therefore, the communication time between the accelerators can be shortened, and communication overhead can be reduced, thereby shortening the duration of an entire model training task, and improving the model training efficiency.

Description

Model training method and device

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 202411290237.8, filed on September 13, 2024, and entitled "A model training method and device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of data processing, and in particular to a model training method and device. BACKGROUND

[0004] With the rapid development of the Internet and the continuous increase in the size of users, recommendation systems play a crucial role in providing personalized user experiences. The rapid development of deep learning technology brings new opportunities and challenges to recommendation systems. Deep learning models can learn more accurate user interest representations from vast amounts of data, thereby improving the effectiveness of recommendation systems. However, with the continuous development of models and the increasing number of parameters, the training efficiency of recommendation models has become a problem to be solved. SUMMARY

[0005] The model training method and device provided by the embodiments of the present application are used to improve the training efficiency of recommendation models.

[0006] In a first aspect, the present application provides an accelerator cluster, which includes a plurality of accelerators, and each accelerator is deployed with a training model. Taking a first accelerator in the plurality of accelerators as an example: the first accelerator is configured to perform a first round of training on the training model to obtain a first parameter set; the first parameter set includes first parameters and second parameters; the first accelerator is further configured to send the first parameters to a second accelerator before the start of a second round of training, and send the second parameters to the second accelerator after the start of the second round of training; wherein the first accelerator is any one of the plurality of accelerators, the second accelerator is an accelerator other than the first accelerator in the plurality of accelerators, and the first round of training and the second round of training can identify any two adjacent rounds of training in a plurality of rounds of training performed on the training model.

[0007] Through the above design, the first accelerator obtains the first type of parameters and the second type of parameters after the first round of training, then transmits the first type of parameters before the second round of training, and hides the transmission of the second type of parameters in the second round of training, so that the model training and the transmission of part of the parameters are performed in parallel. Compared with the prior art in which the first type of parameters and the second type of parameters are transmitted before the second round of training starts, the amount of data to be transmitted between the two rounds of training is reduced, that is, only the first type of parameters needs to be transmitted before the second round of training starts, so that the time for communication between the accelerators can be shortened, thereby shortening the duration of the entire model training task and improving the model training efficiency.

[0008] In a possible design, the first type of parameters are used to update the training parameters of the second accelerator in the second round of training.

[0009] Through the above design, the first accelerator transmits the first type of parameters to the second accelerator after the first round of training, so that the second accelerator can start the second round of training after updating the training parameters required for the second round of training using the first type of parameters, thereby shortening the time interval between the two rounds of training and improving the training efficiency, and training using the updated training parameters can ensure the training effect.

[0010] In a possible design, the cluster further includes a processor; and the first accelerator is further configured to receive communication information, where the communication information is used to indicate the first type of parameters and / or the second type of parameters. For example, the communication information can be determined by the processor.

[0011] Through the above design, the accelerator does not need to determine the first type of parameters and the second type of parameters, so that the computing power of the accelerator can be saved, and the model training efficiency can be improved.

[0012] In a possible design, the first accelerator is further configured to determine the first type of parameters and the second type of parameters.

[0013] Through the above design, after obtaining the parameter set after the first round of training, the first type of parameters and the second type of parameters in the parameter set are determined by the accelerator, without the need for determination by the processor, so that the computing work can be further offloaded from the processor to the accelerator, the utilization rate of the processor can be improved, and the accelerator determines its own communication parameters (such as the first type of parameters and the second type of parameters), so that the accelerator does not need to receive communication information, and the communication bandwidth resources of the accelerator can be saved.

[0014] In a possible design, the second type of parameters include a first sub-type of parameters and a second sub-type of parameters; and when the first accelerator transmits the second type of parameters to the second accelerator, the first accelerator is specifically configured to: transmit the first sub-type of parameters to the second accelerator after the second round of training starts and before a third round of training starts; and transmit the second sub-type of parameters to the second accelerator after the third round of training starts.

[0015] By the above design, the second type of parameters is divided into the first sub-type of parameters and the second sub-type of parameters, the second sub-type of parameters is hidden in the transmission in the third round of training, a more fine-grained communication parameter is realized, the communication time between the second round of training and the third round of training is further shortened, and the model training efficiency is improved.

[0016] In a possible design, the first sub-type of parameters is used to update the training parameters of the second accelerator in the second round of training.

[0017] By the above design, the second accelerator can update the training parameters based on the first sub-type of parameters, and re-train the updated training parameters in the second round of training, so as to ensure the training effect and benefit on the basis of shortening the communication time between the two rounds of training.

[0018] In a possible design, the first type of parameters is determined based on index information of the first round of training (denoted as first index information) and index information of the second round of training (denoted as second index information); wherein the first index information is used to indicate the training parameters of the first accelerator in the first round of training, and the second index information is used to indicate the training parameters of the second accelerator in the second round of training.

[0019] By the above design, the accuracy of the determined first type of parameters can be ensured based on the first index information and the second index information, and the index information of each round of training in the model training is determined, so that the parameters to be transmitted before the two rounds of training, such as the first type of parameters, can be determined in advance, so that the accelerator can start parameter transmission after completing a round of training, avoiding delay.

[0020] In a possible design, the first type of parameters includes the gradient of the training parameters to be updated, and the training parameters are embedding vectors, and the gradient of the embedding vectors is used to determine the updated value of the embedding vectors.

[0021] In a possible design, the training model is a recommendation model, the second accelerator includes an embedding table, and the training parameters are part of the embedding vectors included in the embedding table; after receiving the first type of parameters, the second accelerator is further used to update the first embedding vector in the embedding table based on the first type of parameters to obtain an updated embedding table; and the second embedding vector of the second accelerator in the second round of training includes the first embedding vector in the updated embedding table.

[0022] By the above design, the accelerator cluster can be used to perform multiple rounds of training of the recommendation model, and the gradient of part of the embedding vectors obtained in each round of training can be hidden in the transmission in the next round of training based on the method, so that the gradient of the embedding vectors obtained after training does not need to be transmitted in its entirety before the next round of training is started, thereby shortening the communication waiting time between the two rounds of training and improving the training efficiency of the recommendation model in the recommendation scenario.

[0023] In a second aspect, the present application provides a model training method, which is applied to an accelerator cluster, the cluster comprising a plurality of accelerators, each of which is deployed with a training model; the method comprising: a first accelerator performing a first round of training on the training model to obtain a first parameter set; the parameter set comprising first parameters and second parameters; the first accelerator sending the first parameters to a second accelerator before the start of a second round of training, and sending the second parameters to the second accelerator after the start of the second round of training; the first accelerator being any one of the plurality of accelerators, and the second accelerator being an accelerator other than the first accelerator in the plurality of accelerators.

[0024] In a possible design, the first parameters are used to update training parameters of the second accelerator in the second round of training.

[0025] In a possible design, the cluster further comprises a processor; and the first accelerator is further configured to receive communication information determined by the processor, the communication information being used to indicate the first parameters and / or the second parameters.

[0026] In a possible design, the first accelerator is further configured to determine the first parameters and the second parameters.

[0027] In a possible design, the second parameters comprise first sub-class parameters and second sub-class parameters; and the sending of the second parameters to the second accelerator after the start of the second round of training comprises: transmitting the first sub-class parameters to the second accelerator before the start of a third round of training after the start of the second round of training; and transmitting the second sub-class parameters to the second accelerator after the start of the third round of training.

[0028] In a possible design, the first sub-class parameters are used to update training parameters of the second accelerator in the second round of training.

[0029] In a possible design, the first parameters comprise gradients of training parameters to be updated, the gradients of the training parameters being used to determine updated values of the training parameters, and the training parameters being embedding vectors.

[0030] In a third aspect, the present application further provides a model training apparatus, which has the functions of implementing the behaviors of the first accelerator described in the first aspect, and the beneficial effects can be referred to the description of the first aspect, which will not be repeated here. The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software comprises one or more modules corresponding to the above functions. In a possible design, the apparatus comprises a training module and a communication module; and optionally, further comprises a management module, which can implement the functions of the behaviors in the method of the first aspect, and the specific description can be referred to the detailed description in the method examples, which will not be repeated here.

[0031] In a fourth aspect, the present application provides a computing device, which comprises a processor and a memory. The memory is configured to store instructions and / or data. The memory is coupled to the processor. When the processor executes the program instructions stored in the memory, the function of the method described in the first aspect can be implemented. The computing device further comprises a communication interface configured to communicate with other devices.

[0032] In a fifth aspect, the present application provides a training system, which comprises the accelerator cluster and the processor described in the first aspect. The processor is configured to manage the parameter transmission manner among the accelerator clusters, for example, determine the first type of parameters and the second type of parameters in the first parameter set obtained by the first accelerator in the first round of training, and indicate the first type of parameters and the second type of parameters in the first parameter set through the communication information.

[0033] In a sixth aspect, the present application provides a computer readable storage medium, which stores instructions. When the instructions are executed on a computer, the computer executes the method in the second aspect and each possible design of the second aspect.

[0034] In a seventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, cause the computer to execute the method in the second aspect and each possible design of the second aspect.

[0035] In an eighth aspect, the present application provides a computing chip, which is connected to a memory. The chip is configured to read and execute the software program stored in the memory, and execute the method in the second aspect and each possible design of the second aspect.

[0036] The beneficial effects of any one of the second aspect to the eighth aspect are described in the technical effect description of any one of the possible implementation manners of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0037] FIG. 1 is a schematic diagram of an architecture of a recommendation model;

[0038] FIG. 2 is a schematic diagram of a prior art solution;

[0039] FIG. 3 is a schematic diagram of another prior art solution;

[0040] FIG. 4 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0041] FIG. 5 is a schematic diagram of a process corresponding to a model training method provided by an embodiment of the present application;

[0042] FIG. 6 is a schematic diagram of a deployment architecture of a recommendation model provided by an embodiment of the present application;

[0043] Figure 7 illustrates the deployment architecture intent of another recommended model provided in this application embodiment;

[0044] Figure 8 is a flowchart illustrating a model training task provided in an embodiment of this application;

[0045] Figure 9 is a schematic diagram of a parameter transmission process provided in an embodiment of this application;

[0046] Figure 10 is a schematic diagram of a parameter communication method provided in an embodiment of this application;

[0047] Figure 11 is a schematic diagram of a fine-grained cold update parameter transmission process provided in an embodiment of this application;

[0048] Figure 12 is a schematic diagram of a model training device provided in an embodiment of this application;

[0049] Figure 13 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0050] Figure 14 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0051] "Recommendation" is a common business scenario in current internet applications, such as "You May Like" and "You've Seen Again" in shopping apps, video apps, and browsers. Typically, recommendation models determine recommended content based on user characteristics. Recommendation models in these scenarios often exhibit both large-scale (e.g., hundreds of GB to TB) sparsity and small-scale (e.g., MB to GB) density.

[0052] Figure 1 illustrates a schematic diagram of a recommendation model. As shown in Figure 1, the recommendation model includes a sparse layer (embedding look-up table) and a dense layer.

[0053] For training the recommendation model, sparse layers are used to retrieve and assemble embedding vectors from the embedding table based on the model's input (including feature IDs) using lookup operators. The embedding table can be understood as a complete table mapping all feature IDs within a feature set to their corresponding embedding vectors. A feature ID is a numerical ID encoded to uniquely identify a feature. The sparse layer processes the feature IDs to obtain their corresponding embedding representations, i.e., embedding vectors. Specifically, the embedding vectors corresponding to the input feature IDs are obtained from the embedding table through a lookup. Then, the embedding vectors obtained from the lookup are input into the dense layer for training, and the gradient of these embedding vectors is calculated through backpropagation. Based on this gradient, the embedding vectors in the embedding table are updated.

[0054] At present, improving the computing efficiency of dense layers is a hot research direction in the industry. Several existing technical solutions are provided as follows:

[0055] In the first existing technical solution, the service with heavy dense computing burden is migrated from the parameter server (PS) cluster of the central processing unit (CPU) to the general processing unit (GPU) / neural-network processing units (NPU) cluster, and data parallelism is used to fully utilize the computing power of the GPU / NPU cluster. In addition, in order to further reduce the memory access latency of the parameter server and the influence of the host to device (H2D) memory access latency from CPU to GPU / NPU within the GPU / NPU node on the training performance, the embedding table can also be deployed to the high bandwidth memory (HBM) on the GPU / NPU. Due to the limited capacity of HBM, the current model is generally deployed in parallel.

[0056] Referring to FIG. 2, model parallelism can refer to distributing different layers or different parameters of a model to different nodes (such as GPU / NPU). Here, the model parallelism is for sparse layers. For model parallelism of sparse layers, a single complete embedding table can be divided into multiple data blocks, and each data block is stored in the HBM of the GPU / NPU. It can be understood that the size of each data block is related to the capacity of the HBM. For example, as shown in FIG. 2, the embedding table is divided into four data blocks, and each data block is pre-stored in the local HBM of each node. For example, as shown in FIG. 2, data block 0 is stored in the HBM of node 0, data block 1 is stored in the HBM of node 1, data block 2 is stored in the HBM of node 2, and so on. The node obtains the embedding vector from the local HBM to accelerate the data reading rate, thereby accelerating the training.

[0057] Data parallelism refers to dividing a data set (e.g., including feature IDs) into multiple subsets, each of which is assigned to a different node (e.g., GPU / NPU), and each node has a complete model copy, please note that the data parallelism here is for dense layers, and the data parallelism of the dense layer means that each node deploys a complete dense layer. Each node uses its own model copy (dense layer) to process different data subsets. As shown in FIG. 2, the data set is divided into four data subsets, which are assigned to node 0, node 1, node 2, and node 3, respectively, and each node deploys a complete dense layer. Each node looks up the input data subset to include the feature ID, and forms the embedding vector obtained by the lookup into the batch input to the dense layer for training. After calculating the gradient, the gradient is synchronized with other nodes through collective communication, and the embedding vector in the embedding table is updated based on the gradient.

[0058] This scheme has a long synchronization waiting time for node communication, because the embedding vectors input to the dense layer of each node are usually scattered in the entire embedding table. For example, as shown in FIG. 2, the embedding vectors included in batch0 are scattered in data block 0-data block 4, and the embedding vectors included in batch1 are also scattered in data block 0-data block 4. However, data block 0-data block 4 are stored in different nodes, so each node needs to perform forward communication and reverse communication through collective communication such as all-to-all (A2A) communication.

[0059] Forward communication refers to the interaction of embedding vectors of multiple nodes through A2A, which means two-by-two interaction. For example, in FIG. 2, node 0 extracts the embedding vectors required by node 1 from local data block 0 and sends them to node 1, and node 1 extracts the embedding vectors required by node 0 from local data block 1 and sends them to node 0. Node 1 extracts the embedding vectors required by node 2 from data block 1 and sends them to node 2, and node 2 extracts the embedding vectors required by node 1 from local data block 2 and sends them to node 1, and so on. Each node can form the batch input to the dense layer for training based on the embedding vectors extracted from the local data block and the embedding vectors sent by other nodes. Reverse communication refers to the interaction of the gradients of the embedding vectors after each node is trained in this round. In this way, each node updates the part of the embedding vectors in the local data block based on the gradients sent by the other nodes.

[0060] As can be seen, the more the number of nodes in the above-mentioned manner, the longer the interaction time required for forward communication and reverse communication. All nodes need to be synchronized and waited during forward communication and reverse communication, resulting in poor system performance.

[0061] In a second prior art solution, a hybrid parallel solution is provided. The hybrid parallel solution takes advantage of the multi-table scenario commonly existing in the recommendation system. The dense layer adopts data parallelism, and the sparse layer adopts model parallelism or data parallelism based on the size of the embedding table.

[0062] In combination with FIG. 3, for an embedding table with a large size, model parallelism is adopted, that is, multiple nodes jointly store an entire embedding table, and each node locally stores part of the data of the same embedding table in the HBM. For an embedding table with a small size that can be stored locally, data parallelism is switched, that is, each node locally stores a complete small (sparse) table in the HBM. In this way, the nodes do not need to communicate with each other to exchange the data of the small table, reducing the number of synchronizations and the synchronization waiting time, and having better scale-out performance in a large-scale scenario.

[0063] However, the above solution has the problem of increasing the amount of synchronized data, because in the backward communication of the data parallelism, each node needs to calculate and synchronize the gradient of the locally stored complete embedding table or the gradient of the embedding vectors activated by all nodes. In the model parallelism, each node only needs to synchronize the gradient of the embedding vector activated by the current node, thus leading to an increase in the backward communication time and low training efficiency, which will affect the overall benefits of the solution.

[0064] Therefore, the present application provides a model training method to solve the problem of long node synchronization waiting time, poor system performance, and low training efficiency caused by the communication mode in the existing recommendation model training scenario. Taking the application scenario shown in FIG. 3 as an example, the method takes into account the embedding vector update characteristics of the embedding table in model training, performs multiple rounds of training on the embedding table, hides the cold update parameters of each round of training to the subsequent multiple rounds of training process for transmission, reduces the amount of parameters that need to be transmitted between two rounds of training, thereby reducing the node synchronization waiting time, improving the system performance and training efficiency.

[0065] Next, the technical solutions provided by the embodiments of the present application will be described in detail in combination with the drawings.

[0066] FIG. 4 is a schematic diagram of the architecture of a distributed training system provided by an embodiment of the present application. As shown in FIG. 4, the system 10 includes an accelerator cluster. The accelerator cluster includes multiple accelerators 112. The accelerators 112 can communicate with each other.

[0067] In terms of function, the multiple accelerators 112 are used to perform the model training method provided by the embodiments of the present application, such as performing the model training task.

[0068] On the hardware, the accelerator 112 can include, but is not limited to, a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), an application processor (AP), a modem, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a hardware logic circuit, a processing core, an ASIC, an AI chip, or a programmable logic device (PLD) implementation, which can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), or any combination thereof.

[0069] The plurality of accelerators 112 can be integrated in different computing nodes 110, or integrated in the same computing node 110. The computing node 110 can be a computing device such as a server, a desktop computer, etc., and is not particularly limited.

[0070] Optionally, the system 10 further includes a processor 111, which can be a central processing unit (CPU) for managing the communication manner between the plurality of accelerators 112, which will be described in detail below, and will not be described here. It should be noted that in FIG. 4, the processor 111 and the accelerator 112 are integrated in the same computing node 110, and the embodiments of the present application also support that the processor 111 and the accelerator 112 are located in different computing nodes 110.

[0071] It should be noted that the system 10 shown in FIG. 4 is only an example, and the system to which the embodiments of the present application can be applied can have more or fewer components than FIG. 4. The embodiments of the present application do not limit this.

[0072] The model training method provided by the embodiments of the present application will be described in detail below with reference to FIG. 5, taking the system shown in FIG. 4 as an example.

[0073] Taking the model training method applied to a recommendation scenario as an example, the plurality of accelerators 112 can be used to perform parallel training of a recommendation model. In one training scenario, as shown in FIG. 6, both the sparse layer and the dense layer in the recommendation model are deployed in a data parallel manner, that is, each accelerator 112 (such as GPU0 to GPU7) stores a complete embedding table, and a complete copy of the dense layer is deployed on each accelerator 112. In another training scenario, as shown in FIG. 7, the sparse layer in the recommendation model is deployed in a model parallel manner, and the dense layer is deployed in a data parallel manner, that is, in this scenario, the embedding table is divided into multiple data blocks and stored in multiple accelerators 112, and a complete copy of the dense layer is deployed on each accelerator 112.

[0074] The method will be introduced below taking the method applied to the training scenario shown in FIG. 6 as an example. As shown in FIG. 5, the method can include the following steps:

[0075] Step 500, the processor 111 obtains input index information of multiple rounds of training included in a model training task.

[0076] For example, FIG. 8 is a model training task execution schematic diagram provided by an embodiment of the present application. The model training task can include multiple rounds of training of embedding vectors included in an embedding table in a recommendation model.

[0077] As shown in FIG. 8, the model training task includes multiple rounds of training, each round of training includes multiple subtasks, and the multiple subtasks are executed in parallel by the plurality of accelerators 112 (such as GPU0-GPUn in FIG. 8), wherein each subtask is used to train part of the embedding vectors in the embedding table.

[0078] Taking one round of training as an example, the first round of training includes subtask 0, subtask 1, …, and subtask n. GPU 0 is used to execute subtask 0, GPU 1 is used to execute subtask 1, GPU 2 is used to execute subtask 2, and so on.

[0079] In a fine-grained manner, each subtask has input index information, and the input index information of the subtask is used to indicate the input data of the subtask, where the input data refers to the embedding vectors input to the dense layer. For example, the input index information of subtask 0 indicates the input data batch0 (denoted by b0 in FIG. 8) of subtask 0, the input index information of subtask 1 indicates the input data batch1 (denoted by b1 in FIG. 8) of subtask 1, the input index information of subtask 2 indicates the input data batch2 (denoted by b2 in FIG. 8) of subtask 2, and so on.

[0080] The input index information includes feature IDs for indicating embedding vectors. For example, the input index information of the subtask 0 includes feature IDs (0, 2, 1, 8). The GPU 0 performs table lookup based on the feature IDs (0, 2, 1, 8) to respectively query an embedding vector corresponding to the feature ID = 0 (denoted as embedding vector 0), an embedding vector corresponding to the feature ID = 2 (denoted as embedding vector 1), an embedding vector corresponding to the feature ID = 1 (denoted as embedding vector 2), and an embedding vector corresponding to the feature ID = 8 (denoted as embedding vector 3) from the embedding table. The batch 0 includes the embedding vector 0, the embedding vector 1, the embedding vector 2, and the embedding vector 3.

[0081] For another example, the input index information of the subtask 1 includes feature IDs (1, 3, 4, 7). The GPU 1 performs table lookup based on the feature IDs (1, 3, 4, 7) to respectively query an embedding vector corresponding to the feature ID = 1 (denoted as embedding vector 4), an embedding vector corresponding to the feature ID = 3 (denoted as embedding vector 5), an embedding vector corresponding to the feature ID = 4 (denoted as embedding vector 6), and an embedding vector corresponding to the feature ID = 7 (denoted as embedding vector 7) from the embedding table. The batch 1 includes the embedding vector 4, the embedding vector 5, the embedding vector 6, and the embedding vector 7.

[0082] Coarsely, the input index information of a round of training includes input index information of all subtasks. In step 500, the processor 111 obtains input index information of multiple rounds of training, which indicates input index information of each round of training in the multiple rounds of training. For simplicity of description, the input index information of the multiple rounds of training is referred to as input index information of multiple rounds of training hereinafter.

[0083] The input index information of a round of training is used to indicate input data (denoted as global batch, GB for short) of the round of training, and the input data of the round of training includes input data of all subtasks in the round of training. For example, the input index information of the first round of training includes input index information of the subtask 0, input index information of the subtask 1, and input index information of the subtask n. The input index information of the first round of training indicates input data GB0 of the first round of training, and the input data GB0 includes the batch 0, the batch 1, and the batch n. The input data GB1 of the second round of training includes the batch 0' (denoted as b0' in FIG. 8), the batch 1' (denoted as b1' in FIG. 8), and the batch n' (denoted as bn' in FIG. 8).

[0084] It can be understood that the embedding vectors included in the input data batch are training parameters to be trained by each subtask.

[0085] Specifically, in FIG. 8, each GPU inputs the embedding vectors included in the input data batch to the dense layer for training to obtain a parameter set, which includes the gradients of the input part of the embedding vectors. For example, GPU0 inputs the embedding vectors included in batch0 to the dense layer for training to obtain a parameter set par0, which includes the gradients of the embedding vectors in batch0. For another example, GPU1 inputs the embedding vectors included in batch1 to the dense layer for training to obtain a parameter set par1, which includes the gradients of the embedding vectors in batch1.

[0086] Therefore, for each round of training, a gradient parameter set (denoted as global parameter, GP for short) is obtained, which includes the parameter sets of all sub-tasks in the round of training. For example, the gradient parameter set GP0 of the first round of training includes par0, par1, …, and par n. That is, GP0 includes the gradients of the embedding vectors in GB0. The gradient parameter set GP1 of the second round of training includes par0', par1', …, and par n'. GP1 includes the gradients of the embedding vectors in GB1. The same applies to the subsequent rounds of training.

[0087] In step 501, the processor 111 determines the first-class parameters and / or the second-class parameters in the first parameter set obtained by the first accelerator in the first round of training.

[0088] Macroscopically, in the embodiments of the present application, the gradient parameter set of each round of training includes cold update parameters and non-cold update parameters. The non-cold update parameters are transmitted before the next round of training, or in other words, the next round of training is started after the transmission of the non-cold update parameters is completed. The cold update parameters are transmitted after the start of the next round of training.

[0089] For example, referring to FIG. 9, the first round of training obtains a gradient parameter set GP0, which, as described above, includes the gradients of the embedding vectors in GB0. GP0 includes non-cold update parameters (denoted as GP0-non-cold update parameters) and cold update parameters (denoted as GP0-cold update parameters). That is, GP0-non-cold update parameters include the gradients of part of the embedding vectors in GB0, and GP0-cold update parameters include the gradients of the remaining part of the embedding vectors in GB0. Wherein, GP0-non-cold update parameters are transmitted before the start of the second round of training, and GP0-cold update parameters are transmitted after the start of the second round of training.

[0090] The second round of training obtains a gradient parameter set GP1, which includes the gradients of the embedding vectors in GB1, and GP1 includes non-cold update parameters (denoted as GP1-non-cold update parameters) and cold update parameters (denoted as GP1-cold update parameters), wherein the GP1-non-cold update parameters are transmitted before the start of the third round of training, and the GP1-cold update parameters are transmitted after the start of the third round of training.

[0091] The third round of training obtains a gradient parameter set GP2, which includes the gradients of the embedding vectors in GB2, and GP2 includes non-cold update parameters (denoted as GP2-non-cold update parameters) and cold update parameters (denoted as GP2-cold update parameters), wherein the GP2-non-cold update parameters are transmitted before the start of the fourth round of training, and the GP2-cold update parameters are transmitted after the start of the fourth round of training.

[0092] Similarly.

[0093] For ease of description, the cold update parameters and non-cold update parameters in the gradient parameter set obtained by the previous round of training between any two adjacent rounds of training are referred to as the cold update parameters and non-cold update parameters between the two rounds of training.

[0094] In an embodiment, the processor 111 can determine the cold update parameters and non-cold update parameters between the two rounds of training based on the input index information of any two adjacent rounds of training.

[0095] For example, in step 501, the processor 111 obtains the input index information of the first round of training (denoted as input index information 0) and the input index information of the second round of training (denoted as input index information 1), and then in step 502, the processor 111 determines the cold update parameters / non-cold update parameters in the gradient parameter set GP0 obtained by the first round of training according to the input index information 0 and the input index information 1.

[0096] For another example, in step 501, the processor 111 obtains the input index information of the second round of training (input index information 1) and the input index information of the third round of training (denoted as input index information 2), and then in step 502, the processor 111 determines the cold update parameters / non-cold update parameters in the gradient parameter set GP1 obtained by the second round of training according to the input index information 1 and the input index information 2.

[0097] For another example, in step 501, the processor 111 obtains the input index information of the second round of training (input index information 1) and the input index information of the third round of training (denoted as input index information 2), and then in step 502, the processor 111 determines the cold update parameters / non-cold update parameters in the gradient parameter set GP1 obtained by the second round of training according to the input index information 1 and the input index information 2.

[0098] In summary, the processor 111 repeats the steps 501 and 502 each time when determining the cold update parameters / non-cold update parameters between two adjacent rounds of training. Alternatively, in the step 501, the processor 111 can obtain the input index information of all rounds of training at one time.

[0099] For example, another implementation of the above example: in the step 501, the processor 111 obtains the input index information 0, the input index information 1, the input index information 2 and the input index information 3. In the step 502, the processor 111 determines the cold update parameters / non-cold update parameters in the GP0 based on the input index information 0 and the input index information 1. The step 502 is repeated, and the processor 111 determines the cold update parameters / non-cold update parameters in the GP1 based on the input index information 1 and the input index information 2. The step 502 is repeated again, and the processor 111 determines the cold update parameters / non-cold update parameters in the GP2 based on the input index information 2 and the input index information 3. And so on.

[0100] The following describes how to determine the cold update parameters / non-cold update parameters based on the input index information of two rounds of training.

[0101] Taking the first round of training and the second round of training as an example, in the embodiment of the present application, the non-cold update parameters (GP0-non-cold update parameters) in the GP0 are used to update the embedding vectors of the second round of training. That is, the GB1 includes the embedding vectors updated using the GP0-non-cold update parameters. Therefore, the GP0-non-cold update parameters are determined based on the intersection of the embedding vector set GB0 indicated by the input index information of the first round of training and the embedding vector set GB1 indicated by the input index information of the second round of training. The GP0-non-cold update parameters include the gradients of the embedding vectors in the intersection. And the GP0-cold update parameters include the parameters in the GP0 other than the GP0-non-cold update parameters.

[0102] The following describes in detail how the processor 111 instructs the plurality of accelerators to transmit the non-cold update parameters in combination with All to All (A2A) communication.

[0103] Continuing to take the first round of training as an example, as to each accelerator 112, the GP0-non-cold update parameters include part of the parameter set obtained by the plurality of sub-tasks in the first round of training. The parameter set of each sub-task includes the gradient of the embedding vector trained by the respective sub-task.

[0104] Referring to FIG. 10, each GPU needs to send the gradient of the embedding vector included in the parameter set obtained by itself to the remaining GPUs, and based on whether the opposite end GPU uses the embedding vector in the second round of training, the parameters in the parameter set obtained by each GPU can be divided into two categories, one of which belongs to the non-cold update parameters, and the other of which belongs to the cold update parameters.

[0105] Take GPU0 as an example, and understand in combination with FIG. 8:

[0106] Suppose that GPU0 executes subtask 0 in the first round of training, determines batch0 based on the input index information of subtask 0, and batch0 includes at least one embedding vector. GPU0 inputs the at least one embedding vector included in batch0 to the dense layer to obtain a parameter set par0, and the parameter set par0 includes the gradient of each embedding vector in the at least one embedding vector.

[0107] In addition, GPU1 executes subtask 1' in the second round of training, determines batch1' based on the input index information of subtask 1', and batch1' includes at least one embedding vector. GPU2 executes subtask 2' in the second round of training, determines batch2' based on the input index information of subtask 2'. GPU3 executes subtask 3' in the second round of training, determines batch3' based on the input index information of subtask 3'. In this way, the training is performed.

[0108] For GPU0 (the parameter sending end) and GPU1 (the parameter receiving end), the processor 111 obtains the input index information of subtask 0 and the input index information of subtask 1', determines the intersection 1 of batch0 and batch1', and the processor 111 determines two types of parameters included in par0 based on the intersection 1, which are referred to as the first type of parameter (belonging to the non-cold update parameter) and the second type of parameter (belonging to the cold update parameter) for the sake of distinction, the first type of parameter includes the gradient of the embedding vector included in the intersection 1, and is used to update the embedding vector used by GPU1 in executing subtask 1' in the second round of training, that is, the embedding vector trained by subtask 1'. The second type of parameter includes the parameters in par0 other than the first type of parameter.

[0109] Similarly, for GPU0 (the parameter sending end) and GPU2 (the parameter receiving end), the processor 111 obtains the input index information of subtask 0 and the input index information of subtask 2', determines the intersection 2 of batch0 and batch2', and the processor 111 determines two types of parameters included in par0 based on the intersection 2, which are referred to as the third type of parameter (belonging to the non-cold update parameter) and the fourth type of parameter (belonging to the cold update parameter) for the sake of distinction, the third type of parameter includes the gradient of the embedding vector included in the intersection 2, and is used to update the embedding vector used by GPU2 in executing subtask 2' in the second round of training, that is, the embedding vector trained by subtask 2'. The fourth type of parameter includes the parameters in par0 other than the third type of parameter.

[0110] Similarly, for GPU0 (parameter sending end) and GPU3 (parameter receiving end), the processor 111 obtains the input index information of the subtask 0 and the input index information of the subtask 3', determines the intersection 3 of batch0 and batch3', and the processor 111 determines two types of parameters included in par0 based on the intersection 3, which are the fifth type of parameters (belonging to non-cold update parameters) and the sixth type of parameters (belonging to cold update parameters). The fifth type of parameters include the gradient of the embedding vector included in the intersection 3, which is used to update the embedding vector used by GPU3 in the second round of training to perform the subtask 3', i.e., the trained embedding vector of the subtask 3'. The fifth type of parameters include parameters in par0 other than the sixth type of parameters.

[0111] By analogy.

[0112] In FIG. 10, the remaining accelerators also send non-cold update parameters to the opposite end accelerators based on the manner of GPU0, which will not be repeated here. Similarly, GPU0 will also receive non-cold update parameters sent by other GPUs as a parameter receiving end.

[0113] In step 501, the processor 111 determines the non-cold update parameters to be transmitted between any two GPUs in a round of training, and indicates the parameter sending end through the communication information (see step 502). For ease of description, here it is simplified that the processor 111 determines the first type of parameters in the first parameter set obtained by the first accelerator for the first accelerator and the second accelerator. The first accelerator can be any one of GPU0-GPUn in FIG. 8. Here, the second accelerator can refer to any one of GPU0-GPUn except the first accelerator. The following will not be repeated.

[0114] Step 502, the processor 111 sends the communication information to the first accelerator.

[0115] The processor 111 sends the communication information to the first accelerator, which is used to indicate the first accelerator to transmit the first type of parameters to the second accelerator. For example, in the above example, the processor 111 sends the communication information 0, the communication information 1, the communication information 2, …, the communication information n-1 to GPU0, the communication information 0 indicates that GPU0 sends the first type of parameters to GPU1. The communication information 1 indicates that GPU0 sends the third type of parameters to GPU2. The communication information 2 indicates that GPU0 sends the fifth type of parameters to GPU3. By analogy.

[0116] For example, the communication information 0 can indicate the first type of parameters, such as including an index of the embedding vector corresponding to the first type of parameters, such as a feature ID or a newly generated identifier for uniquely identifying an embedding vector. Wherein, the embedding vector corresponding to the gradient of the first embedding vector is the first embedding vector. Alternatively, the communication information 0 can also indicate the second type of parameters, such as including an index of the embedding vector corresponding to the second type of parameters. The GPU 0 can reversely determine the first type of parameters in the first parameter set based on the second type of parameters. Alternatively, the communication information 0 indicates the first type of parameters and the second type of parameters. In addition, the communication information 0 can also include an identifier of the GPU 1, for indicating a receiving end of the first type of parameters.

[0117] It should be noted that the above is only for convenience of description, and the plurality of communication information can be integrated in one communication information, such as the communication information A including the communication information 0 to the communication information n-1, so that the processor 111 only needs to send the communication information A to the GPU 0.

[0118] Similarly, the processor 111 sends the communication information to the rest of the GPUs, indicating how each GPU sends parameters to other GPUs. For details, please refer to the above description.

[0119] In step 503, the first accelerator completes the first round of training to obtain the first parameter set.

[0120] For example, the GPUs 0-GPUn perform the model training task in parallel, and each GPU performs a respective subtask in each round of training and obtains a parameter set of the respective subtask. For details, please refer to the above description.

[0121] It should be noted that there is no strict time sequence limitation between steps 501-502 and step 503, and steps 501-502 can be executed before step 503, simultaneously, or after step 503.

[0122] In step 504, the first accelerator sends the first type of parameters in the first parameter set to the second accelerator based on the communication information.

[0123] In step 505, the second accelerator updates the embedding vectors in the embedding table based on the received parameters.

[0124] Continuing to understand in combination with FIG. 10, the GPU 0 determines the first type of parameters in the first parameter set based on the received communication information 0, and sends the first type of parameters to the GPU 1. Correspondingly, the GPU 1 receives the first type of parameters sent by the GPU 0, determines the update value of the embedding vector based on the gradient of the embedding vector included in the first type of parameters, and updates the embedding vector in the embedding table stored locally by the GPU 0 based on the determined update value.

[0125] Similarly, GPU0 determines the third type of parameters in the first parameter set based on the received communication information 1, and sends the third type of parameters to GPU2. Correspondingly, GPU2 receives the third type of parameters sent by GPU0, determines the update value of the part of embedding vectors based on the gradient of the embedding vectors included in the third type of parameters, and updates the part of embedding vectors in the embedding table stored locally by GPU2 based on the determined update value.

[0126] GPU0 determines the fifth type of parameters in the first parameter set based on the received communication information 2, and sends the fifth type of parameters to GPU3. Correspondingly, GPU3 receives the fifth type of parameters sent by GPU0, determines the update value of the part of embedding vectors based on the gradient of the embedding vectors included in the fifth type of parameters, and updates the part of embedding vectors in the embedding table stored locally by GPU3 based on the determined update value.

[0127] Similarly.

[0128] Taking GPU1 as an example, it should be noted that in addition to receiving the first type of parameters sent by GPU0, GPU1 can also receive non-cold update parameters sent by other GPUs, and GPU1 needs to update the embedding vectors in the local embedding table based on all the received non-cold update parameters. That is, in step 505, each GPU needs to update the embedding vectors in the embedding table based on all the received non-cold update parameters.

[0129] Step 506, the second accelerator starts the second round of training.

[0130] There are various signs of the start of the second round, for example, based on the received all non-cold update parameters to update the embedding vectors in the embedding table, or start the sub-tasks of the second round of training, or extract the embedding vectors to be trained in the second round of sub-tasks from the embedding table, or input the extracted embedding vectors into the dense layer, etc. As long as the local embedding table is updated using the non-cold update parameters of the first round of training, the second round of training can be started. In an embodiment, all sub-tasks start each round of training synchronously.

[0131] Step 507, after the start of the second round of training, the first accelerator sends the second type of parameters in the first parameter set to the second accelerator.

[0132] The transmission process of the cold update parameters will be introduced below in combination with the A2A mode.

[0133] Continuing to refer to FIG. 10, after the start of the second round of training:

[0134] GPU0 sends the second type of parameters in the first parameter set to GPU1. The second type of parameters and the first type of parameters constitute the first parameter set, and correspondingly, GPU1 receives the second type of parameters and updates the local embedding table based on the second type of parameters.

[0135] GPU0 sends the fourth type of parameters in the first parameter set to GPU2, the fourth type of parameters and the third type of parameters form the first parameter set, and correspondingly, GPU2 receives the fourth type of parameters and updates the local embedding table based on the fourth type of parameters.

[0136] GPU0 sends the sixth type of parameters in the first parameter set to GPU3, the sixth type of parameters and the fifth type of parameters form the first parameter set, and correspondingly, GPU3 receives the sixth type of parameters and updates the local embedding table based on the sixth type of parameters.

[0137] By analogy.

[0138] The rest of the accelerators also send cold update parameters to the opposite end accelerator based on the manner of GPU0, which will not be described here.

[0139] Further, the embodiment of the application provides a more fine-grained transmission manner of cold update parameters.

[0140] Taking the transmission of the second type of parameters from GPU0 to GPU1 as an example, the processor 111 determines a plurality of sub-class parameters in the second type of parameters based on the input index information of the sub-tasks in more than two rounds of training.

[0141] For example, referring to FIG. 11, the processor 111 obtains the input index information (denoted as B0) of the sub-task 0 executed by GPU0 in the first round of training, the input index information (denoted as B1) of the sub-task 1' executed by GPU1 in the second round of training, and the input index information (denoted as B2) of the sub-task 1'' executed by GPU1 in the third round of training.

[0142] Suppose that the feature IDs included in B0 are (2, 1, 3) respectively, the feature IDs included in B1 are (4, 1, 5) respectively, and the feature IDs included in B2 are (2, 4, 5) respectively.

[0143] Wherein, the intersection of B0 and B1 is feature ID=1, so it can be determined that the first type of parameters in par0 include feature ID=1, and the second type of parameters include feature ID=2 and feature ID=3.

[0144] Optionally, the second type of parameters further includes a first sub-class parameter and a second sub-class parameter, the intersection of B0 and B2 is feature ID=2, and it can be determined that the first sub-class parameter includes feature ID=2, which is used to update the embedding vector to be trained by the sub-task 1'' of GPU1 in the third round of training. The second sub-class parameter includes parameters other than the first sub-class parameter in the second type of parameters, that is, the second sub-class parameter includes feature ID=3.

[0145] As described above, the first type of parameters is transmitted before the start of the second round of training. The second type of parameters is transmitted after the start of the second round of training. For this purpose, the first sub-type of parameters is transmitted after the start of the second round of training and before the start of the third round of training, for updating the embedding vectors of the sub-tasks to be trained in the third round of training. The transmission time of the second sub-type of parameters is not strictly required, and the second sub-type of parameters can be transmitted after the start of the second round of training or after the start of the third round of training, and the specific transmission time is not limited and can be determined based on the transmission strategy.

[0146] For example, the priority of each type of parameter is set, and the transmission time of each type of parameter is determined based on the priority. Wherein, the priority of each type of parameter can be determined according to one or more parameters such as the training round involved by the parameter, the parameter size, the transmission time, the training time of each round of training, etc.

[0147] For example, the first parameter set includes the first type of parameter, the first sub-type of parameter and the second sub-type of parameter:

[0148] (1) The earlier the training round involved, the higher the priority.

[0149] The first type of parameter is used to update the embedding vectors of the sub-tasks in the second round of training, and therefore the first type of parameter involves the second round of training.

[0150] The first sub-type of parameter is used to update the embedding vectors of the sub-tasks in the third round of training, and therefore the first sub-type of parameter involves the third round of training.

[0151] The second sub-type of parameter involves an unknown training round.

[0152] Therefore, the priority of each type of parameter is ranked as follows: first type of parameter > first sub-type of parameter > second sub-type of parameter.

[0153] For example, the priority of the first type of parameter in B0 > the priority of the first sub-type of parameter in B0 > the priority of the second sub-type of parameter in B0.

[0154] (2) The priority of the non-cold update parameter is higher than that of the cold update parameter.

[0155] The non-cold update parameter between the second round of training and the third round of training, such as the first type of parameter in B1 involves the third round of training, and the first sub-type of parameter in B0 also involves the third round of training. Since the first type of parameter in B1 belongs to the non-cold update parameter, and the first sub-type of parameter in B0 belongs to the cold update parameter, the priority of the first type of parameter in B1 is higher than that of the first sub-type of parameter in B0.

[0156] The first sub-type of parameter in the second type of parameter included in B1 involves the fourth round of training, and the second sub-type of parameter in B0 involves an unknown training round, and therefore the priority of the first sub-type of parameter in B1 is higher than that of the second sub-type of parameter in B0.

[0157] Based on this, a transmission strategy can include:

[0158] GPU0 first transmits the first subclass parameters of B0 after the start of the second round of training. In order to further shorten the synchronization waiting time of communication between the two rounds of training, if the transmission of the second subclass parameters of B0 can be completed before the end of the second round of training, the second subclass parameters of B0 are transmitted. If the second subclass parameters of B0 are not transmitted or not transmitted completely during the second round of training, after the end of the second round of training, the transmission of other parameters is suspended, the non-cold update parameters of the second round of training such as the first class parameters of B1 are preferentially transmitted, and the first class parameters in B1 are first transmitted after the start of the third round of training, and then the transmission of the second subclass parameters of B0 is continued.

[0159] The above example is the transmission mode of the parameters of multiple classes determined by the processor 111 based on the comparison of the input index information of the three rounds of training. Embodiments of the present application also support more rounds of training to manage cold update parameters in a fine-grained manner. For example, based on B0, B1, B2 and B3 (the input index information of GPU1 performing subtask 1”' in the fourth round of training), the second class parameters in par0 are divided into first subclass parameters, second subclass parameters and third subclass parameters, see the related description above, which will not be repeated here. The granularity of the division of cold update parameters by embodiments of the present application is not limited.

[0160] Through the above design, the first accelerator obtains the first class parameters and the second class parameters after the first round of training, then transmits the first class parameters before the second round of training, and hides the transmission of the second class parameters in the second round of training, so that model training and partial parameter transmission are parallel, compared with the prior art scheme of transmitting the first class parameters and the second class parameters before the start of the second round of training, the present application reduces the amount of data to be transmitted between the two rounds of training, i.e. only the first class parameters need to be transmitted before the start of the second round of training, so that the communication time between accelerators can be shortened, the communication overhead can be reduced, and the duration of the entire model training task can be shortened, thereby improving the model training efficiency. And under the data parallel or model parallel architecture of the sparse layer, the number of synchronization waiting times can be reduced, and better expansion performance can be achieved.

[0161] The method embodiment shown in FIG. 5 is introduced in the training scenario of FIG. 6, and the application of the method in the training scenario of FIG. 7 is introduced as follows:

[0162] Still taking the first accelerator and the second accelerator as an example, the same points of the two scenarios of FIG. 6 and FIG. 7 include that the first accelerator obtains a first parameter set in the first round of training, and the first parameter set includes first type parameters and second type parameters, wherein the first type parameters and the second type parameters are unchanged, and the transmission occasions of the first type parameters and the second type parameters are also unchanged. For details, please refer to the above description, which will not be repeated here.

[0163] The differences between the two scenarios include that although the first type parameters are used to update the embedding vectors of the second accelerator in the second round of training, the first type parameters are not all sent to the second accelerator, because the embedding vectors corresponding to the first type parameters in the embedding table may be stored in multiple accelerators locally. Therefore, the processor 111 determines in which accelerator each embedding vector corresponding to the first type parameters is stored, and instructs the first accelerator to send the gradient of the embedding vector corresponding to the first type parameters in the first type parameters to the accelerator. Similarly, based on the same process, the accelerator where the embedding vector corresponding to the second type parameters is located is determined, and the first accelerator is instructed to send the gradient of the embedding vector corresponding to the second type parameters in the second type parameters to the accelerator.

[0164] Alternatively, in the scenario of FIG. 7, the first accelerator still transmits all the first type parameters to the second accelerator to accelerate the second accelerator to extract the embedding vectors in the second round of training. In addition, the first accelerator also needs to send the gradient of the embedding vector of the first type parameters stored in the remaining accelerators to the corresponding accelerators, so that the embedding vectors in the data blocks of the embedding table stored locally by each accelerator are the latest.

[0165] In addition, the embodiment shown in FIG. 5 is illustrated by taking the processor 111 managing the parameter transmission mode of each accelerator as an example. The embodiments of the present application also support the case where the system 10 does not include the processor 111. Therefore, the embodiments of the present application also provide another implementation manner: each accelerator 112 performs the functions performed by the processor 111 described above, that is, the accelerator 112 itself determines the non-cold update parameters (such as the first type parameters) and the cold update parameters (such as the second type parameters, and the first sub-class parameters and the second sub-class parameters included in the second type parameters) in the parameter set obtained in each round of training that need to be sent to the opposite end accelerator. For details, please refer to the above description of the processor 111, which will not be repeated here.

[0166] It should be noted that (1) the above A2A is only an example, and the embodiments of the present application do not limit that the A2A mode must be used to interact parameters. For example, the All Reduce (AR) mode can also be used for interaction. In addition, the embodiments of the present application do not limit that the A2A mode must be used based on the above process steps, and the embodiments of the present application do not limit this.

[0167] (2) The first round of training and the second round of training in steps 503 to 506 can refer to any two adjacent rounds of training. For example, the second round of training and the third round of training in the model training task shown in FIG. 8, or the third round of training and the fourth round of training, can be understood as the first round of training representing the i th round of training and the second round of training representing the i+1 th round of training in the method embodiment of FIG. 5, where i is an integer not greater than N-1, and N is the number of rounds included in the model training task.

[0168] (3) It should be noted that FIG. 10 is only an example. In a possible case, the batches of sub-tasks in two adjacent rounds of training have no intersection, and therefore the gradients do not need to be transmitted between the corresponding GPUs. For example, batch0 and batch3' have no intersection, and therefore GPU0 does not need to send the gradients in par0 to GPU3.

[0169] In addition, it is worth noting that if there is no intersection between the input data of two rounds of training, the non-cold update parameters do not need to be transmitted between the two rounds of training. For example, if there is no intersection between GB0 of the first round of training and GB1 of the second round of training, the second round of training can be started after the first round of training is completed. That is, step 502 is an optional step and does not need to be performed. For example, when there is no intersection between the input data of two rounds of training, the processor 111 does not need to send the communication information to the accelerators 112.

[0170] If there is no intersection between the input data of two adjacent rounds of training, steps 502 to 504 do not need to be performed. It can be understood that the parameters between the first round of training and the second round of training are all cold update parameters, and the plurality of accelerators can directly start the second round of training and exchange the cold update parameters obtained in the first round of training after the second round of training is started. For the transmission mode of the cold update parameters, please refer to the description above, which will not be repeated here.

[0171] (4) The above method embodiments are exemplified by being applied to the recommendation scenario. The embodiments of the present application can also be applied to other scenarios and used in other model training tasks, and the embodiments of the present application do not limit this.

[0172] (5) In addition to the training scenarios shown in FIGS. 6 and 7, the embodiments of the present application can also be applied to other training scenarios, such as a hybrid parallel deployment of a recommendation model, and the embodiments of the present application do not limit this.

[0173] The above describes the method provided by the embodiments of the present application with reference to the accompanying drawings. The device provided by the embodiments of the present application is described below with reference to the accompanying drawings.

[0174] Based on the same inventive concept as the method embodiments, the embodiments of the present application also provide a model training apparatus, which is configured to perform the method of the method embodiments of FIG. 5. As shown in FIG. 12, the apparatus 1200 includes a training module 1201, a communication module 1202, and optionally, a management module 1203. Specifically, in the apparatus 1200, the modules are connected through a communication path.

[0175] The training module 1201 is configured to perform first round training on the training model to obtain a first parameter set, the first parameter set including first type parameters and second type parameters. For details, refer to the description of step 503 in the method embodiments of FIG. 5, which will not be repeated here.

[0176] The communication module 1202 is configured to send the first type parameters to a second accelerator before the second round training starts, and send the second type parameters to the second accelerator after the second round training starts. The first accelerator is any one of the plurality of accelerators, and the second accelerator is an accelerator other than the first accelerator in the plurality of accelerators. For details, refer to the description of steps 504-507 in the method embodiments of FIG. 5, which will not be repeated here.

[0177] In a possible implementation, the communication module 1202 is further configured to receive communication information, the communication information being configured to indicate the first type parameters and / or the second type parameters. The communication information is determined by the processor. For details, refer to the description of step 502 in the method embodiments of FIG. 5, which will not be repeated here.

[0178] In a possible implementation, the management module 1203 is configured to perform the method steps performed by the processor 111 in the method embodiments of FIG. 5, such as determining the first type parameters and / or the second type parameters in the first parameter set. For details, refer to the description of step 501 in the method embodiments of FIG. 5, which will not be repeated here.

[0179] The present application also provides a computing device 1300. As shown in FIG. 13, the computing device 1300 includes a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, the memory 1306, and the communication interface 1308 communicate through the bus 1302. The computing device 1300 can be a server or a storage array or a hard disk frame, etc. It should be understood that the present application does not limit the number of processors and memories in the computing device 1300.

[0180] The bus 1302 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is represented in FIG. 13, but it does not mean that there is only one bus or only one type of bus. The bus 1302 can include a path for transmitting information between various components (for example, the memory 1306, the processor 1304, the communication interface 1308) of the computing device 1300.

[0181] The processor 1304 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), or the like.

[0182] The memory 1306 can include a volatile memory (for example, a random access memory (RAM)), and the processor 1304 can further include a non-volatile memory (for example, a read-only memory (ROM), a flash memory, a mechanical hard disk drive (HDD), or a solid state drive (SSD), or the like, without limitation.

[0183] In an embodiment, the computing device 1300 is configured to perform the functions of the data processing apparatus 1200, wherein the executable program code is stored in the memory 1306, and the processor 1304 executes the executable program code to respectively implement the functions of the foregoing training module 1201, the communication module 1202, and the management module 1203, thereby implementing the model training method. That is, the memory 1306 has instructions for the computing device 1300 to execute the model training method provided in the present application.

[0184] The communication interface 1308 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, and the like, to realize the communication between the computing device 1300 and other devices or communication networks.

[0185] The embodiments of the present application also provide a computing device cluster. The computing device cluster comprises at least one computing device. The computing device can be a server. In some embodiments, the computing device can also be a desktop computer, a notebook computer, a terminal device such as a smart phone, or the like.

[0186] As shown in FIG. 14, the computing device cluster comprises at least one computing device 1300. The same instructions for performing the model training method can be stored in the memory 1306 of one or more computing devices 1300 in the computing device cluster.

[0187] In some possible implementation manners, partial instructions for performing the model training method can also be respectively stored in the memory 1306 of one or more computing devices 1300 in the computing device cluster. In other words, the combination of one or more computing devices 1300 can collectively execute the instructions for performing the model training method.

[0188] It should be noted that the memory 1306 in different computing devices 1300 in the computing device cluster can store different instructions, respectively used for performing partial functions of the data access apparatus. That is, the instructions stored in the memory 1306 in different computing devices 1300 can implement the functions of one or more of the training module 1201, the communication module 1202, and the management module 1203.

[0189] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network, a local area network, or the like.

[0190] The embodiments of the present application also provide a computer program product comprising instructions. The computer program product can be software or a program product comprising instructions, capable of running on a computing device or stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to perform the model training method.

[0191] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center comprising one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), or the like. The computer readable storage medium comprises instructions, which instruct the computing device to perform the model training method.

[0192] Optionally, the computer execution instructions in the embodiments of the present application can also be referred to as application program codes, which are not specifically limited in the embodiments of the present application.

[0193] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor which is implemented by reading software codes stored in a memory.

[0194] It can be understood by those skilled in the art that the first, second, etc. various numerical designations involved in the present application are only for the convenience of description, and do not limit the scope of the embodiments of the present application, nor represent the order. "And / or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after are a kind of "or" relationship. "At least one" means one or more. At least two means two or more. "At least one", "any one" or similar expressions mean any combination of these items, including any combination of single item (one) or multiple items. For example, at least one (one, kind) of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, and c can be single or multiple. "Multiple" means two or more, and other quantifiers are similar. In addition, for the elements (element) appearing in the singular form "a", "an" and "the", unless the context clearly indicates otherwise, it does not mean "one or only one", but means "one or more than one". For example, "a device" means one or more such devices.

[0195] In the embodiments described above, all or some of the embodiments can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, all or some of the embodiments can be implemented in the form of a computer program product storing computer program instructions. When the computer program instructions are loaded into and executed by a computer, all or some of the procedures or functions described in the embodiments of the present application are performed. The computer can be a general purpose computer, a special purpose computer, a computer network, or another programmable apparatus. The computer program instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer program instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as a coaxial cable, an optical fiber, a digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0196] The various illustrative logical blocks, modules, circuits, and algorithms described in connection with the embodiments disclosed herein can be implemented or performed by a general purpose processor, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the described functions. The general purpose processor can be a microprocessor, optionally the general purpose processor can also be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration.

[0197] The steps of the methods or algorithms described in the embodiments disclosed herein can be directly embedded in hardware, a software unit executed by a processor, or a combination of the two. The software unit can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium in the art. The storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Alternatively, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0199] Although the present application has been described in connection with certain specific features and embodiments thereof, it is to be understood that various modifications and changes can be made thereto without departing from the spirit and scope of the application. Accordingly, it is intended that all such modifications and changes be included within the scope of the application as defined by the following claims and their equivalents. It will be obvious to those having ordinary skill in the art that various changes and modifications can be made therein without departing from the spirit and scope of the application. It is intended that all such changes and modifications be included within the scope of the application as defined by the following claims and their equivalents.

Claims

1. An accelerator cluster, characterized by, The cluster comprises a plurality of accelerators, each of which is deployed with a training model; The first accelerator is configured to perform first round training on the training model to obtain a first parameter set; the first parameter set comprises first type parameters and second type parameters; The first accelerator is further configured to send the first type parameters to a second accelerator before the start of the second round training, and send the second type parameters to the second accelerator after the start of the second round training; the first accelerator is any one of the plurality of accelerators, and the second accelerator is an accelerator other than the first accelerator among the plurality of accelerators.

2. The cluster of claim 1, wherein, The first type parameters are used to update training parameters of the second accelerator in the second round training.

3. The cluster of any of claims 1 or 2, wherein The first accelerator is further configured to receive communication information, the communication information being used to indicate the first type parameters and / or the second type parameters.

4. The cluster of any of claims 1 or 2, wherein The first accelerator is further configured to determine the first type parameters and the second type parameters.

5. The cluster of any of claims 1-4, wherein, The second type parameters comprise first sub-type parameters and second sub-type parameters; When the first accelerator transmits the second type parameters to the second accelerator, the first accelerator is specifically configured to: transmit the first sub-type parameters to the second accelerator before the start of the third round training after the start of the second round training; and transmit the second sub-type parameters to the second accelerator after the start of the third round training.

6. The cluster of claim 5, wherein, The first sub-type parameters are used to update the training parameters of the second accelerator in the second round training.

7. The cluster of any of claims 1-6, wherein, The first type parameters comprise gradients of training parameters to be updated, the gradients of the training parameters being used to determine updated values of the training parameters, and the training parameters are embedding vectors.

8. A model training method, comprising: The method is applied to an accelerator cluster, the cluster comprising a plurality of accelerators, each of which is deployed with a training model; the method comprises: A first accelerator performs first round training on the training model to obtain a first parameter set; the first parameter set comprises first type parameters and second type parameters; The first accelerator sends the first type parameters to a second accelerator before the start of the second round training, and sends the second type parameters to the second accelerator after the start of the second round training; the first accelerator is any one of the plurality of accelerators, and the second accelerator is an accelerator other than the first accelerator among the plurality of accelerators.

9. The method of claim 8, wherein, The method comprises: The first accelerator determines the first type parameters and the second type parameters, or The first accelerator receives communication information, the communication information being used to indicate the first type parameters and / or the second type parameters.

10. The method of claim 8 or 9, wherein, The second type parameters comprise first sub-type parameters and second sub-type parameters; The sending of the second type parameters to the second accelerator after the start of the second round training comprises: the first accelerator transmits the first sub-type parameters to the second accelerator before the start of the third round training after the start of the second round training; and the first accelerator transmits the second sub-type parameters to the second accelerator after the start of the third round training.

11. A model training apparatus, comprising: The apparatus comprises: a training module configured to perform a first round of training on the training model to obtain a first parameter set; the first parameter set comprises first parameters and second parameters; a communication module configured to send the first parameters to a second accelerator before the second round of training starts, and send the second parameters to the second accelerator after the second round of training starts; the first accelerator is any one of the plurality of accelerators, and the second accelerator is any one of the plurality of accelerators except the first accelerator.

12. The apparatus of claim 11, wherein, The apparatus further comprises a management module; the management module is further configured to determine the first parameters and the second parameters.

13. The apparatus of claim 11 or 12, wherein, The second parameters comprise first sub-parameters and second sub-parameters; the communication module sends the second parameters to the second accelerator after the second round of training starts, and specifically configured to: transmit the first sub-parameters to the second accelerator before a third round of training starts after the second round of training starts; and transmit the second sub-parameters to the second accelerator after the third round of training starts.

14. A training system, characterized by The system comprises the accelerator cluster and the processor according to any one of claims 1-7. The processor is configured to send communication information to the first accelerator, the communication information being used to indicate the first parameters and / or the second parameters.

15. A computing device, comprising: The apparatus comprises a processor and a memory, the memory stores instructions, and the processor executes the instructions to cause the apparatus to perform the method according to any one of claims 8-10.

16. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, and when the computer program or instructions are executed by the communication apparatus, the method according to any one of claims 8-10 is implemented.

Images