Model training method and device

Abstract

The invention discloses a model training method and device, and the method is applied to an accelerator cluster, the accelerator cluster comprises a plurality of accelerators, and each accelerator is provided with a training model. Taking a first accelerator in the plurality of accelerators as an example: the first accelerator is used for executing a first round of training on the training model to obtain a first parameter set; the parameter set comprises a first type of parameters and a second type of parameters; the first accelerator is further used for sending the first type of parameters to the second accelerator before the second round of training is started and sending the second type of parameters to the second accelerator after the second round of training is started. According to the mode, the data volume needing to be transmitted between the two rounds of training is reduced, that is, only the first type of parameters need to be transmitted before the second round of training is started, so that the communication time between accelerators can be shortened, the communication overhead is reduced, the duration time of a whole model training task is shortened, and the model training efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a model training method and apparatus. Background Technology

[0002] With the rapid development of the internet and the continuous increase in user scale, recommender systems play a crucial role in providing personalized user experiences. The rapid development of deep learning technology has brought new opportunities and challenges to recommender systems. Deep learning models can learn more accurate representations of user interests from massive amounts of data, thereby improving the performance of recommender systems. However, with the continuous development of models and the increasing number of parameters, the training efficiency of recommender models has become an urgent problem to be solved. Summary of the Invention

[0003] This application provides a model training method and apparatus for improving the training efficiency of recommended models.

[0004] In a first aspect, this application provides an accelerator cluster comprising multiple accelerators, each of which is equipped with a training model. Taking the first accelerator among the multiple accelerators as an example: the first accelerator is used to perform a first round of training on the training model to obtain a first set of parameters; the first set of parameters includes a first type of parameters and a second type of parameters; the first accelerator is also used to send the first type of parameters to the second accelerator before the start of the second round of training, and to send the second type of parameters to the second accelerator after the start of the second round of training; wherein, the first accelerator is any one of the multiple accelerators, the second accelerator is any accelerator other than the first accelerator among the multiple accelerators, and the first round of training and the second round of training can identify any two adjacent rounds of training performed on the training model.

[0005] Through the above design, the first accelerator obtains the first type of parameters and the second type of parameters after the first round of training. Then, the first type of parameters are transmitted before the second round of training, while the transmission of the second type of parameters is hidden during the second round of training. This allows model training and partial parameter transmission to be carried out in parallel. Compared with the existing technology that transmits the first type of parameters and the second type of parameters before the start of the second round of training, this application reduces the amount of data that needs to be transmitted between the two rounds of training. That is, only the first type of parameters need to be transmitted before the start of the second round of training. Therefore, the communication time between accelerators can be shortened, thereby shortening the duration of the entire model training task and improving the model training efficiency.

[0006] In one possible design, the first type of parameters are used to update the training parameters of the second accelerator in the second round of training.

[0007] With the above design, the first accelerator transmits the first type of parameters to the second accelerator after the first round of training. In this way, the second accelerator can start the second round of training after updating the training parameters required for the second round of training with the first type of parameters, thereby shortening the time interval between the two rounds of training, improving training efficiency, and ensuring the training effect by using the updated training parameters.

[0008] In one possible design, the cluster also includes a processor; the first accelerator is also used to receive communication information indicating a first type of parameter and / or a second type of parameter. For example, this communication information may be determined by the processor.

[0009] The above design eliminates the need for an accelerator to determine the first and second types of parameters, saving accelerator computing power and improving model training efficiency.

[0010] In one possible design, the first accelerator is also used to determine a first type of parameter and a second type of parameter.

[0011] With the above design, after obtaining the parameter set after the first round of training, the accelerator determines the first type of parameters and the second type of parameters in the parameter set, which does not need to be determined by the processor. This can further offload the computation work from the processor to the accelerator, which is beneficial to improving the utilization of the processor. In addition, the accelerator determines its own communication parameters (such as the first type of parameters and the second type of parameters), and the accelerator does not need to receive communication information, which can save the accelerator's communication bandwidth resources.

[0012] In one possible design, the second type of parameters includes a first subtype of parameters and a second subtype of parameters; when the first accelerator transmits the second type of parameters to the second accelerator, it is specifically used to: transmit the first subtype of parameters to the second accelerator after the start of the second round of training and before the start of the third round of training; and transmit the second subtype of parameters to the second accelerator after the start of the third round of training.

[0013] Through the above design, the second type of parameters are divided into the first subclass parameters and the second subclass parameters. The second subclass parameters are hidden during the transmission in the third round of training, realizing more granular communication parameters, further shortening the communication time between the second and third rounds of training, and improving the model training efficiency.

[0014] In one possible design, the first subclass parameter is used to update the training parameters of the second accelerator in the second round of training.

[0015] Through the above design, the second accelerator can update the training parameters based on the first subclass parameters, and then retrain the updated training parameters in the second round of training, thereby ensuring the training effect and benefits while shortening the communication time between the two rounds of training.

[0016] In one possible design, the first type of parameters is determined based on the index information of the first round of training (denoted as the first index information) and the index information of the second round of training (denoted as the second index information); wherein the first index information is used to indicate the training parameters of the first accelerator in the first round of training, and the second index information is used to indicate the training parameters of the second accelerator in the second round of training.

[0017] Through the above design, the accuracy of the determined first type of parameters can be guaranteed based on the first index information and the second index information. Furthermore, the index information for each training round in the model training is fixed. Therefore, the index information for each training round can be obtained in advance to determine the parameters to be transmitted before two training rounds, such as the first type of parameters. In this way, the accelerator can start parameter transmission after completing one training round, avoiding delays.

[0018] In one possible design, the first type of parameters includes the gradients of the training parameters to be updated, where the training parameters are embedding vectors, and the gradients of the embedding vectors are used to determine the updated values ​​of the embedding vectors.

[0019] In one possible design, the training model is a recommendation model, the second accelerator includes a sparse table, and the training parameters are the partial embedding vectors included in the sparse table; after receiving the first type of parameters, the second accelerator is also used to: update the first embedding vector in the sparse table based on the first type of parameters to obtain the updated sparse table; the second embedding vector in the second round of training of the second accelerator includes the first embedding vector in the updated sparse table.

[0020] With the above design, accelerator clusters can be used to perform multi-round training of the recommendation model. Based on this method, the gradients of some embedding vectors obtained in each round of training can be hidden and transmitted in the next round of training. It is not necessary to transmit all the gradients of the embedded vectors obtained after training before starting the next round of training, thereby shortening the communication waiting time between two rounds of training and improving the training efficiency of the recommendation model in the recommendation scenario.

[0021] Secondly, this application provides a model training method applied to an accelerator cluster, the cluster comprising multiple accelerators, each accelerator having a training model deployed on it; the method comprising: a first accelerator performing a first round of training on the training model to obtain a first parameter set; the parameter set comprising a first type of parameter and a second type of parameter; the first accelerator sending the first type of parameter to a second accelerator before the start of a second round of training, and sending the second type of parameter to the second accelerator after the start of the second round of training; the first accelerator being any one of the multiple accelerators, and the second accelerator being any one of the multiple accelerators other than the first accelerator.

[0022] In one possible design, the first type of parameters is used to update the training parameters of the second accelerator in the second round of training.

[0023] In one possible design, the cluster also includes a processor; the first accelerator is also used to receive communication information determined by the processor, which indicates a first type of parameter and / or a second type of parameter.

[0024] In one possible design, the first accelerator also needs to determine the first type of parameters and the second type of parameters.

[0025] In one possible design, the second type of parameters includes a first subclass parameter and a second subclass parameter; sending the second type of parameters to the second accelerator after the start of the second round of training includes: transmitting the first subclass parameter to the second accelerator after the start of the second round of training and before the start of the third round of training; and transmitting the second subclass parameter to the second accelerator after the start of the third round of training.

[0026] In one possible design, the first subclass parameter is used to update the training parameters of the second accelerator in the second round of training.

[0027] In one possible design, the first type of parameters includes the gradients of the training parameters to be updated, which are used to determine the updated values ​​of the training parameters, which are embedding vectors.

[0028] Thirdly, this application also provides a model training device that has the function of implementing the first accelerator behavior described in the first aspect above. The beneficial effects can be found in the description of the first aspect, and will not be repeated here. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the device structure includes a training module and a communication module; optionally, it also includes a management module. These modules can implement the functions of the behavior in the method of the first aspect above, as detailed in the method examples, and will not be repeated here.

[0029] Fourthly, this application also provides a computing device, the structure of which includes a processor and a memory, the memory being used to store instructions and / or data. The memory is coupled to the processor, and when the processor executes the program instructions stored in the memory, it can implement the functionality of the method described in the first aspect. The computing device also includes a communication interface for communicating with other devices.

[0030] Fifthly, this application provides a training system that may include an accelerator cluster and a processor as described in the first aspect above. The processor is used to manage the parameter transmission method between the accelerator clusters, such as determining a first type of parameter and a second type of parameter in a first parameter set obtained by a first accelerator in a first round of training, and indicating the first type of parameter and the second type of parameter in the first parameter set through communication information.

[0031] In a sixth aspect, this application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described in the second aspect and various possible designs of the second aspect.

[0032] In a seventh aspect, this application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods described in the second aspect and various possible designs of the second aspect.

[0033] Eighthly, this application also provides a computing chip connected to a memory, the chip being used to read and execute software programs stored in the memory, and to execute the methods in the second aspect and various possible implementations of the second aspect.

[0034] For the description of the beneficial effects of any of the second to the seventh aspects mentioned above, please refer to the description of the technical effects in any possible implementation of the first aspect mentioned above, and this application will not repeat it here. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the architecture of a recommendation model;

[0036] Figure 2 This is a schematic diagram of an existing technical solution;

[0037] Figure 3 This is a schematic diagram of another existing technical solution;

[0038] Figure 4 A schematic diagram of a system architecture provided for an embodiment of this application;

[0039] Figure 5 A flowchart illustrating the model training method provided in the embodiments of this application;

[0040] Figure 6 This application provides an embodiment of a proposed deployment architecture for a recommendation model.

[0041] Figure 7 The intended deployment architecture for another recommended model provided in the embodiments of this application;

[0042] Figure 8 A schematic diagram illustrating a model training task provided in an embodiment of this application;

[0043] Figure 9 This is a schematic diagram of a parameter transmission process provided in an embodiment of this application;

[0044] Figure 10 This is a schematic diagram of a parameter communication method provided in an embodiment of this application;

[0045] Figure 11 A schematic diagram illustrating a fine-grained cold update parameter transmission process provided in an embodiment of this application;

[0046] Figure 12 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0047] Figure 13 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0048] Figure 14 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0049] "Recommendation" is a common business scenario in current internet applications, such as "You May Like" and "You've Seen Again" in shopping apps, video apps, and browsers. Typically, recommendation models determine recommended content based on user characteristics. Recommendation models in these scenarios often exhibit both large-scale (e.g., hundreds of GB to TB) sparsity and small-scale (e.g., MB to GB) density.

[0050] Figure 1 An exemplary diagram of a recommendation model is shown. Figure 1 As shown, the recommendation model includes a sparse layer (embedding look-up table) and a dense layer.

[0051] For training the recommendation model, the sparse layer is used to retrieve and assemble embedding vectors from the sparse embedding table based on the model's input (including feature IDs) using operators such as lookup. The sparse table can be understood as a complete embedding vector table that maps all feature IDs within a feature set to embedding vectors. A feature ID is a numerical ID encoded as a feature, used to uniquely identify it. The feature ID, after passing through the sparse layer, yields its corresponding sparse representation, i.e., the embedding vector. Specifically, the embedding vector corresponding to the input feature ID is obtained from the sparse table through a lookup. Then, the embedding vector obtained from the lookup is input into the dense layer for training, and the gradient of this embedding vector is calculated through backpropagation. Based on this gradient, the embedding vector in the sparse table is updated.

[0052] Currently, improving the computational efficiency of dense layers is a hot research topic in the industry. Several existing technical solutions are provided below:

[0053] In the first existing technical solution, computationally intensive tasks are migrated from the parameter server (PS) cluster on the central processing unit (CPU) to a general processing unit (GPU) / embedded neural network processing unit (NPU) cluster, and data parallelism is used to fully utilize the computing power of the GPU / NPU cluster. Furthermore, to further reduce the impact of memory access latency on the parameter server and the memory access latency of copying memory from the CPU to the GPU / NPU (host to device, H2D) within the GPU / NPU node on training performance, the sparse table can be deployed in the on-chip high-bandwidth memory (HBM) of the GPU / NPU. Since HBM capacity is limited, it is currently commonly deployed using model parallelism.

[0054] See Figure 2 As shown, model parallelism can refer to distributing different layers or parameters of a model across different nodes (such as GPUs / NPUs). Here, model parallelism specifically refers to sparse layers. For sparse layer model parallelism, a single complete sparse table can be divided into multiple data blocks, each stored in the HBM (Hardware Modeling) of the GPU / NPU. It can be understood that the size of each data block is related to the capacity of the HBM. Figure 2 In this approach, the sparse table is divided into four data blocks, each of which is pre-stored in the local HBM of each node, such as... Figure 2 In this setup, data block 0 is stored in the HBM of node 0, data block 1 is stored in the HBM of node 1, data block 2 is stored in the HBM of node 2, and so on. Nodes obtaining embedding vectors from their local HBM can accelerate data retrieval, thereby speeding up training.

[0055] Data parallelism refers to dividing a dataset (including feature IDs) into multiple subsets, each of which is assigned to a different node (e.g., GPU / NPU). Each node has a complete copy of the model. Note that this data parallelism refers to dense layers; dense layer data parallelism means that each node deploys a complete dense layer. Each node uses its own model copy (dense layer) to process a different subset of data. For example, in... Figure 2In this process, the dataset is divided into four data subsets, which are assigned to nodes 0, 1, 2, and 3 respectively. Each node deploys a complete dense layer. Each node looks up the feature IDs included in the input data subset, and uses the embedding vectors obtained from the lookup table to form a batch input into the dense layer for training. After calculating the gradient, the gradient is synchronized with other nodes through set communication, and the embedding vectors in the sparse table are updated based on the gradient.

[0056] This scheme suffers from long node communication synchronization latency because the embedding vectors input by each node to the dense layer are typically scattered across the entire sparse table, for example, ... Figure 2 As shown, the embedding vectors included in batch0 are scattered in data blocks 0-4, and the embedding vectors included in batch1 are also scattered in data blocks 0-4. However, data blocks 0-4 are stored in different nodes. Therefore, each node needs to communicate forward and backward through set communication such as global exchange communication (All to All, A2A).

[0057] Forward communication refers to multiple nodes exchanging their embedding vectors via A2A, where A2A means pairwise interaction. For example, Figure 2 In this process, node 0 extracts the embedding vector needed by node 1 from its local data block 0 and sends it to node 1. Node 1 extracts the embedding vector needed by node 0 from its local data block 1 and sends it to node 0. Node 1 extracts the embedding vector needed by node 2 from data block 1 and sends it to node 2. Node 2 extracts the embedding vector needed by node 1 from its local data block 2 and sends it to node 1, and so on. Each node can combine the embedding vector extracted from its local data block with the embedding vectors sent by other nodes to form the batch input for this round of training in the dense layer. Reverse communication refers to each node exchanging the gradients of its trained embedding vectors after obtaining the gradients of its own embedded vectors. In this way, each node updates the embedding vectors in its local data block based on the gradients sent by the other nodes.

[0058] As can be seen, the more nodes there are in the above method, the longer the interaction time required for forward and reverse communication. All nodes need to wait synchronously during forward and reverse communication, resulting in poor system performance.

[0059] In the second existing technical solution, a hybrid parallel scheme is provided. This hybrid parallel scheme leverages the multi-table scenario commonly found in recommendation systems, employing data parallelism for the dense layer and either model parallelism or data parallelism for the sparse layer based on the size of the sparse table.

[0060] Combination Figure 3To understand this, for large sparse tables, a model-parallel approach is used, where multiple nodes jointly store a complete sparse table, and each node's local HBM stores a portion of the same sparse table's data. For smaller sparse tables that can be stored locally, a data-parallel approach is used, where each node's local HBM stores a complete smaller (sparse) table. This eliminates the need for forward communication between nodes to exchange data in the smaller table, reducing synchronization cycles and latency, resulting in better scale-out performance in large-scale scenarios.

[0061] However, the above scheme suffers from the problem of increasing synchronization data volume. In the data-parallel approach, each node needs to compute and synchronize the gradient of the complete sparse table stored locally or the gradient of the embedding vectors activated by all nodes. In the model-parallel approach, each node only needs to synchronize the gradient of the embedding vector activated by the current node. Therefore, the reverse communication time increases, training efficiency decreases, and this will affect the overall benefit of the scheme.

[0062] In view of this, this application provides a model training method to solve the problems of long node synchronization waiting time, poor system performance, and low training efficiency caused by communication methods in existing recommendation model training scenarios. Figure 3 Taking the application scenario shown as an example, this method takes advantage of the embedded vector update characteristics of sparse tables in model training, performs multiple rounds of training on sparse tables, hides the cold update parameters of each round of training in the subsequent training processes, reduces the amount of parameters that need to be transmitted between two rounds of training, thereby reducing node synchronization waiting time and improving system performance and training efficiency.

[0063] The technical solutions provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0064] Figure 4 This is a schematic diagram of the architecture of a distributed training system provided in an embodiment of this application. Figure 4 As shown, the system 10 includes an accelerator cluster. This accelerator cluster includes multiple accelerators 112. The individual accelerators 112 can communicate with each other.

[0065] Functionally, the multiple accelerators 112 are used to execute the model training methods provided in the embodiments of this application, such as to perform model training tasks.

[0066] In terms of hardware, the accelerator 112 may include, but is not limited to: a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), an application processor (AP), a modem, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, hardware logic circuits, a processing core, an ASIC, an AI chip, or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), or any combination thereof.

[0067] The multiple accelerators 112 can be integrated in different computing nodes 110, or in the same computing node 110. The computing node 110 can be a computing device, such as a server, desktop computer, etc., and is not specifically limited.

[0068] Optionally, the system 10 also includes a processor 111, which may be a central processing unit (CPU) for managing the communication between the multiple accelerators 112. This will be described in detail below and will not be repeated here. It should be noted that... Figure 4 As shown, the processor 111 and the accelerator 112 are integrated in the same computing node 110. This application embodiment also supports the processor 111 and the accelerator 112 being located in different computing nodes 110.

[0069] It should be noted that, Figure 4 The system 10 shown is merely an example; the systems applicable to the embodiments of this application may be relatively different. Figure 4 It may have more or fewer components. This application does not limit this to the embodiments.

[0070] The following can be applied to Figure 4 Taking the system shown as an example, combined with Figure 5The model training method provided in the embodiments of this application will be described in detail. Figure 5 This is a flowchart illustrating the model training method provided in the embodiments of this application.

[0071] Taking the application of this model training method to a recommendation scenario as an example, the multiple accelerators 112 can be used to train the recommendation model in parallel. In one training scenario, such as... Figure 6 As shown, both the sparse and dense layers in this recommendation model are deployed in data parallelism. That is, each accelerator 112 (e.g., GPU0 to GPU7) stores a complete sparse table, and each accelerator 112 deploys a complete copy of the dense layer. In another training scenario, such as... Figure 7 As shown, the sparse layer in this recommendation model is deployed in a model-parallel manner, while the dense layer is deployed in a data-parallel manner. That is, in this scenario, the sparse table is divided into multiple data blocks and stored in multiple accelerators 112, and each accelerator 112 has a complete copy of the dense layer deployed on it.

[0072] The following method is applied to Figure 6 The method will be introduced using the training scenario shown as an example. Figure 5 As shown, the method may include the following steps:

[0073] Step 500: Processor 111 obtains the input index information of the multiple training rounds included in the model training task.

[0074] For example, Figure 8 This diagram illustrates the execution of a model training task according to an embodiment of this application. The model training task may include multiple rounds of training on the embedding vectors included in the sparse table of the recommendation model.

[0075] like Figure 8 As shown, the model training task includes multiple rounds of training, each round of training includes multiple sub-tasks, and these multiple sub-tasks are handled by the multiple accelerators 112 (such as...). Figure 8 The process is executed in parallel (GPU0-GPUn), where each subtask is used to train a portion of the embedding vectors in the sparse table.

[0076] Taking a training round as an example, the first training round includes subtask 0, subtask 1, ..., subtask n. GPU 0 is used to execute subtask 0, GPU 1 is used to execute subtask 1, GPU 2 is used to execute subtask 2, and so on.

[0077] At a finer granular level, for each subtask, each subtask has input index information. This input index information indicates the input data for that subtask, which refers to the embedding vector input to the dense layer. For example, the input index information for subtask 0 indicates the input data batch 0, the input index information for subtask 1 indicates the input data batch 1, the input index information for subtask 2 indicates the input data batch 2, and so on.

[0078] The input index information includes feature IDs that indicate the embedding vectors. For example, the input index information for subtask 0 includes feature IDs (0, 2, 1, 8). GPU0 performs a lookup table based on feature IDs (0, 2, 1, 8), retrieving the embedding vector corresponding to feature ID = 0 (denoted as embedding vector 0), the embedding vector corresponding to feature ID = 2 (denoted as embedding vector 1), the embedding vector corresponding to feature ID = 1 (denoted as embedding vector 2), and the embedding vector corresponding to feature ID = 8 (denoted as embedding vector 3) from the sparse table. Batch 0 includes embedding vector 0, embedding vector 1, embedding vector 2, and embedding vector 3.

[0079] For example, the input index information for subtask 1 includes feature IDs (1, 3, 4, 7). GPU1 looks up the sparse table based on feature IDs (1, 3, 4, 7), retrieving the embedding vectors corresponding to feature ID = 1 (denoted as embedding vector 4), feature ID = 3 (denoted as embedding vector 5), feature ID = 4 (denoted as embedding vector 6), and feature ID = 7 (denoted as embedding vector 7). Batch 1 includes embedding vectors 4, 5, 6, and 7.

[0080] In a coarse-grained manner, for a single training round, the input index information of a single training round includes the input index information of all subtasks. In step 500, the processor 111 obtains the input index information of multiple training rounds. The input index information of multiple training rounds refers to the input index information of each training round in multiple training rounds. For the sake of simplicity, it will be referred to as the input index information of multiple training rounds from now on.

[0081] The input index information for one round of training is used to indicate the input data for that round of training (denoted as globalbatch, or GB for short). The input data for one round of training includes the input data of all subtasks in that round. For example, the input index information for the first round of training includes the input index information of subtask 0, subtask 1, ..., and subtask n. The input index information for the first round of training indicates the input data GB0, which includes batch 0, batch 1, ..., and batch n. The input data GB1 for the second round of training includes batch 0', batch 1', ..., and batch n'.

[0082] It is understandable that the embedding vectors included in the input data batch are the training parameters to be trained for each subtask.

[0083] Specifically, Figure 8 In this process, each GPU inputs the embedding vectors from its input data batch into a dense layer for training, resulting in a parameter set that includes the gradients of the input embedding vectors. For example, GPU0 inputs the embedding vectors from batch 0 into the dense layer for training, resulting in a parameter set par0, which includes the gradients of the embedding vectors in batch 0. Similarly, GPU1 inputs the embedding vectors from batch 1 into the dense layer for training, resulting in a parameter set par1, which includes the gradients of the embedding vectors in batch 1.

[0084] Therefore, for each training round, a gradient parameter set (denoted as globalparameter, or GP for short) is obtained. This gradient parameter set includes the parameter sets of all subtasks in that training round. For example, the gradient parameter set GP0 for the first training round includes par0, par1, ..., par n. That is, GP0 includes the gradients of the embedding vectors in GB0. The gradient parameter set GP1 for the second training round includes par0', par1', ..., par n'. GP1 includes the gradients of the embedding vectors in GB1. And so on.

[0085] Step 501, processor 111 determines the first type of parameters and / or the second type of parameters in the first parameter set obtained in the first round of training of the first accelerator.

[0086] In a macroscopic sense, in the embodiments of this application, the set of gradient parameters for each round of training includes cold update parameters and non-cold update parameters. Non-cold update parameters are transmitted before the next round of training, or in other words, the next round of training starts after the transmission of non-cold update parameters is completed. Cold update parameters are transmitted after the start of the next round of training.

[0087] For example, see Figure 9 As shown, the first round of training yields a gradient parameter set GP0. As mentioned earlier, GP0 includes the gradients of the embedded vectors in GB0. GP0 includes non-cold update parameters (denoted as GP0-non-cold update parameters) and cold update parameters (denoted as GP0-cold update parameters). That is, GP0-non-cold update parameters include the gradients of some embedded vectors in GB0, and GP0-cold update parameters include the gradients of the remaining embedded vectors in GB0. GP0-non-cold update parameters are transmitted before the start of the second round of training, and GP0-cold update parameters are transmitted after the start of the second round of training.

[0088] The second round of training yields a gradient parameter set GP1, which includes the gradients of the embedded vectors in GB1. GP1 includes non-cold update parameters (denoted as GP1-non-cold update parameters) and cold update parameters (denoted as GP1-cold update parameters). The GP1-non-cold update parameters are transmitted before the start of the third round of training, and the GP1-cold update parameters are transmitted after the start of the third round of training.

[0089] The third round of training yields a gradient parameter set GP2, which includes the gradients of the embedded vectors in GB2. GP2 includes non-cold update parameters (denoted as GP2-non-cold update parameters) and cold update parameters (denoted as GP2-cold update parameters). The GP2-non-cold update parameters are transmitted before the start of the fourth round of training, and the GP2-cold update parameters are transmitted after the start of the fourth round of training.

[0090] And so on.

[0091] For ease of explanation, the cold update parameters and non-cold update parameters in the gradient parameter set obtained from the previous training round in two adjacent training rounds will be referred to as the cold update parameters and non-cold update parameters between the two training rounds.

[0092] In one implementation, the processor 111 can determine the cold update parameters and non-cold update parameters between two training rounds based on the input index information of any two adjacent training rounds.

[0093] For example, in step 501, processor 111 obtains the input index information of the first round of training (denoted as input index information 0) and the input index information of the second round of training (denoted as input index information 1). Then, in step 502, processor 111 determines the cold update parameters / non-cold update parameters in the gradient parameter set GP0 obtained from the first round of training based on input index information 0 and input index information 1.

[0094] For example, in step 501, processor 111 obtains the input index information of the second round of training (input index information 1) and the input index information of the third round of training (denoted as input index information 2). Then, in step 502, processor 111 determines the cold update parameters / non-cold update parameters in the gradient parameter set GP1 obtained from the second round of training based on input index information 1 and input index information 2.

[0095] For example, in step 501, processor 111 obtains the input index information of the third round of training (input index information 2) and the input index information of the fourth round of training (denoted as input index information 3). Then, in step 502, processor 111 determines the cold update parameters / non-cold update parameters in the gradient parameter set GP2 obtained from the third round of training based on input index information 2 and input index information 3.

[0096] In summary, processor 111 repeats steps 501 and 502 each time it determines the cold update parameters / non-cold update parameters between two adjacent training rounds. Alternatively, in step 501, processor 111 can obtain the input index information for all training rounds at once.

[0097] For example, another execution method of the above example: In step 501, processor 111 acquires input index information 0, input index information 1, input index information 2, and input index information 3. In step 502, processor 111 determines the cold update parameters / non-cold update parameters in GP0 based on input index information 0 and input index information 1. Step 502 is repeated, and processor 111 determines the cold update parameters / non-cold update parameters in GP1 based on input index information 1 and input index information 2. Step 502 is repeated again, and processor 111 determines the cold update parameters / non-cold update parameters in GP2 based on input index information 2 and input index information 3. And so on.

[0098] The following describes how to determine cold update parameters / non-cold update parameters based on the input index information from two rounds of training.

[0099] Taking the first and second rounds of training as examples, in this embodiment, the non-cold update parameters (GP0-non-cold update parameters) in GP0 are used to update the embedding vectors in the second round of training. That is, GB1 includes the embedding vectors updated using the GP0-non-cold update parameters. Therefore, the GP0-non-cold update parameters are determined based on the intersection of the embedding vector set GB0 indicated by the input index information of the first round of training and the embedding vector set GB1 indicated by the input index information of the second round of training. The GP0-non-cold update parameters include the gradients of the embedding vectors in this intersection. The GP0-cold update parameters include all parameters in GP0 except for the GP0-non-cold update parameters.

[0100] The following section provides a detailed explanation of how processor 111 instructs multiple accelerators to transfer non-cold update parameters, using All-to-All (A2A) communication as an example.

[0101] Continuing with the first round of training as an example, specifically for each accelerator 112, the GP0-non-cold update parameters include a subset of parameters from the parameter sets obtained from multiple subtasks in the first round of training. The parameter set for each subtask includes the gradient of the embedding vector trained on its respective subtask.

[0102] See Figure 10 As shown, each GPU needs to send the gradient of the embedding vector included in its own trained parameter set to the other GPUs. Based on whether the other GPU uses the embedding vector in the second round of training, the parameters in the parameter set obtained by each GPU can be divided into two categories: one category belongs to non-cold update parameters and the other category belongs to cold update parameters.

[0103] Taking GPU0 as the parameter sender as an example, combined with Figure 8 understand:

[0104] In the first round of training, GPU0 executes subtask 0 and determines batch 0 based on the input index information of subtask 0. Batch 0 includes at least one embedding vector. GPU0 inputs the at least one embedding vector included in batch 0 into a dense layer to obtain a parameter set par0, which includes the gradient of each embedding vector in the at least one embedding vector.

[0105] Furthermore, in the second round of training, GPU1 executes subtask 1' and determines batch 1' based on the input index information of subtask 1', where batch 1' includes at least one embedding vector. GPU2 executes subtask 2' in the second round of training and determines batch 2' based on the input index information of subtask 2'. GPU3 executes subtask 3' in the second round of training and determines batch 3' based on the input index information of subtask 3'. And so on.

[0106] For GPU0 (parameter sender) and GPU1 (parameter receiver), processor 111 obtains the input index information of subtask 0 and subtask 1', determines the intersection 1 of batch 0 and batch 1', and determines two types of parameters included in par0 based on this intersection 1, denoted as the first type of parameters (belonging to non-cold update parameters) and the second type of parameters (belonging to cold update parameters). The first type of parameters includes the gradient of the embedding vectors included in the intersection 1, which is used to update the embedding vectors that GPU1 will use when executing subtask 1' in the second round of training, i.e., the embedding vectors that subtask 1' will train. The second type of parameters includes all parameters in par0 except for the first type of parameters.

[0107] Similarly, for GPU0 (parameter sender) and GPU2 (parameter receiver), processor 111 obtains the input index information of subtask 0 and subtask 2', determines the intersection 2 of batch 0 and batch 2', and determines the two types of parameters included in par0 based on this intersection 2. To distinguish them from the above, they are denoted as the third type of parameters (belonging to non-cold update parameters) and the fourth type of parameters (belonging to cold update parameters). The third type of parameters includes the gradient of the embedding vectors included in the intersection 2, which is used to update the embedding vectors that GPU2 will use when executing subtask 2' in the second round of training, that is, the embedding vectors that subtask 2' will train. The fourth type of parameters includes the parameters in par0 other than the third type of parameters.

[0108] Similarly, for GPU0 (parameter sender) and GPU3 (parameter receiver), processor 111 obtains the input index information of subtask 0 and subtask 3', determines the intersection 3 of batch 0 and batch 3', and determines the two types of parameters included in par0 based on this intersection 3, denoted as the fifth type of parameters (belonging to non-cold update parameters) and the sixth type of parameters (belonging to cold update parameters). The fifth type of parameters includes the gradient of the embedding vectors included in the intersection 3, which is used to update the embedding vectors that GPU3 will use when executing subtask 3' in the second round of training, that is, the embedding vectors that subtask 3' will train. The fifth type of parameters includes all parameters in par0 except for the sixth type of parameters.

[0109] And so on.

[0110] exist Figure 10 In this process, the other accelerators also send non-cold update parameters to the peer accelerators using the same method as GPU0, which will not be elaborated here. Similarly, GPU0 will also act as a parameter receiver to receive non-cold update parameters sent by other GPUs.

[0111] In step 501, processor 111 determines the non-cold update parameters to be transferred between any two GPUs in a training round and instructs the parameter sender via communication information (see step 502). For ease of explanation, this is simplified to processor 111 determining, for both the first and second accelerators, a first type of parameter from the first parameter set obtained by the first accelerator. The first accelerator can be... Figure 8 The second accelerator refers to any one of the accelerators from GPU0 to GPUn. Here, the second accelerator can refer to any one of the accelerators from GPU0 to GPUn except for the first accelerator. This will not be repeated below.

[0112] Step 502, processor 111 sends communication information to the first accelerator.

[0113] Processor 111 sends communication information to the first accelerator, which instructs the first accelerator to transmit a first type of parameter to the second accelerator. For example, in the example above, processor 111 sends communication information 0, communication information 1, communication information 2, ..., communication information n-1 to GPU0. Communication information 0 instructs GPU0 to send a first type of parameter to GPU1. Communication information 1 instructs GPU0 to send a third type of parameter to GPU2. Communication information 2 instructs GPU0 to send a fifth type of parameter to GPU3, and so on.

[0114] Taking communication information 0 as an example, communication information 0 may indicate a first type of parameter, such as the index of the embedding vector corresponding to the first type of parameter, such as a feature ID or a newly generated identifier, used to uniquely identify an embedding vector. The embedding vector corresponding to the gradient of the first embedding vector is the first embedding vector. Alternatively, communication information 0 may also indicate a second type of parameter, such as the index of the embedding vector corresponding to the second type of parameter. GPU0 can determine the first type of parameter in the first parameter set based on the second type of parameter. Furthermore, communication information 0 may indicate both the first type of parameter and the second type of parameter. Additionally, communication information 0 may also include the identifier of GPU1, used to indicate the receiving end of the first type of parameter.

[0115] It should be noted that the above is for illustrative purposes only. The above multiple communication information can be integrated into one communication information. For example, communication information A includes communication information 0 to communication information n-1. In this way, the processor 111 only needs to send communication information A to GPU0.

[0116] Similarly, processor 111 sends communication information to the other GPUs, instructing each GPU on how to send parameters to the other GPUs. See the above description; it will not be repeated here.

[0117] Step 503: The first accelerator completes the first round of training and obtains the first parameter set.

[0118] For example, GPU0-GPUn execute model training tasks in parallel. Each GPU executes its corresponding subtask in each round of training and obtains the parameter set of each subtask. See the relevant description above, which will not be repeated here.

[0119] It should be noted that there is no strict time limit between steps 501-502 and step 503. Steps 501-502 can be executed before step 503, simultaneously, or after step 503.

[0120] Step 504: The first accelerator sends the first type of parameters from the first parameter set to the second accelerator based on the communication information.

[0121] Step 505: The second accelerator updates the embedding vector in the sparse table based on the received parameters.

[0122] Continue to combine Figure 10 Understanding is that GPU0 determines the first type of parameters in the first parameter set based on the received communication information 0, and sends the first type of parameters to GPU1. Correspondingly, GPU1 receives the first type of parameters sent by GPU0, determines the update value of this part of the embedding vector based on the gradient of the embedding vector included in the first type of parameters, and updates this part of the embedding vector in the sparse table stored locally by GPU0 based on the determined update value.

[0123] Similarly, GPU0 determines the third type of parameters in the first parameter set based on the received communication information 1, and sends the third type of parameters to GPU2. Accordingly, GPU2 receives the third type of parameters sent by GPU0, determines the update value of this part of the embedding vector based on the gradient of the embedding vector included in the third type of parameters, and updates this part of the embedding vector in the sparse table stored locally by GPU2 based on the determined update value.

[0124] GPU0 determines the fifth type of parameter in the first parameter set based on the received communication information 2, and sends the fifth type of parameter to GPU3. Correspondingly, GPU3 receives the fifth type of parameter sent by GPU0, determines the update value of this part of the embedding vector based on the gradient of the embedding vector included in the fifth type of parameter, and updates this part of the embedding vector in the sparse table stored locally by GPU3 based on the determined update value.

[0125] And so on.

[0126] Taking GPU1 as an example, it should be noted that in addition to receiving the first type of parameters sent by GPU0, GPU1 may also receive non-cold update parameters sent by other GPUs. GPU1 needs to update the embedding vector in its local sparse table based on all the received non-cold update parameters. That is, in step 505, each GPU needs to update the embedding vector in its sparse table based on all the received non-cold update parameters.

[0127] Step 506: The second accelerator begins the second round of training.

[0128] There are several ways to signal the start of the second round of training. For example, updating the embedding vectors in the sparse table based on all received non-cold update parameters; starting the second round of training subtasks; extracting the embedding vectors to be trained in the second round of subtasks from the sparse table; or inputting the extracted embedding vectors into a dense layer, etc. As long as the local sparse table is updated using the non-cold update parameters from the first round of training, the second round of training can begin. In one implementation, all subtasks start each round of training simultaneously.

[0129] Step 507: After the second round of training begins, the first accelerator sends the second type of parameters from the first parameter set to the second accelerator.

[0130] The following section will continue to describe the transmission process of cold update parameters using the A2A method.

[0131] See also Figure 10 After the start of the second round of training:

[0132] GPU0 sends the second type of parameters from the first parameter set to GPU1. The second type of parameters and the first type of parameters together form the first parameter set. Accordingly, GPU1 receives the second type of parameters and updates its local sparse table based on the second type of parameters.

[0133] GPU0 sends the fourth type of parameter from the first parameter set to GPU2. The fourth type of parameter and the third type of parameter together form the first parameter set. Accordingly, GPU2 receives the fourth type of parameter and updates its local sparse table based on the fourth type of parameter.

[0134] GPU0 sends the sixth type of parameter from the first parameter set to GPU3. The sixth type of parameter and the fifth type of parameter together form the first parameter set. Accordingly, GPU3 receives the sixth type of parameter and updates its local sparse table based on the sixth type of parameter.

[0135] And so on.

[0136] The other accelerators also send cold update parameters to the peer accelerators based on GPU0, which will not be elaborated here.

[0137] Furthermore, embodiments of this application provide a more granular method for transmitting cold update parameters.

[0138] Taking the sending of the second type of parameters from GPU0 to GPU1 as an example, processor 111 determines multiple sub-class parameters in the second type of parameters based on the input index information of the subtasks in two or more rounds of training.

[0139] For example, see Figure 11 As shown, processor 111 obtains the input index information (denoted as B0) of GPU0 executing subtask 0 in the first round of training, the input index information (B1) of GPU1 executing subtask 1' in the second round of training, and the input index information (B2) of GPU1 executing subtask 1" in the third round of training.

[0140] Suppose that the feature IDs included in B0 are (2, 1, 3), the feature IDs included in B1 are (4, 1, 5), and the feature IDs included in B2 are (2, 4, 5).

[0141] If the intersection of B0 and B1 is feature ID = 1, then it can be determined that the first type of parameters in par0 includes feature ID = 1, and the second type of parameters includes feature ID = 2 and feature ID = 3.

[0142] Optionally, the second type of parameters also includes the first sub-class parameters and the second sub-class parameters. The intersection of B0 and B2 is feature ID = 2, which determines that the first sub-class parameters include feature ID = 2, used to update the embedding vector that GPU1 will train in subtask 1" in the third round of training. The second sub-class parameters include the parameters in the second type of parameters other than the first sub-class parameters, that is, the second sub-class parameters include feature ID = 3.

[0143] As mentioned earlier, the first type of parameters is transmitted before the start of the second round of training. The second type of parameters is transmitted after the start of the second round of training. Therefore, the first type of parameters are transmitted after the start of the second round of training and before the start of the third round of training, and are used to update the embedding vectors that will be trained in the sub-tasks during the third round of training. The transmission time of the second type of parameters is not mandatory; they can be transmitted either after the start of the second round of training or after the start of the third round of training. There is no specific limitation, and it can be determined based on the transmission strategy.

[0144] For example, a priority is set for each type of parameter, and the transmission time of each type of parameter is determined based on the priority. The priority of each type of parameter can be determined based on one or more parameters such as the number of training rounds involved, the size of the number of parameters, the transmission time, and the training time of each training round.

[0145] For example, let's take a first parameter set that includes a first type of parameter, a first subclass parameter, and a second subclass parameter:

[0146] (1) The earlier the training rounds involved, the higher the priority.

[0147] The first type of parameters is used to update the embedding vectors of subtasks in the second round of training; therefore, the first type of parameters is involved in the second round of training.

[0148] The first subclass parameter is used to update the embedding vector of the subtask in the third round of training; therefore, the first subclass parameter is involved in the third round of training.

[0149] The second subclass parameter involves an unknown training epoch.

[0150] Therefore, the priority order of various parameters is: first type of parameter > first subtype parameter > second subtype parameter;

[0151] For example, the priority of the first type of parameter in B0 is > the priority of the first subclass parameter in B0 > the priority of the second subclass parameter in B0.

[0152] (2) Non-cold update parameters have higher priority than cold update parameters;

[0153] For non-cold update parameters between the second and third training rounds, such as the first type of parameters in B1 which are involved in the third training round, and the first subtype of parameters in B0 which are also involved in the third training round, since the first type of parameters in B1 belong to non-cold update parameters, while the first subtype of parameters in B0 belong to cold update parameters, the priority of the first type of parameters in B1 is greater than that of the first subtype of parameters in B0.

[0154] The first subclass parameter in the second class of parameters included in B1 involves the fourth round of training, while the training rounds involved in the second subclass parameter in B0 are unknown. Therefore, the first subclass parameter in B1 has a higher priority than the second subclass parameter in B0.

[0155] Based on this, a transmission strategy may include:

[0156] After the second round of training begins, GPU0 first transmits the first subclass parameters of B0. To further shorten the synchronization waiting time between the two training rounds, if the second subclass parameters of B0 can be transmitted before the end of the second round of training, then the second subclass parameters of B0 are transmitted. If the second subclass parameters of B0 are not transmitted or are not transmitted completely during the second round of training, then after the end of the second round of training, the transmission of other parameters is paused, and the non-cold update parameters of the second round of training, such as the first class parameters of B1, are transmitted first after the start of the third round of training. Then the second subclass parameters of B0 are transmitted.

[0157] The above example describes a transmission method for parameters of multiple categories determined by processor 111 based on the input index information from three rounds of training. This application embodiment also supports more rounds of training to manage cold update parameters with finer granularity. For example, based on B0, B1, B2, and B3 (the input index information of subtask 1' executed by GPU1 in the fourth round of training), the second category of parameters in par0 is divided into a first subcategory parameter, a second subcategory parameter, and a third subcategory parameter, as described above. This application embodiment does not limit the granularity of cold update parameter division.

[0158] Through the above design, the first accelerator obtains the first and second types of parameters after the first round of training. Then, the first type of parameters are transmitted before the second round of training, while the transmission of the second type of parameters is hidden during the second round of training. This allows model training and partial parameter transmission to proceed in parallel. Compared to existing solutions that transmit the first and second types of parameters before the start of the second round of training, this application reduces the amount of data that needs to be transmitted between the two rounds of training—that is, only the first type of parameters need to be transmitted before the start of the second round of training. Therefore, it can shorten the communication time between accelerators, reduce communication overhead, and thus shorten the overall model training task duration, improving model training efficiency. Furthermore, using a data-parallel or model-parallel architecture in the sparse layer can reduce the number of synchronization waits, resulting in better scalability.

[0159] The above Figure 5 The method embodiment shown is in Figure 6 The training scenario was introduced below, and the application of this method is discussed in the following sections. Figure 7 The training scenario will be introduced as follows:

[0160] Taking the first and second accelerators as examples again, Figure 6 and Figure 7 The similarities between the two scenarios include: the first accelerator obtains a first set of parameters in the first round of training. The first set of parameters includes a first type of parameter and a second type of parameter. The first type of parameter and the second type of parameter remain unchanged, and the transmission timing of the first type of parameter and the second type of parameter also remains unchanged. For details, please refer to the relevant introduction above, which will not be repeated here.

[0161] The differences between the two scenarios include: Although the first type of parameters is used to update the embedding vectors of the second accelerator in the second round of training, not all of the first type of parameters are sent to the second accelerator. This is because the embedding vectors corresponding to the first type of parameters in the sparse table may be scattered and stored locally on multiple accelerators. Therefore, the processor 111 determines which accelerator each embedding vector corresponding to the first type of parameters is stored on and instructs the first accelerator to send the gradient corresponding to that embedding vector in the first type of parameters to that accelerator. Similarly, based on the same process, the accelerator where the embedding vector corresponding to the second type of parameters is located is determined, and the first accelerator is instructed to send the gradient corresponding to that embedding vector in the second type of parameters to that accelerator.

[0162] Or, in Figure 7 In this scenario, the first accelerator still transmits all the first-class parameters to the second accelerator to accelerate the second accelerator's extraction of embedding vectors from the second round of training. In addition, the first accelerator also needs to send the gradients of the embedding vectors stored on the other accelerators in the first-class parameters to the corresponding accelerators, ensuring that the embedding vectors in the data blocks of the sparse table stored locally by each accelerator are up-to-date.

[0163] in addition, Figure 5 The illustrated embodiment takes the parameter transmission method of processor 111 managing each accelerator as an example. This application embodiment also supports the case where system 10 does not include processor 111. Therefore, this application embodiment also provides another implementation method: each accelerator 112 performs the function performed by processor 111: accelerator 112 itself determines the non-cold update parameters (such as the first type of parameters) and cold update parameters (such as the second type of parameters, and the first sub-type parameters and the second sub-type parameters included in the parameter set obtained in each round of training) that need to be sent to the peer accelerator. Refer to the relevant introduction of processor 111 above, which will not be repeated here.

[0164] It should be noted that (1) the above A2A is just an example. The embodiments of this application do not limit the use of A2A to interact parameters. For example, the All Reduce (AR) communication method can also be used to interact. In addition, it does not limit the use of A2A to interact based on the above process steps. The embodiments of this application do not limit this.

[0165] (2) The first and second rounds of training in steps 503 to 506 can refer to any two adjacent rounds of training. For example, Figure 8 The second and third training rounds, or the third and fourth training rounds, in the model training task shown can be understood as follows: Figure 5 In the method embodiment, the first round of training refers to the i-th round of training, the second round of training refers to the (i+1)-th round of training, where i is an integer not greater than N-1, and N is the number of rounds included in the model training task.

[0166] (3) It should be noted that Figure 10 As an example only, in one possible scenario, the batches of subtasks in two adjacent training rounds do not overlap, in which case the corresponding GPUs do not need to transfer gradients. For example, if batch0 and batch3' do not overlap, then GPU0 does not need to send the gradients from par0 to GPU3.

[0167] Additionally, it's worth noting that if the input data from two training rounds do not overlap, then non-cold update parameters do not need to be transmitted between the two rounds. For example, if GB0 from the first training round and GB1 from the second training round do not overlap, then the second training round can begin immediately after the first training round ends. That is, step 502 is an optional step, not a mandatory one. For instance, when the input data from two training rounds do not overlap, the processor 111 does not need to send communication information to each accelerator 112.

[0168] If the input data of two adjacent training rounds do not overlap, then steps 502 to 504 do not need to be executed. It can be understood that the parameters between the first and second training rounds are cold update parameters. Multiple accelerators can directly start the second training round and exchange the cold update parameters obtained from the first training round after the second training round starts. For the transmission method of cold update parameters, please refer to the above introduction, which will not be repeated here.

[0169] (4) The above method embodiment is shown as an example of application in the recommendation scenario. This application embodiment can also be applied to other scenarios and used in other model training tasks. This application embodiment does not limit this.

[0170] (5) Except Figure 6 and Figure 7 In addition to the training scenarios shown, the embodiments of this application can also be applied to other training scenarios, such as the recommendation model adopting a hybrid parallel deployment, and the embodiments of this application do not limit this.

[0171] The methods provided by the embodiments of this application have been described above with reference to the accompanying drawings. The apparatus provided by the embodiments of this application will be described below with reference to the accompanying drawings.

[0172] Based on the same inventive concept as the method embodiments, this application also provides a model training apparatus for performing the above-described... Figure 5 The method of the method embodiment. For example... Figure 12 As shown, the device 1200 includes a training module 1201, a communication module 1202, and optionally a management module 1203. Specifically, in this device 1200, the modules are connected to each other through a communication path.

[0173] Training module 1201 is used to perform the first round of training on the training model to obtain a first parameter set, which includes first-class parameters and second-class parameters; see details below. Figure 5 The description of step 503 in the method embodiment will not be repeated here.

[0174] Communication module 1202 is used to send the first type of parameters to the second accelerator before the start of the second round of training, and to send the second type of parameters to the second accelerator after the start of the second round of training; the first accelerator is any one of the plurality of accelerators, and the second accelerator is any one of the plurality of accelerators other than the first accelerator. See details below. Figure 5 The descriptions of steps 504-507 in the method embodiment will not be repeated here.

[0175] In one possible implementation, the communication module 1202 is further configured to receive communication information, which indicates the first type of parameters and / or the second type of parameters; the communication information is determined by the processor. See details below. Figure 5 The description of step 502 in the method embodiment will not be repeated here.

[0176] In one possible implementation, the management module 1203 is used to perform... Figure 5 In the method embodiment, the method steps executed by processor 111100 include determining a first type of parameter and / or a second type of parameter in the first parameter set. See details [link to relevant documentation]. Figure 5 The description of step 501 in the method embodiment will not be repeated here.

[0177] This application also provides a computing device 1300. For example... Figure 13 As shown, the computing device 1300 includes a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, the memory 1306, and the communication interface 1308 communicate with each other via the bus 1302. The computing device 1300 may be a server, a storage array, or a hard disk enclosure, etc. It should be understood that this application does not limit the number of processors and memories in the computing device 1300.

[0178] Bus 1302 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 13 The bus 1302 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1302 may include a path for transmitting information between various components of the computing device 1300 (e.g., memory 1306, processor 1304, communication interface 1308).

[0179] The processor 1304 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0180] The memory 1306 may include volatile memory, such as random access memory (RAM). The processor 1304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD), etc., without limitation.

[0181] In one embodiment, the computing device 1300 performs the functions of the data processing device 1200. The memory 1306 stores executable program code, and the processor 1304 executes this executable program code to implement the functions of the aforementioned training module 1201, communication module 1202, and management module 1203, thereby implementing the model training method. That is, the memory 1306 stores instructions for the computing device 1300 to execute the model training method provided in this application.

[0182] The communication interface 1308 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1300 and other devices or communication networks.

[0183] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0184] like Figure 14 As shown, the computing device cluster includes at least one computing device 1300. The memory 1306 in one or more computing devices 1300 in the computing device cluster may store the same instructions for executing model training methods.

[0185] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the model training method. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for executing the model training method.

[0186] It should be noted that the memory 1306 in different computing devices 1300 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data access device. That is, the instructions stored in the memory 1306 of different computing devices 1300 can implement the functions of one or more modules among the training module 1201, communication module 1202, and management module 1203.

[0187] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0188] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a model training method.

[0189] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a model training method.

[0190] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0191] It should be understood that the processor mentioned in the embodiments of this application can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0192] Those skilled in the art will understand that the various numerical designations, such as "first," "second," etc., used in this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application, nor do they indicate a sequential order. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" refers to one or more. "At least two" refers to two or more. "At least one," "any one," or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. "Multiple" refers to two or more, and other quantifiers are similar. Furthermore, for elements appearing in the singular forms "a," "an," and "the," unless the context explicitly specifies otherwise, they do not imply "one or only one," but rather "one or more." For example, "a device" implies one or more such devices.

[0193] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0194] The various illustrative logic units and circuits described in the embodiments of this application can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can also be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0195] The steps of the methods or algorithms described in the embodiments of this application can be directly embedded in hardware, software units executed by a processor, or a combination of both. The software units can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC.

[0196] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0197] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely illustrative descriptions of the application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Thus, if such modifications and modifications fall within the scope of the claims and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. An accelerator cluster, characterized in that, The cluster includes multiple accelerators, each of which is equipped with a training model. A first accelerator is used to perform a first round of training on the training model to obtain a first set of parameters; the first set of parameters includes a first type of parameters and a second type of parameters. The first accelerator is also configured to send the first type of parameters to the second accelerator before the start of the second round of training, and to send the second type of parameters to the second accelerator after the start of the second round of training; the first accelerator is any one of the plurality of accelerators, and the second accelerator is any one of the plurality of accelerators other than the first accelerator.

2. The cluster as described in claim 1, characterized in that, The first type of parameters is used to update the training parameters of the second accelerator in the second round of training.

3. The cluster as described in claim 1 or 2, characterized in that, The first accelerator is also configured to receive communication information, which is used to indicate the first type of parameters and / or the second type of parameters.

4. The cluster as described in claim 1 or 2, characterized in that, The first accelerator is also used to determine the first type of parameters and the second type of parameters.

5. The cluster as described in any one of claims 1-4, characterized in that, The second type of parameter includes the first subclass parameter and the second subclass parameter; When the first accelerator transmits the second type of parameters to the second accelerator, it is specifically used to: transmit the first sub-type parameters to the second accelerator after the start of the second round of training and before the start of the third round of training; and transmit the second sub-type parameters to the second accelerator after the start of the third round of training.

6. The cluster as described in claim 5, characterized in that, The first subclass parameter is used to update the training parameters of the second accelerator in the second round of training.

7. The cluster as described in any one of claims 1-6, characterized in that, The first type of parameter includes the gradient of the training parameter to be updated, the gradient of the training parameter is used to determine the updated value of the training parameter, and the training parameter is an embedding vector.

8. A model training method, characterized in that, The method is applied to an accelerator cluster, the cluster comprising multiple accelerators, each of which has a trained model deployed on it; the method includes: The first accelerator performs a first round of training on the training model to obtain a first set of parameters; the first set of parameters includes a first type of parameters and a second type of parameters. The first accelerator sends the first type of parameters to the second accelerator before the start of the second round of training, and sends the second type of parameters to the second accelerator after the start of the second round of training; the first accelerator is any one of the plurality of accelerators, and the second accelerator is any one of the plurality of accelerators other than the first accelerator.

9. The method as described in claim 8, characterized in that, The method includes: The first accelerator determines the first type of parameters and the second type of parameters, or, The first accelerator receives communication information, which is used to indicate the first type of parameters and / or the second type of parameters.

10. The method as described in claim 8 or 9, characterized in that, The second type of parameter includes the first subclass parameter and the second subclass parameter; Sending the second type of parameters to the second accelerator after the start of the second round of training includes: after the start of the second round of training and before the start of the third round of training, the first accelerator transmits the first sub-type parameters to the second accelerator; after the start of the third round of training, the first accelerator transmits the second sub-type parameters to the second accelerator.

11. A model training device, characterized in that, The device includes: The training module is used to perform a first round of training on the training model to obtain a first set of parameters; the first set of parameters includes a first type of parameters and a second type of parameters. The communication module is used to send the first type of parameters to the second accelerator before the start of the second round of training, and to send the second type of parameters to the second accelerator after the start of the second round of training; the first accelerator is any one of the plurality of accelerators, and the second accelerator is an accelerator other than the first accelerator among the plurality of accelerators.

12. The apparatus as claimed in claim 11, characterized in that, The device also includes a management module; The management module is also used to determine the first type of parameters and the second type of parameters.

13. The apparatus as claimed in claim 11 or 12, characterized in that, The second type of parameter includes the first subclass parameter and the second subclass parameter; The communication module sends the second type of parameters to the second accelerator after the second round of training begins. Specifically, it is used to: transmit the first sub-type parameters to the second accelerator after the second round of training begins and before the third round of training begins; and transmit the second sub-type parameters to the second accelerator after the third round of training begins.

14. A training system, characterized in that, The system includes an accelerator cluster and a processor as described in any one of claims 1-7; The processor is configured to send communication information to the first accelerator, the communication information being used to indicate the first type of parameters and / or the second type of parameters.

15. A computing device, characterized in that, The device includes a processor and a memory, the memory storing instructions, which, when executed by the processor, cause the device to perform the method according to any one of claims 8-10.

16. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a communication device, implement the method as described in any one of claims 8-10.

Images