A large-scale recommendation system training method based on distributed heterogeneous computing

Abstract

The application discloses a kind of large-scale recommendation system training method based on distributed heterogeneous computing in the field of communication technology, comprising the following steps: Cache initialization;Cache replacement algorithm construction;Index mapping algorithm construction;In GPU, the calculation of Sparsefeature is carried out using the Embedding or Embedding_bag function provided by PyTorch, and the calculation of the subsequent Dense part of the model, the technical solution can make full use of GPU computing resources, and the calculation speed is increased;Replace in and out by row optimization is replaced in and out by block, and the throughput is improved;Finally, the scale of the model is further expanded by making full use of computing resources, and the model is distributedly expanded, and the method has creativity and market promotion value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a large-scale recommendation system training method based on distributed heterogeneous computing. BACKGROUND

[0002] The recommendation system model is usually composed of an Embedding (sparse) part and a neural network model (Dense) part. Correspondingly, the input sample of the model can be divided into discrete sparse features and continuous dense features. The sparse features are looked up from the Embedding to find the corresponding parameters, and then combined with the dense features to perform subsequent model calculation.

[0003] In actual scenarios, the Embedding parameters are usually very large and cannot be completely placed in the GPU like traditional deep learning neural network models. However, only a small part of the parameters is needed for each step of training, so it is an I / O intensive task. In contrast, all neural network model parameters are needed for each step of training, and the calculation is usually very dependent on the GPU. Therefore, a distributed and heterogeneous approach is considered to expand the scale of the recommendation system model.

[0004] In order to support the distributed and heterogeneous training of large-scale recommendation system models, the existing solutions mainly have two categories according to the distributed strategy. The first category is based on Parameter Server implementation. In this category of solutions, Persia proposed by the team of Kuai Shou can support the training and reasoning of recommendation systems with trillion-level parameters. The system structure of Persia is divided into four parts:

[0005] 1) The Data loader is responsible for reading data from the disk, database, etc., and inputting the data into the recommendation system model for training after forming a batch;

[0006] 2) The Embedding parameter server is responsible for the storage and update of the Embedding parameters;

[0007] 3) The Embedding worker can have multiple, responsible for reading the embedding parameters needed by each batch from the Embedding parameter server to input the model, and returning the gradient returned by the model to the Embedding parameter server for update;

[0008] 4) The NN worker can have multiple, responsible for the storage, calculation and update of the parameters of the recommendation model.

[0009] The isomers of Persia are now used in the embedding parameter server and worker, which use CPU computing, and the computer main memory can expand the size of the embedding parameters, while the NN worker uses GPU computing. In the distributed design, the embedding parameter server and worker use the parameter server method for management, and the NN worker uses the allreduce method to achieve data parallelism.

[0010] The training process of each step of Persia is as follows:

[0011] 1) The Dataloader collects the sparse feature in each training sample and sends it to the EmbeddingWorker. Then the Embedding Worker assigns a unique ID to it, saves the sparse feature and the corresponding ID locally, and sends the ID to the Dataloader. The Dataloader sends the ID, Dense feature, and label of the sample to the NN worker.

[0012] 2) After the NN worker receives the sample data, it sends a request to the embedding worker that holds the ID and sparse feature, requesting the embedding corresponding to the sparse feature. After the embedding worker receives the request, it requests the embedding parameter server for the embedding parameter required by the ID.

[0013] 3) After the Embedding Worker obtains the parameter from the Embedding Parameter server, it can perform some preset calculations and return the results in the appropriate form to the NN worker.

[0014] 4) After the NN worker receives the embedding parameter, it forms a training sample with the Dense feature and label. When the number of samples reaches the preset batch size, it inputs the recommendation system model for training.

[0015] 5) After the forward computation, the loss function is calculated according to the label, and the gradient is returned to the model according to the back propagation algorithm. The gradients are synchronized among multiple NN workers in the form of AllReduce, and the model parameters are updated. The gradient of the Embedding parameter is returned to the corresponding Embedding worker according to the sample ID, and the Embedding worker updates the received gradient according to the preset calculation and returns it to the Embedding Parameter server.

[0016] 6) The Embedding Parameter server updates the Embedding parameter according to the gradient, and completes one step of training.

[0017] The second type of solution is based on AllReduce implementation. In this type of solution, Torchrec proposed by the PyTorch team can also better support distributed heterogeneous computing of large-scale recommendation system models.

[0018] Torchrec is an end-to-end training framework, and its structure is consistent with traditional supervised training. Dataloader reads a batch of training samples, inputs them into the recommendation model for forward calculation, calculates the loss function according to the sample label, then calculates the gradient of the model parameter according to the back propagation algorithm, and updates the model parameter according to the optimization algorithm.

[0019] For the design on distributed computing, Torchrec mainly relies on AllReduce to synchronize the gradient, thereby supporting Dense part data parallelism. For large-scale Embedding parameters, Torchrec supports four kinds of model parallelism: 1) Table-wise, which divides the Embedding parameter according to the type of sparse feature; 2) Row-wise, which divides the Embedding parameter by row; 3) Column-wise, which divides the Embedding parameter by column; 4) Data-parallel, which is data parallel, and for sparse feature types with less embedding, it is trained through redundant means. Moreover, Torchrec supports the mixing of the four kinds of model parallelism for Embedding parameters.

[0020] For the design on heterogeneous computing, Torchrec extends the data structure of PyTorch, Torch Tensor, and uses the library function provided by Nvidia CUDA platform to realize the UVM Tensor of Universal Virtual Memeory. UVM Tensor realizes the calculation on heterogeneous memory by unifying the memory of GPU and CPU as one memory. It should be noted that UVM Tensor cannot reduce the memory copy between GPU and CPU, but only provides a unified memory programming interface, and the actual memory copy is automatically managed by the driver. However, Persia has four main defects:

[0021] 1) Embedding is entirely stored and calculated by CPU, and GPU is only responsible for the storage and calculation of Dense part, which cannot fully utilize the GPU computing resources;

[0022] 2) The calculation speed is slow, mainly because the Embedding parameter is calculated by CPU, which cannot keep up with the multi-thread parallel calculation of GPU;

[0023] 3) The throughput is low, mainly because the sample batch data structure is unreasonable. When constructing the batch at each training step, it needs to traverse each sample and each sparse feature data in the batch;

[0024] 4) In order to improve the throughput, the Embedding Parameter server uses an asynchronous way to update the parameters, which has a certain impact on the accuracy of the recommendation model. In actual business activities, the decline in accuracy means the decline in user experience of product recommendation, which means direct loss of economic benefits.

[0025] At the same time, Torchrec has two main defects:

[0026] 1) Torchrec uses automatic parallel for Embedding parameters, which automatically calculates the optimal configuration of four distributed parallel modes. However, its calculation method only depends on the communication amount, without considering the calculation amount, which leads to uneven load, and in large-scale recommendation model training, it will waste more computing resources and costs;

[0027] 2) Torchrec's UVM Tensor implementation relies on the underlying directly managing UVM memory, which is not compatible with PyTorch's autograd engine, so it is difficult to expand and modify and customize for actual scenarios. Therefore, a calculation method that meets the actual needs is urgently needed during use. Based on this, the present application designs a large-scale recommendation system training method based on distributed heterogeneous computing to solve the above problems. SUMMARY

[0028] The present application aims to provide a large-scale recommendation system training method based on distributed heterogeneous computing to solve the problems raised in the background art.

[0029] To achieve the above object, the present application provides the following technical solution: a large-scale recommendation system training method based on distributed heterogeneous computing, based on the long-tail characteristics of recommendation system data, using Cache mechanism, including the following steps:

[0030] S1: Cache initialization;

[0031] S2: Cache replacement algorithm construction;

[0032] S3: Index mapping algorithm construction;

[0033] S4: In GPU, use the Embedding or Embedding_bag function provided by PyTorch to calculate Sparsefeature and the subsequent Dense part of the model.

[0034] Preferably, the step S1 is specifically: set the number of Embedding as E, the dimension of each Embedding vector as d, and the Embedding parameter as an Exd matrix, which is placed on the host side;

[0035] Create an Nxd Cache on GPU, that is, the number of Cache rows is N and the dimension of each Cache row is d; the Embedding parameter can be randomly initialized or read pre-trained parameters, and the Cache only needs to allocate memory without initialization;

[0036] Create a mapping table with a size of N to store the mapping of Cache index to Embedding index, and initialize it to -1, indicating empty;

[0037] Create a frequency recording table with a size of N to store the number of times each Cache is accessed, initialized to -1, also indicating empty.

[0038] Preferably, the step S2 is specifically: during training, a batch of training samples is received at each step, and all indexes in the Sparse feature of the samples are detected and ensured to be de-duplicated and not exceed N in quantity, and if exceeding N, an error is reported;

[0039] For the de-duplicated Sparse feature indexes, it is checked according to the mapping table whether the current input indexes are saved in the Cache, and according to the result, it is divided into two sets of Cache hit and miss;

[0040] For the miss set, first, the indexes of the Cache are sorted according to the frequency record table, the idle (-1) Cache row is found, or the Cache row with the least access times is accessed, and the Cache rows in the hit set are filtered out, until the number of Cache rows is the size of the miss, and the indexes of this set of Cache rows are recorded as a list pos, indicating the target position of the Embedding parameters to be read from the CPU and put into the Cache; if it is a non-idle Cache row, the index of the Cache row also needs to be copied to the list pos_write_back, that is, the Cache row of the GPU needs to be written back from the Cache first;

[0041] According to the pos_write_back list, the corresponding Embedding parameters are first read out from the Cache, and the Embedding row index of each Embedding on the CPU is read out from the mapping table, so that the parameters can be written back to the Embedding parameter matrix on the CPU according to the index; subsequently, according to the miss set, the Embedding parameters that need to be exchanged into the GPU Cache are read out from the Embedding parameter matrix, and the Cache row where each parameter needs to be put is found according to the pos list; finally, the mapping table is updated according to the pos list, and the records corresponding to the pos list in the frequency record table are cleared, that is, set to -1.

[0042] Preferably, in the step S3, when the Cache exchange in and out is completed, it can be ensured that all required Embedding parameters of the current input Sparse feature are in the Cache, so that each Embedding row index of the Sparse feature is mapped to the index of the Cache row according to the mapping table.

[0043] Preferably, the key to the Cache mechanism is to reduce the exchange in and out of the Embedding parameters between the CPU and the GPU, in order to further improve the throughput of model training, the basic implementation of row exchange in and out needs to be optimized, that is, block exchange in and out of the Cache, and the specific steps are as follows:

[0044] At initialization, first scan the dataset once, count the frequency of each Sparse feature index corresponding to the Embedding row; then, sort the Embedding matrix by frequency, and logically divide it into C blocks, thereby changing the mapping table into an Ex3 mapping table of <Embedding index-block index-in-block offset>, and the size of the Cache is set to an NxCxd matrix;

[0045] When the Cache is swapped in and out, when a batch of samples is read in, the Sparse feature index is mapped to the block index according to the mapping table, and the remaining operations are consistent with the row-by-row swapping process. Each time, read and write a Cxd block into the mapping table;

[0046] When the index is mapped, according to each Embedding index, the in-block offset is found according to the mapping table, and then the block position in the Cache is added to the offset, which is the corresponding Cache row index in the Cache.

[0047] Preferably, in order to fully utilize the computing resources and further expand the size of the model in the Cache mechanism, the model needs to be distributed and expanded, and the specific steps are as follows:

[0048] Based on the automatic parallel technology provided by the open source distributed training framework Colossal-AI, the AllReduce method is used to perform model parallel on the Embedding parameters and data parallel on the Dense part;

[0049] Model parallel is to divide the entire Embedding parameter matrix into multiple parts according to rows and columns, distribute different parts to different nodes in a network, read the required data from each node during training, and collect the complete Embedding parameters of a batch of Sparse features through the AllReduce communication function, thereby further expanding the model parameter size;

[0050] Data parallelism is to copy the Dense part to each node and use AllReduce to synchronize the gradient during training, thereby increasing the throughput.

[0051] Preferably, it also includes a Cache mechanism, and the specific steps are as follows: place the Embedding parameters in the CPU, place the Dense parameters in the GPU, and during training, perform Embedding-related lookup and calculation in the CPU, copy the obtained Embedding parameter results to the GPU, and then perform Dense part calculation.

[0052] Compared with the prior art, the application has the beneficial effects that the application can fully utilize GPU computing resources, the calculation speed is increased, the row-based swap-in and swap-out is optimized to block-based swap-in and swap-out, the throughput is improved, and finally the scale of the model is further expanded by fully utilizing the computing resources to distribute the expansion of the model, so that the method has creativity and market promotion value. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the following embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0054] Figure 1 Flowchart of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0056] In the description of the present application, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0057] Embodiment one

[0058] Please refer to Figure 1 The present application provides a technical solution: a large-scale recommendation system training method based on distributed heterogeneous computing, based on the long-tail characteristics of recommendation system data, using Cache mechanism, comprising the following steps:

[0059] S1: Cache initialization;

[0060] S2: Cache swap-in and swap-out algorithm construction;

[0061] S3: Index mapping algorithm construction;

[0062] S4: In the GPU, the Sparse feature is calculated by using the Embedding or Embedding_bag function provided by PyTorch, and the subsequent Dense part of the model is calculated.

[0063] In step S1, the number of Embeddings is E, the dimension of each Embedding vector is d, the Embedding parameters are taken as an Exd matrix, and placed on the host (CPU) side.

[0064] In the GPU, an Nxd Cache is created, i.e., the number of Cache rows is N and the dimension of each Cache row is d. The Embedding parameters can be randomly initialized or read from pre-trained parameters. The Cache only needs to be allocated memory and does not need to be initialized.

[0065] A mapping table with a size of N is created to store the mapping of Cache index to Embedding index, and all are initialized to -1, indicating empty.

[0066] A frequency recording table with a size of N is created to store the number of times each Cache is accessed, initialized to -1, also indicating empty.

[0067] In step S2, during training, a batch of training samples is received at each step. First, ensure that all indexes in the Sparse feature of the sample are de-duplicated and the number does not exceed N. If it exceeds N, an error is reported.

[0068] For the de-duplicated Sparse feature index, check whether the current input index is saved in the Cache according to the mapping table. According to the result, it is divided into two sets: Cache hit and miss.

[0069] For the miss set, first sort the Cache index according to the frequency recording table to find the idle (-1) Cache row or the Cache row with the least access frequency. Filter out the Cache rows in the hit, until the number of Cache rows is equal to the size of miss. The index of this set of Cache rows is recorded as a list pos, indicating the target position of the Embedding parameters corresponding to miss that will be read into the Cache from the CPU. If it is a non-idle Cache row, the index also needs to be copied to the list pos_write_back, i.e., the Cache row needs to be written back to the GPU first.

[0070] According to the pos_write_back list, first read out the corresponding Embedding parameters from the Cache, and then read out the Embedding row index of each Embedding on the CPU from the mapping table, so that the parameters can be written back to the Embedding parameter matrix on the CPU according to the index; then, according to the miss set, read out the Embedding parameters that need to be swapped into the GPU Cache from the Embedding parameter matrix, and then find the Cache row where each parameter needs to be placed according to the pos list; finally, update the mapping table according to the pos list, and clear the record corresponding to the pos list in the frequency record table, that is, set it to -1.

[0071] After the Cache swap-in and swap-out in step S3 is completed, it can be ensured that all the required Embedding parameters of the current input Sparse feature are in the Cache, so that each Embedding row index of the Sparse feature is mapped to the index of the Cache row according to the mapping table.

[0072] Throughput optimization design:

[0073] The key to the Cache mechanism is to reduce the swap-in and swap-out of Embedding parameters between the CPU and the GPU. In order to further improve the throughput of model training, optimization is needed on the basis of the row-based swap-in and swap-out of the basic implementation, that is, block-based cache swap-in and swap-out. The specific steps are as follows:

[0074] During initialization, first scan the data set once to count the frequency of each Sparse feature index corresponding to the Embedding row; then, sort the Embedding matrix by frequency and logically divide it into C blocks, thereby changing the mapping table into an Ex3 <Embedding index-block index-in-block offset> mapping table, and setting the size of the Cache to an NxCxd matrix;

[0075] During Cache swap-in and swap-out, when a batch of samples is read in, the index of the Sparse feature is mapped to the block index according to the mapping table, and the remaining operations are consistent with the row-based swap-in and swap-out process. Each time, a Cxd block is read and written into the mapping table;

[0076] During index mapping, according to each Embedding index, the in-block offset is found according to the mapping table, and then the offset is added to the position of the block in the Cache, which is the corresponding Cache row index in the Cache.

[0077] Distributed computing design:

[0078] The above is a description of a serial implementation. In order to further expand the scale of the model and make full use of computing resources in the Cache mechanism, the model needs to be distributed and expanded, and the specific steps are as follows:

[0079] Based on the automatic parallel technology provided by the open source distributed training framework Colossal-AI, the Embedding parameters are model-parallel and the Dense part is data-parallel in the AllReduce mode.

[0080] Model parallelism is to divide the entire Embedding parameter matrix into multiple parts according to rows and columns, distribute different parts to different nodes in a network, read the required data from each node during training, collect the complete Embedding parameters of a batch of Sparse features through the AllReduce communication function, and further expand the scale of the model parameters.

[0081] Data parallelism is to copy the Dense part to each node and synchronize the gradient during training using AllReduce to increase throughput.

[0082] Embodiment Two

[0083] A large-scale recommendation system training method based on distributed heterogeneous computing also includes not based on the Cache mechanism, and the specific steps are as follows: the Embedding parameters are placed in the CPU, the Dense parameters are placed in the GPU, during training, the Embedding-related lookup and calculation are performed in the CPU, the obtained Embedding parameter results are copied to the GPU, and then the Dense part is calculated.

[0084] In the description of the specification, the description of the terms "one embodiment", "example", "specific example" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0085] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details and limit the application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the specification. The specification selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and utilize the application. The application is limited only by the claims and their entire scope and equivalents.

Claims

1. A large-scale recommendation system training method based on distributed heterogeneous computing, based on the long-tail characteristics of the recommendation system data, using the Cache mechanism, characterized in that, Comprising the following steps: S1: Cache initialization; S2: Cache swap in and out algorithm construction; During training, each step receives a batch of training samples. First, detect and ensure that all indexes within the Sparse feature of the sample are de-duplicated and the number does not exceed N. If it exceeds N, an error is reported. For the de-duplicated Sparse feature index, check whether the current input index is saved in the Cache according to the mapping table. According to the result, it is divided into two sets of Cache hit and miss; For the miss set, first sort the Cache indexes according to the frequency record table, find the Cache row of the free-1, or the Cache row with the least access times, and filter out the Cache rows in the hit, until the number of Cache rows is the size of the miss. The index of this group of Cache rows is recorded as a list pos, indicating the target position of the Embedding parameter to be read from the CPU and put into the Cache for the miss; If it is a non-free Cache row, it also needs to copy its index to the list pos_write_back, that is, it needs to write back the Cache row of the GPU from the Cache first; According to the pos_write_back list, first read out the corresponding Embedding parameter from the Cache, and then read out the Embedding row index of each Embedding on the CPU from the mapping table. Thus, the parameter can be written back to the Embedding parameter matrix on the CPU according to the index; Subsequently, according to the miss set, read out the Embedding parameter that needs to be swapped into the GPU Cache from the Embedding parameter matrix, and then find the Cache row that each parameter needs to be put into according to the pos list; Finally, update the mapping table according to the pos list, and clear the records corresponding to the pos list in the frequency record table, that is, set them to-1. S3: Index mapping algorithm construction; After the Cache swap in and out is completed, it can be ensured that all required Embedding parameters of the current input Sparse feature are in the Cache. Therefore, according to the mapping table, each Embedding row index of the Sparse feature is mapped to the index of the Cache row; S4: In the GPU, use the Embedding or Embedding_bag function provided by PyTorch to calculate the Sparse feature, and calculate the subsequent Dense part of the model. 2.The method of claim 1, wherein: The step S1 is specifically: setting the number of Embeddings as E, the dimension of each Embedding vector as d, and the Embedding parameter as an Exd matrix on the host. In the GPU, create a Cache with Nxd, that is, the number of Cache lines is N, and the dimension of each Cache line is d; the Embedding parameters can be randomly initialized or read from the pre-trained parameters, and the Cache only needs to allocate memory without initialization; Create a mapping table with size N to store the mapping from Cache index to Embedding index, and initialize all to -1, indicating empty; Create a frequency record table with size N to store the number of times each Cache is accessed, initialized to -1, also indicating empty. 3.The method of claim 1, wherein: The key to the Cache mechanism is to reduce the exchange of Embedding parameters between CPU and GPU. To further improve the throughput of model training, the basic implementation of row-by-row exchange needs to be optimized, that is, block-by-block cache exchange, the specific steps are as follows: In the initialization, first scan the dataset once to count the frequency of each Sparse feature index corresponding to the Embedding row; then, sort the Embedding matrix by frequency and logically divide it into C blocks, thereby changing the mapping table into an Ex3 <Embedding index-block index-block offset> mapping table, and the size of the Cache is set to NxCxd matrix; When Cache is exchanged, when reading a batch of samples, the index of Sparse feature is mapped to block index according to the mapping table, and the remaining operations are consistent with the row-by-row exchange process, and each time Cxd block is read and written to the mapping table; When indexing, according to each Embedding index, the block offset is found according to the mapping table, and then the block position in the Cache is added to the offset, which is the corresponding Cache line index in the Cache. 4.The method of claim 1, wherein: In this Cache mechanism, to fully utilize computing resources and further expand the size of the model, the model needs to be distributed and expanded, the specific steps are as follows: Relying on the automatic parallel technology provided by the open-source distributed training framework Colossal-AI, using AllReduce method, the Embedding parameters are model-parallel, and the Dense part is data-parallel; Model parallel is to divide the entire Embedding parameter matrix into multiple parts by row and column, distribute different parts to different nodes in a network, read the required data from each node during training, and collect the complete Embedding parameters of a batch of Sparse features through the AllReduce communication function, thereby further expanding the size of the model parameters; Data parallel is to copy the Dense part to each node, and use AllReduce to synchronize the gradient during training, thereby increasing the throughput. 5.The method of claim 1, wherein: Also included is not based on Cache mechanism, the specific steps are: Embedding parameters are placed in the CPU, and the Dense parameters are placed in the GPU; during training, the Embedding-related lookup and calculation are performed in the CPU, the obtained Embedding parameter results are copied to the GPU, and then the Dense part calculation is performed.

Images