A recommendation system acceleration method based on operator fusion and multi-flow concurrency

Abstract

The application discloses a kind of based on operator fusion and the acceleration method of recommendation system of multi-flow concurrency, including the following contents: optimization is carried out to the Embedding module commonly in recommendation model, through multi-table parallel search algorithm, and all original Embedding table operation is implemented as a lightweight fusion operator;For Neural Network module, through non-GEMM operator fusion algorithm, reduce operator startup overhead, and carry out fine-grained control to the parallel computation of parallel module at operator level;For the structural characteristics of "double-tower type" recommendation model, an independent computing flow is created for the parallel structure of "double-tower type" recommendation model respectively, and each computing flow is responsible for the computing task of a branch in parallel structure. It respectively carries out careful optimization to the Embedding part and Neural Network part of recommendation model, reduces operator startup overhead, and carries out fine-grained control to the parallel computation of parallel module at operator level.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a method for accelerating recommendation systems based on operator fusion and multi-stream concurrency. Background Technology

[0002] With the rapid development of the internet and the continuous increase in user scale, recommender systems play a crucial role in providing personalized user experiences. The rapid development of deep learning technology has brought new opportunities and challenges to recommender systems. Deep learning models can learn more accurate representations of user interests from massive amounts of data, thereby improving the performance of recommender systems. However, with the continuous development of models and the increasing number of parameters, the training efficiency and inference speed of recommender networks have become pressing issues. Especially in real-time recommender scenarios, low latency and high throughput are crucial requirements.

[0003] In recommendation models, matrix computations account for a relatively small proportion, while vector computations constitute a larger share. This makes it difficult to accelerate the computation of recommendation systems using traditional AI chips. Traditional AI chips are typically optimized for matrix computations, while vector computations are relatively inefficient.

[0004] With the proliferation of internet applications, recommender systems play a crucial role. Click-through rate (CTR) prediction is critical in recommender systems and advertising. Accurate CTR prediction can directly improve user experience and increase advertising revenue. With advancements in deep neural networks, CTR models have evolved from traditional logistic regression and factorization machine models to more complex deep architectures, now typically conforming to the paradigms of embedding and neural networks. Furthermore, in typical industrial datasets, the number of click samples has grown to hundreds of billions and continues to increase daily. Therefore, GPU-equipped data centers need to rapidly execute CTR model inference to recommend relevant ads to numerous users. Unlike models used for other tasks (e.g., image classification, object detection), CTR models have stringent accuracy requirements; even a 0.1% error is significant. Therefore, accelerating CTR model inference on GPUs while maintaining model accuracy is challenging. Summary of the Invention

[0005] Current recommendation models involve a large amount of vector computation rather than AI chip-friendly matrix computation, making it difficult to leverage the powerful parallel computing capabilities of AI chips. Existing deep learning frameworks like PyTorch have small operator granularity, resulting in significant runtime overhead. When using mainstream frameworks like PyTorch for model inference, they default to single-stream computation, meaning operators are executed serially in the scheduled order. Achieving parallel computation between operators on a single GPU presents both an opportunity and a challenge. Therefore, this invention proposes a recommendation system acceleration method based on operator fusion and multi-stream concurrency, including a multi-table parallel search algorithm for embeddings, a fusion mechanism for non-GEMM operators in recommendation models, and a multi-stream parallel acceleration technique for single GPUs. These three techniques construct a recommendation system acceleration method based on operator fusion and multi-stream concurrency, achieving AI affinity-based acceleration optimization for recommendation systems.

[0006] This invention achieves this objective through the following technical solution:

[0007] An acceleration method for recommender systems based on operator fusion and multi-stream concurrency includes:

[0008] The Embedding module, which is common in recommendation models, is optimized by using a multi-table parallel lookup algorithm and transforming all the original Embedding table lookup operations into a lightweight fusion operator.

[0009] For the Neural Network module, a non-GEMM operator fusion algorithm is used to reduce operator startup overhead and to perform fine-grained control over the parallel computation of the parallel module at the operator level.

[0010] To address the structural characteristics of the "dual-tower" recommendation model, an independent computational flow is created for each parallel structure of the "dual-tower" recommendation model, and a width-first computational flow scheduling algorithm is adopted. Each computational flow is responsible for the computational task of one branch in the parallel structure. Specifically, operators from the parallel modules of the same model are scheduled to different computational flows, and operators from different computational flows are started alternately to ensure that operators are executed by the GPU as early as possible and to prevent blocking or waiting.

[0011] According to the aforementioned method for accelerating a recommender system based on operator fusion and multi-stream concurrency, the embedding-oriented merge-pull algorithm includes the following:

[0012] First, construct a blank Embedding result table of the target size to be filled;

[0013] Then, using the Embedding index algorithm, each element in the empty table is precisely filled;

[0014] Finally, leveraging the parallel computing capabilities of AI hardware, the table lookup process is handled concurrently using multi-threading to complete the merging and retrieval of embeddings.

[0015] According to the aforementioned method for accelerating a recommender system based on operator fusion and multi-stream concurrency, the Embedding indexing algorithm includes the following steps:

[0016] The first step is to calculate the size of the output matrix and the number of its columns based on the input size and the Embedding configuration parameters, and then begin traversing the output matrix.

[0017] The second step is to iterate through the matrix row by row. Based on this step, the position of the current element is derived according to the row and column of the current element in the matrix.

[0018] The third step is to calculate and determine the Embedding table to be searched.

[0019] The fourth step is to use the collected information and input ID to determine the specific rows and columns in the Embedding table, and obtain the values ​​to be filled into the output matrix elements of the current traversal.

[0020] According to the aforementioned method for accelerating a recommender system based on operator fusion and multi-stream concurrency, the workload allocation of the recommender system adopts an output-first approach, enabling different threads to serve the same samples and performing embedding table lookups feature by feature. Input reading, embedding table lookups, and output writing all maintain sequential addressing mode.

[0021] According to the aforementioned method for accelerating a recommender system based on operator fusion and multi-stream concurrency, the fusion algorithm of the non-GEMM operators includes the following:

[0022] First, the forward inference process of the model is described as a directed acyclic graph, where nodes represent operators and edges represent tensors;

[0023] Traverse the directed acyclic graph starting from the root node, and mark all non-GEMM nodes that can be connected by edges;

[0024] Define a subgraph as the set of labeled nodes and the edges connecting them, and then merge all operators contained in the same subgraph into a new operator;

[0025] Finally, the custom fusion operator is integrated into the deep learning framework to complete the equivalent replacement of the operator, thereby accelerating computation and reducing memory usage.

[0026] According to the aforementioned method for accelerating a recommendation system based on operator fusion and multi-stream concurrency, the breadth-first computational flow scheduling algorithm includes the following steps:

[0027] The first step is to obtain the operator lists for the feature interaction module and the deep network module, respectively;

[0028] The second step is to calculate the number of operators in the feature interaction module and the deep network module, respectively.

[0029] The third step involves creating CUDA computational flows for the feature interaction module and the deep network module, respectively.

[0030] The fourth step is to add the operators of the feature interaction module and the deep network module to their respective CUDA computation streams;

[0031] The fifth step involves adding operators to the startup queue in alternating order of modules, starting with the modules that have more operators, until all operators have been added.

[0032] Compared to existing technologies, this invention proposes a method for accelerating recommender systems based on operator fusion and multi-stream concurrency. It meticulously optimizes both the embedding and Neural Network parts of the recommender model. The core idea is to design a multi-table parallel lookup algorithm for the embedding and implement it as a lightweight fusion operator. A fusion mechanism is designed for the non-GEMM operators in the Neural Network to reduce operator startup overhead, and fine-grained control over the parallel computation of parallel modules is implemented at the operator level. The beneficial effects of this invention are as follows:

[0033] 1) This invention proposes an efficient parallel lookup algorithm for multiple Embedding tables by predicting the entire workload in advance. Based on this, the Embedding table index can be accurately performed, greatly improving inference performance. Compared with PyTorch, the Embedding latency can be reduced by an average of 12.9 times, and by a maximum of 23.0 times.

[0034] 2) This invention proposes a fusion mechanism that reduces the number of non-GEMM operators by fusing them, thereby saving the overhead of starting them. The number of these operators is reduced by an average of 93.9%, with a maximum reduction of 97.6%.

[0035] 3) This invention designs an operator-level computation flow scheduling algorithm, realizing fine-grained management of parallel computation within a single GPU. This design can further improve performance by 10.4%. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the "dual-tower" recommended model according to an embodiment of the present invention;

[0037] Figure 2This is a schematic representation of the Embedding lookup in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram illustrating workload allocation based on input priority and output priority in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of operator fusion in an embodiment of the present invention. Detailed Implementation

[0040] Exemplary embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] This invention discloses a method for accelerating recommender systems based on operator fusion and multi-stream concurrency, comprising:

[0042] (1) Merging and fetching algorithm for Embedding

[0043] by Figure 1 Taking the DCN model as an example, this invention first optimizes the Embedding module common to the recommendation model, merging all the original Embedding table lookup operations into a single kernel implementation.

[0044] Suppose the input X of the recommendation model has a size of b×k, where b represents the number of samples and k represents the number of feature domains. Each feature domain has a separate embedding vector table. Where n i The denoted represents the number of features contained in the i-th feature domain, and d represents the dimension of the embedding table, i.e., the length of the vector used to represent each feature. For example, the "gender" feature domain contains male and female features, so n is 2 for this feature domain. Some feature domains have a huge number of features, reaching millions or even tens of millions, so their corresponding embedding vector tables are also very large; for the input sample, each of the k feature domains is first encoded using a one-hot encoding. To reduce sparsity and memory consumption, they are usually further represented as numeric IDs. Figure 2This demonstrates an example of the embedding operation: performing a lookup operation on each of the k feature domains. After a total of k lookup operations, their vector representations are obtained. These vectors are then concatenated into a matrix. As input to the next layer of the model.

[0045] Treating all Embedding table lookups as a single operation, the original k lookups can be replaced with a single operation. The size of the Embedding output can be calculated based on the Embedding configuration parameters (i.e., the values ​​of k and d) and the input size. Therefore, only the output matrix needs to be initialized and filled element by element, without any additional intermediate storage. The key challenge is accurately matching each position in the matrix with its corresponding position in the Embedding table. To address this, this invention designs a parallel multi-table lookup algorithm to perform the global Embedding table lookup.

[0046] The multi-table parallel lookup algorithm for Embedding is as follows:

[0047]

[0048] First, the size of the output matrix and the number of its columns are calculated based on the input size and the embedding configuration parameters (lines 1-2). Then, the output matrix is ​​traversed. Next, the matrix is ​​iterated row by row (line 3). Based on this step, the position of the current element is derived according to its row and column in the matrix (lines 4-5). Furthermore, the embedding table to be searched is determined through calculation (line 6). Finally, using the collected information and the input ID, the specific row and column in the embedding table are determined (lines 7-8), which are the values ​​to be filled into the currently traversed elements of the output matrix (line 9).

[0049] After determining the overall embedding table lookup task, the focus shifts to how to distribute the workload across threads to ensure GPU friendliness and thus improve performance. Distribution principles can be categorized into two types: input-first and output-first, such as... Figure 3 As shown. The input-first approach aims to maximize the distribution of different input samples among different threads, while the output-first approach aims to coordinate multiple threads to process the same sample.

[0050] The smallest hardware execution unit on a GPU is a warp, which typically consists of 32 threads. When these 32 threads execute the same instruction, they operate in Single Instruction Multithreaded (SIMT) mode, allowing simultaneous read and write operations. If an input-first approach is used, where different threads prioritize serving different samples, even if the input read phase conforms to sequential addressing, the embedding table lookup and output write phases are random read / write modes, not sequential addressing. This is unfriendly to GPU hardware because it fails to fully utilize memory bandwidth, leading to resource waste and poor performance. Therefore, this invention employs an output-first approach: different threads serve the same samples and perform embedding table lookups feature-by-feature. Thus, input reads, embedding table lookups, and output writes all maintain sequential addressing. This ensures optimal utilization of hardware memory bandwidth and results in performance improvements.

[0051] (2) Fusion mechanism for non-GEMM operators in recommendation models

[0052] This invention designs a fusion algorithm for non-GEMM (non-generic matrix multiplication) operators in recommendation models, which fuses fine-grained operators to reduce the operator startup overhead during runtime.

[0053] In the Neural Network of a deep recommendation model, there are two main components: the feature interaction module and the deep network module. The feature interaction module is used to learn explicit features, while the deep network module is responsible for capturing complex and implicit features. Across different deep recommendation models, the deep network module remains relatively consistent, typically consisting of several multilayer perceptrons (MLPs). Each layer of the deep network module can be represented by the following formula:

[0054] h l+1 =σ(W l h l +b l )

[0055] Where l represents the depth of the layer, h l and h l+1 These are the input and output of the l-th layer, W. l and b l These are the weight matrix and bias vector of the l-th layer, respectively, and σ is the ReLU activation function. This part mainly involves the GEMM (General Matrix Multiplication) operator and element-wise activation operators.

[0056] Furthermore, due to differing design goals, the feature interaction modules of "dual-tower" recommendation models of varying depths often differ. For example, the feature interaction module of the DCNv2 model is a cross-network composed of several cross layers. Each cross layer can be represented by the following formula:

[0057] x l+1 =x0⊙(W l x l +b l )+x l

[0058] Where l represents the depth of the layer, x l and x l+1 These are the input and output of the l-th cross layer, respectively, W l and b l These are the weight matrix and bias vector of layer l, respectively. x0 is the original input of the entire cross-module, and ⊙ represents element-wise multiplication. This formula shows that GEMM operators account for only 25% of the total number of operators in the cross-layer, while non-GEMM operators account for 75%. As the number of cross-layers increases, more non-GEMM operators introduce additional startup overhead and data read / write operations. This phenomenon also exists in the feature interaction modules of other deep "dual-tower" recommendation models.

[0059] Table 1. Number of operators before and after fusion in different feature interaction modules

[0060]

[0061] To address this issue, this invention proposes a fusion mechanism: merging all adjacent non-GEMM operators into a single operator to reduce the time consumption associated with operator startup. First, a directed acyclic graph (DAG) is constructed to represent the forward propagation (inference) process of the model, where nodes represent operators and edges represent tensors. Starting from the root node, this graph is traversed, marking all non-GEMM nodes that can be connected by edges. The set of these nodes and the edges connecting them is defined as a subgraph. Then, all operators contained within the same subgraph are merged into a new operator. This invention validates the proposed operator fusion mechanism using four well-known industry models: DCN, DCNv2, Wide&Deep, and DeepFM. Schematic diagrams before and after operator fusion are shown below. Figure 4 As shown in Table 1, the number of operators before and after fusion is compared. This fusion mechanism can reduce the number of these operators by an average of 93.9% and a maximum reduction of 97.6%. Furthermore, the workload allocation follows an output-first principle, traversing the output matrix element by element and assigning the filling task of each matrix element to each thread, ensuring that data reading and writing are in sequential addressing mode.

[0062] The inner product fusion algorithm in DeepFM is shown as an example. The specific algorithm is as follows:

[0063]

[0064] (3) Multi-computational stream parallel acceleration technology for single GPU

[0065] This invention addresses the structural characteristics of "dual-tower" recommendation models by proposing a multi-computational stream parallel acceleration technique for single-GPU systems. The computational streams are independent of each other, with no data dependencies, maintaining the computational characteristics of the original model and achieving computational acceleration while preserving accuracy.

[0066] When deploying models on GPUs, there is an opportunity for operators scheduled to different computation streams (CUDA streams) to execute in parallel. However, existing deep learning frameworks typically use only a single default computation stream to schedule operators, so operators are executed serially according to the scheduling order, even for models with parallel architectures. Although computation streams can be managed at the module level, achieving optimal performance remains a challenge due to their depth-first scheduling pattern. These methods do not fully utilize the computing power of GPUs, resulting in longer inference times and failing to meet the requirements for fast response.

[0067] To address this issue, this invention designs an operator-level computational flow scheduler to manage the parallel computation of the feature interaction module and the deep network module in a "dual-tower" recommendation model with fine-grained control. The key idea behind this design is to use a breadth-first scheduling pattern to schedule the operators of these two modules into separate computational flows. In other words, operators from these two modules are added to two different computational flows. These operators are then added to a startup queue in an alternating order of modules, starting with the module with more operators. This ensures that operators from both modules execute on the GPU as early as possible, avoiding blocking or waiting.

[0068] The breadth-first computation flow scheduling algorithm is as follows:

[0069]

[0070] First, obtain the operator lists for the feature interaction module and the deep network module respectively (lines 1-2), and calculate the number of operators for each module (lines 3-4). Then, create CUDA computation streams for the feature interaction module and the deep network module respectively (lines 5-6), and add the operators for each module to their respective CUDA computation streams (lines 7-8). Finally, add the operators to the startup queue in alternating order, starting with the module with the most operators, until all operators are added (lines 9-20).

Claims

1. A method for accelerating recommender systems based on operator fusion and multi-stream concurrency, characterized in that, The Embedding module, which is common in recommendation models, is optimized by using a multi-table parallel lookup algorithm and transforming all the original Embedding table lookup operations into a lightweight fusion operator. The multi-table parallel lookup algorithm for Embedding includes the following: First, construct a blank Embedding result table of the target size to be filled; Then, using the Embedding index algorithm, each element in the empty table is precisely filled; Finally, leveraging the parallel computing capabilities of AI hardware, the table lookup process is handled concurrently using multi-threading to complete the merging and retrieval of embeddings. The Embedding indexing algorithm includes the following steps: The first step is to calculate the size of the output matrix and the number of its columns based on the input size and the Embedding configuration parameters, and then begin traversing the output matrix. The second step is to iterate through the matrix row by row. Based on this step, the position of the current element is derived according to the row and column of the current element in the matrix. The third step is to calculate and determine the Embedding table to be searched. The fourth step is to use the collected information and input ID to determine the specific rows and columns in the Embedding table, and obtain the values ​​to be filled into the output matrix elements of the current traversal. For the Neural Network module, a non-GEMM operator fusion algorithm is used to reduce operator startup overhead and to perform fine-grained control over the parallel computation of the parallel module at the operator level. The fusion algorithm for the non-GEMM operators includes the following: First, the forward inference process of the recommendation model is described as a directed acyclic graph, where nodes represent operators and edges represent tensors; Traverse the directed acyclic graph starting from the root node, and mark all non-GEMM nodes that can be connected by edges; Define a subgraph as the set of labeled nodes and the edges connecting them, and then merge all operators contained in the same subgraph into a new operator; Finally, the custom fusion operator is integrated into the deep learning framework to complete the equivalent replacement of the operator, thereby accelerating computation and reducing memory usage; In response to the structural characteristics of the "dual-tower" recommendation model, an independent computational flow is created for the parallel structure of the "dual-tower" recommendation model and a width-first computational flow scheduling algorithm is adopted. Each computational flow is responsible for the computational task of one branch in the parallel structure. The breadth-first computational flow scheduling algorithm includes the following steps: The first step is to obtain the operator lists for the feature interaction module and the deep network module, respectively; The second step is to calculate the number of operators in the feature interaction module and the deep network module, respectively. The third step involves creating CUDA computational flows for the feature interaction module and the deep network module, respectively. The fourth step is to add the operators of the feature interaction module and the deep network module to their respective CUDA computation streams; The fifth step involves adding operators to the startup queue in alternating order of modules, starting with the modules that have more operators, until all operators have been added.

2. The method for accelerating a recommender system based on operator fusion and multi-stream concurrency as described in claim 1, characterized in that, The workload allocation of the recommendation model adopts an output-first approach, so that different threads serve the same samples and perform embedding table lookups for each feature. Input reading, embedding table lookups, and output writing all maintain sequential addressing mode.

Images