A hybrid expert model sparse inference method and system for generative recommendation
By employing a hybrid expert model sparse inference method specifically designed for recommendation scenarios, and combining hierarchical attention and sparse attention mechanisms with a load-balanced loss function, we optimize expert selection and computational resource allocation. This solves the computational efficiency and accuracy issues of generative recommendation models in real-time inference, and achieves a highly efficient recommendation system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANKAI UNIV
- Filing Date
- 2026-03-13
- Publication Date
- 2026-06-23
AI Technical Summary
Existing generative recommendation models struggle to balance computational efficiency and inference performance, especially in real-time inference with hybrid expert models. They cannot fully utilize the parallel computing capabilities of hardware, resulting in high inference latency and limited model accuracy.
We design a hybrid expert model sparse inference method specifically for recommendation scenarios. It adopts a hierarchical attention mechanism and a sparse attention mechanism, and optimizes expert selection and computational resource allocation by decoupling the gating network and expert network through a load balancing loss function. We also achieve hardware parallel efficiency through a low-level fusion operator.
With low time overhead and low computational complexity, it achieves recommendation accuracy comparable to or even higher than dense models, improves the real-time inference performance and computational efficiency of the model, and solves the problem of balancing accuracy and efficiency in recommendation systems.
Smart Images
Figure CN121835927B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning and recommendation system technology, and in particular to a hybrid expert model sparse inference method and system for generative recommendation. Background Technology
[0002] With the continuous expansion of internet information scale, recommender systems have become a core technology connecting users with information, goods, and services. In recent years, generative recommender models based on sequence modeling have received widespread attention due to their ability to effectively capture users' dynamic interests. However, in pursuing higher accuracy and larger model capacity, these models generally face severe challenges in computational efficiency and inference performance. First, the attention mechanism used in the core of generative recommender models has a computational complexity that increases quadratically with the sequence length, while a user's next action is often only related to a few key items in the historical sequence, leading to a large amount of redundant computation and limiting the model's ability to process long sequence data and its real-time response speed. Second, the hybrid expert model structure introduced to improve the model's expressive power is mainly designed for natural language processing tasks. Its routing mechanism and expert structure are difficult to optimally adapt to the highly sparse and dynamic characteristics of recommender system data, limiting the upper limit of the model's accuracy.
[0003] The aforementioned challenges are particularly pronounced during the deployment and inference phases of generative recommendation models. Hybrid expert models require dynamic activation of certain expert networks based on input during real-time inference. In this scenario, existing mainstream deep learning frameworks typically use a serial approach at the Python level to sequentially call the activated experts for computation. This execution method is severely incompatible with the highly parallel design characteristics of hardware such as GPUs, failing to fully utilize their massively parallel computing capabilities. This results in persistently high end-to-end inference latency when providing real-time recommendation services to users, becoming a key bottleneck restricting its large-scale online application.
[0004] Therefore, in the existing technology, at the algorithmic structure level of generative recommendation models, there is a lack of hybrid expert and attention mechanism designs that can simultaneously take into account the characteristics of recommendation tasks, high accuracy and low complexity; at the underlying hardware execution level, there is a lack of efficient hybrid expert model inference operators that can fully utilize hardware parallel performance and reduce inference latency.
[0005] In summary, existing optimization methods for hybrid expert models for generative recommendations struggle to balance model accuracy, computational efficiency, and inference speed. On one hand, while dense model structures or complex attention mechanisms can achieve high recommendation accuracy, they introduce significant computational redundancy, leading to high training and inference latency, which is insufficient for real-time service requirements. On the other hand, while general simplification strategies, such as directly reducing the number of experts or attention heads, can improve computational efficiency, they severely impair the model's ability to capture the dynamic expression of complex user interests, resulting in a significant decrease in recommendation accuracy. Summary of the Invention
[0006] This invention aims to solve the aforementioned problems. To this end, this invention provides a hybrid expert model sparse inference method and system for generative recommendation. By designing a hybrid expert model structure specifically for recommendation scenarios, a sparse attention mechanism based on expert selection, and an efficient underlying fusion operator, it achieves recommendation accuracy comparable to or even higher than advanced dense models with low time overhead and low computational complexity. This provides a high-performance and high-efficiency solution for generative recommendation systems, systematically addressing the balance between accuracy and efficiency.
[0007] This invention provides a sparse inference method using a hybrid expert model for generative recommendation, employing the following technical solution: (Including...)
[0008] S1: Obtain the original input data and, through the embedding layer, obtain the input vector;
[0009] S2: The input vector is normalized and the maximum value is selected to obtain the hybrid expert weights; based on the hierarchical attention mechanism, the attention enhancement features are calculated according to the hybrid expert weights.
[0010] The calculation process for hybrid expert weights is as follows:
[0011] The input vector is passed through a gating mechanism to obtain the total weight of the shared header and the total weight of the routing header;
[0012] The total weight of the shared header is normalized to obtain a fine-grained weight; the total weight of the routing header is selected by selecting the maximum value to obtain a sparse weight.
[0013] By fusing fine-grained weights with sparse weights, a hybrid expert weight is obtained.
[0014] S3: Input the input vector and attention-enhanced features into the hybrid expert model to calculate the recommendation result;
[0015] Among them, the parallel gating mechanism based on Sigmoid activation calculates expert weights according to attention enhancement features; the activated expert layer is selected according to the expert weights, and the recommendation result is calculated using the input vector.
[0016] Furthermore, in the calculation of the hybrid expert weights in step S2, the gating mechanism is implemented through the first linear layer combined with the Softmax function; the maximum value selection operation is implemented through the second linear layer combined with Gumbel noise and Top-k operation.
[0017] Furthermore, in step S2, when calculating the attention enhancement features, the matrix projection calculation of query, key, and value is performed only for the selected attention head; during the attention score calculation process, a gating weight-aware adjustment mechanism is introduced, and the hybrid expert weights are incorporated into the attention score calculation as weighting factors; the outputs of each activated attention head are weighted and fused according to their gating weights, and gating attention output features are formed through residual connections and layer normalization operations; the gating attention output features are added to the input vector to obtain the attention enhancement features.
[0018] Furthermore, in step S2, the formula for calculating the gated attention output features is:
[0019]
[0020]
[0021]
[0022]
[0023]
[0024]
[0025] in, The weights are the values corresponding to the input vector. The weights corresponding to the projection information are... To query the weights corresponding to the projection information, The weights corresponding to the key projection information. For attention weights, For the partitioning operation, For intermediate attention variables, For activation function, For Einstein's summation agreement operation, For mixed expert weights, Scaling factor The intermediate attention variable after masking. For the mask, For time information and The combined variables, This is a bias model based on bucketing time intervals and relative positions. For time information, For location information and The combined variables, For location information, for and The combined variables, This is for intermediate splicing. For splicing operations, For gating attention output features, This is for root mean square normalization.
[0026] Furthermore, in step S3, the attention enhancement features are passed through a gating layer and a scoring layer to obtain expert scores, and the expert scores are passed through Sigmoid activation to obtain expert weights; Sigmoid activation realizes parallel and independent expert evaluation based on absolute confidence, so that the activation state calculation of each expert layer is independent of each other.
[0027] Furthermore, in step S3, the activated expert layer is selected according to the expert weights, and the output weights of the gating network are calculated; the activated expert layer calculates the expert prediction result based on the input vector; and the recommendation result is calculated based on the input vector, the output weights of the gating network, and the expert prediction result.
[0028] Furthermore, the total loss function during hybrid expert model training includes load balancing loss;
[0029] The load balancing loss is used to control the concentration of activation distribution within experts and the balance of load distribution among experts. Its calculation formula is as follows:
[0030]
[0031] in, For load balancing losses, For expert internal smoothness adjustment factor, The total number of experts, For expert indexing, For batch size, For sample index, Assign probabilities to the sample-expert pair. This serves as a factor for adjusting the balance among experts. This represents the average activation probability of experts.
[0032] Furthermore, in step S2, the input vector is first subjected to layer normalization processing, and then used to calculate the hybrid expert weights;
[0033] In step S3, the input vector is first subjected to RMSNorm normalization and then input into the activated expert layer.
[0034] Furthermore, in step S3,
[0035] Before the expert layer calculation, the tokens in the input vector are grouped and rearranged continuously according to the activated expert layers. After rearrangement, the token layout in memory becomes continuous expert groups: a two-dimensional GPU kernel grid structure is constructed. The first dimension of the grid structure corresponds to the total number of activated expert layers, and the second dimension is the maximum number of tokens that each activated expert layer needs to process in this time divided by the number of threads contained in each GPU kernel thread block, and rounded up.
[0036] During expert layer computation, the GPU kernel continuously reads tokens from memory according to the grid structure and performs computations. Before each thread starts computation, it first queries the number of tokens actually allocated to its corresponding expert layer. If the token index exceeds the number of tokens, the thread immediately exits the computation process.
[0037] This invention also provides a hybrid expert model sparse inference system for generative recommendation, employing the following technical solution: it includes a data acquisition module, a dynamic head selection module, and a hybrid expert model.
[0038] The data acquisition module is used to acquire raw input data and obtain an input vector through the embedding layer;
[0039] The dynamic head selection module is used to obtain hybrid expert weights by normalizing the input vector and selecting the maximum value; and to calculate attention enhancement features based on the hierarchical attention mechanism and the hybrid expert weights.
[0040] The calculation process for hybrid expert weights is as follows:
[0041] The input vector is passed through a gating mechanism to obtain the total weight of the shared header and the total weight of the routing header;
[0042] The total weight of the shared header is normalized to obtain a fine-grained weight; the total weight of the routing header is selected by selecting the maximum value to obtain a sparse weight.
[0043] By fusing fine-grained weights with sparse weights, a hybrid expert weight is obtained.
[0044] The hybrid expert model is used to calculate the recommendation result based on the input vector and attention enhancement features. Specifically, a parallel gating mechanism based on Sigmoid activation is used to calculate the expert weights based on the attention enhancement features. The activated expert layer is selected based on the expert weights, and the recommendation result is calculated using the input vector.
[0045] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:
[0046] 1. This invention designs a hybrid expert model based on hierarchical feature decoupling specifically for recommendation scenarios. This model decouples the inputs of the gating network and the expert network, enabling routing decisions to be based on richer contextual information. It employs a Sigmoid gating and linear normalization strategy to transform expert selection from competitive allocation to parallel confidence evaluation, while maintaining the original proportional relationship among experts. Simultaneously, an improved load balancing loss function is introduced, using hyperparameters to separately adjust the smoothness of the expert's built-in confidence and the uniformity of the load among experts, effectively preventing expert degradation.
[0047] 2. This invention introduces a hybrid expert approach into the attention layer, proposing a hierarchical gated multi-head attention mechanism based on expert selection. This mechanism calculates weights for each attention head through a lightweight gating network and performs dynamic head selection, activating only the most relevant attention heads for computation. By weighted fusion of the outputs of the selected heads, this mechanism can significantly reduce computational complexity while maintaining model accuracy comparable to dense attention.
[0048] 3. To optimize real-time inference performance, this invention designs and implements a low-level fusion inference operator. This operator first rearranges the input tokens based on expert groups according to the routing results, arranging tokens belonging to the same expert consecutively in memory to optimize memory access efficiency. Subsequently, an adaptive kernel design is employed, with its grid dimension aligned with the number of experts and the maximum token processing capacity. Invalid computation units are skipped through internal conditional checks, eliminating GPU core idle time and achieving extreme hardware parallelism efficiency.
[0049] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0050] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0051] Figure 1 This is a flowchart of the method provided by the present invention.
[0052] Figure 2 This is a model architecture diagram of the hierarchical gated multi-head attention mechanism based on expert selection provided by the present invention.
[0053] Figure 3This is a schematic diagram of the two-level gating architecture of the AttentionHeadMoE layer provided by the present invention.
[0054] Figure 4 This is an architecture diagram of the hybrid expert model based on hierarchical feature decoupling provided by the present invention.
[0055] Figure 5 This is a schematic diagram of the token grouping and rearrangement process provided by the present invention. Detailed Implementation
[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.
[0057] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0058] The following is combined with Figures 1 to 5 The present invention will be further described in detail below, presenting a sparse inference method and system based on a hybrid expert model for generative recommendation:
[0059] In this embodiment, as Figure 1 As shown, a hybrid expert model sparse inference method for generative recommendation is provided, including the following steps:
[0060] S1: Obtain the original input data and, through the embedding layer, obtain the input vector.
[0061] The overall process of this method begins with a practical application scenario of a recommendation system, using a sequence of user historical behavior in text form (such as the IDs of five consecutively clicked movies) and its features (such as movie tags) as the raw input data. The raw input data is represented as an input vector of a uniformly shaped embedding through an embedding layer.
[0062] S2: The input vector is normalized and the maximum value is selected to obtain the hybrid expert weights; based on the hierarchical attention mechanism, the attention enhancement features are calculated according to the hybrid expert weights.
[0063] This embodiment designs a hierarchical gated multi-head attention mechanism based on expert selection, introducing a hybrid expert approach into the computation to achieve dynamic sparsity. For example... Figure 2 As shown, the input vector x serves as the model input. After layer normalization, it enters the AttentionHeadMoE layer. Through normalization and maximum value selection, hybrid expert weights are obtained, which serve as the basis for head selection. Subsequently, it passes through the Moe-Attention layer to calculate the gated attention output features, which are then added to the input vector to obtain the attention enhancement features.
[0064] This embodiment innovatively introduces a hierarchical gating network architecture in the AttentionHeadMoE layer. For example... Figure 3 As shown, this embodiment employs a clear two-level gating architecture to achieve hierarchical allocation of attention heads. At the first level, broad category allocation is performed: the input vector after layer normalization is gating to obtain the total weight of the shared head and the total weight of the routing head. Specifically, the first linear layer and the Softmax function allocate the overall weight ratio of each input feature decision to the shared head and routing head categories. At the second level, two types of fine-grained allocation are performed in parallel: for the shared head path, the total weight of the shared head is processed by the third linear layer to generate a smooth internal expert weight distribution, and then normalized by softmax to obtain fine-grained weights. For the routing head path, the total weight of the routing head is processed by the second linear layer and the Gumbel-Topk layer to obtain sparse weights. The second linear layer combines Gumbel noise and Top-k operations to achieve sparse expert selection. Finally, the two-level allocation results (fine-grained weights and sparse weights) are weighted and fused to form a complete gating matrix, resulting in hybrid expert weights. This matrix dynamically weights the output of the attention layer through a broadcast mechanism, thereby achieving adaptive optimization of computational resources.
[0065] The AttentionHeadMoE layer takes the features of the normalized output of the previous layer as input and generates hierarchical attention head activation weights through multiple linear transformation layers. This allows the model to adaptively allocate attention computational resources for different categories based on the semantic characteristics of the input sequence. This design ensures that the gating network can adaptively evaluate the importance of each attention head based on the semantic features of the input sequence.
[0066] In this embodiment, a complete attention calculation process is implemented through the Moe-Attention layer on the subset of activated attention heads. The matrix projection calculation of query, key, and value is performed only on the selected attention head. During the attention score calculation, a gating weight-aware adjustment mechanism is introduced, incorporating hybrid expert weights as weighting factors into the attention score calculation. The outputs of each activated attention head are weighted and fused according to their gating weights, and gating attention output features are formed through residual connections and layer normalization operations.
[0067] The formula for calculating the gated attention output features is:
[0068]
[0069]
[0070]
[0071]
[0072]
[0073]
[0074] in, The weights are the values corresponding to the input vector. The weights corresponding to the projection information are... To query the weights corresponding to the projection information, The weights corresponding to the key projection information. For attention weights, For the partitioning operation, For intermediate attention variables, For activation function, For Einstein's summation agreement operation, For mixed expert weights, Scaling factor The intermediate attention variable after masking. For the mask, For time information and The combined variables, This is a bias model based on bucketed time intervals and relative positions, used for the fusion learning operation of temporal position information and relative internal information. For time information, For location information and The combined variables, For location information, for and The combined variables, This is for intermediate splicing. For splicing operations, For gating attention output features, This is for root mean square normalization.
[0075] This embodiment utilizes a Top-k selection strategy to achieve dynamic sparsity of attention heads based on the hybrid expert weights output by the gating network. During model training, random noise is added to the gating scores based on the hybrid expert weights to enhance the exploratory nature of route selection. This method controls the flexibility of the selection process through a learnable temperature parameter and uses a zeroing operation to mask unselected attention heads, thereby achieving dynamic allocation of computational resources and avoiding redundant computation. This embodiment sorts the hybrid expert weights and retains the k attention heads with the highest weights, while setting the weights of the remaining heads to zero using a Scatter operation. This dynamic selection mechanism enables the model to autonomously allocate the most relevant attention computational resources for different input sequence features, effectively avoiding computational redundancy in a fixed mode.
[0076] S3: Input the input vector and attention-enhanced features into the hybrid expert model to calculate the recommendation result; wherein, based on the parallel gating mechanism of Sigmoid activation, the expert weights are calculated according to the attention-enhanced features; the activated expert layer is selected according to the expert weights, and the recommendation result is calculated using the input vector.
[0077] This embodiment makes structured improvements to the standard hybrid expert model to suit the data characteristics of recommendation systems, such as... Figure 4 As shown, the improved part of this embodiment is circled in red. Traditional MoE models use a single input for routing decisions and expert computation, resulting in a mismatch in feature representation levels. This embodiment designs a dual-path input architecture based on hierarchical feature decoupling, decoupling routing decisions and expert computation at the input level. Specifically, the gating network receives attention-enhanced features, which retain rich semantic information and sequence dependencies, used for global route selection; while the expert network receives input vectors normalized by RMSnorm, which have better distribution stability and representation quality, used for refined feature transformations by each expert. This hierarchical decoupling design allows routing decisions to be based on more comprehensive contextual information, while ensuring that expert computation obtains high-quality feature input.
[0078] Furthermore, in the gated network, to address the issue of uneven expert utilization caused by strong competition in the traditional Softmax gating function, this embodiment employs a parallel gating mechanism based on Sigmoid activation. This mechanism transforms expert selection from a relative evaluation based on ranking to a parallel independent evaluation based on absolute confidence, ensuring that the activation state calculations of each expert are independent and effectively alleviating the load imbalance caused by expert resource competition. To further improve numerical stability, this embodiment performs linear normalization on the Sigmoid output, preserving the relative relationship of the original confidence to the greatest extent while satisfying probability distribution constraints. This gating mechanism can flexibly adapt to complex scenarios in recommendation systems where users have multiple interests, supporting fine-grained expert scheduling across different interest dimensions, significantly improving the diversity and robustness of model expression.
[0079] The specific working process of the hybrid expert model based on hierarchical feature decoupling in this embodiment includes:
[0080] Attention-enhancing features are passed through gating layers and layer-by-layer to obtain expert scores, and the expert scores are then used to obtain expert weights through Sigmoid activation;
[0081] The expert weights are used as indexes to select the activated expert layers. Each activated expert layer calculates the expert prediction result based on the input vector after RMSNorm normalization.
[0082] The expert weights are used as weights. After summing and taking the reciprocal (1 / sum), they are multiplied by the expert prediction results. Finally, they are added to the input vector after RMSNorm normalization to obtain the recommendation results.
[0083] This embodiment also designs a multi-objective collaborative load balancing optimization strategy. To overcome the expert degradation problem and achieve load balancing, this embodiment proposes a multi-objective collaborative optimization strategy. This strategy introduces two adjustable hyperparameters to control the concentration of activation distribution within experts and the balance of load distribution among experts, respectively. The intra-expert constraint term enhances the specialization of experts by promoting significant responses from each expert to specific sample subsets; the inter-expert constraint term ensures that all experts are adequately trained by suppressing extreme differentiation in load distribution. These two constraint terms (load balancing loss) together with the main loss function constitute a complete optimization objective, achieving reasonable resource allocation while maintaining the model's expressive power.
[0084] The formula for calculating load balancing losses is:
[0085]
[0086] in, This is due to load balancing losses. is the expert intra-smoothness adjustment factor, which is a trainable hyperparameter used to control the weights of the first part (intra-expert uniformity constraint) in the loss function. The total number of experts represents the number of all experts in the model. This serves as an expert index, used to iterate through the variables of each expert. The value range is from 1 to . The batch size represents the number of samples contained in a training batch. This is a sample index used to iterate through the variables of each sample in the batch. The value range is from 1 to . Assign probabilities to the sample-expert pair, representing the first... The sample was assigned to the first The probability of an expert. This is the inter-expert equilibrium adjustment factor, used to control the weight of the second part (inter-expert uniformity constraint) in the loss function. The average activation probability of experts represents the activation probability of all experts in the first stage. The average probability distribution over the samples. Through hyperparameters and Adjust the smoothness of the built-in confidence of experts and the uniformity of the load among experts respectively.
[0087] The main loss function uses a binary cross-entropy function. Then the total loss function... The calculation formula is:
[0088]
[0089] in, The total number of samples represents the total number of samples used in the loss calculation. The sample weights represent the weights of the samples. The importance of each sample in loss calculation. For the model to the first The difference between the predicted value of each sample and the positive class label (1); For the model to the first The difference between the predicted value of each sample and the negative class label (0). The loss weight is used for load balancing.
[0090] As shown in Table 1, compared with the original SwiGLU and moe structures, the hybrid expert model based on hierarchical feature decoupling proposed in this invention (this model) is significantly better than the two in terms of accuracy.
[0091] Table 1
[0092]
[0093] In Table 1, the evaluation metrics used are: HR@10: Hit Rate@10, the proportion of samples containing the user's target in the top 10 recommendations; HR@50: Hit Rate@50, both measuring the hit rate of the recommendations. NDCG@10: Normalized Diminished Cumulative Gain@10, combining the relevance and ranking position of the recommendation results and normalizing them; NDCG@50: Normalized Diminished Cumulative Gain@50, both measuring the ranking quality and relevance of the recommendations. MRR: Mean Reciprocal Rank, calculating the average of the reciprocals of the first result that hits the target in the recommendation list, measuring the ranking position of the first relevant result.
[0094] This method also designs a low-level fusion inference operator for expert-level computation. It fundamentally optimizes the inference process of the MoE model on GPUs from a hardware execution perspective, overcoming the insufficient parallelism and high overhead of traditional serial scheduling through the collaborative design of kernel fusion and memory access modes. Its core operation includes two key steps:
[0095] Step 1: Before the expert layer calculation, the tokens in the input vector are grouped and rearranged continuously according to the activated expert layer. After the rearrangement, the token layout in memory becomes a continuous expert group.
[0096] This step implements a hardware-oriented data layout optimization, which arranges tokens from the same expert layer contiguously in memory simply by sorting them. This aims to solve the problem of GPU core idling and discontinuous memory access caused by uneven load during inference, directly optimizing hardware execution efficiency. For example... Figure 5As shown, based on the routing results of the gating network, the tokens in the input sequence are grouped and rearranged consecutively according to the experts they are assigned to. Assuming the original input sequence contains 9 tokens (Token1-Token9, from left to right) and the number of expert layers is 4 (Expert1-Expert4), the routing results assign the tokens as follows: Expert 1 (E1): Token1, Token5; Expert 2 (E2): Token2, Token7; Expert 3 (E3): Token3, Token6, Token8; Expert 4 (E4): Token4, Token9. After rearrangement, the token layout in memory becomes a consecutive expert group: [Token1, Token5, Token2, Token7, Token3, Token6, Token8, Token4, Token9]. This data layout ensures that each GPU thread block can achieve memory access when processing a specific expert, i.e., consecutive threads access consecutive memory addresses. Compared to the original interleaved layout, this allows each GPU computing block to continuously read and process all tokens assigned to the same expert, optimizing the memory access pattern and reducing memory latency.
[0097] Step 2: This step designs a two-dimensional GPU kernel mesh structure, achieving efficient utilization of computing resources through fine-grained mesh partitioning and thread management mechanisms. The first dimension (x-dimensional) of the mesh structure corresponds to the total number of activated expert layers, ensuring that each thread block can independently handle all the computing tasks of a specific expert; the second dimension (y-dimensional) is dynamically calculated based on the maximum number of tokens that each activated expert layer needs to process in this instance, specifically by dividing the maximum number of tokens that each activated expert layer needs to process in this instance by the number of threads (block_size) contained in each GPU kernel thread block, and rounding up.
[0098] The first dimension is strongly correlated with the routing results, strictly bound to the total number of activated expert layers generated by the dynamic routing of the gated network. This enables a precise and dynamic mapping between hardware resources and software decisions for the first time. The second dimension is adaptive to the load limit; its value is not a fixed or empirical value, but is dynamically calculated based on the maximum number of tokens that a single expert layer in the current batch needs to process. This design ensures that grid resources can cover the worst-case scenario while providing a precise scope for the idle elimination mechanism.
[0099] Consider a MoE layer with 8 experts, where each expert can process a maximum of 128 tokens, and the block size is set to 32. The grid structure dimension is then configured as (8,4), resulting in a total of 8 × 4 = 32 thread blocks. Each thread block specializes in handling the computational tasks of one expert layer, and each expert layer is processed collaboratively by a maximum of 4 thread blocks, ensuring that the computational requirements for the maximum number of tokens can be covered.
[0100] During expert-level computation, the GPU core continuously reads tokens from memory based on the grid structure and performs calculations. Before starting computation, each thread first checks the actual number of tokens allocated to it in the expert layer. If the token index exceeds the token limit, the thread immediately exits the computation process. By comparing the token index with the token limit, the thread can autonomously determine the legitimacy of its computation task.
[0101] This design outperforms the traditional MoE structure in inference speed while maintaining accuracy. As shown in Table 2, with the model accuracy remaining unchanged, the inference speed of the underlying fusion inference operator is significantly faster than the two traditional implementations of expert loop and token loop when the number of experts is 16, the length of the input sequence is 100, and the dimension of the input embedding vector is 256.
[0102] Table 2
[0103]
[0104] The three core innovations of this method are activated sequentially and work synergistically: A hierarchical gated multi-head attention mechanism dynamically selects heads from the input based on expert selection, activating only the most relevant subset of attention for computation, significantly reducing computational complexity while maintaining model expressiveness; subsequently, a hybrid expert model based on hierarchical feature decoupling is used, where the attention input is the direct input to the network, and the original input is the input to the expert gating network. Through the decoupled gating network and sigmoid activation mechanism, precise routing and expert computation are achieved, effectively capturing diverse user interests and directly improving recommendation accuracy. These two parts form the overall network module, which is repeatedly stacked to form the backbone network. To ensure efficient execution of the above algorithm, the underlying fusion inference operator performs memory-level rearrangement of tokens based on the routing results, and uses a two-dimensional grid adaptive GPU kernel to achieve precise scheduling and elimination of idle resources. Finally, the deeply fused expert outputs are weighted and aggregated to generate the prediction result for the next candidate item. This method combines the advantages of accuracy and efficiency through a hierarchical gated multi-head attention mechanism based on expert selection, a hybrid expert model based on hierarchical feature decoupling, and a low-level fusion inference operator. It achieves high recommendation accuracy with low time overhead and high resource utilization.
[0105] This embodiment also provides a hybrid expert model sparse inference system for generative recommendation, which adopts the following technical solution: including a data acquisition module, a dynamic head selection module, and a hybrid expert model.
[0106] The data acquisition module is used to acquire raw input data and obtain an input vector through the embedding layer;
[0107] The dynamic head selection module is used to obtain hybrid expert weights by normalizing the input vector and selecting the maximum value; and to calculate attention enhancement features based on the hierarchical attention mechanism and the hybrid expert weights.
[0108] The calculation process for hybrid expert weights is as follows:
[0109] The input vector is passed through a gating mechanism to obtain the total weight of the shared header and the total weight of the routing header;
[0110] The total weight of the shared header is normalized to obtain a fine-grained weight; the total weight of the routing header is selected by selecting the maximum value to obtain a sparse weight.
[0111] By fusing fine-grained weights with sparse weights, a hybrid expert weight is obtained.
[0112] The hybrid expert model is used to calculate the recommendation result based on the input vector and attention enhancement features. Specifically, a parallel gating mechanism based on Sigmoid activation is used to calculate the expert weights based on the attention enhancement features. The activated expert layer is selected based on the expert weights, and the recommendation result is calculated using the input vector.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A sparse inference method using a hybrid expert model for generative recommendation, characterized in that, include: S1: Obtain the original input data and, through the embedding layer, obtain the input vector; The user's historical behavior sequence and its characteristics in text form are used as the raw input data; S2: The input vector is normalized and the maximum value is selected to obtain the hybrid expert weights; based on the hierarchical attention mechanism, the attention enhancement features are calculated according to the hybrid expert weights. The calculation process for hybrid expert weights is as follows: The input vector is passed through a gating mechanism to obtain the total weight of the shared header and the total weight of the routing header; The total weight of the shared head is normalized to obtain fine-grained weights; The total weight of the routing header is obtained by selecting the maximum value to obtain the sparse weight; By fusing fine-grained weights with sparse weights, a hybrid expert weight is obtained. S3: Input the input vector and attention-enhanced features into the hybrid expert model to calculate the recommendation result; Among them, the parallel gating mechanism based on Sigmoid activation calculates expert weights according to attention enhancement features; the activated expert layer is selected according to the expert weights, and the recommendation result is calculated using the input vector; The specific workflow includes: attention enhancement features are passed through gating layers and layer divisions to obtain expert scores, and expert scores are activated by Sigmoid to obtain expert weights; The expert weights are used as indexes to select the activated expert layers. Each activated expert layer calculates the expert prediction result based on the input vector after RMSNorm normalization. The expert weights are used as weights. After summing and taking the reciprocal, they are multiplied with the expert prediction results by matrix multiplication. Finally, they are added to the input vector after RMSNorm normalization to obtain the recommendation result.
2. The sparse inference method of hybrid expert model for generative recommendation as described in claim 1, characterized in that, In the calculation of the hybrid expert weights in step S2, the gating mechanism is implemented through the first linear layer combined with the Softmax function; the maximum value selection operation is implemented through the second linear layer combined with Gumbel noise and Top-k operation.
3. The sparse inference method of hybrid expert model for generative recommendation as described in claim 1, characterized in that, In step S2, when calculating attention enhancement features, the matrix projection calculation of query, key, and value is performed only for the selected attention head; during the attention score calculation process, a gating weight perception adjustment mechanism is introduced, and the hybrid expert weights are incorporated as weighting factors into the attention score calculation. The outputs of each activated attention head are weighted and fused according to their gating weights, and then the gating attention output features are formed through residual connections and layer normalization operations. The gated attention output feature is added to the input vector to obtain the attention-enhanced feature.
4. The sparse inference method of hybrid expert model for generative recommendation as described in claim 3, characterized in that, In step S2, the formula for calculating the gated attention output features is: in, The weights are the values corresponding to the input vector. The weights corresponding to the projection information are... To query the weights corresponding to the projection information, The weights corresponding to the key projection information. For attention weights, For the partitioning operation, For intermediate attention variables, For activation function, For Einstein's summation agreement operation, For mixed expert weights, Scaling factor The intermediate attention variable after masking. For the mask, For time information and The combined variables, This is a bias model based on bucketing time intervals and relative positions. For time information, For location information and The combined variables, For location information, for and The combined variables, This is for intermediate splicing. For splicing operations, For gating attention output features, This is for root mean square normalization.
5. The sparse inference method of hybrid expert model for generative recommendation as described in claim 1, characterized in that, In step S3, the attention-enhanced features are passed through a gating layer and a scoring layer to obtain expert scores. The expert scores are then processed by Sigmoid activation to obtain expert weights. Sigmoid activation enables parallel and independent expert evaluation based on absolute confidence, making the activation state calculation of each expert layer independent of each other.
6. A sparse inference method for hybrid expert models oriented towards generative recommendation as described in claim 1 or 5, characterized in that, In step S3, the activated expert layer is selected according to the expert weights, and the output weights of the gating network are calculated; the activated expert layer calculates the expert prediction result based on the input vector; and the recommendation result is calculated based on the input vector, the output weights of the gating network, and the expert prediction result.
7. The sparse inference method of hybrid expert model for generative recommendation as described in claim 5, characterized in that, The total loss function during hybrid expert model training includes load balancing loss; The load balancing loss is used to control the concentration of activation distribution within experts and the balance of load distribution among experts. Its calculation formula is as follows: in, For load balancing losses, For expert internal smoothness adjustment factor, The total number of experts, For expert indexing, For batch size, For sample index, Assign probabilities to the sample-expert pair. This serves as a factor for adjusting the balance among experts. This represents the average activation probability of experts.
8. The sparse inference method of hybrid expert model for generative recommendation as described in claim 1, characterized in that, In step S2, the input vector is first subjected to layer normalization and then used to calculate the hybrid expert weights; In step S3, the input vector is first subjected to RMSNorm normalization and then input into the activated expert layer.
9. The sparse inference method of hybrid expert model for generative recommendation as described in claim 1, characterized in that, In step S3, Before the expert layer calculation, the tokens in the input vector are grouped and rearranged continuously according to the activated expert layers. After rearrangement, the token layout in memory becomes continuous expert groups: a two-dimensional GPU kernel grid structure is constructed. The first dimension of the grid structure corresponds to the total number of activated expert layers, and the second dimension is the maximum number of tokens that each activated expert layer needs to process in this time divided by the number of threads contained in each GPU kernel thread block, and rounded up. During expert layer computation, the GPU kernel continuously reads tokens from memory according to the grid structure and performs computations. Before each thread starts computation, it first queries the number of tokens actually allocated to its corresponding expert layer. If the token index exceeds the number of tokens, the thread immediately exits the computation process.
10. A hybrid expert model sparse reasoning system for generative recommendation, characterized in that, A hybrid expert model sparse inference method for generative recommendation as described in any one of claims 1 to 9, comprising: a data acquisition module, a dynamic head selection module, and a hybrid expert model. The data acquisition module is used to acquire raw input data and obtain an input vector through the embedding layer; The dynamic head selection module is used to obtain hybrid expert weights by normalizing the input vector and selecting the maximum value; and to calculate attention enhancement features based on the hierarchical attention mechanism and the hybrid expert weights. The calculation process for hybrid expert weights is as follows: The input vector is passed through a gating mechanism to obtain the total weight of the shared header and the total weight of the routing header; The total weight of the shared header is normalized to obtain a fine-grained weight; the total weight of the routing header is selected by selecting the maximum value to obtain a sparse weight. By fusing fine-grained weights with sparse weights, a hybrid expert weight is obtained. The hybrid expert model is used to calculate the recommendation result based on the input vector and attention enhancement features. Specifically, a parallel gating mechanism based on Sigmoid activation is used to calculate the expert weights based on the attention enhancement features. The activated expert layer is selected based on the expert weights, and the recommendation result is calculated using the input vector.