A hybrid sparse expert inference acceleration method and device based on routing guidance shared base

Abstract

The application discloses a hybrid sparse expert reasoning acceleration method and device based on routing guidance shared base. The hybrid sparse expert reasoning acceleration method based on routing guidance shared base comprises the following steps: deploying an original pre-trained MoE model according to a modified final shared base expert weight matrix and an optimized final expert-to-base projection matrix, so as to obtain a deployed MoE hybrid reasoning model; obtaining input information of a user; inputting the input information of the user into the deployed MoE hybrid reasoning model, so as to obtain a reasoning output result. The hybrid sparse expert reasoning acceleration method based on routing guidance shared base can greatly reduce the number of activated experts, dynamically compensate for discarded experts by using shared base experts, and effectively maintain model accuracy while significantly improving reasoning efficiency by combining the initialization strategies of Fisher information weighting and co-activation perception without gradient-based fine-tuning training.

Description

Technical Field

[0001] This application relates to the field of large language model technology, specifically to a hybrid sparse expert reasoning acceleration method and device based on a route-guided shared base. Background Technology

[0002] Optimizing the inference performance of large-scale language models (LLMs) has become a core focus of research and industry. The Mixture-of-Experts (MoE) architecture, through sparse activation mechanisms, significantly expands the model parameter scale while maintaining low inference computation, and has become an important technical approach for large-scale language models.

[0003] Typical MoE models (such as GLM-4.5-Air and DeepSeek-V3) typically contain dozens to hundreds of expert subnetworks. Each input token dynamically selects several experts for computation via a router. It's worth noting that some MoE models (such as DeepSeek-V3) introduce a hybrid structure of "shared experts + routing experts" during the architecture design phase. This means that several experts are reserved and permanently active for all input tokens, without participating in routing selection, to provide general basic capabilities across tokens.

[0004] However, this mechanism is an architectural design choice in the model training phase and requires training from scratch or full fine-tuning to take effect; shared experts provide the same fixed compensation for all inputs and cannot adaptively adjust based on expert combinations where different input tokens are discarded; and the introduction of shared experts does not reduce the number of activated routing experts, so it cannot alleviate the bandwidth fragmentation problem in high-throughput scenarios.

[0005] In high-throughput inference scenarios (i.e., OTPS > 100 tokens per second), activating a large number of experts (such as the standard Top-8) can lead to severe memory bandwidth fragmentation. Specifically, the activation of a large number of experts will divide the global batch into multiple micro-batches. Each expert is allocated too few tokens, which cannot fully utilize the parallel computing power of the GPU Tensor Cores, resulting in low arithmetic intensity. The entire system then falls into a memory access bottleneck rather than a computational bottleneck.

[0006] The existing technical solutions for MoE model compression and inference optimization mainly include the following three types of solutions: (1) Expert pruning method directly reduces the number of activated experts (e.g., from Top-8 to Top-2) to reduce the number of memory accesses and computational overhead. REAP proposed a pruning criterion based on routing weight and expert activation norm, proving that pruning is better than merging in generation tasks. Advantages: Simple to implement, can directly reduce the amount of computation and memory access during inference. Disadvantages: Aggressive expert pruning (e.g., from Top-8 to Top-2) will lead to a serious decrease in accuracy because the specialized knowledge carried by the discarded experts is completely ignored. This work also proved that expert merging will introduce an unavoidable error due to "functional subspace collapse" (i.e., the knowledge of multiple experts is forcibly mapped into the representation space of a single expert, resulting in the irreversible loss of the unique specialized features of each expert).

[0007] (2) Low-rank decomposition methods based on shared bases utilize matrix decomposition techniques to represent multiple experts as a linear combination of shared basis matrices, thereby reducing parameter redundancy. MoBE decomposes the weight of each expert into W=AB, where the larger matrix B is further represented as a linear combination of the basis matrices shared by all experts. Advantages: It can effectively reduce storage usage, with only a 1%-2% decrease in accuracy at a compression rate of 24%-30%. Disadvantages: It mainly involves mixing in the weight space, i.e., calculating the weighted weight matrix before forward propagation. This approach can lead to internal covariate shift when using nonlinear activation functions such as SwiGLU, affecting numerical stability. In addition, these methods typically use static coefficient allocation and fail to utilize the dynamic information of the router for adaptive compensation.

[0008] (3) The Delta decomposition method decomposes expert weights into shared basis weights and unique Delta residuals for each expert. The decomposition is guided by the Fisher information matrix, and SVD is used to perform low-rank compression of the Delta weights. Advantages: It utilizes the similarity between expert weights for efficient compression; the Fisher information-weighted decomposition preserves the characteristics of key experts. Disadvantages: Delta decomposition primarily focuses on model compression (reducing storage footprint) and does not specifically address bandwidth fragmentation during inference; multiple independent low-rank residuals still need to be loaded after compression, failing to improve the computational density for each expert.

[0009] The existing technology has the following drawbacks: (1) The contradiction between accuracy and efficiency: Although the simple expert pruning method can improve the inference efficiency, it will cause unacceptable loss of accuracy. Taking the GLM-4.5-Air model as an example, after the Top-8 is directly pruned to the Top-2, the model output degenerates into repetitive and meaningless text, and completely loses the ability to generate language.

[0010] (2) Numerical instability of weight space mixing: Existing methods based on shared base perform linear mixing in the weight space, but MoE experts generally use the SwiGLU nonlinear activation function (i.e., the element-wise product structure of gated projection and up projection) to linearly combine the weights and then activate them nonlinearly, which will produce serious internal covariate shift, resulting in numerical instability and decreased accuracy.

[0011] (3) Lack of dynamic adaptability: Existing shared expert or expert merging schemes adopt static strategies and fail to utilize the dynamic routing information generated by the router during inference. The expert sets that are discarded for different input tokens are different, and static compensation mechanisms cannot specifically fill the functional gaps unique to each token.

[0012] (4) Lack of theoretical support for initialization: When constructing a shared base, existing methods usually use simple mean or random initialization, without fully considering the differences in importance of each expert and the co-activation mode among experts, resulting in low initialization quality and requiring additional fine-tuning training to restore accuracy. Summary of the Invention

[0013] The purpose of this invention is to provide a hybrid sparse expert inference acceleration method based on a route-guided shared base to at least solve one of the above-mentioned technical problems.

[0014] One aspect of the present invention provides a method for accelerating hybrid sparse expert inference based on a route-bootstrapping shared base, the method comprising: Obtain the original pre-trained MoE model; Obtain a representative calibration dataset; Inference is performed on the original pre-trained MoE model based on a representative calibration dataset to obtain the expert importance matrix and the expert co-activation matrix; Construct an input-side expert weight stacking matrix and an output-side expert weight stacking matrix. Use the expert importance matrix to weight the expert weight stacking matrix. Perform SVD decomposition on the weighted input-side expert weight stacking matrix and the output-side expert weight stacking matrix respectively to obtain the initial weight matrix, singular value matrix and initial projection matrix of the shared base experts. The final, corrected shared base expert weight matrix is ​​generated based on the initial weight matrix and singular value matrix of the shared base experts. The optimized final expert-to-base projection matrix is ​​generated based on the expert co-activation matrix and the initial projection matrix. The original pre-trained MoE model is deployed based on the corrected final shared pedestal expert weight matrix and the optimized final expert-to-pedestal projection matrix, thereby obtaining the deployed MoE hybrid inference model. Obtain user input information; The user's input information is fed into the deployed MoE hybrid inference model to obtain the inference output result.

[0015] Optionally, the step of inferring the original pre-trained MoE model based on a representative calibration dataset to obtain the expert importance matrix and the expert co-activation matrix includes: For each input token feature vector x in the calibration dataset D, perform pure forward inference of the original MoE model to obtain the N-dimensional gate vector G(x) output by the router, which contains the routing scores of all N original experts, where N is the total number of original experts in the MoE layer. Perform Top-k' segmentation on the gate vector G(x) according to the preset k', to obtain the N-dimensional discard gate vector corresponding to the pruned expert. This vector only retains The routing scores of experts who are discarded are set to 0 in all other positions. Calculate the Fisher information score for each original expert. ; Construct an N×N diagonal expert importance matrix Ω based on the Fisher information scores of all original experts; Capture the semantic clustering patterns among the original experts and calculate the N×N dimensional expert co-activation matrix C.

[0016] Optionally, the step of constructing the input-side expert weight stack matrix and the output-side expert weight stack matrix, weighting the expert weight stack matrix with the expert importance matrix, and performing SVD decomposition on the weighted input-side expert weight stack matrix and the output-side expert weight stack matrix respectively to obtain the shared base expert initial weight matrix, singular value matrix, and initial projection matrix includes: Obtain the weight parameters of N experts in the original pre-trained MoE model, where the weight parameters of the N experts include the gated projection matrix of the i-th expert. Upper projection matrix Lower projection matrix ; According to the gated projection matrix Upper projection matrix Constructing an input-side expert weight stack matrix ; According to the lower projection matrix Construct the output-side expert weight stack matrix ; Perform singular value decomposition on the Fisher-weighted input-side stacked matrix; Perform singular value decomposition on the Fisher-weighted output-side stacked matrix; Pick and The first M right singular vectors are reshaped into the input-side initial weight matrix and the output-side initial weight matrix of the shared base expert, respectively, and then merged to obtain the complete initial weight matrix of the M shared base experts. Left singular matrix obtained from SVD decomposition , and singular value diagonal matrix Extract and concatenate to obtain the N×M dimensional initial projection matrix. .

[0017] Optionally, generating the corrected final shared base expert weight matrix based on the initial shared base expert weight matrix and the singular value matrix includes: The matrix obtained by SVD decomposition is scaled and the element values ​​of the unit orthogonal vectors are corrected to obtain the corrected final shared base expert weight matrix.

[0018] Optionally, generating the optimized final expert-to-base projection matrix based on the expert co-activation matrix and the initial projection matrix includes: Normalize the expert co-activation matrix C to obtain an N×N dimensional normalized co-activation matrix. ; Based on the normalized coactivation matrix Solving for the N×M dimensional final expert-to-base projection matrix yields... .

[0019] Optionally, the step of inputting user input information into the deployed MoE hybrid inference model to obtain inference output results includes: The user's input information is segmented into routes to obtain the retained gating vector and the discarded gating vector; Based on the retained gating vector, the user's input information is used to perform sparse path forward computation to obtain the final output vector of the sparse path; The final output vector of the base compensation path is generated based on the corrected final shared base expert weight matrix, the optimized final expert-to-base projection matrix, the discard gating vector, and the user's input information. The final output vector of the sparse path and the final output vector of the base compensation path are used to generate the inference output result.

[0020] Optionally, the step of performing routing segmentation processing on the user's input information to obtain the retention gate vector and the discard gate vector includes: User input enters the router, and through linear projection and Softmax normalization calculations, it outputs an N-dimensional gating vector containing all N original expert routing scores. ; For gated vectors Perform Top-k' splitting to split the gated vectors into retained gated vectors and discarded gated vectors.

[0021] Optionally, generating the final output vector of the base compensation path based on the modified final shared base expert weight matrix, the optimized final expert-to-base projection matrix, the discard gating vector, and the user's input information includes: Calculate the dynamic hybrid weight of each shared base expert based on the routing information of the discarded experts; SwiGLU forward computation is performed on each of the shared base experts to obtain the independent forward computation results for each base expert. An activation space mixing strategy is adopted. First, the independent forward calculations of all base experts are completed. Then, the output results are weighted and summed according to the mixing coefficient to obtain the final output vector of the base compensation path.

[0022] Optionally, generating the inference output result by combining the final output vector of the sparse path with the final output vector of the base compensation path includes: The fusion is performed using the following formula:

[0023] ; in, This is the final output vector of the sparse path; The final output vector for the base compensation path; The user's input information x corresponds to the routing score of the i-th original expert; This is the output result of the forward computation of the i-th original expert; Output the forward computation result for the m-th shared base expert; Forward computation output for the m-th shared base expert The weighted weights.

[0024] This application also provides a hybrid sparse expert inference acceleration device based on a route-guided shared base, the hybrid sparse expert inference acceleration device based on a route-guided shared base includes: Original pre-trained MoE model acquisition module, which is used to acquire the original pre-trained MoE model; A representative calibration dataset acquisition module, which is used to acquire a representative calibration dataset; The inference module is used to infer the original pre-trained MoE model based on a representative calibration dataset, thereby obtaining the expert importance matrix and the expert co-activation matrix. The SVD decomposition module is used to construct an input-side expert weight stacking matrix and an output-side expert weight stacking matrix, weight the expert weight stacking matrix with the expert importance matrix, and perform SVD decomposition on the weighted input-side expert weight stacking matrix and the output-side expert weight stacking matrix respectively, thereby obtaining the shared base expert initial weight matrix, singular value matrix and initial projection matrix. The correction module is used to generate a corrected final shared base expert weight matrix based on the initial weight matrix of the shared base experts and the singular value matrix. The optimization module is used to generate an optimized final expert-to-base projection matrix based on the expert co-activation matrix and the initial projection matrix. The deployment module is used to deploy the original pre-trained MoE model according to the corrected final shared pedestal expert weight matrix and the optimized final expert-to-pedestal projection matrix, thereby obtaining the deployed MoE hybrid inference model. An input information acquisition module is used to acquire user input information. The actual inference module is used to input user input information into the deployed MoE hybrid inference model to obtain inference output results.

[0025] The proposed method for accelerating hybrid sparse expert inference based on route-guided shared pedestals significantly reduces the number of activated experts while dynamically compensating for discarded experts using shared pedestal experts. Combined with Fisher information weighting and co-activation awareness initialization strategies, it achieves a significant improvement in inference efficiency and effective maintenance of model accuracy without the need for gradient-based fine-tuning training. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a method for accelerating hybrid sparse expert inference based on a route-guided shared base according to an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0028] like Figure 1 The hybrid sparse expert inference acceleration method based on route-guided shared base shown includes: Obtain the original pre-trained MoE model; Obtain a representative calibration dataset; Inference is performed on the original pre-trained MoE model based on a representative calibration dataset to obtain the expert importance matrix and the expert co-activation matrix; Construct an input-side expert weight stacking matrix and an output-side expert weight stacking matrix. Use the expert importance matrix to weight the expert weight stacking matrix. Perform SVD decomposition on the weighted input-side expert weight stacking matrix and the output-side expert weight stacking matrix respectively to obtain the initial weight matrix, singular value matrix and initial projection matrix of the shared base experts. The final, corrected shared base expert weight matrix is ​​generated based on the initial weight matrix and singular value matrix of the shared base experts. The optimized final expert-to-base projection matrix is ​​generated based on the expert co-activation matrix and the initial projection matrix. The original pre-trained MoE model is deployed based on the corrected final shared pedestal expert weight matrix and the optimized final expert-to-pedestal projection matrix, thereby obtaining the deployed MoE hybrid inference model. Obtain user input information; The user's input information is fed into the deployed MoE hybrid inference model to obtain the inference output result.

[0029] In this embodiment, the step of performing inference on the original pre-trained MoE model based on a representative calibration dataset to obtain the expert importance matrix and the expert co-activation matrix includes: Freeze all weights of the original pre-trained MoE model; obtain the preset number of sparsely retained experts k′ for the inference stage.

[0030] For each input token feature vector x in the calibration dataset D, perform pure forward inference of the original MoE model to obtain the N-dimensional gate vector G(x) output by the router, which contains the routing scores of all N original experts, where N is the total number of original experts in the MoE layer. Perform Top-k' segmentation on the gate vector G(x) according to the preset number of sparsely retained experts k′ in the inference stage, to obtain the N-dimensional discard gate vector corresponding to the pruned experts. This vector only retains The routing scores of experts who are discarded are set to 0 in all other positions, and the segmentation rule is completely consistent with the subsequent reasoning stage. Calculate the Fisher information score for each original expert. The sensitivity of the i-th expert to the model's output loss is quantified by the following formula: ; in, Given the routing score of the i-th original expert corresponding to the input token x, Given the 2-norm of the input token feature vector x, the summation operation covers all input tokens in the calibration dataset D.

[0031] An N×N diagonal expert importance matrix Ω is constructed based on the Fisher information scores of all original experts, as shown in the following formula: ; in Construct a diagonal matrix, where the elements on the main diagonal are the square roots of the Fisher information scores of each expert, and the off-diagonal elements are all 0.

[0032] Calculate the N×N dimensional expert co-activation matrix C to capture the semantic clustering patterns among the original experts (i.e., which experts are often jointly discarded), providing constraints for the optimization of the projection matrix. The formula is as follows (for example, "programming" experts are often jointly discarded): ; in, To calculate the expected value of all input tokens x in the calibration dataset D, To discard the gate vector The transpose of and the product of the two is a cross product operation.

[0033] In this embodiment, the steps of constructing the input-side expert weight stack matrix and the output-side expert weight stack matrix, weighting the expert weight stack matrix with the expert importance matrix, and performing SVD decomposition on the weighted input-side expert weight stack matrix and the output-side expert weight stack matrix respectively to obtain the shared base expert initial weight matrix, singular value matrix, and initial projection matrix include: Obtain the weight parameters of N experts in the original pre-trained MoE model, where the weight parameters of the N experts include the gated projection matrix of the i-th expert. Upper projection matrix Lower projection matrix ; According to the gated projection matrix Upper projection matrix Constructing an input-side expert weight stack matrix Specifically, to capture the correlation between expert weights, an input-side expert weight stacking matrix is ​​constructed. Each original expert's and After flattening, the layers are stacked along the expert dimensions, using the following formula: ; in, The matrix flattening operation converts a two-dimensional matrix into a one-dimensional row vector; For the i-th expert and Perform column concatenation; subscript This represents the result of stacking the flattened results of N original experts along the row dimension, ultimately... For N×(2· · (Model hidden layer dimension) dimensional matrix.

[0034] According to the lower projection matrix Construct the output-side expert weight stack matrix Specifically, constructing an output-side expert weight stacking matrix. Each original expert's After flattening, stack along the expert dimensions, as shown in the following formula: ; Among them, subscript Represents the relationship between N original experts. The flattened results are stacked along the row dimensions, ultimately For N×( (Model hidden layer dimension) )-dimensional matrix.

[0035] Singular Value Decomposition (SVD) is performed on the Fisher-weighted input-side stacked matrix. Specifically, the Fisher-weighted input-side stacked matrix is ​​weighted using the importance matrix Ω and then subjected to SVD, prioritizing the retention of the weight features of highly important experts. The formula is as follows: ; in, The expert importance matrix Ω and the input-side stacking matrix Matrix multiplication; It is a left singular matrix. It is a singular value diagonal matrix. A right singular matrix The transpose of .

[0036] Singular value decomposition (SVD) is performed on the Fisher-weighted output-side stacked matrix; specifically, the formula is as follows: ; in, The expert importance matrix Ω and the output-side stacking matrix Matrix multiplication; It is a left singular matrix. It is a singular value diagonal matrix. A right singular matrix The transpose of .

[0037] Pick and The first M right singular vectors are reshaped into the input-side and output-side initial weight matrices of the shared base experts, respectively, and then merged to obtain the complete initial weight matrices of the M shared base experts (including the base expert gated projection matrix). Upper projection matrix Lower projection matrix (m=1,2,...,M). Left singular matrix obtained from SVD decomposition , and singular value diagonal matrix Extract and concatenate to obtain the N×M dimensional initial projection matrix. .

[0038] In this embodiment, generating the corrected final shared base expert weight matrix based on the initial shared base expert weight matrix and the singular value matrix includes: The matrix obtained by SVD decomposition is scaled and the element values ​​of the unit orthogonal vectors are corrected to obtain the corrected final shared base expert weight matrix.

[0039] In this embodiment, the matrix obtained by SVD decomposition is scaled and the element values ​​of the unit orthogonal vectors are corrected to obtain the corrected final shared pedestal expert weight matrix, which includes: Drawing on the Delta Decompression strategy of D²-MoE, a scale allocation is performed on the matrix obtained by SVD decomposition to correct the problem of excessively small element values ​​in the identity orthogonal vectors, ensuring that the SwiGLU activation values ​​are within the effective range in the forward computation of the pedestal expert. The formula is as follows: ; Where W is the weighted stacking matrix before SVD decomposition ( or U is a left singular matrix ( or ), where Σ is a singular value diagonal matrix ( or ), VT is the transpose of the right singular matrix ( or ); For singular valued diagonal matrices The diagonal matrix obtained by taking the square root of the singular values ​​of the main diagonal; The magnitude is allocated to the initial weight matrix of the shared base experts, completing the weighting of all base experts ( , , The scale is corrected to obtain the final weight matrix of the M shared base experts.

[0040] In this embodiment, generating the optimized final expert-to-base projection matrix based on the expert co-activation matrix and the initial projection matrix includes: Normalize the expert co-activation matrix C to obtain an N×N dimensional normalized co-activation matrix. ; Based on the normalized coactivation matrix Solving for the N×M dimensional final expert-to-base projection matrix yields... .

[0041] Specifically, the expert co-activation matrix C is normalized to obtain an N×N dimensional normalized co-activation matrix. The formula is as follows: ; Where diag(C) is the main diagonal element extracted from the co-activation matrix C, and mean is the arithmetic mean of the extracted main diagonal elements.

[0042] The projection matrix optimization problem is formalized as a generalized ridge regression problem, and the optimal N×M dimensional projection matrix is ​​obtained by solving it. The formula is as follows: ; in, λ is the regularization term, λ is the regularization coefficient, and I is the N×N identity matrix; It is the inverse matrix of the sum of the normalized coactivation matrix and the regularization term; by introducing coactivation matrix constraints, the projection matrix can learn to activate shared bases not only based on expert similarity, but also based on which experts are often discarded in common patterns, ensuring that the base combination can cover the most common functional gaps.

[0043] In this embodiment, the original pre-trained MoE model is deployed based on the corrected final shared pedestal expert weight matrix and the optimized final expert-to-pedestal projection matrix to obtain the deployed MoE hybrid inference model, including: Hybrid Inference Mode Configuration and Module Mounting: The hybrid sparse-pedestal inference mode can be enabled and disabled by configuring parameters, while remaining compatible with the standard inference mode of the original model. For each MoE layer of the original model, the corresponding shared base module and projection matrix are matched to complete the mounting and binding of the module with the original model. No weight parameters of the original model are modified throughout the process, thereby obtaining the MoE model file with the hybrid architecture mounted.

[0044] Model loading and interface encapsulation During the model loading phase of the inference framework, a shared base module is automatically initialized for each MoE layer to complete the weight loading; It remains completely transparent to the upper-level inference interface, and upper-level calls do not require any modification to the inference code; it only automatically adds base compensation calculations during inference. The computational logic of the pedestal experts is optimized using Grouped GEMM, merging multiple independent pedestal expert matrix multiplications into a single kernel function startup, reducing GPU kernel function scheduling overhead. Finally, a fully encapsulated MoE inference instance is obtained.

[0045] Inference service deployment and startup Deploy the inference instance as a stable online inference service capable of receiving user requests, complete service startup and availability verification, and ensure that it can process input tokens and output inference results correctly. Finally, obtain a fully deployed and operational MoE hybrid inference service.

[0046] In this embodiment, the step of inputting user input information into the deployed MoE hybrid inference model to obtain inference output results includes: The user's input information is segmented into routes to obtain the retained gating vector and the discarded gating vector; Based on the retained gating vector, the user's input information is used to perform sparse path forward computation to obtain the final output vector of the sparse path; The final output vector of the base compensation path is generated based on the corrected final shared base expert weight matrix, the optimized final expert-to-base projection matrix, the discard gating vector, and the user's input information. The final output vector of the sparse path and the final output vector of the base compensation path are used to generate the inference output result.

[0047] In this embodiment, the step of performing routing segmentation processing on the user's input information to obtain the retention gate vector and the discard gate vector includes: User input information (let the user input token feature vector be...) The data enters the router, and through linear projection and Softmax normalization calculations, outputs an N-dimensional gating vector containing all N original expert routing scores. ; For gated vectors Perform Top-k' splitting to divide the gated vector into retained gated vectors and discarded gated vectors; the formula is as follows: ; in, N-dimensional preserving gated vectors, only retain Set the original values ​​of the positions corresponding to the top-k' highest route scores, and set all other positions to 0; Discard the gated vector for N dimensions, and only retain The original values ​​of the experts whose positions are removed outside of Top-k′ are set to 0, and all other positions are set to 0.

[0048] In this embodiment, the sparse path forward computation is performed on the user's input information based on the retained gating vector to obtain the final output vector of the sparse path, including: right For the top-k' retained original experts, perform standard SwiGLU structured forward computation on each one. The forward computation formula for the i-th retained expert is: ; in, It is the Sigmoid linear unit activation function; x is the matrix multiplication of the gated projection matrix and the input vector, and the output is the gated branch result; The expression represents the matrix multiplication of the projection matrix and the input vector, and outputs the projection branch result; ⊙ represents the element-wise multiplication of the two branch results; The final projection of the activation result onto the lower projection matrix yields the expert's forward output.

[0049] For all Top-k retained experts' forward outputs, the weighted sums based on their corresponding route scores yield the final output of the sparse path, as shown in the formula: ; Where i∈Top−k′ indicates that the summation range covers all experts retained by Top-k′; The routing score corresponding to the i-th reserved expert; This is the output of the forward computation for the i-th retained expert. This is the final output vector of the sparse path.

[0050] In this embodiment, generating the final output vector of the base compensation path based on the corrected final shared base expert weight matrix, the optimized final expert-to-base projection matrix, the discard gating vector, and the user's input information includes: Based on the routing information of the discarded experts, calculate the dynamic hybrid weight of each shared base expert, using the following formula: ; To discard the transpose of the gated vector; The optimal projection matrix is ​​used to map the N-dimensional discarded routing vector to the M-dimensional base mixing coefficient space. The final output is an M-dimensional hybrid coefficient vector, where each element corresponds to a weighted weight of a shared pedestal expert. The weights change dynamically as the expert combination discards the input token.

[0051] For each of the M shared base experts, an independent SwiGLU forward computation, identical to the original expert structure, is performed. The forward computation formula for the m-th base expert is: ; in, , , The weight parameters of the m-th shared base expert output above; the calculation logic within the formula is completely consistent with the forward calculation of the original expert, ensuring that the base expert has the same feature expression ability as the original expert.

[0052] An activation space mixing strategy is adopted. First, the independent forward calculations of all base experts are completed. Then, the output results are weighted and summed according to the mixing coefficient to obtain the final output vector of the base compensation path.

[0053] Specifically, this application employs activation-space mixing instead of weight-space mixing. That is, each plinth expert first independently completes the forward computation involving nonlinear activation, and then the outputs are linearly combined. This avoids the internal covariate shift problem caused by mixing weights before the SwiGLU nonlinearity, ensuring numerical stability. The specific formula is as follows: ;

[0054] in, Mixing coefficient vector The weight corresponding to the m-th base expert; This is the independent forward computation output of the m-th pedestal expert; the summation range covers all M shared pedestal experts. This strategy completely avoids the internal covariate shift problem caused by the mixing of weight space before combining weights in the SwiGLU nonlinear activation, ensuring numerical stability.

[0055] In this embodiment, generating the inference output result by combining the final output vector of the sparse path with the final output vector of the base compensation path includes: The retained expert calculation results of the sparse path are directly added to the discarded expert function approximation results of the base compensation path to complete the dual-output fusion, obtaining the final inference result of the MoE layer corresponding to the current input token. The fusion is performed using the following formula:

[0056] ; in, This is the final output vector of the sparse path; The final output vector for the base compensation path; The user's input information x corresponds to the routing score of the i-th original expert; This is the output result of the forward computation of the i-th original expert; Output the forward computation result for the m-th shared base expert; Forward computation output for the m-th shared base expert The weighted weights.

[0057] This application has the following advantages: (1) Dynamic base compensation mechanism for route guidance: The discarded expert route scores generated by the router are used to dynamically calculate the base mixing coefficient through the projection matrix to achieve adaptive compensation for different input tokens. The projection matrix P maps the N-dimensional discarded route vector to the M-dimensional base coefficient space, so that the compensation intensity changes dynamically with the combination of discarded experts.

[0058] (2) Activation-space mixing strategy: The activation-space mixing strategy is adopted by first independently forward computing and then linearly weighting. This avoids the internal covariate shift problem caused by the linear combination of weights before the nonlinear activation of SwiGLU in the weight space mixing, thus ensuring numerical stability.

[0059] (3) Fisher information weighted SVD base initialization: The importance difference of each expert is quantified by Fisher information. Before SVD decomposition, the expert weights are weighted by importance so that the decomposition prioritizes the retention of the expert features that have the greatest impact on the model output, thus achieving high-quality base initialization.

[0060] (4) Optimization of the projection matrix of co-activation sensing: Using the expert co-activation matrix collected from calibration data, the projection matrix is ​​optimized by generalized ridge regression, so that the base combination can cover the most common discarded expert combination patterns and improve the global consistency of compensation.

[0061] (5) D²-MoE style scale correction (referencing Delta Decompression method): After SVD decomposition, the scale of the base weights is redistributed, and the square root of the singular value is assigned to the base weight matrix, which solves the problem of too small activation value caused by directly using unit orthogonal basis vectors.

[0062] (6) Plug-and-play solution without gradient fine-tuning: The entire base initialization and inference process only relies on the statistical analysis of calibration data, does not modify the original model weights, does not require additional gradient training, and can be directly applied to the deployed MoE model.

[0063] Taking the GLM-4.5-Air model (128 experts, standard Top-8 routing) as an example, the advantages of this invention over existing technologies can be clearly demonstrated: (1) Compared to the simple expert pruning scheme: The simple pruning scheme, which directly reduces Top-8 to Top-2, will cause the model output to completely degenerate into repetitive and meaningless text. However, the hybrid architecture (Top-2 + pedestal compensation) of this invention maintains the sparsity of Top-2 while effectively recovering the functional contributions of the discarded experts through dynamic pedestal compensation. In single-layer offline testing, compared with the complete discarding scheme, the pedestal compensation method of this invention can significantly reduce the output error and does not require additional storage overhead.

[0064] (2) Compared with the weight space mixing scheme: Existing methods such as MoBE mix the weight space, which leads to numerical instability in expert structures containing SwiGLU activation. The activation space mixing strategy adopted in this invention completely avoids this problem. Experiments show that the compensation effect after combining projection matrix optimization with co-activation sensing and activation space mixing is far better than the pure weight space decomposition method, verifying the superiority of the technical route of this invention.

[0065] (3) Compared with static shared expert schemes: traditional shared experts provide the same compensation for all tokens. The routing guidance mechanism of this invention makes the compensation change dynamically with the input: different expert combinations are discarded for different tokens, and the corresponding base mixing coefficients are also different, realizing refined adaptive compensation.

[0066] (4) Compared with schemes that require gradient-based fine-tuning: This invention only requires a small amount of calibration data (such as 500 samples) to complete the base initialization. The entire process does not require gradient calculation and backpropagation, which greatly reduces the deployment cost. Fisher information weighted and co-activation sensing optimization ensures the initialization quality under the condition of no fine-tuning.

[0067] (5) Improved inference efficiency: By reducing Top-8 to Top-2, the number of tokens allocated to each active expert is significantly increased, and the utilization of GPU tensor cores is significantly improved. The computation of pedestal experts can be efficiently executed through Grouped GEMM, and the arithmetic strength of the overall inference is changed from a memory access bottleneck to a computational bottleneck, which is conducive to leveraging hardware performance in high-throughput scenarios.

[0068] This application also provides a hybrid sparse expert inference acceleration device based on a route-guided shared base, the hybrid sparse expert inference acceleration device based on a route-guided shared base includes: Original pre-trained MoE model acquisition module, which is used to acquire the original pre-trained MoE model; A representative calibration dataset acquisition module, which is used to acquire a representative calibration dataset; The inference module is used to infer the original pre-trained MoE model based on a representative calibration dataset, thereby obtaining the expert importance matrix and the expert co-activation matrix. The SVD decomposition module is used to construct an input-side expert weight stacking matrix and an output-side expert weight stacking matrix, weight the expert weight stacking matrix with the expert importance matrix, and perform SVD decomposition on the weighted input-side expert weight stacking matrix and the output-side expert weight stacking matrix respectively, thereby obtaining the shared base expert initial weight matrix, singular value matrix and initial projection matrix. The correction module is used to generate a corrected final shared base expert weight matrix based on the initial weight matrix of the shared base experts and the singular value matrix. The optimization module is used to generate an optimized final expert-to-base projection matrix based on the expert co-activation matrix and the initial projection matrix. The deployment module is used to deploy the original pre-trained MoE model according to the corrected final shared pedestal expert weight matrix and the optimized final expert-to-pedestal projection matrix, thereby obtaining the deployed MoE hybrid inference model. An input information acquisition module is used to acquire user input information.

[0069] The actual inference module is used to input user input information into the deployed MoE hybrid inference model to obtain inference output results.

[0070] Definitions:

[0071] MoE (Mixture-of-Experts) is a sparse activation neural network extension technique. MoE sets up multiple expert subnetworks in deep network layers and introduces a dynamic routing gating / router mechanism to automatically select a few experts for activation and computation based on input features. Compared to dense models with full activation, MoE only calls on a subset of experts for each input, resulting in a linear increase in parameter size while the computational cost increases only by a constant.

[0072] Router: The router is the core scheduling component in the MoE architecture. It is responsible for calculating the routing score of each expert based on the feature vector of the input token and selecting the experts with the highest scores for activation. Routers are typically implemented as a linear projection layer with Softmax normalization.

[0073] SwiGLU: SwiGLU is a gated linear unit activation function widely used in the feedforward network layers of modern large language models. Its calculation formula is SwiGLU(x) = SiLU(Wg·x)⊙(Wu·x), where SiLU is a Sigmoid linear unit, and ⊙ represents element-wise multiplication. The SwiGLU structure enables experts to have stronger feature representation capabilities.

[0074] Singular Value Decomposition (SVD) is a matrix factorization technique that decomposes a matrix W into the form U·Σ·V^T, where U and V are orthogonal matrices and Σ is a diagonal matrix. In this invention, SVD is used to extract the principal feature orientation from stacked expert weights as initial weights for a shared base.

[0075] Fisher information is a statistical measure of the influence of parameters on a likelihood function. In this invention, Fisher information is used to quantify the sensitivity of each expert to the model's output loss when that expert is removed, thereby determining the importance weights of the experts.

[0076] OTPS: Output Tokens Per Second (OTPS) is a core metric for measuring the inference throughput of a language model. High OTPS scenarios (e.g., OTPS > 100) are the primary application scenarios targeted by this invention.

[0077] Grouped GEMM: Grouped General Matrix Multiply (GroupedGEMM) is a technique that combines multiple independent matrix multiplication operations into a single kernel function startup, which can significantly reduce GPU kernel function scheduling overhead and improve computational efficiency.

[0078] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A hybrid sparse expert inference acceleration method based on route-guided shared base, characterized in that, The hybrid sparse expert inference acceleration method based on route-guided shared base includes: Obtain the original pre-trained MoE model; Obtain a representative calibration dataset; Inference is performed on the original pre-trained MoE model based on a representative calibration dataset to obtain the expert importance matrix and the expert co-activation matrix; Construct an input-side expert weight stacking matrix and an output-side expert weight stacking matrix. Use the expert importance matrix to weight the expert weight stacking matrix. Perform SVD decomposition on the weighted input-side expert weight stacking matrix and the output-side expert weight stacking matrix respectively to obtain the initial weight matrix, singular value matrix and initial projection matrix of the shared base experts. The final, corrected shared base expert weight matrix is ​​generated based on the initial weight matrix and singular value matrix of the shared base experts. The optimized final expert-to-base projection matrix is ​​generated based on the expert co-activation matrix and the initial projection matrix. The original pre-trained MoE model is deployed based on the corrected final shared pedestal expert weight matrix and the optimized final expert-to-pedestal projection matrix, thereby obtaining the deployed MoE hybrid inference model. Obtain user input information; The user's input information is fed into the deployed MoE hybrid inference model to obtain the inference output result.

2. The hybrid sparse expert inference acceleration method based on route-guided shared base as described in claim 1, characterized in that, The step of inferring the original pre-trained MoE model based on a representative calibration dataset to obtain the expert importance matrix and expert co-activation matrix includes: For each input token feature vector x in the calibration dataset D, perform pure forward inference of the original MoE model to obtain the N-dimensional gate vector G(x) output by the router, which contains the routing scores of all N original experts, where N is the total number of original experts in the MoE layer. Perform Top-k' segmentation on the gate vector G(x) according to the preset number of sparsely retained experts k′ in the inference stage, to obtain the N-dimensional discard gate vector corresponding to the pruned experts. This vector only retains The routing scores of experts who are discarded are set to 0 in all other positions. Calculate the Fisher information score for each original expert. ; Construct an N×N diagonal expert importance matrix Ω based on the Fisher information scores of all original experts; Capture the semantic clustering patterns among the original experts and calculate the N×N dimensional expert co-activation matrix C.

3. The method for accelerating hybrid sparse expert inference based on a route-guided shared base as described in claim 2, characterized in that, The process of constructing the input-side expert weight stacking matrix and the output-side expert weight stacking matrix, weighting the expert weight stacking matrix using the expert importance matrix, and performing SVD decomposition on the weighted input-side expert weight stacking matrix and the output-side expert weight stacking matrix respectively to obtain the shared base expert initial weight matrix, singular value matrix, and initial projection matrix includes: Obtain the weight parameters of N experts in the original pre-trained MoE model, where the weight parameters of the N experts include the gated projection matrix of the i-th expert. Upper projection matrix Lower projection matrix ; According to the gated projection matrix Upper projection matrix Constructing an input-side expert weight stack matrix ; According to the lower projection matrix Construct the output-side expert weight stack matrix ; Perform singular value decomposition on the Fisher-weighted input-side stacked matrix; Perform singular value decomposition on the Fisher-weighted output-side stacked matrix; Pick and The first M right singular vectors are reshaped into the input-side initial weight matrix and the output-side initial weight matrix of the shared base expert, respectively, and then merged to obtain the complete initial weight matrix of the M shared base experts. Left singular matrix obtained from SVD decomposition , and singular value diagonal matrix Extract and concatenate to obtain the N×M dimensional initial projection matrix. .

4. The hybrid sparse expert inference acceleration method based on route-guided shared base as described in claim 3, characterized in that, The process of generating the corrected final shared base expert weight matrix based on the initial shared base expert weight matrix and the singular value matrix includes: The matrix obtained by SVD decomposition is scaled and the element values ​​of the unit orthogonal vectors are corrected to obtain the corrected final shared base expert weight matrix.

5. The hybrid sparse expert inference acceleration method based on route-guided shared base as described in claim 4, characterized in that, The process of generating the optimized final expert-to-base projection matrix based on the expert co-activation matrix and the initial projection matrix includes: Normalize the expert co-activation matrix C to obtain an N×N dimensional normalized co-activation matrix. ; Based on the normalized coactivation matrix Solving for the N×M dimensional final expert-to-base projection matrix yields... .

6. The method for accelerating hybrid sparse expert inference based on a route-guided shared base as described in claim 5, characterized in that, The step of inputting user input information into the deployed MoE hybrid inference model to obtain inference output results includes: The user's input information is segmented into routes to obtain the retained gating vector and the discarded gating vector; Based on the retained gating vector, the user's input information is used to perform sparse path forward computation to obtain the final output vector of the sparse path; The final output vector of the base compensation path is generated based on the corrected final shared base expert weight matrix, the optimized final expert-to-base projection matrix, the discard gating vector, and the user's input information. The final output vector of the sparse path and the final output vector of the base compensation path are used to generate the inference output result.

7. The method for accelerating hybrid sparse expert inference based on a route-guided shared base as described in claim 6, characterized in that, The step of performing routing segmentation on the user's input information to obtain the retention gate vector and the discard gate vector includes: User input enters the router, and through linear projection and Softmax normalization calculations, it outputs an N-dimensional gating vector containing all N original expert routing scores. ; For gated vectors Perform Top-k' splitting to split the gated vectors into retained gated vectors and discarded gated vectors.

8. The method for accelerating hybrid sparse expert inference based on a route-guided shared base as described in claim 7, characterized in that, The step of generating the final output vector of the base compensation path based on the modified final shared base expert weight matrix, the optimized final expert-to-base projection matrix, the discard gating vector, and the user's input information includes: Calculate the dynamic hybrid weight of each shared base expert based on the routing information of the discarded experts; SwiGLU forward computation is performed on each of the shared base experts to obtain the independent forward computation results for each base expert. An activation space mixing strategy is adopted. First, the independent forward calculations of all base experts are completed. Then, the output results are weighted and summed according to the mixing coefficient to obtain the final output vector of the base compensation path.

9. The hybrid sparse expert inference acceleration method based on route-guided shared base as described in claim 8, characterized in that, The step of generating the inference output result by combining the final output vector of the sparse path with the final output vector of the base compensation path includes: The fusion is performed using the following formula: ； in, This is the final output vector of the sparse path; The final output vector for the base compensation path; The user's input information x corresponds to the routing score of the i-th original expert; This is the output result of the forward computation of the i-th original expert; Output the forward computation result for the m-th shared base expert; Forward computation output for the m-th shared base expert The weighted weights.

10. A hybrid sparse expert inference acceleration device based on a route-guided shared base, characterized in that, The hybrid sparse expert inference acceleration device based on the route-guided shared base includes: Original pre-trained MoE model acquisition module, which is used to acquire the original pre-trained MoE model; A representative calibration dataset acquisition module, which is used to acquire a representative calibration dataset; The inference module is used to infer the original pre-trained MoE model based on a representative calibration dataset, thereby obtaining the expert importance matrix and the expert co-activation matrix. The SVD decomposition module is used to construct an input-side expert weight stacking matrix and an output-side expert weight stacking matrix, weight the expert weight stacking matrix with the expert importance matrix, and perform SVD decomposition on the weighted input-side expert weight stacking matrix and the output-side expert weight stacking matrix respectively, thereby obtaining the shared base expert initial weight matrix, singular value matrix and initial projection matrix. The correction module is used to generate a corrected final shared base expert weight matrix based on the initial weight matrix of the shared base experts and the singular value matrix. The optimization module is used to generate an optimized final expert-to-base projection matrix based on the expert co-activation matrix and the initial projection matrix. The deployment module is used to deploy the original pre-trained MoE model according to the corrected final shared pedestal expert weight matrix and the optimized final expert-to-pedestal projection matrix, thereby obtaining the deployed MoE hybrid inference model. An input information acquisition module is used to acquire user input information. The actual inference module is used to input user input information into the deployed MoE hybrid inference model to obtain inference output results.

Images