Sparse expert hybrid large model's retraining-free pruning and reorganization method and system

Abstract

This invention discloses a retraining-free pruning and recombination method for large sparse expert hybrid models, belonging to the field of large-scale model compression and optimization technology. The method aims to solve the problems of excessive memory consumption and deployment difficulties caused by the need to load all expert parameters in existing sparse expert hybrid (SMoE) models. The method first identifies and prunes redundant experts based on route activation statistics; then, the pruned experts are decomposed into neuron-level functional fragments, and these fragments are redistributed to the retained experts based on structural similarity; finally, a weighted clustering algorithm is used to merge the original fragments and newly assigned fragments within the retained experts to reconstruct compact experts with fewer parameters but stronger expressive power. This invention performs fine-grained operations at the neuron level, effectively solving the inherent representational conflicts and misalignments among experts, significantly improving the performance of the compressed model, and providing reliable technical support for deploying large-scale SMoE models.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer system optimization technology, and particularly relates to a compression method for sparse expert hybrid (SMoE) architecture in large language models. Background Technology

[0002] The Sparse Expert Hybrid (SMoE) architecture is a key technology for building and scaling large-scale language models. It effectively reduces inference costs by dynamically selecting a small subset of expert subnetworks for each input token during inference, thereby significantly improving computational efficiency while maintaining or even surpassing the performance of dense models of the same size.

[0003] However, existing SMoE models still face significant challenges in deployment and application. First, although SMoE models activate only a small number of experts during inference, their complete parameter set, including parameters from all active and inactive experts, must be loaded into the computing device's memory. This results in extremely high memory consumption, placing enormous pressure on hardware resources and severely limiting their deployment capabilities in resource-constrained environments. Second, to address this issue, existing technologies primarily employ expert pruning or expert merging strategies. Direct pruning removes the entire expert, potentially leading to the permanent loss of useful knowledge and severely impairing model performance. Expert merging methods typically perform weighted averaging at a macroscopic expert level, ignoring the inherent structural misalignments and semantic conflicts between neurons within the expert. This coarse-grained merging approach cannot effectively integrate knowledge, limiting the expressive power of the compressed model. Therefore, a compression method that can effectively reduce the memory overhead of SMoE models, deeply integrate expert knowledge, and without compromising model performance is urgently needed. Summary of the Invention

[0004] This invention aims to provide a retraining-free pruning and recombination method for sparse expert hybrid large models, in order to solve the problems of large memory overhead of SMoE models, severe performance loss of traditional pruning methods, and neglect of neuronal-level structural misalignment in merging methods in the prior art.

[0005] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0006] In a first aspect, the present invention provides a method for retraining-free pruning and recombination of a sparse expert hybrid large model, which includes the following steps:

[0007] S1: Read all network parameters of the sparse expert mixture model into memory, and use a text dataset as a preset calibration dataset. Input each text sample in the preset calibration dataset into the sparse expert mixture model to perform a text question answering task, and collect the routing behavior of each expert in the sparse expert mixture model. Based on the routing behavior on the preset calibration dataset, calculate the importance score of each expert in the sparse expert mixture model, and retain the preset number of experts with the highest scores, while the remaining experts are removed.

[0008] S2: Decompose the projection matrix of each pruned expert into multiple independent neuron-level segments. Each segment is composed of vectors from each projection matrix. Add all segments of all pruned experts to the segment pool. Then calculate the local structural similarity between each segment and each retained expert. Reassign each segment in the segment pool that passes the lowest similarity threshold to the retained expert with the highest local structural similarity.

[0009] S3: Within each retained expert that has received a new segment, merge its original segment with the newly assigned segment, and use a clustering algorithm to cluster all segments to obtain the same number of cluster centers as the original segments. Reconstruct all cluster centers into the projection matrix of the corresponding retained expert. After completing the reconstruction of the projection matrices of all retained experts, a compressed sparse expert mixture model is obtained. The parameters of all retained experts are persistently stored according to the preset storage path and then the memory is released. Before executing the inference task, for each expert layer, only the parameters of all retained experts are read into memory to participate in the forward propagation process, thereby reducing memory usage.

[0010] As a preferred embodiment of the first aspect above, in step S1, the importance score of each expert is the expected value of the normalized routing weight of the expert when processing the calibration dataset, which is calculated by combining the expert's activation frequency and routing weight strength.

[0011] As a preferred embodiment of the first aspect above, in step S2, each expert in the sparse expert hybrid large model is composed of gated linear units (GLUs), which include a gated projection matrix W. g Upward projection matrix W u and the downlink projection matrix W d The i-th neuron-level segment of each expert is determined by the gated projection matrix W. g The transpose of the i-th row vector and the upward projection matrix W u The transpose of the i-th row vector and the downlink projection matrix W d The parameter triplet (w) formed by the i-th column vector g,i ,w u,i ,w d,i), i = 1, 2, ..., h, where h is the dimension of the intermediate hidden layer of the gated linear unit.

[0012] As a preferred embodiment of the first aspect above, in step S2, the method for calculating the local structural similarity between any segment and a retention expert is as follows: calculate the cosine similarity between the vectorized representation of the segment and the vectorized representation of each segment within the retention expert pairwise, and take the maximum value of the cosine similarity as the local structural similarity between the segment and the retention expert.

[0013] As a preferred embodiment of the first aspect above, the vectorization of each segment is obtained by sequentially concatenating the vectors in the parameter triplet of the segment into a column vector.

[0014] As a preferred embodiment of the first aspect, in step S2, when reassigning segments in the segment pool, it is necessary to first determine whether the maximum local structural similarity between each segment and all experts exceeds a preset minimum similarity threshold. Only when the maximum local structural similarity exceeds the minimum similarity threshold will the segment be assigned to the retained expert with the highest local structural similarity.

[0015] As a preferred embodiment of the first aspect above, in step S3, the clustering algorithm is a spherical weighted k-means clustering algorithm, and the weight of each segment during the clustering process is the importance score of the original expert to which the segment originally belonged.

[0016] Secondly, this invention provides a retraining-free pruning and recombination system for sparse expert hybrid large models, comprising:

[0017] The expert pruning module is used to read all network parameters of the sparse expert mixture model into memory, and use a text dataset as a preset calibration dataset. Each text sample in the preset calibration dataset is input into the sparse expert mixture model to perform a text question answering task, and the routing behavior of each expert in the sparse expert mixture model is collected. Based on the routing behavior on the preset calibration dataset, the importance score of each expert in the sparse expert mixture model is calculated, and the experts with the highest scores are retained, while the rest of the experts are pruned.

[0018] The segmentation and reorganization module is used to decompose the projection matrix of each pruned expert into multiple independent neuron-level segments. Each segment is composed of vectors from each projection matrix. All segments of all pruned experts are added to the segment pool. Then, the local structural similarity between each segment and each retained expert is calculated. Each segment in the segment pool that passes the lowest similarity threshold is reassigned to the retained expert with the highest local structural similarity.

[0019] The expert reconstruction module is used to merge the original segments and the newly assigned segments within each retained expert that has received a new segment, and to cluster all segments using a clustering algorithm to obtain the same number of cluster centers as the original segments. All cluster centers are then reconstructed into the projection matrix of the corresponding retained expert. After the reconstruction of the projection matrices of all retained experts is completed, a compressed sparse expert mixture model is obtained. The parameters of all retained experts are persistently stored according to a preset storage path and then the memory is released. Before executing the inference task, only the parameters of all retained experts are read into memory for each expert layer to participate in the forward propagation process, thereby reducing memory usage.

[0020] Thirdly, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the retraining pruning and recombination method for sparse expert hybrid large models as described in any of the first aspects above.

[0021] Fourthly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0022] The memory is used to store computer programs;

[0023] The processor is configured to, when executing the computer program, implement the retraining pruning and recombination method for sparse expert hybrid large models as described in any of the first aspects above.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] This invention proposes a SMoE model compression method based on neuron-level decomposition and recombination, which effectively solves the problems of large performance loss and neglect of microstructure in existing pruning and merging techniques. By treating experts as a collection of independently functioning neuron segments, this invention achieves fine-grained knowledge transfer across experts, breaking the rigid limitation of experts as indivisible units. The pruning strategy based on routing behavior accurately identifies redundant experts; the segmentation and recombination mechanism based on local structural similarity ensures the compatibility of knowledge transfer; and the expert reconstruction process based on weighted clustering generates information-dense, compact experts. This method requires no retraining, significantly reducing model memory usage and inference latency while effectively maintaining or even improving model performance on various benchmark tests, demonstrating superior compression efficiency and practical value. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method steps of the present invention;

[0027] Figure 2 This is a schematic diagram of the overall architecture of the sparse expert hybrid model compression method proposed in this invention;

[0028] Figure 3 This is a schematic diagram of the system module composition of the present invention;

[0029] Figure 4 This is a schematic diagram of the computer electronic device structure of the present invention. Detailed Implementation

[0030] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in various embodiments of the present invention can be combined accordingly without mutual conflict.

[0031] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0032] 1. Large-scale language models

[0033] Natural language processing (NLP) refers to a large-scale neural network model based on deep learning, particularly the Transformer architecture. These models are trained on massive amounts of text data to learn complex linguistic patterns, semantic relationships, and world knowledge, thereby possessing powerful natural language understanding and generation capabilities. A significant characteristic is their enormous number of parameters, typically reaching billions or even trillions, which presents challenges for efficient deployment.

[0034] 2. Pruning

[0035] This refers to a compression technique aimed at reducing the complexity of neural network models. This technique systematically identifies and removes redundant or low-contribution components from the model, such as individual weights, neurons, or structured modules like experts, thereby reducing the model's storage size, memory footprint, and inference computation load without excessively impacting model performance.

[0036] 3. Expert Mix

[0037] This refers to a neural network architecture whose core idea is to replace a single, dense network layer (such as a feedforward network layer) with a set of parallel subnetworks that are functionally similar but have independent parameters, i.e., "experts".

[0038] 4. Sparse expert hybrid

[0039] SMoE refers to a variant of the MoE architecture, characterized by the introduction of a gating network, also known as a router. This router dynamically and sparsely selects one or a few experts to participate in the computation based on the features of each input data (such as a lexical unit). This sparse activation mechanism allows the model to maintain low computational cost per inference even with a huge number of total parameters. However, its deployment still requires loading the parameters of all experts into memory, resulting in high memory overhead. Large language models based on the SMoE architecture are called Sparse Expert Hybrid Large Models, and may also be referred to as SMoE models.

[0040] 5. Experts

[0041] In a MoE or SMoE architecture, an expert is a single computational unit that constitutes a set of parallel subnetworks. In the large-scale Transformer-based language model involved in this invention, each expert is typically an independent Feed-Forward Network (FFN) responsible for performing nonlinear transformations on the input lexical representations.

[0042] 6. Feedforward Networks

[0043] A Freeform Neural Network (FFN) refers to a neural network layer in which information flows unidirectionally without loops. In the Transformer architecture, an FFN typically consists of two linear transformation layers and a nonlinear activation function, used to enhance the model's representational power. In the SMoE architecture, each expert is essentially an FFN.

[0044] 7. Gated Linear Unit

[0045] This refers to an advanced network layer structure commonly used in the expert layers of modern large-scale language models. A standard GLU expert layer consists of three trainable weight matrices: a gated projection matrix W... g Upward projection matrix W u and the downlink projection matrix W d Its computational method allows the output to be decomposed into the sum of multiple functionally independent transformations dominated by a single hidden dimension. This characteristic is the structural basis for the "expert segmentation" defined in this invention.

[0046] 8. Expert segmentation

[0047] This invention defines a functionally independent minimum computational unit for fine-grained decomposition and recombination of experts. Specifically, for experts employing a GLU architecture, an expert segment is formed by a gated projection matrix W. g The transpose of the i-th row vector and the upward projection matrix W u The transpose of the i-th row vector and the downlink projection matrix W dThe parameter triple (w) formed by the i-th column vector g,i ,w u,i ,w d,i ), where i is the index of the segment. Each segment is functionally self-contained, and its parameter updates are localized on the gradient.

[0048] 9. Routing behavior

[0049] This refers to the decision-making process of the gated network (routers) in the SMoE architecture. This process determines which experts to activate based on input characteristics and assigns a weight to the output of each selected expert. In this invention, routing behavior is used to evaluate the importance of each expert, serving as the basis for the expert pruning step decision.

[0050] Based on the above explanation of the nouns and terms involved in the embodiments of this application, the implementation scenarios of the present invention will be described next.

[0051] To address the shortcomings of existing SMoE model compression techniques, this invention proposes a retraining-free pruning and recombination method for large sparse expert hybrid models. The specific steps are as follows: Figure 1 As shown, the method includes steps S1 to S3. This method is a solution combining expert pruning, neuron decomposition, segmented recombination, and clustering reconstruction. Its overall architecture can be found in [reference needed]. Figure 2 The following is combined with Figure 1 and Figure 2 The specific steps and procedures of this method are described in detail.

[0052] S1. Expert pruning based on routing behavior

[0053] Before performing the pruning task, all network parameters of the sparse expert mixture model to be pruned need to be loaded into memory. This includes all expert parameters of all expert layers and other non-expert layer parameters (such as router network parameters, parameters of all Transformer layers, and parameters of input and output layers, etc.). Simultaneously, a text dataset is obtained as a pre-defined calibration dataset. The text dataset is not limited; for example, the C4 (Colossal Clean Crawled Corpus) dataset can be used. Each text sample in the pre-defined calibration dataset is segmented and converted into tokens, then input into the sparse expert mixture model (SMoE model, such as Mixtral-8x7B, Qwen2-57B, DeepSeek-MoE-16B). The routing behavior of each expert in the SMoE model is collected. Then, based on the routing behavior collected on the pre-defined calibration dataset, the importance score of each expert in the sparse expert mixture model is calculated. A predetermined number of experts with the highest scores are retained, and the remaining experts are pruned. The retained experts are called retained experts, and the pruned experts are called pruned experts.

[0054] In this step, redundant experts are identified and pruned by analyzing the routing behavior of the SMoE model when processing a small calibration dataset. Each SMoE layer in the SMoE model contains N experts ε={E1,…,E N} and a router G: The router outputs routing weights for each input token's feature vector x, and selects activated experts based on these routing weights according to the routing mechanism. The routing mechanism of this invention employs a soft routing mechanism; therefore, the router outputs soft routing weights G(x) = (G1(x), ..., G...). N (x)). Given a calibration set make Indicates router is x m The selected Top-k expert index set allows us to calculate the importance scores of the experts.

[0055] Specifically, the present invention is for each expert E i Calculate its importance score S i This score is defined as the expert's performance in processing calibration datasets. At that time, the expected value of its normalized route weight is:

[0056]

[0057] in:

[0058] For indicator functions, when the condition The value is 1 when it is true, and 0 otherwise.

[0059] This indicates that the router is the input x. m The set of indices for the selected Top-k experts.

[0060] G i (x m ) is the router for input x m Assigned to expert E i The weight value.

[0061] The importance score S i It takes into account the activation frequency of experts (whether they are selected). ) and activation intensity (G i (x m The importance score (or weight) of all experts can effectively measure their contribution in real-world application scenarios. Subsequently, this invention selects the Top-K experts with the highest importance scores as retained experts, while the remaining NK experts are listed as pruned experts. Here, K is the preset number of retained experts, a parameter that can be adjusted according to actual needs.

[0062] S2. Expert Decomposition and Fragment Reconstruction

[0063] The projection matrix of each pruned expert is decomposed into multiple independent neuron-level segments, each segment consisting of vectors from each projection matrix. All segments of all pruned experts are added to the segment pool. Then, the local structural similarity between each segment and each retained expert is calculated. Each segment in the segment pool that passes the minimum similarity threshold is reassigned to the retained expert with the highest local structural similarity.

[0064] This step aims to decompose the parameters of pruned experts into independently functioning neuron-level fragments instead of discarding them directly. These fragments can be selectively assigned to compatible retained experts. The fragments from pruned experts are pooled into a fragment pool and redistributed based on local structural similarity. This method achieves neuron-level knowledge transfer while preserving the functional coherence of experts.

[0065] S21. Expert-to-Piece Decomposition: In the SMoE model applied in this invention, each expert adopts a gated linear unit (GLU) architecture, which is represented by three learnable weight matrices, namely: gated projection matrix. Upward projection matrix and downlink projection matrix Where d is the input dimension and h is the dimension of the intermediate hidden layer. Assuming the current input token is x, the forward computation process in GLU is represented as:

[0066] f(x) = Wd (σ(W g x)⊙(W u x))

[0067] Where σ is the activation function and ⊙ represents element-wise multiplication. The formula for this forward computation process can be rewritten as the sum of the independent contributions of each hidden dimension:

[0068]

[0069] in, and They are matrices W g and W u The i-th row, w d,i It is matrix W d The i-th column.

[0070] Based on the structural independence revealed by the rewritten formula, this invention defines an expert segment as the smallest self-contained computational unit composed of parameter triples. The i-th neuron-level segment is seg... i The triplet parameters are the gated projection matrix W. g The transpose of the i-th row vector and the upward projection matrix W u The transpose of the i-th row vector and the downlink projection matrix W d The i-th column vector, i = 1, 2, ..., h, that is, the i-th segment seg i Defined as:

[0071] seg i =(w g,i ,w u,i ,w d,i )

[0072] The segmentation defined above possesses three core characteristics suitable for recombination: functional independence (each segment only affects a single output dimension), gradient locality (the gradient depends only on its own parameters), and structural regularity (triples can be flattened into fixed-dimensional vectors for easy comparison of parameter space similarity). Based on this, this invention extracts all segments of all pruned experts to form a global segmentation pool. Awaiting reallocation.

[0073] S22. Segmentation and Reorganization Based on Local Structural Similarity:

[0074] In order to segment the pool The segments in the data are selectively assigned to the most suitable retention experts. This invention employs a matching scheme based on structural similarity. First, each segment is seg... i Vectorization is performed by concatenating the vectors in the parameter triplet in order to obtain the column vector v(seg).i ):

[0075]

[0076] Segmentation i With a certain preservation expert E r The local structural similarity sim is defined as the relationship between the segment and E. r The maximum cosine similarity between the vectorized representations of all existing segments is calculated by first calculating the cosine similarity between each pair of the vectorized representations of the segment and the vectorized representations of each segment within the preserved expert, and then taking the maximum cosine similarity as the local structural similarity between the segment and the preserved expert. For any segment seg i and preservation expert E r The local structural similarity sim between the two can be expressed by the following formula:

[0077]

[0078] in It is expert E r The existing set of segments.

[0079] For each segment seg i We need to find the highest value sim from the local structural similarity sim between this segment and all retained experts. max The corresponding experts who retain their expertise are denoted as Segmentation i They will be reassigned to the retention experts whose local structural similarity to sim is the highest. However, only when the local structural similarity value sim is at its highest... max Assignment is only performed when the similarity exceeds a preset minimum threshold α∈[0,1]. If a segment seg... i The highest value of local structural similarity among all experts (sim) max If the threshold still does not exceed α, the segment is discarded and will not be assigned to any expert. This threshold mechanism is used to filter out structurally incompatible segments.

[0080] S3. Expert Reconstruction through Segmented Clustering

[0081] After the S2 recombination step, the number of internal segments in each retained expert increases, including its original segments and the newly absorbed segments. To reduce redundancy while preserving knowledge diversity, this step uses a clustering algorithm to compress the expanded segment set into a smaller, more semantically coherent, representative set of neurons. Specifically, within each retained expert that has received a new segment, its original segments are merged with the newly assigned segments, and a clustering algorithm is used to cluster all segments, obtaining cluster centers with the same number of segments as the original. All cluster centers are then reconstructed into the projection matrix of the corresponding retained expert. After reconstructing the projection matrices of all retained experts, a compressed sparse expert mixture model is obtained. The parameters of all retained experts are persistently stored according to a preset storage path before memory is released. Before executing the inference task, for each expert layer, only the parameters of all retained experts are read into memory to participate in the forward propagation process, thereby reducing memory usage.

[0082] In a preferred embodiment, the above clustering algorithm employs a spherically weighted k-means clustering algorithm. For each retained expert, the algorithm aims to minimize the following clustering objective function based on weighted cosine distance:

[0083]

[0084] in, It is the set of cluster centers. The number of clusters k during clustering needs to be the same as the number of segments originally retained by the experts, that is, k is the dimension h of the intermediate hidden layer. v(seg) represents the set of segments that are clustered into the j-th cluster; i ) is a segmentation (seg) i The vectorized form of w i It is the weight reflecting the importance of the segment seg, and its value is the importance score S of the original expert to which the segment belongs. i .

[0085] To stabilize the iterative process, this invention introduces norm equalization before updating the cluster centers. For each segment in each cluster j... Its vector v k First it is scaled:

[0086]

[0087] This operation unifies the norm of all piecewise vectors within a cluster to the average norm of the cluster. Subsequently, each cluster center c j The update is achieved by normalizing and weighting the equalized vectors within the cluster:

[0088]

[0089] After clustering convergence, the final set of cluster centers {c1,c2,…,c} is obtained. k This constitutes the new compressed expert. Each cluster center vector c j It is reverse-converted into a new parameter triple, thereby allowing the new projection matrix in the GLU to be reconstructed by reversing the process of decomposing the projection matrix into parameter triples. This results in a compact and efficient expert representation with fewer parameters, but which retains its expressive power through knowledge fusion, while reducing memory and latency.

[0090] It should be noted that after the reconstruction is completed and a compressed sparse expert mixture model is obtained, the network parameters of all retained experts (the three reconstructed projection matrices) need to be saved to an external storage device (such as a hard drive) according to the actual specified storage path, while the network parameters in memory can be released. Before executing the inference task, for each expert layer, only the projection matrix parameters of all retained experts are read into memory to participate in the forward propagation process, thereby reducing memory usage. Of course, in addition to reading the network parameters of the retained experts in the expert layers, the parameters of other non-expert layers also need to be read into memory. The only difference from the parameter reading operation in step S1 is that the network parameters of the pruned experts no longer need to be read.

[0091] Therefore, through the above technical solution, the present invention achieves efficient compression of the SMoE model without retraining, and the number of parameters that need to be read into memory is greatly reduced.

[0092] The present invention will now demonstrate the application effect of the non-retraining pruning and recombination method for sparse expert hybrid large models described in S1 to S3 of the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0093] Example

[0094] To verify the effectiveness of the method described in S1 to S3 of this invention (hereinafter referred to as the method of this invention), this embodiment conducted experiments on several publicly available large-scale language models (including Mixtral-8x7B, Qwen2-57B, and DeepSeek-MoE-16B) and several standard benchmarks (including the commonsense reasoning dataset and MMLU). All experiments were performed on 2×NVIDIA H100 GPUs.

[0095] To achieve task-independent pruning, this embodiment uses the C4 dataset as the preset calibration dataset. 128 sequences (2048 tokens each) randomly sampled from the C4 corpus are used to collect the routing behavior of each expert in the sparse expert mixture model, which is then used to calculate the importance score of the experts.

[0096] Furthermore, this invention employs the LLM-Prune evaluation protocol to perform zero-shot evaluations on commonsense reasoning datasets: BoolQ, PIQA, HellaSwag, ARC-e and ARC-c, OpenbookQA, and WinoGrande. It also utilizes small-shot hints from MMLU to evaluate multi-domain reasoning capabilities.

[0097] All tasks are set to be evaluated based on generated data. The model is tuned to directly output the answer, and correctness is determined by matching template strings. The hint format and matching criteria follow OpenCompass, and the inference process uses vLLM.

[0098] In addition, this embodiment sets up a series of baseline methods for comparison. Specifically, this embodiment compares a series of representative structured sparse hybrid expert model pruning methods: NAEE (Expert Pruning), MC-SMoE (Expert Merging), and LLM-Pruner (General Structured Pruning). For NAEE on Qwen2 and DeepSeek, this embodiment approximates expert selection by sampling 10k combinations per layer and selecting the scheme with the smallest output bias. To maintain fairness under the retraining-free constraint, intra-expert compression is disabled in this embodiment and it is denoted as M-SMoE. LLM-Pruner is evaluated using a data- and gradient-based Taylor version. All methods are compared fairly under the retraining-free setting.

[0099] The experimental results of this invention and baseline methods on commonsense question answering and multi-domain benchmarks are shown in Tables 1 and 2. On the Mixtral model, when pruning 25% (retaining 6 experts) and 50% (retaining 4 experts) of the experts, the method of this invention significantly outperforms other baseline methods in both the average score and MMLU of commonsense reasoning, with far less performance loss. In particular, under a 6x7B setting, the performance of the method of this invention is very close to that of the original 8x7B model. On the more lightweight DeepSeek-MoE and the larger-scale Qwen2 model, the method of this invention also exhibits optimal performance, with almost no performance loss after pruning.

[0100] Table 1 Performance evaluation of the present invention on Mixtral-8x7B and Qwen2-57B models.

[0101]

[0102] Table 2 Performance evaluation of the present invention on the DeepSeek-MoE-16b-Chat model

[0103]

[0104] Table 3 shows the performance of the models obtained before and after pruning in terms of inference acceleration and memory consumption. Table 3 illustrates the inference efficiency before and after vLLM pruning (evaluated using vLLM on 1024 C4 samples under 64-way concurrency). It is evident that reducing the number of experts from 8 to 6 and then to 4 significantly improves throughput and latency, with the highest token throughput increase of 38% and the first token response time (TTFT) reduced by over 900ms. This acceleration effect mainly stems from kernel-level parallel optimization, where fewer experts allow more tokens to be routed to active experts, resulting in more efficient large-scale matrix multiplication. Therefore, the method of this invention can significantly reduce memory consumption and support larger batch processing scales. Notably, the 6-expert variant achieves near-perfect accuracy while reducing memory consumption by 25% and increasing throughput by 18%, demonstrating that the method of this invention effectively balances redundancy and speed.

[0105] Table 3. Inference efficiency of Mixtral before and after expert pruning

[0106]

[0107] In summary, the experiments demonstrate that the method of this invention exhibits significant advantages under different models and compression ratios, and possesses stronger performance preservation capabilities, robustness, and practical application potential.

[0108] It should be noted that the method steps shown in S1 to S3 above can essentially be implemented in the form of computer programs or software functional modules.

[0109] Therefore, based on the same inventive concept, such as Figure 3 As shown, the present invention also provides a system for retraining and recombining sparse expert hybrid large models, corresponding to the retraining-free pruning and recombination method for sparse expert hybrid large models provided in the above embodiments, comprising:

[0110] The expert pruning module is used to read all network parameters of the sparse expert mixture model into memory, and use a text dataset as a preset calibration dataset. Each text sample in the preset calibration dataset is input into the sparse expert mixture model to perform a text question answering task, and the routing behavior of each expert in the sparse expert mixture model is collected. Based on the routing behavior on the preset calibration dataset, the importance score of each expert in the sparse expert mixture model is calculated, and the experts with the highest scores are retained, while the rest of the experts are pruned.

[0111] The segmentation and reorganization module is used to decompose the projection matrix of each pruned expert into multiple independent neuron-level segments. Each segment is composed of vectors from each projection matrix. All segments of all pruned experts are added to the segment pool. Then, the local structural similarity between each segment and each retained expert is calculated. Each segment in the segment pool that passes the lowest similarity threshold is reassigned to the retained expert with the highest local structural similarity.

[0112] The expert reconstruction module is used to merge the original segments and the newly assigned segments within each retained expert that has received a new segment, and to cluster all segments using a clustering algorithm to obtain the same number of cluster centers as the original segments. All cluster centers are then reconstructed into the projection matrix of the corresponding retained expert. After the reconstruction of the projection matrices of all retained experts is completed, a compressed sparse expert mixture model is obtained. The parameters of all retained experts are persistently stored according to a preset storage path and then the memory is released. Before executing the inference task, only the parameters of all retained experts are read into memory for each expert layer to participate in the forward propagation process, thereby reducing memory usage.

[0113] Furthermore, based on the same inventive concept, such as Figure 4 As shown, the present invention also provides a computer electronic device corresponding to the non-retraining pruning and recombination method for a sparse expert hybrid large model provided in the above embodiments, which includes a memory and a processor;

[0114] The memory is used to store computer programs;

[0115] The processor is configured to implement the retraining-free pruning and recombination method for sparse expert hybrid large models as described above when executing the computer program.

[0116] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0117] Therefore, based on the same inventive concept, this invention provides a computer-readable storage medium corresponding to the method for retraining and reorganizing a sparse expert hybrid large model without repetition. The storage medium stores a computer program, which, when executed by a processor, can realize the method for retraining and reorganizing a sparse expert hybrid large model without repetition as described above.

[0118] Therefore, based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can realize the retraining pruning and recombination method for sparse expert hybrid large models as described above.

[0119] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by a processor, which can perform the aforementioned steps S1 to S3.

[0120] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0121] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0122] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0123] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for repetitive training pruning and recombination of a sparse expert hybrid large model, characterized in that, Includes the following steps: S1: Read all network parameters of the sparse expert hybrid model into memory, and use a text dataset as a preset calibration dataset. Input each text sample in the preset calibration dataset into the sparse expert hybrid model to perform a text question answering task, and collect the routing behavior of each expert in the sparse expert hybrid model. Based on the routing behavior on the preset calibration dataset, the importance score of each expert in the sparse expert mixture model is calculated, and the preset number of experts with the highest scores are retained, while the rest of the experts are removed. S2: Decompose the projection matrix of each pruned expert into multiple independent neuron-level segments. Each segment is composed of vectors from each projection matrix. Add all segments of all pruned experts to the segment pool. Then, the local structural similarity between each segment and each retention expert is calculated, and each segment in the segment pool that passes the lowest similarity threshold is reassigned to the retention expert with the highest local structural similarity. S3: Within each retained expert that has received a new segment, merge its original segment with the newly assigned segment, and use a clustering algorithm to cluster all segments to obtain the same number of cluster centers as the original segments. Reconstruct all cluster centers into the projection matrix of the corresponding retained expert. After completing the reconstruction of the projection matrices of all retained experts, a compressed sparse expert mixture model is obtained. The parameters of all retained experts are persistently stored according to the preset storage path and then the memory is released. Before executing the inference task, for each expert layer, only the parameters of all retained experts are read into memory to participate in the forward propagation process, thereby reducing memory usage.

2. The method for retraining-free pruning and recombination of sparse expert hybrid large models as described in claim 1, characterized in that, In step S1, the importance score of each expert is the expected value of the normalized routing weight of that expert when processing the calibration dataset, which is calculated by combining the expert's activation frequency and routing weight strength.

3. The method for retraining and pruning / recombining sparse expert hybrid large models as described in claim 1, characterized in that, In step S2, each expert in the sparse expert hybrid large model is composed of gated linear units (GLUs), which contain a gated projection matrix W. g Upward projection matrix W u and the downlink projection matrix W d The i-th neuron-level segment of each expert is determined by the gated projection matrix W. g The transpose of the i-th row vector and the upward projection matrix W u The transpose of the i-th row vector and the downlink projection matrix W d The parameter triplet (w) formed by the i-th column vector g,i ,w u,i ,w d,i ), i = 1, 2, ..., h, where h is the dimension of the intermediate hidden layer of the gated linear unit.

4. The method for retraining and pruning / recombining a sparse expert hybrid large model as described in claim 1, characterized in that, In step S2, the method for calculating the local structural similarity between any segment and a retention expert is as follows: calculate the cosine similarity between the vectorized representation of the segment and the vectorized representation of each segment within the retention expert pairwise, and take the maximum value of the cosine similarity as the local structural similarity between the segment and the retention expert.

5. The method for retraining and pruning / recombining a sparse expert hybrid large model as described in claim 4, characterized in that, The vectorization of each segment is obtained by concatenating the vectors in the parameter triplet of that segment in sequence to form a column vector.

6. The method for retraining and pruning / recombining a sparse expert hybrid large model as described in claim 4, characterized in that, In step S2, when reassigning segments in the segment pool, it is necessary to first determine whether the maximum local structural similarity between each segment and all experts exceeds the preset minimum similarity threshold. Only when the maximum local structural similarity exceeds the minimum similarity threshold will the segment be assigned to the retained expert with the highest local structural similarity.

7. The method for retraining and pruning / recombining a sparse expert hybrid large model as described in claim 1, characterized in that, In step S3, the clustering algorithm is a spherical weighted k-means clustering algorithm, and the weight of each segment during the clustering process is the importance score of the original expert to which the segment originally belonged.

8. A repetitive training pruning and recombination system for sparse expert hybrid large models, characterized in that, include: The expert pruning module is used to read all the network parameters of the sparse expert mixture model into memory, and use a text dataset as a preset calibration dataset. Each text sample in the preset calibration dataset is input into the sparse expert mixture model to perform a text question answering task, and the routing behavior of each expert in the sparse expert mixture model is collected. Based on the routing behavior on the preset calibration dataset, the importance score of each expert in the sparse expert mixture model is calculated, and the preset number of experts with the highest scores are retained, while the rest of the experts are removed. The segmentation and reorganization module is used to decompose the projection matrix of each pruned expert into multiple independent neuron-level segments. Each segment is composed of vectors in each projection matrix. All segments of all pruned experts are added to the segment pool. Then, the local structural similarity between each segment and each retention expert is calculated, and each segment in the segment pool that passes the lowest similarity threshold is reassigned to the retention expert with the highest local structural similarity. The expert reconstruction module is used to merge the original segments and the newly assigned segments within each retained expert that has received a new segment, and to cluster all segments using a clustering algorithm to obtain the same number of cluster centers as the original segments. All cluster centers are then reconstructed into the projection matrix of the corresponding retained expert. After the reconstruction of the projection matrices of all retained experts is completed, a compressed sparse expert mixture model is obtained. The parameters of all retained experts are persistently stored according to a preset storage path and then the memory is released. Before executing the inference task, only the parameters of all retained experts are read into memory for each expert layer to participate in the forward propagation process, thereby reducing memory usage.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the retraining-free pruning and recombination method for sparse expert hybrid large models as described in any one of claims 1 to 7.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the retraining-free pruning and recombination method for sparse expert hybrid large models as described in any one of claims 1 to 7.

Images