Hybrid expert model structured pruning and acceleration method and system based on micro expert sorting

Abstract

The invention discloses a hybrid expert model structured pruning and acceleration method and system based on micro expert sorting, belongs to the technical field of large language models, and solves the problems that an existing MoE pruning method cannot consider the requirements of fine-grained pruning, reasoning acceleration and structural analysis generalization at the same time, coarse-grained expert-level pruning damages the performance of a model, and the reliability of the model is poor. And fine-grained compression lacks speed increase and lacks a unified micro-analysis method. The method comprises the following steps: splitting each expert network in a hybrid expert model into a plurality of micro experts, and modeling the plurality of micro experts, so that the micro experts in different expert networks have comparability; sorting all the micro experts according to the energy indexes of the micro experts by adopting a micro expert sorting algorithm; and processing the sorted micro-experts by adopting a pruning algorithm, selecting a core micro-expert for reservation, and directly deleting the rest of the micro-experts. The method is suitable for application scenes such as edge calculation and multi-task learning.

Description

Technical Field

[0001] This invention belongs to the field of large language model technology, specifically relating to a hybrid expert model structured pruning acceleration method. Background Technology

[0002] As the parameter size of Large Language Models (LLMs) continues to expand, inference overhead has become a major bottleneck in their practical deployment. Sparse structures such as Mixture-of-Experts (MoEs) effectively reduce computation during the inference phase by activating only a subset of expert networks, and are currently one of the mainstream acceleration methods. However, MoE structures still suffer from significant parameter redundancy, particularly the extremely low parameter utilization within each expert network. Existing MoE pruning methods have the following problems: Coarse-grained operations mask the differences in the internal structure of experts, resulting in low pruning precision; The inference speed after pruning was not significantly improved, limiting its practical deployment value. The lack of a unified method for microstructure analysis hinders the generalization of model compression strategies.

[0003] Therefore, how to further compress the MoE structure to reduce resource consumption while ensuring performance is a pressing issue. Existing methods mainly focus on two categories: 1. Expert-level pruning: Some methods coarsely remove entire expert groups by statistically analyzing expert activation frequency or importance metrics. This method is simple to operate, but due to the strong heterogeneity within experts, it often leads to the unintended pruning of effective micro-experts, affecting model performance.

[0004] 2. Expert merging into compression methods: For example, D2MoE achieves compression through SVD decomposition of the expert layer matrix, but the complete parameters need to be restored during inference, resulting in no significant improvement in decoding speed.

[0005] In summary, existing methods struggle to balance pruning granularity, pruning efficiency, and inference acceleration. Summary of the Invention

[0006] This invention aims to propose a fine-grained, high-performance, and inference-friendly pruning method. Its purpose is to address the problems that existing MoE pruning methods cannot simultaneously meet the needs of fine-grained pruning, inference acceleration, and generalization of structural analysis. Coarse-grained expert-level pruning damages model performance, fine-grained compression lacks speed improvement, and there is a lack of a unified microscopic analysis method.

[0007] The present invention proposes a structured pruning and acceleration method for hybrid expert models based on micro-expert ranking, comprising: S1: Decompose each expert network in the hybrid expert model into several micro-experts, model several micro-experts, and make the micro-experts of different expert networks comparable; S2: Employ the micro-expert ranking algorithm to rank all micro-experts based on their energy index; S3: Use a pruning algorithm to process the sorted micro-experts, select the core micro-experts to keep, and delete the rest directly.

[0008] Furthermore, a preferred embodiment is provided: the micro-expert is a linear mapping unit of the feedforward substructure of each expert network.

[0009] Furthermore, a preferred solution is provided: In S1, the modeling formula for the micro-expert is: , in, It is the input hidden state, where, It is the weight matrix of the i-th micro-expert. It is an activation function.

[0010] Furthermore, a preferred embodiment is provided: S2 includes: S21: Integrate the activation values ​​and parameter weights of micro-experts to construct a ranking standard based on energy indicators; S22: Determine the balance coefficient, weight and fuse the two types of information, and perform a global ranking of all micro-experts based on the energy index to obtain the importance ranking of micro-experts during model decoding.

[0011] Furthermore, a preferred solution is provided: In S22, the balance coefficient is the value that optimizes the model performance on the test task; the two types of information are: the overall activation value and the maximum activation value.

[0012] Furthermore, a preferred solution is provided: the energy index is defined as follows: , in, Let represent the combination coefficient vector of the i-th micro-expert.

[0013] Furthermore, a preferred solution is provided: in S3, the core micro-experts are the top k% of micro-experts, and the value of k ranges from 40 to 80.

[0014] The present invention also proposes a computer device, the computer device including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a hybrid expert model structured pruning and acceleration method based on micro-expert ranking, according to any one or more of the above-described schemes.

[0015] The present invention also proposes a computer-readable storage medium for storing a computer program that executes a hybrid expert model structured pruning and acceleration method based on micro-expert ranking, as described in any one or more of the above-described schemes.

[0016] This invention also proposes a hybrid expert model structured pruning and acceleration system based on micro-expert ranking. The system is implemented using a hybrid expert model structured pruning and acceleration method based on micro-expert ranking, as described in any one or more of the above-mentioned schemes. The system includes: Model splitting module: This module is used to split each expert network in the hybrid expert model into several micro-experts, model these micro-experts, and make the micro-experts of different expert networks comparable. Model ranking module: used to rank all micro-experts according to the energy index of the micro-experts based on the micro-expert ranking algorithm; Model pruning module: This module uses a pruning algorithm to process the sorted micro-experts, selecting and retaining the core micro-experts while deleting the rest.

[0017] Compared with the prior art, the advantages of the present invention are: The method proposed in this invention establishes a unified "micro-expert" perspective across experts to perform unified modeling of the MoE structure.

[0018] The method described in this invention constructs a micro-expert ranking framework and ranks micro-experts based on global importance indicators.

[0019] This invention proposes an efficient pruning strategy that supports the direct removal of redundant micro-experts, compressing parameters while accelerating decoding.

[0020] Experiments have verified that the method proposed in this invention, when applied to models such as DeepSeek-MoE-16B, achieves almost no performance loss after pruning 20% ​​of parameters, while significantly improving decoding speed. These results have been validated on multiple open datasets, and their preservation accuracy has been quantified using cosine similarity and L2 norm.

[0021] This invention is applicable to application scenarios such as edge computing and multi-task learning. Attached Figure Description

[0022] Figure 1 This is a flowchart of a structured pruning and acceleration method for a hybrid expert model based on micro-expert ranking, as described in a specific embodiment of the present invention. Detailed Implementation

[0023] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0024] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0028] Implementation Method 1: Reference Figure 1 This implementation method is described below.

[0029] A structured pruning and acceleration method for hybrid expert models based on micro-expert ranking includes: S1: Each expert network in the hybrid expert model is split into several micro-experts, and each micro-expert is a linear mapping unit of the feedforward substructure of each expert network; several micro-experts are modeled to make the micro-experts of different expert networks comparable; S2: Employ the micro-expert ranking algorithm to rank all micro-experts based on their energy index; S21: Integrate the activation values ​​and parameter weights of micro-experts to construct a ranking standard based on energy indicators; S22: Determine the balance coefficient, weight and fuse the two types of information, and perform a global ranking of all micro-experts in conjunction with the energy index to obtain the importance ranking of micro-experts during model decoding; the balance coefficient is the value that optimizes the model performance on the test task; the two types of information are: overall activation value and maximum activation value.

[0030] S3: A pruning algorithm is used to process the ranked micro-experts, selecting core micro-experts for retention and deleting the rest. The core micro-experts are the top k% of the micro-experts, with k ranging from 40 to 80. In this step, k is typically set to 40-80% of the total number of micro-experts; the larger the value, the smaller the performance loss of the pruned model.

[0031] Specifically: Each expert in the hybrid expert model is represented as: (1) Among them, It is the input hidden state. It is the weight matrix of the i-th expert in the hybrid expert layer. It is an activation function.

[0032] Given the combination coefficient A for each expert, the output of the hybrid expert layer is: A linear combination of experts: (2) At this point, the weight matrix , Split by line, Split by column to get These weight vectors constitute the micro-expert, represented as: (3) in, It is the weight matrix of the i-th micro-expert.

[0033] Extract all scalars from the above process and define them as follows: , represented as: (4) in, It is the combination coefficient of each micro-expert, which is equal to the combination coefficient A of the expert to which it belongs.

[0034] Based on this, the hybrid micro-expert representation of the hybrid expert model is obtained as follows: (5) in, This represents the total number of micro-experts.

[0035] This linear combination of vectors can be viewed as a matrix product. Separating the coefficient matrix and the weight vector set, it can be written in the following form: (6) Based on this form, the energy of each micro-expert is defined as the product of the norm of its combined coefficient vector and the norm of its corresponding weight vector, which is also known as the energy index, expressed as: (7) in, Let represent the combination coefficient vector of the i-th micro-expert.

[0036] This energy index is used as the ranking reference value for micro-experts in S21, that is, the ranking standard based on the energy index. After ranking based on energy, all micro-experts are ranked according to their importance.

[0037] The method described in this embodiment is based on a micro-expert perspective and can directly optimize the output of the hybrid expert layer, approximating the original model. In contrast, existing methods optimize individual substructures of the model, i.e., approximate experts or approximate weight matrices, and cannot approximate the original model as effectively as the method proposed in this embodiment.

[0038] Table 1. Changes in evaluation indicators after 20% pruning compared to the original model.

[0039] The pruning algorithm proposed in this embodiment does not rely on gradients or retraining, quickly compresses the parameters of the MoE layer, and the deleted MoE layer maintains structural integrity. Evaluation and pruning are completed with only a very small amount of general data without the need for additional labeled data. No recovery is required during inference, thus truly accelerating decoding.

[0040] Implementation Method 2 A hybrid expert model structured pruning and acceleration system based on micro-expert ranking is disclosed. The system is implemented using the hybrid expert model structured pruning and acceleration method based on micro-expert ranking as described in Embodiment 1. The system includes: Model splitting module: This module is used to split each expert network in the hybrid expert model into several micro-experts, model these micro-experts, and make the micro-experts of different expert networks comparable. Model ranking module: used to rank all micro-experts according to the energy index of the micro-experts based on the micro-expert ranking algorithm; Model pruning module: This module uses a pruning algorithm to process the sorted micro-experts, selecting and retaining the core micro-experts while deleting the rest.

Claims

1. A structured pruning and acceleration method for hybrid expert models based on micro-expert ranking, characterized in that, The method includes: S1: Decompose each expert network in the hybrid expert model into several micro-experts, model several micro-experts, and make the micro-experts of different expert networks comparable; S2: Employ the micro-expert ranking algorithm to rank all micro-experts based on their energy index; S3: Use a pruning algorithm to process the sorted micro-experts, select the core micro-experts to keep, and delete the rest directly.

2. The structured pruning and acceleration method for a hybrid expert model based on micro-expert ranking as described in claim 1, characterized in that, The micro-expert is a linear mapping unit of the feedforward substructure of each expert network.

3. The structured pruning and acceleration method for hybrid expert models based on micro-expert ranking according to claim 1, characterized in that, In S1, the modeling formula for the micro-expert is: ， in, It is the input hidden state, where, It is the weight matrix of the i-th micro-expert. It is an activation function.

4. The structured pruning and acceleration method for a hybrid expert model based on micro-expert ranking as described in claim 1, characterized in that, S2 includes: S21: Integrate the activation values ​​and parameter weights of micro-experts to construct a ranking standard based on energy indicators; S22: Determine the balance coefficient, weight and fuse the two types of information, and perform a global ranking of all micro-experts based on the energy index to obtain the importance ranking of micro-experts during model decoding.

5. The structured pruning and acceleration method for a hybrid expert model based on micro-expert ranking according to claim 4, characterized in that, In S22, the balance coefficient is the value that optimizes the model's performance on the test task; the two types of information are: the overall activation value and the maximum activation value.

6. The structured pruning and acceleration method for a hybrid expert model based on micro-expert ranking according to claim 1, characterized in that, The energy index is defined as follows: ， in, Let represent the combination coefficient vector of the i-th micro-expert.

7. The structured pruning and acceleration method for a hybrid expert model based on micro-expert ranking according to claim 1, characterized in that, In S3, the core micro-experts are the top k% of micro-experts, and the value of k ranges from 40 to 80.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes a hybrid expert model structured pruning and acceleration method based on micro-expert ranking according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that executes a hybrid expert model structured pruning and acceleration method based on micro-expert ranking, as described in any one of claims 1-7.

10. A hybrid expert model structured pruning and acceleration system based on micro-expert ranking, characterized in that, The system is implemented using a hybrid expert model structured pruning and acceleration method based on micro-expert ranking as described in any one of claims 1-7, and the system includes: Model splitting module: This module is used to split each expert network in the hybrid expert model into several micro-experts, model these micro-experts, and make the micro-experts of different expert networks comparable. Model ranking module: used to rank all micro-experts according to the energy index of the micro-experts based on the micro-expert ranking algorithm; Model pruning module: This module uses a pruning algorithm to process the sorted micro-experts, selecting and retaining the core micro-experts while deleting the rest.

Images