Model pruning method and device based on hierarchical importance measurement, medium and equipment

Abstract

The invention discloses a model pruning method based on hierarchical importance measurement, and belongs to the field of model pruning, and the method comprises the steps: dividing a target MoE model into a universal layer section and a domain layer section in the depth direction according to the function division of the target MoE model between different layer sections; determining a first importance score of each first expert by taking an access frequency, an output difference degree and an output amplitude as indexes for a plurality of first experts contained in the general layer section through a calibration set of the target domain, and obtaining a first expert sequence; and for a plurality of second experts contained in the domain layer section, determining a second importance score of each second expert by taking the access frequency and the output difference degree as indexes, and obtaining a second expert sequence; and pruning the target MoE model by taking a preset model pruning proportion as a target according to the first expert sequence and the second expert sequence to obtain a compressed target MoE model which is used for completing a question and answer task of the target domain.

Description

Technical Field

[0001] This application belongs to the field of model pruning, specifically relating to a model pruning method, apparatus, medium, and equipment based on hierarchical importance measurement. Background Technology

[0002] In recent years, the Mixture-of-Experts (MoE) architecture, which expands the model parameter scale by activating a small number of expert networks, has become the mainstream paradigm for building ultra-large-scale language models (such as DeepSeek-R1, with a total of 671B parameters). However, its huge parameter storage overhead poses a severe challenge to model deployment, especially for specific domains such as finance, healthcare, and law, where full model deployment is costly and inefficient.

[0003] Previous research has shown that experts in MoE models exhibit significant domain-specificity. For example, the paper "Domain Specific Pruning of Large Mixture-of-ExpertsModels with Few-shot Demonstrations" found that only a small number of target domain demonstration samples are needed to stably activate a sparse, domain-relevant subset of experts. Based on this, the paper proposed methods such as EASY-EP, which use statistical measures such as the difference in word output before and after and gating scores to evaluate expert importance and prune redundant experts. These methods demonstrate the great potential of domain expert pruning: it can significantly compress the parameter scale while maintaining the model's performance in the target domain. This work represents the closest existing technical solution.

[0004] However, such statistical pruning methods have a fundamental flaw: their globally uniform evaluation system ignores the functional differentiation of MoE models in the depth dimension and the key heterogeneity of expert mechanisms. Specifically, this problem manifests in two aspects: First, it ignores the existence and fatal impact of "super experts": A very small number of super experts exist in the MoE model. These experts are typically located in the shallow or middle layers of the model and are characterized by generating extremely large output activations. They are crucial for forming the attention sink and maintaining the model's normal inference mechanism. Once these super experts are pruned (e.g., removing 3 from Qwen2-30B-A3B), the model performance will catastrophically decline, manifesting as the generation of a large amount of meaningless repetitive content and a complete collapse of mathematical reasoning ability (Pass@1 drops to near 0). Existing methods such as EASY-EP completely fail to identify and protect these crucial experts, and their pruning strategies carry a high risk of causing model failure.

[0005] Second, it ignores the functional differences between different layers and uses a single evaluation criterion: the functions of different layers in the MoE model are not homogeneous. Shallow layers deal more with general language representations (such as syntax and semantics), while deep layers focus more on the integration and output of specific knowledge in the target domain. Existing methods use the same set of importance metrics (such as frequency) throughout the network, failing to differentiate evaluations based on the functions of different layers. This may lead to the incorrect pruning of experts crucial to general language fluency in shallow layers, or the retention of experts with insufficient domain contributions in deep layers, making it difficult to achieve the optimal balance between model compression ratio, domain-specific performance, and general language capability. Summary of the Invention

[0006] The purpose of this application is to provide a model pruning method, apparatus, medium, and device based on hierarchical importance measurement, which can solve the above-mentioned problems.

[0007] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a model pruning method based on hierarchical importance measurement, the method comprising: Based on the functional division of the target MoE model among different layers, the target MoE model is divided into a general layer and a domain layer in the depth direction. The general layer consists of the first two-thirds of the target MoE model and is used to focus on general reasoning capabilities. The domain layer consists of the last one-third of the target MoE model and is used to focus on domain-specific capabilities. Using a calibration set for the target domain, for multiple first experts included in the general layer segment, a first importance score is determined for each first expert based on access frequency, output difference, and output amplitude, and a ranking of the first experts is obtained based on the first importance scores of the multiple first experts; and for multiple second experts included in the domain layer segment, a second importance score is determined for each second expert based on the access frequency and output difference, and a ranking of the second experts is obtained based on the second importance scores of the multiple second experts; Based on the first expert ranking and the second expert ranking, the target MoE model is pruned with a preset model pruning ratio as the target to obtain a compressed target MoE model. The compressed target MoE model is used to complete the question answering task in the target domain.

[0008] Optionally, the primary importance score for each first expert is determined using access frequency, output variability, and output amplitude as indicators, including: For each token sequence in the calibration set, count the total number of times each first expert in the token sequence is activated by the router. A first expert with a gating value greater than 0 is considered to be activated by the router. The access frequency of each first expert is determined based on the average access frequency of each first expert for multiple token sequences. When the first expert is activated by the router, the first cosine similarity between the input representation of a single token before it passes through the first expert and the output representation of the token after it passes through the first expert is calculated, and the difference between 1 and the first cosine similarity is used as the output difference of the first expert on a single token. The output dissimilarity of each first expert is determined based on the average of the output dissimilarity of each first expert across multiple tokens; In each case where the first expert is activated by the router, obtain the output tensor of each token after the forward computation by the first expert. L2 norm and output tensor L2 norm Multiply by the corresponding gating value to obtain the output amplitude of the first expert for a single token; The output amplitude of the first expert is determined based on the average of the output amplitudes of the first expert across multiple tokens; The first importance score of the first expert is determined by multiplying the first expert's access frequency, the first expert's output difference, and the first expert's output amplitude.

[0009] Optionally, the second importance score for each second expert is determined based on access frequency and output variability, including: For each token sequence in the calibration set, count the number of times each second expert in the token sequence is activated by the router, and obtain the access frequency of each second expert in each token sequence based on the number of times each second expert in the token sequence is activated by the router. The access frequency of each second expert is determined by the average access frequency of each expert across multiple token sequences. When the second expert is activated by the router, the second cosine similarity between the input representation of a single token before it passes through the second expert and the output representation of the token after it passes through the second expert is calculated, and the difference between 1 and the second cosine similarity is used as the output difference of the second expert on a single token. The output dissimilarity of each second expert is determined based on the average output dissimilarity of each expert across multiple tokens. The second importance score of the second expert is determined by multiplying the access frequency of the second expert by the output difference of the second expert.

[0010] Optionally, based on the first expert ranking and the second expert ranking, the target MoE model is pruned with a preset model pruning ratio as the objective, to obtain a compressed target MoE model, including: Based on the first expert ranking and the preset model pruning ratio, a number of first experts to be eliminated are determined from the multiple first experts; Based on the second expert ranking and the preset model pruning ratio, a number of second experts to be eliminated are determined from the multiple second experts; Based on the multiple first-to-be-eliminated experts, the general segments of the target MoE model are pruned to retain the first experts who contribute to the target domain task and play a key role in the model stability. Based on the multiple second-to-be-eliminated experts, the general segments of the target MoE model are pruned to retain the second experts who are most proficient in the target domain task, thus obtaining the compressed target MoE model.

[0011] Optionally, after obtaining the compressed target MoE model, the following steps are also included: The compressed target MoE model is used to perform inference on a general validation set to obtain the duplicate token rate of the text generated after inference of each token sequence contained in the general validation set by the compressed target MoE model. Determine whether the duplicate token rate exceeds the preset duplicate rate; If the duplicate token rate exceeds a preset duplicate rate, determine the domain to which the token sequence with the duplicate token rate exceeding the preset duplicate rate belongs; If the domain to which the token sequence with a duplicate token rate exceeding a preset duplicate rate belongs is the target domain, the preset model pruning ratio is reduced, and the target MoE model is pruned according to the first expert ranking and the second expert ranking, with the reduced preset model pruning ratio as the target, to obtain the compressed target MoE model.

[0012] Optionally, after obtaining the compressed target MoE model, the following steps are also included: The compressed target MoE model is used to perform inference on a general validation set to obtain the duplicate token rate of the text generated after inference of each token sequence contained in the general validation set by the compressed target MoE model. Determine whether the duplicate token rate exceeds the preset duplicate rate; If the duplicate token rate exceeds a preset duplicate rate, determine the domain to which the token sequence with the duplicate token rate exceeding the preset duplicate rate belongs; If the domain to which the token sequence with a duplicate token rate exceeding the preset duplicate rate belongs is the target domain, weights are assigned to access frequency, output difference degree, and output amplitude, with the weight of output amplitude being greater than the weights of access frequency and output difference degree. Based on the weighted access frequency, output difference, and output amplitude, new first importance scores for multiple first experts are obtained, and a new ranking of first experts is determined. Based on the new first expert ranking and the second expert ranking, the target MoE model is pruned with the preset model pruning ratio as the target, resulting in a compressed target MoE model.

[0013] Optionally, if the domain to which the token sequence with a duplicate token rate exceeding a preset duplicate rate belongs is not the target domain, the method further includes: Determine whether the token sequence with a repetition rate exceeding a preset repetition rate belongs to a general reasoning task; If a token sequence whose repetition rate exceeds a preset repetition rate belongs to a general reasoning task, a token sequence related to the general reasoning task is added to the calibration set of the target domain to obtain an updated calibration set of the target domain. Using the updated calibration set for the target domain, for multiple first experts included in the general segment, a first importance score is determined for each first expert based on access frequency, output difference, and output amplitude, and a ranking of the first experts is obtained based on the first importance scores of the multiple first experts; and for multiple second experts included in the domain segment, a second importance score is determined for each second expert based on the access frequency and output difference, and a ranking of the second experts is obtained based on the second importance scores of the multiple second experts; Based on the first expert ranking and the second expert ranking, the target MoE model is pruned again with the preset model pruning ratio as the target, resulting in a compressed target MoE model.

[0014] Secondly, embodiments of this application provide a model pruning device based on hierarchical importance measurement, the device comprising: The model partitioning module is used to divide the target MoE model into general-purpose segments and domain-specific segments in the depth direction according to the functional division of different segments. The general-purpose segments are the first 2 / 3 of the target MoE model, which are used to focus on general reasoning capabilities; the domain-specific segments are the last 1 / 3 of the target MoE model, which are used to focus on domain-specific capabilities. The ranking module is used to determine the first importance score of each first expert within the general layer segment using access frequency, output difference, and output amplitude as indicators, based on the calibration set of the target domain; and to obtain a ranking of the first experts based on the first importance scores of the multiple first experts. Additionally, it is used to determine the second importance score of each second expert within the domain segment using the access frequency and output difference as indicators, and to obtain a ranking of the second experts based on the second importance scores of the multiple second experts. The pruning module is used to prune the target MoE model according to the first expert ranking and the second expert ranking, with a preset model pruning ratio as the target, to obtain a compressed target MoE model. The compressed target MoE model is used to complete the question answering task in the target domain.

[0015] Thirdly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0017] The technical solution of this application employs a hierarchical processing mechanism for the target MoE model, dividing it into "general layer" and "domain layer," and using different evaluation criteria for each layer to achieve efficient pruning and optimization. Specifically, for the general layer, a three-dimensional evaluation criterion of access frequency, output dissimilarity, and output amplitude is used. This not only accurately evaluates the contribution of the first expert but also identifies and protects the "super experts" who are crucial to the model's stability by introducing "output amplitude," effectively avoiding potential issues such as duplicate outputs or logical collapses after pruning, such as the high repetition rate of generated answer tokens in question-answering tasks. For the domain layer, access frequency and output dissimilarity are combined to accurately measure the expertise and contribution of the second expert in specific domain knowledge processing, ensuring that the core capabilities of the target domain are not lost. This hierarchical and refined pruning strategy significantly compresses the model parameter size and reduces storage and computation costs while maintaining the model's high performance in the target domain, providing support for the model's practical deployment and application, thereby improving the model's usability. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a model pruning method based on hierarchical importance measurement provided in an embodiment of this application; Figure 2 This is a complete framework diagram of a model pruning method based on hierarchical importance measurement provided in an embodiment of this application; Figure 3 This is a schematic diagram of the framework of a model pruning device based on hierarchical importance measurement provided in an embodiment of this application; Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] To overcome the aforementioned challenges, this application proposes a model pruning method based on hierarchical importance measurement. This method first divides the network into general and domain layers according to model depth, acknowledging their functional differentiation. In the general layer, it innovatively integrates three-dimensional indicators: access frequency, output dissimilarity, and output amplitude, aiming to accurately identify and strictly protect "super experts" and other important general experts who play a crucial role in model stability. In the domain layer, it uses frequency and output dissimilarity indicators, which are more relevant to the domain task, for refined screening. Simultaneously, it introduces duplicate token rate as a feedback signal to dynamically optimize the pruning strategy, thereby fundamentally solving the shortcomings of existing solutions and achieving safe, efficient, and reliable domain-adaptive pruning.

[0022] The following description, in conjunction with the accompanying drawings, details a model pruning method based on hierarchical importance measurement provided by the embodiments of this application, through specific implementations and application scenarios.

[0023] Mixture-of-Experts Language Model (MoE model): A Transformer structure based on sparse activation, in which the traditional feedforward network (FFN) layer is replaced by a mixed expert layer containing multiple expert networks; during each forward propagation, the router selects the Top-K experts for each token based on the input, and only activates these experts (i.e., the experts activated by the router) for computation, thereby significantly increasing the total number of model parameters while greatly reducing computational overhead.

[0024] General Layers: These refer to the set of layers in the MoE model that span from 0% to 75% along the depth direction from bottom to top. Their expert parameters make a significant contribution to general language capabilities (such as grammar and common sense reasoning) and are weakly coupled with specific downstream domains.

[0025] Domain layers: These refer to the set of 25% of the layers in the MoE model from top to bottom along the depth direction. Their expert parameters are highly correlated with specific domain tasks and determine the accuracy of the domain knowledge representation.

[0026] Pruning and compression: A technique that reduces the overall number of parameters and inference latency by filtering and permanently removing low-value expert networks through importance scoring.

[0027] Super experts: Experts that generate extreme activation values ​​within the general layer, whose attention mechanism outputs activation values ​​far higher than other experts; these experts are key nodes (Attention Sink) in the model's attention mechanism, which are crucial for maintaining generation stability and avoiding duplicate outputs. Deleting them can easily lead to duplicate outputs or inference interruptions in the model.

[0028] Domain experts: A group of experts located within the domain layer who play a key role in the performance of domain tasks. Their importance is determined by the frequency of access to domain data and the differences in output.

[0029] Importance score: a scalar indicator used to quantify the contribution of a single expert to the overall performance of the model; the general layer uses a three-dimensional (access frequency, output difference, output amplitude) fusion calculation, while the domain layer uses a two-dimensional (access frequency, output difference) fusion calculation.

[0030] Calibration set: This refers to a representative set of text samples extracted from the target domain (such as finance or healthcare) and tokenized. This dataset is not used for training but only for forward propagation to collect activation statistics from experts, which serve as the basis for importance assessment.

[0031] Figure 1 This is a flowchart illustrating a model pruning method based on hierarchical importance measurement provided in an embodiment of this application. Figure 2 This is a complete framework diagram of a model pruning method based on hierarchical importance measurement provided in an embodiment of this application.

[0032] refer to Figure 1 This application provides a model pruning method based on hierarchical importance measurement, the method comprising steps S11 to S13: Step S11: Based on the functional division of the target MoE model among different layers, the target MoE model is divided into a general layer and a domain layer in the depth direction. The general layer consists of the first 2 / 3 layers of the target MoE model and is used to focus on general reasoning capabilities. The domain layer consists of the last 1 / 3 layers of the target MoE model and is used to focus on domain-specific capabilities.

[0033] In this embodiment, reference Figure 2It receives the complete target MoE model as input. Considering the different functional divisions of the target MoE model in different layers, according to the differences in the functional divisions of different layers, in the depth direction, the first 2 / 3 of the target MoE model is divided into general layers, and the last 1 / 3 of the target MoE model is divided into domain layers.

[0034] Among them, the domain layer is closer to the output layer, and the general layer is closer to the input layer. The general layer focuses on general reasoning ability, such as the layer that focuses on processing general language representations such as syntax and grammar. The domain layer focuses on domain-specific ability, such as the layer that focuses more on integrating contextual information and outputting knowledge closely related to a specific domain (such as mathematics, code, etc.).

[0035] Step S12: Using the calibration set of the target domain, for the multiple first experts included in the general layer segment, determine the first importance score of each first expert based on access frequency, output difference, and output amplitude, and obtain the first expert ranking based on the first importance scores of the multiple first experts; and for the multiple second experts included in the domain layer segment, determine the second importance score of each second expert based on the access frequency and output difference, and obtain the second expert ranking based on the second importance scores of the multiple second experts.

[0036] In this embodiment, the target domain refers to the specific application area that the model needs to optimize, such as finance, healthcare, and law. The target domain calibration set is used to adjust and optimize the target MoE model to fit the target domain dataset. It contains a series of text samples related to the target domain. After preprocessing, the text samples are converted into token sequences that the target MoE model can understand, so as to help the target MoE model understand and generate target domain-related text content.

[0037] The importance of experts at different levels is assessed through a calibration set in order to rank these experts.

[0038] Specifically, the first expert is the expert in the general layer. For each first expert in the general layer, three metrics are calculated: access frequency, output variability, and output amplitude. Access frequency refers to how often the first expert is activated by the router; output variability refers to the degree of change between the first expert's input and output representations; and output amplitude refers to the L2 norm of the output tensor. It should be noted that the output amplitude can be used to identify and protect experts who play a crucial role in the stability of the model in the target domain, avoiding problems such as duplicate outputs or logical collapses in the model after pruning.

[0039] Then, based on the three indicators of access frequency, output difference, and output amplitude, the first importance score of each first expert is obtained. The first experts are then sorted in descending order of their first importance scores to obtain the first expert ranking. In the first expert ranking, the first experts with higher first importance scores are ranked first, and the first experts with lower first importance scores are ranked last.

[0040] The second expert is the expert in the domain layer. For each second expert in the domain layer, two metrics are calculated: access frequency and output variability. Access frequency refers to how often the second expert is activated by the router; output variability refers to the degree of change between the input representation and the output representation of the second expert.

[0041] Then, based on the two indicators of access frequency and output difference, the second importance score of each second expert is obtained. The second experts are sorted in descending order of their second importance scores to obtain the second expert ranking. In the second expert ranking, the second experts with higher second importance scores are ranked first, and the second experts with lower second importance scores are ranked last.

[0042] Step S13: Based on the first expert ranking and the second expert ranking, the target MoE model is pruned with a preset model pruning ratio as the target to obtain a compressed target MoE model. The compressed target MoE model is used to complete the question-answering task in the target domain.

[0043] In this embodiment, the preset model pruning ratio refers to the degree to which the target MoE model needs to be pruned. The preset model pruning ratio can be determined based on the storage space limitations of the hardware device where the model is deployed. For example, assuming the hardware device has only 1GB of storage space, and the unpruned model is 3GB, a higher pruning ratio (e.g., above 60%) may be needed to ensure the model can be deployed on that device. The preset model pruning ratio can also be determined based on the model's computational speed requirements in practical applications. For example, in autonomous driving systems, the model needs to make decisions in a very short time, thus requiring a higher pruning ratio to ensure the model can run quickly in a real-time environment. Furthermore, the preset model pruning ratio can also be determined based on the power consumption and heat dissipation requirements of the hardware device. For example, when deploying a model on a hardware device, a higher pruning ratio needs to be set to reduce power consumption and heat dissipation issues, ensuring that the device does not overheat or run out of power during long-term operation.

[0044] The higher the preset model pruning ratio, the more experts are needed for pruning, and the smaller the number of parameters in the target MoE model. Conversely, the lower the preset model pruning ratio, the fewer experts are needed for pruning, and the larger the number of parameters in the target MoE model.

[0045] After obtaining the first and second expert rankings, the first experts with lower first importance scores in the general layer and the second experts with lower second importance scores in the domain layer are pruned according to the preset model pruning ratio. Since the target MoE model retains the first experts with higher first importance scores in the general layer and the second experts with higher second importance scores in the domain layer, that is, the experts who perform well in the validation set of the target domain, the number of parameters of the compressed target MoE model is reduced after pruning, but the functionality of the model in the target domain is retained, so it can be used to complete the question answering task in the target domain.

[0046] The technical solution described above employs a hierarchical processing mechanism for the target MoE model, dividing it into "general layer" and "domain layer," and using different evaluation criteria for each layer to achieve efficient pruning and optimization. Specifically, for the general layer, a three-dimensional evaluation criterion of access frequency, output dissimilarity, and output amplitude is used. This not only accurately evaluates the contribution of the first expert but also identifies and protects the "super experts" crucial to model stability by introducing "output amplitude," effectively avoiding potential issues such as duplicate outputs or logical collapses after pruning, such as the high repetition rate of generated answer tokens in question-answering tasks. For the domain layer, access frequency and output dissimilarity are combined to accurately measure the expertise and contribution of the second expert in specific domain knowledge processing, ensuring that the core capabilities of the target domain are not lost. This hierarchical and refined pruning strategy significantly compresses the model parameter size and reduces storage and computation costs while maintaining the model's high performance in the target domain, providing support for the model's practical deployment and application, thereby improving the model's usability.

[0047] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another model pruning method based on hierarchical importance measurement. In this method, step S12, "determining the first importance score of each first expert using access frequency, output difference, and output amplitude as indicators," specifically includes steps S12-1-1 to S12-1-7: Step S12-1-1: For each token sequence in the calibration set, count the total number of times each first expert is activated by the router among all tokens in the token sequence. A first expert with a gating value greater than 0 indicates that it has been activated by the router.

[0048] In this embodiment, the calibration set of the target domain contains multiple data entries, each of which is converted into a token sequence. For each token sequence in the calibration set, the number of times each first expert is activated by the router is recorded.

[0049] For example, if a token sequence has 100 tokens, and a certain first expert is activated 30 times for that token sequence (meaning 30 out of 100 tokens are processed by that first expert), then the access frequency of that first expert for this token sequence is 30%. The access frequency can be used to determine the activity level of each first expert when processing token sequences in the target domain.

[0050] Step S12-1-2: Determine the access frequency of each first expert based on the average access frequency of each first expert for multiple token sequences.

[0051] In this embodiment, after obtaining the access frequency of each first expert for each token sequence, the average access frequency of each first expert across all token sequences is calculated as the access frequency of each first expert. For example, if a first expert's access frequencies across three token sequences are 30%, 40%, and 20%, respectively, then the access frequency of that first expert is the average of these access frequencies (30%, 40%, 20%) / 3 = 30%.

[0052] The frequency of access to the first expert reflects its activity level across the entire calibration set. Being selected indicates that the first expert is activated; frequently activated first experts typically perform fundamental and important functions (such as general language modeling and high-frequency word processing). Retaining these frequently activated first experts (i.e., experts in the Top-K) during pruning is a prerequisite for maintaining the model's basic capabilities.

[0053] Step S12-1-3: When the first expert is activated by the router, calculate the first cosine similarity between the input representation of a single token before it passes through the first expert and the output representation of the token after it passes through the first expert, and use the difference between 1 and the first cosine similarity as the output difference of the first expert on a single token.

[0054] In this embodiment, for each first expert, when the router selects that first expert to process multiple tokens in a token sequence, for each first expert selected to process a token, when the token flows through that first expert, the input representation of the token before passing through that first expert is calculated. The token is then processed by the first expert, and the output indicates that... Calculate their cosine similarity That is, the first cosine similarity, then the output dissimilarity of the first expert on this token is defined as... .

[0055] Step S12-1-4: Determine the output dissimilarity of each first expert based on the average output dissimilarity of each first expert across multiple tokens.

[0056] In this embodiment, the average output dissimilarity of each first expert across all tokens processed by that first expert is calculated. For example, if an expert's output dissimilarity across 100 tokens in the validation set is 0.2, 0.3, 0.1, etc., then the expert's output dissimilarity is the average of these values. The output dissimilarity reflects the average influence of the first expert in processing all tokens.

[0057] Output dissimilarity measures the degree to which a first expert alters the input representation. In the general layer, a first expert with high output dissimilarity indicates that the expert has performed significant non-linear transformations on the input information, possibly involving critical feature extraction or semantic conversion. For target domain data, first experts with high output dissimilarity in the general layer demonstrate that their behavior is sensitive to target domain inputs and serve as a crucial bridge connecting general knowledge with downstream domain tasks; therefore, they require special protection during pruning.

[0058] Step S12-1-5: When each first expert is activated by the router, obtain the output tensor of each token after forward computation by the first expert. L2 norm and output tensor L2 norm Multiply by the corresponding gating value to obtain the output amplitude of the first expert for a single token.

[0059] In this embodiment, when each first expert is activated by the router, it means that the first transition is selected to process a token, and the output tensor of the token after the first expert's computation is calculated. L2 norm and the resulting output tensor L2 norm Multiplying this value by the corresponding gating value yields the output amplitude of the first expert on that token. Step S12-1-6: Determine the output amplitude of the first expert based on the average output amplitude of the first expert across multiple tokens.

[0060] In this embodiment, after obtaining the output amplitude of the first expert at each token, the average value of the first expert's output amplitude at these tokens is calculated, and the average value is used as the output amplitude of the first expert.

[0061] Because a few super-experts exist in the general layers with extremely high output amplitudes—potentially tens of thousands of times greater than those of ordinary experts—these massive activation values ​​act as "attention sinks," helping to stabilize the model's attention mechanism and prevent duplicate tokens or logical inconsistencies during generation. Introducing output amplitude is a core technique for protecting these critical super-experts and preventing model collapse after pruning.

[0062] Step S12-1-7: Determine the first importance score of the first expert based on the product of the first expert's access frequency, the first expert's output difference degree, and the first expert's output amplitude.

[0063] In this embodiment, after obtaining the access frequency, output difference degree, and output amplitude of each first expert, the access frequency, output difference degree, and output amplitude of the first expert are multiplied together to obtain the first importance score of each first expert. For example, if a first expert has an access frequency of 0.25, an output difference degree of 0.3, and an output amplitude of 1.8, then the first importance score of the first expert is 0.135.

[0064] Using a product rather than a weighted sum implies a collaborative constraint among the three factors. An expert must simultaneously meet three conditions to achieve a very high first importance score: "frequently used" (high access frequency), "capable of making significant changes to domain data" (high output variability), and "sufficiently large output strength" (high output amplitude). This mechanism can extremely accurately select the top experts (i.e., key experts and super experts with high access frequency, large output variability, and large output amplitude) in the general layer who contribute to both the domain task and the model's stability, thus prioritizing their retention during pruning.

[0065] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another model pruning method based on hierarchical importance measurement. In this method, step S12, "determining the second importance score of each second expert using access frequency and output difference as indicators," includes steps S12-2-1 to S12-2-5: Step S12-2-1: For each token sequence in the calibration set, count the number of times each second expert in the token sequence is activated by the router, and obtain the access frequency of each second expert in each token sequence based on the number of times each second expert in the token sequence is activated by the router.

[0066] In this embodiment, the core task of the second expert is to integrate and output high-level domain knowledge. Therefore, the evaluation criteria focus more on its relevance and expertise to the domain task. The calibration set of the target domain contains multiple data points, each of which is converted into a token sequence. For each token sequence in the calibration set, the number of times each second expert is activated by the router is recorded. For example, if a token sequence has 100 tokens, and a certain second expert is activated 30 times for that token sequence (i.e., 30 out of 100 tokens are processed by the first expert), then the access frequency of that second expert for that token sequence is 30%. The access frequency can be used to determine the activity level of each second expert when processing a specific token sequence.

[0067] Step S12-2-2: Determine the access frequency of each second expert based on the average access frequency of each second expert across multiple token sequences.

[0068] In this embodiment, after obtaining the access frequency of each second expert for each token sequence, the average access frequency of each second expert across all token sequences is calculated as the access frequency of each second expert. For example, if a second expert's access frequencies across three token sequences are 30%, 40%, and 20%, respectively, then the access frequency of that second expert is the average of these access frequencies (30%, 40%, 20%) / 3 = 30%.

[0069] The frequency of access to a second expert reflects its activity level across the entire calibration set. Selection indicates that the second expert is activated; frequently activated second experts play important roles in the target domain (such as integrating and outputting target domain knowledge). Preserving these frequently activated second experts during pruning is a prerequisite for maintaining the model's functionality in the target domain.

[0070] Step S12-2-3: When the second expert is activated by the router, calculate the second cosine similarity between the input representation of a single token before it passes through the second expert and the output representation of the token after it passes through the second expert, and use the difference between 1 and the second cosine similarity as the output difference of the second expert on a single token.

[0071] In this embodiment, for each second expert, when the router selects that second expert to process multiple tokens in a token sequence, for each second expert selected to process a token, when the token flows through that second expert, the input representation of the token before passing through the second expert is calculated. The output of the token after passing through the second expert represents... Calculate their cosine similarity That is, the second cosine similarity, then the output difference on this token is defined as .

[0072] Step S12-2-4: Determine the output dissimilarity of each second expert based on the average of the output dissimilarity of each second expert across multiple tokens.

[0073] In this embodiment, the average output dissimilarity of each second expert across all tokens processed by that second expert is calculated. For example, if a second expert's output dissimilarity across 100 tokens in the validation set is 0.2, 0.3, 0.1, etc., then the second expert's output dissimilarity is the average of these values. The output dissimilarity reflects the average influence of the second expert when processing all tokens.

[0074] At the domain level, output dissimilarity measures the intensity of a second expert's processing of specific information within the target domain. A second expert who can produce significant changes in representation when encountering domain keywords or core concepts indicates deep involvement in the processing of target domain knowledge, thus signifying greater importance.

[0075] Step S12-2-5: Determine the second importance score of the second expert based on the product of the access frequency of the second expert and the output difference of the second expert.

[0076] In this embodiment, after obtaining the access frequency and output dissimilarity of each second expert, the access frequency and output dissimilarity of the second expert are multiplied together to obtain the second importance score of each second expert. For example, if a second expert has an access frequency of 0.5 and an output dissimilarity of 0.3, then the second importance score of the second expert is 0.15.

[0077] At the domain level, the focus is more on the functionality and transformative ability of the second expert within the target domain context, rather than the absolute strength of the output. Therefore, the second importance score is determined by both access frequency and output diversity, selecting the second expert who is most proficient in the target domain task.

[0078] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another model pruning method based on hierarchical importance measurement. In this method, step S13, "pruning the target MoE model according to the first expert ranking and the second expert ranking, with a preset model pruning ratio as the target, to obtain the compressed target MoE model," specifically includes steps S13-1 to S13-3: Step S13-1: Based on the first expert ranking and the preset model pruning ratio, determine a number of first experts to be eliminated from the multiple first experts.

[0079] In this embodiment, based on a preset pruning ratio, multiple first experts in the general segment are identified as first experts to be eliminated, starting from the end of the first expert ranking. These first experts to be eliminated are those in the general segment whose contribution to the model is relatively small and can be pruned to reduce the number of model parameters. For example, if the preset pruning ratio is 30%, there are 100 first experts, and the number of first experts to be eliminated is 30% * 100 = 30, then 30 first experts are identified from the end of the first expert ranking as first experts to be eliminated.

[0080] Step S13-2: Based on the second expert ranking and the preset model pruning ratio, determine a number of second experts to be eliminated from the multiple second experts.

[0081] In this embodiment, based on a preset pruning ratio, multiple second experts in the general segment are identified as second experts to be eliminated, starting from the end of the second expert ranking. These second experts to be eliminated are those in the domain segment whose contribution to the model in the target domain is relatively small and can be pruned to reduce the number of model parameters. For example, if the preset pruning ratio is 30%, and there are 100 second experts, then the number of second experts to be eliminated is 30% * 100 = 30. Therefore, 30 second experts are identified from the end of the second expert ranking as second experts to be eliminated.

[0082] Step S13-3: Based on the multiple first experts to be eliminated, the general segments of the target MoE model are pruned to retain the first experts who contribute to the target domain task and play a key role in the stability of the model; and based on the multiple second experts to be eliminated, the general segments of the target MoE model are pruned to retain the second experts who are most proficient in the target domain task, thus obtaining the compressed target MoE model.

[0083] In this embodiment, based on all the experts to be eliminated identified in steps S13-1 and S13-2, the general and domain layers of the target MoE model are pruned to retain the first expert who contributes to the target domain task and plays a key role in the model's stability, as well as the second expert who is most proficient in the target domain task. Pruning reduces the number of model parameters while ensuring that the model's performance in the target domain is not affected, thus maintaining the model's stability and effectiveness.

[0084] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another model pruning method based on hierarchical importance measurement. In this method, after obtaining the compressed target MoE model in step S13, steps S21 to S24 are further included: Step S21: Perform inference on a general validation set using the compressed target MoE model to obtain the duplicate token rate of the text generated by the compressed target MoE model after inference on each token sequence contained in the general validation set.

[0085] In this embodiment, in order to further verify the performance of the compressed target MoE model, it is necessary to use the compressed target MoE model to perform inference on a general validation set after pruning. The general validation set includes not only data from the target domain, but also data from other general domains. The compressed target MoE model is used to perform inference on the token sequences obtained after transforming these data. For each token sequence, the proportion of consecutive repeated tokens in the text generated by the compressed target MoE model (repeated token rate) is calculated.

[0086] The formula for calculating the duplicate token rate is:

[0087] in, This represents the total number of tokens generated by the compressed target MoE model in a single test case (i.e., a token sequence) on the validation set. This indicates the number of consecutively repeated tokens.

[0088] Step S22: Determine whether the duplicate token rate exceeds the preset duplicate rate. The duplicate token rate can be used to measure the performance of the compressed target MoE model on a general validation set, such as the quality of the generated text.

[0089] In this embodiment, for each token sequence, it is determined whether the proportion of consecutive repeated tokens in the text generated by the compressed target MoE model exceeds a preset repetition rate, such as 10%.

[0090] Step S23: If the duplicate token rate exceeds a preset duplicate rate, determine the domain to which the token sequence with the duplicate token rate exceeding the preset duplicate rate belongs.

[0091] In this embodiment, if the duplicate token rate exceeds a preset threshold, the domain to which the token sequence belongs is further analyzed to determine whether the problem occurs in the target domain.

[0092] Step S24: If the domain to which the token sequence with a duplicate token rate exceeding a preset duplicate rate belongs is the target domain, reduce the preset model pruning ratio, and prune the target MoE model according to the first expert ranking and the second expert ranking, with the reduced preset model pruning ratio as the target, to obtain the compressed target MoE model.

[0093] In this embodiment, if the problem is determined to occur in the target domain, it indicates that over-pruning has damaged key experts in the model. In this case, it is necessary to reduce the preset model pruning ratio and then re-prune the original target MoE model according to the reduced preset model pruning ratio to retain more first and second experts who are important to the target domain.

[0094] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another model pruning method based on hierarchical importance measurement. In this method, after obtaining the compressed target MoE model in step S13, steps S31 to S36 are further included: Step S31: Perform inference on a general validation set using the compressed target MoE model to obtain the duplicate token rate of the text generated by the compressed target MoE model after inference on each token sequence contained in the general validation set.

[0095] In this embodiment, step S31 is the same as step S21 and will not be described again.

[0096] Step S32: Determine whether the duplicate token rate exceeds the preset duplicate rate.

[0097] In this embodiment, step S32 is the same as step S22, and will not be described again.

[0098] Step S33: If the duplicate token rate exceeds a preset duplicate rate, determine the domain to which the token sequence with the duplicate token rate exceeding the preset duplicate rate belongs.

[0099] In this embodiment, step S33 is the same as step S23, and will not be described again.

[0100] Step S34: If the domain to which the token sequence with a duplicate token rate exceeding the preset duplicate rate belongs is the target domain, weights are set for access frequency, output difference degree and output amplitude, wherein the weight of output amplitude is greater than the weight of access frequency and the weight of output difference degree.

[0101] In this embodiment, if the problem is determined to occur in the target domain, it indicates that over-pruning has damaged key experts in the model. Since the high repetition rate is due to over-pruning of the first expert with the highest output amplitude in the general layer, it is necessary to adjust the weights of each indicator in the first importance score, so that the weight of output amplitude is greater than the weights of access frequency and output difference.

[0102] Step S35: Based on the weighted access frequency, output difference, and output amplitude, obtain new first importance scores for multiple first experts and determine the new first expert ranking.

[0103] In this embodiment, since the output amplitude is given a higher weight, new first importance scores for multiple first experts are obtained according to the weighted access frequency, output difference, and output amplitude. This improves the first importance score of the first experts with higher output amplitudes and determines a new ranking of the first experts.

[0104] For example, the weight of output amplitude is 0.5, the weight of access frequency is 0.25, the weight of output difference is 0.25, and the sum of the weights is 1.

[0105] Then, when calculating the first importance score, the weight of the output amplitude is combined with the output amplitude to obtain the weighted output amplitude, the weight of the access frequency is combined with the access frequency to obtain the weighted access frequency, and the weight of the output difference is combined with the output difference to obtain the weighted output difference. Finally, the weighted output amplitude, the weighted access frequency, and the weighted output difference are combined to obtain the new first importance score.

[0106] Step S36: Based on the new first expert ranking and the second expert ranking, prune the target MoE model with the preset model pruning ratio as the target to obtain the compressed target MoE model.

[0107] In this embodiment, the original target MoE model is re-pruned according to the new first expert ranking and second expert ranking, with the preset model pruning ratio as the target, to obtain the compressed target MoE model. Since the first expert with the higher output amplitude is retained as much as possible, the duplicate token rate of the compressed target MoE model can be reduced.

[0108] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another model pruning method based on hierarchical importance measurement. In this method, when the domain to which the token sequence with a repetition rate exceeding a preset repetition rate belongs is not the target domain, steps S41 to S44 are further included: Step S41: Determine whether the token sequence with a repetition rate exceeding a preset repetition rate belongs to a general inference task.

[0109] In this embodiment, if in step S22 or step S32 it is found that the duplicate token rate of certain token sequences exceeds a preset threshold, and these token sequences do not belong to the target domain, then it is necessary to further determine whether these token sequences belong to a general reasoning task. A general reasoning task refers to those language processing tasks that are not specific to a certain domain and are broader in scope, such as logical reasoning, common sense question answering, etc.

[0110] Step S42: If the token sequence with a repetition rate exceeding a preset repetition rate belongs to a general inference task, add the token sequence related to the general inference task to the calibration set of the target domain to obtain an updated calibration set of the target domain.

[0111] In this embodiment, if it is confirmed that these highly repetitive token sequences belong to a general inference task, then more token sequences related to the general inference task need to be added to the calibration set of the target domain to update the calibration set of the target domain, so as to enhance the training of the model on the general inference task and thus improve the model's performance on these tasks.

[0112] Step S43: Using the updated calibration set of the target domain, for the multiple first experts included in the general segment, determine the updated first importance score of each first expert using access frequency, output difference, and output amplitude as indicators, and obtain the updated first expert ranking based on the updated first importance scores of the multiple first experts; and for the multiple second experts included in the domain segment, determine the updated second importance score of each second expert using the access frequency and output difference as indicators, and obtain the updated second expert ranking based on the updated second importance scores of the multiple second experts.

[0113] In this embodiment, the importance scores of each first and second expert are recalculated using the updated calibration set. For the first expert in the general layer, three indicators—access frequency, output dissimilarity, and output amplitude—are used to assess their importance; for the second expert in the domain layer, two indicators—access frequency and output dissimilarity—are used to assess their importance. Then, the experts are ranked according to these importance scores, as detailed above.

[0114] Step S44: Based on the updated first expert ranking and the updated second expert ranking, and with the preset model pruning ratio as the target, the target MoE model is pruned again to obtain the compressed target MoE model.

[0115] In this embodiment, after obtaining the updated first and second expert rankings, the first experts with lower first importance scores in the general layer and the second experts with lower second importance scores in the domain layer are pruned according to the preset model pruning ratio. Since the target MoE model retains the first experts with higher first importance scores in the general layer and the second experts with higher second importance scores in the domain layer, that is, the experts who perform well in the validation set of the target domain, the number of parameters of the compressed target MoE model is reduced after pruning, but the function of the model in the target domain is retained, so that it can be used to complete the question answering task in the target domain.

[0116] The technical solution of this application is illustrated below with a complete example: Input: A complete pre-trained hybrid expert model (MoE model, such as DeepSeek-R1), a calibration set for the target domain (such as AIME math problems), and a general validation set (such as WikiText-2).

[0117] Layering: The model is divided into general layers and domain layers according to a preset ratio.

[0118] Data forward propagation: Input the calibration set and validation set into the model respectively, and collect expert activation records (gating values) and input-output representations for all layers.

[0119] Importance assessment: For the first expert in the general segment, calculate their first importance score.

[0120] For the second expert in the domain segment, calculate their second importance score.

[0121] Pruning Decisions: All experts are sorted by expert, and a pruning threshold is determined based on the global preset model pruning ratio as the target. Experts with scores below the threshold are removed.

[0122] Validation and Feedback: Evaluate the duplicate token rate of the pruned, compressed MoE model using a validation set. If it fails, return to readjustment; if it succeeds, proceed to the next step.

[0123] Output: The final result is a compressed target MoE model oriented towards the target domain.

[0124] The process of using the compressed target MoE model: The user enters a sequence of questions.

[0125] The problem sequence is fed into a pruned, lightweight MoE model.

[0126] The model's router dynamically activates the Top-K remaining experts based on the learned policy.

[0127] The model outputs fluent, accurate, and non-repetitive degenerate responses.

[0128] This application uses the Deepseek-0528 model to comprehensively evaluate the pruned domain model across multiple dimensions, including text generation quality, knowledge application ability, problem reasoning performance, value alignment, security and trustworthiness, and instruction compliance. Experimental results show that the performance of the pruned model with 160 experts based on the EASY-EP algorithm reaches 94% of that of the original model with 256 experts, and the performance is further improved to 97% after introducing the hierarchical super-expert algorithm.

[0129] Regarding repetition rate, in a pruning experiment involving 80 experts out of 160 experts, conducted on a test set of multiple-choice questions across 100 domains, the main statistical data exceeded [a certain percentage]. The number of examples shows that without the hierarchical super expert algorithm, the EASY-EP duplicate example rate was 36%, while after introducing the algorithm, the duplicate example rate was significantly reduced to 8%, which fully verifies the effectiveness and superiority of the present invention.

[0130] The hierarchical importance domain pruning method proposed in this application is specifically designed to address the pain points of large-scale MoE (Hybrid Expert) language models in practical deployments, such as huge memory consumption, high inference latency, and high domain customization costs. Its applications are wide-ranging, specifically reflected in the following aspects: 1. Domain-specific high-performance model deployment Financial quantitative analysis: Pruning MoE models with hundreds of billions of users (such as DeepSeek-R1) based on financial texts, retaining risk prediction and quantitative inference experts, to obtain a dedicated model with half the number of parameters but no loss of performance, which is then deployed in a quantitative trading system to achieve low-latency market sentiment analysis and high-frequency trading decisions.

[0131] Legal text retrieval and contract review: By tailoring a general model and accurately retaining legal text interpretation and compliance review experts, we empower the intelligent contract review SaaS platform to provide law firms and corporate legal departments with efficient and accurate text retrieval and risk identification services, significantly improving work efficiency.

[0132] 2. Lightweight Training of Domain Models: This application provides an efficient starting point for domain-adaptive training. After pruning, the model parameter size is significantly reduced (e.g., by 50%), greatly shortening the training cycle and computational cost of subsequent full-fine-tuning or efficient fine-tuning (such as LoRA). Users can quickly inject a small amount of domain data into the pruned "skeleton model" that already contains key domain knowledge to complete model customization, realizing an efficient domain adaptation pipeline from "pre-training -> pruning -> lightweight fine-tuning," lowering the development threshold for domain AI applications.

[0133] It should be noted that the model pruning method based on hierarchical importance measurement provided in this application embodiment can be executed by a model pruning device based on hierarchical importance measurement, or by a control module within that device for executing the method. This application embodiment uses the execution of the loading method based on hierarchical importance measurement by a model pruning device as an example to illustrate the model pruning method based on hierarchical importance measurement provided in this application embodiment.

[0134] Figure 3 This is a schematic diagram of the framework of a model pruning device based on hierarchical importance measurement according to an embodiment of this application. (Refer to...) Figure 3 One embodiment of this application provides a model pruning device based on hierarchical importance measurement, the device comprising: The model partitioning module 11 is used to divide the target MoE model into a general layer and a domain layer in the depth direction according to the functional division of the target MoE model in different layers. The general layer consists of the first 2 / 3 layers of the target MoE model and is used to focus on general reasoning capabilities. The domain layer consists of the last 1 / 3 layers of the target MoE model and is used to focus on domain-specific capabilities. The ranking module 12 is used to determine the first importance score of each first expert included in the general segment using access frequency, output difference, and output amplitude as indicators, based on the calibration set of the target domain; and to obtain the ranking of the first experts based on the first importance scores of the multiple first experts; and to determine the second importance score of each second expert included in the domain segment using the access frequency and output difference as indicators, and to obtain the ranking of the second experts based on the second importance scores of the multiple second experts. The pruning module 13 is used to prune the target MoE model according to the first expert ranking and the second expert ranking, with a preset model pruning ratio as the target, to obtain a compressed target MoE model. The compressed target MoE model is used to complete the question answering task in the target domain.

[0135] Optionally, the sorting module 12 includes: The first access frequency calculation unit is used to calculate the total number of times each first expert is activated by the router among all tokens in the calibration set for each token sequence. A first expert with a gating value greater than 0 indicates that it has been activated by the router. The first access frequency determination unit is used to determine the access frequency of each first expert based on the average access frequency of each first expert for multiple token sequences. The first output difference calculation unit is used to calculate the first cosine similarity between the input representation of a single token before it passes through the first expert and the output representation of the token after it passes through the first expert when the first expert is activated by the router, and use the difference between 1 and the first cosine similarity as the output difference of the first expert on a single token. The first output difference determination unit is used to determine the output difference of each first expert based on the average output difference of each first expert on multiple tokens; The output tensor computation unit is used to obtain the output tensor of each token after forward computation by the first expert, in the case where each first expert is activated by the router. L2 norm and output tensor L2 norm Multiply by the corresponding gating value to obtain the output amplitude of the first expert for a single token; The output amplitude determination unit is used to determine the output amplitude of the first expert based on the average of the output amplitudes of the first expert across multiple tokens. The first importance score determination unit is used to determine the first importance score of the first expert based on the product of the first expert's access frequency, the first expert's output difference degree, and the first expert's output amplitude.

[0136] Optionally, the sorting module 12 includes: The second access frequency calculation unit is used to count the number of times each second expert in each token sequence is activated by the router for each token sequence in the calibration set, and to obtain the access frequency of each second expert in each token sequence based on the number of times each second expert is activated by the router in each token sequence. The second access frequency determination unit is used to determine the access frequency of each second expert based on the average access frequency of each second expert in multiple token sequences. The second output difference calculation unit is used to calculate the second cosine similarity between the input representation of a single token before it passes through the second expert and the output representation of the token after it passes through the second expert when the second expert is activated by the router, and to use the difference between 1 and the second cosine similarity as the output difference of the second expert on a single token. The second output difference determination unit is used to determine the output difference of each second expert based on the average output difference of each second expert on multiple tokens; The second importance score determination unit is used to determine the second importance score of the second expert based on the product of the access frequency of the second expert and the output difference of the second expert.

[0137] Optionally, the pruning module 13 includes: The first expert to be eliminated unit is used to determine multiple first experts to be eliminated from multiple first experts based on the first expert ranking and the preset model pruning ratio. The second elimination expert determination unit is used to determine multiple second elimination experts from multiple second experts based on the second expert ranking and the preset model pruning ratio. The pruning unit is used to prune the general segments of the target MoE model based on the plurality of first experts to be eliminated, so as to retain the first experts who contribute to the target domain task and play a key role in the stability of the model, and to prune the general segments of the target MoE model based on the plurality of second experts to be eliminated, so as to retain the second experts who are most proficient in the target domain task, thereby obtaining the compressed target MoE model.

[0138] Optionally, the device further includes: The first duplicate token rate determination module is used to perform inference on a general validation set using the compressed target MoE model after obtaining the compressed target MoE model, and to obtain the duplicate token rate of the text generated after inference of each token sequence contained in the general validation set by the compressed target MoE model. The first judgment module is used to determine whether the duplicate token rate exceeds a preset duplicate rate; The first domain determination module is used to determine the domain to which the token sequence with a duplicate token rate exceeding a preset duplicate rate belongs when the duplicate token rate exceeds a preset duplicate rate. The first adjustment module is used to reduce the preset model pruning ratio when the domain to which the token sequence with a duplicate token rate exceeding the preset duplicate rate belongs is the target domain, and to prune the target MoE model according to the first expert ranking and the second expert ranking, with the reduced preset model pruning ratio as the target, to obtain the compressed target MoE model.

[0139] Optionally, the device further includes: The second duplicate token rate determination module is used to perform inference on a general validation set using the compressed target MoE model after obtaining the compressed target MoE model, and to obtain the duplicate token rate of the text generated after inference of each token sequence contained in the general validation set by the compressed target MoE model. The second judgment module is used to determine whether the duplicate token rate exceeds a preset duplicate rate; The second domain determination module is used to determine the domain to which the token sequence with a duplicate token rate exceeding a preset duplicate rate belongs when the duplicate token rate exceeds a preset duplicate rate. The second adjustment module is used to set weights for access frequency, output difference degree and output amplitude when the domain to which the token sequence with the duplicate token rate exceeds the preset duplicate rate belongs is the target domain. The weight of the output amplitude is greater than the weight of the access frequency and the weight of the output difference degree. The first expert ranking determination module is used to obtain new first importance scores for multiple first experts based on weighted access frequency, output difference degree and output amplitude, and to determine the new first expert ranking. The first re-pruning module is used to prune the target MoE model according to the new first expert ranking and the second expert ranking, with the preset model pruning ratio as the target, to obtain the compressed target MoE model.

[0140] Optionally, the device further includes: The third judgment module is used to determine whether the token sequence with a repeating token rate exceeding the preset repeating rate belongs to a general reasoning task when the domain to which the token sequence belongs is not the target domain. The calibration set update module is used to add token sequences related to the general reasoning task to the calibration set of the target domain when the token sequence with a repetition rate exceeding a preset repetition rate belongs to the general reasoning task, so as to obtain an updated calibration set of the target domain. The reordering module is used to determine, using the updated calibration set of the target domain, the updated first importance score of each first expert included in the general segment based on access frequency, output difference, and output amplitude, and to obtain an updated first expert ranking based on the updated first importance scores of the multiple first experts; and to determine, using the access frequency and output difference as indicators, the updated second importance score of each second expert included in the domain segment, and to obtain an updated second expert ranking based on the updated second importance scores of the multiple second experts. The second re-pruning module is used to re-prune the target MoE model based on the updated first expert ranking and the updated second expert ranking, with the preset model pruning ratio as the target, to obtain the compressed target MoE model.

[0141] The model pruning device based on hierarchical importance measurement in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not make specific limitations.

[0142] The model pruning device based on hierarchical importance measurement in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0143] The model pruning device based on hierarchical importance measurement provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented by the model pruning device based on hierarchical importance measurement in the method embodiments will not be described again here to avoid repetition.

[0144] Optionally, Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. This application also provides an electronic device; it should be noted that the electronic device in this application includes the mobile electronic device and non-mobile electronic device described above.

[0145] The electronic device includes, but is not limited to, components such as: radio frequency unit, network module, audio output unit, input unit, sensor, display unit, user input unit, interface unit, memory, and processor.

[0146] Those skilled in the art will understand that electronic devices may also include power supplies (such as batteries) that supply power to various components. The power supply may be connected to the processor logic through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0147] As an example, such as Figure 4 As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps in the model pruning method based on hierarchical importance measurement disclosed in the above embodiments of this application.

[0148] As the apparatus is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects.

[0151] Furthermore, this application embodiment also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described model pruning method embodiment based on hierarchical importance measurement and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0152] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0153] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the model pruning method based on hierarchical importance measurement, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0154] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0155] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0157] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model pruning method based on hierarchical importance measurement, characterized in that, The method includes: Based on the functional division of the target MoE model among different layers, the target MoE model is divided into a general layer and a domain layer in the depth direction. The general layer consists of the first two-thirds of the target MoE model and is used to focus on general reasoning capabilities. The domain layer consists of the last one-third of the target MoE model and is used to focus on domain-specific capabilities. Using a calibration set for the target domain, for multiple first experts included in the general layer, a first importance score is determined for each first expert based on access frequency, output difference, and output amplitude, and a ranking of the first experts is obtained based on the first importance scores of the multiple first experts; and for multiple second experts included in the domain layer, a second importance score is determined for each second expert based on the access frequency and output difference, and a ranking of the second experts is obtained based on the second importance scores of the multiple second experts; Based on the first expert ranking and the second expert ranking, the target MoE model is pruned with a preset model pruning ratio as the target to obtain a compressed target MoE model. The compressed target MoE model is used to complete the question answering task in the target domain.

2. The model pruning method based on hierarchical importance measurement according to claim 1, characterized in that, The primary importance score for each top expert was determined using access frequency, output dissimilarity, and output amplitude as indicators, including: For each token sequence in the calibration set, count the total number of times each first expert in the token sequence is activated by the router. A first expert with a gating value greater than 0 is considered to be activated by the router. The access frequency of each first expert is determined based on the average access frequency of each first expert for multiple token sequences. When the first expert is activated by the router, the first cosine similarity between the input representation of a single token before it passes through the first expert and the output representation of the token after it passes through the first expert is calculated, and the difference between 1 and the first cosine similarity is used as the output difference of the first expert on a single token. The output dissimilarity of each first expert is determined based on the average of the output dissimilarity of each first expert across multiple tokens; In each case where the first expert is activated by the router, obtain the output tensor of each token after the forward computation by the first expert. L2 norm and output tensor L2 norm Multiply by the corresponding gating value to obtain the output amplitude of the first expert for a single token; The output amplitude of the first expert is determined based on the average of the output amplitudes of the first expert across multiple tokens; The first importance score of the first expert is determined based on the product of the first expert's access frequency, the first expert's output difference, and the first expert's output amplitude.

3. The model pruning method based on hierarchical importance measurement according to claim 1, characterized in that, The second importance score for each second expert was determined based on access frequency and output variability, including: For each token sequence in the calibration set, count the number of times each second expert in the token sequence is activated by the router, and obtain the access frequency of each second expert in each token sequence based on the number of times each second expert in the token sequence is activated by the router. The access frequency of each second expert is determined by the average access frequency of each expert across multiple token sequences. When the second expert is activated by the router, the second cosine similarity between the input representation of a single token before it passes through the second expert and the output representation of the token after it passes through the second expert is calculated, and the difference between 1 and the second cosine similarity is used as the output difference of the second expert on a single token. The output dissimilarity of each second expert is determined based on the average output dissimilarity of each expert across multiple tokens. The second importance score of the second expert is determined by multiplying the access frequency of the second expert by the output difference of the second expert.

4. The model pruning method based on hierarchical importance measurement according to claim 1, characterized in that, Based on the first expert ranking and the second expert ranking, and with a preset model pruning ratio as the target, the target MoE model is pruned to obtain a compressed target MoE model, including: Based on the first expert ranking and the preset model pruning ratio, a number of first experts to be eliminated are determined from the multiple first experts; Based on the second expert ranking and the preset model pruning ratio, a number of second experts to be eliminated are determined from the multiple second experts; Based on the multiple first-to-be-eliminated experts, the general segments of the target MoE model are pruned to retain the first experts who contribute to the target domain task and play a key role in the model stability. Based on the multiple second-to-be-eliminated experts, the general segments of the target MoE model are pruned to retain the second experts who are most proficient in the target domain task, thus obtaining the compressed target MoE model.

5. The model pruning method based on hierarchical importance measurement according to claim 1, characterized in that, After obtaining the compressed target MoE model, the following is also included: The compressed target MoE model is used to perform inference on a general validation set to obtain the duplicate token rate of the text generated after inference of each token sequence contained in the general validation set by the compressed target MoE model. Determine whether the duplicate token rate exceeds the preset duplicate rate; If the duplicate token rate exceeds a preset duplicate rate, determine the domain to which the token sequence with the duplicate token rate exceeding the preset duplicate rate belongs; If the domain to which the token sequence with a duplicate token rate exceeding a preset duplicate rate belongs is the target domain, the preset model pruning ratio is reduced, and the target MoE model is pruned according to the first expert ranking and the second expert ranking, with the reduced preset model pruning ratio as the target, to obtain the compressed target MoE model.

6. The model pruning method based on hierarchical importance measurement according to claim 1, characterized in that, After obtaining the compressed target MoE model, the following is also included: The compressed target MoE model is used to perform inference on a general validation set to obtain the duplicate token rate of the text generated after inference of each token sequence contained in the general validation set by the compressed target MoE model. Determine whether the duplicate token rate exceeds the preset duplicate rate; If the duplicate token rate exceeds a preset duplicate rate, determine the domain to which the token sequence with the duplicate token rate exceeding the preset duplicate rate belongs; If the domain to which the token sequence with a duplicate token rate exceeding the preset duplicate rate belongs is the target domain, weights are assigned to access frequency, output difference degree, and output amplitude, with the weight of output amplitude being greater than the weights of access frequency and output difference degree. Based on the weighted access frequency, output difference, and output amplitude, new first importance scores for multiple first experts are obtained, and a new ranking of first experts is determined. Based on the new first expert ranking and the second expert ranking, the target MoE model is pruned with the preset model pruning ratio as the target, resulting in a compressed target MoE model.

7. The model pruning method based on hierarchical importance measurement according to claim 5 or 6, characterized in that, If the domain of the token sequence whose repetition rate exceeds a preset repetition rate is not the target domain, the following additional steps are also included: Determine whether the token sequence with a repetition rate exceeding a preset repetition rate belongs to a general reasoning task; If a token sequence whose repetition rate exceeds a preset repetition rate belongs to a general reasoning task, a token sequence related to the general reasoning task is added to the calibration set of the target domain to obtain an updated calibration set of the target domain. Using the updated calibration set of the target domain, for multiple first experts included in the general segment, an updated first importance score is determined for each first expert based on access frequency, output difference, and output amplitude. An updated ranking of the first experts is then obtained based on these updated first importance scores. Similarly, for multiple second experts included in the domain segment, an updated second importance score is determined for each second expert based on the access frequency and output difference. An updated ranking of the second experts is then obtained based on these updated second importance scores. Based on the updated first expert ranking and the updated second expert ranking, the target MoE model is pruned again with the preset model pruning ratio as the target, resulting in a compressed target MoE model.

8. A model pruning device based on hierarchical importance measurement, characterized in that, The device includes: The model partitioning module is used to divide the target MoE model into general-purpose segments and domain-specific segments in the depth direction according to the functional division of different segments. The general-purpose segments are the first 2 / 3 of the target MoE model, which are used to focus on general reasoning capabilities; the domain-specific segments are the last 1 / 3 of the target MoE model, which are used to focus on domain-specific capabilities. The ranking module is used to determine the first importance score of each first expert within the general layer segment using access frequency, output difference, and output amplitude as indicators, based on the calibration set of the target domain; and to obtain a ranking of the first experts based on the first importance scores of the multiple first experts. Additionally, it is used to determine the second importance score of each second expert within the domain segment using the access frequency and output difference as indicators, and to obtain a ranking of the second experts based on the second importance scores of the multiple second experts. The pruning module is used to prune the target MoE model according to the first expert ranking and the second expert ranking, with a preset model pruning ratio as the target, to obtain a compressed target MoE model. The compressed target MoE model is used to complete the question answering task in the target domain.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the model pruning method based on hierarchical importance measurement as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the model pruning method based on hierarchical importance metric as described in any one of claims 1-7.

Images