Hybrid expert model heterogeneous pruning method based on local disturbance agent

By evaluating the output similarity of the MoE layer through local perturbation proxy and modeling it as a multi-choice knapsack problem, and solving the optimal activation number through dynamic programming, the problem of unreasonable allocation of computational resources in the MoE model is solved, and efficient pruning and performance improvement are achieved under a fixed budget.

CN121835791APending Publication Date: 2026-04-10NANJING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511942548.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-22
Publication Date
2026-04-10

AI Technical Summary

Technical Problem

In existing technologies, hybrid expert models (MoEs) have high computational and communication overhead during the inference phase, and existing pruning strategies cannot effectively reflect the differences in semantic functions and computational sensitivity of different MoE layers, resulting in unreasonable allocation of computational resources and affecting model performance.

Method used

A method based on local perturbation proxy is adopted. The similarity of the output representation of each layer is evaluated by self-distillation calibration, a performance value function is constructed, and the hierarchical Top-K activation pruning problem is modeled as a multi-choice knapsack problem. The optimal expert activation number configuration is solved by dynamic programming to realize inter-layer heterogeneous pruning.

Benefits of technology

With a fixed computational budget, the allocation of computational resources was optimized, the model inference performance was improved, and the computational overhead was reduced. At the same time, the globally optimal or near-globally optimal pruning scheme was obtained, avoiding suboptimal solutions from heuristic configuration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121835791A_ABST
    Figure CN121835791A_ABST
Patent Text Reader

Abstract

The invention discloses a local disturbance agent-based hybrid expert model heterogeneous pruning method, which comprises the following steps of: S1, acquiring hierarchical structure information of a MoE model to be pruned, determining the number of MoE layers contained in the model, and setting a total expert activation budget after pruning; s2, evaluating output representation of each MoE layer under different activation expert numbers by using a pre-selected calibration data set, calculating representation similarity of the output representation relative to high-quality baseline configuration, and obtaining a value function of the layer when an expert is adopted; s3, modeling a hierarchical Top-K activation pruning problem into a multi-choice knapsack problem under the constraint of total budget based on a value function, and solving the optimal expert activation number configuration of each MoE layer; s4, according to the optimal expert activation number configuration, the reasoning routing logic of the MoE model is updated; the method is used for solving the problem that in the prior art, a uniform pruning strategy with the low Top-K value is adopted for all layers in a unified mode, the calculation cost is reduced, and meanwhile the model performance is obviously reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and computer system optimization technology, specifically to a heterogeneous pruning method for hybrid expert models based on local perturbation agents. Background Technology

[0002] With the development of large-scale language models, the scale of model parameters continues to expand, and model inference cost has become a significant bottleneck limiting their widespread application. Hybrid expert models, as an effective technique for expanding model capacity through sparse activation mechanisms, are widely used in scenarios such as natural language processing, code generation, and multi-task learning.

[0003] MoE models typically control computational load while maintaining high model performance by introducing multiple parallel expert networks in some feedforward network layers and having a router select a subset of experts for computation for each input token. Mainstream MoE models generally employ a Top-K routing strategy, meaning that for each token, only the K highest-scoring experts are activated in each MoE layer.

[0004] In the original design, to ensure model performance, MoE models typically use high Top-K values ​​(e.g., Top-8) across all MoE layers. However, this strategy incurs significant computational and communication overhead during the inference phase, especially in large-scale models and high-concurrency inference scenarios, making it difficult to meet practical deployment requirements.

[0005] To address this, current practices typically employ a naive uniform activation pruning strategy, which applies the same and relatively low Top-K value to all MoE layers during the inference phase (e.g., setting all layers to Top-4) to simplify deployment and reduce overall computational overhead. However, this strategy implicitly assumes that different MoE layers play the same role in the model and have the same requirement for the number of expert activations, thus failing to reflect the differences in semantic modeling capabilities and computational sensitivity between layers.

[0006] In fact, different MoE layers in the model have different semantic functions and computational sensitivities: some layers are more sensitive to the number of expert activations, and their output representation will degrade significantly when the number of expert activations is excessively reduced; while for other layers, reducing the number of expert activations has a relatively small impact on the overall model output. The uniform Top-K pruning strategy cannot characterize the above-mentioned differences between layers, and therefore it is difficult to achieve a reasonable allocation of computational resources among different layers, thus introducing unnecessary model performance loss while reducing inference computational overhead.

[0007] To address the aforementioned issues, some research has begun to explore the introduction of layer-adaptive expert activation mechanisms. For example, LExI (Layer-Adaptive Active Experts for Efficient MoE Model Inference) proposes assigning different numbers of activation experts to different MoE layers to overcome the limitations of a unified Top-K strategy. However, this method does not use task-related or real input samples during the calibration phase; instead, it evaluates model behavior based on random tensors, making it difficult to accurately reflect the performance impact in real-world inference scenarios. Furthermore, LExI uses the average Frobenius norm of the output differences under different configurations as a performance metric. This metric is an absolute measure and does not normalize the differences in output amplitude between different layers. Since the output norm of each layer in the network typically increases with depth, directly using absolute norm bias may have a disproportionate impact on deeper layers, thus reducing the accuracy of inter-layer comparisons. Additionally, the determination of the number of expert activations relies on heuristic strategies such as evolutionary search, making it difficult to guarantee that the resulting allocation is the globally optimal solution.

[0008] Therefore, existing naive uniform Top-K pruning strategies and hierarchical adaptive methods can generally only obtain suboptimal solutions under the condition of a fixed total expert activation budget. There is an urgent need for an activation pruning method that can perform differentiated expert activation allocation at the hierarchical granularity and achieve global optimum under controllable computational complexity. Summary of the Invention

[0009] To address the problems existing in the prior art, this invention provides a heterogeneous pruning method for hybrid expert models based on local perturbation proxies. This method overcomes the performance degradation caused by the naive, uniform Top-K activation pruning strategy in the prior art under a fixed computational budget. This invention achieves this through the following technical solution.

[0010] A heterogeneous pruning method for hybrid expert models based on local perturbation proxies, characterized in that the method includes the following steps: S1, Obtain the hierarchical structure information of the MoE model to be pruned, and determine the number of MoE layers contained in the model. And set the total expert activation budget after pruning. ; S2, Value estimation based on self-distillation calibration: using a pre-selected calibration dataset, for each MoE layer Different number of activated experts The output representation of the layer is evaluated, and its representation similarity relative to the high-quality baseline configuration is calculated to obtain the similarity of the layer's representation with respect to the high-quality baseline configuration. The value function of the expected performance gain value for an expert ; S3, Global Optimization Solution: Based on the aforementioned value function In the total budget Under constraints, the hierarchical Top-K activation pruning problem is modeled as a multi-choice knapsack problem, and the optimal expert activation number configuration for each MoE layer is solved. ; S4. Update the inference routing logic of the MoE model according to the optimal expert activation number configuration.

[0011] Furthermore, step S2 specifically includes: S21, using the output of the original MoE model under high-quality configuration as a supervision signal, performs forward inference on samples in the calibration dataset to obtain each MoE layer. Baseline output representation; S22, for each MoE layer and the number of each candidate expert The system obtains the top-level value of this layer under the same input. Output representation under configuration; S23, calculate the similarity between the output representation under the Top-k configuration and the baseline output representation to quantify the value of the configuration and obtain the performance value. .

[0012] Furthermore, the similarity is cosine similarity, and the performance value... This involves accumulating the similarity results for all calibration samples and averaging them over the sample dimension. The relevant formula is: .

[0013] Furthermore, the similarity is calculated based on normalized Euclidean distance, and the performance value... The value obtained by subtracting the normalized Euclidean distance from 1 is averaged over the sample dimension. The relevant formula is: .

[0014] Furthermore, step S3, which models the hierarchical Top-K activation pruning problem as a multi-choice knapsack problem, specifically formalizes the hierarchical Top-K activation pruning problem into a combinatorial optimization model that satisfies grouping selection constraints and global budget constraints. This is equivalently represented as a multi-choice knapsack problem, where: In the MoE model, each hybrid expert layer is treated as an independent group of items, with different expert activation numbers in each layer. Select and activate the candidates that constitute mutual exclusion within the group. The resource consumption for each expert is Its performance value is The optimization objective is to ensure that at each level, only one candidate must be selected, and the sum of the number of expert activations selected across all levels does not exceed the total budget. Under the given conditions, maximize the sum of the performance values ​​of all selected candidate options across all layers.

[0015] Furthermore, step S3, which solves for the optimal expert activation number configuration of each MoE layer, employs a dynamic programming algorithm.

[0016] Furthermore, the dynamic programming algorithm includes: S1, Construct a two-dimensional dynamic programming table The two-dimensional dynamic programming table Indicates that only considering the prior A hybrid expert layer, and the expert activation budget has been used for Under the given conditions, the maximum performance value that can be obtained; S2, Initialization For each layer and every feasible budget Iterate through all candidate expert activation numbers in this layer. The update is performed using the state transition equation, with the relevant formula: ; S3, Dynamic Programming Table Fill in the layers in ascending order, followed by the budget index in ascending order.

[0017] Furthermore, the calibration dataset contains samples from multiple task domains.

[0018] The present invention adopts the above technical solution and has the following beneficial effects: (1) By introducing a performance value estimation method based on local perturbation agent, this invention only needs to evaluate the impact of changes in the number of expert activations in a single layer on the intermediate representation of the model, and can approximately measure the contribution of different layers to the overall model performance under different activation configurations. This avoids the exponential computational complexity caused by end-to-end evaluation of all possible activation combinations, and significantly improves the efficiency of the pruning configuration search stage.

[0019] (2) Under the constraint of the total expert activation budget, the present invention performs joint optimization allocation of the number of expert activations in each hybrid expert layer, so that computing resources can be configured differently according to the actual contribution of each layer. By solving the corresponding combinatorial optimization problem, the obtained inter-layer heterogeneous Top-K activation scheme can achieve the global optimum or near-global optimum of performance value under the budget condition, rather than relying on empirical rules or heuristic configuration strategies.

[0020] (3) This invention optimizes the allocation of expert activations in different layers of the model in a static but heterogeneous manner. Under the premise of keeping the total expert activation budget unchanged, it effectively reduces redundant expert activations in layers that contribute less to performance, so that more computing resources are allocated to layers that are more critical to model performance, thereby achieving better model inference performance under the same inference computing overhead constraints. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the applicable hybrid expert MoE model structure for a specific implementation.

[0022] Figure 2 This is a flowchart illustrating the optimization method in the heterogeneous pruning method of the hybrid expert model based on local perturbation agent as described in the specific implementation.

[0023] Figure 3 This is a schematic diagram illustrating the principle of mapping the MoE hierarchical allocation problem to a multi-choice knapsack problem as described in a specific implementation.

[0024] Figure 4 This is a schematic diagram illustrating the implementation of the dynamic programming algorithm for solving the multi-choice knapsack problem in a specific embodiment. Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0026] This invention provides a heterogeneous pruning method for hybrid expert models based on local perturbation proxies, such as... Figure 1 As shown, the method of the present invention can be applied to inference systems that include hybrid expert (MoE) models. The system includes a processor 101 (e.g., a CPU, GPU, or dedicated AI accelerator chip) and a memory 102. The MoE model 103 is stored in the memory 102, and its structure is as follows... Figure 1 As shown, the model comprises multiple network layers, at least some of which are MoE layers. In the hybrid expert layer, the model does not perform computation on all experts simultaneously. Instead, for each input sample or token in the input sequence, the routing module within the layer evaluates multiple candidate experts and selects a subset to participate in the current computation process. Specifically, the routing module calculates the relevance score of each expert based on the input features and selects several experts as activated experts according to their scores. Only the activated experts process the input, while the remaining experts do not participate in the computation. In this way, the model can avoid redundant computation without significantly affecting the output representation capability, thereby effectively reducing the computational overhead in the inference stage. The number of experts actually participating in the computation in each hybrid expert layer is usually limited by a preset Top-K parameter to control the computational scale and resource consumption of the model during the inference process.

[0027] The method of this invention can be implemented by an offline optimization module 104, which runs before the inference task begins. The optimization module 104 reads the structural information of the MoE model 103 and performs [operations] using the calibration dataset. Figure 2 The optimization process shown generates the optimal interlayer heterogeneous Top-K activation pruning scheme. And based on this scheme, the routing logic of model 103 is configured.

[0028] In one embodiment, the calibration dataset contains a predetermined number of samples, such as 50 to 500 samples, but the invention is not limited thereto.

[0029] See Figure 2 The optimization method of this invention includes the following two stages: Phase 1 (Steps S201–S207): Performance value determination stage based on local perturbations, used to estimate the value of the first step. Layer activation Performance value per expert ; Phase 2 (Steps S208–S212): Combinatorial optimization solution phase, used to activate the total expert budget. Under constraints, a Top-K activation allocation scheme for interlayer heterogeneity is determined based on the performance value information.

[0030] The following details the two stages of the optimization method in this embodiment.

[0031] Phase 1: Performance value determination based on local perturbations, aiming to construct a performance value matrix for subsequent combinatorial optimization, specifically including: Step S201, Model and Initial Data Input; In step S201, the system receives the hybrid expert model (MoE model) to be optimized and input data for performance value evaluation, wherein the MoE model contains multiple sequentially connected network layers, and at least some of the network layers are hybrid expert layers.

[0032] Step S202, Construction of Distillation Calibration Dataset; In step S202, the system constructs a calibration dataset for performance evaluation in Phase 1. Unlike end-to-end model training, this calibration dataset is used to characterize the changes in the intermediate representation of the model under different expert activation configurations, and its size is much smaller than the training dataset.

[0033] In a preferred embodiment, the calibration dataset is obtained by distillation, specifically including: Sample several input prompts from a pre-defined task distribution; The prompt is forward-inferred using the original configuration of the MoE model (i.e., the baseline expert activation configuration); (equivalent to using the output of the original MoE model itself under a high-quality configuration as a supervision signal in the original invention, so that the behavior of the pruned model is as consistent as possible with the original model) During decoding or forward propagation, extract the intermediate or output representations corresponding to each layer of the model; The prompt and its corresponding intermediate representation are associated and saved to form a calibration sample set.

[0034] This invention establishes the local perturbation proxy hypothesis based on the following core idea: Given the large number of layers in the hybrid expert model, the combination space of expert activation numbers in each layer grows exponentially. If all possible combinations of expert activation numbers between layers are considered... Performing end-to-end performance evaluations one by one would incur unacceptable computational overhead. Therefore, this invention avoids direct global combined evaluation and instead introduces local output representation similarity as a proxy metric for changes in overall model performance.

[0035] Specifically, for the first A hybrid expert layer, where the number of expert activations in this layer is... When configuring a layer, if the output representation produced by that layer under that configuration has a high similarity to the output representation under the baseline expert activation configuration, it indicates that the local perturbation of that layer configuration is small, and its impact on subsequent layers and the overall model output is limited. Therefore, it is considered that the negative impact of this configuration on the overall model performance is small, and its corresponding performance value is relatively low. If the difference is large, then the configuration is considered to introduce more information loss and has lower performance value.

[0036] In this invention, the output representation similarity can be measured in various ways, including but not limited to cosine similarity between vectors, or normalized Euclidean distance. Different measurement methods can all serve as implementations of the performance value function and do not affect the basic idea of ​​this invention.

[0037] This proxy assumption is based on at least the following two understandings: Information fidelity assumption: If the output representation of a certain layer changes significantly relative to the baseline configuration after the number of expert activations is compressed, it indicates that there is a significant information loss in that layer. The interlayer chain propagation effect hypothesis states that changes in early representations in a hierarchical network will propagate layer by layer, and configurations with smaller local perturbations will have a more limited impact on subsequent layers.

[0038] Steps S203–S206 are the performance value function Based on the above assumptions, the calculation process in Stage 1 involves calculating the performance value matrix through the following nested loop process.

[0039] Step S203: Perform calibration evaluation sample by sample; in step S203, each sample in the calibration dataset is evaluated sequentially. For the current sample, its input is first forward-propagated according to the baseline expert activation configuration.

[0040] Step S204, baseline intermediate representation is obtained; in step S204, the calibration samples are forward propagated, and at each hybrid expert layer At that point, obtain the layer's baseline configuration. The output representation under (for example, the original Top-8 configuration of the model) is denoted as: The baseline output will serve as a reference for subsequent local disturbance assessments.

[0041] Step S205, calculate the output representation under the local perturbation configuration; in step S205, calculate the current calibration sample and the current layer. The system, while maintaining the baseline expert activation configuration in all its preceding layers, only applies the first layer... Apply local perturbations to the layers, i.e., use different expert activation numbers respectively. Perform forward computation to obtain the corresponding output representation: This ensures that the disturbance only acts on a single layer, avoiding the coupling effects caused by multiple layers of superimposed disturbances.

[0042] Step S206, Sample-level performance value calculation and accumulation: In step S206, the system calculates the similarity between the two types of output representations and uses it as the performance value estimate of the current sample, the current layer, and the current number of expert activations.

[0043] In a preferred embodiment, the similarity is measured using cosine similarity. Specifically, for each input sample in the calibration dataset, at the... Each layer obtains its expert activation count. Output representation under the configuration And in the baseline expert activation configuration (the number of expert activations is...) The output representation under ) The cosine similarity between the two samples is calculated. Then, the similarity results for all calibration samples are accumulated and averaged over the sample dimension to obtain the performance value function.

[0044] In another alternative implementation, the performance value function can also be defined based on the normalized Euclidean distance, for example:

[0045] Among them, by the difference of the output vector The norm is normalized to make the obtained performance values ​​comparable. The values ​​obtained using any of the above methods... All of these can serve as a basis for value assessment in the subsequent interlayer heterogeneous Top-K activation pruning optimization process.

[0046] Step S207: Performance value matrix generation; after calculating the activation numbers of all calibration samples, all hybrid expert layers, and all candidate experts, the system will obtain a complete performance value set. and organize it into sizes of The performance value matrix is ​​used as the input for the stage 2 combinatorial optimization solution.

[0047] For details on the implementation of the Phase 2 combinatorial optimization solution phase, please refer to [link / reference]. Figure 2 , Figure 3 , Figure 4 The second stage corresponds to Figure 2 Steps S208–S212 aim to determine the optimal Top-K expert activation allocation scheme for inter-layer heterogeneity under a given total expert activation budget constraint; the specific analysis is as follows: Step S208: Construct a multi-choice knapsack combination optimization model; in step S208, the system receives the performance value matrix generated in stage 1. and the pre-set total expert activation budget .in, Indicates the first Activated in a hybrid expert layer The performance value corresponding to each expert; the total budget This represents the upper limit of the total number of experts allowed to be activated across all hybrid expert layers. In one exemplary implementation, if the uniform pruning strategy uses a Top-4 configuration, then the total budget... Can be set to This is to achieve fair comparison under the same computing resource conditions.

[0048] like Figure 3 As shown, this invention formalizes the hierarchical Top-K activation pruning problem into a combinatorial optimization model that satisfies grouping selection constraints and global budget constraints. Its equivalent representation is the Multiple-Choice Knapsack Problem (MCKP), which specifically includes: Item Groups: Groups each hybrid expert layer in the MoE model Treat it as a separate group of items; Candidate Items: For each level Different number of experts activated These constitute mutually exclusive candidates within the group; Resource consumption (Cost): in the first Layer selection activation The resource consumption for each expert is ; Performance Value: The performance value corresponding to the candidate is the one calculated in Phase 1. ; Constraints: Each layer must select one and only one candidate, and the sum of the number of expert activations selected across all layers cannot exceed the total budget. ; Optimization objective: Under the constraints described above, maximize the sum of the performance values ​​of all selected candidate options across all layers. Step S209: Solve the combinatorial optimization model using dynamic programming; In step S209, in order to solve the above-mentioned multi-choice knapsack combinatorial optimization model, dynamic programming algorithm is preferably used in this embodiment to obtain the global optimal solution under budget constraints.

[0049] In one implementation, the system constructs a two-dimensional dynamic programming table: ,in, Indicates that only considering the prior A hybrid expert layer, and the expert activation budget has been used for Under the given conditions, the maximum performance value that can be obtained. The dynamic programming solution process includes the following steps: Initialization: Initialization The remaining states are initialized to negative infinity or unreachable values; State transition: For each layer and every feasible budget Iterate through all candidate expert activation numbers in this layer. And update according to the following state transition equation:

[0050] This transfer process indicates that: in the... Layer selection activation When there are multiple experts, their corresponding optimal performance value is determined by the previous... Layer in budget The optimal value under the given conditions is determined by both the performance value of the current layer and the overall performance. Table filling order: The dynamic programming table is filled in the order of increasing layer index and increasing budget index.

[0051] In this implementation, the time complexity of the dynamic programming algorithm is O(n log n). The space complexity is This is significantly better than the approach of enumerating all combinations.

[0052] Steps S210–S211: Result backtracking and optimal solution determination; After the dynamic programming table is populated, the system, in the corresponding dynamic programming state at the last level, ensures that the total number of activated experts does not exceed the budget. Under the given conditions, select the state with the highest performance value, that is: ,in, Indicates that only considering the prior A hybrid expert layer, and the total number of activated experts is [number missing]. The maximum performance value that can be obtained at that time.

[0053] Subsequently, the system selects a budget value to maximize the aforementioned performance value. The corresponding termination state Starting from this point, the optimal expert activation allocation scheme is determined layer by layer through a backtracking process, which specifically includes: by state As the current backtracking state; In the In a hybrid expert layer, determine the number of expert activations that satisfy the following equation. :

[0054] Determine the number of expert activations at the current layer. Then, update the current remaining budget to and enter the previous level. ; Repeat the above backtracking steps until you backtrack to the first hybrid expert layer.

[0055] Through the above backtracking process, the system finally obtains the optimal inter-layer heterogeneous Top-K activation pruning scheme that satisfies the budget constraint: Step S212, pruning scheme applied to model output; in step S212, the system updates the MoE model configuration according to the optimal inter-layer heterogeneous Top-K activation pruning scheme. In one implementation, this process is achieved by modifying the model configuration file or routing module parameters, so that the model updates the MoE model during inference at the specified time. Layer fixed activation The system generates a hybrid expert model without requiring retraining of the model structure or parameter weights. After the configuration update is complete, the system outputs the optimized hybrid expert model.

[0056] The above description is merely a preferred embodiment of the present invention, but the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Any simple modifications, equivalent changes, and alterations made by those skilled in the art to the above embodiments based on the technical essence of the present invention without departing from the principles and spirit of the present invention should be included within the protection scope of the present invention.

Claims

1. A heterogeneous pruning method for hybrid expert models based on local perturbation proxies, characterized in that, The method includes the following steps: S1, Obtain the hierarchical structure information of the MoE model to be pruned, and determine the number of MoE layers contained in the model. And set the total expert activation budget after pruning. ; S2, Value estimation based on self-distillation calibration: using a pre-selected calibration dataset, for each MoE layer Different number of activated experts The output representation of the layer is evaluated, and its representation similarity relative to the high-quality baseline configuration is calculated to obtain the similarity of the layer's representation with respect to the high-quality baseline configuration. The value function of the expected performance gain value for an expert ; S3, Global Optimization Solution: Based on the aforementioned value function In the total budget Under constraints, the hierarchical Top-K activation pruning problem is modeled as a multi-choice knapsack problem, and the optimal expert activation number configuration for each MoE layer is solved. ; S4. Update the inference routing logic of the MoE model according to the optimal expert activation number configuration.

2. The heterogeneous pruning method for hybrid expert models based on local perturbation proxies according to claim 1, characterized in that, The S2 step specifically includes: S21, using the output of the original MoE model under high-quality configuration as a supervision signal, performs forward inference on samples in the calibration dataset to obtain each MoE layer. Baseline output representation; S22, for each MoE layer and the number of each candidate expert The system obtains the top-level value of this layer under the same input. Output representation under configuration; S23, calculate the similarity between the output representation under the Top-k configuration and the baseline output representation to quantify the value of the configuration and obtain the performance value. .

3. The heterogeneous pruning method for hybrid expert models based on local perturbation proxies according to claim 2, characterized in that, The similarity is cosine similarity, and the performance value This involves accumulating the similarity results for all calibration samples and averaging them over the sample dimension. The relevant formula is: 。 4. The heterogeneous pruning method for hybrid expert models based on local perturbation proxies according to claim 2, characterized in that, The similarity is calculated based on normalized Euclidean distance, and the performance value... The value obtained by subtracting the normalized Euclidean distance from 1 is averaged over the sample dimension. The relevant formula is: 。 5. The heterogeneous pruning method for hybrid expert models based on local perturbation proxies according to claim 1, characterized in that, Step S3 models the hierarchical Top-K activation pruning problem as a multi-choice knapsack problem. Specifically, it formalizes the hierarchical Top-K activation pruning problem into a combinatorial optimization model that satisfies group selection constraints and global budget constraints. Its equivalent representation is a multi-choice knapsack problem, where: In the MoE model, each hybrid expert layer is treated as an independent group of items, with different expert activation numbers in each layer. Select and activate the candidates that constitute mutual exclusion within the group. The resource consumption for each expert is Its performance value is The optimization objective is to ensure that at each level, only one candidate must be selected, and the sum of the number of expert activations selected across all levels does not exceed the total budget. Under the given conditions, maximize the sum of the performance values ​​of all selected candidate options across all layers.

6. The heterogeneous pruning method for hybrid expert models based on local perturbation proxies according to claim 1, characterized in that, The optimal expert activation number configuration for each MoE layer in step S3 is solved using a dynamic programming algorithm.

7. The heterogeneous pruning method for hybrid expert models based on local perturbation proxies according to claim 6, characterized in that, The dynamic programming algorithm includes: S1, Construct a two-dimensional dynamic programming table The two-dimensional dynamic programming table Indicates that only considering the prior A hybrid expert layer, and the expert activation budget has been used for Under the given conditions, the maximum performance value that can be obtained; S2, Initialization For each layer and every feasible budget Iterate through all candidate expert activation numbers in this layer. The update is performed using the state transition equation, with the relevant formula: ; S3, Dynamic Programming Table Fill in the layers in ascending order, followed by the budget index in ascending order.

8. The heterogeneous pruning method for hybrid expert models based on local perturbation proxies according to any one of claims 1-7, characterized in that, The calibration dataset contains samples from multiple task domains.