A hybrid expert model collaborative pruning method and system for embedded deployment

Abstract

The application discloses a hybrid expert model collaborative pruning method and system for embedded deployment, and the method comprises the following steps: obtaining a pre-training sparse hybrid expert model to be pruned and a calibration data set, and determining a target retention rate according to resource constraint parameters of target deployment hardware; performing offline forward inference on the calibration data set by using the model, recording the token activation expert routing result of each hybrid expert layer, and constructing an expert combination support set; calculating the expert collaborative contribution value and the total positive contribution quality for each expert in each layer; and determining the maximum feasible quality coverage threshold value through binary search; performing hierarchical minimum prefix selection according to the threshold value to obtain a final reserved expert set, and exporting a pruned hybrid expert model. The application completes the collaborative pruning of the hybrid expert model based on the routing track and co-occurrence statistics of the inference process, and does not need to perform additional loss reevaluation or model retraining on the candidate pruning result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model compression and deployment optimization technology, and more specifically to a hybrid expert model collaborative pruning method and system for embedded deployment. Background Technology

[0002] Currently, sparse hybrid expert models (MoE) have become the mainstream architecture for high-performance inference due to their large parameters and low activation characteristics. However, their deployment still faces severe challenges on resource-constrained platforms such as domestic embedded devices and edge nodes. Although only a small number of experts are activated in a single inference, the server must reside all expert parameters in high-speed storage, resulting in enormous pressure on GPU memory and memory access, forming an insurmountable "memory wall" that severely restricts the feasibility of model deployment.

[0003] However, existing pruning methods mostly rely on single-expert statistical metrics such as activation frequency, gating score, or weight norm for independent scoring and removal. These methods treat experts as isolated units, ignoring the dynamic combination and collaborative work among experts in the MoE model. In fact, the value of an expert lies not only in the frequency of activation but also in its marginal contribution to the overall output in a specific combination. Simply relying on local metrics can easily lead to the accidental removal of experts who play a supplementary role in key combinations, disrupting the collaborative structure and causing model performance degradation or even functional collapse.

[0004] In addition, existing technologies have many shortcomings: they lack a quantitative mechanism for the collaborative contributions of expert groups; they mostly use fixed quota pruning, which is difficult to adapt to the differences in contribution distribution between different layers and is prone to network topology imbalance; some solutions rely on complex retraining or evaluation processes, which are costly and difficult to integrate; and they generally lack control logic that is linked to target hardware resources (video memory, bandwidth, latency), which cannot meet the automated deployment requirements of embedded scenarios.

[0005] Therefore, providing a hybrid expert model pruning method that can take into account synergistic effects, adaptive hierarchical structure, low engineering cost, and adaptability to hardware constraints is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a hybrid expert model collaborative pruning method and system for embedded deployment that overcomes or at least partially solves the above problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, embodiments of the present invention provide a collaborative pruning method for hybrid expert models for embedded deployment, comprising the following steps:

[0009] S1. Obtain the pre-trained sparse hybrid expert model to be pruned and the calibration dataset, and determine the target retention rate based on the resource constraint parameters of the target deployment hardware;

[0010] S2. Using the model, perform offline forward inference on the calibration dataset, record the expert routing results and gating statistics for each token activation in each hybrid expert layer, and construct the expert combination support set actually observed.

[0011] S3. Treat the expert portfolio support set as the observation alliance space of empirical cooperative game and define the alliance empirical utility;

[0012] S4. For each expert in each hybrid expert layer, based on the set of observation combinations they have participated in and the co-occurrence statistics of the sub-combinations after removing the expert, calculate the sub-combination proxy utility corrected by monotonicity constraints, and calculate the expert collaborative contribution value based on the difference between the original alliance experience utility and the sub-combination proxy utility.

[0013] S5. Perform non-negative truncation on the expert collaborative contribution values ​​of each hybrid expert layer and calculate the total positive contribution quality; and determine the maximum feasible quality coverage threshold that makes the global retention rate lower than the target retention rate through binary search.

[0014] S6. Perform hierarchical minimum prefix selection based on the threshold to obtain the final set of retained experts, delete the experts that were not retained, and export the pruned hybrid expert model.

[0015] Further, step S1 specifically includes:

[0016] Obtain the pre-trained sparse hybrid expert model to be pruned, and identify all hybrid expert layers in the model, the number of experts in each hybrid expert layer, the number of experts activated by each token, and the parameter mapping relationship corresponding to each expert;

[0017] Obtain the calibration dataset corresponding to the target task, which is divided into multiple calibration subsets based on the task type;

[0018] Obtain the resource constraint parameters of the target embedded deployment hardware, and determine the target retention rate or target resource limit based on the resource constraint parameters; the determination method is based on the total budget of the target hardware. Resident overhead of basic parameters Expert pool parameter overhead and runtime caching overhead Determine the target retention rate This ensures that the pruned model satisfies resource constraints. .

[0019] Furthermore, step S2 specifically includes:

[0020] The model is used to perform offline forward inference on the calibration dataset. The sample number, token position, hybrid expert layer number, top-k expert number set and corresponding gating weight are recorded during the inference process. The top-k expert routing results, activation frequency and gating statistics of each token in each hybrid expert layer are also statistically analyzed.

[0021] The top-k expert routing results of each token in each hybrid expert layer are deduplicated and aggregated. The same expert combinations that appear repeatedly in the same layer are aggregated into a mapping table of combination identifier and activation frequency. The actual observed expert combination support set is constructed layer by layer.

[0022] Furthermore, step S3 specifically includes:

[0023] S3. Treat the expert portfolio support set as the observation alliance space of empirical cooperative game and define the alliance empirical utility;

[0024] The expert portfolio support set is regarded as the observation alliance space of empirical cooperative game, and the number of times the expert portfolio appears, the gating weighted result, or the task weighted statistics are defined as the alliance empirical utility.

[0025] No. Layered hybrid expert layer expert combination The utility of alliance experience can be expressed by the following formula:

[0026]

[0027] in, Indicates task The corresponding task weight, Indicates task In the Layer-by-layer expert combination The count statistics, gated weighted statistics, or normalized cumulative values ​​are given, where T represents the total number of tasks participating in the weighted aggregation.

[0028] Furthermore, step S4 specifically includes:

[0029] S41. For each expert in each hybrid expert layer, construct a set of observation combinations that they have participated in, and based on the inclusion relationship statistics of the sub-combinations in the observation alliance space after removing the corresponding experts, establish the original co-occurrence statistics of the sub-combinations.

[0030] S42. Calculate the scaling factor of the corresponding expert based on the original alliance experience utility and the original co-occurrence statistics of the sub-combination to obtain the sub-combination proxy utility corrected by monotonicity constraints, and calculate the Shapley-style expert collaborative contribution value based on the difference between the original alliance experience utility and the sub-combination proxy utility.

[0031] Furthermore, in step S42, regarding the first... Experts in a hybrid expert layer The scaling factor is expressed by the formula:

[0032]

[0033] The sub-combination agency utility is expressed by the following formula:

[0034]

[0035] in, Indicates the first Layered experts The scaling factor, Indicates the first Layered experts The set of observation combinations Indicates removal of experts The original co-occurrence statistics of the sub-combinations. This indicates the modified sub-combination proxy utility.

[0036] Furthermore, in step S42, the collaborative contribution value is expressed by the formula:

[0037]

[0038] in, This represents the collaborative contribution value of each hybrid expert layer.

[0039] Furthermore, step S5 specifically includes:

[0040] The collaborative contribution values ​​of experts in each hybrid expert layer are non-negatively truncated, and expressed by the formula:

[0041]

[0042] The total positive contribution mass is calculated using the following formula:

[0043]

[0044] in, Indicates non-negative collaborative contribution. Indicates the total positive contribution quality. Indicates the first The total number of experts at each level;

[0045] For each layer of hybrid experts, sort them from largest to smallest according to the non-negative collaborative contribution, and select experts in turn until the cumulative non-negative collaborative contribution of the selected experts reaches the preset proportion α of the total positive contribution quality of the layer.

[0046] By iteratively adjusting α using a binary search approach, the maximum feasible quality coverage value is achieved, where the global retention rate corresponding to the total number of experts retained in all hybrid expert layers is lower than the target retention rate. This is to avoid layer collapse and maintain the balance of interlayer structure.

[0047] Secondly, embodiments of the present invention provide a hybrid expert model collaborative pruning system for embedded deployment, employing the method described in any one of the first aspects, comprising the following modules:

[0048] Acquisition module: used to acquire the pre-trained sparse hybrid expert model to be pruned and the calibration dataset, and to determine the target retention rate based on the resource constraint parameters of the target deployment hardware;

[0049] The routing trajectory acquisition module is used to perform offline forward inference on the calibration dataset using the model, record the routing results of each token activation expert and gating statistics of each hybrid expert layer, and construct the expert combination support set actually observed.

[0050] The Module for Constructing Experienced Cooperative Game Theory is used to view the expert portfolio support set as the observation alliance space for experienced cooperative game theory and to define the alliance's empirical utility.

[0051] The module for constructing the utility of the sub-combination of agents and evaluating collaborative contributions is used to calculate the sub-combination agent utility corrected for monotonicity constraints for each expert in each hybrid expert layer, based on the set of observation combinations they have participated in and the co-occurrence statistics of the sub-combinations after removing the expert. It also calculates the expert collaborative contribution value based on the difference between the original alliance experience utility and the sub-combination agent utility.

[0052] Resource constraint matching module: used to perform non-negative truncation on the expert collaborative contribution values ​​of each hybrid expert layer and calculate the total positive contribution quality; and to determine the maximum feasible quality coverage threshold that makes the global retention rate lower than the target retention rate through binary search;

[0053] Model export module: used to perform hierarchical minimum prefix selection based on the threshold to obtain the final set of retained experts, delete the experts that were not retained, and export the pruned hybrid expert model.

[0054] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a hybrid expert model collaborative pruning method and system for embedded deployment, which has the following beneficial effects:

[0055] 1. Accurately quantify the value of expert collaboration and protect key portfolio structures.

[0056] By introducing the collaborative contribution calculation of "expert portfolio support set" and "Shapley style," this invention breaks through the limitation of traditional pruning relying solely on single-expert statistics (such as activation frequency and weight norm). It can identify and retain "hidden experts" who play a key supplementary role in specific portfolios from a global perspective, effectively preventing the destruction of collaborative structure caused by isolated evaluation, thereby maintaining the model's high accuracy and semantic coherence even after significant pruning.

[0057] 2. Adaptive hierarchical pruning to maintain network topology balance.

[0058] This invention abandons static pruning strategies that rely on a globally uniform threshold or fixed quotas layer by layer. By analyzing the differences in contribution distribution among different MoE layers, this invention can adaptively allocate expert retention capacity based on the actual collaborative value density of each layer. This avoids the problem of critical layers being over-compressed or redundant layers retaining too many resources, ensuring the stability and rationality of the overall network topology after pruning.

[0059] 3. Significantly reduces resource consumption and breaks through the bottleneck of embedded deployment.

[0060] This invention directly targets the expert module with structured trimming, which can substantially reduce the number of parameters residing in the model and the amount of video memory / memory used. This feature effectively overcomes the "memory wall" limitation faced by domestic embedded devices and edge nodes, reduces memory access bandwidth pressure and power consumption budget, and makes it possible to efficiently and stably deploy large-scale sparse MoE models in resource-constrained environments.

[0061] 4. It balances engineering efficiency and hardware compatibility, making it easy to integrate and deploy.

[0062] This invention enables collaborative value assessment through forward inference based on a calibration dataset, eliminating the need for complex multi-round retraining or additional fine-tuning processes, thus significantly reducing engineering implementation costs. Furthermore, this invention incorporates control logic linked to target hardware resource constraints, allowing the output pruning results to be directly converted into configuration instructions for deployment toolchains, demonstrating excellent hardware friendliness and automated integration capabilities. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0064] Figure 1 This is a flowchart of a hybrid expert model collaborative pruning method for embedded deployment provided in an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of the collaborative contribution calculation process provided in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of hierarchical expert selection under quality coverage constraints provided in an embodiment of the present invention;

[0067] Figure 4 This is a framework diagram of a hybrid expert model collaborative pruning system for embedded deployment provided in an embodiment of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example 1

[0070] This invention discloses a hybrid expert model collaborative pruning method for embedded deployment, referring to... Figure 1 As shown, it includes the following steps:

[0071] S1. Obtain the pre-trained sparse hybrid expert model to be pruned and the calibration dataset, and determine the target retention rate based on the resource constraint parameters of the target deployment hardware;

[0072] S2. Use the model to perform offline forward inference on the calibration dataset, record the expert routing results and gating statistics of each token activation in each hybrid expert layer, and construct the expert combination support set actually observed.

[0073] S3. Treat the expert support set as the observation alliance space of empirical cooperative game and define the alliance empirical utility;

[0074] S4. For each expert in each hybrid expert layer, based on the set of observation combinations they have participated in and the co-occurrence statistics of the sub-combinations after removing the expert, calculate the sub-combination proxy utility corrected by monotonicity constraints, and calculate the expert collaborative contribution value based on the difference between the original alliance experience utility and the sub-combination proxy utility.

[0075] S5. Perform non-negative truncation on the collaborative contribution values ​​of experts in each hybrid expert layer and calculate the total positive contribution quality; and determine the maximum feasible quality coverage threshold that makes the global retention rate lower than the target retention rate through binary search.

[0076] S6. Perform hierarchical minimum prefix selection based on the threshold to obtain the final set of retained experts, delete the experts that were not retained, and export the pruned hybrid expert model.

[0077] This embodiment applies "a hybrid expert model collaborative pruning method for embedded deployment" to a smart camera equipped with a domestically produced edge AI chip. The model collaborative pruning method achieves high-precision real-time target detection and behavior analysis, while solving the problems of limited memory resources and strict power consumption budget of the device.

[0078] The smart camera in this embodiment uses a typical domestically produced embedded SoC chip with an NPU computing power of 4 TOPS, an on-chip SRAM (storage) of only 4MB, and limited external DDR bandwidth. The original model was a visual Transformer based on the MoE architecture, used to distinguish complex scenarios such as "wearing a safety helmet," "smoke and fire," and "personnel intrusion." This model had 12 layers, including 6 MoE layers, with 16 experts in each layer. The original model had a huge number of parameters, and loading them all to DDR would lead to frequent cache replacements, resulting in an inference latency of up to 200ms, which could not meet the requirements of real-time monitoring.

[0079] This embodiment addresses the memory wall problem in the deployment of sparse hybrid expert models. It utilizes a small number of task-related calibration samples to perform offline forward inference on the unpruned model, recording the top-k expert routing trajectories of each hybrid expert layer. The observed expert combinations are considered as the support set of an alliance based on empirical cooperative game theory. The alliance's empirical utility is calculated using combination activation frequency, gated weighted statistics, or task-weighted statistics. A scaled-corrected proxy utility is constructed by combining the co-occurrence statistics of sub-combinations after removing the target expert, and Shapley-style expert collaborative contribution values ​​are calculated accordingly. Hierarchical expert selection is then performed according to the α-coverage criterion of positive contribution quality at each layer. A binary search is used to satisfy the target global retention rate or target resource budget, removing unretained experts and deriving the pruned model. This embodiment eliminates the need for additional retraining or loss re-evaluation, reducing the size of resident expert parameters, memory usage, and deployment costs while maintaining stable task performance. It is suitable for domestic embedded platforms, edge devices, and heterogeneous acceleration platforms.

[0080] The execution steps of collaborative pruning in this embodiment specifically include:

[0081] S1. Model acquisition, runtime environment initialization, and target constraint setting.

[0082] This embodiment first obtains the pre-trained sparse hybrid expert model to be deployed, the target task data, and the resource constraint information of the target embedded deployment hardware; and identifies the position, layer number, and number of experts in each layer of all MoE layers in the model. First, the number of experts (k) activated per token and the parameter subgraph corresponding to each expert are defined. Second, the inference environment is initialized and the model is set to inference mode. Finally, the constraints of the target embedded deployment hardware are converted into the target retention rate (r), target parameter limit, or target memory limit used in the subsequent pruning process.

[0083] This embodiment reads the model structure definition, weight parameter file, word segmenter, inference configuration, and routing configuration; disables random perturbation in the training state to ensure stable and reproducible routing behavior; and jointly estimates the resident usage of basic parameters, the usage of the expert pool, and the runtime cache usage to more accurately map the hardware budget to executable pruning targets.

[0084] The specific calculation method is based on the total budget of the target hardware. Resident overhead of basic parameters Expert pool parameter overhead and runtime caching overhead Determine the target retention rate This ensures that the pruned model satisfies resource constraints. When the target hardware provides memory budget, bandwidth budget, or latency budget simultaneously, multiple candidate retention rate intervals can be calculated separately first. Then, the intersection interval that satisfies all constraints or the most stringent target retention rate can be selected to improve the certainty of the deployment results.

[0085] S2, Calibration dataset construction and task distribution alignment.

[0086] Before deployment, this embodiment selects a calibration dataset containing various typical scenarios (such as nighttime construction and dense crowds) based on the target task. The calibration dataset in this embodiment preferably contains 20 to 100 task-related samples, more preferably 20 to 50, and even more preferably approximately 25 samples, to achieve a balance between statistical stability and computational cost. If joint deployment for multiple tasks is required, multiple calibration subsets can be constructed according to task type, and a unified pruning decision can be generated through weighted fusion.

[0087] S3. Route trajectory acquisition and expert combination observation space construction.

[0088] This embodiment uses an unpruned model to perform forward inference on the calibration dataset, recording the top-k expert routing results for each token at each MoE layer, thereby obtaining the expert portfolio set divided by layer and the frequency of occurrence of each expert portfolio. Since the theoretical total number of portfolios is C(n,k), while the actual number of portfolios occurring under the real task distribution is far less than this theoretical upper limit, this embodiment only performs statistical modeling on the support set of observed portfolios.

[0089] In addition to recording the expert ID set, this embodiment also records the average gating weight vector, sample category, context length, and timestamp information corresponding to the combination, so as to be used in subsequent construction of more granular combined experience utility.

[0090] For the task In the The expert support set obtained from the observations at each layer is expressed by the following formula:

[0091]

[0092] in, Indicates the first A group of experts at the level Indicates in the task C corresponds to the top-k expert portfolio support set observed in the calibration dataset. j Let represent the j-th observed expert combination, J represent the number of observed expert combinations, and |C j |=k indicates that each expert portfolio contains k experts activated by the route. Compared to the theoretically possible consortia, the size of the expert portfolio support set is usually much smaller than the total number of consortia, making it more suitable for rapid statistics and analysis before deployment.

[0093] S4, Definition of Alliance Experience Utility.

[0094] From the perspective of empirical cooperative game theory, this embodiment will... The expert portfolio is considered as a set of players, the expert portfolio support set is considered as the observational coalition space, and the number of times each expert portfolio appears on the calibration dataset is defined as the coalition experience utility. .

[0095] In this embodiment, the number of times each expert group is selected in calibration inference, the cumulative value of gated weighting, or the cumulative value after task weight correction are defined as the alliance experience utility. This alliance experience utility is used to approximate the degree to which the pre-trained model allocates conditional computational resources to each alliance under the target task conditions.

[0096] In multi-task weighting, the first Layered expert team The utility of alliance experience can be expressed by the following formula:

[0097]

[0098] in, Indicates task The corresponding task weight, Indicates task In the Layer-by-layer expert combination The count statistics, gated weighted statistics, or normalized cumulative values ​​are used, where T represents the total number of tasks participating in the weighted aggregation. In this way, this embodiment integrates the routing trajectories collected under different task scenarios into the same contribution evaluation framework.

[0099] To reduce statistical overhead, the expert portfolio support set in this embodiment is preferably stored using a sparse mapping table with sorted expert ID tuples as keys, and an inverted index is maintained for each expert or each sub-portion to support fast retrieval of expert portfolios that "contain a specified expert" or "contain a specified sub-portion".

[0100] S5, Sub-combination Agent Utility Construction.

[0101] This embodiment refers to the experts in expert group C. First, construct a removal expert The remaining sub-combinations Then, count the number of all observed combinations that contain the remaining subcombinations to obtain the original co-occurrence statistic. Furthermore, to avoid overestimating the utility of sub-combinations, for experts... Calculate the scaling factor And based on this, the modified sub-combination agency utility is obtained. .

[0102] The scaling factor and proxy utility can be expressed as:

[0103]

[0104]

[0105] in, Indicates the first Layered experts The scaling factor, Indicates the first Layered experts The set of observation combinations Indicates removal of experts The original co-occurrence statistics of the sub-combinations. This indicates the modified sub-combination proxy utility.

[0106] The above modifications ensure a more conservative estimate of the agency utility of the sub-combination, thus avoiding overestimation of the extent to which the remaining experts can maintain the original combination utility after removing a certain expert; the scaling factor corresponds to satisfying the requirement of including all experts. The maximum feasible correction coefficient for observation combinations that do not result in out-of-bounds utility of surrogate sub-combinations is obtained, thereby improving the stability of subsequent Shapley style attribution.

[0107] S6. Calculation of expert collaborative contribution value.

[0108] This embodiment calculates the marginal contribution of an expert in different observation combinations based on the difference between the original coalition's experience utility and the sub-combination proxy utility after removing the target expert. It then averages the contributions of all observation combinations the expert has participated in to obtain the expert's Shapley-style collaborative contribution value in the corresponding layer. .

[0109] Reference Figure 2 As shown, the observation combination A+B+C begins with a complete "expert combination" consisting of experts A, B, and C. To evaluate the contribution of expert A, it is first removed from the combination. After removing expert A, a "residual sub-combination" is obtained, consisting only of experts B and C. The performance of the complete combination (A+B+C) is represented by the empirical utility U(A+B+C), which can be understood as a measure of the accuracy, confidence, or other performance metrics brought by this expert combination when the model is processing a specific task or data. The performance of the residual sub-combination after removing expert A is represented by the empirical utility U'(B+C). By calculating the difference between the two utility values, the collaborative contribution value φ of expert A is obtained. A This difference precisely quantifies the marginal performance gain that expert A brings to the entire team, given that B and C are present. If φ A A large value indicates that A is crucial to the combination; conversely, a small value indicates that A's role is relatively minor.

[0110] The collaborative contribution value of each hybrid expert layer is expressed by the following formula:

[0111]

[0112] Among them, in the formula This approach, which reflects the proportional apportionment of marginal utility of the alliance by experts within the alliance, can reflect both the frequency of expert participation in the observation alliance and their irreplaceability in the specific alliance context.

[0113] Subsequently, in this embodiment, collaborative contribution values ​​less than zero are truncated to zero, denoted as... And calculate the total positive contribution quality of the current layer. Then the th layer... The total positive contribution quality and quality coverage constraints of a layer are expressed by the following formula:

[0114]

[0115]

[0116] in, Indicates non-negative collaborative contribution. Let n represent the total positive contribution quality, and n represent the th... The total number of experts at each level.

[0117] S7, Layered Quality Coverage Selection Mechanism.

[0118] In this embodiment, each layer of hybrid expert is assigned a non-negative collaborative contribution. Experts are sorted from largest to smallest, and selected sequentially until the cumulative non-negative collaborative contributions of the selected experts reach a preset proportion α of the total positive contribution quality of that layer. Compared with a scheme that retains the same number of experts in each layer, this mechanism can retain a small number of key experts for layers with highly concentrated contributions, and retain more experts for layers with dispersed contributions, thereby avoiding layer collapse and structural imbalance.

[0119] Reference Figure 3 As shown, in this embodiment, experts in the current layer are first sorted from highest to lowest based on their non-negative collaborative contributions. Then, starting from the top of the sorted list, they are added to the candidate set sequentially while accumulating their contributions simultaneously. When the accumulated contribution first reaches a preset proportion of the total positive contribution quality of the layer, selection stops, and the experts in the current prefix constitute the set of experts to be retained for that layer. Thus, for layers with highly concentrated contributions, only a small number of high-value experts need to be retained to cover the main collaborative quality; for layers with more dispersed contributions, more experts are retained to balance deployment budget and the stability of the layer structure.

[0120] If the truncated contribution values ​​of all experts in a certain layer are all zero, the system can adopt a strategy of sorting by activation frequency, sorting by gating mean, or retaining at least a preset number of experts to prevent the layer from being completely deleted under special task distributions.

[0121] S8, Threshold search under global retention rate constraint.

[0122] Since different quality coverage thresholds can lead to variations in the number of experts retained at each layer, this embodiment iteratively adjusts the α value using a binary search approach to ensure that the global retention rate corresponding to the total number of experts retained across all MoE layers does not exceed the target retention rate. If the current global retention rate is higher than the target retention rate, the α value is decreased; conversely, the α value is increased until the difference between the upper and lower bounds of the threshold is less than a preset tolerance or the deviation between the global retention rate and the target retention rate is less than a preset tolerance.

[0123] The global retention rate corresponding to the preset ratio α is expressed by the formula:

[0124]

[0125] in, Indicates the th value under α. The layer-generated set of retained experts Indicates the first The number of original experts at each layer. Binary search optimally returns results that satisfy... The maximum feasible quality coverage threshold is determined to preserve the quality of intra-layer collaborative value as much as possible without exceeding the resource budget.

[0126] In one alternative implementation, the global retention rate can be determined by the ratio of the total number of experts retained across all layers to the original total number of experts; the binary search preferably returns the maximum feasible quality coverage threshold that satisfies the target retention rate constraint, so as to retain as much intra-layer collaborative value quality as possible without exceeding the resource budget.

[0127] In a preferred implementation, a binary search returns the maximum feasible quality coverage threshold that satisfies the global retention rate constraint, so as to retain as much collaborative value quality as possible across layers without exceeding the hardware resource budget. Contribution estimation, quality coverage selection, and threshold search are all performed solely based on routing trajectories and co-occurrence statistics collected during calibration inference, without requiring additional forward loss verification on each candidate retention set.

[0128] Furthermore, to avoid a certain layer being completely pruned due to abnormal quality distribution or insufficient task samples, a minimum number of experts to be retained can be set for each hybrid expert layer. For hybrid structure models with shared experts, common routing modules, or non-MoE dense sub-layers, it is preferable to perform the above selection process only on the pruningable routing expert pool, while keeping the shared modules and dense paths unpruned.

[0129] S9, Pruning Execution, Structure Rewriting, and Model Export Deployment.

[0130] In this embodiment, based on the final retention results, the parameter tensors corresponding to the experts not retained are deleted, the expert index mapping relationship is reconstructed, the target number table output by the router to the expert subnetwork is modified, the retained experts are renumbered, and the model configuration file is rewritten. Subsequently, the pruned model is requantized and compiled and deployed to the edge chip to complete model conversion, operator optimization, graph compilation, and target hardware deployment.

[0131] The deployment and access process includes at least the following: exporting the pruned model into a unified exchange format or the framework's native format; performing operator replacement, graph simplification, or memory layout optimization on the MoE-related subgraphs; completing graph compilation and execution plan generation based on the target hardware instruction set or runtime library; performing deployment verification on the target device; and statistically analyzing peak memory usage, inference latency, and task performance metrics.

[0132] From the perspective of implementation complexity, the statistical and contribution estimation in this embodiment mainly relies on the size of the actual observed combinatorial support set and the inclusion relationship between the combinatorial and sub-combinations, rather than on the complete enumeration of the theoretical combinatorial set. Therefore, in most deployment scenarios where tasks are relatively concentrated, the overall offline analysis overhead can be controlled within an acceptable range for engineering purposes, and the processing time can be further shortened by layer-by-layer parallelism or task-by-task parallelism.

[0133] Compared to the unpruned model, the pruned model can significantly reduce the number of expert parameters residing and the storage overhead required for deployment while maintaining basic stability of task performance, thereby improving the deployability of the model on embedded platforms.

[0134] Example 2

[0135] This invention discloses a hybrid expert model collaborative pruning system for embedded deployment, employing the method described in any of Embodiment 1, with reference to... Figure 4 As shown, it includes the following modules:

[0136] Acquisition module: used to acquire the pre-trained sparse hybrid expert model to be pruned and the calibration dataset, and to determine the target retention rate based on the resource constraint parameters of the target embedded deployment hardware;

[0137] The routing trajectory acquisition module is used to perform offline forward inference on the calibration dataset using the model, record the routing results of each token activation expert and gating statistics of each hybrid expert layer, and construct the expert combination support set actually observed.

[0138] The Module for Constructing Empirical Cooperative Games is used to view the expert portfolio support set as the observation alliance space for empirical cooperative games and to define the alliance's empirical utility.

[0139] The module for constructing the utility of the sub-combination of agents and evaluating collaborative contributions is used to calculate the sub-combination agent utility corrected for monotonicity constraints for each expert in each hybrid expert layer, based on the set of observation combinations they have participated in and the co-occurrence statistics of the sub-combinations after removing the expert. It also calculates the expert collaborative contribution value based on the difference between the original alliance experience utility and the sub-combination agent utility.

[0140] Resource constraint matching module: used to perform non-negative truncation on the collaborative contribution values ​​of experts in each hybrid expert layer and calculate the total positive contribution quality; and to determine the maximum feasible quality coverage threshold that makes the global retention rate lower than the target retention rate through binary search;

[0141] Model Export Module: Used to perform hierarchical minimum prefix selection based on a threshold to obtain the final set of retained experts, delete the experts that were not retained, and export the pruned hybrid expert model.

[0142] This embodiment is used for collaborative pruning of hybrid expert models, specifically including:

[0143] The calibration set construction module is used to build a calibration dataset that matches the distribution of the target task;

[0144] The routing trajectory acquisition module is used to record the top-k expert combinations and their statistical information for each hybrid expert layer;

[0145] The Experience Cooperative Game Theory Builder module is used to construct the observation alliance support set and generate the alliance's experience utility.

[0146] A proxy sub-composite utility building module for generating scaled-corrected proxy sub-composite utilities;

[0147] A collaborative contribution assessment module based on Shapley value approximation is used to calculate the collaborative contribution value of each expert in each layer.

[0148] The α quality coverage selection module is used to generate a hierarchical expert retention set under global retention rate constraints.

[0149] The resource constraint matching module is used to map the target hardware resource budget to the model pruning target retention rate or resource constraints.

[0150] The model export and deployment module is used to output the pruned, compact hybrid expert model and connect it to the target hardware deployment toolchain.

[0151] The target hardware platform in this embodiment includes one or more of the Rockchip RK3588 platform, Huawei Ascend310 platform, and Cambricon MLU370 platform; the compilation and deployment toolchain can be called as an expert compression module in the model optimization stage, or as an independent compression preprocessing module before the model is connected to the compiler.

[0152] For hybrid structural models that simultaneously include routing expert pools, shared experts, and dense residual branches, this invention preferably performs collaborative contribution evaluation and hierarchical quality coverage selection only on task-related pruningable routing expert pools, while maintaining an unpruned state for shared experts, router common parameters, and non-MoE dense sublayers. This implementation can achieve the memory gains from expert pool pruning while maintaining basic expressive power and post-training structural consistency.

[0153] This embodiment also validates the obtained pruning model. Deployment validation includes at least one or more of the following: task performance, peak memory usage, parameter residency size, average inference latency, and energy consumption per unit task, to evaluate the balance between resource consumption and task performance of the pruning model.

[0154] In one validation scenario, three sparse MoE backbone models—Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite—were selected and validated on tasks including mathematical reasoning, code generation, knowledge-based question answering, authenticity assessment, sequence labeling, and medical question answering. Routing trajectories for each task were collected using approximately 25 calibration samples.

[0155] In the above verification scenario, when the system executes the method of the present invention with a 20% expert pruning ratio, the average task performance of Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite is similar to the performance ratio when the method of the present invention is executed with a 40% expert pruning ratio, indicating that the present invention can still maintain good task robustness under medium to high pruning intensity.

[0156] Furthermore, when compared with global sorting pruning or layer-by-layer fixed quota pruning, the present invention can better maintain the collaborative value quality of each layer through the layered α quality coverage mechanism, and can especially suppress the performance drop caused by the imbalance of inter-layer resource allocation in the 40% pruning scenario.

[0157] Regarding the verification of memory utilization benefits: Under 32K and 8K context lengths, peak memory usage was measured for the pruned models obtained using the method of this invention. It was observed that as the pruning ratio increased from 0% to 20% and 40%, the peak memory usage of different MoE backbone models showed a stable decreasing trend. This indicates that memory overhead during deployment is mainly affected by the resident size of the expert pool, and this invention can reduce expert resident overhead without additional training or architectural modifications.

[0158] Application instructions for resource-constrained deployment scenarios. When the target device's GPU memory budget cannot accommodate the entire expert pool, the resource constraints can first be converted into a target retention rate based on the device's available GPU memory, parameter resident capacity, and target latency limit. Then, the method of this invention can be used to generate a compact MoE model that meets the budget requirements. For various tasks such as code generation, mathematical reasoning, and knowledge question answering, this invention can complete pre-deployment expert pruning without relying on additional training, thereby shortening the preparation cycle for integrating the model into the actual business system.

[0159] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0160] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A hybrid expert model collaborative pruning method for embedded deployment, characterized in that, Includes the following steps: S1. Obtain the pre-trained sparse hybrid expert model to be pruned and the calibration dataset, and determine the target retention rate based on the resource constraint parameters of the target embedded deployment hardware; S2. Using the model, perform offline forward inference on the calibration dataset, record the expert routing results and gating statistics for each token activation in each hybrid expert layer, and construct the expert combination support set actually observed. S3. Treat the expert portfolio support set as the observation alliance space of empirical cooperative game and define the alliance empirical utility; S4. For each expert in each hybrid expert layer, based on the set of observation combinations they have participated in and the co-occurrence statistics of the sub-combinations after removing the expert, calculate the sub-combination proxy utility corrected by monotonicity constraints, and calculate the expert collaborative contribution value based on the difference between the original alliance experience utility and the sub-combination proxy utility. S5. Perform non-negative truncation on the expert collaborative contribution values ​​of each hybrid expert layer and calculate the total positive contribution quality; and determine the maximum feasible quality coverage threshold that makes the global retention rate lower than the target retention rate through binary search. S6. Perform hierarchical minimum prefix selection based on the threshold to obtain the final set of retained experts, delete the experts that were not retained, and export the pruned hybrid expert model.

2. The method as described in claim 1, characterized in that, Step S1 specifically includes: Obtain the pre-trained sparse hybrid expert model to be pruned, and identify all hybrid expert layers in the model, the number of experts in each hybrid expert layer, the number of experts activated by each token, and the parameter mapping relationship corresponding to each expert; Obtain the calibration dataset corresponding to the target task, which is divided into multiple calibration subsets based on the task type; Obtain the resource constraint parameters of the target embedded deployment hardware, and determine the target retention rate or target resource limit based on the resource constraint parameters; the determination method is based on the total budget of the target hardware. Resident overhead of basic parameters Expert pool parameter overhead and runtime caching overhead Determine the target retention rate This ensures that the pruned model satisfies resource constraints. .

3. The method as described in claim 2, characterized in that, Step S2 specifically includes: The model is used to perform offline forward inference on the calibration dataset. The sample number, token position, hybrid expert layer number, top-k expert number set and corresponding gating weight are recorded during the inference process. The top-k expert routing results, activation frequency and gating statistics of each token in each hybrid expert layer are also statistically analyzed. The top-k expert routing results of each token in each hybrid expert layer are deduplicated and aggregated. The same expert combinations that appear repeatedly in the same layer are aggregated into a mapping table of combination identifier and activation frequency. The actual observed expert combination support set is constructed layer by layer.

4. The method as described in claim 3, characterized in that, Step S3 specifically includes: S3. Treat the expert portfolio support set as the observation alliance space of empirical cooperative game and define the alliance empirical utility; The expert portfolio support set is regarded as the observation alliance space of empirical cooperative game, and the number of times the expert portfolio appears, the gating weighted result, or the task weighted statistics are defined as the alliance empirical utility. No. Layered hybrid expert layer expert combination The utility of alliance experience can be expressed by the following formula: in, Indicates task The corresponding task weight, Indicates task In the Layer-by-layer expert combination The count statistics, gated weighted statistics, or normalized cumulative values ​​are given, where T represents the total number of tasks participating in the weighted aggregation.

5. The method as described in claim 4, characterized in that, Step S4 specifically includes: S41. For each expert in each hybrid expert layer, construct a set of observation combinations that they have participated in, and based on the inclusion relationship statistics of the sub-combinations in the observation alliance space after removing the corresponding experts, establish the original co-occurrence statistics of the sub-combinations. S42. Calculate the scaling factor of the corresponding expert based on the original alliance experience utility and the original co-occurrence statistics of the sub-combination to obtain the sub-combination proxy utility corrected by monotonicity constraints, and calculate the Shapley-style expert collaborative contribution value based on the difference between the original alliance experience utility and the sub-combination proxy utility.

6. The method as described in claim 5, characterized in that, In step S42, for the first Experts in a hybrid expert layer The scaling factor is expressed by the formula: The sub-combination agency utility is expressed by the following formula: in, Indicates the first Layered experts The scaling factor, Indicates the first Layered experts The set of observation combinations Indicates removal of experts The original co-occurrence statistics of the sub-combinations. This indicates the modified sub-combination proxy utility.

7. The method as described in claim 6, characterized in that, In step S42, the collaborative contribution value is expressed by the formula: in, This represents the collaborative contribution value of each hybrid expert layer.

8. The method as described in claim 7, characterized in that, Step S5 specifically includes: The collaborative contribution values ​​of experts in each hybrid expert layer are non-negatively truncated, and expressed by the formula: The total positive contribution mass is calculated using the following formula: in, Indicates non-negative collaborative contribution. Indicates the total positive contribution quality. Indicates the first The total number of experts at each level; For each layer of hybrid experts, sort them from largest to smallest according to the non-negative collaborative contribution, and select experts in turn until the cumulative non-negative collaborative contribution of the selected experts reaches the preset proportion α of the total positive contribution quality of the layer. By iteratively adjusting α using a binary search approach, the maximum feasible quality coverage value is achieved, where the global retention rate corresponding to the total number of experts retained in all hybrid expert layers is lower than the target retention rate. This is to avoid layer collapse and maintain the balance of interlayer structure.

9. A hybrid expert model collaborative pruning system for embedded deployment, employing the method described in any one of claims 1-8, characterized in that, Includes the following modules: Acquisition module: used to acquire the pre-trained sparse hybrid expert model to be pruned and the calibration dataset, and to determine the target retention rate based on the resource constraint parameters of the target embedded deployment hardware; The routing trajectory acquisition module is used to perform offline forward inference on the calibration dataset using the model, record the routing results of each token activation expert and gating statistics of each hybrid expert layer, and construct the expert combination support set actually observed. The Module for Constructing Experienced Cooperative Game Theory is used to view the expert portfolio support set as the observation alliance space for experienced cooperative game theory and to define the alliance's empirical utility. The module for constructing the utility of the sub-combination of agents and evaluating collaborative contributions is used to calculate the sub-combination agent utility corrected for monotonicity constraints for each expert in each hybrid expert layer, based on the set of observation combinations they have participated in and the co-occurrence statistics of the sub-combinations after removing the expert. It also calculates the expert collaborative contribution value based on the difference between the original alliance experience utility and the sub-combination agent utility. Resource constraint matching module: used to perform non-negative truncation on the expert collaborative contribution values ​​of each hybrid expert layer and calculate the total positive contribution quality; and to determine the maximum feasible quality coverage threshold that makes the global retention rate lower than the target retention rate through binary search; Model export module: used to perform hierarchical minimum prefix selection based on the threshold to obtain the final set of retained experts, delete the experts that were not retained, and export the pruned hybrid expert model.

Images