A structured pruning and mixed precision quantization method based on a hybrid expert model

Abstract

The application discloses a structured pruning and mixed precision quantization method based on a hybrid expert model, and relates to the technical field of deep learning model compression and edge computing. The application calculates the importance score of experts by counting the activation frequency and average routing weight of each expert subnetwork in the hybrid expert model on the calibration dataset, combined with the field characteristics; performs differentiated structured pruning on the expert subnetworks according to the importance score, removes structural units such as attention heads and feedforward network channels; configures different quantization precisions for the routing gate network, core experts and low-frequency experts respectively, and restores the model precision through quantization-aware training and knowledge distillation; finally, the compressed model is converted into an edge inference format and deployed to a POCT edge computing device. The application solves the problems of large parameter quantity, high inference delay and high storage occupation of the hybrid expert model in the POCT mobile robot scene, and realizes efficient inference on the edge device.

Description

Technical Field

[0001] This invention relates to the field of deep learning model compression and edge computing technology, specifically to a method for joint optimization of hybrid expert models using structured pruning and hybrid precision quantization for POCT mobile robot edge computing devices. This invention is particularly applicable to POCT mobile robots, medical edge terminals, and embedded artificial intelligence devices with limited computing power, storage capacity, and power consumption. Background Technology

[0002] With the widespread application of deep learning in the medical field, multi-task medical models employing a Mixture of Experts (MoE) architecture have shown promising application prospects in POCT detection data parsing, task classification, and auxiliary information processing due to their ability to invoke different expert sub-networks based on different inputs. However, deploying such models to edge computing devices for POCT mobile robots faces the following prominent challenges: First, the computing power of mobile robot edge computing platforms is far lower than that of cloud servers, making it difficult to directly run complete MoE models. MoE models typically contain multiple expert subnetworks and routing gating networks, resulting in a large number of parameters and placing significant pressure on the processor and memory resources of edge devices.

[0003] Second, POCT testing tasks typically have high real-time requirements. If cloud inference is relied upon, network latency and network stability will affect the testing response time, which is unacceptable in scenarios such as emergency rooms.

[0004] Third, mobile robots are powered by batteries, and continuous operation of large-scale models will significantly increase energy consumption and shorten the working time of a single charge.

[0005] Fourth, the actual call frequency, routing weight, and contribution to different tasks vary significantly among different expert subnetworks in the MoE model. Applying a uniform compression ratio to all experts may result in accuracy loss for frequently used experts while retaining too many redundant parameters for low-frequency experts, making it impossible to achieve an optimal balance between compression efficiency and accuracy.

[0006] Fifth, while the traditional method of sparsifying only the model weights reduces the number of theoretical parameters, sparse weights cannot achieve corresponding computational acceleration on actual edge hardware because general-purpose inference engines usually require dense matrix operation structures.

[0007] Sixth, directly reducing the model's accuracy by using post-training quantization can easily lead to a significant decrease in recognition accuracy, classification accuracy, or output stability in POCT medical tasks.

[0008] Therefore, there is an urgent need for a model compression and deployment method that can adapt to PoCT edge computing scenarios, perform differentiated compression based on the characteristics of MoE model architecture, and take into account both model accuracy and inference acceleration. Summary of the Invention

[0009] To address the shortcomings of the existing technologies, a structured pruning and hybrid precision quantization method based on a hybrid expert model is provided for POCT edge computing. This method adapts to POCT edge computing scenarios, performs differentiated compression based on the characteristics of hybrid expert model architectures, and balances model accuracy and inference acceleration.

[0010] To address the aforementioned problems, this invention provides a structured pruning and hybrid precision quantization method based on a hybrid expert model, comprising: Obtain the hybrid expert model to be compressed and the calibration dataset for representing the POCT edge computing task. The hybrid expert model includes a routing gated network and multiple expert subnetworks. The calibration dataset is input into the hybrid expert model, and the activation frequency and average routing weight of each expert subnetwork activated by the routing gating network are statistically analyzed. Based on the activation frequency, average routing weight, and domain characteristics related to the POCT edge computing task domain of each expert subnetwork, the importance score of each expert subnetwork is calculated. Based on the importance score, a structured pruning strategy is determined for each expert subnetwork, and at least one structural unit in each expert subnetwork is removed; wherein, the structural unit specifically includes attention head, feedforward network channel, convolutional channel, intermediate layer or neuron group; Using the removed complete structural units as the object, the corresponding intermediate feature dimensions are adjusted after removal to maintain network dimension alignment, so that the pruned expert subnetwork maintains a dense structure; Different quantization accuracies are configured for the routing gated network, different expert sub-networks, and the embedding layer, with the quantization accuracies of the routing gated network and core experts being higher than those of the low-frequency experts. Quantization-aware training is performed on the models configured with the different quantization accuracies, and knowledge distillation is performed on the structured pruning and mixed-accuracies quantized models using an uncompressed hybrid expert model as the teacher model. The trained model is converted into a model format supported by edge inference devices and deployed to POCT edge computing devices.

[0011] Furthermore, the activation frequency is determined based on the ratio of the number of times each expert subnetwork is selected by the routing gating network during the calibration dataset input process to the total number of routing decisions.

[0012] Furthermore, the importance score is a weighted sum of activation frequency, average routing weight, and domain feature corresponding indicators; the domain feature corresponding indicator is the activation distribution entropy of the expert subnetwork under different task categories, and the activation distribution entropy is used to characterize the activation distribution characteristics of the expert subnetwork under different task categories.

[0013] Furthermore, based on the importance score, the expert subnetwork is divided into core experts, regular experts, and low-frequency experts, and different structured pruning ratios are configured for each.

[0014] Furthermore, the pruning rate of the core experts is 10% to 20%, the pruning rate of the regular experts is 30% to 40%, and the pruning rate of the low-frequency experts is 50% to 70%.

[0015] Furthermore, the routing gating network and core experts use INT8 quantization, the low-frequency experts use INT4 quantization, and the embedding layer uses FP16 quantization or maintains FP16 precision.

[0016] Furthermore, the quantization-aware training includes inserting pseudo-quantization nodes in the weight and / or activation value processing path of the model, performing forward simulation of quantization errors, and updating model parameters based on the backpropagation results.

[0017] Furthermore, the knowledge distillation includes aligning at least one of the output distributions of the teacher model and the student model, intermediate layer features, or routing distributions.

[0018] A hybrid expert model deployment system, characterized in that it includes: The calibration data management module is used to acquire the hybrid expert model to be compressed and the POCT task calibration dataset; The expert statistics module is used to calculate the activation frequency and average routing weight of each expert sub-network. An importance assessment module is used to calculate an expert importance score based on the activation frequency, average routing weight, and domain characteristics. A structured pruning module is used to perform differential removal of structural units in the expert subnetwork according to the importance score; A hybrid precision quantization module is used to configure different quantization precisions for routing gated networks, expert subnetworks, and the embedded layer; The quantization perception training module is used to perform quantization perception training on the quantization model; The knowledge distillation module is used to perform knowledge distillation on a compressed hybrid expert model using an uncompressed hybrid expert model. The model deployment module is used to convert the trained model into a format supported by edge inference devices and deploy it.

[0019] Furthermore, when the computer program runs on the processor, it causes the processor to perform the above-described method.

[0020] This invention provides a structured pruning and hybrid precision quantization method based on a hybrid expert model, which has the following beneficial effects: This invention employs a structured pruning approach to remove structural units from the expert subnetwork, including attention heads and feedforward network channels. The pruned network retains a dense structure, enabling it to be directly executed by edge hardware using conventional dense matrix operations, thereby achieving a computational acceleration in actual inference that matches the theoretical parameter reduction.

[0021] This invention enables the model to achieve low-latency, low-power local inference on the embedded AI computing module of the POCT mobile robot by converting the compressed model into an edge inference format and performing deployment optimization. This reduces the dependence on cloud network connections and improves the real-time detection response and local data processing capabilities. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0023] Figure 1 A flowchart of the method provided by the present invention; Figure 2 This is a schematic diagram of the system structure provided by the present invention. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0025] The following detailed description of the implementation method of the present invention is in conjunction with the accompanying drawings. The description is only a partial embodiment and not all embodiments. For clarity, representations and descriptions unrelated to the present invention are omitted in the drawings and description.

[0026] To provide a clearer understanding of the technical features, objectives, and beneficial effects of this invention, the following detailed description of the technical solution is provided. Obviously, the described embodiments are only a portion of the embodiments of this invention, not all of them, and should not be construed as limiting the scope of implementation of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the protection scope of this invention.

[0027] like Figure 1 As shown, this invention provides a structured pruning and hybrid precision quantization method based on a hybrid expert model, comprising: Obtain the hybrid expert model to be compressed and the calibration dataset for representing the POCT edge computing task. The hybrid expert model includes a routing gated network and multiple expert subnetworks. The calibration dataset is input into the hybrid expert model, and the activation frequency and average routing weight of each expert subnetwork activated by the routing gating network are statistically analyzed. Based on the activation frequency, average routing weight, and domain characteristics related to the POCT edge computing task domain of each expert subnetwork, the importance score of each expert subnetwork is calculated. Based on the importance score, a structured pruning strategy is determined for each expert subnetwork, and at least one structural unit in each expert subnetwork is removed; wherein, the structural unit specifically includes attention head, feedforward network channel, convolutional channel, intermediate layer or neuron group; Using the removed complete structural units as the object, the corresponding intermediate feature dimensions are adjusted after removal to maintain network dimension alignment, so that the pruned expert subnetwork maintains a dense structure; Different quantization accuracies are configured for the routing gated network, different expert sub-networks, and the embedding layer, with the quantization accuracies of the routing gated network and core experts being higher than those of the low-frequency experts. Quantization-aware training is performed on the models configured with the different quantization accuracies, and knowledge distillation is performed on the structured pruning and mixed-accuracies quantized models using an uncompressed hybrid expert model as the teacher model. The trained model is converted into a model format supported by edge inference devices and deployed to POCT edge computing devices.

[0028] Furthermore, the activation frequency is determined based on the ratio of the number of times each expert subnetwork is selected by the routing gating network during the calibration dataset input process to the total number of routing decisions.

[0029] Furthermore, the importance score is a weighted sum of activation frequency, average routing weight, and domain feature corresponding indicators; the domain feature corresponding indicator is the activation distribution entropy of the expert subnetwork under different task categories, and the activation distribution entropy is used to characterize the activation distribution characteristics of the expert subnetwork under different task categories.

[0030] Furthermore, based on the importance score, the expert subnetwork is divided into core experts, regular experts, and low-frequency experts, and different structured pruning ratios are configured for each.

[0031] Furthermore, the pruning rate of the core experts is 10% to 20%, the pruning rate of the regular experts is 30% to 40%, and the pruning rate of the low-frequency experts is 50% to 70%.

[0032] Furthermore, the routing gating network and core experts use INT8 quantization, the low-frequency experts use INT4 quantization, and the embedding layer uses FP16 quantization or maintains FP16 precision.

[0033] Furthermore, the quantization-aware training includes inserting pseudo-quantization nodes in the weight and / or activation value processing path of the model, performing forward simulation of quantization errors, and updating model parameters based on the backpropagation results.

[0034] Furthermore, the knowledge distillation includes aligning at least one of the output distributions of the teacher model and the student model, intermediate layer features, or routing distributions.

[0035] A hybrid expert model deployment system, such as Figure 2 As shown, the feature is that it includes: The calibration data management module is used to acquire the hybrid expert model to be compressed and the POCT task calibration dataset; The expert statistics module is used to calculate the activation frequency and average routing weight of each expert sub-network. An importance assessment module is used to calculate an expert importance score based on the activation frequency, average routing weight, and domain characteristics. A structured pruning module is used to perform differential removal of structural units in the expert subnetwork according to the importance score; A hybrid precision quantization module is used to configure different quantization precisions for routing gated networks, expert subnetworks, and the embedded layer; The quantization perception training module is used to perform quantization perception training on the quantization model; The knowledge distillation module is used to perform knowledge distillation on a compressed hybrid expert model using an uncompressed hybrid expert model. The model deployment module is used to convert the trained model into a format supported by edge inference devices and deploy it.

[0036] Furthermore, when the computer program runs on the processor, it causes the processor to perform the above-described method.

[0037] Example 1: Compressed Deployment of MoE Model for POCT Detection Data Parsing This embodiment uses a MoE model for POCT test data parsing as the object to be compressed. The MoE model includes a routing gating network and 12 expert subnetworks, covering different specialty areas such as emergency triage, laboratory test interpretation, chronic disease management, medication guidance, traditional Chinese medicine diagnosis, and rare disease screening. The model also includes a shared embedding layer and an output layer.

[0038] Step 1: Obtain the calibration dataset. Collect representative testing data and task classification data from the hospital information system and POCT testing equipment to construct a calibration dataset containing approximately 50,000 samples, covering various practical deployment scenarios such as emergency care, routine testing, and chronic disease follow-up.

[0039] Step 2: Statistical Analysis of Expert Features. Input the calibration dataset into the original MoE model, recording the expert ID and corresponding routing weight selected in each routing decision. After inference using the full calibration data, the activation frequency Fi and average routing weight Wi of each expert are statistically obtained. For example, the activation frequency of an emergency triage expert is approximately 0.35, and the average routing weight is approximately 0.42; while the activation frequency of a rare disease screening expert is approximately 0.02, and the average routing weight is approximately 0.08.

[0040] Step 3: Calculate Importance Scores. The importance score for each expert is calculated using the formula Ii = αFi + βWi + γGi, where α = 0.5, β = 0.3, and γ = 0.2. The domain characteristic index Gi is calculated based on the activation distribution entropy of experts across different task categories; the more concentrated the distribution (the higher the specificity), the smaller the Gi value. Calculation results: Emergency triage expert Ii ≈ 0.39, test interpretation expert Ii ≈ 0.33, chronic disease management expert Ii ≈ 0.21, traditional Chinese medicine diagnosis expert Ii ≈ 0.08, rare disease screening expert Ii ≈ 0.04.

[0041] The domain feature corresponding index can be represented by the activation distribution entropy of the expert subnetwork on different task categories, and the activation distribution entropy can be used as the domain feature index in the importance score.

[0042] Step 4: Differentiated Structured Pruning. The 12 experts were divided into three groups based on their importance scores: Core experts (Ii≥0.25): Emergency triage expert, laboratory interpretation expert, intensive care expert—pruning rate 15%, retaining 85% of structural units; Routine experts (0.10≤Ii<0.25): 5 experts including chronic disease management expert, medication guidance expert, and infection screening expert—pruning rate 35%, retaining 65% of structural units; Low-frequency experts (Ii<0.10): 4 experts including TCM syndrome differentiation expert and rare disease screening expert - pruning rate 60%, retaining 40% of structural units.

[0043] During the pruning process, some attention heads in the multi-head attention mechanism and some channels in the feedforward network within each expert sub-network are structurally removed. After removal, the intermediate feature dimensions of the network are adjusted accordingly to maintain dimensional alignment.

[0044] When performing structured pruning based on expert importance scores, the objects removed are complete attention heads, feedforward network channels, convolutional channels, intermediate layers, or groups of neurons. After removal, the corresponding intermediate feature dimensions are adjusted to maintain dimensional alignment between adjacent network structures and to enable the pruned network to continue using a dense structure for edge reasoning.

[0045] Step 5: Hybrid precision quantization. Routing gating networks and core experts use INT8 quantization (symmetric quantization, per order of magnitude), regular experts use INT8 quantization, low-frequency experts use INT4 quantization (symmetric quantization, per channel level), and the shared embedding layer maintains FP16 precision.

[0046] Step 6: Quantization-Aware Training. Based on the quantization configuration in Step 5, pseudo-quantization nodes are inserted into the model. Quantization-aware fine-tuning training is performed using a calibration dataset for 5 training epochs with an initial learning rate of 1e-4. The forward propagation simulates the quantization and dequantization process of INT8 / INT4, while the backpropagation updates the original full-precision weights.

[0047] Step 7: Knowledge Distillation. The original uncompressed MoE model is used as the teacher model, and the model trained in Step 6 is used as the student model. The distillation loss includes the KL divergence loss of the output soft label, the MSE alignment loss of the intermediate layer features, and the JS divergence loss of the route distribution. The temperature parameter is set to 4, and the distillation training epochs are 10.

[0048] Step 8: Edge Deployment. The trained compressed model is converted to ONNX format, and after operator fusion and memory optimization using the inference engine, it is deployed to the embedded AI computing module of the POCT mobile robot. After deployment, the model's parameter count is approximately 25% of the original model, and the inference latency is reduced to approximately 30% of the original model, meeting the real-time requirements of the POCT mobile robot for detection data parsing tasks.

[0049] It should be noted that the specific values ​​in the above embodiments (including the number of experts, pruning rate, quantization accuracy, training rounds, etc.) are only used to illustrate the technical solution of the present invention and do not constitute a limitation on the scope of protection of the present invention. In practical applications, the above parameters can be adaptively adjusted according to the specific MoE model structure, POCT task requirements, and edge hardware capabilities.

[0050] This invention achieves differentiated compression based on the actual usage characteristics of experts by statistically analyzing the activation frequency and routing weights of each expert subnetwork in the MoE model and calculating their importance scores. This approach avoids the practice of applying a uniform compression ratio to all experts in existing technologies, allowing high-frequency key experts to retain more parameters and low-frequency experts to achieve a higher compression ratio, thus achieving a better balance between the overall model compression rate and the accuracy of key tasks.

[0051] This invention employs a structured pruning approach to remove structural units from the expert subnetwork, including attention heads and feedforward network channels. The pruned network retains a dense structure, enabling it to be directly executed by edge hardware using conventional dense matrix operations, thereby achieving a computational acceleration in actual inference that matches the theoretical parameter reduction.

[0052] This invention configures differentiated hybrid precision quantization strategies based on the varying precision sensitivities of routing gating networks, core experts, and low-frequency experts. This allows key modules to maintain high precision while compressing secondary modules to a greater extent. Combined with quantization-aware training, quantization errors are simulated and specifically optimized during training, effectively reducing the precision loss caused by quantization.

[0053] Knowledge distillation is performed using the uncompressed MoE model as the teacher model. By aligning the output distribution, intermediate layer features, and routing distribution, the model performance loss caused by structured pruning and mixed precision quantization is further recovered.

[0054] This invention enables the model to achieve low-latency, low-power local inference on the embedded AI computing module of the POCT mobile robot by converting the compressed model into an edge inference format and performing deployment optimization. This reduces the dependence on cloud network connections and improves the real-time detection response and local data processing capabilities.

[0055] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A structured pruning and hybrid precision quantization method based on a hybrid expert model, characterized in that, include: Obtain the hybrid expert model to be compressed and the calibration dataset for representing the POCT edge computing task. The hybrid expert model includes a routing gated network and multiple expert subnetworks. The calibration dataset is input into the hybrid expert model, and the activation frequency and average routing weight of each expert subnetwork activated by the routing gating network are statistically analyzed. Based on the activation frequency, average routing weight, and domain characteristics related to the POCT edge computing task domain of each expert subnetwork, the importance score of each expert subnetwork is calculated. Based on the importance score, a structured pruning strategy is determined for each expert subnetwork, and at least one structural unit in each expert subnetwork is removed; wherein, the structural unit specifically includes attention head, feedforward network channel, convolutional channel, intermediate layer or neuron group; Using the removed complete structural units as the object, the corresponding intermediate feature dimensions are adjusted after removal to maintain network dimension alignment, so that the pruned expert subnetwork maintains a dense structure; Different quantization accuracies are configured for the routing gated network, different expert sub-networks, and the embedding layer, with the quantization accuracies of the routing gated network and core experts being higher than those of the low-frequency experts. Quantization-aware training is performed on the models configured with the different quantization accuracies, and knowledge distillation is performed on the structured pruning and mixed-accuracies quantized models using an uncompressed hybrid expert model as the teacher model. The trained model is converted into a model format supported by edge inference devices and deployed to POCT edge computing devices.

2. The method according to claim 1, characterized in that, The activation frequency is determined based on the ratio of the number of times each expert subnetwork is selected by the routing gating network during the calibration dataset input process to the total number of routing decisions.

3. The method according to claim 1, characterized in that, The importance score is a weighted sum of activation frequency, average routing weight, and domain feature corresponding indicators; the domain feature corresponding indicator is the activation distribution entropy of the expert subnetwork under different task categories, and the activation distribution entropy is used to characterize the activation distribution characteristics of the expert subnetwork under different task categories.

4. The method according to claim 3, characterized in that, Based on the importance score, the expert subnetwork is divided into core experts, regular experts, and low-frequency experts, and different structured pruning ratios are configured for each.

5. The method according to claim 4, characterized in that, The pruning rate for core experts is 10% to 20%, the pruning rate for regular experts is 30% to 40%, and the pruning rate for low-frequency experts is 50% to 70%.

6. The method according to claim 1, characterized in that, The routing gating network and core experts use INT8 quantization, the low-frequency experts use INT4 quantization, and the embedded layer uses FP16 quantization or maintains FP16 precision.

7. The method according to claim 1, characterized in that, The quantization-aware training includes inserting pseudo-quantization nodes into the weight and / or activation value processing path of the model, performing forward simulation of quantization errors, and updating model parameters based on the backpropagation results.

8. The method according to claim 1, characterized in that, The knowledge distillation includes aligning at least one of the output distributions of the teacher model and the student model, intermediate layer features, or routing distributions.

9. A hybrid expert model deployment system for implementing the method according to any one of claims 1 to 8, characterized in that, include: The calibration data management module is used to acquire the hybrid expert model to be compressed and the POCT task calibration dataset; The expert statistics module is used to calculate the activation frequency and average routing weight of each expert sub-network. An importance assessment module is used to calculate an expert importance score based on the activation frequency, average routing weight, and domain characteristics. A structured pruning module is used to perform differential removal of structural units in the expert subnetwork according to the importance score; A hybrid precision quantization module is used to configure different quantization precisions for routing gated networks, expert subnetworks, and the embedded layer; The quantization perception training module is used to perform quantization perception training on the quantization model; The knowledge distillation module is used to perform knowledge distillation on a compressed hybrid expert model using an uncompressed hybrid expert model. The model deployment module is used to convert the trained model into a format supported by edge inference devices and deploy it.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run on a processor, it causes the processor to perform the method according to any one of claims 1 to 8.

Images