Large model compression method and device, task processing method and device, and storage medium
By performing layer-by-layer quantization and routing calibration on the large MoE model, combined with dynamic pruning, the problem of low running efficiency of large MoE sparse architecture models on lightweight devices is solved, and efficient operation on lightweight devices is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-15
- Publication Date
- 2026-03-03
AI Technical Summary
Existing model compression methods are not suitable for large models in the MoE sparse architecture, resulting in high demand for video memory resources and difficulty in running efficiently on lightweight devices.
By performing layer-by-layer quantization and route calibration on the large model of the MoE architecture, the importance of expert sub-models in the target task is evaluated, and dynamic pruning is performed to reduce the participation of unimportant expert sub-models in the computation.
While ensuring model performance, it reduces the memory and computational overhead required for large model inference, making it suitable for lightweight devices with limited video memory resources.
Smart Images
Figure CN120952084B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of model compression technology, and in particular to a method, apparatus, task processing method, device and storage medium for large model compression. Background Technology
[0002] Large Language Models (LLMs), also known as large language models or large-scale language models, have demonstrated exceptional capabilities in general language understanding. The Mixture of Experts (MoE) architecture is an advanced neural network architecture designed to improve overall model performance through the predictions of multiple models or experts. Its core idea is to distribute input data to different expert sub-models and then merge the outputs of all sub-models to generate the final result. Large models under the MoE architecture, with their sparse structure, can be effectively pre-trained with fewer computational resources, thus significantly increasing the model size, while achieving the same quality level as dense models during inference by activating only a small number of parameters.
[0003] Although the MoE architecture significantly reduces the number of activation parameters used during inference, its overall parameter count remains large, resulting in high demands on GPU memory resources and making it difficult to run efficiently on lightweight devices. Currently, most existing model compression methods are designed for large language models with dense architectures and are not suitable for the sparse architecture of MoE. Summary of the Invention
[0004] This invention provides a large model compression method, apparatus, task processing method, device, and storage medium to address the shortcomings of existing model compression methods that are not suitable for the sparse architecture of MoE, where large models in the MoE architecture have high requirements for video memory resources and are difficult to run efficiently on lightweight devices.
[0005] This invention provides a method for compressing large models, comprising the following steps:
[0006] The linear layers of the initial large model to be compressed are quantized layer by layer to obtain the first large model; the initial large model is a large language model built based on an expert hybrid architecture and pre-trained.
[0007] The routing of each expert sub-model in the first large model is calibrated to obtain the second large model;
[0008] During the inference process of the second major model, the importance of each expert sub-model in the task type is evaluated based on the task type of the target task to be performed;
[0009] Based on the aforementioned importance, each expert sub-model is dynamically pruned to compress the second major model.
[0010] According to the large model compression method provided by the present invention, the evaluation of the importance of each expert sub-model on the task type based on the task type of the target task to be executed includes:
[0011] Obtain the frequency information of each expert sub-model being selected in terms of the task type of the target task to be executed;
[0012] Calculate the average frequency at which each expert sub-model is selected for the task type based on the frequency information;
[0013] The pruning conditions are obtained by multiplying the average frequency by the preset pruning threshold.
[0014] The frequency information of each expert sub-model is compared with the pruning conditions, and the importance of each expert sub-model in the task type is determined based on the comparison results.
[0015] According to the large model compression method provided by the present invention, the dynamic pruning of each expert sub-model based on the importance includes:
[0016] Based on the aforementioned importance, determine which expert sub-models need to be skipped in each of the aforementioned expert sub-models;
[0017] Dynamic pruning is performed on expert sub-models that need to be skipped.
[0018] According to the large model compression method provided by the present invention, the step of performing route calibration on each expert sub-model in the first large model to obtain the second large model includes:
[0019] A calibration set is randomly selected from the training set of the initial large model, and the first output of each expert sub-model in the initial large model to the calibration set is obtained.
[0020] The calibration set is input into the first large model to obtain the second output of each expert sub-model in the first large model to the calibration set;
[0021] Based on the output difference between the first output and the second output, the routing calibration of each expert sub-model in the first large model is performed to obtain the second large model.
[0022] According to the large model compression method provided by the present invention, the step of performing route calibration on each expert sub-model in the first large model based on the output difference between the first output and the second output to obtain the second large model includes:
[0023] Based on the first output, the output score of each expert sub-model in the initial large model is calculated; the output score of the target expert sub-model represents the proportion of the output of the target expert sub-model on the calibration set in the total output, the total output is obtained by weighted summation of the outputs of each expert sub-model in the initial large model on the calibration set, and the target expert sub-model is any one of the expert sub-models in the initial large model;
[0024] The expert sub-models in the initial large model are sorted according to the output scores, and the K first expert sub-models with the highest output scores are determined according to the sorting order; K is a positive integer.
[0025] Based on the difference between the outputs of the first output and the second output, and the difference in the target output corresponding to the first expert sub-model, the offset loss of the first expert sub-model is calculated.
[0026] In the first large model, the offset loss is backpropagated to update the routing weights of the first expert sub-model in the first large model, so as to perform routing calibration on each expert sub-model in the first large model and obtain the second large model.
[0027] According to the large model compression method provided by the present invention, the step of performing layer-by-layer quantization on the linear layers of the initial large model to be compressed to obtain a first large model includes:
[0028] The linear layers of the initial large model to be compressed are quantized layer by layer, and the weight parameters of each linear layer are quantized into integer variables to obtain the first large model.
[0029] The linear layer includes a multi-head attention layer and an expert hybrid layer, and the initial large model also includes an expert routing layer. The expert hybrid layer has multiple expert sub-models, and the expert routing layer contains the routing weight matrix of the expert sub-models. The expert routing layer is used to select routes for the expert sub-models in the expert hybrid layer based on the routing weight matrix.
[0030] The present invention also provides a task processing method, comprising:
[0031] Retrieve the text information corresponding to the task to be processed;
[0032] The text information is input into the target large model, and the target text output by the target large model is obtained to obtain the processing result of the task to be processed.
[0033] The target large model is obtained by compressing the initial large model using any of the large model compression methods described above.
[0034] The present invention also provides a large model compression device, comprising the following modules:
[0035] The quantization compression module is used to quantize the linear layers of the initial large model to be compressed layer by layer to obtain the first large model; the initial large model is a large language model built based on an expert hybrid architecture and pre-trained.
[0036] The routing calibration module is used to perform routing calibration on each expert sub-model in the first large model to obtain the second large model.
[0037] The importance assessment module is used to assess the importance of each expert sub-model in the task type based on the task type of the target task to be performed during the inference process of the second major model.
[0038] A dynamic pruning module is used to dynamically prune each of the expert sub-models based on their importance in order to compress the second large model.
[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the large model compression method as described above, or the task processing method as described above.
[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the large model compression method as described above, or the task processing method as described above.
[0041] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the large model compression method as described above, or the task processing method as described above.
[0042] The large model compression method, apparatus, task processing method, device, and storage medium provided by this invention achieve initial model compression by performing layer-by-layer quantization on the linear layers of a pre-trained initial large model under an expert hybrid architecture. Furthermore, it reduces performance degradation caused by quantization through routing calibration of expert sub-models. During large model inference, it dynamically prunes expert sub-models based on their importance to the target task type, reducing the possibility of unimportant expert sub-models participating in computation, thereby further compressing the model. By combining static quantization and dynamic pruning on the initial large model under this sparse expert hybrid architecture, and performing routing calibration based on quantization to reduce quantization loss, and dynamically pruning based on task importance during the inference stage, it can reduce the memory and computational overhead required for large model inference while ensuring model performance. This is beneficial for the efficient operation of large models on lightweight devices with limited GPU memory resources and meets the low-power requirements of such devices. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0044] Figure 1 This is a flowchart illustrating the large model compression method provided by the present invention.
[0045] Figure 2 This is a schematic diagram of the quantization and expert route calibration process provided by the present invention.
[0046] Figure 3 This is a schematic diagram of the pruning process provided by the present invention.
[0047] Figure 4 This is a flowchart illustrating the task processing method provided by the present invention.
[0048] Figure 5 This is a schematic diagram of the structure of the large model compression device provided by the present invention.
[0049] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0051] This invention provides a large model compression method for compressing large models with a MoE architecture (hereinafter referred to as MoE large models). It should be noted that the MoE large model is similar to the dense large model in that it is based on the Transformer architecture. The difference lies in that the expert hybrid layer (MoE layer) of the MoE large model replaces the forward propagation layer of the dense large model. Each MoE layer of the MoE large model contains multiple expert sub-models. For each input vector token x, the routing probability r={r0,…,rN-1} and the expert selection score s=Softmax(r) of each expert sub-model are calculated. The top K expert sub-models with the highest scores are selected, and the outputs z of these K expert sub-models are weighted and summed to obtain the final output of the large model.
[0052] It should be noted that the large MoE model to be compressed mentioned in the various embodiments of the present invention can be applied to edge devices with limited video memory resources. These edge devices include smart home devices, specifically applied to interactive assistants in smart home devices, providing users with functions such as device control and information query.
[0053] The initial large model is pre-trained based on user interaction data from smart home devices. The user interaction data is text information, including text information input by the user, and may also include text information obtained by converting the user's voice control information into text.
[0054] Edge devices such as smart home devices, as terminals that directly provide diversified services to users, have limited resources available for large models. If large models consume too many resources, it may affect the use of smart home devices. Therefore, it is necessary to quantize and compress the deployed large models into lightweight models so as to achieve efficient operation of large models on edge devices without affecting the use of device functions.
[0055] This invention provides a compression method combining static quantization and dynamic pruning, specifically addressing the unique mechanism of the MoE large model. Figure 1 This is a flowchart illustrating the large model compression method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:
[0056] Step 100: Quantize the linear layers of the initial large model to be compressed layer by layer to obtain the first large model; the initial large model is a large language model built on an expert hybrid architecture and pre-trained.
[0057] Step 200: Perform route calibration on each expert sub-model in the first large model to obtain the second large model;
[0058] Step 300: During the inference process of the second major model, the importance of each expert sub-model in the task type is evaluated based on the task type of the target task to be executed.
[0059] Step 400: Dynamically prune each of the expert sub-models based on their importance to compress the second large model.
[0060] For the initial large model to be compressed, the linear layers of the initial large model are quantized layer by layer, thereby performing preliminary compression of the initial large model to obtain the first large model. The quantization of the linear layers is to reduce the number of parameters in the linear layers. The initial large model is a large language model built on the expert hybrid architecture MoE and pre-trained.
[0061] The routing calibration of each expert sub-model in the first quantized model is performed to obtain the second quantized model. The routing calibration of the expert sub-models is to ensure that the quantized large model and the unquantized large model can have the same output, thereby ensuring that the quantized model has the same performance.
[0062] During the second major model inference process, based on the task type of the target task to be executed, the importance of each expert sub-model in the task type is evaluated. Based on the importance of the expert sub-model to the target task to be executed, the expert sub-models are dynamically pruned, thereby further compressing the second major model to obtain the compressed target major model.
[0063] Compression methods for quantization and pruning of large language models in the Transformer architecture are primarily aimed at dense, large models. However, because they fail to consider the characteristics of expert routing and sparse activation expert mechanisms in the MoE model, they cannot achieve satisfactory compression results on MoE models. Firstly, in terms of quantization, quantization methods for dense, large models only focus on reducing quantization errors in linear layers, neglecting the fact that quantization loss in the MoE model can shift the expert selection weights in the expert routing output, leading to biased expert selection and ultimately severe performance degradation. Secondly, in terms of pruning, pruning for dense, large models often targets specific parameters of linear layers. However, the sparse activation expert mechanism in the MoE model offers the possibility of pruning from an expert perspective, potentially achieving higher pruning rates and less performance loss.
[0064] In the MoE large-scale model, the training objective is to enable each expert sub-model to master knowledge of different types of tasks. Simultaneously, training allows expert routing to select the most suitable expert sub-model based on the current input. Therefore, expert routing is the core of the MoE large-scale model; selecting an expert sub-model unsuitable for the current input will lead to severe performance degradation. Furthermore, quantization errors introduced by quantization often cause significant expert selection bias. Additionally, due to the generally uneven distribution of expert load during MoE model pre-training, mainstream MoE architectures pre-trained large language models exhibit redundancy in expert sub-model selection and computation, meaning some selected expert sub-models are less important to the current task, increasing computational latency.
[0065] Based on this, after quantizing the initial large model, the expert sub-models are route-calibrated. Furthermore, during the model inference process, the expert sub-models are dynamically pruned based on their importance to the task type of the target task to be executed. This reduces the possibility of less important expert sub-models participating in the computation, thereby compressing the large model and reducing model computational overhead and latency.
[0066] The target task is to identify the user's interaction intent based on the user's current input interaction data, generate an inference task based on the user's interaction intent, and then have the large model execute this inference task, outputting text information corresponding to the task processing result based on the user's interaction intent. Optionally, if the user's interaction intent is device control, the target task is a code generation task, and the corresponding text information is the control command for the smart home device; if the user's interaction intent is information query, the target task is a common sense question-and-answer task, and the corresponding text information is the query text obtained based on the user's query requirements.
[0067] During the inference process of executing the target task, the large model prunes expert sub-models with lower importance based on the importance of each expert sub-model in the current target task, skipping the inference calculation of unimportant expert sub-models, thereby reducing computational overhead and ensuring the efficient operation of the large model on edge devices such as smart homes.
[0068] In this embodiment, preliminary model compression is achieved by performing layer-by-layer quantization on the linear layers of the pre-trained initial large model under the expert hybrid architecture. Furthermore, performance degradation caused by quantization is reduced through routing calibration of the expert sub-models. During the large model inference process, expert sub-models are dynamically pruned based on their importance to the target task type, reducing the likelihood of unimportant expert sub-models participating in computation, thus further compressing the model. By combining static quantization and dynamic pruning on the initial large model under this sparse expert hybrid architecture, routing calibration is performed on the basis of quantization to reduce quantization loss, and dynamic pruning is performed during the inference stage based on the importance of the task. This approach reduces the memory and computational overhead required for large model inference while ensuring model performance, which is beneficial for the efficient operation of large models on lightweight devices with limited GPU memory resources and meets the device's low-power requirements.
[0069] In one embodiment, layer-by-layer quantization of the linear layers of the initial large model involves quantizing the weight parameters of the linear layers into low-bit integer variables, thereby reducing the number of parameters in the linear layers. Based on this, step 100 includes:
[0070] Step 101: Perform layer-by-layer quantization on the linear layers of the initial large model to be compressed, and quantize the weight parameters of each linear layer into integer variables to obtain the first large model;
[0071] The linear layer includes a multi-head attention layer and an expert hybrid layer, and the initial large model also includes an expert routing layer. The expert hybrid layer has multiple expert sub-models, and the expert routing layer contains the routing weight matrix of the expert sub-models. The expert routing layer is used to select routes for the expert sub-models in the expert hybrid layer based on the routing weight matrix.
[0072] The initial large model's linear layers include a multi-head attention layer and an expert hybrid layer. The expert hybrid layer contains multiple expert sub-models. The initial large model also includes an expert routing layer, which contains the routing weight matrix of the expert sub-models. The expert routing layer selects routes for the expert sub-models in the expert hybrid layer based on the routing weight matrix of the expert sub-models.
[0073] Layer-by-layer quantization of the linear layers of the initial large model involves quantizing the weight parameters of the linear layers of the initial large model into integer variables, including quantizing the weight parameters of the multi-head attention layer into integer variables and quantizing the weight parameters of the expert hybrid layer into integer variables.
[0074] Based on the quantization of the initial large model, the route calibration is performed on the quantized large model so that the quantized large model has basically the same output as the unquantized large model, thereby ensuring model performance.
[0075] In one embodiment, step 200 includes:
[0076] Step 201: Randomly select a calibration set from the training set of the initial large model, and the first expert route of each expert sub-model in the initial large model to the calibration set;
[0077] Step 202: Input the calibration set into the first large model to obtain the second expert route of each expert sub-model in the first large model to the calibration set;
[0078] Step 203: Based on the routing differences between the first expert route and the second expert route, perform routing calibration on each expert sub-model in the first large model to obtain the second large model.
[0079] A calibration set is randomly selected from the training set of the initial large model, along with the first expert routes of each expert submodel in the initial large model to the calibration set. These first expert routes contain the expert route selections of the initial large model for each input sample in the calibration set. The calibration set is then input into the quantized first large model, resulting in the second expert routes of each expert submodel in the first large model to the calibration set. These second expert routes also contain the expert route selections of the first large model for each input sample in the calibration set.
[0080] Using the output route of the initial large model before quantization as the standard, and based on the route difference between the first expert route of the expert sub-model before quantization and the second expert route of the expert sub-model after quantization, the expert sub-model of the first large model is route-calibrated to obtain the second large model, ensuring the consistency of the output of the second large model with the initial large model.
[0081] Optionally, step 203 further includes:
[0082] Step 213: Calculate the output score of each expert sub-model in the initial large model based on the first output; the output score of the target expert sub-model represents the proportion of the output of the target expert sub-model on the calibration set in the total output, the total output is obtained by weighted summation of the outputs of each expert sub-model in the initial large model on the calibration set, and the target expert sub-model is any one of the expert sub-models in the initial large model;
[0083] Step 223: Sort the expert sub-models in the initial large model according to the output scores, and determine the K first expert sub-models with the highest output scores according to the sorting order; K is a positive integer;
[0084] Step 233: Based on the difference between the first output and the second output, and the difference between the target output of the first expert sub-model, calculate the offset loss of the first expert sub-model.
[0085] Step 243: Backpropagate the offset loss in the first large model to update the routing weights of the first expert sub-model in the first large model, so as to perform routing calibration on each expert sub-model in the first large model and obtain the second large model.
[0086] Based on the first output of the expert sub-model on the calibration set before quantization, the output score of each expert sub-model before quantization is calculated. For any expert sub-model, denoted as the target expert sub-model, its output score represents the proportion of its output on the calibration set in the total output. The total output is obtained by weighted summation of the outputs of each expert sub-model on the calibration set before quantization.
[0087] The output score of any expert sub-model on the calibration set is calculated as the proportion of its output on the calibration set to the total output of all expert sub-models on the calibration set before quantization. The output score of each expert sub-model on the calibration set before quantization is calculated in this way. The expert sub-models are then sorted according to their output scores, either in ascending or descending order, without any specific limitation.
[0088] Furthermore, the top K (K is a positive integer) expert sub-models with the highest output scores are selected as the first expert sub-model according to the sorting order. Based on the difference between the output of the first output and the second output, the difference between the target output before and after quantization of the first expert sub-model is calculated. The offset loss of the first expert sub-model is backpropagated in the first large model to update the routing weight of the first expert sub-model in the first large model. Thus, the routing of each expert sub-model in the first large model is calibrated to obtain the second large model.
[0089] In one embodiment, refer to Figure 2 The quantization and expert route calibration process shown is as follows: For the initial large model, its linear layer includes a multi-headed self-attention (MHSA) layer and an expert hybrid layer. The expert hybrid layer is configured with multiple expert sub-models (expert sub-models can also be simply referred to as experts). The MHSA layer and the expert hybrid layer are quantized layer by layer, and the weight parameters of the linear layer are quantized into integer variables to obtain the quantized MHSA layer and the expert hybrid layer.
[0090] Furthermore, to offset the expert selection bias caused by quantization loss, the expert routing of the expert hybrid layer is calibrated. Figure 2The paper exemplifies eight expert sub-models, from expert 1 to expert 8. To calibrate the expert routing selection of the quantized MoE model, the outputs of the expert routes of the large model before and after quantization are first collected on the calibration set. Using the output of the expert routes of the large model before quantization as the correct standard, the relative loss of the output of the expert routes of the large model after quantization is calculated. The gradient of the expert route weights relative to this loss is obtained through backpropagation. The weights of the expert routes of the large model after quantization are updated by using the AdamW optimizer and other methods, which can effectively offset the expert selection bias caused by the quantization loss.
[0091] exist Figure 2 The example further illustrates that for an input sample in the calibration set, the expert routes of the quantized large model select experts 2 and 4, while the unquantized large model selects experts 2 and 7 for the same input sample. Routing calibration of the quantized large model is performed based on the routing selection of the unquantized large model to obtain the calibrated expert routes. The routing calibration of the quantized large model is achieved by adjusting the routing weights of the expert sub-models on that input sample using the AdamW optimizer, etc.
[0092] For the offset loss of the quantized large model, expert sub-models selected in the expert routing before quantization but not in the expert routing after quantization for the same input sample are considered offset experts. Among these offset experts, most have high output scores in the expert routing of the quantized large model, but their corresponding loss proportions are low. This phenomenon indicates that if the mean squared error loss is directly used to calculate the offset loss of expert sub-models, most of the low-probability selected expert sub-models will account for the majority of the loss, introducing noise and interference into the optimization process. Therefore, by analyzing the sources of loss caused by offset expert offsets and the output score rankings of the offset expert sub-models before and after quantization, the offset loss is calculated based on the Top K expert sub-models in output score. This allows the optimization process to focus on aligning with expert sub-models that are likely to be selected.
[0093] Optionally, for the K expert sub-models with the highest output scores, the offset loss... The calculation method is shown in Formula 1 below:
[0094] (1)
[0095] In Formula 1, This indicates that among the K expert sub-models with the highest output scores, the k-th... Each expert sub-model on the input sample The first output, Let represent the K expert sub-models with the highest output scores, i.e., the th... Each expert sub-model on the input sample The second output, This indicates that among the K expert sub-models with the highest output scores, the k-th... Routing weights for each expert sub-model. and These are the outputs of the multi-head attention layer before and after quantization, and the input of the expert routing layer. In this embodiment, based on the output differences of the K expert sub-models with the highest output scores, the offset loss of the multi-head attention layer is calculated. This offset loss is backpropagated to adjust the routing weights of the expert sub-models in the expert routing layer, thereby offsetting the output differences from the multi-head attention layer and ensuring the consistency of the expert sub-models' outputs before and after quantization.
[0096] Under the condition of using the mean squared error loss of the expert sub-model with TopK output scores, the quantization method combined with expert route calibration effectively reduces the quantization loss and maintains the accuracy of the quantized large model well.
[0097] Furthermore, the dynamic pruning of expert sub-models during the inference phase is based on the importance of each expert sub-model to the task type of the target task to be executed. In step 300, the importance of each expert sub-model to the task type of the target task to be executed is evaluated, including:
[0098] Step 301: Obtain the frequency information of each expert sub-model being selected in terms of the task type of the target task to be executed;
[0099] Step 302: Calculate the average frequency of each expert sub-model being selected for the task type based on the frequency information;
[0100] Step 303: Calculate the product of the average frequency and the preset pruning threshold to obtain the pruning conditions;
[0101] Step 304: Compare the frequency information of each expert sub-model with the pruning conditions, and determine the importance of each expert sub-model in the task type based on the comparison results.
[0102] The process involves obtaining the frequency of selection for each expert sub-model on the target task type, calculating the average selection frequency of each sub-model on that task type, multiplying the average frequency by a preset pruning threshold to obtain the pruning condition, comparing the frequency of each sub-model on that task type with the pruning condition, and determining the importance of each sub-model on that task type based on the comparison results. The pruning threshold is a configurable constant and is an adjustment factor used for pruning, while the number of times each sub-model is selected on each task type is statistically obtained during the model pre-training phase.
[0103] Optionally, during the large model inference stage, the expert sub-model is dynamically pruned based on its importance in the current type of task. Step 400 may also include:
[0104] Step 401: Based on the importance, determine the expert sub-models that need to be skipped in each of the expert sub-models;
[0105] Step 402: Dynamically prune the expert sub-models that need to be skipped.
[0106] Based on the importance of each expert sub-model in the task type of the target task to be executed, the expert sub-models that need to be skipped are determined, and dynamic pruning is performed on the expert sub-models that need to be skipped.
[0107] To address the issue of uneven expert load distribution in the MoE pre-trained large model, analysis revealed that for any given task, some experts are frequently selected for computation, while others are rarely selected. Therefore, it can be determined that the expert selection pattern is strongly correlated with the task type. Specifically, by calculating the number of expert selections on multiple datasets across four common task types (mathematical reasoning, code generation, commonsense reasoning question answering, and specific languages), the cosine similarity between the number of expert selections on each pair of task types is calculated. The results show that on different datasets for the same type of task, the selection frequency of the MoE large model for expert sub-models is highly similar, while on datasets for different task types, the selection frequency varies significantly. This indicates that the selection frequency and importance of expert sub-models change with the task type.
[0108] Based on this, during the large model inference stage, according to the task type of the target task to be executed, the importance of each expert sub-model on the target task to be executed is inferred by obtaining the frequency information of the selection of expert sub-models on that type of task. Thus, expert sub-models with lower importance are dynamically pruned to reduce the computational overhead of those with low contribution to the target task.
[0109] For example, if each expert hybrid layer in the MoE large model has N expert sub-models, and the sequence length of the input samples in the current batch is... The preset pruning threshold is defined as (0 < ≤1), if a certain expert sub-model is selected ≤1 times The pruning conditions shown in Formula 2 must be met:
[0110] (2)
[0111] Then, the expert sub-model that meets the conditions is pruned; that is, if an expert sub-model is selected a certain number of times in the current task type to be executed, then the pruning is performed. It is less than the average number of times all expert sub-models are selected on the current task type to be performed and the pruning threshold. If the product of the two factors is considered, then the expert submodel is deemed to have low inference importance for the current task to be performed, and therefore the expert submodel is pruned to skip its inference computation.
[0112] In Formula 2, the sequence length is This indicates that the input sample contains Each of the given tokens is used to select the token with the highest score. A total of expert sub-models were selected. Each expert sub-model selects any token. The outputs of each expert sub-model are weighted and summed to obtain the model's output on that token. This is because different tokens are chosen... Each expert sub-model may be different. Therefore, when calculating the average selection frequency of the expert sub-model on the input sample sequence, the average value is calculated uniformly for all N expert sub-models.
[0113] In one embodiment, refer to Figure 3 The dynamic pruning process for expert sub-models, as shown, involves considering the text sequence corresponding to the current input text "Large Language Model" and the eight expert sub-models corresponding to experts 1 to 8. The number of times each expert sub-model is selected in the current task type is counted. For the current task to be executed, expert routing selects expert sub-models corresponding to experts 1, 5, and 8 for inference computation. Expert 5 is selected significantly less often than the average number of times the three selected expert sub-models are selected, satisfying the pruning condition. Therefore, expert 5's inference computation is skipped, and its output is not included in the weighted summation of the final output. This method can significantly reduce the model's inference latency and improve inference efficiency with minimal impact on the accuracy of the MoE large model.
[0114] In this embodiment, a combination of static quantization and dynamic pruning during the inference phase is used to achieve quantization compression of multiple expert sub-models in a large model. In static quantization, the weight parameters of the linear layers of the expert sub-models in the MoE large model are quantized into low-bit integer variables. Furthermore, expert routing is fine-tuned based on a small number of samples to ensure that the quantized expert selection is closer to the initial large model, thereby reducing quantization error. In dynamic pruning, inference computations of expert sub-models with lower importance in the current task type are skipped, thus reducing computational overhead. This compression method, combining static quantization and dynamic pruning, can reduce the memory and computational overhead required for inference of large models under the MoE architecture with minimal loss of model accuracy. It is particularly suitable for deployment of large models in environments with limited GPU memory resources, such as edge devices, meeting the low-power requirements of these devices.
[0115] Furthermore, traditional quantization methods only focus on reducing the quantization error of the linear layer itself, ignoring the fact that quantization loss in the MoE architecture model can cause a shift in the expert selection weights of the expert routing output, leading to biased expert selection and ultimately severe performance degradation. In this embodiment of the invention, expert routing calibration is used to reduce the rate of change in expert selection, thereby reducing quantization loss. Moreover, during expert routing calibration, the expert sub-model with the highest output score is selected for calculating the offset loss, avoiding the introduction of noise and interference during the calibration process and ensuring the performance of the quantized model.
[0116] Regarding pruning, pruning methods for dense models often prune specific parameters of linear layers. However, the sparse activation expert mechanism in the MoE architecture provides the possibility of pruning from the expert dimension, thereby achieving a higher pruning rate and less model performance loss. During the pre-training phase of the MoE architecture, there is some redundancy in expert selection and computation. Some selected expert sub-models have low importance to the task at hand, increasing computational latency. To address this, dynamic pruning of expert sub-models is performed during the inference phase, skipping the inference calculations of less important expert sub-models. This significantly reduces inference latency and improves inference efficiency with minimal impact on model accuracy.
[0117] Reference Figure 4 This invention also provides a task processing method, which includes:
[0118] S1, obtain the text information corresponding to the task to be processed;
[0119] S2, input the text information into the target large model, obtain the target text output by the target large model, and obtain the processing result of the task to be processed;
[0120] The target large model is obtained by compressing the initial large model using the large model compression method described in the above embodiments.
[0121] First, obtain the text information corresponding to the task to be processed, input the text information into the target large model, obtain the target text output by the target large model, and obtain the processing result of the task to be processed.
[0122] The target text is obtained by the target big model through reasoning on the input text information. It represents the reasoning result of the target big model on the input text information, that is, the processing result of the task to be processed corresponding to the text information.
[0123] The target large model is obtained by compressing the initial large model to be compressed using the large model compression method described in the above embodiments. That is, by using the model compression method described in the above embodiments, the initial large model is quantized and compressed through a combination of static quantization and dynamic pruning to obtain the target large model. The text information corresponding to the task to be processed is input into the target large model for inference calculation, and the target text output by the target large model is the processing result of the task to be processed.
[0124] Optionally, the target large model includes a multi-head attention layer, an expert routing layer, and an expert hybrid layer. The expert hybrid layer has multiple expert sub-models. When the target large model performs inference calculations on the text information of the task to be processed, it calculates the importance of each expert sub-model on the task type according to the task type of the task to be processed, performs dynamic pruning on the expert sub-models with lower importance, and performs weighted summation on the outputs of the expert sub-models with higher importance to obtain the target text.
[0125] Optionally, after obtaining the text information corresponding to the task to be processed, the text information can be converted into word vectors, and the converted word vectors can be input into the target large model for inference calculation to obtain the target text output by the target large model. There are no specific limitations on this.
[0126] It should be noted that the task processing method provided in this embodiment of the invention is applied to edge devices with limited video memory resources. These edge devices include, but are not limited to, smart home devices. A target large model is deployed on the edge device to realize the interactive functions of the smart home device. The target large model is obtained by quantization compression using the large model compression method described in the above embodiments. The quantization compression process of the target large model can be referred to in conjunction with the content described in the above embodiments, and will not be repeated here.
[0127] Furthermore, the text information corresponding to the task to be processed is obtained based on the user's input interaction data, and the task to be executed is generated based on this text information. The text information corresponding to the task to be processed is input into the target large model. The target large model recognizes the user's interaction intent, executes the inference task corresponding to the text information, and outputs the target text corresponding to the user's interaction intent, thus obtaining the processing result of the task to be executed.
[0128] Optionally, for smart home devices, the user's interaction intent can be device control, in which case the task to be processed is a code generation task. After recognizing the user's device control intent, the target large model performs inference calculations based on the user's control needs contained in the text information and outputs the target text, which is the control instruction for the smart home device. Alternatively, the user's interaction intent can be information query, in which case the task to be processed is a common sense reasoning question-and-answer task. After recognizing the user's information query intent, the target large model performs inference calculations based on the user's query needs contained in the text information and outputs the queried target text.
[0129] When the target large model performs the inference task corresponding to the task to be processed, it dynamically prunes each expert sub-model based on the importance of each expert sub-model in the task to be processed, skipping the inference calculation of text information by unimportant expert sub-models. Thus, based on the quantification of the target large model, the computational overhead required for inference can be further dynamically reduced during the inference process, ensuring the efficient operation of the target large model on edge devices and reducing the power consumption of edge devices.
[0130] In this embodiment, a compression method combining static quantization and dynamic pruning is used to quantize and compress the initial large model, resulting in a target large model that can be deployed and applied on edge devices. This target large model is used to process the tasks to be processed, reducing the number of parameters involved in the computation. Furthermore, during the inference computation process, expert sub-models with lower importance to the tasks to be processed can be dynamically pruned, further reducing the computational overhead of inference, lowering computational latency, and improving inference efficiency. This is beneficial for the deployment and application of the large model in environments with limited video memory resources, such as edge devices, thereby improving the task processing efficiency of edge devices and reducing device power consumption, meeting the device's low power consumption requirements.
[0131] The large model compression apparatus provided by the present invention is described below. The large model compression apparatus described below and the large model compression method described above can be referred to in correspondence.
[0132] Reference Figure 5 This invention provides a large model compression device, comprising:
[0133] The quantization compression module 10 is used to quantize the linear layers of the initial large model to be compressed layer by layer to obtain the first large model; the initial large model is a large language model built based on an expert hybrid architecture and pre-trained.
[0134] The routing calibration module 20 is used to perform routing calibration on each expert sub-model in the first large model to obtain the second large model.
[0135] Importance assessment module 30 is used to assess the importance of each expert sub-model in the task type based on the task type of the target task to be performed during the inference process of the second major model.
[0136] The dynamic pruning module 40 is used to dynamically prune each of the expert sub-models based on the importance, so as to compress the second large model.
[0137] In one embodiment, the importance assessment module 30 is further configured to:
[0138] Obtain the frequency information of each expert sub-model being selected in terms of the task type of the target task to be executed;
[0139] Calculate the average frequency at which each expert sub-model is selected for the task type based on the frequency information;
[0140] The pruning conditions are obtained by multiplying the average frequency by the preset pruning threshold.
[0141] The frequency information of each expert sub-model is compared with the pruning conditions, and the importance of each expert sub-model in the task type is determined based on the comparison results.
[0142] In one embodiment, the dynamic pruning module 40 is further configured to:
[0143] Based on the aforementioned importance, determine which expert sub-models need to be skipped in each of the aforementioned expert sub-models;
[0144] Dynamic pruning is performed on expert sub-models that need to be skipped.
[0145] In one embodiment, the routing calibration module 20 is further configured to:
[0146] A calibration set is randomly selected from the training set of the initial large model, and the first output of each expert sub-model in the initial large model to the calibration set is obtained.
[0147] The calibration set is input into the first large model to obtain the second output of each expert sub-model in the first large model to the calibration set;
[0148] Based on the output difference between the first output and the second output, the routing calibration of each expert sub-model in the first large model is performed to obtain the second large model.
[0149] In one embodiment, the routing calibration module 20 is further configured to:
[0150] Based on the first output, the output score of each expert sub-model in the initial large model is calculated; the output score of the target expert sub-model represents the proportion of the output of the target expert sub-model on the calibration set in the total output, the total output is obtained by weighted summation of the outputs of each expert sub-model in the initial large model on the calibration set, and the target expert sub-model is any one of the expert sub-models in the initial large model;
[0151] The expert sub-models in the initial large model are sorted according to the output scores, and the K first expert sub-models with the highest output scores are determined according to the sorting order; K is a positive integer.
[0152] Based on the difference between the outputs of the first output and the second output, and the difference in the target output corresponding to the first expert sub-model, the offset loss of the first expert sub-model is calculated.
[0153] In the first large model, the offset loss is backpropagated to update the routing weights of the first expert sub-model in the first large model, so as to perform routing calibration on each expert sub-model in the first large model and obtain the second large model.
[0154] In one embodiment, the quantization compression module 10 is further configured to:
[0155] The linear layers of the initial large model to be compressed are quantized layer by layer, and the weight parameters of each linear layer are quantized into integer variables to obtain the first large model.
[0156] The linear layer includes a multi-head attention layer and an expert hybrid layer, and the initial large model also includes an expert routing layer. The expert hybrid layer has multiple expert sub-models, and the expert routing layer contains the routing weight matrix of the expert sub-models. The expert routing layer is used to select routes for the expert sub-models in the expert hybrid layer based on the routing weight matrix.
[0157] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute steps of the large model compression method, such as:
[0158] The linear layers of the initial large model to be compressed are quantized layer by layer to obtain the first large model; the initial large model is a large language model built based on an expert hybrid architecture and pre-trained.
[0159] The routing of each expert sub-model in the first large model is calibrated to obtain the second large model;
[0160] During the inference process of the second major model, the importance of each expert sub-model in the task type is evaluated based on the task type of the target task to be performed;
[0161] Based on the aforementioned importance, each expert sub-model is dynamically pruned to compress the second major model.
[0162] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0163] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to perform the steps of the large model compression method provided by the above methods, for example including:
[0164] The linear layers of the initial large model to be compressed are quantized layer by layer to obtain the first large model; the initial large model is a large language model built based on an expert hybrid architecture and pre-trained.
[0165] The routing of each expert sub-model in the first large model is calibrated to obtain the second large model;
[0166] During the inference process of the second major model, the importance of each expert sub-model in the task type is evaluated based on the task type of the target task to be performed;
[0167] Based on the aforementioned importance, each expert sub-model is dynamically pruned to compress the second major model.
[0168] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the large model compression method provided by the above methods, for example including:
[0169] The linear layers of the initial large model to be compressed are quantized layer by layer to obtain the first large model; the initial large model is a large language model built based on an expert hybrid architecture and pre-trained.
[0170] The routing of each expert sub-model in the first large model is calibrated to obtain the second large model;
[0171] During the inference process of the second major model, the importance of each expert sub-model in the task type is evaluated based on the task type of the target task to be performed;
[0172] Based on the aforementioned importance, each expert sub-model is dynamically pruned to compress the second major model.
[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A large model compression method, characterized in that, The method comprises the following steps: linear layers of an initial large model to be compressed are quantized layer by layer to obtain a first large model; the initial large model is a large language model constructed based on an expert hybrid architecture and pre-trained; each expert sub-model in the first large model is subjected to routing calibration to obtain a second large model; in an inference process of the second large model, the importance of each expert sub-model on a task type of a target task to be executed is evaluated based on the task type; each expert sub-model is dynamically pruned based on the importance to compress the second large model; the routing calibration of each expert sub-model in the first large model to obtain a second large model comprises the following steps: a calibration set is randomly selected from a training set of the initial large model, and a first output of each expert sub-model in the initial large model on the calibration set is obtained; the calibration set is input into the first large model to obtain a second output of each expert sub-model in the first large model on the calibration set; each expert sub-model in the first large model is subjected to routing calibration based on output differences between the first output and the second output to obtain a second large model; the routing calibration of each expert sub-model in the first large model to obtain a second large model based on the output differences between the first output and the second output comprises the following steps: an output score of each expert sub-model in the initial large model is calculated based on the first output; the output score of a target expert sub-model represents a proportion of an output of the target expert sub-model on the calibration set in an output total sum, and the output total sum is obtained by weighting and summing the outputs of each expert sub-model in the initial large model on the calibration set; the target expert sub-model is any one of the expert sub-models in the initial large model; each expert sub-model in the initial large model is sorted according to the output scores, and K first expert sub-models with the highest output scores are determined according to the sorting order; K is a positive integer; an offset loss of the first expert sub-model is calculated based on a target output difference corresponding to the first expert sub-model in the output differences between the first output and the second output; the offset loss is back propagated in the first large model to update routing weights of the first expert sub-model in the first large model, so as to calibrate the routing of each expert sub-model in the first large model to obtain a second large model.
2. The large model compression method of claim 1, wherein, the importance of each expert sub-model on a task type of a target task to be executed is evaluated based on the task type, which comprises the following steps: frequency information of each expert sub-model being selected on the task type of the target task to be executed is obtained; an average frequency of each expert sub-model being selected on the task type is calculated based on the frequency information; a product of the average frequency and a preset pruning threshold is calculated to obtain a pruning condition; the frequency information of each expert sub-model is compared with the pruning condition, and the importance of each expert sub-model on the task type is determined according to a comparison result.
3. The large model compression method of claim 2, wherein, the dynamic pruning of each expert sub-model based on the importance comprises the following steps: determine an expert sub-model that needs to be skipped in each of the expert sub-models based on the importance; perform dynamic pruning on the expert sub-model that needs to be skipped.
4. The large model compression method of claim 1, wherein, The linear layer of the initial large model to be compressed is quantized layer by layer to obtain a first large model, including: The linear layer of the initial large model to be compressed is quantized layer by layer to obtain a first large model, including: The linear layer includes a multi-head attention layer and an expert mixing layer, and the initial large model further includes an expert routing layer; the expert mixing layer is provided with a plurality of expert sub-models, and the expert routing layer includes a routing weight matrix of the expert sub-models, and the expert routing layer is used for routing selection of the expert sub-models in the expert mixing layer based on the routing weight matrix.
5. A task processing method characterized by, including: obtaining text information corresponding to a to-be-processed task; inputting the text information into a target large model to obtain a target text output by the target large model, and obtaining a processing result of the to-be-processed task; The target large model is obtained by compressing an initial large model using the large model compression method of any one of claims 1 to 4.
6. A large model compression apparatus, characterized by, including: The linear layer of the initial large model to be compressed is quantized layer by layer to obtain a first large model; The initial large model is a large language model constructed based on an expert mixing architecture and pre-trained; The routing calibration module is configured to perform routing calibration on each expert sub-model in the first large model to obtain a second large model. The importance evaluation module is configured to evaluate the importance of each expert sub-model for a task type of a target task to be executed in an inference process of the second large model. The dynamic pruning module is configured to perform dynamic pruning on each expert sub-model based on the importance to compress the second large model. The routing calibration module is further configured to: randomly select a calibration set from a training set of the initial large model, and obtain a first output of each expert sub-model in the initial large model on the calibration set; input the calibration set into the first large model to obtain a second output of each expert sub-model in the first large model on the calibration set; and perform routing calibration on each expert sub-model in the first large model based on an output difference between the first output and the second output to obtain a second large model. The routing calibration module is further configured to: calculate an output score of each expert sub-model in the initial large model based on the first output; the output score of a target expert sub-model represents a proportion of an output of the target expert sub-model in the calibration set in an output sum, the output sum is obtained by weighted summation of the outputs of each expert sub-model in the initial large model on the calibration set, and the target expert sub-model is any one of the expert sub-models in the initial large model; sort the expert sub-models in the initial large model according to the output scores, and determine K first expert sub-models with the highest output scores according to the sorting order; K is a positive integer; calculate an offset loss of the first expert sub-model based on a target output difference corresponding to the first expert sub-model in an output difference between the first output and the second output; and perform back propagation on the offset loss in the first large model, update a routing weight of the first expert sub-model in the first large model, and perform routing calibration on the expert sub-models in the first large model to obtain a second large model.
7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the large model compression method of any one of claims 1 to 4, or the task processing method of claim 5.
8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the large model compression method of any one of claims 1 to 4, or the task processing method of claim 5.