Mixture-of-experts model quantization method and apparatus, and device and storage medium

Abstract

The present application relates to the field of deep learning. Disclosed are a mixture-of-experts model quantization method and apparatus, and a device and a storage medium. The method comprises: determining model activation values of a mixture-of-experts model and first weight quantization bit widths of non-expert models; determining a current hardware support situation and expert model precision requirements of the mixture-of-experts model, so as to determine second weight quantization bit widths of expert models; determining memory limits of the expert models on the basis of the first weight quantization bit widths, and performing inference on models in each layer of the mixture-of-experts model, so as to determine the quantization loss of each layer on the basis of the model activation values and the second weight quantization bit widths; and determining an optimization objective on the basis of the memory limits and the quantization loss, and optimizing the second weight quantization bit widths on the basis of the optimization objective, so as to perform model quantization on the mixture-of-experts model. By means of analyzing the precision loss under quantization bit widths of expert models in each layer of a mixture-of-experts model and memory limits thereof, optimization is performed under certain memory limits to obtain optimal quantization bit widths, thereby reducing the model precision loss.

Description

A hybrid expert model quantization method, apparatus, device, and storage medium Technical Field

[0001] This invention relates to the field of deep learning, and in particular to a hybrid expert model quantization method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of deep learning technologies, neural network models have been widely applied in many industries and scenarios. However, deep learning models have a large number of parameters and computational demands, requiring significant hardware resources, which often conflicts with the limitations of hardware resources, especially in the field of large models, where model parameters often restrict the hardware devices that can be deployed. To address these issues, model quantization, especially mixed-precision model quantization, is an effective method to reduce model memory usage.

[0003] Mixture of Experts (MoE) is a sparse-gated deep learning model that consists of a set of expert models and a gating model. The basic idea of ​​MoE is to divide the input data into multiple regions based on the task type and assign one or more expert models to each region. Each expert model can focus on processing its assigned region of input data, thereby improving the overall performance of the model. However, since each expert specializes in different domains, their importance varies across different business scenarios. Therefore, how to select different quantization bits for the weights of different models is a problem that needs to be solved in this field.

[0004] Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a hybrid expert model quantization method, apparatus, device, and storage medium. By analyzing the accuracy loss of all expert models at each layer of the hybrid expert model under various possible bit widths, and considering the memory limitations of the expert model weights, optimization is performed under certain memory constraints to obtain the optimal quantization bit width for different experts at each layer, thereby reducing model accuracy loss. The specific solution is as follows:

[0006] Firstly, this application provides a hybrid expert model quantization method, including:

[0007] The activation values ​​of each model in the hybrid expert model are determined according to a preset activation function, and several first weight quantization bits of the non-expert models in the hybrid expert model are determined; the hybrid expert model includes the non-expert models and expert models;

[0008] Determine the hardware support of the current device and the accuracy requirements of the expert model in the hybrid expert model, and determine several second weight quantization bits of the expert model in the hybrid expert model based on the hardware support and the accuracy requirements of the expert model;

[0009] The memory limit of the expert model is determined based on the first weight quantization bit number according to the preset memory determination rule, and model inference is performed on the model of each layer in the hybrid expert model based on the preset loss calculation dataset, so as to determine the quantization loss of each layer of the hybrid expert model according to the model activation value and the second weight quantization bit number.

[0010] The model optimization objective of the hybrid expert model is determined based on the memory usage limit and the quantization loss, and the second weight quantization bit is optimized based on the model optimization objective, so as to perform model quantization on the hybrid expert model based on the optimized second weight quantization bit.

[0011] Optionally, determining the number of first weight quantization bits for the non-expert models in the hybrid expert model includes:

[0012] The accuracy requirement of the non-expert model in the hybrid expert model is determined, and a number of first weight quantization bits of the non-expert model are determined based on the accuracy requirement of the non-expert model; the first weight quantization bits are represented based on bit width.

[0013] Construct a first set of quantization bits for the non-expert model based on the first set of weighted quantization bits;

[0014] Accordingly, determining the number of second weight quantization bits for the expert model in the hybrid expert model includes:

[0015] Based on the accuracy requirements of the expert model, a number of second weight quantization bits are determined for the expert model; the second weight quantization bits are represented based on bit width.

[0016] The second set of quantization bits for the expert model is constructed based on the second weight quantization bits.

[0017] Optionally, before determining the quantization loss of each layer in the hybrid expert model based on the model activation value and the second weight quantization bit, the method further includes:

[0018] The first operator of the hybrid expert model is determined based on the preset activation value weights and the model input values ​​corresponding to the model activation values, and the non-quantized activation value data of the hybrid expert model is determined based on the first operator; the preset activation value weights are activation value weights in floating-point form;

[0019] The preset activation value weights and the model input values ​​corresponding to the model activation values ​​are quantized respectively. The second operator of the hybrid expert model is determined based on the quantized preset activation value weights and the model input values. The second operator is dequantized, and the quantized activation value data of the hybrid expert model is determined based on the dequantized second operator.

[0020] The data difference between the non-quantized activation value data and the quantized activation value data is determined, and the quantization loss evaluation index of the hybrid expert model is determined based on the data difference. The quantization loss of each layer in the hybrid expert model is determined based on the quantization loss evaluation index, the model activation value, the first weight quantization bit number, and the second weight quantization bit number.

[0021] Optionally, determining the memory usage limit of the expert model includes:

[0022] Determine the target memory (excluding the model-dependent memory of the hybrid expert model) and the hardware memory of the current device, and determine the model-dependent memory limit of the hybrid expert model based on the target memory and the hardware memory;

[0023] The memory usage limit of the expert model is determined based on the model-related memory limit of the hybrid expert model.

[0024] Optionally, determining the memory usage limit of the expert model based on the model-related memory limit of the hybrid expert model includes:

[0025] The memory usage of the hybrid expert model's activation value and the memory usage of the non-expert model's weights are determined based on the model's activation value and the first weight quantization bit depth.

[0026] The difference between the model-related memory limit, the memory occupied by the activation value, and the memory occupied by the non-expert model weights is used as the memory limit of the expert model.

[0027] Optionally, determining the quantization loss of each layer in the hybrid expert model based on the model activation value and the second weight quantization bit depth includes:

[0028] The target activation value of each layer in the hybrid expert model is determined based on the model activation value, and the corresponding activation value quantization loss is determined based on each second weight quantization bit based on the target activation value.

[0029] Furthermore, after determining the quantization loss of each layer in the hybrid expert model based on the model activation value and the second weight quantization bit depth, the method further includes:

[0030] The activation value quantization loss of each layer in the hybrid expert model is summarized based on a preset data induction method to obtain the total quantization loss of the hybrid expert model.

[0031] Optionally, determining the model optimization objective of the hybrid expert model based on the memory limitation and the quantization loss, and optimizing the second weight quantization bit depth based on the model optimization objective, includes:

[0032] The theoretical minimum loss of the total quantitative loss is determined, and the theoretical minimum loss is used as the optimization objective of the hybrid expert model. The actual memory occupied by the expert model is less than the memory limit, which is used as the constraint condition of the hybrid expert model. The model optimization objective is constructed through the optimization objective and the constraint condition.

[0033] The actual minimum loss of the total quantized loss is determined by quantizing the loss of the activation value of each layer in the hybrid expert model, and the actual memory occupied by the expert model is determined by the number of models in the expert model and the second weight quantization bit.

[0034] Based on the model optimization objective, the second weight quantization bit depth of each expert model is optimized according to the actual minimum loss and actual memory usage, to obtain the target quantization bit depth of the hybrid expert model, so as to perform model quantization on the hybrid expert model according to the target quantization bit depth; the second weight quantization bit depth is always kept in integer form during the model optimization process.

[0035] Secondly, this application provides a hybrid expert model quantization device, comprising:

[0036] The first parameter determination module is used to determine the model activation value of each model in the hybrid expert model according to a preset activation function, and to determine a number of first weight quantization bits of the non-expert models in the hybrid expert model; the hybrid expert model includes the non-expert models and expert models;

[0037] The second parameter determination module is used to determine the hardware support status of the current device and the expert model accuracy requirements of the hybrid expert model, and to determine several second weight quantization bits of the expert model in the hybrid expert model based on the hardware support status and the expert model accuracy requirements;

[0038] The loss determination module is used to determine the memory limit of the expert model based on the first weight quantization bit number according to the preset memory determination rule, and to perform model inference on the model of each layer in the hybrid expert model based on the preset loss calculation dataset, so as to determine the quantization loss of each layer of the hybrid expert model according to the model activation value and the second weight quantization bit number.

[0039] The model quantization module is used to determine the model optimization objective of the hybrid expert model based on the memory limitation and the quantization loss, and to optimize the second weight quantization bit based on the model optimization objective, so as to perform model quantization on the hybrid expert model based on the optimized second weight quantization bit.

[0040] Thirdly, this application provides an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the aforementioned hybrid expert model quantization method.

[0041] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the aforementioned hybrid expert model quantization method.

[0042] This application determines the model activation values ​​of each model in the hybrid expert model based on a preset activation function, and determines several first weight quantization bits for the non-expert models in the hybrid expert model. Then, it determines the hardware support of the current device and the accuracy requirements of the expert models in the hybrid expert model, and determines several second weight quantization bits for the expert models in the hybrid expert model based on the hardware support and accuracy requirements. Next, it determines the memory usage limit of the expert models based on the first weight quantization bits according to a preset memory determination rule, and performs model inference on the models of each layer in the hybrid expert model based on a preset loss calculation dataset to determine the quantization loss of each layer in the hybrid expert model based on the model activation values ​​and the second weight quantization bits. Finally, it determines the model optimization objective of the hybrid expert model based on the memory usage limit and the quantization loss, and optimizes the second weight quantization bits based on the model optimization objective, and performs model quantization on the hybrid expert model based on the optimized second weight quantization bits. In this way, this application analyzes the accuracy loss of all expert models in each layer of the hybrid expert model under various possible bit widths, and the memory usage limit of the expert model weights, optimizes under certain memory constraints, obtains the optimal accuracy loss, and thus obtains the optimal quantization bits for different experts in each layer. In this way, under the condition of hardware memory constraints, this application can select the optimal number of quantization bits for the expert weights of the expert model, reduce the loss of model accuracy, and improve the practical value of the hybrid expert model. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 is a flowchart of a hybrid expert model quantization method provided in this application;

[0045] Figure 2 is a flowchart of a specific hybrid expert model quantization process provided in this application;

[0046] Figure 3 is a flowchart of a specific hybrid expert model quantization method provided in this application;

[0047] Figure 4 is a schematic diagram of a hybrid expert model quantization device provided in this application;

[0048] Figure 5 is a structural diagram of an electronic device provided in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Hybrid expert models can segment input data into multiple regions based on task type and assign one or more expert models to each region. Each expert model can focus on processing its assigned input data, thereby improving the overall performance of the model. However, since each expert specializes in different domains, their importance varies in different business scenarios. This application analyzes the accuracy loss of all expert models at each layer of the hybrid expert model under various possible bit widths, as well as the memory constraints of expert model weights. Under certain memory constraints, optimization is performed to obtain the optimal accuracy loss, and thus the optimal number of quantization bits used for different experts at each layer, reducing the model's accuracy loss.

[0051] Referring to Figure 1, an embodiment of the present invention discloses a hybrid expert model quantization method, including:

[0052] Step S11: Determine the model activation value of each model in the hybrid expert model according to the preset activation function, and determine a number of first weight quantization bits of the non-expert model in the hybrid expert model; the hybrid expert model includes the non-expert model and the expert model.

[0053] In this embodiment, it is understood that before performing model quantization of the hybrid expert model, it is necessary to define a quantization loss evaluation index. The above index is mainly determined based on the activation value of the model. Therefore, in this embodiment, the model activation value of each model in the hybrid expert model is first determined according to the preset activation function, and several first weight quantization bits of the non-expert models in the hybrid expert model are determined. It is understood that the hybrid expert model includes non-expert models, such as gated models and expert models.

[0054] When determining the number of first weights for the non-expert model in a hybrid expert model, the accuracy requirement of the non-expert model in the hybrid expert model is first determined, and the number of first weights for the non-expert model is determined based on the accuracy requirement. The number of first weights is represented based on bit width. Then, a set of first quantization bits for the non-expert model is constructed based on the number of first weights. It can be understood that the above process also applies to determining the number of quantization bits for activation values. When pre-determining the number of quantization bits for model activation values ​​and non-expert model weights, the number of quantization bits for model activation values ​​and non-expert model weights can be, but is not limited to, 2, 4, 6, 8, 16, etc., specifically determined according to the model accuracy requirement.

[0055] Step S12: Determine the hardware support status of the current device and the accuracy requirements of the expert model in the hybrid expert model, and determine several second weight quantization bits of the expert model in the hybrid expert model based on the hardware support status and the accuracy requirements of the expert model.

[0056] In this embodiment, when determining the quantization bit depth of the expert model, it is necessary to determine the hardware support of the current device and the accuracy requirements of the expert model in the hybrid expert model. Based on the hardware support and accuracy requirements, several second weight quantization bit depths of the expert model in the hybrid expert model are determined. Specifically, after determining several second weight quantization bit depths of the expert model based on the accuracy requirements, a second quantization bit depth set of the expert model is constructed according to the second weight quantization bit depths. The second weight quantization bit depths are represented based on bit width. When defining the expert model weight quantization bit depth set based on hardware support and model accuracy requirements, the expert model weight quantization bit depths can be, but are not limited to, 2, 4, 6, 8, 16 bits, etc., and the bit width is used to represent the quantization bit depth set. For example, bit width = {2, 4, 6, 8, 16}.

[0057] Step S13: Determine the memory limit of the expert model based on the first weight quantization bit number according to the preset memory determination rule, and perform model inference on the model of each layer in the hybrid expert model based on the preset loss calculation dataset, so as to determine the quantization loss of each layer in the hybrid expert model according to the model activation value and the second weight quantization bit number.

[0058] In this embodiment, it should be noted that quantization loss calculation and model inference can be performed based on a pre-prepared dataset containing partial test data. This dataset can serve as both a quantization calibration dataset and a quantization loss calculation dataset. Next, this embodiment can determine the memory limit of the expert model based on a preset memory determination rule and the first weight quantization bit depth. Then, based on the preset loss calculation dataset, model inference is performed on each layer of the hybrid expert model to determine the quantization loss of each layer based on the model activation value and the second weight quantization bit depth.

[0059] When determining the memory limit of the expert model, it is first necessary to determine the target memory (excluding the model-related memory of the hybrid expert model) and the current device's hardware memory. Based on the target memory and hardware memory, the model-related memory limit of the hybrid expert model is determined. Then, based on the model-related memory limit of the hybrid expert model, the memory limit of the expert model is determined. Specifically, the memory occupied by the activation values ​​of the hybrid expert model and the memory occupied by the weights of the non-expert model can be determined based on the model activation values ​​and the first weight quantization bit depth. The difference between the model-related memory limit, the activation value memory occupation, and the non-expert model weight memory occupation is then used as the memory limit of the expert model. It is understood that during model inference, inference-related memory occupation includes inference framework memory occupation, code data memory occupation, model weight memory occupation, and activation value memory occupation, among which model-related memory occupation mainly includes model weight memory occupation and activation value memory occupation. In this embodiment, it is assumed that the hardware memory size is m. total The memory usage excluding model-related memory is m. other Remaining memory size m model That is, the model-related memory limit is: m model =m total -m other ;

[0060] Since the quantization bits of the model activation values ​​and the non-expert model weights are already determined, their memory usage can be calculated in advance. In this embodiment, it is assumed that the maximum memory usage for the model activation values ​​is `mactivation`, and the memory usage for the non-expert model weights is `mnon-experts`. Let `m...` limit Indicates the memory limit of the expert model, then m limit =m model -mactivation-mnon-experts.

[0061] It should be noted that, in this embodiment, before determining the quantization loss of each layer in the hybrid expert model based on the model activation value and the second weight quantization bit depth, based on the above steps, it is also necessary to determine the first operator of the hybrid expert model based on the preset activation value weight and the model input value corresponding to the model activation value, and determine the non-quantized activation value data of the hybrid expert model based on the first operator; wherein the preset activation value weight is the activation value weight in floating-point form. Then, the preset activation value weight and the model input value corresponding to the model activation value are quantized respectively, and the second operator of the hybrid expert model is determined based on the quantized preset activation value weight and the model input value. After the second operator is dequantized, the quantized activation value data of the hybrid expert model is determined based on the dequantized second operator. Then, the data difference between the non-quantized activation value data and the quantized activation value data can be determined, and the quantization loss evaluation index of the hybrid expert model can be determined based on the data difference. Based on the quantization loss evaluation index, the quantization loss of each layer in the hybrid expert model is determined according to the model activation value, the first weight quantization bit depth, and the second weight quantization bit depth. Understandably, quantization loss calculation involves calculating the difference between non-quantized activation value data and quantized activation value data. This difference can be calculated using, but is not limited to, KL divergence (Kullback-Leibler Divergence) as the quantization loss. Non-quantized activation value data refers to the data obtained by calculating the corresponding operator, such as a fully connected layer, using floating-point weights and the input (the previous activation value data). The result is the non-quantized activation value data. Quantized activation value data, on the other hand, involves quantizing the floating-point weights and the input data separately, calculating the corresponding operator using the quantized data, and then dequantizing the result to obtain the quantized activation value data.

[0062] Step S14: Determine the model optimization objective of the hybrid expert model based on the memory usage limit and the quantization loss, and optimize the second weight quantization bit depth based on the model optimization objective, so as to perform model quantization on the hybrid expert model based on the optimized second weight quantization bit depth.

[0063] In this embodiment, the parameters obtained through the above steps can be used to determine the model optimization objective of the hybrid expert model based on memory constraints and quantization loss. The second weight quantization bit depth is then optimized based on the model optimization objective, and the hybrid expert model is quantized using the optimized second weight quantization bit depth. This allows the optimized quantization bit depth to be used for model quantization and subsequently for model inference.

[0064] In this embodiment, the activation values ​​of each model in the hybrid expert model are first determined, and the accuracy requirements of the non-expert models in the hybrid expert model are determined. Based on the accuracy requirements of the non-expert models, several first weight quantization bits are determined for the non-expert models. Then, the hardware support of the current device and the accuracy requirements of the expert models in the hybrid expert model are determined, and several second weight quantization bits are determined for the expert models. Then, based on the first weight quantization bits, the memory limit of the expert models can be determined. Based on a preset loss calculation dataset, model inference is performed on the models of each layer in the hybrid expert model to determine the quantization loss of each layer in the hybrid expert model. Then, the model optimization target of the hybrid expert model can be determined according to the memory limit and the quantization loss. Based on the model optimization target, the second weight quantization bits are optimized, and the hybrid expert model is quantized based on the optimized second weight quantization bits. As shown in Figure 2, this embodiment prepares a dataset containing partial test data, defines a quantization loss evaluation index, determines the quantization bit depth of the model activation value and non-expert model weights, defines a set of quantization bit depths for expert model weights, obtains the memory constraints of the expert model, analyzes the quantization loss of each layer of the hybrid expert model, optimizes problem construction, and solves the problem using an integer linear programming method. Through model quantization and inference, a hybrid expert model quantization method is proposed. In the hybrid expert model, higher quantization bit depths are needed for important experts, and lower quantization bit depths are needed for less important experts. Choosing different quantization bit depths for different model weights is crucial to the model accuracy loss; the lower the model quantization bit depth, the higher the model accuracy loss. This embodiment solves the problem of choosing the quantization bit depth for different expert models under memory constraints. It can select the optimal quantization bit depth for expert weights under hardware memory constraints, reducing accuracy loss, and has high practical and innovative value.

[0065] Based on the previous embodiment, this application can optimize the bit width under certain memory constraints by analyzing the accuracy loss of all expert models in each layer of the hybrid expert model under various possible bit widths and the memory limitations of the expert model weights. Next, this embodiment will describe in detail the process of optimizing the quantization bit width of the expert model. Referring to Figure 3, this application embodiment discloses a specific hybrid expert model quantization method, including:

[0066] Step S21: Perform model inference on the model of each layer in the hybrid expert model based on the preset loss calculation dataset, so as to determine the quantization loss of each layer in the hybrid expert model according to the model activation value and the second weight quantization bit.

[0067] In this embodiment, when determining the quantization loss, it is first necessary to perform model inference on the model of each layer in the hybrid expert model based on a preset loss calculation dataset. As shown in the previous embodiment, the quantization bit is a set containing multiple quantization bit values. Therefore, after determining the target activation value of each layer in the hybrid expert model based on the model activation value, it is necessary to determine the corresponding activation value quantization loss based on each second weight quantization bit. Furthermore, after determining the quantization loss of each layer in the hybrid expert model based on the model activation value and the second weight quantization bit, it is also necessary to summarize the activation value quantization loss of each layer in the hybrid expert model based on a preset data induction method to obtain the total quantization loss of the hybrid expert model.

[0068] Based on the above process, when analyzing the quantization loss of each layer of the hybrid expert model, it is necessary to perform inference using the pre-quantization model for each sample data in the loss calculation dataset and obtain the activation value data of all expert model layers. Simultaneously, for the activation value data of all expert model layers, the weights of different expert models in that layer are calculated using all possible quantization bit widths, and the quantization loss of the activation values ​​of all experts in that layer is calculated under various possible bit widths. After completing the inference of all sample data, the quantization loss of the activation values ​​of all experts in each layer under various possible bit widths is summarized, and the quantization loss set Loss of each layer of all sample data is calculated using data induction methods, including but not limited to the mean. The calculation process is as follows: Loss = {loss l}, l∈Layers;

[0069] Where loss l This represents the set of quantization losses for layer l, containing the quantization losses of all experts in that layer using different quantization bits. Layers represents all layers of the model. Experts l This represents all experts at level l.

[0070] Step S22: Determine the theoretical minimum loss of the total quantitative loss, and use the theoretical minimum loss as the optimization objective of the hybrid expert model, and use the actual memory occupied by the expert model being less than the memory limit as the constraint condition of the hybrid expert model, and construct the model optimization objective through the optimization objective and the constraint condition.

[0071] In this embodiment, during model optimization, it is understood that an optimization objective needs to be determined for the current model. Therefore, it is first necessary to determine the theoretical minimum loss of the total quantitative loss and use the theoretical minimum loss as the optimization objective of the hybrid expert model, and use the actual memory occupied by the expert model being less than the memory limit as the constraint condition of the hybrid expert model. The model optimization objective is constructed through the optimization objective and the constraint condition.

[0072] Step S23: Determine the actual minimum loss of the total quantized loss by quantizing the loss of the activation value of each layer in the hybrid expert model, and determine the actual memory occupied by the expert model by the number of models and the second weight quantization bit.

[0073] In this embodiment, during the optimization process, the actual minimum loss of the total quantized loss can be determined by summing the quantized loss of the activation values ​​of each layer in the hybrid expert model, and the actual memory occupied by the expert model can be determined by the number of models and the second weight quantization bit.

[0074] Step S24: Based on the model optimization objective, optimize the second weight quantization bit number corresponding to each expert model according to the actual minimum loss and actual memory usage of the expert model to obtain the target quantization bit number of the hybrid expert model, so as to perform model quantization on the hybrid expert model according to the target quantization bit number; the second weight quantization bit number is always kept in integer form during the model optimization process.

[0075] In this embodiment, by combining steps S23 and S24 above, the second weight quantization bits corresponding to each expert model can be optimized based on the actual minimum loss and actual memory usage of the expert model according to the model optimization objective, so as to obtain the target quantization bits of the hybrid expert model. The hybrid expert model is then quantized according to the target quantization bits. It should be noted that the second weight quantization bits are always kept in integer form during the optimization process.

[0076] It should be noted in this embodiment that the optimization problem is solved using integer linear programming. First, the optimization objective is to minimize the total quantitative loss, with the constraint being the memory occupied by all expert model weights, m. experts Less than the expert model weight memory limit m limit Then let total_loss represent the totalized loss, size le b represents the number of elements in expert e at level l. le This represents the number of quantization bits for expert e in layer l. After determining the number of quantization bits for each expert in layer l, the loss is determined by searching the loss set. l .

[0077] Optimization objective: min total_loss;

[0078] Total loss: total_loss = ∑ l∈Layers loss l ;

[0079] Constraints: m experts ≤m limit ;

[0080] in: b le ∈Bitwidth.

[0081] It is understandable that the above optimization problem is an integer linear programming problem, and the optimization variable is the quantization bit b of the weight of each expert at each layer. le This can be solved using integer linear programming, thereby obtaining the quantization bits of each expert weight in each layer.

[0082] In this embodiment, the target activation value of each layer in the hybrid expert model is first determined based on the model activation value. Then, based on the target activation value, the corresponding activation value quantization loss is determined according to the quantization bit depth of each second weight. The theoretical minimum loss of the total quantization loss is used as the optimization objective of the hybrid expert model, and the actual memory usage of the expert model is less than the memory limit as the constraint condition. Based on the model optimization objective, the quantization bit depth of the second weight corresponding to each expert model is optimized according to the actual minimum loss and actual memory usage, resulting in the target quantization bit depth of the hybrid expert model. The hybrid expert model is then quantized based on the target quantization bit depth using an integer linear programming problem. Through the above technical solution, this embodiment first analyzes the quantization loss of each layer of the hybrid expert model, including the accuracy loss of all experts in each layer under various possible bit widths. Secondly, it analyzes the memory limit of the expert model weights, constructs an optimization problem, and under certain memory constraints, optimizes the optimal accuracy loss using an integer linear programming method, thereby obtaining the optimal quantization bit depth for different experts in each layer. In this way, the optimal quantization bit depth for the expert weights of the expert model can be selected under hardware memory constraints, which has high practical and innovative value.

[0083] Referring to Figure 4, this application also discloses a hybrid expert model quantization device, comprising:

[0084] The first parameter determination module 11 is used to determine the model activation value of each model in the hybrid expert model according to a preset activation function, and to determine a number of first weight quantization bits of the non-expert models in the hybrid expert model; the hybrid expert model includes the non-expert models and expert models;

[0085] The second parameter determination module 12 is used to determine the hardware support status of the current device and the expert model accuracy requirements of the hybrid expert model, and to determine a number of second weight quantization bits of the expert model in the hybrid expert model based on the hardware support status and the expert model accuracy requirements;

[0086] The loss determination module 13 is used to determine the memory limit of the expert model based on the first weight quantization bit number according to the preset memory determination rule, and to perform model inference on the model of each layer in the hybrid expert model based on the preset loss calculation dataset, so as to determine the quantization loss of each layer in the hybrid expert model according to the model activation value and the second weight quantization bit number.

[0087] The model quantization module 14 is used to determine the model optimization objective of the hybrid expert model based on the memory limitation and the quantization loss, and optimize the second weight quantization bit based on the model optimization objective, so as to perform model quantization on the hybrid expert model based on the optimized second weight quantization bit.

[0088] In this embodiment, the model activation values ​​of each model in the hybrid expert model are determined according to a preset activation function, and several first weight quantization bits of the non-expert models in the hybrid expert model are determined. Then, the hardware support of the current device and the accuracy requirements of the expert models in the hybrid expert model are determined, and several second weight quantization bits of the expert models in the hybrid expert model are determined based on the hardware support and the accuracy requirements of the expert models. Then, the memory usage limit of the expert models is determined based on the first weight quantization bits according to a preset memory determination rule, and model inference is performed on the models of each layer in the hybrid expert model based on a preset loss calculation dataset to determine the quantization loss of each layer in the hybrid expert model based on the model activation values ​​and the second weight quantization bits. Finally, the model optimization objective of the hybrid expert model is determined based on the memory usage limit and the quantization loss, and the second weight quantization bits are optimized based on the model optimization objective to perform model quantization on the hybrid expert model based on the optimized second weight quantization bits. Through the above technical solution, this embodiment analyzes the accuracy loss of all expert models in each layer of the hybrid expert model under various possible bit widths, as well as the memory usage limit of the expert model weights, and optimizes under certain memory constraints to obtain the optimal accuracy loss, and thus obtains the optimal quantization bits used for different experts in each layer. In this way, under the condition of hardware memory constraints, this application can select the optimal number of quantization bits for the expert weights of the expert model, reduce the loss of model accuracy, and improve the practical value of the hybrid expert model.

[0089] In some specific embodiments, the first parameter determining module 11 specifically includes:

[0090] The first parameter determination unit is used to determine the accuracy requirement of the non-expert model in the hybrid expert model, and to determine a number of first weight quantization bits of the non-expert model based on the accuracy requirement of the non-expert model; the first weight quantization bits are represented based on bit width.

[0091] The first set construction unit is used to construct a first set of quantization bits for the non-expert model based on the first weight quantization bits.

[0092] Accordingly, the second parameter determining module 12 specifically includes:

[0093] The second parameter determination unit is used to determine a number of second weight quantization bits of the expert model based on the accuracy requirements of the expert model; the second weight quantization bits are represented based on the bit width.

[0094] The second set construction unit is used to construct the second set of quantization bits of the expert model based on the second weight quantization bits.

[0095] In some specific embodiments, the loss determination module 13 further includes:

[0096] The first activation value determination unit is used to determine the first operator of the hybrid expert model based on the preset activation value weight and the model input value corresponding to the model activation value, and to determine the non-quantized activation value data of the hybrid expert model according to the first operator; the preset activation value weight is the activation value weight in floating-point form;

[0097] The second activation value determination unit is used to quantize the preset activation value weight and the model input value corresponding to the model activation value, respectively, and determine the second operator of the hybrid expert model based on the quantized preset activation value weight and the model input value, and after dequantizing the second operator, determine the quantized activation value data of the hybrid expert model based on the dequantized second operator;

[0098] The first loss determination unit is used to determine the data difference between the non-quantized activation value data and the quantized activation value data, and to determine the quantization loss evaluation index of the hybrid expert model based on the data difference, so as to determine the quantization loss of each layer in the hybrid expert model based on the quantization loss evaluation index according to the model activation value, the first weight quantization bit number and the second weight quantization bit number.

[0099] In some specific embodiments, the loss determination module 13 specifically includes:

[0100] The first memory limit determination unit is used to determine the target memory other than the model-related memory of the hybrid expert model and the hardware memory of the current device, and to determine the model-related memory limit of the hybrid expert model based on the target memory and the hardware memory.

[0101] The second memory limit determination unit is used to determine the memory limit occupied by the expert model based on the model-related memory limit of the hybrid expert model.

[0102] In some specific embodiments, the loss determination module 13 specifically includes:

[0103] The first memory determination unit is used to determine the memory occupied by the activation value of the hybrid expert model and the memory occupied by the weights of the non-expert model based on the model activation value and the first weight quantization bit.

[0104] The third memory limit determination unit is used to take the difference between the model-related memory limit, the memory occupied by the activation value, and the memory occupied by the non-expert model weights as the memory limit occupied by the expert model.

[0105] In some specific embodiments, the loss determination module 13 specifically includes:

[0106] The second loss determination unit is used to determine the target activation value of each layer in the hybrid expert model based on the model activation value, and to determine the corresponding activation value quantization loss based on each second weight quantization bit based on the target activation value.

[0107] Furthermore, the loss determination module 13 also includes:

[0108] The third loss determination unit is used to summarize the activation value quantization loss of each layer in the hybrid expert model based on a preset data induction method, so as to obtain the total quantization loss of the hybrid expert model.

[0109] In some specific embodiments, the model quantization module 14 specifically includes:

[0110] The objective determination unit is used to determine the theoretical minimum loss of the total quantitative loss, and use the theoretical minimum loss as the optimization objective of the hybrid expert model, and use the actual memory occupied by the expert model being less than the memory limit as the constraint condition of the hybrid expert model, and construct the model optimization objective through the optimization objective and the constraint condition;

[0111] The second memory determination unit is used to determine the actual minimum loss of the total quantized loss by quantizing the loss of the activation value of each layer in the hybrid expert model, and to determine the actual memory occupied by the expert model by the number of models in the expert model and the second weight quantization bit.

[0112] The model quantization unit is used to optimize the second weight quantization bit number corresponding to each expert model based on the actual minimum loss and actual memory usage of the expert model according to the model optimization objective, so as to obtain the target quantization bit number of the hybrid expert model, and to perform model quantization on the hybrid expert model according to the target quantization bit number; the second weight quantization bit number is always kept in integer form during the model optimization process.

[0113] Furthermore, this application also discloses an electronic device. FIG5 is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the figure should not be considered as any limitation on the scope of use of this application.

[0114] Figure 5 is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the hybrid expert model quantization method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0115] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0116] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0117] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the hybrid expert model quantization method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0118] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed hybrid expert model quantization method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0120] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0122] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0123] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A hybrid expert model quantization method, characterized in that, include: The activation values ​​of each model in the hybrid expert model are determined according to a preset activation function, and several first weight quantization bits of the non-expert models in the hybrid expert model are determined. The hybrid expert model includes both the non-expert model and the expert model. Determine the hardware support of the current device and the accuracy requirements of the expert model in the hybrid expert model, and determine several second weight quantization bits of the expert model in the hybrid expert model based on the hardware support and the accuracy requirements of the expert model; The memory limit of the expert model is determined based on the first weight quantization bit number according to the preset memory determination rule, and model inference is performed on the model of each layer in the hybrid expert model based on the preset loss calculation dataset, so as to determine the quantization loss of each layer of the hybrid expert model according to the model activation value and the second weight quantization bit number. The model optimization objective of the hybrid expert model is determined based on the memory usage limit and the quantization loss, and the second weight quantization bit is optimized based on the model optimization objective, so as to perform model quantization on the hybrid expert model based on the optimized second weight quantization bit.

2. The hybrid expert model quantization method according to claim 1, characterized in that, The determination of several first weight quantization bits for the non-expert models in the hybrid expert model includes: The accuracy requirement of the non-expert model in the hybrid expert model is determined, and a number of first weight quantization bits of the non-expert model are determined based on the accuracy requirement of the non-expert model; the first weight quantization bits are represented based on bit width. Construct a first set of quantization bits for the non-expert model based on the first set of weighted quantization bits; Accordingly, determining the number of second weight quantization bits for the expert model in the hybrid expert model includes: Based on the accuracy requirements of the expert model, a number of second weight quantization bits are determined for the expert model; the second weight quantization bits are represented based on bit width. The second set of quantization bits for the expert model is constructed based on the second weight quantization bits.

3. The hybrid expert model quantization method according to claim 1, characterized in that, Before determining the quantization loss of each layer in the hybrid expert model based on the model activation value and the second weight quantization bit, the method further includes: The first operator of the hybrid expert model is determined based on the preset activation value weights and the model input values ​​corresponding to the model activation values, and the non-quantized activation value data of the hybrid expert model is determined based on the first operator; the preset activation value weights are activation value weights in floating-point form; The preset activation value weights and the model input values ​​corresponding to the model activation values ​​are quantized respectively. The second operator of the hybrid expert model is determined based on the quantized preset activation value weights and the model input values. The second operator is dequantized, and the quantized activation value data of the hybrid expert model is determined based on the dequantized second operator. The data difference between the non-quantized activation value data and the quantized activation value data is determined, and the quantization loss evaluation index of the hybrid expert model is determined based on the data difference. The quantization loss of each layer in the hybrid expert model is determined based on the quantization loss evaluation index, the model activation value, the first weight quantization bit number, and the second weight quantization bit number.

4. The hybrid expert model quantization method according to claim 1, characterized in that, Determining the memory usage limit of the expert model includes: Determine the target memory (excluding the model-dependent memory of the hybrid expert model) and the hardware memory of the current device, and determine the model-dependent memory limit of the hybrid expert model based on the target memory and the hardware memory; The memory usage limit of the expert model is determined based on the model-related memory limit of the hybrid expert model.

5. The hybrid expert model quantization method according to claim 4, characterized in that, The step of determining the memory usage limit of the expert model based on the model-related memory limit of the hybrid expert model includes: The memory usage of the hybrid expert model's activation value and the memory usage of the non-expert model's weights are determined based on the model's activation value and the first weight quantization bit depth. The difference between the model-related memory limit, the memory occupied by the activation value, and the memory occupied by the non-expert model weights is used as the memory limit of the expert model.

6. The hybrid expert model quantization method according to any one of claims 1 to 5, characterized in that, Determining the quantization loss of each layer in the hybrid expert model based on the model activation value and the second weight quantization bit depth includes: The target activation value of each layer in the hybrid expert model is determined based on the model activation value, and the corresponding activation value quantization loss is determined based on each second weight quantization bit based on the target activation value. Furthermore, after determining the quantization loss of each layer in the hybrid expert model based on the model activation value and the second weight quantization bit depth, the method further includes: The activation value quantization loss of each layer in the hybrid expert model is summarized based on a preset data induction method to obtain the total quantization loss of the hybrid expert model.

7. The hybrid expert model quantization method according to claim 6, characterized in that, The step of determining the model optimization objective of the hybrid expert model based on the memory usage limit and the quantization loss, and optimizing the second weight quantization bit depth based on the model optimization objective, includes: The theoretical minimum loss of the total quantitative loss is determined, and the theoretical minimum loss is used as the optimization objective of the hybrid expert model. The actual memory occupied by the expert model is less than the memory limit, which is used as the constraint condition of the hybrid expert model. The model optimization objective is constructed through the optimization objective and the constraint condition. The actual minimum loss of the total quantized loss is determined by quantizing the loss of the activation value of each layer in the hybrid expert model, and the actual memory occupied by the expert model is determined by the number of models in the expert model and the second weight quantization bit. Based on the model optimization objective, the second weight quantization bit depth of each expert model is optimized according to the actual minimum loss and actual memory usage, to obtain the target quantization bit depth of the hybrid expert model, so as to perform model quantization on the hybrid expert model according to the target quantization bit depth; the second weight quantization bit depth is always kept in integer form during the model optimization process.

8. A hybrid expert model quantization device, characterized in that, include: The first parameter determination module is used to determine the model activation value of each model in the hybrid expert model according to a preset activation function, and to determine a number of first weight quantization bits of the non-expert models in the hybrid expert model; the hybrid expert model includes the non-expert models and expert models; The second parameter determination module is used to determine the hardware support status of the current device and the expert model accuracy requirements of the hybrid expert model, and to determine several second weight quantization bits of the expert model in the hybrid expert model based on the hardware support status and the expert model accuracy requirements; The loss determination module is used to determine the memory limit of the expert model based on the first weight quantization bit number according to the preset memory determination rule, and to perform model inference on the model of each layer in the hybrid expert model based on the preset loss calculation dataset, so as to determine the quantization loss of each layer of the hybrid expert model according to the model activation value and the second weight quantization bit number. The model quantization module is used to determine the model optimization objective of the hybrid expert model based on the memory limitation and the quantization loss, and to optimize the second weight quantization bit based on the model optimization objective, so as to perform model quantization on the hybrid expert model based on the optimized second weight quantization bit.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the hybrid expert model quantization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the hybrid expert model quantization method as described in any one of claims 1 to 7.

Images