Model quantization method, model operation method, device, medium, and program product

Abstract

Disclosed in the present disclosure are a model quantization method, a model operation method, a device, a medium, and a program product. The model quantization method comprises: grouping model parameters of each network layer in a model to be quantized; determining a quantization parameter group of any group on the basis of a target integer bit number required by a network layer to which the any group belongs, and quantifying the model parameters of the any group on the basis of the quantization parameter group; and replacing the corresponding model parameters in said model with the quantized model parameters of each group to obtain a quantized model. The model parameters are grouped for quantization to reduce the influence of outliers present in the model parameters on quantization errors.

Description

[0001] Technical field of model quantification method, model operation method, device, medium and program product

[0002]

[0001] The present disclosure relates to the field of computer technology, and more particularly to a model quantization method, a model operation method, a device, a medium, and a program product.

[0003]

[0002] With the development of deep learning technology, the number of model parameters and model structure has become increasingly complex, requiring a large amount of computing resources and storage space to run. Typically, high-performance computing devices are needed to run the model, such as GPUs (Graphics Processing Units) and TPUs (Tensor Processing Units).

[0004]

[0003] In related technologies, the model is also quantized, that is, the model parameters in the model are quantized from floating-point data format to integer data format, and then the quantized model is stored to reduce the storage space occupied by the model.

[0005]

[0004] However, the existing model quantization scheme will affect the operation effect of the model, resulting in a relatively large model loss.

[0006]

[0005] The present disclosure provides a model quantization method, a model operation method, a device, a medium and a program product.

[0007]

[0006] A first aspect of the present disclosure provides a model quantization method, comprising: grouping model parameters of each network layer in a model to be quantized; for each group, determining a quantization parameter group for the group according to a target number of integer bits required by the network layer to which the group belongs; quantizing the model parameters of the group according to the quantization parameter group to obtain quantized model parameters of the group; the quantization parameter group includes multiple quantization parameters, and the multiple quantization parameters are nonlinearly related; and replacing corresponding model parameters in the model to be quantized with the quantized model parameters of each group to obtain a quantized model.

[0008]

[0007] A second aspect of the present disclosure proposes a model operation method, wherein the model is quantized using the method described in the first aspect, and the method includes: for each operation in the model operation process, obtaining a stored quantization parameter group corresponding to the model parameters required for the current operation; the model parameters required for each operation belong to the same group; based on the quantization parameter group, dequantizing the model parameters required for the current operation; and executing the current operation using the dequantized model parameters.

[0009]

[0008] A third aspect of the present disclosure provides a model quantization device, comprising: a grouping module for grouping model parameters of each network layer in a model to be quantized; a parameter quantization module for determining, for each group, a quantization parameter group of the group according to a target number of integer bits required by the network layer to which the group belongs, and quantizing the model parameters of the group according to the quantization parameter group to obtain quantized model parameters of the group; the quantization parameter group includes multiple quantization parameters, and the multiple quantization parameters are nonlinearly related; and a parameter replacement module for replacing corresponding model parameters in the model to be quantized with the quantized model parameters of each group to obtain a quantized model.

[0010]

[0009] A fourth aspect of the present disclosure proposes a model operation device, wherein the model is quantized using the device described in the third aspect, and the device includes: a quantization parameter acquisition module, which is used to obtain a stored quantization parameter group corresponding to the model parameters required for the current operation for each operation during the model operation; the model parameters required for each operation belong to the same group; a dequantization module, which is used to dequantize the model parameters required for the current operation based on the quantization parameter group; and an operation module, which is used to perform the current operation using the dequantized model parameters.

[0011]

[0010] A fifth aspect of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first or second aspect above.

[0012]

[0011] A sixth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method described in the first aspect or the second aspect above.

[0013]

[0012] An embodiment of the seventh aspect of the present disclosure provides a computer program product, including a computer program, wherein the computer program is executed by a processor to implement the method described in the first aspect or the second aspect.

[0014] Based on the above-described model quantization method and model operation method, the present disclosure has at least the following beneficial effects or advantages: quantization by grouping model parameters reduces the impact of outliers in the model parameters on quantization errors. When quantizing by group, the quantization parameter group of the group is determined based on the target integer bit number of the network layer to which it belongs. Since the quantization parameters in the quantization parameter group have a nonlinear relationship, the model parameters in the group can be quantized in a nonlinear mapping manner based on the quantization parameter group to ensure the result generation capability during model operation. Moreover, since each network layer corresponds to a target integer bit number, the model parameters of each network layer can be quantized to different integer spaces, thereby minimizing the storage space occupied by the model.

[0015]

[0014] For each operation during the model operation, since the model parameters required for each operation belong to the same group, the same quantization parameter group can be used to dequantize the obtained model parameters and perform the operation. In this way, only the acquisition of the quantization parameter group is required to complete the model operation. Under the premise of ensuring that the model operation speed is not affected, a generation result that is more similar to the original model can be obtained, thereby reducing model loss.

[0016]

[0015] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented according to the contents of the description. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the following specific embodiments of the present disclosure are specifically cited.

[0017]

[0016] The drawings described herein are intended to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are intended to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0018] FIG1 is a flow chart of an embodiment of a model quantization method according to an exemplary embodiment;

[0019] FIG2 is a flow chart of an embodiment of a model operation method according to an exemplary embodiment;

[0020] FIG3 is a schematic diagram of an overall process of model quantization and operation according to an exemplary embodiment;

[0021] FIG4 is a schematic diagram showing the structure of a model quantization device according to an exemplary embodiment;

[0022] FIG5 is a schematic diagram of a structure of a model operation device according to an exemplary embodiment;

[0023] FIG6 is a schematic diagram of a hardware structure of an electronic device according to an exemplary embodiment;

[0024]

[0023] FIG7 is a schematic diagram showing the structure of a storage medium according to an exemplary embodiment.

[0025]

[0024] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, like numbers in different drawings represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of systems and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0026]

[0025] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0027]

[0026] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to a determination".

[0027] Currently, the number of model parameters and the model structure are becoming increasingly complex, and the storage space occupied is increasing.

[0028]

[0028] Taking the large language model OPT (Open Pre-trained Transformer) as an example, it is divided into OPT-1.3B, OPT-66B, OPT-175B, etc. according to the number of parameters. For the OPT-175B model, the model parameters require at least 350GB of storage space at FP16 (Floating Point 16, 16-bit floating point) precision, and at least 700GB of video memory space at FP32 (32-bit floating point) precision. Currently, a common GPU used for model inference has only 80GB of video memory per GPU. Therefore, at least five GPUs are required to complete the operation of the OPT-175B model with FP16 precision.

[0029]

[0029] Considering that practical applications are all long text generation scenarios, the text that the model needs to process is relatively long, and the input data set is also relatively large (Batch size) each time the model is trained, so the storage space required for model operation will further increase.

[0030]

[0030] In the related art, the original floating-point model parameters in the model are usually converted into integer data format through quantization and then stored to reduce the space occupied by the model.

[0031]

[0031] However, the existing model quantization scheme has an impact on the generation result of the model, which is quite different from the generation result of the original model, resulting in a relatively large model loss.

[0032]

[0032] To solve the above technical problems, the present disclosure proposes a model quantization method and a model operation method. Considering that a large number of outliers usually exist in model parameters, the model parameters are grouped and quantized to reduce the impact of outliers on quantization errors. Then, when quantizing any group, a quantization parameter group for any group is determined based on the target integer bit number of the network layer to which the group belongs. The quantization parameters in the quantization parameter group have a nonlinear relationship. Therefore, based on the quantization parameter group, the model parameters in the group can be quantized in a nonlinear mapping manner to ensure the result generation capability during model operation. Moreover, since each network layer corresponds to a target integer bit number, the model parameters of each network layer can be quantized to different integer spaces, thereby minimizing the storage space occupied by the model parameters.

[0033]

[0033] During the model operation process, for each operation in the model operation process, since the model parameters required for each operation belong to the same group, the same quantization parameter group can be used to perform dequantization on the obtained model parameters before performing the operation. In this way, only one quantization parameter group needs to be obtained to complete one operation of the model, ensuring that the model operation speed is not affected and that a generation result that is more similar to the original model can be obtained, thereby reducing model loss.

[0034]

[0034] The following describes in detail the technical solution of the present invention and how the technical solution of the present invention solves the aforementioned technical problems using specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following describes the embodiments of the present invention in detail with reference to the accompanying drawings.

[0035]

[0035] FIG1 is a flow chart of an embodiment of a model quantization method according to an exemplary embodiment, including the following steps 101 to 103.

[0036]

[0036] Step 101: Group the model parameters of each network layer in the model to be quantized.

[0037]

[0037] Step 102: For each group, determine a quantization parameter group for the group according to a target integer bit number required by the network layer to which the group belongs, and quantize the model parameters of the group according to the quantization parameter group to obtain quantized model parameters of the group; wherein the quantization parameter group includes multiple quantization parameters, and the multiple quantization parameters are nonlinearly related.

[0038]

[0038] Step 103: Use the quantized model parameters of each group to replace the corresponding model parameters in the model to be quantized to obtain the quantized model.

[0039] In the disclosed embodiment, the model to be quantized is the original network model before quantization, and the data format of the model parameters is floating point. For example, the model parameters in the model to be quantized may be in a half-precision floating point (FP16) data format, and each parameter requires 16 bits to be stored in a computer. For another example, the model parameters in the model to be quantized may be in a 32-bit floating point (FP32) data format, and each parameter requires 32 bits to be stored in a computer. The model to be quantized may be a model to be quantized specified by a user according to actual needs.

[0040] The model structure of the model to be quantized consists of a large number of network layers, each of which contains a large number of model parameters. Taking a large language model as an example, the number of network layers is on the order of hundreds, and each layer contains a large number of model parameters. Typically, the model parameters in a network layer include weights and / or biases.

[0041]

[0041] Generally, the model parameters are initialized in the pre-training stage using the Gaussian distribution as the prior distribution. At the same time, during the model training process, the Gaussian distribution is also used as an assumption to constrain the model parameters. Therefore, it is inevitable that there will be outliers in the model parameters. Moreover, due to the large number of parameters, the number of outliers will also be relatively large. If the quantization parameter group of each network layer is directly determined, it will inevitably lead to a relatively large quantization error. Considering that the model operation process involves matrix operations of the model parameters of each network layer, outliers usually exist in local positions of the matrix composed of the model parameters. Therefore, the present disclosure groups the model parameters of each network layer and determines a quantization parameter group according to each group to reduce the impact of outliers on the quantization error.

[0042]

[0042] For the grouping of model parameters in the same network layer, the number of model parameters in each group is the same. Since the matrix sizes of model parameters in different network layers are different, the number of model parameters in groups belonging to different network layers is different.

[0043]

[0043] The target integer number represents the number of bits required to convert the model parameters in the floating point data format in the corresponding network layer into the integer data format. The target integer number can be 8-bit integer (INT8), 4-bit integer (INT4), 3-bit integer (INT3),

[0044] Any 2-bit integer (INT2). Different target integer numbers have different quantization losses on model parameters. Smaller target integer numbers result in greater quantization losses than larger target integer numbers.

[0045] The target number of integer bits required for each network layer in the model to be quantized can be predetermined, and the target number of integer bits for different network layers is different. Taking 4-bit integers, 3-bit integers, and 2-bit integers as examples, of the total number of network layers included in the model to be quantized, the number of network layers with 4-bit integers accounts for 50%, the number of network layers with 3-bit integers accounts for 25%, and the number of network layers with 2-bit integers accounts for 25%.

[0046] The quantization parameter group is a floating-point parameter group used when quantizing the model parameters in the corresponding group. The group includes multiple quantization parameters. The multiple quantization parameters are in a nonlinear relationship. The nonlinear relationship can be understood as the difference between adjacent quantization parameters in the sorting result after the quantization parameters are sorted in a certain order. The data format of the quantization parameter group is consistent with the data format of the model parameters in the model to be quantized, and both are floating-point data formats. If the data format of the model parameters is 32-bit floating-point, then the data format of the quantization parameters in the quantization parameter group is also 32-bit floating-point. If the data format of the model parameters is 16-bit floating-point, then the data format of the quantization parameters in the quantization parameter group is also 16-bit floating-point.

[0047]

[0046] It should be noted here that the number of quantization parameters in the quantization parameter group is determined by the target integer bit number. If the target integer bit number is 4 bits, then the number of quantization parameters in the corresponding quantization parameter group is 24 = 16. If the target integer bit number is 3 bits, then the number of quantization parameters in the corresponding quantization parameter group is 23 = 8. If the target integer bit number is 2 bits, then the number of quantization parameters in the corresponding quantization parameter group is 22 = 4.

[0048]

[0047] The quantized model parameters are parameters quantized to the target integer bit number by the quantization parameter group. If the target integer bit number is 2, the corresponding integer values ​​are 0 (binary 00), 1 (binary 01), 2 (binary 10), and 3 (binary 11). Then the values ​​of the quantized model parameters include 0, 1, 2, and 3.

[0049]

[0048] Each quantization parameter in the quantization parameter group corresponds to an integer value. Assuming a 2-bit integer, there are 4 quantization parameters in the quantization parameter group, the largest quantization parameter corresponds to the integer value 3, the second largest quantization parameter corresponds to the integer value 2, the third largest quantization parameter corresponds to the integer value 1, and the smallest quantization parameter corresponds to the integer value 0. Therefore, during quantization, for any model parameter in the group, the quantization parameter closest to the any model parameter is determined in the quantization parameter group, and the integer value corresponding to the closest quantization parameter is used as the quantized model parameter of the any model parameter.

[0050]

[0049] Optionally, a model generally includes two model parameters, namely weight and bias. Since the value of the bias is generally small, the embodiment of the present disclosure can quantify the model parameters such as weight in the model.

[0051] The quantized model is a model obtained by converting the data format of the model parameters in the model to be quantized into an integer data format. In the quantized model, the model parameters of each network layer are all in integer data format, and the number of integer bits in different network layers varies. Assuming that the data format of the model parameters in the model to be quantized is 16-bit floating point, the corresponding data formats of the model parameters in the quantized model are 2-bit integer, 3-bit integer, and 4-bit integer. In other words, the model parameters of the model to be quantized are mapped into a 2-bit to 4-bit integer space, which is equivalent to compressing the size of the model to be quantized to approximately 25% of its original size, significantly reducing the storage space occupied by the model.

[0052]

[0051] For example, taking the large language model OPT-175b model as an example, the model parameters of each network layer in the OPT-175b model are grouped to obtain multiple groups. Then, a quantization parameter group of any group is determined according to the target integer bit number required by the network layer to which any group belongs. The model parameters of any group are quantized according to the quantization parameter group to obtain the quantized model parameters of any group. Finally, the quantized model parameters of each group are used to replace the corresponding model parameters in the OPT-175b model to obtain the quantized OPT-175b model.

[0053] At this point, the model quantization process shown in FIG. 1 is completed. Model parameters are grouped and quantized to reduce the impact of outliers in the model parameters on quantization errors. When quantizing by group, the quantization parameter group of the group is determined based on the target integer bit number of the network layer to which it belongs. Since the quantization parameters in the quantization parameter group have a nonlinear relationship, the model parameters in the group can be quantized in an asymmetric manner based on the quantization parameter group to ensure the result generation capability during model operation. Moreover, since each network layer corresponds to a target integer bit number, the model parameters of each network layer can be quantized to different integer spaces, thereby minimizing the storage space occupied by the model.

[0054]

[0053] In some embodiments of the present disclosure, the process of determining the target number of integer bits required for each network layer in the model to be quantized specifically includes the following steps: obtaining the network layer allocation ratio set for each integer bit number; for each network layer, determining the model loss when the network layer is quantized to each integer bit number; the model loss is obtained when the network layer in the model to be quantized is quantized to a data format of the corresponding integer bit number and other network layers remain unchanged; and determining the target number of integer bits required for each network layer based on the model loss and the network layer allocation ratio of each integer bit number.

[0055] Each integer bit number can be understood as a different integer space into which the model to be quantized is compressed and quantized. The network layer allocation ratio is the proportion of the number of network layers of any integer bit number preset by the user relative to the total number of network layers in the entire model. Taking the integer bit number including 2-bit integer, 3-bit integer and 4-bit integer as an example, the network layer allocation ratio of 2-bit integer is set to 25%,

[0056] The network layer allocation ratio of a 3-bit integer is set to 25%, and the network layer allocation ratio of a 4-bit integer is set to 50%.

[0057] Model loss is the output loss of the prediction result of the model to be quantized relative to the prediction result of the model after quantizing only one network layer. It is used to characterize the impact of converting the model parameters in the network layer into a data format with a corresponding integer number of bits on the model generation capability. The greater the model loss, the greater the impact on the model generation capability. The process of obtaining the model loss is specifically as follows: quantizing the corresponding network layer in the model to be quantized to the corresponding integer number of bits, while keeping the other network layers unchanged, to obtain a test model; then inputting a test sample into the test model to obtain the prediction result of the test model; and then inputting the test sample into the model to be quantized to obtain the prediction result of the model to be quantized. The model loss is then determined using the prediction results of the test model and the prediction results of the model to be quantized. Quantizing the corresponding network layer in the model to be quantized to the corresponding integer number of bits can be achieved using the quantization method described in steps 101 and 102 above.

[0058] For each network layer, there is a model loss corresponding to each integer bit number. Taking the integer bit number including 2-bit integer, 3-bit integer and 4-bit integer as an example, for each network layer, when quantized to 2-bit integer, there is a corresponding model loss, when quantized to 3-bit integer, there is a corresponding model loss, and when quantized to 4-bit integer, there is a corresponding model loss.

[0059]

[0057] The target integer bit number is one of the integer bit numbers, and is used to characterize the data format of the model parameters in the corresponding network layer being quantized into the data format of the target integer bit number.

[0060]

[0058] Optionally, the process of determining the target number of integer bits required for each network layer based on the model loss and the network layer allocation ratio can be performed by obtaining the model loss difference between adjacent integer bits of each network layer, and determining the target number of integer bits required for each network layer based on the model loss difference between adjacent integer bits of each network layer and the network layer allocation ratio of each integer bit.

[0061]

[0059] For example, taking the case where the number of integer bits includes 2-bit integer, 3-bit integer and 4-bit integer, the network layer allocation ratio of 2-bit integer is 25%, the network layer allocation ratio of 3-bit integer is 25%, and the network layer allocation ratio of 4-bit integer is 50%. For each network layer, the first model loss difference between the 4-bit integer and the 3-bit integer, and the second model loss difference between the 3-bit integer and the 2-bit integer are obtained. Then, from each network layer, 50% of the network layers (i.e., half of the network layers in the model) are selected and the target integer number of bits is determined to be 4-bit integer, and the first model loss difference of the 50% network layers is smaller than the first model loss difference of the remaining 50% network layers. Then, from the remaining 50% of the network layers, 25% of the network layers are selected and the target integer number is determined to be 3 integer digits, and the second model loss difference of these 25% of the network layers is smaller than the second model loss difference of the remaining 25% of the network layers. Finally, the target integer number of the remaining 25% of the network layers is determined to be 2 integer digits.

[0062] In this embodiment, by determining the model loss when the network layers are quantized into different integer bit numbers, and determining the target integer bit number required for each network layer based on the model loss and the network layer allocation ratio of each integer bit number, it is possible to ensure that different network layers in the model are quantized into different integer spaces without reducing the model generation capability.

[0063]

[0061] In some embodiments of the present disclosure, the model parameters of each network layer in the to-be-quantized model are grouped in step 101 above. Specifically, the model parameter matrices in each network layer are grouped by column.

[0064] The model parameters of the network layer are calculated in matrix form. During the matrix calculation process, they are read into the register in columns and multiplied and added with the input features. In the register, each column of parameters in the model parameter matrix is ​​read and multiplied and added with the input features once. For example, if the input features are m*k, the model parameter matrix is ​​k*n, and the matrix calculation result is m*n,

[0065]

[0063] It can be seen that in actual model operation, outliers affect the results of the operations in which the columns are involved. By grouping the model parameters in the network layer by columns, the impact of quantization errors can be minimized.

[0066]

[0064] Exemplarily, assuming that the model parameter matrix of the network layer is k*n, the model parameter matrix can be divided into n groups, each group having k model parameters.

[0067]

[0065] In this embodiment, the influence of outliers on quantization error can be minimized by grouping the model parameter matrix of the network layer by columns.

[0068]

[0066] In some embodiments of the present disclosure, the process of determining the quantization parameter group of the group according to the target number of integer bits required by the network layer to which the group belongs in the above step 102 specifically includes the following steps: determining an initial quantization parameter group for the group according to the target number of integer bits required by the network layer to which the group belongs, wherein each quantization parameter in the initial quantization parameter group has a linear relationship with each other; and determining the quantization parameter group for the group based on the initial quantization parameter group and the model parameters of the group, wherein the data format of the quantization parameters in the quantization parameter group is consistent with the data format of the model parameters of the group.

[0069]

[0067] The target integer bit number indicates the number of bits required to convert the model parameters in the floating point data format in the corresponding network layer into the integer data format. The target integer bit number can be any one of an 8-bit integer (INT8), a 4-bit integer (INT4), a 3-bit integer (INT3), and a 2-bit integer (INT2).

[0070] The initial quantization parameter group is an initial floating-point quantization parameter group determined based on a target integer bit number. The initial quantization parameter group includes multiple quantization parameters, and the multiple quantization parameters are linearly related. This linear relationship can be understood as that after the quantization parameters are sorted in a certain order, the differences between adjacent quantization parameters in the sorting result are the same. The data format of the initial quantization parameter group is consistent with the data format of the model parameters in the model to be quantized, both of which are floating-point data formats. If the data format of the model parameters is 32-bit floating-point, the data format of the quantization parameters in the initial quantization parameter group is also 32-bit floating-point. If the data format of the model parameters is 16-bit floating-point, the data format of the quantization parameters in the initial quantization parameter group is also 16-bit floating-point.

[0071]

[0069] The quantization parameter group can be understood as a floating-point quantization parameter obtained by optimizing the quantization parameters in the initial quantization parameter group according to the grouped model parameters.

[0072]

[0070] Based on the initial quantization parameter group and the model parameters of the group, the quantization parameter group of the group is determined, specifically by using an EM (Expectation-Maximization algorithm) algorithm to calculate the quantization parameter group when the expectation is maximized using the model parameters of the group and the initial quantization parameter group.

[0073]

[0071] In the initialization stage before training, the model parameters are initialized using the Gaussian distribution as a prior distribution. During the model training process, the Gaussian distribution is also used as an assumption to constrain the model parameters. Therefore, the model parameters of each network layer can be regarded as a mixed Gaussian (GM) distribution composed of multiple Gaussian distributions. Then, each model parameter in the network layer can be attributed to a certain Gaussian distribution, and the expected value of the Gaussian distribution can be used to replace the model parameter. Therefore, each quantization parameter in the initial quantization parameter group can be regarded as the expected value of a Gaussian distribution. In order to make each quantization parameter in the initial quantization parameter group as close as possible to the expected value of the actual distribution, the EM algorithm is used to iteratively solve the problem using the grouped model parameters and the initial quantization parameter group. The expected combination with the maximum expectation is the most appropriate set of quantization parameters.

[0074]

[0072] Optionally, the process of determining an initial quantization parameter group for a group according to a target number of integer bits required by the network layer to which the group belongs specifically includes the following steps: determining the required number of quantization parameters according to the target number of integer bits; obtaining a maximum model parameter and a minimum model parameter in the group; and selecting quantization parameters at equal intervals between the maximum model parameter and the minimum model parameter according to the number of quantization parameters to obtain an initial quantization parameter group.

[0075]

[0073] The initial quantization parameter group may include a maximum model parameter, a minimum model parameter, and a value selected between the maximum model parameter and the minimum model parameter.

[0076] For example, assuming that the target integer bit number required by the network layer to which the group belongs is a 2-bit integer, then the number of required quantization parameters is 4. The range of the model parameters in the group is -0.6 to 1.5. -0.6 and 1.5 are used as two of the quantization parameters. Then, two quantization parameters are selected at equal intervals from -0.6 to 1.5: 0.1 and 0.8, resulting in an initial quantization parameter set of {-0.6, 0.1, 0.8, 1.5}. Then, {-0.6, 0.1, 0.8, 1.5} and the model parameters in the group are input into the EM algorithm for iterative solution, obtaining a quantization parameter set output by the EM algorithm of {-0.5, 0.1, 0.9, 1.1}.

[0077] In this embodiment, the quantization parameters in the initial quantization parameter group with a linear relationship are tuned by using the model parameters in the group to obtain an optimized quantization parameter group with a nonlinear relationship, so that the quantization parameters in the quantization parameter group are more consistent with the expected values ​​in the actual distribution of the model parameters, thereby ensuring that the generation capability of the quantized model is not affected.

[0078]

[0076] In some embodiments of the present disclosure, the process of quantizing the model parameters of the group according to the quantization parameter group in the above-mentioned step 102 to obtain the quantized model parameters of the group specifically includes the following steps: determining the integer value corresponding to each quantization parameter in the quantization parameter group according to the target integer bit number required by the network layer to which the group belongs; for each model parameter of the group, obtaining the target quantization parameter closest to the model parameter in the quantization parameter group; and using the integer value corresponding to the target quantization parameter as the quantized model parameter of the model parameter.

[0079]

[0077] The target integer bit number indicates the number of bits required to convert the model parameters in the floating-point data format in the corresponding network layer into the integer data format. For example, if the target integer bit number is 2, the corresponding integer values ​​are 0 (binary 00), 1 (binary 01), 2 (binary 10), and 3 (binary 11). Assuming that the quantization parameter group = {-0.5, 0.1, 0.9, 1.1}, then the integer value corresponding to the quantization parameter -0.5 is 0, the integer value corresponding to the quantization parameter 0.1 is 1, the integer value corresponding to the quantization parameter 0.9 is 2, and the integer value corresponding to the quantization parameter 1.1 is 3.

[0080]

[0078] The target quantization parameter closest to the model parameter is obtained in the quantization parameter group. The specific method is: calculating the difference between the model parameter and each quantization parameter in the quantization parameter group, and selecting the quantization parameter with the smallest difference as the target quantization parameter.

[0081]

[0079] The quantized model parameters are quantized parameters, and their data format is the data format of the target integer bit number.

[0082]

[0080] For example, assuming that the quantization parameter group = {-0.5, 0.1, 0.9, 1.1}, the corresponding integer values ​​are 0, 1, 2, 3 respectively. If a model parameter in the group is 0.8, the difference between the model parameter and each quantization parameter in the quantization parameter group is calculated to obtain 1.3, 0.7, -0.1, -0.3, among which -0.1 is the minimum difference. Therefore, the target quantization parameter is 0.9, and the integer value corresponding to the quantization parameter of 0.9 is 2. Therefore, the model parameter of 0.8 can be quantized to 2.

[0083]

[0081] In this embodiment, the corresponding model parameters are quantized by using integer values ​​corresponding to target quantization parameters that are closest to the model parameters, so as to ensure that the generation capability of the quantized model is not affected.

[0084]

[0082] In some other embodiments of the present disclosure, after obtaining the quantization parameter groups of each group, the quantization parameter groups of each group can be stored in a preset shared memory, so as to quickly read the quantization parameter groups from the shared memory when the model is running, thereby accelerating the model running speed.

[0085]

[0083] Since the size of the quantization parameters of each grouped quantization parameter group is relatively small, the quantization parameter group is stored in a shared memory using a proximal storage method to speed up the running speed of the model.

[0086]

[0084] And, after obtaining the quantized model, the quantized model is stored in a preset memory so that the model parameters can be read from the memory when the model needs to be run.

[0087] Based on the above-mentioned embodiment of model quantization, FIG2 is a flow chart of an embodiment of a model operation method according to an exemplary embodiment. The model operated in this embodiment is obtained using the model quantization method shown in FIG1 . As shown in FIG2 , the model operation method includes the following steps 201 to 203.

[0088]

[0086] Step 201: For each operation in the model running process, obtain the stored quantization parameter group corresponding to the model parameters required for the current operation; wherein the model parameters required for each operation belong to the same group.

[0089]

[0087] Step 202: Dequantize the model parameters required for the current operation according to the quantization parameter group.

[0090]

[0088] Step 203: Utilize the dequantized model parameters to perform the current operation.

[0091]

[0089] In the embodiment of the present disclosure, each operation in the model running process can be understood as each multiplication and addition operation in the matrix operation. Each multiplication and addition operation is to first multiply the input feature with a column of parameters in the model parameter matrix of the network layer and then add the products. The multiplication and addition operation is implemented by using an operation unit of the computer. For example, in a GPU, a register is used to implement a multiplication and addition operation. Assume that the input feature of the network layer is m*k, and the model parameter matrix of the network layer is k*n. The input feature parameters of m*k are input into the register. Then, each time a column of parameters is read in the model parameter matrix k*n, the control register performs an operation between the input feature and the column of parameters to obtain a column of output data. Until the last column of parameters is read in the model parameter matrix k*n, the control register performs an operation between the input feature and the last column of parameters to obtain the last column of output data, and the operation of the network layer ends.

[0092] The model parameters required for the current operation are a column of parameters in the model parameter matrix of the network layer. These are quantized model parameters in integer format. As described in the previous embodiment, the model parameters of each network layer are grouped by column, so the model parameters required for each operation belong to the same group.

[0093] The quantization parameter group is a floating-point parameter group used when quantizing the model parameters in the corresponding group. The group includes multiple quantization parameters, and the multiple quantization parameters have a nonlinear relationship. This nonlinear relationship can be understood as the difference between adjacent quantization parameters in the sorting result after the quantization parameters are sorted in a certain order. The data format of the quantization parameter group is floating-point, which is the same as the data format of the model parameters before model quantization. Specifically, the quantization parameter group corresponding to the group to which the model parameters required for the current operation belong can be obtained from a preset shared memory.

[0094]

[0092] Dequantization refers to converting the data format of the model parameters required for the current operation into a floating-point data format before quantization for use in executing the current operation.

[0095] The dequantized model parameters are floating-point parameters of the model parameters required for the current operation after data format conversion. The floating-point type of the dequantized model parameters is the same as the floating-point type of the quantization parameters in the corresponding quantization parameter group. For example, if the quantization parameters in the quantization parameter group are 16-bit floating-point types, the floating-point type of the quantized model parameters is also 16-bit floating-point types.

[0096]

[0094] For example, taking the quantized large language model OPT-175b model as an example, for each operation in the running process of the OPT-175b model, the model parameters required for the current operation are obtained from the OPT-175b model, and a quantization parameter group corresponding to the model parameters required for the current operation is obtained. The model parameters required for the current operation are dequantized according to the obtained quantization parameter group, so that the current operation is performed using the dequantized model parameters.

[0097] Based on the above-mentioned second embodiment, for each operation during the model operation, since the model parameters required for each operation belong to the same group, the same quantization parameter group can be used to dequantize the obtained model parameters and perform the operation. In this way, the model operation can be completed by only adding the acquisition of the quantization parameter group. Under the premise of ensuring that the model operation speed is not affected, a generation result that is more similar to the original model can be obtained, thereby reducing model loss.

[0098]

[0096] In some embodiments of the present disclosure, the process of dequantizing the model parameters required for the current operation according to the quantization parameter group in step 202 specifically includes the following steps: obtaining a target quantization parameter corresponding to any model parameter in the model parameters required for the current operation in the quantization parameter group; and using the target quantization parameter as the dequantized model parameter of any model parameter.

[0099]

[0097] The data format of the model parameters required for the current operation is integer, which is a column of parameters in the model parameter matrix. Therefore, it is necessary to obtain the corresponding target quantization parameter for each model parameter.

[0100]

[0098] For example, assuming that the quantization parameter group = {-0.5, 0.1, 0.9, 1.1}, then among these four quantization parameters, the integer value corresponding to the quantization parameter -0.5 is 0, the integer value corresponding to the quantization parameter 0.1 is 1, the integer value corresponding to the quantization parameter 0.9 is 2, and the integer value corresponding to the quantization parameter 1.1 is 3. Therefore, if the model parameter is an integer value 3, then the corresponding target quantization parameter is 1.1, and 1.1 is used as the model parameter after dequantization.

[0101]

[0099] In this embodiment, by obtaining the target quantization parameter corresponding to the model parameter in the quantization parameter group, the inverse quantization can be directly completed without any complex calculation, thereby speeding up the model operation.

[0102] Based on the above-described model quantization embodiment and model operation embodiment, FIG3 is a schematic diagram of the overall process of model quantization and operation according to an exemplary embodiment. First, a model before quantization, i.e., the model to be quantized, is obtained. Since the data format of the model parameters in the model to be quantized is floating point and the number of parameters is large, the current storage space is insufficient to store the model parameters of the model to be quantized. The solution of the above-described model quantization embodiment is used to quantize the currently obtained model to obtain a quantized model. Since the size of the quantized model is significantly reduced compared to the pre-quantization model, the quantized model can be stored in the current storage space, and the quantization parameter group generated during the quantization process can be stored in the shared memory. The storage space used to store the model parameters can be a memory.

[0103]

[0101] When the quantized model stored in the storage space starts to run, for each operation in the model running process, the model parameters required for the current operation are read from the storage space to the shared memory, and then the quantization parameter group corresponding to the current model parameters is read, and the current model parameters are dequantized using the read quantization parameter group, so that the current operation is executed using the dequantized model parameters, until all operations are completed and the model running is completed.

[0104]

[0102] The execution subject of the embodiments of the present disclosure may be an application, service, instance, software-based functional module, virtual machine (VM), container, or cloud server, or a hardware device (such as a server or terminal device) or hardware chip (such as a CPU, GPU, FPGA, NPU, AI accelerator card, or DPU) with data processing capabilities. The device for implementing model quantization or model operation may be deployed on a computing device of the application providing the corresponding service or on a cloud computing platform that provides computing power, storage, and network resources. The cloud computing platform may provide external services in the form of IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service), or DaaS (Data as a Service). Taking the platform providing SaaS Software as a Service as an example, the cloud computing platform may use its own computing resources to provide model quantization or model operation, model training, or the functional execution of the model quantization or model operation module. The specific application architecture may be built according to service requirements. For example, the platform can provide construction services based on the above-mentioned model to application parties or individuals using platform resources, and further call the above-mentioned model and implement online or offline model quantification or model operation functions based on model quantification or model operation requests submitted by relevant client or server devices.

[0105]

[0103] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0106]

[0104] Corresponding to the embodiments of the aforementioned model quantization method, the present disclosure also provides embodiments of a model quantization device.

[0107]

[0105] FIG4 is a schematic structural diagram of a model quantization device according to an exemplary embodiment. The device is used to execute the model quantization method provided in any of the above embodiments. As shown in FIG4, the model quantization device includes a grouping module 410, a parameter quantization module 420, and a parameter replacement module 430.

[0108]

[0106] The grouping module 410 is used to group the model parameters of each network layer in the quantization model.

[0109]

[0107] A parameter quantization module 420 is configured to determine, for each group, a quantization parameter group for the group based on a target number of integer bits required by the network layer to which the group belongs, and quantize the model parameters of the group based on the quantization parameter group to obtain quantized model parameters of the group; the quantization parameter group includes multiple quantization parameters, and the multiple quantization parameters are nonlinearly related.

[0110]

[0108] The parameter replacement module 430 is used to replace the corresponding model parameters in the to-be-quantized model with the quantized model parameters of each group to obtain the quantized model.

[0111]

[0109] In an optional implementation, the device further includes (not shown in FIG4 ): a bit number determination module, configured to obtain a network layer allocation ratio set for each integer bit number; determining, for each network layer, a model loss when the network layer is quantized into each integer bit number; the model loss is obtained when the network layer in the to-be-quantized model is quantized into a data format of a corresponding integer bit number while other network layers remain unchanged; and determining a target integer bit number required for each network layer based on the model loss and the network layer allocation ratio of each integer bit number.

[0112]

[0110] In an optional implementation, the parameter quantization module 420 is specifically configured to, in a process of determining the quantization parameter group for the group according to the target number of integer bits required by the network layer to which the group belongs, determine an initial quantization parameter group for the group according to the target number of integer bits required by the network layer to which the group belongs; quantization parameters in the initial quantization parameter group are linearly related to each other; and determine the quantization parameter group for the group based on the initial quantization parameter group and the model parameters of the group; wherein a data format of the quantization parameters in the quantization parameter group is consistent with a data format of the model parameters of the group.

[0113]

[0111] In an optional implementation, the parameter quantization module 420 is specifically configured to, in determining the initial quantization parameter group of the group according to the target number of integer bits required by the network layer to which the group belongs, determine the required number of quantization parameters according to the target number of integer bits; obtain the maximum model parameter and the minimum model parameter in the group; and select quantization parameters at equal intervals between the maximum model parameter and the minimum model parameter according to the number of quantization parameters to obtain the initial quantization parameter group.

[0114]

[0112] In an optional implementation, the parameter quantization module 420 is specifically configured to, in a process of quantizing the model parameters of the group according to the quantization parameter group to obtain the quantized model parameters of the group, determine the integer value corresponding to each quantization parameter in the quantization parameter group according to a target integer bit number required by the network layer to which the group belongs; for each model parameter of the group, obtain a target quantization parameter in the quantization parameter group that is closest to the model parameter; and use the integer value corresponding to the target quantization parameter as the quantized model parameter of the model parameter.

[0115]

[0113] In an optional implementation, the device further includes (not shown in FIG4 ): a storage module configured to store the quantization parameter groups of each group in a preset shared memory; and to store the quantized model in a preset memory.

[0116]

[0114] In an optional implementation, the grouping module 410 is specifically configured to group the model parameter matrices in each network layer by columns.

[0117]

[0115] Corresponding to the embodiments of the aforementioned model operation method, the present disclosure also provides embodiments of a model operation device.

[0118]

[0116] Figure 5 is a structural diagram of a model running device according to an exemplary embodiment. The device is used to execute the model running method provided by any of the above embodiments. As shown in Figure 5, the model running device includes a quantization parameter acquisition module 510, an inverse quantization module 520 and an operation module 530.

[0119]

[0117] The quantization parameter acquisition module 510 is used to acquire, for each operation in the model running process, a stored quantization parameter group corresponding to the model parameters required for the current operation; the model parameters required for each operation belong to the same group.

[0120]

[0118] The dequantization module 520 is used to dequantize the model parameters required for the current operation according to the quantization parameter group.

[0121]

[0119] The operation module 530 is configured to perform the current operation using the inverse quantized model parameters.

[0120] In an optional implementation, the inverse quantization module 520 is specifically configured to obtain, from the quantization parameter group, a target quantization parameter corresponding to any model parameter in the model parameters required for the current operation; and use the target quantization parameter as the inverse quantized model parameter of the any model parameter.

[0122]

[0121] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0123]

[0122] As for the device embodiments, since they basically correspond to the method embodiments, reference will be made to the partial description of the method embodiments for relevant details. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the objectives of the disclosed solution. Those skilled in the art can understand and implement the present invention without inventive effort.

[0124]

[0123] The embodiment of the present disclosure also provides an electronic device corresponding to the model quantization method and model operation method provided in the above embodiment to execute the above model quantization method and model operation method.

[0125] FIG6 is a hardware structure diagram of an electronic device according to an exemplary embodiment. The electronic device includes a communication interface 601, a processor 602, a memory 603, and a bus 604. The communication interface 601, the processor 602, and the memory 603 communicate with each other via the bus 604. The processor 602 can execute the model quantization method or model operation method described above by reading and executing machine-executable instructions corresponding to the control logic of the model quantization method or model operation method in the memory 603. The specific content of the method is described in the above embodiment and will not be repeated here.

[0126]

[0125] The memory 603 mentioned in the present disclosure may be any electronic, magnetic, optical, or other physical storage system, and may contain stored information, such as executable instructions, data, and the like. Specifically, the memory 603 may be RAM (Random Access Memory), flash memory, a storage drive (such as a hard disk drive), any type of storage disk (such as an optical disk, a DVD, etc.), or similar storage media, or a combination thereof. The communication connection between the system network element and at least one other network element is achieved through at least one communication interface 601 (which may be wired or wireless), and may use the Internet, a wide area network, a local area network, a metropolitan area network, and the like.

[0127]

[0126] The bus 604 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 603 is used to store programs, and the processor 602 executes the programs after receiving an execution instruction.

[0128]

[0127] The processor 602 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be completed by hardware integrated logic circuits or software instructions in the processor 602. The processor 602 may be a general-purpose processor, including a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor.

[0129]

[0128] The electronic device provided by the embodiment of the present disclosure and the model quantization method and model operation method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0130]

[0129] The embodiments of the present disclosure also provide a computer-readable storage medium corresponding to the model quantization method and model operation method provided in the aforementioned embodiments. Please refer to Figure 7, where the computer-readable storage medium shown is a CD 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the model quantization method and model operation method provided in any of the aforementioned embodiments.

[0131]

[0130] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0132]

[0131] The computer-readable storage medium provided by the above-mentioned embodiments of the present disclosure and the model quantization method and model operation method provided by the embodiments of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the methods adopted, operated or implemented by the application programs stored therein.

[0133]

[0132] An embodiment of the present disclosure also provides a computer program product corresponding to the model quantization method and model operation method provided in the aforementioned embodiment. The computer program product includes a computer program, which is executed by a processor to implement the model quantization method and model operation method provided in the aforementioned embodiment.

[0134]

[0133] The computer program product provided by the above-mentioned embodiments of the present disclosure and the model quantization method or model operation method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by the application programs stored therein.

[0135]

[0134] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0136]

[0135] It should also be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the process, method, commodity, or apparatus comprising the element.

[0137]

[0136] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

Claims 1. A model quantification method, comprising: Group the model parameters of each network layer in the model to be quantized; For each group, determining a quantization parameter group for the group based on a target number of integer bits required by a network layer to which the group belongs, and quantizing a model parameter of the group based on the quantization parameter group to obtain a quantized model parameter of the group; the quantization parameter group includes a plurality of quantization parameters, and the plurality of quantization parameters are nonlinearly related. The quantized model parameters of each group are used to replace the corresponding model parameters in the model to be quantized to obtain a quantized model.

2. The method according to claim 1, wherein: The process of determining the target number of integer bits required for each network layer in the model to be quantized includes: obtaining a network layer allocation ratio set for each integer bit number; determining, for each network layer, a model loss when the network layer is quantized to each integer bit number; the model loss is obtained when the network layer in the model to be quantized is quantized to a data format of a corresponding integer bit number while other network layers remain unchanged; and determining the target number of integer bits required for each network layer based on the model loss and the network layer allocation ratio of each integer bit number.

3. The method according to claim 1 or 2, wherein: Determining the quantization parameter group for the group based on a target number of integer bits required by the network layer to which the group belongs includes: determining an initial quantization parameter group for the group based on the target number of integer bits required by the network layer to which the group belongs; wherein each quantization parameter in the initial quantization parameter group has a linear relationship with each other; and determining the quantization parameter group for the group based on the initial quantization parameter group and model parameters of the group; wherein a data format of the quantization parameters in the quantization parameter group is consistent with a data format of the model parameters of the group.

4. The method according to claim 3, wherein: Determining an initial quantization parameter group for the group based on a target number of integer bits required by a network layer to which the group belongs includes: determining a required number of quantization parameters based on the target number of integer bits; obtaining a maximum model parameter and a minimum model parameter in the group; and selecting quantization parameters at equal intervals between the maximum model parameter and the minimum model parameter based on the number of quantization parameters to obtain the initial quantization parameter group.

5. The method according to claim 1 or 2, wherein: The quantizing the grouped model parameters according to the quantization parameter group to obtain the quantized model parameters of the group includes: Determine the integer value corresponding to each quantization parameter in the quantization parameter group based on the target number of integer bits required by the network layer to which the group belongs; for each model parameter of the group, obtain the target quantization parameter closest to the model parameter in the quantization parameter group; and use the integer value corresponding to the target quantization parameter as the quantized model parameter of the model parameter.

6. The method according to claim 1 or 2, further comprising: Storing the quantization parameter groups of each group in a preset shared memory; The quantized model is stored in a preset memory.

7. The method according to claim 1 or 2, wherein: The grouping of the model parameters of each network layer in the model to be quantized includes: grouping the model parameter matrices in each network layer by column.

8. A model operation method, wherein: The model is quantized using the method described in any one of claims 1 to 7, the method comprising: obtaining, for each operation during the model operation, a stored quantization parameter group corresponding to the model parameters required for the current operation; the model parameters required for each operation belong to the same group; dequantizing the model parameters required for the current operation based on the quantization parameter group; and executing the current operation using the dequantized model parameters.

9. The method according to claim 8, wherein: The dequantizing of the model parameters required for the current operation according to the quantization parameter group includes: obtaining, in the quantization parameter group, a target quantization parameter corresponding to any model parameter in the model parameters required for the current operation; and using the target quantization parameter as the dequantized model parameter of the any model parameter.

10. A model quantization device, comprising: A grouping module, used to group the model parameters of each network layer in the quantization model; a parameter quantization module, configured to determine, for each group, a quantization parameter group for the group based on a target number of integer bits required by the network layer to which the group belongs, and quantize the model parameters of the group based on the quantization parameter group to obtain quantized model parameters of the group; the quantization parameter group includes a plurality of quantization parameters, and the plurality of quantization parameters have a nonlinear relationship; The parameter replacement module is used to replace the corresponding model parameters in the to-be-quantized model with the quantized model parameters of each group to obtain the quantized model.

11. A model operation device, wherein: The model is quantitatively obtained by using the device described in claim 10, The device includes: a quantization parameter acquisition module, configured to acquire, for each operation in a model running process, a stored quantization parameter group corresponding to the model parameters required for the current operation; the model parameters required for each operation belong to the same group; The inverse quantization module is used to inversely quantize the model parameters required for the current operation according to the quantization parameter group; and the operation module is used to perform the current operation using the inverse quantized model parameters.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the program to implement the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having a computer program stored thereon, wherein: The program is executed by a processor to implement the method according to any one of claims 1 to 9.

14. A computer program product, comprising a computer program, wherein: The computer program is executed by a processor to implement the method according to any one of claims 1 to 9.

Images