Weight data quantification method and device and electronic equipment
Through the layer-by-layer quantization parameter determination method, combined with activation value error and weight loss function, the problem of high bandwidth reading bandwidth of large model weight data and difficulty in training quantization parameters is solved, and efficient weight data quantization is achieved, taking into account prediction accuracy and generalization ability.
Patent Information
- Application Number
- CN202411217119.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-17
AI Technical Summary
The weighted data reading bandwidth consumption of large models during the decoding stage is high, resulting in performance bottlenecks, and multi-processing layers lead to difficulty in training quantization parameters, and traditional quantization methods may lead to loss of prediction accuracy.
The layer-by-layer quantization parameter determination method is adopted, combining the activation value error and weight loss function, and gradually transitioning to global quantization through local quantization, the quantization parameters of each layer are updated to reduce the quantization loss and avoid the accuracy loss caused by independent quantization.
It effectively reduces the resource consumption of weighted data, solves the difficulty of training quantization parameters in multi-processing layers, and takes into account prediction accuracy and generalization capabilities, improving quantization accuracy and efficiency.
Smart Images
Figure CN119150138A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of model processing technology, and more specifically, to a method, device and electronic device for quantizing weight data. Background Art
[0002] The current performance bottleneck of large models is mainly the bandwidth consumption for reading weight data. During the decoding phase of a large model, each time a token is generated, the complete weight data needs to be read once. The weight data of a large model is very large and requires a lot of resources, which affects the overall performance of the large model. Therefore, quantization of the weight data of the large model is very necessary. Since the large model contains many processing layers, the quantization of its weight data also has the problem of difficulty in training quantization parameters. Summary of the invention
[0003] In view of this, this application provides the following technical solutions:
[0004] The first aspect of the present application provides a method for quantifying weight data, comprising:
[0005] Determine quantization parameters for all processing layers of the artificial intelligence model to be quantized layer by layer, wherein the quantization parameters determined for each processing layer include the quantization parameter of the current processing layer and the quantization parameters of all processing layers before the current processing layer, the quantization parameter of the current processing layer is determined for the first time, and the quantization parameters of all processing layers before the current processing layer are updated based on their original quantization parameters;
[0006] After the quantization parameters of all processing layers of the artificial intelligence model to be quantized are determined, the quantized weights of each processing layer are determined based on the quantization parameters of each processing layer.
[0007] In one possible implementation, the loss function used to determine the quantization parameter includes a first sub-function and a second sub-function, wherein the first sub-function represents the activation value error of the current processing layer, and the second sub-function represents the weight loss of the current quantization parameter determination process.
[0008] In a possible implementation, the weight loss represents the accumulation of weight errors of the current processing layer and all previous processing layers, and is determined based on the weight error of the current processing layer and the weight errors of each processing layer before the current processing layer.
[0009] In a possible implementation, determining quantization parameters layer by layer for all processing layers of the artificial intelligence model to be quantized includes:
[0010] Determine an initial quantization parameter and a quantized weight of the i-th processing layer based on the original weight of the i-th processing layer;
[0011] Based on the loss function, the initial quantization parameter of the i-th processing layer and the quantization parameters of the 1st to (i-1) processing layers are trained to obtain target quantization parameters, which include N quantization parameters from the 1st processing layer to the current processing layer, where N is the same as i.
[0012] In a possible implementation, determining the initial quantization parameter and the quantized weight of the i-th processing layer based on the original weight of the i-th processing layer includes:
[0013] Based on the nearest neighbor method, the initial quantization parameter of the i-th processing layer and the corresponding initial quantized weight are calculated through the original weight of the i-th processing layer.
[0014] In one possible implementation, the activation value error represents the norm of the output value of the current processing layer after the training data is processed by each processing layer from the first processing layer to the current processing layer before and after quantization of the weight data of the current processing layer; and / or, the weight loss represents the result of weighted summation of the weight errors of the current processing layer and all the previous processing layers.
[0015] In a possible implementation, the input data corresponding to the output value of the weight data of the current processing layer before quantization is the same as the input data corresponding to the output value after quantization of the weight data.
[0016] In a possible implementation, determining the quantized weight of each processing layer based on the quantization parameter of each processing layer includes:
[0017] Based on the quantization parameters of each processing layer, a target quantization method is used to determine the quantized weight of each processing layer, and the target quantization method is any one of the following: group quantization, tensor quantization, and channel quantization.
[0018] A second aspect of the present application provides a weight data quantization device, comprising:
[0019] A parameter determination module, used to determine the quantization parameters of all processing layers of the artificial intelligence model to be quantized layer by layer, wherein the quantization parameters determined for each processing layer include the quantization parameters of the current processing layer and the quantization parameters of all processing layers before the current processing layer, wherein the quantization parameters of the current processing layer are determined for the first time, and the quantization parameters of all processing layers before the current processing layer are updates based on their original quantization parameters;
[0020] The weight determination module is used to determine the quantized weight of each processing layer based on the quantization parameters of each processing layer after the quantization parameters of all processing layers of the artificial intelligence model to be quantized are determined.
[0021] A third aspect of the present application provides an electronic device on which an artificial intelligence model is deployed, and the weights of each processing layer of the artificial intelligence model are determined based on any one of the above-mentioned weight data quantization methods.
[0022] The present application provides a method, device and electronic device for quantizing weight data, wherein the method includes: determining quantization parameters for all processing layers of the artificial intelligence model to be quantized layer by layer, wherein the quantization parameters determined for each processing layer include the quantization parameters of the current processing layer and the quantization parameters of all processing layers before the current processing layer, wherein the quantization parameters of the current processing layer are determined for the first time, and the quantization parameters of all processing layers before the current processing layer are updated based on their original quantization parameters; after the quantization parameters of all processing layers of the artificial intelligence model to be quantized are determined, the quantized weights of each processing layer are determined based on the quantization parameters of each processing layer. The above scheme adopts a quantization method of accumulating multiple processing layers of the artificial intelligence model layer by layer, gradually transitioning from local quantization to global quantization, which can solve the problem of difficulty in training quantization parameters when the number of processing layers of the artificial intelligence model is large, and at the same time avoids the problem of loss of prediction accuracy that may be caused by independent quantization of each processing layer in the traditional scheme. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0024] Figure 1 A flow chart of a method for quantizing weight data disclosed in an embodiment of the present application;
[0025] Figure 2 A flowchart of determining quantization parameters layer by layer disclosed in an embodiment of the present application;
[0026] Figure 3 A schematic diagram of the structure of a weight data quantization device disclosed in an embodiment of the present application. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] Figure 1This is a flow chart of a method for quantizing weight data disclosed in an embodiment of the present application. Figure 1 As shown, the quantization method of weight data may include:
[0029] Step 101: Determine quantization parameters for all processing layers of the artificial intelligence model to be quantized layer by layer, wherein the quantization parameters determined for each processing layer include the quantization parameters of the current processing layer and the quantization parameters of all processing layers before the current processing layer.
[0030] The artificial intelligence module to be quantified can be a machine learning model with a large number of parameters and complex structure, which can be understood as the "big model" in the field. This application does not impose fixed restrictions on the specific type of the big model. For example, it can be a language processing model, image processing model, etc., especially a large language model based on a transform structure.
[0031] Among them, the quantization parameter of the current processing layer is determined for the first time, and the quantization parameters of all processing layers before the current processing layer are updated based on their original quantization parameters. That is, the artificial intelligence model includes multiple processing layers. When the artificial intelligence model is quantized, starting from the first processing layer, the quantization parameters are determined layer by layer in the order of the first processing layer, the second processing layer, the third processing layer...; when the quantization parameters of a certain processing layer are determined, the determined quantization parameters include not only the quantization parameters of the current layer, but also the quantization parameters of all processing layers before the current processing layer. Since the quantization parameters of the processing layers before the current processing layer have been determined before, the quantization parameters originally determined for the processing layers before the current processing layer will be updated during the quantization parameter determination process of the current processing layer.
[0032] For example, when determining the quantization parameters of the first processing layer, the determined quantization parameters only include the quantization parameters of the first processing layer; when determining the quantization parameters of the second processing layer, the determined quantization parameters include the quantization parameters of the first processing layer and the second processing layer, and the originally recorded quantization parameters of the first processing layer will be updated to the latest determined quantization parameters of the first processing layer; when determining the quantization parameters of the third processing layer, the determined quantization parameters include the quantization parameters of the first processing layer, the second processing layer and the third processing layer, and the originally recorded quantization parameters of the first processing layer and the second processing layer will be updated to the latest determined quantization parameters of the first processing layer and the second processing layer, and so on.
[0033] In the above implementation, in addition to determining the quantization parameter of the current processing layer, the quantization parameters of all processing layers before the current processing layer are also updated during the process of determining the quantization parameter of the current processing layer. That is, when determining the quantization parameter of the current processing layer, all previous data related to the current processing layer that may affect the accuracy of the weight / quantization parameter of the current processing layer are considered, thereby reducing the quantization loss and improving the prediction accuracy as a whole.
[0034] Step 102: After the quantization parameters of all processing layers of the artificial intelligence model to be quantized are determined, the quantized weights of each processing layer are determined based on the quantization parameters of each processing layer.
[0035] After determining the quantization parameters of the last processing layer of the artificial intelligence model to be quantized according to the layer-by-layer processing method described in the above content, the quantization parameters determined by the last processing layer can be determined as the final quantization parameters of the artificial intelligence model to be quantized, and the final quantization parameters include the quantization parameters of all processing layers of the artificial intelligence model to be quantized.
[0036] Based on the quantization parameters of each processing layer, the quantized weights of each processing layer can be determined based on the quantization parameters of each processing layer. Since weight data accounts for a large proportion of various data in the artificial intelligence model, after quantizing the original weight data of the artificial intelligence model, the resource overhead on weight data in the artificial intelligence model processing work will be greatly saved.
[0037] The quantization method of weight data described in the embodiment of the present application adopts a quantization method of accumulating multiple processing layers of the artificial intelligence model layer by layer, gradually transitioning from local quantization to global quantization, which can solve the problem of difficulty in training quantization parameters when the artificial intelligence model has a large number of processing layers, and at the same time avoids the problem of loss of prediction accuracy that may be caused by independent quantization of each processing layer in the traditional solution.
[0038] In the above embodiment, the loss function used for determining the quantization parameter includes a first sub-function and a second sub-function, wherein the first sub-function characterizes the activation value error of the current processing layer, and the second sub-function characterizes the weight loss of the current quantization parameter determination process. The current quantization parameter determination process refers to the quantization parameter determination process performed on the current processing layer, and each processing layer performs a quantization parameter determination process. The weight loss of the current quantization parameter determination process may refer only to the weight loss before and after quantization of the current processing layer, or may refer to the accumulation of weight errors of all processing layers including the current processing layer that have undergone quantization parameter determination, or may refer to the accumulation of other weight errors that can characterize at least one processing layer.
[0039] There are two main types of traditional weight compression schemes. One type considers how to minimize the error of activation values before and after compression during the compression process, and the other type considers how to minimize the error of weight values before and after compression during the compression process. The first type of method requires the input of a small batch of training data during the quantization process. Since the original weight values are not considered in the quantization, it is easy to overfit to this batch of given data, making the generalization performance of the quantized artificial intelligence model worse. The second type of method does not require the input of training data at all during the quantization process, and only considers minimizing the error of weight values before and after quantization. The problem with this processing method is that it may reduce the prediction accuracy of the original artificial intelligence model. In addition, the above two types of methods usually have more complicated mathematical derivations and solution processes.
[0040] In the embodiment of the present application, in the process of determining the quantization parameters of the artificial intelligence model processing layer, the loss function used includes not only the activation value error part corresponding to the accuracy of the artificial intelligence model output result, but also the weight loss part corresponding to the generalization ability of the artificial intelligence model. Therefore, it can take into account the prediction accuracy and generalization ability of the original artificial intelligence model and ensure the processing effect of the artificial intelligence model after quantization.
[0041] In one implementation, the weight loss may represent only the weight error of the current processing layer. That is, when determining the quantization parameters of the current processing layer, only the weight error of the current processing layer before and after quantization is considered independently, without considering the weight error caused by the quantization parameters of other processing layers. Since the quantization parameters are determined layer by layer, the weight error of the layer will be considered in each loss function. In the process of determining the quantization parameters layer by layer, the weight error of the layer will be considered in the process of confirming the quantization parameters of all processing layers, and finally the weight errors of all layers are covered. Through local to local layer-by-layer quantization, the fine-grained quantization parameters of the artificial intelligence model are determined, which can improve the quantization accuracy to a certain extent. In this implementation, since the loss function combines the activation value error and the weight error, it can also better guarantee the prediction accuracy and generalization ability of the artificial intelligence model after quantization.
[0042] It is understandable that there is mutual influence between different processing layers in the artificial intelligence model. The errors of the front processing layers will be continuously accumulated in the subsequent processing layers, affecting the overall processing effect of the artificial intelligence model. Therefore, in another implementation of the present application, the weight loss represents not only the weight error of the current processing layer, but also the accumulated weight errors of all processing layers before the current processing layer. In this implementation, the weight loss can be determined based on the weight error of the current processing layer and the weight errors of each processing layer before the current processing layer. Each time the quantization parameters are determined, the accumulation of the weight errors of the processing layers corresponding to all confirmed or updated weight parameters is considered to further improve the quantization accuracy.
[0043] In this implementation, the weight loss in the loss function used in the process of determining the quantization parameters represents the accumulation of weight errors of the current processing layer and all previous processing layers. Therefore, combined with the gradual transition from local quantization to global quantization, the global weight loss can be reasonably taken into account, ensuring that the quantized artificial intelligence model has good generalization ability.
[0044] Since the activation function value is transitive, the error of the activation function of the current layer can already reflect the error of the activation function of the previous processing layer. Therefore, the activation value part corresponding to the first sub-function in the loss function only needs to consider the activation value error of the current processing layer, and there is no need to consider the activation value error of the processing layer before the current processing layer.
[0045] Figure 2 This is a flowchart of determining quantization parameters layer by layer disclosed in the embodiment of the present application, combined with Figure 2 As shown, the determination of quantization parameters for all processing layers of the artificial intelligence model to be quantized layer by layer may include:
[0046] Step 201: Determine the initial quantization parameter and the quantized weight of the i-th processing layer based on the original weight of the i-th processing layer.
[0047] Specifically, the initial quantization parameter of the i-th processing layer and the corresponding initial quantized weight may be calculated based on the nearest neighbor method through the original weight of the i-th processing layer.
[0048] Step 202: Based on the loss function, the initial quantization parameter of the i-th processing layer and the quantization parameters of the 1st to (i-1) processing layers are trained to obtain target quantization parameters, wherein the target quantization parameters include N quantization parameters from the 1st processing layer to the current processing layer, where N=i.
[0049] In one implementation, the activation value error represents the norm of the output value of the current processing layer subtracted from the training data after it is processed by each processing layer from the first processing layer to the current processing layer before and after quantization of the weight data of the current processing layer; and / or, the weight loss represents the result of weighted summation of the weight errors of the current processing layer and all the previous processing layers.
[0050] In order to facilitate a better understanding of the technical solution of the present application, a specific example is given below.
[0051] In the process of determining the quantization parameters of the artificial intelligence model to be quantified:
[0052] 1. Establish a mathematical description, assuming that the original weight of the i-th layer of the original artificial intelligence model is w i , the weight after quantization is w i (a i ), a i is the quantization parameter of this layer, fi is the inference function of this layer. For an N-layer model, assuming the input is x, then,
[0053] The output of the first layer is y1=f1(x, w1),
[0054] The output of the second layer is y2=f2(y1,w2),
[0055] By analogy, the output of the Nth layer is y N =f N (y N-1 , w N ),
[0056] 2. When i=1,
[0057] First, the initial quantization method can be used to calculate the initial a1 and w1(a1) through w1, that is, the initial quantization parameter of the i-th processing layer and the corresponding initial post-quantization weight are calculated through the original weight of the i-th processing layer based on the nearest neighbor method; the initial quantization method can be the nearest neighbor method, AWQ method, GPTQ method, etc., and this application does not have fixed restrictions.
[0058] Then according to the loss function Training adjustment a1, in implementation, the automatic differentiation function of the deep learning framework can be used to implement the training adjustment of the quantization parameters;
[0059] The above process can be summarized as follows: first, based on the initial weight value, the nearest neighbor method or other appropriate methods are used to obtain the initial value of the quantization parameter, and then the quantization parameter is trained and adjusted on this basis to obtain a better solution;
[0060] In the loss function It is used to measure the error of the activation value, that is, the norm of the difference between the activation values before and after quantization; ||w1-w1(a1)|| is used to measure the error of the weight, that is, the norm of the difference between the weights before and after quantization.
[0061] 3. When i=2,
[0062] First, use the nearest neighbor method to calculate w2 to get the initial a2 and w2(a2).
[0063] Then according to the loss function Training adjusts a1 and a2.
[0064] In the loss function Used to measure the error of the activation value, It is used to measure the error of weights, representing the weighted average sum of the weight errors of each layer from the 1st layer to the i-th layer. Among them, λ represents the weight of the weight error, and the value of λ can be 1 / i, that is, the weight of the weight error of each processing layer is equal. For example, if i=2, then λ is 1 / 2. Therefore, the weighted average sum of the weight errors of each layer is realized to achieve the accumulation of the weight errors of each processing layer with quantized parameters, which can effectively guarantee the generalization ability of the quantized model.
[0065] Alternatively, in other implementations, the corresponding weight λ for weight errors of different processing layers may also be different. For example, the closer to the current processing layer, the larger the corresponding weight λ. For example, when determining the quantization parameters of the third layer, the weight λ of the weight error of the third layer is the largest, the weight λ of the weight error of the second layer is medium, and the weight λ of the weight error of the third layer is the smallest.
[0066] Of course, in other implementations, the value of λ can be 1, that is, λ does not exist, then the weight errors of the current processing layer and all previous processing layers can be directly arithmetic averaged; that is, the weight loss in the loss function is The corresponding loss function formula is:
[0067] 4. Similarly, when i=N,
[0068] First, use the nearest neighbor method to pass w N Calculate the initial a N and w N (a N );
[0069] Then according to the loss function Training adjustment a1,a2,…,a N ;
[0070] In the loss function Used to measure the error of the activation value, Used to measure the error of weights.
[0071] 5. After N layers of training and adjustment, a1, a2, ..., a N It is the final quantization parameter.
[0072] The weight data quantization method described in the embodiment of the present application, the loss function includes activation value error and weight loss, which takes into account the prediction accuracy and generalization ability of the model after quantization, and can greatly ensure the effect of the artificial intelligence model after quantization.
[0073] In actual applications, the input data corresponding to the output value of the weight data of the current processing layer before quantization is the same as the input data corresponding to the output value after the weight data is quantized. That is, in the process of determining the quantization parameters, the artificial intelligence model uses the same input data to determine the excitation values of each processing layer before and after quantization, and then determines the excitation value error of each processing layer before and after quantization.
[0074] In the implementation, the same input data is used to input the model before and after the weight data is quantized, which can effectively ensure the accuracy of the error of the excitation value before and after the weight data is quantized, thereby ensuring the accuracy of the loss function and improving the determination effect of the quantization parameters. Among them, the input data can be a small part of the large batch training data used by the artificial intelligence model in the training stage. Therefore, the quantization of the weight data is post-quantization based on the small batch training data, and there is no need to prepare additional training data for the quantization of the weight data.
[0075] In one implementation, the method of determining the quantized weights of each processing layer based on the quantization parameters of each processing layer may include: determining the quantized weights of each processing layer based on the quantization parameters of each processing layer using a target quantization method, and the target quantization method may be, but is not limited to, any of the following: group quantization, tensor quantization, and channel quantization. The target quantization method may be symmetric quantization or asymmetric quantization. It can be understood that the quantization granularity of group quantization is the smallest, the quantization granularity of channel quantization is the second largest, and the quantization granularity of tensor quantization is the largest. Correspondingly, the computing power required for group quantization is the largest, the computing power required for channel quantization is the second largest, and the computing power required for tensor quantization is the smallest. In implementation, a suitable target quantization method may be selected based on the actual application scenario requirements and hardware performance.
[0076] The quantization method for weight data disclosed in the embodiment of the present application, in the process of determining the quantization parameters of the artificial intelligence model processing layer, adopts a loss function that includes not only the activation value error part corresponding to the accuracy of the artificial intelligence model output result, but also the weight loss part corresponding to the generalization ability of the artificial intelligence model. Therefore, it can take into account the prediction accuracy and generalization ability of the original artificial intelligence model; and the quantization process adopts a layer-by-layer cumulative quantization method, gradually transitioning from local quantization to global quantization, which solves the problem of difficulty in training quantization parameters when the number of artificial intelligence model layers is too large.
[0077] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0078] The method is described in detail in the embodiments disclosed in the above-mentioned application. The method of the application can be implemented by various forms of devices. Therefore, the application also discloses a device, and a specific embodiment is given below for detailed description.
[0079] Figure 3 This is a schematic diagram of the structure of a weight data quantization device disclosed in an embodiment of the present application. Figure 3 As shown, the weight data quantization device 30 may include:
[0080] The parameter determination module 301 is used to determine the quantization parameters of all processing layers of the artificial intelligence model to be quantized layer by layer, wherein the quantization parameters determined for each processing layer include the quantization parameters of the current processing layer and the quantization parameters of all processing layers before the current processing layer, the quantization parameters of the current processing layer are determined for the first time, and the quantization parameters of all processing layers before the current processing layer are updated based on their original quantization parameters.
[0081] The weight determination module 302 is used to determine the quantized weight of each processing layer based on the quantization parameters of each processing layer after the quantization parameters of all processing layers of the artificial intelligence model to be quantized are determined.
[0082] The quantization device for weight data described in the embodiment of the present application adopts a quantization method of accumulating multiple processing layers of the artificial intelligence model layer by layer, gradually transitioning from local quantization to global quantization, which can solve the problem of difficulty in training quantization parameters when the artificial intelligence model has a large number of processing layers, and at the same time avoids the problem of loss of prediction accuracy that may be caused by independent quantization of each processing layer in the traditional solution.
[0083] In one implementation, the loss function used to determine the quantization parameter includes a first sub-function and a second sub-function, wherein the first sub-function represents the activation value error of the current processing layer, and the second sub-function represents the weight loss of the current quantization parameter determination process.
[0084] In one implementation, the weight loss represents the accumulation of weight errors of the current processing layer and all previous processing layers, and is determined based on the weight error of the current processing layer and the weight errors of each processing layer before the current processing layer.
[0085] In one implementation, the parameter determination module may include: an initial processing module, used to determine the initial quantization parameters and quantized weights of the i-th processing layer based on the original weights of the i-th processing layer; an initial processing module, used to train the initial quantization parameters of the i-th processing layer and the quantization parameters of the 1st to (i-1) processing layers based on the loss function to obtain target quantization parameters, wherein the target quantization parameters include i quantization parameters from the 1st processing layer to the current processing layer.
[0086] In one implementation, the initial processing module may be specifically used to: obtain the initial quantization parameter of the i-th processing layer and the corresponding initial quantized weight by calculating the original weight of the i-th processing layer based on the nearest neighbor method.
[0087] In one implementation, the activation value error represents the norm of the output value of the current processing layer subtracted from the training data after it is processed by each processing layer from the first processing layer to the current processing layer before and after quantization of the weight data of the current processing layer; and / or, the weight loss represents the result of weighted summation of the weight errors of the current processing layer and all the previous processing layers.
[0088] In one implementation, the input data corresponding to the output value of the weight data of the current processing layer before quantization is the same as the input data corresponding to the output value after quantization of the weight data.
[0089] In one implementation, the weight determination module may be specifically configured to: determine the quantized weight of each processing layer using a group quantization method based on the quantization parameters of each processing layer.
[0090] The specific implementation of the above-mentioned weight data quantization device and the various modules it contains, as well as other possible implementations, can be found in the corresponding parts of the method embodiment, and will not be repeated here.
[0091] Based on the above implementation content, the quantization device of weight data has the following advantages:
[0092] 1. Since the loss function including activation value error and weight loss is adopted, the prediction accuracy and generalization ability of the original artificial intelligence model can be taken into account, so the quantization accuracy is very high and the application scenarios are rich;
[0093] 2. Compared with the traditional model quantization scheme, the implementation scheme of the weight data quantization device does not involve complex mathematical derivation and solution, so the computing power requirement is low and the quantization efficiency is high;
[0094] 3. The quantization process of the quantization device of the weight data can be completed based on the model training process, and the model quantization parameters can be determined based on small batch training data.
[0095] Any one of the weight data quantization devices described in the above embodiments includes a processor and a memory. The parameter determination module, weight determination module, initial processing module, initial processing module, etc. in the above embodiments are all stored in the memory as program modules, and the processor executes the above program modules stored in the memory to realize corresponding functions.
[0096] The processor includes a kernel, which retrieves the corresponding program module from the memory. One or more kernels can be set, and the processing of the access data can be realized by adjusting the kernel parameters.
[0097] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0098] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the above-mentioned weight data quantization method.
[0099] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer, and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the weight data quantization method described above.
[0100] Furthermore, an embodiment of the present application provides an electronic device on which an artificial intelligence model is deployed, and the weights of each processing layer of the artificial intelligence model are determined based on any one of the weight data quantization methods in the aforementioned embodiments. In combination with the contents of the aforementioned embodiments, the artificial intelligence model deployed on the electronic device uses the quantization method introduced in the contents of the aforementioned embodiments to quantize the weight data. The corresponding quantization method, while ensuring the prediction accuracy and generalization ability of the quantized model, makes the quantized weight data more compressed, thereby reducing the hardware capability requirements of the electronic device for the operation of the artificial intelligence model.
[0101] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0102] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0103] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0104] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for quantifying weight data, include: Determine quantization parameters for all processing layers of the artificial intelligence model to be quantized layer by layer, wherein the quantization parameters determined for each processing layer include the quantization parameter of the current processing layer and the quantization parameters of all processing layers before the current processing layer, the quantization parameter of the current processing layer is determined for the first time, and the quantization parameters of all processing layers before the current processing layer are updated based on their original quantization parameters; After the quantization parameters of all processing layers of the artificial intelligence model to be quantized are determined, the quantized weights of each processing layer are determined based on the quantization parameters of each processing layer.
2. The method for quantizing weight data according to claim 1, in, The loss function used to determine the quantization parameter includes a first sub-function and a second sub-function, wherein the first sub-function represents the activation value error of the current processing layer, and the second sub-function represents the weight loss of the current quantization parameter determination process.
3. The method for quantizing weight data according to claim 2, in, The weight loss represents the accumulation of weight errors of the current processing layer and all the processing layers before it, and is determined based on the weight error of the current processing layer and the weight errors of each processing layer before the current processing layer.
4. The method for quantizing weight data according to claim 2, wherein the quantization parameters of all processing layers of the artificial intelligence model to be quantized are determined layer by layer, include: Determine an initial quantization parameter and a quantized weight of the i-th processing layer based on the original weight of the i-th processing layer; Based on the loss function, the initial quantization parameter of the i-th processing layer and the quantization parameters of the 1st to (i-1) processing layers are trained to obtain target quantization parameters, which include N quantization parameters from the 1st processing layer to the current processing layer, where N is the same as i.
5. The method for quantizing weight data according to claim 4, wherein the initial quantization parameter and the quantized weight of the i-th processing layer are determined based on the original weight of the i-th processing layer, include: Based on the nearest neighbor method, the initial quantization parameter of the i-th processing layer and the corresponding initial quantized weight are calculated through the original weight of the i-th processing layer.
6. The method for quantizing weight data according to claim 2, in, The activation value error represents the norm of the output value of the current processing layer subtracted from the training data after being processed by each processing layer from the first processing layer to the current processing layer before and after quantization of the weight data of the current processing layer; and / or, the weight loss represents the result of weighted summation of the weight errors of the current processing layer and all the previous processing layers.
7. The method for quantizing weight data according to claim 6, in, The input data corresponding to the output value of the weight data of the current processing layer before quantization is the same as the input data corresponding to the output value after quantization of the weight data.
8. The method for quantizing weight data according to claim 1, wherein the quantized weight of each processing layer is determined based on the quantization parameters of each processing layer. include: Based on the quantization parameters of each processing layer, a target quantization method is used to determine the quantized weight of each processing layer, and the target quantization method is any one of the following: group quantization, tensor quantization, and channel quantization.
9. A device for quantizing weight data, include: A parameter determination module, used to determine the quantization parameters of all processing layers of the artificial intelligence model to be quantized layer by layer, wherein the quantization parameters determined for each processing layer include the quantization parameters of the current processing layer and the quantization parameters of all processing layers before the current processing layer, wherein the quantization parameters of the current processing layer are determined for the first time, and the quantization parameters of all processing layers before the current processing layer are updates based on their original quantization parameters; The weight determination module is used to determine the quantized weight of each processing layer based on the quantization parameters of each processing layer after the quantization parameters of all processing layers of the artificial intelligence model to be quantized are determined.
10. An electronic device having an artificial intelligence model deployed thereon, wherein the weights of each processing layer of the artificial intelligence model are determined based on the quantization method of weight data described in any one of claims 1-8.