Hierarchical non-training large model hybrid quantification method and system

Abstract

The invention discloses a hierarchy-based untrained large model hybrid quantization method and system, and the method comprises the steps: determining a target large model, and carrying out the layer-by-layer quantization processing, and obtaining an original floating point weight function and a quantization floating point weight function corresponding to each layer in the target large model; constructing a square error equation corresponding to each layer according to the original floating point weight function and the quantitative floating point weight function corresponding to each layer, and calculating the sensitivity corresponding to each layer in the target large model according to the square error equation corresponding to each layer; bit width distribution is carried out on each corresponding layer in the target large model according to the sensitivity corresponding to each layer, and an initial quantification model is obtained; performing hybrid quantization processing on the initial quantization model to obtain an optimal quantization model; and obtaining target data, inputting the target data into the optimal quantification model, and outputting a target result. According to the method, quantization errors can be effectively reduced, so that high precision of a large model is kept under the condition of low bit width.

Description

Technical Field

[0001] The present invention relates to the technical field of model quantization processing, and particularly to a hierarchical untrained large model hybrid quantization method, system, terminal, and computer-readable storage medium. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, large language models have developed rapidly, greatly changing the interaction mode of users' daily lives. Large language models can understand human language and reply in the style and tone expected by users, but their capabilities go far beyond that of a chatbot. They can generate various types of text, including news reports, papers, PPTs, code, and novels.

[0003] For large language models (LLMs), such as OPT, LLaMA, etc., they perform excellently in various tasks. However, deploying these large language models on edge devices is challenging because they require extremely high computational requirements and memory occupancy. To improve the efficiency and portability of these large language models, the currently common method is model quantization.

[0004] However, in the prior art, the training time required for quantifying large models is too long, and the quantization accuracy loss of the obtained quantized models is relatively large, which cannot meet the needs of users for using large models.

[0005] Therefore, the prior art still needs to be improved and developed. Summary of the Invention

[0006] The main purpose of the present invention is to provide a hierarchical untrained large model hybrid quantization method, system, terminal, and computer-readable storage medium, aiming to solve the problem that when deploying a large model on the edge for quantization in the prior art, the training time required is long and the accuracy of the obtained quantized model is not high, resulting in a slow parsing process and inaccurate parsing results for the large model deployed on the edge when parsing user output data.

[0007] To achieve the above object, the present invention provides a hierarchical untrained large model hybrid quantization method, and the hierarchical untrained large model hybrid quantization method includes the following steps:

[0008] Determine a target large model, and perform layer-by-layer quantization processing on the target large model to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model;

[0009] Construct a squared error equation for each layer based on the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculate the sensitivity corresponding to each layer in the target large model according to the squared error equation corresponding to each layer;

[0010] Perform bit-width allocation for each layer corresponding to the target large model according to the sensitivity corresponding to each layer, and obtain an initial quantization model;

[0011] Perform hybrid quantization processing on the initial quantization model to obtain an optimal quantization model;

[0012] Obtain target data, input the target data into the optimal quantization model, and output a target result.

[0013] Optionally, in the hierarchical untrained large model hybrid quantization method, the step of determining a target large model and performing layer-by-layer quantization processing on the target large model to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model specifically includes:

[0014] Determine a target large model that needs to be quantized, and obtain the input dimension and weight corresponding to each layer in the target large model;

[0015] Perform layer-by-layer matrix splitting processing on each layer in the target large model according to the input dimension and the weight corresponding to each layer, and obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model.

[0016] Optionally, in the hierarchical untrained large model hybrid quantization method, the step of constructing a squared error equation for each layer based on the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculating the sensitivity corresponding to each layer in the target large model specifically includes:

[0017] Perform mean squared error solving processing according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and obtain a squared error equation corresponding to each layer;

[0018] Calculate the differential result and the Hessian matrix corresponding to the squared error equation corresponding to each layer, and perform optimization solving processing according to the differential result and the Hessian matrix to obtain the Hessian value corresponding to each layer in the target large model;

[0019] Obtain the diagonal value corresponding to the Hessian value, and perform accumulation processing on the diagonal value to obtain the sensitivity corresponding to each layer in the target large model.

[0020] Optionally, in the hierarchical untrained large model hybrid quantization method, the expression of the mean squared error equation is as follows:

[0021]

[0022] where loss is the mean squared error, argmin is the minimization calculation, W is the original floating-point weight value, is the quantized floating-point weight value, F(W) is the original floating-point weight function, is the quantized floating-point weight function, is the least squares method calculation.

[0023] Optionally, in the hierarchical untrained large model hybrid quantization method, the expression of the Hessian value is as follows:

[0024]

[0025] where is the Hessian value corresponding to each layer in the target large model, is the differential result corresponding to the mean squared error equation, is the Hessian matrix corresponding to the mean squared error equation.

[0026] Optionally, in the hierarchical untrained large model hybrid quantization method, the process of performing bit-width allocation on each layer in the target large model according to the sensitivity corresponding to each layer to obtain an initial quantization model specifically includes:

[0027] Arrange the Hessian values corresponding to each layer in the target large model in descending order, and select the first target layer number corresponding to the target large model that is greater than the preset first Hessian value;

[0028] Obtain a preset sensitivity-bit width correspondence table, determine the first bit width corresponding to the first target layer number according to the preset sensitivity-bit width correspondence table, and allocate the first bit width to the first target layer number;

[0029] Determine the other layer numbers in the target large model except the first target layer number, determine the second bit width corresponding to the other layer numbers according to the preset sensitivity-bit width correspondence table, and allocate the second bit width to the other layer numbers;

[0030] Obtain an initial quantization model according to the first bit width corresponding to the first target layer number and the second bit width corresponding to the other layer numbers.

[0031] Optionally, in the hierarchical untrained large model hybrid quantization method, the process of performing hybrid quantization processing on the initial quantization model to obtain an optimal quantization model specifically includes:

[0032] Determine whether the quantization bit weights corresponding to the initial quantization model meet the preset requirements. If so, recalculate the target Hessian value corresponding to each layer of the first target layer in the initial quantization model;

[0033] Select the second target layer in the first target layer where the target Hessian value is less than the preset second Hessian value, and adjust the first bit width corresponding to the second target layer to the second bit width;

[0034] When the bit width of the first target layer in the initial quantization model is adjusted, the optimal quantization model is obtained.

[0035] In addition, to achieve the above object, the present invention also provides a hierarchical untrained large model hybrid quantization system, wherein the hierarchical untrained large model hybrid quantization system includes:

[0036] A layer-by-layer quantization processing module, configured to determine a target large model and perform layer-by-layer quantization processing on the target large model to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model;

[0037] A sensitivity calculation module, configured to construct a squared error equation corresponding to each layer according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculate the sensitivity corresponding to each layer in the target large model according to the squared error equation corresponding to each layer;

[0038] A bit width allocation module, configured to perform bit width allocation on each corresponding layer in the target large model according to the sensitivity corresponding to each layer to obtain an initial quantization model;

[0039] A hybrid quantization processing module, configured to perform hybrid quantization processing on the initial quantization model to obtain an optimal quantization model;

[0040] A target result output module, configured to obtain target data, input the target data into the optimal quantization model, and output a target result.

[0041] In addition, to achieve the above object, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a hierarchical untrained large model hybrid quantization program stored on the memory and executable on the processor. When the hierarchical untrained large model hybrid quantization program is executed by the processor, the steps of the above-mentioned hierarchical untrained large model hybrid quantization method are implemented.

[0042] In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a hierarchical untrained large model hybrid quantization program, and when the hierarchical untrained large model hybrid quantization program is executed by a processor, the steps of the above-mentioned hierarchical untrained large model hybrid quantization method are implemented.

[0043] In the present invention, a target large model is determined, and the target large model is quantized layer by layer to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model; a mean square error equation corresponding to each layer is constructed according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and the sensitivity corresponding to each layer in the target large model is calculated according to the mean square error equation corresponding to each layer; bit width allocation is performed on the corresponding layer in the target large model according to the sensitivity corresponding to each layer to obtain an initial quantization model; the initial quantization model is subjected to hybrid quantization processing to obtain an optimal quantization model; target data is acquired, and the target data is input into the optimal quantization model to output a target result. By integrating attention gradient and second-order Hessian optimization, the present invention is used for quantization of large language models. This integration not only brings a more refined update mechanism, but also significantly improves the accuracy in the quantization process. Through this method, quantization errors can be effectively reduced, so as to maintain high precision of the model in the case of low bit width. At the same time, according to the sensitivity of different layers, the present invention skillfully allocates high and low bit widths, which can effectively optimize the overall model performance and ensure computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flowchart of a preferred embodiment of the hierarchical untrained large model hybrid quantization method of the present invention;

[0045] Figure 2 is a schematic diagram of a quantization example and quantization schemes with different granularities of a preferred embodiment of the hierarchical untrained large model hybrid quantization method of the present invention;

[0046] Figure 3 is a schematic diagram of the decomposition of a fully connected layer to support group quantization in hardware in a preferred embodiment of the hierarchical untrained large model hybrid quantization method of the present invention;

[0047] Figure 4 is a schematic diagram of the overall architecture of a large model in a preferred embodiment of the hierarchical untrained large model hybrid quantization method of the present invention;

[0048] Figure 5 is a comparison schematic diagram of the quantization calculation of an attention module and the traditional quantization scheme GPTQ in a preferred embodiment of the hierarchical untrained large model hybrid quantization method of the present invention;

[0049] Figure 6 It is a schematic diagram of the Hessian value of a preferred embodiment of the hierarchical untrained large model hybrid quantization method of the present invention in different blocks;

[0050] Figure 7 It is a schematic diagram of the search space of a preferred embodiment of the hierarchical untrained large model hybrid quantization method of the present invention;

[0051] Figure 8 It is a schematic diagram of the UI design of a preferred embodiment of the hierarchical untrained large model hybrid quantization method of the present invention;

[0052] Figure 9 It is a structural diagram of a preferred embodiment of the hierarchical untrained large model hybrid quantization system of the present invention;

[0053] Figure 10 It is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed implementation manners

[0054] To make the objectives, technical solutions and advantages of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] In recent years, with the rapid development of artificial intelligence technology, large language models have developed rapidly. For example, ChatGPT has recently become popular globally, greatly changing the interaction mode of users' daily lives. ChatGPT can understand human language and reply in the style and tone expected by users. However, its capabilities are far more than just a chatbot. It can generate various types of texts, including news reports, papers, PPTs, code, novels, etc. However, with the rapid development of large language models in recent years, these models usually have billions or even tens of billions of parameters, occupying a large amount of storage space and computing resources. To improve the efficiency and portability of these large models, a commonly used method is model quantization.

[0056] Large language models (LLMs), such as ChatGPT, OPT, LLaMA, etc., perform excellently in various tasks. However, deploying these models on edge devices is challenging because they require extremely high computational requirements and memory occupancy. Existing model compression solutions, such as pruning and neural architecture search, usually require model retraining, which is very time-consuming and expensive for models with billions of parameters. Recently, the post-training quantization (PTQ) method has achieved overall quantization of the model without retraining, but the challenge lies in how to maintain the accuracy of the model.

[0057] There are two main categories of conventional quantization algorithms. One is quantization during training, which realizes quantization processing during the training process. The other is post-training quantization, which only realizes quantization according to a small amount of data sets without training, and is also called post-quantization technology. Quantization technology for large models generally requires too long training time and a large amount of computational resource requirements. And the post-quantization technology has a large loss of quantization accuracy for large models, making the quantized model unacceptable.

[0058] To solve the above problems, the present invention realizes high-accuracy mixed-precision quantization of large models by group quantization (group quantization is a model quantization technology that divides the weight matrix of the model into groups and then quantizes each group of weights. This method can reduce computational and memory requirements while maintaining the accuracy of the model. It is particularly suitable for deploying large language models on edge devices, enabling efficient model compression and quantization without retraining), designing quantization algorithms, and analyzing weight sensitivity.

[0059] The large model mixed quantization technology refers to a technology for performing mixed-precision quantization on neural network models in deep learning. This technology aims to balance the computational speed and accuracy of the model by using different numerical precisions. Specifically, quantization refers to converting the weights and activation values in the model from floating-point representation to a lower-bitwidth representation, such as 8-bit integers or 4-bit integers. This helps reduce the memory occupancy and computational requirements of the model. And mixed-precision computing uses numerical values of different precisions for computing. In deep learning, low precision (e.g., 4-bit integers) is usually used for inference. In the present invention, large models generally refer to deep neural networks with a large number of parameters, such as open-source neural networks like LLaMA.

[0060] The present invention proposes a hybrid quantization technique. By comprehensively considering the advantages and disadvantages of quantization during training and post-training quantization, and combining attention gradients and second-order Hessian optimization, it achieves the effect of 8-bit, 4-bit, and 2-bit hybrid quantization (different layers use different bit widths). The present invention uses the gradient information of the attention output and the Hessian trace to perform hierarchical sensitivity analysis, and then assigns an appropriate bit width to each layer, thereby optimizing the overall model performance and ensuring computational efficiency. The APTQ (Attention-aware Post-Training Mixed-Precision Quantization) technology in the present invention significantly reduces the computational resource requirements while ensuring the model accuracy, and is applicable to large models that are efficiently deployed on edge devices.

[0061] The method for hierarchical untrained large model hybrid quantization according to a preferred embodiment of the present invention is as Figure 1 shown, and the method for hierarchical untrained large model hybrid quantization includes the following steps:

[0062] Step S10: Determine the target large model, and perform layer-by-layer quantization processing on the target large model to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model.

[0063] The present invention relates to a method for achieving lower bit widths on edge devices without compromising their accuracy, namely Attention-aware Post-Training Mixed-Precision Quantization (APTQ). By considering the quantization optimization problem, especially within the attention block range, including the non-linear softmax operation, the present invention provides a mixed-precision quantization method. APTQ utilizes the gradients derived from the attention output and adopts a second-order Hessian-based optimization strategy to quantize the weights, thereby significantly reducing the quantization error and ensuring the integrity of the model.

[0064] Specifically, determine the target large model that needs to be quantized, and obtain the input dimension and weight corresponding to each layer in the target large model; perform layer-by-layer matrix splitting processing on each layer in the target large model according to the input dimension and the weight corresponding to each layer to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model.

[0065] As Figure 2 shown, Figure 2In [A], it is an example of a conventional quantization operation, that is, floating-point numbers are quantized to the integer range from 0 to 15 by calculating the zero point value (zeropoint) and the scale value. Mathematically, it is (floating-point number - zero point value) divided by the scale value, and then rounded to an integer. For a neural network, the quantization granularity can be that an entire weight W shares a zero point value and a scale value, or a column of weights shares a zero point value and a scale value. Therefore Figure 2 In [B], there are a total of M zero point values and scale values. What the present invention proposes is group-based quantization, which quantizes with a granularity smaller than a column. Figure 2 In [B], 2 floating-point numbers share a zero point value and a scale value, which can achieve a smaller quantization error.

[0066] Such as Figure 3 As shown, it is a conversion method optimized for graphics. Here, it mainly aims at that some neural network accelerators only support column quantization (column quantization is a matrix splitting and quantization method, mainly used for the optimization of neural network accelerators. It quantizes the matrix by grouping columns, thereby splitting a fully connected layer into two matrix multiplications and additions. In this way, the neural network accelerator can process the fully connected layer more efficiently and achieve a high-precision quantization scheme). Therefore, the present invention can perform matrix splitting and then achieve group quantization. Such as Figure 3 As shown, for an input of m*2k dimensions (i.e., the input dimension in the present invention) and a weight of 2k*n, assuming the group size is k, then the original fully connected layer can be equivalently split into two matrix multiplications and additions. That is, a fully connected layer can be split into two matrix layers. Through such splitting, the AI accelerator can support the fully connected layer in the way of column quantization, and then achieve the high-precision quantization scheme of the present invention.

[0067] Such as Figure 4 As shown, Figure 4 are several important modules in a large model, including an embedding layer, a normalization layer, and an overall main calculation module (where Figure 3 "What are the contents recorded in the 24 Histories of China" is the input of the large model, and "The Fourteen Histories, the general name of the twenty-four official histories written in successive dynasties of China" is the output of the large model). The quantization scheme of the present invention mainly supports the middle GLM layer (because the quantized model is called ChatGLM2-6B, so the GLM layer refers to a basic unit block of this model), and here only the GLM block is taken as an example, which contains an attention layer and a fully connected layer. The present invention is applicable to Figure 4 the fully connected layer marked in pink on the right.

[0068] Step S20: Construct a squared error equation corresponding to each layer based on the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculate the sensitivity corresponding to each layer in the target large model according to the squared error equation corresponding to each layer.

[0069] Specifically, perform mean squared error solution processing based on the original floating-point weight function and the quantized floating-point weight function corresponding to each layer to obtain a squared error equation corresponding to each layer; the expression of the squared error equation is: where loss is the mean squared error, argmin is the minimization calculation, W is the original floating-point weight value, is the quantized floating-point weight value, F(W) is the original floating-point weight function, is the quantized floating-point weight function, is the least squares method calculation. Calculate the differential result and the Hessian matrix corresponding to the squared error equation corresponding to each layer, and perform optimization solution processing according to the differential result and the Hessian matrix to obtain the Hessian value corresponding to each layer in the target large model; the expression of the Hessian value is: where is the Hessian value corresponding to each layer in the target large model, is the differential result corresponding to the squared error equation, is the Hessian matrix corresponding to the squared error equation. Obtain the diagonal value corresponding to the Hessian value, and perform an accumulation process on the diagonal value to obtain the sensitivity corresponding to each layer in the target large model.

[0070] Such as Figure 5As shown, it demonstrates the advanced architecture of APTQ in the present invention and presents its comprehensive quantization strategy. Different from GPTQ in the prior art which mainly deals with the loss of the current layer, APTQ integrates a comprehensive analysis of the attention mechanism, including the Q, K, V, O matrices, matmul, and non-linear activation layers such as softmax. This extensive approach not only focuses on the complexity beyond simple weight matrix multiplication but also significantly reduces quantization errors, providing a robust solution in low-bit quantization scenarios. At the macro level, the present invention adopts a layer-by-layer quantization method to solve the quantization reconstruction problem of the weights of each layer. In the Transformer architecture, there are two main structural levels: the attention layer and the feedforward layer. Specifically, GPTQ treats each weight matrix as a linear layer and ignores the influence of other structures on the output, while the present invention processes all structures of the same layer as a whole, represented by the function F, which represents the attention output Multihead(Q, K, V). The goal of the present invention is to reconstruct the equation and minimize (since the proposed is a loss function for quantization, minimizing the quantization error function can minimize the model quantization loss) the new mean squared error equation as follows:

[0071] where W is the original floating-point weight value (the weight that has not been quantized originally, called floating-point FP weight), is the quantized floating-point weight value (the weight after quantization). For the above optimization objective function, it is optimized and solved by calculating its differential and Hessian matrix. Among them, the solution formula of the Hessian matrix is as follows:

[0072] Meanwhile, the accumulated value of the diagonal values of the Hessian values (i.e., the Hessian values in the present invention) obtained by the optimization solution is regarded as the sensitivity of each layer, and then different bit widths can be allocated to different layers according to this sensitivity (by calculating the sensitivity, the layer with a greater sensitivity is allocated a higher bit width, such as 4 bits, and the layer with a smaller sensitivity is allocated a lower bit width such as 2 bits).

[0073] Step S30: Allocate bit widths to each corresponding layer in the target large model according to the sensitivity corresponding to each layer, and obtain an initial quantization model.

[0074] Specifically, arrange the Hessian values corresponding to each layer in the target large model in descending order, and screen out the first target number of layers in the target large model that are greater than the preset first Hessian value; obtain the preset sensitivity and bitwidth correspondence table, determine the first bitwidth corresponding to the first target number of layers according to the preset sensitivity and bitwidth correspondence table, and allocate the first bitwidth to the first target number of layers; determine the other layers in the target large model except the first target number of layers, determine the second bitwidth corresponding to the other layers according to the preset sensitivity and bitwidth correspondence table, and allocate the second bitwidth to the other layers; obtain the initial quantization model according to the first bitwidth corresponding to the first target number of layers and the second bitwidth corresponding to the other layers.

[0075] As Figure 6 shown, Figure 6 shows the Hessian values for the Q, V, and MLP layers of a 32-module. According to Figure 6 it can be seen that the value of the Hessian is the largest at layer 0, and then gradually decreases as the layer changes. As Figure 6 shown, if 60% (60% is the preferred ratio of the present invention, and the ratio can be arbitrarily selected according to the actual situation) 4 bits (i.e., the first bitwidth preferably used in the present invention) are required, then according to the sorting of the Hessian values, the first 60% of the layers with the largest Hessian values (i.e., the first target number of layers in the present invention, which is to sort the trace of the Hessian (i.e., the cumulative value of the diagonal) from large to small, and the top 60% of the values are allocated 4 bits) can be selected and arranged with 4 bits, and the other 2 bits (i.e., the second bitwidth preferably used in the present invention, the higher the Hessian value, the 4 bits are given, and the lower the 2 bits are given).

[0076] Step S40: Perform hybrid quantization processing on the initial quantization model to obtain the optimal quantization model.

[0077] Specifically, determine whether the quantization bit weights corresponding to the initial quantization model meet the preset requirements. If so, recalculate the target Hessian values corresponding to each layer in the first target number of layers in the initial quantization model; screen out the second target number of layers in the first target number of layers whose target Hessian values are less than the preset second Hessian value, and adjust the first bitwidth corresponding to the second target number of layers to the second bitwidth; when the bitwidth of the first target number of layers in the initial quantization model is adjusted, the optimal quantization model is obtained.

[0078] Figure 7For the search space, in the attention module, the present invention supports two cases of 2-bit or 4-bit weights, while for the MLP layer, the present invention supports three cases of 2-bit, 4-bit or 8-bit. For example, when it is determined that the 4-bit weight quantization of the overall model (i.e., the first target number of layers corresponding to 4 bits has been set) meets the requirements. In this case, further 2-bit and 4-bit mixed quantization can be performed, that is, calculate the Hessian trace of different layers and then change the layers with small Hessian traces (low sensitivity), that is, the second target number of layers in the present invention, to 2 bits (that is, change the layer originally allocated 4 bits to 2 bits, that is, adjust the first bit width to the second bit width), so as to achieve a higher compression ratio while maintaining the accuracy unchanged.

[0079] As Figure 8 shown, it is a UI design for providing inter-layer analysis for large models, and different quantization algorithms and the bit widths of weight and activation data can be selected. The current UI design can configure the size of the group, select the models supported by Hugging Face, and support other quantization methods.

[0080] Step S50: Obtain target data and input the target data into the best quantization model to output a target result.

[0081] The target data in the present invention can refer to many natural language data, such as large models like ChatGPT mentioned before. The function of the present invention is to perform mixed quantization processing on these large models so that the original model accuracy can still be guaranteed when used at the edge. For example, the present invention can be applied to an in-vehicle intelligent system. At this time, the target data can refer to user instructions or natural language output by the user. When the quantized large model is installed on the in-vehicle intelligent system, it can quickly respond according to the user instructions or natural language and output the corresponding answer results, effectively improving the user experience.

[0082] The present invention proposes a hierarchical untrained large model mixed quantization method, named attention-based quantization technology (APTQ).

[0083] The beneficial effects of the present invention are as follows:

[0084] 1. Integrate attention gradient and second-order Hessian optimization: APTQ introduces an innovative method to integrate attention gradient and second-order Hessian optimization for the quantization of large language models. This integration not only brings a more refined update mechanism but also significantly improves the accuracy in the quantization process. Through this method, quantization errors can be effectively reduced, so as to maintain high model accuracy in the case of low bit widths.

[0085] 2. Hessian Trace-based Mixed Precision Quantization Scheme: The present invention proposes a novel quantization scheme based on Hessian trace. According to the sensitivity of different layers, high or low bitwidths are cleverly allocated to optimize the overall model performance and ensure computational efficiency. Specifically, the Hessian trace is used as a metric for quantization sensitivity. Through this sensitivity analysis, different layers can be allocated a mixed precision of 2 bits and 4 bits (i.e., some layers are allocated 4 bits and some layers are allocated 2 bits, determined according to the sensitivity), thereby achieving efficient quantization.

[0086] Compared with the prior art, the present invention has significant differences in the following key points:

[0087] 1. In terms of comprehensively considering the attention mechanism and non-linear operations: Traditional quantization methods such as GPTQ only focus on the weight quantization of a single layer, while APTQ in the present invention can more accurately quantify complex neural network structures by considering the entire attention module (including the softmax operation) and its non-linear effects.

[0088] 2. For the second-order Hessian optimization strategy: APTQ adopts the optimization strategy of the second-order Hessian matrix, making the quantization process more accurate, especially significantly reducing the quantization error when dealing with large models.

[0089] 3. For sensitivity-driven mixed precision quantization: Based on the sensitivity analysis of the Hessian trace, different layers can be allocated appropriate bitwidths, thereby further compressing the model size and computational resource requirements while maintaining the model performance.

[0090] Furthermore, the best effects of the key technical points set in the present invention can be achieved through the following combined schemes:

[0091] 1. Combine Attention-aware gradient and second-order Hessian: Combine the gradient information of the attention output and the second-order Hessian matrix optimization strategy to perform fine quantization processing within the entire attention block.

[0092] 2. Hessian trace-driven inter-layer mixed precision quantization: Use the Hessian trace to sort the sensitivities of different layers and allocate different bit widths (such as 2 bits and 4 bits) to ensure the optimal quantization accuracy and computational efficiency for each layer.

[0093] 3. Efficient group quantization strategy: During the group quantization process, combine the weight sensitivities of different layers to perform appropriate group partitioning and zero-point value calculation to further reduce the quantization error.

[0094] In summary, the APTQ technology can perform high-precision 2-bit and 4-bit hybrid large model quantization schemes, enabling the quantized large model to perform excellently in zero-shot tasks and being superior to other quantization methods. At the same time, the quantization technology designed in the present invention is robust and effective in diverse language processing scenarios and is suitable for efficient deployment on edge devices.

[0095] During the implementation of the present invention, there are several possible directions for design changes or deformation schemes. First, from the perspective of overall alternative schemes, different quantization strategies can be considered, such as adopting different post-training quantization methods. These alternative schemes can provide different performance and efficiency trade-offs in certain cases. Second, alternative schemes may also exist for core technical features such as the calculation method of the Hessian trace. For example, other sensitivity measurement methods can be adopted to achieve similar effects. These alternative schemes may provide different computational complexity and accuracy trade-offs, thus providing more choices for practical applications.

[0096] In addition, in order to further expand the application scope of the APTQ method and optimize its performance, the APTQ method can be applied not only to Transformer models but also extended to other types of neural network architectures. This extended application will enable more types of models to benefit from the efficient quantization technology of APTQ.

[0097] Second, based on the existing foundation, the computational efficiency of APTQ can be further optimized. For example, more efficient Hessian calculation methods can be introduced to reduce the consumption of computing resources, thereby further improving the efficiency of the quantization process. These optimization measures will help to achieve efficient quantization on larger-scale models.

[0098] Furthermore, as Figure 9 shown, based on the above-mentioned hierarchical untrained large model hybrid quantization method, the present invention also correspondingly provides a hierarchical untrained large model hybrid quantization system. Among them, the hierarchical untrained large model hybrid quantization system includes:

[0099] A layer-by-layer quantization processing module 51, configured to determine a target large model and perform layer-by-layer quantization processing on the target large model to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer of the target large model;

[0100] A sensitivity calculation module 52, configured to construct a squared error equation corresponding to each layer according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculate the sensitivity corresponding to each layer of the target large model according to the squared error equation corresponding to each layer;

[0101] The bit-width allocation module 53 is configured to perform bit-width allocation on each layer in the target large model according to the sensitivity corresponding to each layer, so as to obtain an initial quantization model;

[0102] The hybrid quantization processing module 54 is configured to perform hybrid quantization processing on the initial quantization model to obtain an optimal quantization model;

[0103] The target result output module 55 is configured to obtain target data, input the target data into the optimal quantization model, and output a target result.

[0104] Further, as Figure 10 shown, based on the above-mentioned hierarchical untrained large model hybrid quantization method and system, the present invention also correspondingly provides a terminal, which includes a processor 10, a memory 20, and a display 30. Figure 10 Only some components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0105] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as the hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk equipped on the terminal, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 20 may also include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as program codes installed on the terminal, etc. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, a hierarchical untrained large model hybrid quantization program 40 is stored on the memory 20, and the hierarchical untrained large model hybrid quantization program 40 can be executed by the processor 10, so as to implement the hierarchical untrained large model hybrid quantization method in the present application.

[0106] In some embodiments, the processor 10 may be a Central Processing Unit (CPU), a microprocessor, or other data processing chips, and is used to run the program codes stored in the memory 20 or process data, such as executing the hierarchical untrained large model hybrid quantization method, etc.

[0107] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0108] In one embodiment, when the processor 10 executes the hierarchical untrained large model hybrid quantization program 40 in the memory 20, the following steps are implemented:

[0109] Determine a target large model, and perform layer-by-layer quantization processing on the target large model to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model;

[0110] Construct a squared error equation corresponding to each layer according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculate the sensitivity corresponding to each layer in the target large model according to the squared error equation corresponding to each layer;

[0111] Perform bit-width allocation on each corresponding layer in the target large model according to the sensitivity corresponding to each layer to obtain an initial quantization model;

[0112] Perform hybrid quantization processing on the initial quantization model to obtain an optimal quantization model;

[0113] Obtain target data, and input the target data into the optimal quantization model to output a target result.

[0114] Among them, the determining the target large model, and performing layer-by-layer quantization processing on the target large model to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model specifically includes:

[0115] Determine a target large model that needs to be quantized, and obtain the input dimension and weight corresponding to each layer in the target large model;

[0116] Perform layer-by-layer matrix splitting processing on each layer in the target large model according to the input dimension and the weight corresponding to each layer to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model.

[0117] Among them, the constructing a squared error equation corresponding to each layer according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculating the sensitivity corresponding to each layer in the target large model according to the squared error equation corresponding to each layer specifically includes:

[0118] Solve the mean square error according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and obtain the square error equation corresponding to each layer;

[0119] Calculate the differential result and the Hessian matrix corresponding to the square error equation of each layer, and perform an optimization solution process according to the differential result and the Hessian matrix to obtain the Hessian value corresponding to each layer in the target large model;

[0120] Obtain the diagonal value corresponding to the Hessian value, and perform an accumulation process on the diagonal value to obtain the sensitivity corresponding to each layer in the target large model.

[0121] Among them, the expression of the square error equation is:

[0122]

[0123] Among them, loss is the mean square error, argmin is the minimization calculation, W is the original floating-point weight value, is the quantized floating-point weight value, F(W) is the original floating-point weight function, is the quantized floating-point weight function, is the least squares method calculation.

[0124] Among them, the expression of the Hessian value is:

[0125]

[0126] Among them, is the Hessian value corresponding to each layer in the target large model, is the differential result corresponding to the square error equation, is the Hessian matrix corresponding to the square error equation.

[0127] Among them, performing bit width allocation on each layer corresponding to the target large model according to the sensitivity corresponding to each layer to obtain an initial quantization model specifically includes:

[0128] Arrange the Hessian values corresponding to each layer in the target large model in descending order, and screen out the first target layer numbers corresponding to the target large model that are greater than the preset first Hessian value;

[0129] Obtain the preset sensitivity-bit width correspondence table, determine the first bit width corresponding to the first target layer number according to the preset sensitivity-bit width correspondence table, and allocate the first bit width to the first target layer number;

[0130] Determine the other layers in the target large model except the first target layer, determine the second bitwidth corresponding to the other layers according to the preset sensitivity-bitwidth correspondence table, and allocate the second bitwidth to the other layers;

[0131] Obtain an initial quantization model based on the first bitwidth corresponding to the first target layer and the second bitwidth corresponding to the other layers.

[0132] Among them, the step of performing hybrid quantization processing on the initial quantization model to obtain an optimal quantization model specifically includes:

[0133] Judge whether the quantization bit weights corresponding to the initial quantization model meet the preset requirements. If so, recalculate the target Hessian value corresponding to each layer in the first target layer of the initial quantization model;

[0134] Select the second target layer in the first target layer whose target Hessian value is less than the preset second Hessian value, and adjust the first bitwidth corresponding to the second target layer to the second bitwidth;

[0135] When the bitwidth of the first target layer in the initial quantization model is adjusted, an optimal quantization model is obtained.

[0136] The present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a hierarchical untrained large model hybrid quantization program. When the hierarchical untrained large model hybrid quantization program is executed by a processor, the steps of the above-mentioned hierarchical untrained large model hybrid quantization method are implemented.

[0137] In summary, the present invention provides a hierarchical untrained large model hybrid quantization method and system. The method includes: determining a target large model, and performing layer-by-layer quantization processing on the target large model to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model; constructing a squared error equation corresponding to each layer according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculating the sensitivity corresponding to each layer in the target large model according to the squared error equation corresponding to each layer; performing bit-width allocation on the corresponding layer in the target large model according to the sensitivity corresponding to each layer to obtain an initial quantization model; performing hybrid quantization processing on the initial quantization model to obtain an optimal quantization model; obtaining target data, and inputting the target data into the optimal quantization model to output a target result. The present invention integrates the attention gradient and second-order Hessian optimization for quantization of large language models. This integration not only brings a more refined update mechanism, but also significantly improves the accuracy in the quantization process. By this method, the quantization error can be effectively reduced, so as to maintain the high accuracy of the model at a low bit width. At the same time, the present invention cleverly allocates high and low bit widths according to the sensitivity of different layers, which can effectively optimize the overall model performance and ensure the calculation efficiency.

[0138] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal. Without further limitation, the element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or terminal including the element.

[0139] Of course, those of ordinary skill in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium readable by a computer. When the program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0140] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description. All such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A hierarchical, non-trained, large-model hybrid quantization method, characterized in that: The hierarchical, non-trained large-model hybrid quantization method includes: Determine a target large model, and perform layer-by-layer quantization processing on the target large model to obtain an original floating-point weight function and a quantized floating-point weight function corresponding to each layer in the target large model; Constructing a square error equation corresponding to each layer according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculating the sensitivity corresponding to each layer in the target large model according to the square error equation corresponding to each layer; According to the sensitivity corresponding to each layer, the bit width of each layer in the target large model is allocated to obtain an initial quantization model; Performing hybrid quantization processing on the initial quantization model to obtain an optimal quantization model; Obtain target data, input the target data into the optimal quantization model, and output the target result.

2. The hierarchical, non-training, large-model hybrid quantization method according to claim 1, characterized in that: The step of determining a target large model and performing layer-by-layer quantization processing on the target large model to obtain an original floating-point weight function and a quantized floating-point weight function corresponding to each layer in the target large model specifically includes: Determine the target large model that needs to be quantized, and obtain the input dimension and weight corresponding to each layer in the target large model; Each layer in the target large model is subjected to matrix splitting processing layer by layer according to the input dimension and the weight corresponding to each layer, so as to obtain the original floating-point weight function and the quantized floating-point weight function corresponding to each layer in the target large model.

3. The hierarchical, non-trained, large-model hybrid quantization method according to claim 1, characterized in that: The constructing a square error equation corresponding to each layer according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculating the sensitivity corresponding to each layer in the target large model according to the square error equation corresponding to each layer, specifically includes: Perform mean square error solving processing according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer to obtain a square error equation corresponding to each layer; Calculate the differential result and the Hessian matrix corresponding to the square error equation corresponding to each layer, and perform optimization and solution processing according to the differential result and the Hessian matrix to obtain the Hessian value corresponding to each layer in the target large model; The diagonal value corresponding to the Hessian value is obtained, and the diagonal value is accumulated to obtain the sensitivity corresponding to each layer in the target large model.

4. The hierarchical, non-training, large-model hybrid quantization method according to claim 3, characterized in that: The expression of the square error equation is: Among them, loss is the mean square error, argmin is the minimization calculation, W is the original floating point weight value, is the quantized floating-point weight value, F(W) is the original floating-point weight function, is a quantized floating-point weight function, Calculated by the least squares method.

5. The hierarchical, non-training, large-model hybrid quantization method according to claim 3, characterized in that: The expression of the Hessian value is: in, is the Hessian value corresponding to each layer in the target large model, is the differential result corresponding to the square error equation, is the Hessian matrix corresponding to the square error equation.

6. The hierarchical, non-training, large-model hybrid quantization method according to claim 3, characterized in that: The bit width of each layer in the target large model is allocated according to the sensitivity corresponding to each layer to obtain an initial quantization model, specifically including: Arrange the Hessen values ​​corresponding to each layer in the target large model in descending order, and filter out the first target layer number in the target large model that is greater than the preset first Hessen value; Obtaining a preset sensitivity and bit width correspondence table, determining a first bit width corresponding to the first target number of layers according to the preset sensitivity and bit width correspondence table, and allocating the first bit width to the first target number of layers; Determine other numbers of layers in the target large model except the first target number of layers, determine a second bit width corresponding to the other numbers of layers according to the preset sensitivity and bit width correspondence table, and allocate the second bit width to the other numbers of layers; An initial quantization model is obtained according to a first bit width corresponding to the first target number of layers and a second bit width corresponding to the other number of layers.

7. The hierarchical, non-training, large-model hybrid quantization method according to claim 6, characterized in that: The performing hybrid quantization processing on the initial quantization model to obtain the optimal quantization model specifically includes: Determine whether the quantization bit weight corresponding to the initial quantization model meets the preset requirements, and if so, recalculate the target Hessian value corresponding to each layer of the first target number of layers in the initial quantization model; Filter out a second target number of layers whose target Hessen value is less than a preset second Hessen value from the first target number of layers, and adjust the first bit width corresponding to the second target number of layers to the second bit width; When the bit width of the first target number of layers in the initial quantization model is adjusted, an optimal quantization model is obtained.

8. A hierarchical, non-trained, large-model hybrid quantization system, characterized in that: The hierarchical, non-trained, large-model hybrid quantization system includes: A layer-by-layer quantization processing module is used to determine a target large model and perform layer-by-layer quantization processing on the target large model to obtain an original floating-point weight function and a quantized floating-point weight function corresponding to each layer in the target large model; A sensitivity calculation module, used to construct a square error equation corresponding to each layer according to the original floating-point weight function and the quantized floating-point weight function corresponding to each layer, and calculate the sensitivity corresponding to each layer in the target large model according to the square error equation corresponding to each layer; A bit width allocation module, used for allocating bit width to each layer corresponding to the target large model according to the sensitivity corresponding to each layer, so as to obtain an initial quantization model; A hybrid quantization processing module, used for performing hybrid quantization processing on the initial quantization model to obtain an optimal quantization model; The target result output module is used to obtain target data, input the target data into the optimal quantization model, and output the target result.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a hierarchical-based, untrained large model hybrid quantization program stored in the memory and executable on the processor. When the hierarchical-based, untrained large model hybrid quantization program is executed by the processor, the steps of the hierarchical-based, untrained large model hybrid quantization method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a hierarchical-based, untrained large model hybrid quantization program, which, when executed by a processor, implements the steps of the hierarchical-based, untrained large model hybrid quantization method as described in any one of claims 1 to 7.

Images