Large model quantification method based on non-uniform grouping Adamama transformation and activation distribution self-adaption

Abstract

The invention belongs to the technical field of large model quantification, and discloses a large model quantification method based on non-uniform grouping Adamage transform and activation distribution self-adaption, which comprises the following steps: firstly, eliminating abnormal values in an activation matrix through non-uniform grouping Adamage transform, and solving the problem of abnormal distribution caused by uniform grouping; secondly, constructing an equivalent transformation error elimination algorithm, and eliminating a weight quantization error by adopting a GPTQ algorithm; and finally, distributing an optimal quantization data type (Int4 or Nint4) for each activation matrix through an activation distribution adaptive algorithm so as to minimize a quantization error. According to the method, the deployment efficiency of online rotation is improved through non-uniform grouping Adamage transformation, meanwhile, the quantization error is remarkably reduced in combination with self-adaptive data type selection, and the model quantization precision and the hardware deployment efficiency are improved. The method is suitable for compression and optimization of a large language model, and has a wide application prospect.

Description

Technical Field

[0001] This invention belongs to the field of large model quantization technology, and in particular relates to a large model quantization method based on non-uniform grouping Adama transform and activation distribution adaptation. Background Technology

[0002] With the exponential growth in the size of large language models (LLMs), breakthroughs have been achieved in various language-related tasks. However, these advances are accompanied by significant challenges, particularly the dramatic increase in computational cost, memory footprint, and energy consumption. To reduce the deployment cost of these models, researchers have proposed various model compression techniques to address these challenges.

[0003] Among numerous compression techniques, quantization has become a mainstream method due to its significant effects in improving inference performance and reducing memory footprint. Given the extremely high training cost of LLM, Post-Training Quantization (PTQ) has become the de facto standard for quantizing LLM.

[0004] Quantization of weights is extremely effective because generation tasks are often memory-bound and have low arithmetic strength. However, when aggressive weight quantization schemes are used, the computational load is significantly reduced, allowing the task to gradually shift towards the computational bottleneck region. This necessitates simultaneous activation quantization to achieve further performance improvements.

[0005] Existing research indicates that certain input channels in the LLM activation tensor continuously generate values ​​with anomalous amplitudes (i.e., outliers / outliers), leading to significant accuracy loss in traditional quantization methods.

[0006] Existing methods for addressing outlier issues in activation quantization mainly include: equivalent transformation and nonlinear data types.

[0007] SmoothQuant, QuaRot, SpinQuant, AffineQuant, and FlatQuant employ equivalent transformation methods to eliminate outliers. SmoothQuant uses channel-level scaling, calculating scaling coefficients based on validation data and weights, and then absorbing these coefficients through the weights. QuaRot uses a randomized Hadamard matrix to eliminate outliers; since some Hadamard matrices cannot be absorbed into the weights, it uses an online Hadamard transformation and the GPTQ algorithm to eliminate weight quantization and equivalent transformation errors. SpinQuant, building on QuaRot, uses a learning method to obtain the optimal transformation matrix. AffineQuant and FlatQuant, among others, obtain approximate equivalent transformations through learning. LightRot uses a grouped Hadamard transformation instead of the traditional Hadamard transformation to reduce the deployment cost of online rotation.

[0008] QLORA constructs a NormalFloat(NF4) that best fits the normal distribution based on the quantiles of the normal distribution. M-ANT constructs a data representation method to fit different data distributions.

[0009] However, existing large-scale modeling techniques still have the following shortcomings:

[0010] The single NF4 is difficult to handle outliers well, thus introducing large quantization errors; the complex construction method of M-ANT makes it difficult to apply this dynamic data type to activation, and it is also difficult to handle outliers using only this data type.

[0011] The SmoothQuant method collects scaling coefficients by calibrating a dataset, which makes it difficult to handle occasional outliers. Therefore, when performing 4-bit quantization, it introduces a large quantization error.

[0012] QuaRot and SpinQuant introduce online Hadamard matrices, but the Fast Hadamard Transform algorithm requires the matrix dimension to be a power of 2. The hidden layer dimensions of some mainstream large-scale models do not meet this constraint, leading to increased deployment costs. SpinQuant finds the most suitable rotation matrix through end-to-end learning, making the activations more closely fit a uniform distribution, but this introduces significant training costs. LightRot uses a uniformly grouped Hadamard Transform, which can cause some groups to exhibit abnormal distributions. Therefore, it introduces rearrangement and asymmetric quantization, further increasing deployment overhead.

[0013] Although existing equivalent transformation methods can effectively eliminate outliers, the inherent representational errors in quantization still affect the quantization accuracy of large models. Quantization is essentially a clustering process of data distribution; using different data types for quantization will result in different quantization errors. According to the central limit theorem, data after the Adamas transform tends to follow a normal distribution. Experiments have shown that activation often tends to be uniform / normally distributed, but this characteristic has not been fully and efficiently utilized by existing solutions.

[0014] Furthermore, online rotation requires the use of the fast Hadamard transform; otherwise, deployment costs would increase significantly, thus limiting model inference accuracy. The Hadamard transform has a potential problem: when the vector rotated by the Hadamard transform contains only one maximum, the rotated vector exhibits a bi-directional distribution, with the data concentrated between ±0.5 and ±1 after normalization. When using the full Hadamard transform, the situation of having only one maximum across the entire channel dimension is almost nonexistent; however, when using the uniformly grouped Hadamard transform, such as rotating 128 elements using a 128-dimensional Hadamard matrix, the probability of having only a single maximum among the 128 elements is relatively high. Therefore, using the uniformly grouped Hadamard transform may lead to a certain decrease in accuracy. Summary of the Invention

[0015] The purpose of this invention is to provide a large model quantization method based on non-uniform grouping Adama transform and activation distribution adaptation to solve the above-mentioned technical problems.

[0016] To address the aforementioned technical problems, the present invention provides a specific technical solution for a large model quantization method based on non-uniform grouping Hadamard transform and adaptive activation distribution, as follows:

[0017] A large model quantization method based on non-uniform grouping Hadamard transform and activation distribution adaptation includes the following steps:

[0018] Step 1: Construct the overall architecture of the model;

[0019] Step 2: Non-uniform grouping Hadamard transform;

[0020] Step 3: Construct an equivalent transformation error elimination algorithm;

[0021] Step 4: Construct an adaptive activation distribution algorithm.

[0022] Furthermore, step 1 includes the following steps:

[0023] By performing a grouped Hadamard transform on the input activations to eliminate outliers, and based on computational equivalence, the partial grouped Hadamard transform is used to generate a new weight matrix from the original weight matrix. The equivalence calculation formula is shown below:

[0024] XO R = X I WR=X I (WR)

[0025] The outputs of both MHA and FFN blocks rotate the output activation by absorbing the R1 matrix; the inputs of both MHA and FFN recover the activation by absorbing the inverse of the R1 matrix. There are nonlinear calculations after the outputs of some linear layers, so online grouped Adama R3 transform is used to eliminate outliers.

[0026] Furthermore, step 1 includes a simplified quantization process: first, outliers in the original activation matrix are eliminated by grouped Adama transform; then, based on the distribution of the activation matrix after outlier elimination through offline statistics, the mean square error introduced by quantization is calculated, and the optimal quantization data type is selected for the activation matrix according to the magnitude of the error.

[0027] Furthermore, step 2 includes the following steps:

[0028] A uniformly grouped Hadamard matrix is ​​constructed from submatrices arranged diagonally. Each submatrix is ​​a standard Hadamard matrix with dimensions that are powers of 2, and all submatrices have the same dimension. A non-uniformly grouped Hadamard matrix has submatrices with dimensions that are powers of 2, but the dimensions of each submatrix can be different.

[0029] Since R1 must be consistent across different layers, there is only a small search space. Therefore, R1 uses a uniform Hadamard matrix, and the dimension of the standard Hadamard matrix in R1 is the largest power of 2 that can be divided by the dimension of the activated matrix.

[0030] R2 also uses a uniform Adamant matrix, and the dimension of the standard Adamant matrix in R2 must be smaller than the dimension of each head;

[0031] R3 is an online non-uniform Adamas transform, and the R3 remains different for different layers.

[0032] Furthermore, the specific algorithms for searching the dimensions of different layers of the R3 submatrix are described below:

[0033] 1. Input the submatrix dimension selection range sub_dim_range, and input the weight matrix feature dimension dim;

[0034] 2. Determine the initial sub-matrix dimension list sub_dim_list, where each element in the sub-matrix list is the dimension of each sub-matrix;

[0035] (7) The elements of the list are the maximum values ​​in the selection range, Max(sub_dim_range), and the length of the list is dim / Max(sub_dim_range);

[0036] 3. Loop through the list, incrementing the index i from 0 to len(sub_dim_list)-1;

[0037] (8) The current search element is sub_dim_list[i];

[0038] (9) Calculate pseudo-quantization using the current submatrix dimension list sub_dim_list and calculate the representation error;

[0039] (10) Replace old elements with N new elements, where the value of each new element is a selection range.

[0040] The minimum value of sub_dim_range, where N is

[0041] Max(sub_dim_range) / Min(sub_dim_range);

[0042] (11) Calculate pseudo-quantization using the modified sub-matrix dimension list sub_dim_list and calculate the representation error;

[0043] (12) Modify sub_dim_list to a configuration with lower error between the two;

[0044] 4. Output the final submatrix dimension list, sub_dim_list;

[0045] The expression error is calculated as follows:

[0046] 1. Full-precision input activation X, sub-matrix dimension list sub_dim_list;

[0047] 2. Perform non-uniform grouped Hadamard transformation. Use the non-uniform Hadamard matrix constructed by sub_dim_list to perform Hadamard transformation on X, and the result is Xa.

[0048] 3. Perform pseudo-quantification:

[0049] (3) Quantize Xa after the transformation is completed;

[0050] (4) Dequantize the quantized Qa to get FQa;

[0051] 4. Perform a non-uniform grouped Hadamard transformation. Use the non-uniform Hadamard matrix constructed by sub_dim_list to perform an Hadamard transformation on FQa, and the result is FQ.

[0052] 5. Calculate the expression error and the mean square error of the FQ and X matrices.

[0053] Furthermore, step 3 employs the GPTQ algorithm to eliminate equivalent transformation errors and weight quantization errors. The weight quantization data type is selected as a simple INT4 type, and group quantization is used, with each group containing 128 elements. 128 groups of data are randomly selected from the C4 data training set as the calibration dataset for the GPTQ algorithm.

[0054] Furthermore, step 4 includes the following steps:

[0055] After completing matrix absorption, weight quantization, and GPTQ calibration, the new model weights W are obtained. H The calibration data reuses the data used in the GPTQ algorithm, in which a data type is assigned to each activation matrix. The adaptive algorithm for the input activation data type is as follows:

[0056] 7. Input linear layer weights W H Linear layer full-precision input activation X;

[0057] 8. Perform pseudo-quantization on X using the Int4 data type. The pseudo-quantization algorithm is the same as above, and the result is Qint4.

[0058] 9. Calculate the quantization error:

[0059] (1) Multiply the weights by the full-precision activation, XW;

[0060] (2) Multiply the weight by the quantized activation, Qint4W;

[0061] (3) Calculate the root mean square error of XW and Qint4W;

[0062] 10. Perform pseudo-quantization on X using the Nint4 data type. The pseudo-quantization algorithm is the same as described above, and the result is Qnint4.

[0063] 11. Calculate the quantization error:

[0064] (1) Multiply the weights by the full-precision activation, XW;

[0065] (2) Multiply the weight by the quantized activation, Qnint4W;

[0066] (3) Calculate the root mean square error of XW and Qnint4W;

[0067] 12. The optimal data type is the one with the smaller mean square error; output best_dtype.

[0068] The adaptive algorithm for output activation data type is as follows:

[0069] 6. Full-precision output activated: X1, X2;

[0070] 7. Perform pseudo-quantization on X1 and X2 using the Int4 data type. The pseudo-quantization algorithm is the same as above, and the result is Q1int4 and Q2int4.

[0071] 8. Perform pseudo-quantization on X1 and X2 using the Nnt4 data type. The pseudo-quantization algorithm is the same as above, and the result is Q1nint4 and Q2nint4.

[0072] 9. Calculate the quantization error:

[0073] (5) Multiply Q1int4 and Q2int4, and calculate the root mean square error of the product with X1X2;

[0074] (6) Multiply Q1int4 and Q2nint4, and calculate the root mean square error of the product with X1X2;

[0075] (7) Multiply Q1nint4 and Q2int4, and calculate the root mean square error of the product with X1X2;

[0076] (8) Multiply Q1nint4 and Q2nint4, and calculate the root mean square error of the product with X1X2;

[0077] 10. The optimal data type is the one with the smaller mean square error. Output best_dtype1 and best_dtype2.

[0078] Where nint4 is an approximate data type constructed using the NormalFloat4 data type, multiplying each element by 32 and rounding it to the nearest integer, modifying the positive 32 in the element to 31;

[0079] The int4 data type quantization uses traditional symmetric max-min quantization. The specific quantization algorithm for the nint4 data type is as follows:

[0080] 5. Enter activation X;

[0081] 6. Find the largest absolute value in X, Max(Abs(X));

[0082] 7. Map the range of X to -32, 32, X = (X * 32 / Max(Abs(X)));

[0083] 8. Find the closest number for each element in the nint4 data set; this is the quantization result.

[0084] The large model quantization method based on non-uniform grouping Hadamard transform and activation distribution adaptation of the present invention has the following advantages:

[0085] 1. Use non-uniform grouping Hadamard transformation to eliminate outliers and solve the problem of abnormal distribution caused by uniform grouping Hadamard transformation.

[0086] 2. The nint4 data type was proposed to fit the data distribution after non-uniform Hadamard transformation.

[0087] 3. Using the calibration dataset, assign quantization data types offline to each activation matrix.

[0088] The large model quantization method proposed in this invention shows advantages in both hardware deployment efficiency and quantization accuracy.

[0089] The use of non-uniform grouping Hadamard transform improves the deployment efficiency of online rotation; at the same time, it solves the abnormal distribution problem caused by uniform grouping Hadamard transform, thus improving the model quantization accuracy.

[0090] We propose the nint4 data type to fit a normally distributed activation; and the int4 data type to fit a uniformly distributed activation. Adaptive data allocation using the activation distribution significantly reduces quantization errors. Attached Figure Description

[0091] Figure 1 This is a diagram of the quantization algorithm architecture of the present invention;

[0092] Figure 2 This is a flowchart illustrating the quantization process of the present invention;

[0093] Figure 3 A schematic diagram of a uniformly grouped Adamant matrix;

[0094] Figure 4 A schematic diagram of a non-uniformly grouped Adamant matrix; Detailed Implementation

[0095] To better understand the purpose, structure, and function of this invention, the following description, in conjunction with the accompanying drawings, provides a more detailed account of a large model quantization method based on non-uniform grouping Adama transform and adaptive activation distribution.

[0096] The present invention provides a large model quantization method based on non-uniform grouping Hadamard transform and activation distribution adaptation, comprising the following steps:

[0097] Step 1: Construct the overall model architecture:

[0098] Figure 1 The overall architecture of the technical solution of this invention is shown. Outliers in the input activation are eliminated by performing a grouped Hadamard transform on the input activation. Based on computational equivalence, a partial grouped Hadamard transform can generate a new weight matrix from the original weight matrix. The formula for calculating equivalence is shown below:

[0099] X O R = X I WR=X I (WR)

[0100] Specifically, the outputs of both the MHA and FFN blocks rotate the output activation by absorbing the R1 matrix; the inputs of both the MHA and FFN blocks restore the activation by absorbing the inverse of the R1 matrix. Some linear layers exhibit nonlinear computation after output, failing to meet computational equivalence requirements. Therefore, an online grouped Adamas R3 transform is used to eliminate outliers.

[0101] A simplified quantitative process is as follows: Figure 2 As shown, the original activation matrix contains many outliers, making it difficult to quantize. First, outliers in the original activation matrix are eliminated through grouped Hadamard transform. Then, based on the distribution of the activation matrix after outlier elimination through offline statistics, the mean square error introduced by quantization is calculated, and the optimal quantization data type is selected for the activation matrix according to the magnitude of the error.

[0102] Step 2: Non-uniform grouping Adama transform:

[0103] Next, we will introduce uniformly and non-uniformly grouped Hadamard matrices. A uniformly grouped Hadamard matrix is ​​shown below. Figure 3 As shown; non-uniformly grouped Adamant matrices are as follows Figure 4 As shown.

[0104] A uniformly grouped Hadamard matrix is ​​constructed from submatrices arranged diagonally. Each submatrix is ​​a standard Hadamard matrix with dimensions that are powers of 2, and all submatrices have the same dimension.

[0105] In a non-uniformly grouped Hadamard matrix, each submatrix has a dimension that is an integer power of 2, but the dimensions of each submatrix can be different.

[0106] Since R1 must be consistent across different layers, there is only a small search space. Therefore, R1 uses a uniform Hadamard matrix, and the dimension of the standard Hadamard matrix in R1 is the largest power of 2 that can be divided by the dimension of the activated matrix.

[0107] Since the dimension of the R2 submatrix must be less than or equal to 128, resulting in a smaller search space, R2 also employs a uniform Hadamard matrix. The dimension of the standard Hadamard matrix in R2 must be smaller than the dimension of each head due to optimization methods such as GHA; in this method, it is maintained at 128.

[0108] R3 is an online non-uniform Hadamard transform, and the R3 values ​​of different layers can remain different. The specific algorithm for searching the dimensions of the R3 submatrix at different layers is described below:

[0109] 2. Input submatrix dimension selection range sub_dim_range (e.g. [64, 256]), weight matrix input feature dimension dim (4096)

[0110] 3. Determine the initial submatrix dimension list sub_dim_list, where each element in the submatrix list is the dimension of each submatrix;

[0111] (1) The elements of the list are the maximum values ​​in the selection range, Max(sub_dim_range), and the length of the list is dim / Max(sub_dim_range).

[0112] 4. Loop through the list, incrementing the list index i from 0 to len(sub_dim_list)-1.

[0113] (1) The current search element is sub_dim_list[i]

[0114] (2) Calculate pseudo-quantization using the current submatrix dimension list sub_dim_list and calculate the representation error.

[0115] (3) Replace the old elements with N new elements. The value of the new elements is the minimum value of the selection range sub_dim_range, where N is Max(sub_dim_range) / Min(sub_dim_range).

[0116] (4) Calculate pseudo-quantization using the modified sub-matrix dimension list sub_dim_list and calculate the representation error.

[0117] (5) Modify sub_dim_list to a configuration with lower error between the two.

[0118] 5. Output the final sub-matrix dimension list, sub_dim_list.

[0119] The above-mentioned expression error calculation method is as follows:

[0120] 2. Full-precision input activation X, sub-matrix dimension list sub_dim_list

[0121] 3. Perform a non-uniformly grouped Hadamard transform. Use the non-uniform Hadamard matrix constructed from sub_dim_list to perform the Hadamard transform on X, resulting in Xa.

[0122] 4. Perform pseudo-quantization

[0123] (1) Quantize the Xa after the transformation.

[0124] (2) Dequantize the quantized Qa (restore it to full precision based on the quantization parameters), and the result is FQa.

[0125] 5. Perform a non-uniformly grouped Hadamard transform. Use the non-uniform Hadamard matrix constructed from sub_dim_list to perform an Hadamard transform on FQa, resulting in FQ.

[0126] 6. Calculate the expression error, and calculate the mean square error of the FQ and X matrices.

[0127] Step 3: Construct an equivalent transformation error elimination algorithm:

[0128] This method employs the GPTQ algorithm to eliminate equivalent transformation errors and weight quantization errors. Since the GPTQ algorithm can effectively eliminate weight quantization errors, the weight quantization data type is chosen to be a simple INT4 type, using grouped quantization with 128 elements per group.

[0129] 128 sets of data were randomly selected from the C4 training set as the calibration dataset for the GPTQ algorithm.

[0130] Step 4: Construct an adaptive activation distribution algorithm:

[0131] After completing matrix absorption, weight quantization, and GPTQ calibration, the new model weights W are obtained. H The calibration data reuses the data used in the GPTQ algorithm. In this algorithm, a data type is assigned to each activation matrix. The adaptive algorithm for the input activation data type is shown below:

[0132] 9. Input linear layer weights W H Linear layer full-precision input activation X

[0133] 10. Perform pseudo-quantization on X using the Int4 data type. The pseudo-quantization algorithm is the same as described above, and the result is Qint4.

[0134] 11. Calculate quantization error

[0135] (1) Multiply the weights by the full-precision activation, XW

[0136] (2) Multiply the weights by the quantized activation, Qint4W

[0137] (3) Calculate the root mean square error of XW and Qint4W

[0138] 12. Perform pseudo-quantization on X using the Nint4 data type. The pseudo-quantization algorithm is the same as described above, and the result is Qnint4.

[0139] 13. Calculate quantization error

[0140] (1) Multiply the weights by the full-precision activation, XW

[0141] (2) Multiply the weights by the quantized activation, Qnint4W

[0142] (3) Calculate the root mean square error of XW and Qnint4W

[0143] 14. The optimal data type is the one with the smaller mean square error, and the output is best_dtype.

[0144] The adaptive algorithm for output activation data type is as follows:

[0145] 2. Full-precision output activated: X1, X2

[0146] 3. Perform pseudo-quantization on X1 and X2 using the Int4 data type. The pseudo-quantization algorithm is the same as described above, and the result is Q1int4 and Q2int4.

[0147] 4. Perform pseudo-quantization on X1 and X2 using the Nnt4 data type. The pseudo-quantization algorithm is the same as described above, and the result is Q1nint4 and Q2nint4.

[0148] 5. Calculate quantization error

[0149] (1) Multiply Q1int4 and Q2int4, and calculate the root mean square error of the product with X1X2.

[0150] (2) Multiply Q1int4 and Q2nint4, and calculate the root mean square error of the product with X1X2.

[0151] (3) Multiply Q1nint4 and Q2int4, and calculate the root mean square error of the product with X1X2.

[0152] (4) Multiply Q1nint4 and Q2nint4, and calculate the root mean square error of the product with X1X2.

[0153] 6. The optimal data type is the one with the smaller mean square error. Output best_dtype1 and best_dtype2.

[0154] Where nint4 is an approximate data type constructed using the NormalFloat4 data type, the construction process of which is shown below:

[0155] 1. Input NormalFloat4, the values ​​are shown in the table below.

[0156]

[0157]

[0158] 2. Multiply each element by 32 and round it to the nearest integer.

[0159] 3. Change the value of '32' in the element to '31'.

[0160] The int4 data type quantization uses traditional symmetric max-min quantization. The specific quantization algorithm for the nint4 data type is as follows:

[0161] 1. Enter activation X

[0162] 2. To find the largest absolute value in X, use Max(Abs(X)).

[0163] 3. Map the range of X to -32, 32. X = (X * 32 / Max(Abs(X)))

[0164] 4. Find the closest number for each element in the nint4 data set; this is the quantization result.

[0165] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A large model quantization method based on non-uniform grouping Hadamard transform and adaptive activation distribution, characterized in that, Includes the following steps: Step 1: Construct the overall architecture of the model; Step 2: Non-uniform grouping Hadamard transform; Step 3: Construct an equivalent transformation error elimination algorithm; Step 4: Construct an adaptive activation distribution algorithm.

2. The large model quantization method based on non-uniform grouping Hadamard transform and activation distribution adaptation as described in claim 1, characterized in that, Step 1 includes the following steps: By performing a grouped Hadamard transform on the input activations to eliminate outliers, and based on computational equivalence, the partial grouped Hadamard transform is used to generate a new weight matrix from the original weight matrix. The equivalence calculation formula is shown below: X O R=X I WR=X I (WR) The outputs of both MHA and FFN blocks rotate the output activation by absorbing the R1 matrix; the inputs of both MHA and FFN recover the activation by absorbing the inverse of the R1 matrix. There are nonlinear calculations after the outputs of some linear layers, so online grouped Adama R3 transform is used to eliminate outliers.

3. The large model quantization method based on non-uniform grouping Hadamard transform and adaptive activation distribution as described in claim 1, characterized in that, Step 1 includes a simplified quantization process: first, outliers in the original activation matrix are eliminated by grouped Adama transform; then, the mean square error introduced by quantization is calculated based on the distribution of the activation matrix after offline statistical elimination of outliers, and the optimal quantization data type is selected for the activation matrix according to the magnitude of the error.

4. The large model quantization method based on non-uniform grouping Hadamard transform and adaptive activation distribution as described in claim 1, characterized in that, Step 2 includes the following steps: A uniformly grouped Hadamard matrix is ​​constructed from submatrices arranged diagonally. Each submatrix is ​​a standard Hadamard matrix with dimensions that are powers of 2, and all submatrices have the same dimension. A non-uniformly grouped Hadamard matrix has submatrices with dimensions that are powers of 2, but the dimensions of each submatrix can be different. Since R1 must be consistent across different layers, there is only a small search space. Therefore, R1 uses a uniform Hadamard matrix, and the dimension of the standard Hadamard matrix in R1 is the largest power of 2 that can be divided by the dimension of the activated matrix. R2 also uses a uniform Adamant matrix, and the dimension of the standard Adamant matrix in R2 must be smaller than the dimension of each head; R3 is an online non-uniform Adamas transform, and the R3 remains different for different layers.

5. The large model quantization method based on non-uniform grouping Hadamard transform and activation distribution adaptation as described in claim 4, characterized in that, The specific algorithm for searching the dimensions of different layers of R3 submatrices is described below: 1) Input the submatrix dimension selection range sub_dim_range, and input the weight matrix feature dimension dim; 2) Determine the initial sub-matrix dimension list sub_dim_list, where each element in the sub-matrix list is the dimension of each sub-matrix; The elements of the list are the maximum values ​​in the selection range, Max(sub_dim_range), and the length of the list is dim / Max(sub_dim_range). 3) Loop through the list, incrementing the list index i from 0 to len(sub_dim_list)-1; (1) The current element being searched is sub_dim_list[i]; (2) Calculate pseudo-quantization using the current submatrix dimension list sub_dim_list and calculate the representation error; (3) Replace the old elements with N new elements. The value of the new elements is the minimum value of the selection range sub_dim_range, where N is Max(sub_dim_range) / Min(sub_dim_range). (4) Calculate pseudo-quantization using the modified sub-matrix dimension list sub_dim_list and calculate the representation error; (5) Modify sub_dim_list to a configuration with lower error between the two; 4) Output the final list of submatrix dimensions, sub_dim_list; The expression error is calculated as follows: 1) Full-precision input activation X, sub-matrix dimension list sub_dim_list; 2) Perform a non-uniform grouped Hadamard transformation. Use the non-uniform Hadamard matrix constructed by sub_dim_list to perform an Hadamard transformation on X, and the result is Xa; 3) Perform pseudo-quantization: (1) Quantize Xa after the transformation is completed; (2) Dequantize the quantized Qa to get FQa; 4) Perform a non-uniform grouped Hadamard transformation. Use the non-uniform Hadamard matrix constructed by sub_dim_list to perform an Hadamard transformation on FQa, and the result is FQ. 5) Calculate the expression error and the mean square error of the FQ and X matrices.

6. The large model quantization method based on non-uniform grouping Hadamard transform and activation distribution adaptation as described in claim 1, characterized in that, Step 3 uses the GPTQ algorithm to eliminate equivalent transformation error and weight quantization error. The weight quantization data type is selected as the simple INT4 type, and group quantization is used with 128 elements in each group. 128 groups of data are randomly selected from the C4 data training set as the calibration dataset for the GPTQ algorithm.

7. The large model quantization method based on non-uniform grouping Hadamard transform and activation distribution adaptation as described in claim 1, characterized in that, Step 4 includes the following steps: After completing matrix absorption, weight quantization, and GPTQ calibration, the new model weights W are obtained. H The calibration data reuses the data used in the GPTQ algorithm, in which a data type is assigned to each activation matrix. The adaptive algorithm for the input activation data type is as follows: 1) Input linear layer weights W H Linear layer full-precision input activation X; 2) Perform pseudo-quantization on X using the Int4 data type. The pseudo-quantization algorithm is the same as described above, and the result is Qint4. 3) Calculate the quantization error: (1) Multiply the weights by the full-precision activation, XW; (2) Multiply the weight by the quantized activation, Qint4W; (3) Calculate the root mean square error of XW and Qint4W; 4) Perform pseudo-quantization on X using the Nint4 data type. The pseudo-quantization algorithm is the same as described above, and the result is Qnint4. 5) Calculate the quantization error: (1) Multiply the weights by the full-precision activation, XW; (2) Multiply the weight by the quantized activation, Qnint4W; (3) Calculate the root mean square error of XW and Qnint4W; 6) The optimal data type is the one with the smaller mean square error; output best_dtype. The adaptive algorithm for output activation data type is as follows: 1) Full-precision output activated: X1, X2; 2) Use the Int4 data type to perform pseudo-quantization on X1 and X2. The pseudo-quantization algorithm is the same as above, and the result is Q1int4 and Q2int4. 3) Use the Nnt4 data type to perform pseudo-quantization on X1 and X2. The pseudo-quantization algorithm is the same as above, and the result is Q1nint4 and Q2nint4. 4) Calculate the quantization error: (1) Multiply Q1int4 and Q2int4, and calculate the root mean square error of multiplying Q1int4 and X1X2; (2) Multiply Q1int4 and Q2nint4, and calculate the root mean square error of the product with X1X2; (3) Multiply Q1nint4 and Q2int4, and calculate the root mean square error of multiplying with X1X2; (4) Multiply Q1nint4 and Q2nint4, and calculate the root mean square error of the product with X1X2; 5) The optimal data type is the one with the smaller mean square error. Output best_dtype1 and best_dtype2. Where nint4 is an approximate data type constructed using the NormalFloat4 data type, multiplying each element by 32 and rounding it to the nearest integer, modifying the positive 32 in the element to 31; The int4 data type quantization uses traditional symmetric max-min quantization. The specific quantization algorithm for the nint4 data type is as follows: 1) Enter activation X; 2) Get the largest absolute value in X, Max(Abs(X)); 3) Map the range of X to -32, 32, X = (X * 32 / Max(Abs(X))); 4) Find the closest number for each element in the nint4 data, which is the quantization result.

Images