Large-model quantization compensation method, and device and storage medium

Abstract

Provided in the embodiments of the present disclosure are a large model quantization compensation method, and a device and a storage medium. The method comprises: determining a first quantization error matrix on the basis of a first quantized weight matrix of a target weight matrix corresponding to a target network layer of a model; determining a first auxiliary matrix that makes the singular value distribution of a first product matrix consistent with that of a second product matrix, wherein the first product matrix is the product of an inverse matrix of the first auxiliary matrix and an activation matrix of the target network layer, and the second product matrix is the product of the target weight matrix and the activation matrix; performing singular value decomposition on a third product matrix, and truncating the first r channels of an obtained first singular value decomposition matrix to obtain a second singular value decomposition matrix that contains the larger singular values, wherein the third product matrix is the product of the first quantization error matrix and the first auxiliary matrix; and on the basis of the second singular value decomposition matrix and the first auxiliary matrix, determining a compensation matrix for the first quantized weight matrix, wherein the compensation matrix occupies relatively low storage resources.

Description

Large-scale model quantization compensation methods, equipment, and storage media

[0001] This disclosure claims priority to Chinese Patent Application No. 202411455049.6, filed with the China Patent Office on October 17, 2024, entitled “Large Model Quantization Compensation Method, Device, and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, and storage medium for large-scale model quantization compensation. Background Technology

[0003] As large language model technology matures, its applications are becoming increasingly widespread. To reduce deployment costs, large language models often require quantization before deployment. Quantization is a crucial method for achieving lightweight large language models, reducing model size, memory usage, performance, and power consumption.

[0004] While quantization of large language models can reduce model size and optimize performance, it inevitably leads to a loss of accuracy. Therefore, quantization error compensation is necessary to restore accuracy during model inference. Different methods of determining the compensation matrix not only result in varying accuracy but also have different impacts on computational and storage resources. Therefore, a more resource-efficient quantization compensation method is needed. Summary of the Invention

[0005] This disclosure provides a method, device, and storage medium for large model quantization compensation, which can reduce the parameter size of the compensation matrix for quantization error, thereby reducing the consumption of storage and other resources.

[0006] In a first aspect, embodiments of this disclosure provide a large-model quantization compensation method, the method comprising:

[0007] The first quantization error matrix is ​​determined based on the target weight matrix corresponding to the target network layer of the model and the first quantization weight matrix corresponding to the target weight matrix. The target network layer is any one of the multiple network layers contained in the model.

[0008] A first auxiliary matrix is ​​determined, which makes the singular value distributions of the first product matrix and the second product matrix consistent. The first product matrix is ​​the product of the inverse of the first auxiliary matrix and the activation matrix corresponding to the target network layer. The second product matrix is ​​the product of the target weight matrix and the activation matrix.

[0009] Singular value decomposition is performed on the third product matrix to obtain the first singular value decomposition matrix. The first singular value decomposition matrix is ​​then truncated for the first r channels to obtain the second singular value decomposition matrix containing larger singular values, where r is the set rank truncation length. The third product matrix is ​​the product of the first quantization error matrix and the first auxiliary matrix.

[0010] The compensation matrix of the first quantization weight matrix is ​​determined based on the second singular value decomposition matrix and the first auxiliary matrix.

[0011] Secondly, embodiments of this disclosure provide a large model quantization compensation device, the device comprising:

[0012] The quantization module is used to determine the first quantization error matrix based on the target weight matrix corresponding to the target network layer of the model and the first quantization weight matrix corresponding to the target weight matrix, wherein the target network layer is any one of the multiple network layers contained in the model;

[0013] A first determining module is used to determine a first auxiliary matrix, wherein the first auxiliary matrix makes the singular value distributions of the first product matrix and the second product matrix consistent, the first product matrix is ​​the product of the inverse of the first auxiliary matrix and the activation matrix corresponding to the target network layer, and the second product matrix is ​​the product of the target weight matrix and the activation matrix.

[0014] The decomposition module is used to perform singular value decomposition on the third product matrix to obtain the first singular value decomposition matrix, and to truncate the first singular value decomposition matrix by the first r channels to obtain the second singular value decomposition matrix containing larger singular values, where r is a set rank truncation length, and the third product matrix is ​​the product of the first quantization error matrix and the first auxiliary matrix.

[0015] The second determining module is used to determine the compensation matrix of the first quantization weight matrix based on the second singular value decomposition matrix and the first auxiliary matrix.

[0016] Thirdly, embodiments of this disclosure provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the large model quantization compensation method as described in the first aspect.

[0017] Fourthly, embodiments of this disclosure provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the large model quantization compensation method as described in the first aspect.

[0018] Fifthly, embodiments of this disclosure provide a computer program product, the computer program product including a computer program, which, when executed by a processor, can implement the large model quantization compensation method as described in the first aspect.

[0019] The model quantization scheme provided in this embodiment first obtains the target weight matrix and activation matrix corresponding to the target network layer in the model during quantization compensation processing. The target network layer is any one of the multiple network layers included in the model. Then, the target weight matrix corresponding to the target network layer is quantized to obtain a first quantized weight matrix, and a first quantization error matrix is ​​determined based on the first quantized weight matrix and the target weight matrix. A first auxiliary matrix is ​​determined, and its inverse is multiplied by the activation matrix to obtain a first product matrix. The target weight matrix and activation matrix are then multiplied to obtain a second product matrix, where the first auxiliary matrix ensures that the singular value distributions of the first and second product matrices are consistent. Next, the first quantization error matrix and the first auxiliary matrix are multiplied to obtain a third product matrix, and singular value decomposition (SVD) is performed on the third product matrix to obtain a first singular value decomposition (SVD) matrix. Then, the first SVD matrix is ​​truncated for the first r channels to obtain a second SVD matrix containing larger singular values, where r is a set rank truncation length. Finally, the compensation matrix of the first quantization weight matrix is ​​determined based on the second singular value decomposition matrix.

[0020] In the above scheme, singular value decomposition is performed on the product of the first quantization error matrix and the first auxiliary matrix. The largest r singular values ​​are then extracted from the resulting first singular value decomposition matrix to form a second singular value decomposition matrix. The number r of non-zero elements in the second singular value decomposition matrix is ​​much lower than the rank of the first singular value decomposition matrix, exhibiting a low-rank characteristic. Based on this, a compensation matrix for the quantization error of the target weight matrix is ​​constructed, thereby achieving low-rank compensation for the first quantization weight matrix. Since the parameter size of the second singular value decomposition matrix is ​​relatively small, it consumes less storage resources, and the computational load required for restoring model accuracy during the subsequent model inference stage is also relatively small. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a flowchart of a large model quantization compensation method provided in an embodiment of this disclosure;

[0023] Figure 2 is a schematic diagram illustrating the application of a model reasoning stage according to an embodiment of this disclosure;

[0024] Figure 3 is a flowchart of a process for determining a scaling matrix according to an embodiment of this disclosure;

[0025] Figure 4 is a flowchart of another large model quantization compensation method provided in an embodiment of this disclosure;

[0026] Figure 5 is a schematic diagram illustrating another application of the model inference stage provided in an embodiment of this disclosure;

[0027] Figure 6 is a schematic diagram of a large model quantization compensation device provided in an embodiment of this disclosure;

[0028] Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] The following detailed description of some embodiments of this disclosure is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0032] First, the terms or concepts involved in the embodiments of this disclosure will be explained:

[0033] Quantized models: These are models whose weights and / or activation values ​​are converted from floating-point representations to integer representations using quantization techniques. The main purpose of quantized models is to reduce the model's storage space and computational resource requirements while maintaining the model's performance as much as possible.

[0034] Weight quantization: The process of converting the weights in a model from a high-precision floating-point representation to a low-precision integer representation.

[0035] Activation quantization: The process of converting activation values ​​in a neural network from high-precision floating-point representation to low-precision integer representation.

[0036] Post-training quantization: The process of quantizing the trained model after training is complete. This process does not require retraining the model, but rather quantizes the model's weights and / or activation values ​​to reduce the model's storage and computational requirements while maintaining the model's performance as much as possible.

[0037] Quantization loss: Information loss or error caused during the quantization process due to the conversion of high-precision floating-point representation in the model to low-precision integer representation.

[0038] Cholesky decomposition: A matrix decomposition method that can decompose a symmetric positive definite matrix into the product of a lower triangular matrix and its transpose.

[0039] Singular Value Decomposition (SVD): An important matrix decomposition method in linear algebra, used to decompose an arbitrary-size matrix into the product of three matrices. The middle matrix is ​​a diagonal matrix, and the two outer matrices are orthogonal matrices.

[0040] Large language models typically require hundreds of gigabytes of GPU memory and are usually deployed in multi-GPU cluster data centers. To enable the deployment of large language models in resource-constrained environments and improve computational efficiency, researchers have been exploring various compression techniques. Quantization, as an important optimization strategy, aims to reduce computational and storage requirements without significantly sacrificing model performance.

[0041] While quantization of large language models can reduce model size and optimize performance, it inevitably leads to a loss of accuracy. Therefore, quantization error compensation is necessary to restore accuracy during model inference. Different methods of determining the compensation matrix not only affect the accuracy of the compensation results but also have varying impacts on computational and storage resources. Therefore, a more resource-efficient quantization compensation method is needed.

[0042] The model quantization compensation scheme provided in this disclosure is applicable to various deep learning models, including but not limited to large language models, diffusion models, etc. Generally speaking, these models typically contain several network layers (such as convolutional layers, but not limited to this), and each network layer is often connected to an activation layer. In this disclosure, the model parameters corresponding to the network layers are called weight matrices, and the model parameters corresponding to the activation layers are called activation matrices. Taking any target network layer and its connected activation layers as an example, after the input data is processed by the target weight matrix W corresponding to the target network layer, the calculation result is input to the activation layer, and the output result of the activation layer is used as the input data for the next network layer. Therefore, the target weight matrix W corresponding to the target network layer and the activation matrix X corresponding to the activation layer are equivalent to performing continuous matrix multiplication calculations on the input data of the target network layer: input data * (target weight matrix * activation matrix). Based on this, the quantization objective can be expressed as: minimizing the following quantization error matrix: ||WX-W q X|| F Where W and X represent the target weight matrix and activation matrix, respectively, W q This represents the quantized weight matrix (i.e., the first quantized weight matrix below), |||| F Let E represent the F-norm. Here, we can let E... q =WW q , where represents the first quantization error matrix corresponding to the target weight matrix W.

[0043] Research has revealed that quantization error matrices often exhibit low-rank characteristics, meaning they have a small number of high singular values ​​and a large number of low singular values ​​in the long tail. Based on this finding, this disclosure provides a novel quantization compensation method for large models.

[0044] Before detailing the quantization compensation method, let's briefly introduce how to use the compensation matrix obtained through this method during the inference phase of a large model. Taking any target network layer in the model as an example, the result of quantizing its target weight matrix using a defined quantization algorithm is called the first quantized weight matrix. The final compensation matrix determined is the one corresponding to this first quantized weight matrix. Based on this, the execution flow during the model inference phase can be summarized as follows:

[0045] Obtain the input data of the target task corresponding to the model, as well as the first quantization weight matrix and its corresponding compensation matrix corresponding to the target network layer; compensate the first quantization weight matrix with the compensation matrix (e.g., sum them together) to obtain the compensated weight matrix; after the input data is processed by the calculation of the compensated weight matrix, it is further calculated with the activation matrix of the target network layer to obtain the output data of the target network layer; then, based on the output data of the target network layer, through the calculation and processing of multiple subsequent network layers and their corresponding activation layers, the final output data of the target task is obtained.

[0046] The target task can be a task applicable to one or more application scenarios, such as summarization, text-to-image, image-to-image, and text generation tasks suitable for large language models. The input and output data differ for different tasks. For example, in a text-to-image task, the input data may include textual descriptions of the image to be generated. In an image-to-image task, the input data may include textual descriptions of the target image to be generated and the input image. In a text generation task, the input data may be prompts describing the text content to be generated, such as "Please generate a travel guide for a certain place."

[0047] Based on the above reasoning process, it can be seen that during the reasoning stage, only the first quantized weight matrix can be obtained, not the target weight matrix before quantization. The purpose of compensation through the compensation matrix is ​​to recover the target weight matrix with high accuracy. This compensation matrix is ​​actually used to compensate for the aforementioned E... q Therefore, the main goal in the quantitative compensation process is to determine a more accurate approximation. To estimate E q This is used to determine the compensation matrix; that is, the quantization objective can also be expressed as: minimizing Furthermore, if the parameter size of the compensation matrix is ​​large, it means that more storage space, such as GPU memory, needs to be allocated to store the compensation matrix and the first quantization weight matrix. Compared to directly storing the target weight matrix, this offers only a limited reduction in storage space usage. Moreover, a larger compensation matrix also means increased computational load, consuming more computing power from GPUs and other processors. However, the quantization compensation method provided in this disclosure can effectively reduce the parameter size of the compensation matrix, thereby consuming less storage resources and using less computing power.

[0048] The model quantization compensation scheme provided in the embodiments of this disclosure will be described in detail below.

[0049] Figure 1 is a flowchart of a large model quantization compensation method provided in an embodiment of this disclosure. As shown in Figure 1, the method may include the following steps:

[0050] 101. Determine the first quantization error matrix based on the target weight matrix corresponding to the target network layer of the model and the first quantization weight matrix corresponding to the target weight matrix. The target network layer is any one of the multiple network layers contained in the model.

[0051] 102. Determine the first auxiliary matrix, which makes the singular value distributions of the first product matrix and the second product matrix consistent. The first product matrix is ​​the product of the inverse of the first auxiliary matrix and the activation matrix corresponding to the target network layer. The second product matrix is ​​the product of the target weight matrix and the activation matrix.

[0052] 103. Perform singular value decomposition on the third product matrix to obtain the first singular value decomposition matrix. Truncate the first singular value decomposition matrix for the first r channels to obtain the second singular value decomposition matrix containing larger singular values, where r is the set rank truncation length. The third product matrix is ​​the product of the first quantization error matrix and the first auxiliary matrix.

[0053] 104. Determine the compensation matrix of the first quantization weight matrix based on the second singular value decomposition matrix and the first auxiliary matrix.

[0054] The large model quantization compensation method provided in this disclosure can perform quantization processing after the large model has been trained. Specifically, each network layer in the model can be quantized, and after quantization, a corresponding compensation matrix can be constructed for each network layer in the model to compensate for the quantization error caused during the quantization process. The trained large model can be a large language model, etc.

[0055] When quantizing the model, the target weight matrix and activation matrix of the target network layer are first obtained. The target network layer is any one of the multiple network layers in the model. Since the processing for multiple network layers in the model is the same, this embodiment uses only one network layer as the target network layer, and the processing of the target network layer is described using this as an example. The target weight matrix refers to the weight parameters of the target network layer in the model; these weight parameters determine how the target network layer processes the input data. The activation matrix consists of the parameters of the activation layers connected to the output side of the target network layer.

[0056] After obtaining the target weight matrix and activation matrix corresponding to the target network layer, the target weight matrix is ​​quantized to obtain the first quantized weight matrix. In specific implementations, quantization methods such as Recursive Ternary Quantization (RTN), Mixed-Precision Quantization (MPP), and Generalized Post-Training Quantization (GPTQ) can be used to quantize the target weight matrix; this disclosure does not limit the specific methods used.

[0057] The quantization process primarily involves converting a high-precision target weight matrix into a low-precision first quantized weight matrix. This process introduces quantization error. Specifically, after determining the first quantized weight matrix, the first quantization error matrix can be determined based on the target weight matrix and the first quantized weight matrix. This first quantization error matrix can be obtained by subtracting the target weight matrix from the first quantized weight matrix. For example, suppose the target weight matrix for the target network layer is W, and the first quantized weight matrix is ​​W0. q The first quantization error matrix is ​​E q Therefore, the first quantization error matrix can be determined as E. q =WW q Then, the compensation matrix corresponding to the first quantization weight matrix can be determined using this first quantization error matrix.

[0058] As described above, in this embodiment of the disclosure, the low-rank characteristic of the quantization error matrix is ​​used to determine the compensation matrix corresponding to the first quantization weight matrix, as detailed below.

[0059] First, determine an invertible first auxiliary matrix, where the determined first auxiliary matrix can make the singular value distributions of the first product matrix and the second product matrix consistent. The first product matrix is ​​the product of the inverse of the first auxiliary matrix and the activation matrix, and the second product matrix is ​​the product of the target weight matrix and the activation matrix.

[0060] For example, assuming the activation matrix of the target network layer is X, the target weight matrix of the target network layer is W, and the determined first auxiliary matrix is ​​S, then the inverse matrix of the first auxiliary matrix can be determined as Sinverse. -1 The product of the inverse of the first auxiliary matrix and the activation matrix is ​​S. -1 X, and this product is determined as the first product matrix. The product of the target weight matrix and the activation matrix is ​​obtained as WX, and this product is determined as the second product matrix, i.e., the second product matrix is ​​WX. Where S -1The singular value distribution of X is consistent with the singular value distribution of WX.

[0061] The reason for introducing the first auxiliary matrix is ​​that it is necessary to compensate for the quantization result of the target weight matrix W, not for WX. Therefore, the result of WX decomposition cannot be used to compensate for the quantization result of W in subsequent decomposition processes. Thus, WX cannot be decomposed. Instead, the first auxiliary matrix S is introduced, making its inverse matrix Si. -1 The product of X and S -1 The singular value distribution of X is consistent with the singular value distribution of WX, so the first auxiliary matrix can be used to replace the target weight matrix W in the subsequent decomposition process. Furthermore, because S... -1 The singular value distributions of X and WX are consistent. Therefore, selecting the largest subset of singular values ​​will more accurately compensate for the integration error ||WX-W q X|| F That is, better compensation for E q S.

[0062] In one optional implementation of determining the first auxiliary matrix, the activation matrix corresponding to the target network layer can be decomposed into a matrix, such that the first auxiliary matrix ensures that the singular value distributions of the first product matrix and the second product matrix are consistent. Specifically, the process of determining the first auxiliary matrix can be as follows: perform matrix decomposition on the activation matrix corresponding to the target network layer to decompose the activation matrix into the product of the first auxiliary matrix and its transpose.

[0063] In practical applications, Cholesky decomposition can be used to decompose the activation matrix corresponding to the target network layer to obtain the first auxiliary matrix. For example, assuming the activation matrix corresponding to the target network layer is X, Cholesky decomposition can be performed on the activation matrix X to obtain the first auxiliary matrix S and its transpose S. T The product of, i.e., X = SS T .

[0064] Furthermore, when the first auxiliary matrix S satisfies (S -1 X)(S -1 X) T =S -1 XX T (S -1 ) T When = I, it can be guaranteed that the first auxiliary matrix S can make the first product matrix S -1 X and the second product matrix WX have the same singular value distribution. Here, I is the identity matrix.

[0065] Next, singular value decomposition is performed on the product of the first quantization error matrix and the first auxiliary matrix to obtain the first singular value decomposition matrix.

[0066] Specifically, the first quantization error matrix and the first auxiliary matrix are multiplied to obtain the third product matrix, and the third product matrix is ​​decomposed into singular values ​​to obtain the first left unitary matrix, the first diagonal matrix, and the first right unitary matrix (that is, the first singular value decomposition matrix mentioned above may include these three matrices).

[0067] In this embodiment, the third product matrix can be decomposed using methods such as SVD to obtain the product of the first left unitary matrix, the first diagonal matrix, and the first right unitary matrix. The first left unitary matrix and the first right unitary matrix are both orthogonal matrices; the first diagonal matrix is ​​a diagonal matrix whose diagonal elements are non-negative real singular values ​​arranged in descending order, and the off-diagonal elements are 0.

[0068] Continuing with the example above, the first quantization error matrix is ​​E q Let S be the first auxiliary matrix. Multiplying the first quantization error matrix by the first auxiliary matrix yields the third product matrix E. q S. For the third product matrix E q S undergoes singular value decomposition, yielding the product of the first left unitary matrix U, the first diagonal matrix Σ, and the first right unitary matrix V, i.e., E. q S=UΣV T Among them, E q S is an m×n matrix, U=[u1,u2,…u n ],Σ=diag(σ1,σ2…σ n V = [v1, v2, ... v] n ] T , σ1,σ2…σ n It is a singular value.

[0069] The third product matrix E here q S is equivalent to a generalized quantization error matrix (the first quantization error matrix E of the target weight matrix). q The product of the first auxiliary matrix S and the singular value decomposition (SV) results in U, Σ, and V after performing SV decomposition on the quantization error matrix. U describes the row space of the quantization error matrix, Σ describes the distribution of singular values ​​of the quantization error matrix, and V describes the column space of the quantization error matrix.

[0070] In practical applications, the third product matrix is ​​usually a very large matrix. After performing singular value decomposition on the third product matrix, the first left unitary matrix U, the first diagonal matrix Σ, and the first right unitary matrix V are also large matrices. If the compensation matrix is ​​directly determined based on the first left unitary matrix U, the first diagonal matrix Σ, and the first right unitary matrix V, the determined compensation matrix will occupy a large amount of storage space and consume more computation, making it impossible to deploy in resource-constrained environments.

[0071] Therefore, in this embodiment of the present disclosure, after obtaining the first left unitary matrix, the first diagonal matrix and the first right unitary matrix, the original matrix can be approximated by retaining multiple largest singular values ​​in the first left unitary matrix, the first diagonal matrix and the first right unitary matrix, thereby achieving a low-rank approximation of the matrix. This can reduce the dimension and complexity of the matrix while retaining the main information of the original matrix, thereby improving computational efficiency and reducing storage requirements.

[0072] In practice, the first r channels of the first left-unitary matrix, the first diagonal matrix, and the first right-unitary matrix can be truncated to obtain the second left-unitary matrix, the second diagonal matrix, and the second right-unitary matrix. Here, r is the set rank truncation length. In practical applications, the rank truncation length can be set according to actual needs. For example, the rank truncation length can be set to r = 64. A channel refers to a column or row of the matrix. In essence, it involves extracting the first r singular values ​​from the first singular value decomposition matrix.

[0073] Specifically, the second left-unitary matrix can be obtained by extracting the rows corresponding to the first r largest singular values ​​from the first left-unitary matrix. Similarly, the second diagonal matrix can be obtained by extracting the first r diagonal elements from the first diagonal matrix, i.e., by extracting the first r largest singular values ​​from the first diagonal matrix. Finally, the second right-unitary matrix can be obtained by extracting the columns corresponding to the first r singular values ​​from the first right-unitary matrix.

[0074] In other words, the second left unitary matrix, the second diagonal matrix, and the second right unitary matrix are low-rank approximation matrices. The second left unitary matrix retains only the rows corresponding to the first r singular values ​​in the first left unitary matrix, the second diagonal matrix retains the first r diagonal elements in the first diagonal matrix, and the second right unitary matrix retains only the columns corresponding to the first r singular values ​​in the second right unitary matrix.

[0075] For example, the first left-unitary matrix U, the first diagonal matrix Σ, and the first right-unitary matrix V are respectively truncated through the first r channels of the first left-unitary matrix U, the first diagonal matrix Σ, and the first right-unitary matrix V to obtain the second left-unitary matrix U. r The second diagonal matrix Σ r The second right-unitary matrix V r The second left-unitary matrix U can be used. r The second diagonal matrix Σ rSecond right unitary matrix V r The product approximates the third product matrix, that is, the third product matrix E is obtained. q Approximation of S

[0076] Finally, based on the second left unitary matrix, the second diagonal matrix, the second right unitary matrix, and the first auxiliary matrix, the compensation matrix for the first quantization weight matrix is ​​determined: That is, the first quantization weight matrix W q The compensation matrix.

[0077] In addition, in practical applications, in order to improve computational efficiency and reduce storage requirements, the compensation matrix can be split into two smaller compensation matrices for separate storage and retrieval, so that the first quantization weight matrix can be compensated by the two smaller compensation matrices during the inference phase of the model.

[0078] Specifically, the specific implementation of determining the compensation matrix of the first quantization weight matrix based on the second left unitary matrix, the second diagonal matrix, the second right unitary matrix, and the first auxiliary matrix can be as follows: determine the product of the second left unitary matrix and the second diagonal matrix as the first compensation matrix; determine the product of the transpose of the second right unitary matrix and the inverse of the first auxiliary matrix as the second compensation matrix; store the first compensation matrix and the second compensation matrix so that the first quantization weight matrix is ​​compensated by the product of the first compensation matrix and the second compensation matrix during the inference phase of the model.

[0079] Continuing with the example above, the second left-unitary matrix is ​​U. r The second diagonal matrix is ​​Σ r The second right-unitary matrix is ​​V r Therefore, the product of the second left unitary matrix and the second diagonal matrix can be determined to be U. r Σ r And this product is determined as the first compensation matrix L. A That is, the first compensation matrix L A =U r Σ r Next, we determine that the product of the transpose of the second right-unitary matrix and the inverse of the first auxiliary matrix is ​​V. r T S -1 And this product is determined as the second compensation matrix L. B That is, the second compensation matrix L B =V r T S -1 .thereby, In practical applications, when the second singular value decomposition matrix (such as the second left unitary matrix U mentioned above) is determined... r The second diagonal matrix Σ r Second right unitary matrix V rAfter that, the storage space for storing the second singular value decomposition matrix can be determined in the target device. Taking this storage space as the GPU's video memory space as an example, then the second singular value decomposition matrix can be divided into its entirety or into parts (such as the following four parts U) based on the remaining storage capacity of each GPU in the target device. r , Σ r V r T S -1 These parts are stored in the video memory of at least one GPU. Since these parts are small-scale matrices obtained by truncating r singular values, their video memory usage is relatively small. Furthermore, when using L... A and L B When storing data in this way, you can determine the memory space of one or two GPUs. For example, you can store L in the memory space of one GPU. A L B Alternatively, these two parts can be stored in the memory spaces of two different GPUs, which will reduce GPU usage. Then, during the subsequent model inference stage, only the compensation matrices need to be read from one or two GPU memory spaces. The communication overhead during the reading process is minimal, and since only the multiplication of these two compensation matrices is required, the computational load is also reduced, thus improving the resource utilization and computational efficiency of the target device. In practical applications, the target device can be a terminal device or a server. Terminal devices can be smartphones, PCs, intelligent robots, etc. By determining the results using the aforementioned compensation matrices with relatively small parameter sizes, the quantized model can be stored on these terminal devices, facilitating direct use of the model on these devices. Compared to loading the model from a server, this method offers higher computational efficiency.

[0080] In summary, based on the low-rank characteristic of the quantization error matrix of the model weights, this embodiment introduces a first auxiliary matrix that meets certain conditions. By performing singular value decomposition on the quantization error matrix and extracting a singular value decomposition matrix composed of a small number of higher singular values, the compensation matrix of the quantization weight matrix is ​​approximated. This achieves low-rank compensation for the quantization weight matrix, ensuring good accuracy of the compensation matrix while reducing excessive consumption of storage and computing resources. Consequently, in the model inference stage based on this compensation matrix, the inference computation efficiency of the target task can be improved.

[0081] After obtaining the compensation matrix of the first quantized weight matrix of the target network layer based on the above embodiment, Figure 2 illustrates how to use this compensation matrix during the model inference stage. As shown in Figure 2, the first quantized weight matrix W corresponding to the target network layer can be stored... q and the first compensation matrix L ASecond compensation matrix L B These matrices are read from the video memory space, and then the first compensation matrix L is... A Second compensation matrix L B Multiplying them together yields the compensation matrix. Then, the compensation matrix With the first quantization weight matrix W q The weights are added together to determine the compensated weight matrix H corresponding to the target network layer. Then, the input data of the target network layer is sequentially processed by the compensated weight matrix H and the activation matrix X to obtain the output, which serves as the input to the next network layer. As shown in Figure 2, the first quantization weight matrix W... q A matrix is ​​a matrix whose element values ​​are of low precision (e.g., 4 bits, 8 bits, etc.), while a compensation matrix is ​​a matrix whose element values ​​are of high precision (e.g., 32 bits, etc.).

[0082] In reality, most values ​​in the activation matrix are relatively small, but there are a few outliers (values ​​that are significantly larger than the others). These outliers can increase the quantization error. To reduce the impact of these outliers, they can be extracted and processed by smoothing.

[0083] In practical implementation, taking the target network layer as an example, outlier analysis can be performed on the activation matrix corresponding to the target network layer to identify and extract outliers, which are then smoothed. Smoothing outliers in the activation matrix results in a more uniform numerical distribution and further reduces the quantization difficulty of the target weight matrix, making it easier to quantize. By extracting and transferring the outliers from the activation matrix to the target weight matrix, the outlier portion of the target weight matrix can then be segmented during the construction of the compensation matrix. That is, the outlier portion of the target weight matrix does not need to be quantized; only the non-outlier portion needs to be quantized. Therefore, smoothing the activation matrix can transfer the quantization difficulty of the activation matrix to the target weight matrix.

[0084] In this embodiment of the disclosure, in order to smooth the activation matrix, a scaling matrix needs to be introduced to smooth the activation matrix. The specific implementation process of determining the scaling matrix is ​​illustrated in Figure 3.

[0085] Figure 3 is a flowchart of a process for determining a scaling matrix according to an embodiment of this disclosure. As shown in Figure 3, the method may include the following steps:

[0086] 301. Determine the weight mean vector corresponding to the target weight matrix and the activation mean vector corresponding to the activation matrix. The weight mean vector is composed of the weight mean of each of the n channels in the target weight matrix, and the activation mean vector is composed of the activation mean of each of the n channels in the activation matrix.

[0087] 302. Perform a point-by-point multiplication of the weight mean vector and the activation mean vector to obtain the product vector.

[0088] 303. Determine the f outliers in the product vector and the channel indices corresponding to the f outliers. The f outliers are the f largest values ​​in the product vector.

[0089] 304. Determine the n element values ​​corresponding to the n channels in the scaling matrix based on the f outliers to obtain the scaling matrix, which is a diagonal matrix.

[0090] The target weight matrix comprises m row channels and n column channels. Therefore, the n channels in the target weight matrix refer to n column channels. Each column channel corresponds to a weight mean, which is determined by the mean of the element values ​​of the corresponding m row channels. This results in a weight mean vector composed of the weight mean values ​​of the n column channels. Similarly, the activation matrix comprises n row channels and m column channels. The n channels in the activation matrix refer to n row channels. Each row channel corresponds to an activation mean, which is determined by the mean of the element values ​​of the corresponding m column channels. This results in an activation mean vector composed of the activation mean values ​​of the n row channels.

[0091] After determining the weight mean vector and the activation mean vector, we can perform a pointwise multiplication on the weight mean vector and the activation mean vector to obtain the product vector. Pointwise multiplication means multiplying the corresponding elements of the weight mean vector and the activation mean vector. That is, the first element of the weight mean vector is multiplied by the first element of the activation mean vector, and so on, until all elements have been multiplied.

[0092] Next, determine the f outliers in the product vector and their corresponding channel indices. Here, f is less than or equal to a preset value of r, and the f outliers are the f largest values ​​in the product vector. For example, f can be set to 32 or 48, and r to 64. In practical applications, the elements in the product vector can be sorted in descending order, and the first f elements are identified as the f outliers. Since the product vector contains n elements, corresponding to the n row channels in the activation matrix and the n column channels in the target weight matrix, the channel index of the first element in the product vector is 1, the channel index of the second element is 2, and so on.

[0093] Finally, based on the f outliers, determine the n element values ​​corresponding to the n channels in the scaling matrix. The scaling matrix is ​​an n-dimensional diagonal matrix: M = diag(m1, m2, ..., mn). n ), where diag() represents a diagonal matrix, and the diagonal element values ​​are the values ​​of the above n elements.

[0094] In one optional implementation, the process of determining the n element values ​​corresponding to the n channels in the scaling matrix can be achieved by determining the minimum value among the f outliers. For the i-th channel among the n channels, if the i-th channel is included in the channel index corresponding to the f outliers, then the corresponding i-th element value is determined to be the ratio of the activation mean of the i-th channel in the activation matrix to the minimum value; if the i-th channel is not included in the channel index corresponding to the f outliers, then the corresponding i-th element value is determined to be 1.

[0095] In other words, when determining the n element values ​​corresponding to the n channels in the scaling matrix, we can first find the f largest outliers in the product vector and the channel indices corresponding to these f largest outliers. Then, we find the smallest outlier among these f largest outliers. For each channel in the scaling matrix: if the channel's index is included in the indices corresponding to the f outliers, we calculate the ratio of the channel's activation mean in the activation matrix to the smallest outlier among the f outliers, and determine this ratio as the element value corresponding to the channel. If the channel's index is not included in the indices corresponding to the f outliers, then the element value corresponding to the channel is determined to be 1.

[0096] For example, suppose the weight mean vector is Activation mean vector is The product vector is The set of channel indices corresponding to the f largest outliers in the product vector is I. f ={i1,i2,i3…i f} can be adopted To determine the n element values ​​corresponding to the n channels in the scaling matrix M, we need to obtain the scaling matrix M.

[0097] The following section, with reference to Figure 4, explains how to achieve model quantization compensation based on the scaling matrix described above.

[0098] Figure 4 is a flowchart of another large model quantization compensation method provided in this embodiment of the present disclosure. As shown in Figure 4, the method includes the following steps:

[0099] 401. Determine the scaling matrix for smoothing the activation matrix corresponding to the target network layer of the model, and determine the fourth product matrix, which is the product of the target weight matrix and the scaling matrix corresponding to the target network layer.

[0100] 402. Based on the scaling matrix, the fourth product matrix is ​​split into the sum of the first sub-weight matrix that needs to be quantized and the second sub-weight matrix that does not need to be quantized, where the second sub-weight matrix corresponds to the channel where the outlier is located in the fourth product matrix.

[0101] 403. Quantize the first sub-weight matrix to obtain the second quantized weight matrix, and determine the second quantization error matrix based on the target weight matrix, the second quantized weight matrix, and the second sub-weight matrix.

[0102] 404. Determine a second auxiliary matrix such that the first singular value distribution is consistent with the second singular value distribution, wherein the first singular value distribution is the singular value distribution of the product of the inverse matrix of the second auxiliary matrix and the activation matrix, and the second singular value distribution is the singular value distribution of the product of the target weight matrix and the activation matrix.

[0103] 405. Perform singular value decomposition on the fifth product matrix to obtain the third singular value decomposition matrix. Extract the first r channels from the third singular value decomposition matrix to obtain the fourth singular value decomposition matrix. The fifth product matrix is ​​the product of the second quantization error matrix and the second auxiliary matrix.

[0104] 406. Based on the fourth singular value decomposition matrix, determine the compensation matrix of the second quantization weight matrix.

[0105] The purpose of steps 401-402 above is to transfer the quantization difficulty of the activation matrix to the target weight matrix. Specifically, the scaling matrix M can be used to represent the WX corresponding to the target network layer as WX = WM·M. -1 X, where M -1 X represents the smoothed result of the activation matrix X corresponding to the target network layer, and the fourth product matrix is ​​WM, which is essentially the transformed result of the target weight matrix W. Because the activation matrix X has been smoothed, it contains more uniform numerical values, which is more conducive to quantization.

[0106] Based on the above migration results, it is clear that the next step is to compensate for the integral quantization error WM.

[0107] In practice, based on the scaling matrix M, the fourth product matrix WM is split into the first sub-weight matrix W that needs to be quantized. S and the second sub-weight matrix W that does not require quantization O The sum of the two, i.e., WM = W S +W O Among them, the second sub-weight matrix W O This corresponds to the channel where the outlier is located in the fourth product matrix WM.

[0108] Specifically, based on the channel indices of the aforementioned f outliers, the corresponding f channels (f column channels) can be extracted from the fourth product matrix WM and used as the second sub-weight matrix W. O The first f columns of channels, the second sub-weight matrix W O The remaining channels all have elements of 0, thus obtaining the second sub-weight matrix W. O After that, with WM-W O Then W was obtained S As can be seen, the first sub-weight matrix W S There will be elements with a value of 0 in column f, while other columns need to be quantized.

[0109] Next, for the first sub-weight matrix W S Quantization is performed to obtain the second quantization weight matrix Q(W). S Based on the target weight matrix W and the second quantization weight matrix Q(W S ) and the second sub-weight matrix W O Determine the second quantization error matrix E q1 E q1 =WQ(W S )+W O .

[0110] Then based on the second quantization error matrix E q1 For the second quantization weight matrix Q(W) S The process of constructing the corresponding compensation matrix is ​​similar to that in the previous embodiment based on the first quantization error matrix E. q For the first quantization weight matrix W q The process of constructing the corresponding compensation matrix is ​​similar.

[0111] Specifically, a second auxiliary matrix S1 is determined such that the first singular value distribution is consistent with the second singular value distribution, wherein the first singular value distribution is the inverse of the second auxiliary matrix S1. -1 The product S1 with the activation matrix X -1 The singular value distribution of X is the first singular value distribution of X, and the second singular value distribution is the singular value distribution of the product WX of the target weight matrix W and the activation matrix X.

[0112] In practical applications, the Cholesky decomposition method can be used to calculate the inverse matrix M of the scaling matrix. -1 Multiplying the product with the activation matrix X and performing matrix decomposition yields the second auxiliary matrix S1. The decomposition result is the second auxiliary matrix S1 and the transpose S1 of the first auxiliary matrix. T The product of.

[0113] Furthermore, when the second auxiliary matrix S1 satisfies (S1 -1 M -1X)(S1 -1 M -1 X) T When = I, it can be guaranteed that the second auxiliary matrix S1 is such that the product S1 of the inverse of the second auxiliary matrix and the activation matrix is ​​S1 -1 X is consistent with the singular value distribution of WX, the product of the target weight matrix and the activation matrix. Here, I is the identity matrix.

[0114] Once the second auxiliary matrix S1 is determined, the second quantization error matrix E can be processed first. q1 Multiplying the product matrix E with the second auxiliary matrix S1 yields the fifth product matrix E. q1 S1, then on the fifth product matrix E q1 S1 performs singular value decomposition to obtain the third singular value decomposition matrix. The first r channels of the third singular value decomposition matrix are then extracted to obtain the fourth singular value decomposition matrix.

[0115] Specifically, the SVD decomposition method can be used to decompose the fifth product matrix E. q1 S1 performs singular value decomposition to obtain the product of the third left unitary matrix U1, the third diagonal matrix Σ1, and the third right unitary matrix V1, i.e., E. q1 S1=U1Σ1V1 T Among them, E q1 S1 is an m×n matrix, U1=[u1,u2,…u n ],Σ1=diag(σ1,σ2…σ n V1 = [v1, v2, ... v1] n ] T , σ1,σ2…σ n It is a singular value.

[0116] Then, the first r channels of the third left unitary matrix U1, the third diagonal matrix Σ1, and the third right unitary matrix V1 are extracted to obtain the fourth left unitary matrix U. 1r The fourth diagonal matrix Σ 1r The fourth right-unitary matrix V 1r The fourth left-unitary matrix U can be used. 1r The fourth diagonal matrix Σ 1r and the fourth right unitary matrix V 1r The product approximates the fifth product matrix, that is, we obtain an approximation of the fifth product matrix.

[0117] In other words, the fourth left-unitary matrix retains only the rows corresponding to the first r singular values ​​of the third left-unitary matrix, the fourth diagonal matrix retains the first r diagonal elements of the third diagonal matrix, and the fourth right-unitary matrix retains only the columns corresponding to the first r singular values ​​of the third right-unitary matrix. Therefore, the fourth left-unitary matrix, the fourth diagonal matrix, and the fourth right-unitary matrix are low-rank approximation matrices, allowing us to approximate the matrix using these three low-rank approximation matrices.

[0118] Finally, based on the fourth left-unitary matrix, the fourth diagonal matrix, the fourth right-unitary matrix, and the second auxiliary matrix, the second quantization weight matrix Q(W) is determined. S The compensation matrix

[0119] In addition, in practical applications, to improve computational efficiency and reduce storage requirements, the compensation matrix can also be... The model splits the weights into two smaller compensation matrices for storage and retrieval, respectively, to compensate the second quantization weight matrix Q(W) during the model's inference phase. S ).

[0120] Specifically, in an optional embodiment, the specific implementation of determining the compensation matrix of the second quantization weight matrix based on the fourth left unitary matrix, the fourth diagonal matrix, the fourth right unitary matrix, and the second auxiliary matrix can be as follows: the product of the fourth left unitary matrix and the fourth diagonal matrix is ​​determined to be the third compensation matrix L. A1 =U 1r Σ 1r The product of the transpose of the fourth right-unitary matrix and the inverse of the second auxiliary matrix is ​​determined to be the fourth compensation matrix L. B1 =V 1r T S1 -1 Store the third and fourth compensation matrices.

[0121] As mentioned above, the memory spaces of the two GPUs can be determined, and they will be used to store the third compensation matrix and the fourth compensation matrix, respectively.

[0122] In summary, in this embodiment, by smoothing the activation matrix, the adverse effects of the activation matrix of the preceding network layer on the quantization of the weight matrix of the following network layer can be reduced. Based on the scaling matrix used in the smoothing process, the quantization difficulty of the activation matrix is ​​transferred to the weight matrix. The weight matrix can be split into sub-weight matrices that do not require quantization and sub-weight matrices that do require quantization. Quantization is only required for the sub-weight matrices that require quantization, reducing the computational power demand for quantization. Furthermore, the quantization error matrices corresponding to the sub-weight matrices that do not require quantization and those that do require quantization can be compensated using low-rank decomposition. The compensation matrix has low-rank characteristics, resulting in lower storage and computational requirements.

[0123] Based on the above embodiment, the second sub-quantization weight matrix Q(W) of the target network layer is obtained. S After obtaining the corresponding compensation matrix, Figure 5 illustrates how to use this compensation matrix during the model inference stage. As shown in Figure 5, the second quantization weight matrix Q(W) corresponding to the stored target network layer can be used. S ) and the third compensation matrix L A1 and the fourth compensation matrix L B1 These matrices are read from the video memory space, and then the third compensation matrix L is... A1 and the fourth compensation matrix L B1 Multiplying them together yields the compensation matrix. Then, the compensation matrix With the second quantization weight matrix Q(W) S The weights are summed to determine the compensated weight matrix H corresponding to the target network layer. Then, the input data of the target network layer sequentially passes through the compensated weight matrix H and the smoothed activation matrix M. -1 After X is calculated, the output result is obtained and used as the input to the next network layer. The scaling matrix M is pre-stored for retrieval when needed. As shown in Figure 5, the second quantization weight matrix Q(W) S The matrix L is a matrix whose elements are of low precision (e.g., 4-bit, 8-bit, etc.), while the third compensation matrix L... A1 and the fourth compensation matrix L B1 It is a matrix whose element values ​​are of high precision (e.g., 32 bits).

[0124] In practical applications, many fields require quantization of pre-trained models, and the technical solutions of this disclosure can be used in all of these cases. The following is an illustrative description of specific embodiments. In practice, it is necessary to quantize a pre-trained large language model to obtain a quantized model, and then deploy the quantized model to a mobile device.

[0125] It should be noted that in this embodiment, the target weight matrix corresponding to each network layer in the large language model is quantized and quantization compensation is performed separately to obtain the output results of each network layer. Since the processing procedure is the same for multiple network layers in the model, this embodiment only uses any one of the network layers as the target network layer, and the processing procedure of the target network layer is used as an example for explanation.

[0126] First, obtain the target weight matrix W and calibration data D corresponding to the target network layer in the pre-trained large language model. Input the calibration data into the target network layer in the pre-trained large language model to obtain the activation matrix X corresponding to the target network layer, and store the activation matrix X.

[0127] Next, it is determined whether to smooth the activation matrix X. If it is determined not to smooth the activation matrix, the target weight matrix is ​​quantized to obtain the first quantized weight matrix W. q Based on the first quantization weight matrix and the target weight matrix, the first quantization error matrix E is determined. q =WW q Determine the first auxiliary matrix S, and the product E of the first quantization error matrix and the first auxiliary matrix. q S. To E q S undergoes singular value decomposition, yielding the product of the first left unitary matrix U, the first diagonal matrix Σ, and the first right unitary matrix V, i.e., E. q S=UΣV T The first r channels of the first left unitary matrix, the first diagonal matrix, and the first right unitary matrix are truncated to obtain the second left unitary matrix U. r The second diagonal matrix Σ r Second right unitary matrix V r The product of the second left unitary matrix and the second diagonal matrix is ​​used to determine the first compensation matrix L. A L A =U r Σ r The product of the transpose of the second right-unitary matrix and the inverse of the first auxiliary matrix is ​​used to determine the second compensation matrix L. B L B =V r T S -1 .

[0128] If it is determined that the activation matrix will be smoothed, then the scaling matrix M for smoothing the activation matrix and the product WM of the scaling matrix and the target weight matrix are determined. WM is then decomposed into the first sub-weight matrix W to be quantized. S and the second sub-weight matrix W that does not require quantization O The sum of the two, i.e., WM = WS +W O The first sub-weight matrix is ​​quantized to obtain the second quantized weight matrix Q(W). S Based on the target weight matrix, the second quantization weight matrix, and the second sub-weight matrix, the second quantization error matrix E is determined. q1 =WQ(W S )+W O Determine the second auxiliary matrix S1 and the product E of the second quantization error matrix and the second auxiliary matrix. q1 S1, for E q1 S1 performs singular value decomposition to obtain the product of the third left unitary matrix U1, the third diagonal matrix Σ1, and the third right unitary matrix V1, i.e., E. q1 S1=U1Σ1V1 T The first r channels of the third left-unitary matrix, the third diagonal matrix, and the third right-unitary matrix are extracted respectively to obtain the fourth left-unitary matrix U. 1r The fourth diagonal matrix Σ 1r and the fourth right unitary matrix V 1r The product of the fourth left unitary matrix and the fourth diagonal matrix is ​​used to determine the third compensation matrix L. A1 L A1 =U 1r Σ 1r The product of the transpose of the fourth right-unitary matrix and the inverse of the second auxiliary matrix is ​​used to determine the fourth compensation matrix L. B1 L B1 =V 1r S1 -1 .

[0129] This allows for flexible selection of the appropriate model quantization compensation method to perform model quantization compensation processing on pre-trained large language models.

[0130] The large model quantization compensation apparatus of one or more embodiments of this disclosure will be described in detail below. Those skilled in the art will understand that these apparatuses can be configured using commercially available hardware components through the steps taught in this solution.

[0131] Figure 6 is a schematic diagram of a large model quantization compensation device provided in an embodiment of this disclosure. As shown in Figure 6, the device includes: a quantization module 11, a first determination module 12, a decomposition module 13, and a second determination module 14.

[0132] The quantization module 11 is used to determine the first quantization error matrix based on the target weight matrix corresponding to the target network layer of the model and the first quantization weight matrix corresponding to the target weight matrix, wherein the target network layer is any one of the multiple network layers contained in the model.

[0133] The first determining module 12 is used to determine a first auxiliary matrix, wherein the first auxiliary matrix makes the singular value distributions of the first product matrix and the second product matrix consistent, the first product matrix is ​​the product of the inverse matrix of the first auxiliary matrix and the activation matrix corresponding to the target network layer, and the second product matrix is ​​the product of the target weight matrix and the activation matrix.

[0134] The decomposition module 13 is used to perform singular value decomposition on the third product matrix to obtain a first singular value decomposition matrix, and to truncate the first singular value decomposition matrix by the first r channels to obtain a second singular value decomposition matrix containing larger singular values, where r is a set rank truncation length, and the third product matrix is ​​the product of the first quantization error matrix and the first auxiliary matrix.

[0135] The second determining module 14 is used to determine the compensation matrix of the first quantization weight matrix based on the second singular value decomposition matrix and the first auxiliary matrix.

[0136] Optionally, the decomposition module 13 is used to perform singular value decomposition on the third product matrix to obtain a first left-unitary matrix, a first diagonal matrix, and a first right-unitary matrix; and to truncate the first left-unitary matrix, the first diagonal matrix, and the first right-unitary matrix for the first r channels respectively to obtain a second left-unitary matrix, a second diagonal matrix, and a second right-unitary matrix. Therefore, the second determining module 14 is used to determine the compensation matrix of the first quantization weight matrix based on the second left-unitary matrix, the second diagonal matrix, the second right-unitary matrix, and the first auxiliary matrix.

[0137] Optionally, the second determining module 14 is specifically used to: determine that the product of the second left unitary matrix and the second diagonal matrix is ​​a first compensation matrix; determine that the product of the transpose of the second right unitary matrix and the inverse of the first auxiliary matrix is ​​a second compensation matrix; and store the first compensation matrix and the second compensation matrix to compensate the first quantization weight matrix with the product of the first compensation matrix and the second compensation matrix during the inference phase of the model.

[0138] Optionally, the first determining module 12 is specifically used to: perform matrix decomposition on the activation matrix to decompose the activation matrix into the product of the first auxiliary matrix and the transpose of the first auxiliary matrix.

[0139] Optionally, the device further includes: a third determining module, a splitting module, and a compensation module.

[0140] The third determining module is used to determine the scaling matrix for smoothing the activation matrix, and to determine the fourth product matrix corresponding to the product of the target weight matrix and the scaling matrix.

[0141] The splitting module is used to split the fourth product matrix into a sum of a first sub-weight matrix that needs to be quantized and a second sub-weight matrix that does not need to be quantized, based on the scaling matrix, wherein the second sub-weight matrix corresponds to the channel where the outlier is located in the fourth product matrix.

[0142] The compensation module is used to quantize the first sub-weight matrix to obtain a second quantized weight matrix, and determine a second quantization error matrix based on the target weight matrix, the second quantized weight matrix, and the second sub-weight matrix; determine a second auxiliary matrix, which makes the first singular value distribution consistent with the second singular value distribution, wherein the first singular value distribution is the singular value distribution of the product of the inverse matrix of the second auxiliary matrix and the activation matrix, and the second singular value distribution is the singular value distribution of the product of the target weight matrix and the activation matrix; perform singular value decomposition on the fifth product matrix to obtain a third singular value decomposition matrix, and truncate the third singular value decomposition matrix by the first r channels to obtain a fourth singular value decomposition matrix, wherein the fifth product matrix is ​​the product of the second quantization error matrix and the second auxiliary matrix; and determine the compensation matrix of the second quantization weight matrix based on the fourth singular value decomposition matrix.

[0143] Specifically, the compensation module is used to: perform singular value decomposition on the fifth product matrix to obtain a third left-unitary matrix, a third diagonal matrix, and a third right-unitary matrix; truncate the third left-unitary matrix, the third diagonal matrix, and the third right-unitary matrix for the first r channels to obtain a fourth left-unitary matrix, a fourth diagonal matrix, and a fourth right-unitary matrix; and determine the compensation matrix of the second quantization weight matrix based on the fourth left-unitary matrix, the fourth diagonal matrix, the fourth right-unitary matrix, and the auxiliary matrix.

[0144] The third determining module is specifically used to determine the weight mean vector corresponding to the target weight matrix and the activation mean vector corresponding to the activation matrix. The weight mean vector is composed of the weight mean values ​​corresponding to each of the n channels in the target weight matrix, and the activation mean vector is composed of the activation mean values ​​corresponding to each of the n channels in the activation matrix. The module performs a pointwise multiplication of the weight mean vector and the activation mean vector to obtain a product vector. It determines f outliers in the product vector and the channel indices corresponding to the f outliers, where f is less than or equal to a preset value of r, and the f outliers are the f largest values ​​in the product vector. Based on the f outliers, it determines the n element values ​​corresponding to the n channels in the scaling matrix to obtain the scaling matrix, which is a diagonal matrix.

[0145] The third determining module is used to determine the minimum value among the f outliers; for the i-th channel among the n channels, if the i-th channel is included in the channel index corresponding to the f outliers, then the corresponding i-th element value is determined to be the ratio of the activation mean of the i-th channel in the activation matrix to the minimum value; if the i-th channel is not included in the channel index corresponding to the f outliers, then the corresponding i-th element value is determined to be 1.

[0146] The compensation module is further configured to: extract the corresponding f channels from the fourth product matrix based on the channel indices corresponding to the f outliers, so as to generate the second sub-weight matrix, wherein the values ​​of the other channels in the second sub-weight matrix other than the f channels are all 0.

[0147] The device shown in Figure 6 can perform the steps in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0148] In one possible design, the structure of the large model quantization compensation device shown in Figure 6 can be implemented as an electronic device. As shown in Figure 7, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code, which, when executed by the processor 21, enables the processor 21 to at least implement the large model quantization compensation method provided in the foregoing embodiments.

[0149] In addition, this disclosure provides a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the large model quantization compensation method provided in the foregoing embodiments.

[0150] The invention also provides a computer program product, which includes a computer program that, when executed by a processor, enables the processor to at least implement the large model quantization compensation method provided in the foregoing embodiments.

[0151] The device embodiments described above are merely illustrative. The network elements described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. This disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A large-scale model quantization compensation method, wherein, include: The first quantization error matrix is ​​determined based on the target weight matrix corresponding to the target network layer of the model and the first quantization weight matrix corresponding to the target weight matrix. The target network layer is any one of the multiple network layers contained in the model. A first auxiliary matrix is ​​determined, which makes the singular value distributions of the first product matrix and the second product matrix consistent. The first product matrix is ​​the product of the inverse of the first auxiliary matrix and the activation matrix corresponding to the target network layer. The second product matrix is ​​the product of the target weight matrix and the activation matrix. Singular value decomposition is performed on the third product matrix to obtain the first singular value decomposition matrix. The first singular value decomposition matrix is ​​then truncated for the first r channels to obtain the second singular value decomposition matrix containing larger singular values, where r is the set rank truncation length. The third product matrix is ​​the product of the first quantization error matrix and the first auxiliary matrix. The compensation matrix of the first quantization weight matrix is ​​determined based on the second singular value decomposition matrix and the first auxiliary matrix.

2. The method according to claim 1, wherein, The step of performing singular value decomposition on the third product matrix to obtain a first singular value decomposition matrix, and then truncating the first singular value decomposition matrix by extracting the first r channels to obtain a second singular value decomposition matrix containing larger singular values, includes: Singular value decomposition is performed on the third product matrix to obtain the first left unitary matrix, the first diagonal matrix, and the first right unitary matrix; The first left unitary matrix, the first diagonal matrix, and the first right unitary matrix are truncated for the first r channels to obtain the second left unitary matrix, the second diagonal matrix, and the second right unitary matrix. The step of determining the compensation matrix for the first quantization weight matrix based on the second singular value decomposition matrix and the first auxiliary matrix includes: The compensation matrix of the first quantization weight matrix is ​​determined based on the second left unitary matrix, the second diagonal matrix, the second right unitary matrix, and the first auxiliary matrix.

3. The method according to claim 2, wherein, The step of determining the compensation matrix for the first quantization weight matrix based on the second left unitary matrix, the second diagonal matrix, the second right unitary matrix, and the first auxiliary matrix includes: The product of the second left unitary matrix and the second diagonal matrix is ​​determined to be the first compensation matrix; The product of the transpose of the second right unitary matrix and the inverse of the first auxiliary matrix is ​​determined to be the second compensation matrix; The first compensation matrix and the second compensation matrix are stored to compensate the first quantization weight matrix with the product of the first compensation matrix and the second compensation matrix during the inference phase of the model.

4. The method according to any one of claims 1-3, wherein, Determining the first auxiliary matrix includes: The activation matrix is ​​decomposed into the product of the first auxiliary matrix and the transpose of the first auxiliary matrix.

5. The method according to any one of claims 1-4, wherein, The method further includes: Determine the scaling matrix for smoothing the activation matrix, and determine the fourth product matrix corresponding to the product of the target weight matrix and the scaling matrix; According to the scaling matrix, the fourth product matrix is ​​split into the sum of a first sub-weight matrix that needs to be quantized and a second sub-weight matrix that does not need to be quantized, wherein the second sub-weight matrix corresponds to the channel where the outlier is located in the fourth product matrix; The first sub-weight matrix is ​​quantized to obtain the second quantized weight matrix, and the second quantization error matrix is ​​determined based on the target weight matrix, the second quantized weight matrix, and the second sub-weight matrix. A second auxiliary matrix is ​​determined such that the first singular value distribution is consistent with the second singular value distribution, wherein the first singular value distribution is the singular value distribution of the product of the inverse matrix of the second auxiliary matrix and the activation matrix, and the second singular value distribution is the singular value distribution of the product of the target weight matrix and the activation matrix. Singular value decomposition is performed on the fifth product matrix to obtain the third singular value decomposition matrix. The first r channels of the third singular value decomposition matrix are then truncated to obtain the fourth singular value decomposition matrix. The fifth product matrix is ​​the product of the second quantization error matrix and the second auxiliary matrix. The compensation matrix for the second quantization weight matrix is ​​determined based on the fourth singular value decomposition matrix.

6. The method according to claim 5, wherein, The process of performing singular value decomposition on the fifth product matrix to obtain the third singular value decomposition matrix, and then truncating the third singular value decomposition matrix by extracting the first r channels to obtain the fourth singular value decomposition matrix, includes: Singular value decomposition is performed on the fifth product matrix to obtain the third left-unitary matrix, the third diagonal matrix, and the third right-unitary matrix; The first r channels of the third left unitary matrix, the third diagonal matrix, and the third right unitary matrix are respectively extracted to obtain the fourth left unitary matrix, the fourth diagonal matrix, and the fourth right unitary matrix; The step of determining the compensation matrix for the second quantization weight matrix based on the fourth singular value decomposition matrix includes: The compensation matrix of the second quantization weight matrix is ​​determined based on the fourth left unitary matrix, the fourth diagonal matrix, the fourth right unitary matrix, and the auxiliary matrix.

7. The method according to claim 5 or 6, wherein, The step of determining the scaling matrix for smoothing the activation matrix includes: Determine the weight mean vector corresponding to the target weight matrix and the activation mean vector corresponding to the activation matrix, wherein the weight mean vector is composed of the weight mean of each of the n channels in the target weight matrix, and the activation mean vector is composed of the activation mean of each of the n channels in the activation matrix; The weight mean vector and the activation mean vector are multiplied by a dot product to obtain a product vector; Determine f outliers in the product vector and the channel indices corresponding to the f outliers, wherein f is less than or equal to a preset value of r, and the f outliers are the f larger values ​​in the product vector; Based on the f outliers, determine the n element values ​​corresponding to the n channels in the scaling matrix to obtain the scaling matrix, which is a diagonal matrix.

8. The method according to claim 7, wherein, The step of determining the n element values ​​corresponding to the n channels in the scaling matrix based on the f outliers includes: Determine the minimum value among the f outliers; For the i-th channel among the n channels, if the i-th channel is included in the channel index corresponding to the f outliers, then the corresponding i-th element value is determined to be: the ratio of the activation mean to the minimum value of the i-th channel in the activation matrix; if the i-th channel is not included in the channel index corresponding to the f outliers, then the corresponding i-th element value is determined to be 1.

9. The method according to claim 8, wherein, The method further includes: Based on the channel indices corresponding to the f outliers, the corresponding f channels are extracted from the fourth product matrix to generate the second sub-weight matrix. In the second sub-weight matrix, all channels other than the f channels have a value of 0.

10. The method according to any one of claims 1-9, wherein, The step of quantizing the target weight matrix to obtain a first quantized weight matrix includes: The target weight matrix is ​​quantized using any one of recursive ternary quantization, mixed precision quantization, or generalized post-training quantization, converting the high-precision target weight matrix into a low-precision first quantized weight matrix. The high precision is 32 bits, and the low precision is 4 bits or 8 bits.

11. The method according to claim 3, wherein, The method further includes: During the inference phase of the model, the stored first quantization weight matrix, first compensation matrix, and second compensation matrix are read. Calculate the product of the first compensation matrix and the second compensation matrix to obtain the complete compensation matrix; Add the complete compensation matrix to the first quantization weight matrix to obtain the compensated weight matrix; Based on the calculation results of the compensated weight matrix and the activation matrix, the output data of the target network layer is generated and used as the input of the next network layer.

12. The method according to claim 3, wherein, The storage of the first compensation matrix and the second compensation matrix includes: Obtain the remaining storage capacity of each GPU in the target device; Depending on the remaining storage capacity, the first compensation matrix and the second compensation matrix are stored in the same GPU's memory space, or stored in the memory spaces of two different GPUs respectively. The target device is a terminal device or a server, and the terminal device includes a smartphone, a PC, or an intelligent robot.

13. The method according to claim 5, wherein, The step of quantizing the first sub-weight matrix to obtain the second quantized weight matrix includes: Using any one of the following quantization methods—recursive ternary quantization, mixed precision quantization, or generalized post-training quantization—the channels other than the f channels with an element value of 0 in the first sub-weight matrix are quantized to obtain a low-precision second quantized weight matrix. The second quantization weight matrix has a precision of 4 bits or 8 bits, and the original precision of the first sub-weight matrix is ​​32 bits.

14. The method according to claim 5, wherein, Determining the second auxiliary matrix includes: Perform matrix decomposition on the product of the inverse of the scaling matrix and the activation matrix to obtain the product of the second auxiliary matrix and the transpose of the second auxiliary matrix; The matrix decomposition adopts the Cholesky decomposition method, and the second auxiliary matrix satisfies the following: the transpose of the product of the inverse of the second auxiliary matrix, the inverse of the scaling matrix, and the activation matrix, multiplied by the product of the inverse of the second auxiliary matrix, the inverse of the scaling matrix, and the activation matrix, results in an identity matrix.

15. The method according to claim 6, wherein, The step of determining the compensation matrix for the second quantization weight matrix based on the fourth left-unitary matrix, the fourth diagonal matrix, the fourth right-unitary matrix, and the second auxiliary matrix includes: The product of the fourth left unitary matrix and the fourth diagonal matrix is ​​determined to be the third compensation matrix; The product of the transpose of the fourth right-unitary matrix and the inverse of the second auxiliary matrix is ​​determined to be the fourth compensation matrix; The third compensation matrix and the fourth compensation matrix are stored so that the second quantization weight matrix can be compensated by multiplying the third compensation matrix and the fourth compensation matrix during the inference phase of the model.

16. The method according to claim 9, wherein, The method further includes: During the inference phase of the model, the stored second quantization weight matrix, the third compensation matrix, the fourth compensation matrix, and the scaling matrix are read. Calculate the product of the third compensation matrix and the fourth compensation matrix to obtain the complete compensation matrix; The complete compensation matrix is ​​added to the second quantization weight matrix to obtain the compensated weight matrix; Calculate the product of the inverse of the scaling matrix and the activation matrix to obtain the smoothed activation matrix; Based on the calculation results of the compensated weight matrix and the smoothed activation matrix, the output data of the target network layer is generated and used as the input of the next network layer.

17. The method according to claim 9, wherein, The storage of the third compensation matrix and the fourth compensation matrix includes: Obtain the remaining storage capacity of each GPU in the target device; Depending on the remaining storage capacity, the third compensation matrix and the fourth compensation matrix are stored in the same GPU's memory space, or stored in the memory spaces of two different GPUs respectively. The target device is a terminal device or a server, and the terminal device includes a smartphone, a PC, or an intelligent robot.

18. An electronic device, wherein, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code, which, when executed by the processor, causes the processor to perform the large model quantization compensation method as described in any one of claims 1 to 17.

19. A non-transitory machine-readable storage medium, wherein, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the large model quantization compensation method as described in any one of claims 1 to 7.

20. A computer program product, wherein, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the large model quantization compensation method as described in any one of claims 1 to 17.

Images