A large language model compression method and system based on low-rank quantization

Abstract

The present invention proposes a large language model compression method and system based on low-rank quantization, which belongs to the technical field of large language model compression, including: obtaining calibration data, loading a large language model to be compressed; quantizing the weight of the large language model, replacing the full-precision linear layer weight W of the large language model with the sum of the product of the quantized linear layer weight Q and quantized low-rank factors L and R, so as to achieve quantization and low-rank approximation of the weight matrix; compressing the large language model according to the quantized weights; obtaining a compressed large language model with an accuracy almost equivalent to that of the original model, and significantly reducing the calculation and storage overhead of the large language model while maximally maintaining the accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language model compression, and specifically relates to a large language model compression method and system based on low-rank quantization. Background Art

[0002] Large Language Models (LLMs) have become an important breakthrough in the field of artificial intelligence. These models use deep learning technology to learn the grammar, semantics, and regularities of language from massive amounts of text data, and can generate natural and fluent text. However, the huge number of parameters in large language models leads to high computing resource requirements, limiting their application in resource-constrained environments. In addition, the huge number of parameters also increases the storage cost and inference time of the model, reducing the deployment and inference efficiency.

[0003] In order to solve the resource consumption and efficiency problems of large language models in practical applications, quantization and low-rank decomposition techniques have played an important role. Quantization refers to the process of converting the parameters and calculation results in a large model from high-precision representation to low-precision representation. The advantage is that it can significantly reduce the size of the model weights without almost reducing the accuracy, speed up the transfer of weights on the GPU, improve computational efficiency, and even speed up the calculation. The rank represents the amount of non-redundant information in the matrix. Low-rank decomposition can decompose the feature matrix into the product of two small matrices to approximate it, thereby greatly reducing the number of parameters of the feature matrix with a slight loss of accuracy. However, due to the complex structure and large number of parameters of large language models, traditional quantization methods often find it difficult to maintain the accuracy of the original model. Summary of the invention

[0004] The present invention aims at solving the problems in the prior art, and the technical solution adopted in this application is as follows:

[0005] In a first aspect, the present invention provides a large language model compression method based on low-rank quantization, comprising:

[0006] Load the large language model to be compressed and obtain calibration data;

[0007] The model infers the calibration data and saves the activation values ​​of each linear layer of the model as the calibration matrix X;

[0008] Quantize the weights of the large language model and replace the full-precision linear layer weight W of the large language model with the sum of the quantized linear layer weight Q and the product of the quantized low-rank factors L and R with a rank of k, to achieve quantization and low-rank approximation of the weight matrix;

[0009] Use quantized weights to compress large language models.

[0010] Furthermore, the step of quantizing the large language model weight and replacing the full-precision linear layer weight W of the large language model with the sum of the product of the quantized linear layer weight Q and the quantized low-rank factors L and R includes:

[0011] Quantize each linear layer matrix in the large language model column by column to obtain the quantized model weight Q;

[0012] For each linear layer of the large language model, a residual matrix A=WQ is calculated, where W is the linear layer matrix of each layer of the loaded model weights, and Q is the linear layer matrix of the corresponding quantized model obtained; the right factor matrix R is initialized using the singular value decomposition of the calibration matrix X, and the residual matrix A=WQ is calculated by minimizing To obtain the left factor matrix L; through the obtained L, minimize To obtain the right factor matrix R; update L and R alternately multiple times, and take L and R with the smallest norm error.

[0013] The obtained L and R are quantified separately by MinMax algorithm;

[0014] Perform forward inference on the original linear layer of the model Operation, replace with .

[0015] Furthermore, the step of obtaining the quantized model weight Q includes: the quantization algorithm uses MinMax quantization, which is expressed by the following formula: in, and Respectively represent the full-precision vector before and after quantization of each column weight, s is the scaling factor, z is the zero point value; let N be the number of quantization bits, for symmetric quantization: , ; For asymmetric quantization: , ; Finally, a quantized model weight Q is obtained.

[0016] Furthermore, the initial value of the right factor matrix R is obtained by the formula The solution is calculated, where is the k-dimensional identity matrix, the singular value decomposition formula , .

[0017] Furthermore, the right factor matrix R is obtained by the formula Perform calculations to obtain the solution.

[0018] Furthermore, the left factor matrix L is obtained by the formula Perform calculations to obtain the solution.

[0019] In a second aspect, the present invention provides a large language model compression system based on low-rank quantization, comprising:

[0020] The data preparation module is configured to: obtain calibration data and load a large language model to be compressed;

[0021] The model processing module is configured to: quantize the weight of the large language model, replace the full-precision linear layer weight W of the large language model with the sum of the quantized linear layer weight Q and the product of the quantized low-rank factors L and R, and realize the quantization and low-rank approximation of the weight matrix;

[0022] The model compression module is configured to compress the large language model according to the quantized weights.

[0023] In a third aspect, the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model compression method based on low-rank quantization described in the first aspect.

[0024] In a fourth aspect, the present invention provides an electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the electronic device is triggered to execute the large language model compression method based on low-rank quantization described in the first aspect.

[0025] Compared with the prior art, the advantages and positive effects of the present invention are:

[0026] Based on the combined use of quantization and low-rank decomposition, the present invention proposes a large language model compression method based on low-rank quantization, in which the large model weights are decomposed into the sum of the products of quantized weights and quantized low-rank factors, that is, the full-precision linear layer weight W of the model is replaced by the sum of the products of the quantized linear layer weights Q and the quantized low-rank factors L and R, thereby realizing the quantization and low-rank approximation of the weight matrix, and obtaining a compressed large language model with an accuracy almost equivalent to that of the original model, which significantly reduces the computational and storage overhead of the large language model while maximizing the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 It is a flowchart of the large language model compression method based on low-rank quantization of the present invention. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0031] Embodiment 1, as Figure 1 As shown, this embodiment provides a large language model compression method based on low-rank quantization, which specifically includes the following steps: S1: prepare model and calibration data; S2: save the activation of each layer after the model inputs the calibration data; S3: quantize the model weight; S4: low-rank decomposition residual matrix; S5: quantize the factors of low-rank decomposition; S6: save the quantized weight.

[0032] Step S1 includes:

[0033] S11: Load the full-precision large language model Qwen1.5-72B-Chat that is ready for compression. It is a large language model that supports Chinese and English and is trained using more than 3 trillion texts. It has 72 billion parameters and supports a context length of up to 32,768 tokens. It is suitable for tasks such as text generation, translation, summarization, and question-answering systems.

[0034] S12: Prepare a batch of calibration data. The calibration data is task-related. For a large language model, each calibration data is a text composed of a continuous text that is as relevant to the application scenario as possible. This embodiment prepares 64 calibration data with a length of 32768. The calibration data production process includes: downloading the open source LCCC-base dataset, which is a Chinese conversation dataset; then sampling the conversation text in the dataset, selecting 5000 texts to splice into a long text string, and then using the tokenizer of the model loaded in step S11 to segment the long text string, converting it into an integer sequence (a list of token_id), and then preparing 64 groups of lists with 32768 numbers (token_id) as a group as calibration data.

[0035] Step S2 includes:

[0036] A hook function is defined in the code to capture the activation values ​​before and after all linear layers in the model loaded in S11. Then, the calibration dataset prepared in S12 is used to perform forward inference on the model. The hook function automatically saves the activation values ​​of the linear layer input into a set, the calibration matrix X, for use in subsequent steps. For the first layer of the model, X contains the input embedding from the calibration dataset, and for the middle layer, X contains the input activation of the current layer, that is, the output activation of the previous layer.

[0037] Calculate the Hessian matrix H for future use, formula: , where m is the batch size of the calibration matrix X.

[0038] Step S3 includes:

[0039] The weights loaded in step S11 are quantized. Specifically, each linear layer matrix in the model is quantized column by column. The quantization algorithm uses MinMax quantization, which is expressed by the following formula: in, and Respectively represent the full-precision vector before and after quantization of each column weight, s is the scaling factor, z is the zero point value; let N be the number of quantization bits, for symmetric quantization: , ; For asymmetric quantization: , ; Finally, a quantized model weight Q is obtained.

[0040] Step S4 is specifically as follows:

[0041] S41: For each linear layer of the model, calculate a residual matrix A=WQ, assuming that the matrix shape of A is nx d, W is the linear layer matrix of one of the layers of the model weights loaded in S11, and Q is the quantized model obtained in step S3 The linear layer weight matrix corresponding to a layer of quantization W.

[0042] S42: Calculate the singular value decomposition (SVD) of the current layer calibration matrix X, and decompose it to obtain Three matrices, then calculate The SVD of the matrix product gives Three matrices, let The right factor matrix R is obtained, and the matrix shape is kxd, where k represents the rank of the low-rank matrices L and R after decomposing A. In this embodiment, it is 128. is the k-dimensional identity matrix.

[0043] S43: By minimizing To obtain the left factor matrix L, the shape is kxd, the minimization can be done by formula Calculate the value of L. Then, by using the obtained L and the formula To update the value of the right factor matrix R.

[0044] S44: Repeat step S43, alternately update the values ​​of L and R each time, and record the norm error The values ​​of L and R with the smallest norm error are taken as their respective final values.

[0045] Step S5 is specifically as follows:

[0046] The low-rank matrices L and R obtained in step S4 are quantized respectively by the MinMax algorithm used in step S3.

[0047] Step S6 is specifically as follows:

[0048] Perform forward inference on the original linear layer of the model Operation, replace with .

[0049] Finally, by replacing the large model weights before quantization with the quantized large model weights, the space occupied in the inference system will be greatly reduced, thus achieving compression of the large language model.

[0050] The full-precision linear layer weight W of the model is replaced by the sum of the quantized linear layer weight Q and the product of the quantized low-rank factors L and R, thereby achieving quantization and low-rank approximation of the weight matrix, and obtaining a compressed large language model with an accuracy almost equivalent to that of the original model. This significantly reduces the computational and storage overhead of the large language model while maintaining accuracy to the maximum extent.

[0051] Embodiment 2: This embodiment provides a large language model compression system based on low-rank quantization, including:

[0052] The data preparation module is configured to: obtain calibration data and load a large language model to be compressed;

[0053] The model processing module is configured to: quantize the weight of the large language model, replace the full-precision linear layer weight W of the large language model with the sum of the quantized linear layer weight Q and the product of the quantized low-rank factors L and R, and realize the quantization and low-rank approximation of the weight matrix;

[0054] The model compression module is configured to compress the large language model according to the quantized weights.

[0055] Embodiment 3, this embodiment provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model compression method based on low-rank quantization described in Embodiment 1.

[0056] Embodiment 4. This embodiment provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the large language model compression method based on low-rank quantization described in Embodiment 1.

[0057] The electronic device may include: a processor, a memory and a communication unit. These components communicate via one or more buses. Those skilled in the art will appreciate that the structure of the electronic device does not limit the embodiments of the present invention, and it may be a bus structure or a star structure, or a combination of certain components, or different component arrangements.

[0058] The communication unit is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0059] The processor is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0060] The memory is used to store the execution instructions of the processor. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0061] When the execution instructions in the memory are executed by the processor, the electronic device is enabled to execute part or all of the steps of Embodiment 1.

[0062] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A large language model compression method based on low-rank quantization, characterized in that: include: Load the large language model to be compressed and obtain calibration data. The calibration data production process includes: downloading the open source LCCC-base dataset, which is a Chinese conversation dataset; then sampling and word segmentation conversion processing of the conversation text in the dataset; The model infers the calibration data and saves the activation values ​​of each linear layer of the model as the calibration matrix X; a hook function is defined in the code to capture the activation values ​​before and after all linear layers in the loaded model; then forward inference is performed on the model using the prepared calibration dataset, and the hook function automatically saves the activation values ​​of the linear layer input into a set, the calibration matrix X; for the first layer of the model, X contains the input embedding from the calibration dataset, and for the middle layer, X contains the input activation of the current layer, which is the output activation of the previous layer; the Hessian matrix H is calculated for later use, formula: , where m is the batch size of the calibration matrix X; Quantize the weights of the large language model, replace the full-precision linear layer weight W of the large language model with the sum of the quantized linear layer weight Q and the product of the quantized low-rank factors L and R with a rank of k, to achieve quantization and low-rank approximation of the weight matrix; specifically: quantize each linear layer matrix in the large language model column by column, and use MinMax quantization as the quantization algorithm, as expressed by the following formula: in, and Respectively represent the full-precision vector before and after quantization of each column weight, s is the scaling factor, z is the zero point value; let N be the number of quantization bits, for symmetric quantization: , ; For asymmetric quantization: , ;Finally, a quantized model weight Q is obtained; For each linear layer of the large language model, a residual matrix A=WQ is calculated, where W is the linear layer matrix of each layer of the loaded model weights, and Q is the linear layer matrix of the corresponding quantized model obtained; the right factor matrix R is initialized using the singular value decomposition of the calibration matrix X, and the residual matrix A=WQ is calculated by minimizing To obtain the left factor matrix L; through the obtained L, minimize To obtain the right factor matrix R; update L and R alternately multiple times, and take L and R with the smallest norm error; Use quantized weights to compress large language models; The obtained L and R are quantified separately by MinMax algorithm; Perform forward inference on the original linear layer of the model Operation, replace with ; The initial value of the right factor matrix R is obtained by the formula The solution is calculated, where is the k-dimensional identity matrix, the singular value decomposition formula , ; The right factor matrix R is given by the formula Perform calculations to obtain the solution; The left factor matrix L is given by the formula Perform calculations to obtain the solution.

2. A large language model compression system based on low-rank quantization, characterized in that: include: The data preparation module is configured to: obtain calibration data and load the large language model to be compressed. The calibration data production process includes: downloading the open source LCCC-base dataset, which is a Chinese conversation dataset; then sampling and word segmentation conversion processing of the conversation text in the dataset; model inference calibration data and saving the activation values ​​of each linear layer of the model as the calibration matrix X; define a hook function in the code to capture the activation values ​​before and after all linear layers in the loaded model; then use the prepared calibration dataset to perform forward inference on the model, and the hook function will automatically save the activation values ​​of the linear layer input into a set, the calibration matrix X; for the first layer of the model, X contains the input embedding from the calibration dataset, and for the middle layer, X contains the input activation of the current layer, that is, the output activation of the previous layer; calculate the Hessian matrix H for standby, formula: , where m is the batch size of the calibration matrix X; The model processing module is configured to: quantize the weight of the large language model, replace the full-precision linear layer weight W of the large language model with the sum of the quantized linear layer weight Q and the product of the quantized low-rank factors L and R, and realize the quantization and low-rank approximation of the weight matrix. Specifically, each linear layer matrix in the large language model is quantized column by column, and the quantization algorithm uses MinMax quantization, which is expressed by the following formula: in, and Respectively represent the full-precision vector before and after quantization of each column weight, s is the scaling factor, z is the zero point value; let N be the number of quantization bits, for symmetric quantization: , ; For asymmetric quantization: , ;Finally, a quantized model weight Q is obtained; For each linear layer of the large language model, a residual matrix A=WQ is calculated, where W is the linear layer matrix of each layer of the loaded model weights, and Q is the linear layer matrix of the corresponding quantized model obtained; the right factor matrix R is initialized using the singular value decomposition of the calibration matrix X, and the residual matrix A=WQ is calculated by minimizing To obtain the left factor matrix L; through the obtained L, minimize To obtain the right factor matrix R; update L and R alternately multiple times, and take L and R with the smallest norm error; The model compression module is configured to: compress the large language model according to the quantized weights, and quantize the obtained L and R respectively through the MinMax algorithm; Perform forward inference on the original linear layer of the model Operation, replace with ; The initial value of the right factor matrix R is obtained by the formula The solution is calculated, where is the k-dimensional identity matrix, the singular value decomposition formula , ; The right factor matrix R is given by the formula Perform calculations to obtain the solution; The left factor matrix L is given by the formula Perform calculations to obtain the solution.

3. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model compression method based on low-rank quantization described in claim 1.

4. An electronic device, characterized in that: It includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the large language model compression method based on low-rank quantization as described in claim 1.

Images