Fine-tuning-free large language model weight matrix decomposition and compression method, system and application

Abstract

The invention discloses a large language model weight matrix compression decomposition method without fine tuning, and the method comprises the following steps: 1, carrying out the decomposition sensitive value detection of a to-be-decomposed large language model weight matrix, and determining one or more decomposition sensitive values; step 2, performing low-rank decomposition on the weight matrix, and reserving the decomposition sensitive value in the decomposition process to obtain one or more decomposed low-rank matrixes; and 3, reconstructing an original weight matrix by using the decomposition sensitive value and the low-rank matrix stored in the step 2. The invention further discloses a system for realizing the method and application of the method or the system in deploying a complex deep learning model in a resource-limited environment, and the method and the system have wide application scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language models, and relates to a large language model weight matrix decomposition compression method, system and application that do not require fine-tuning. Background Art

[0002] Pretrained Large Language Model (LLM) has become the hottest research topic in the field of artificial intelligence in the past two years. It is trained on massive amounts of text data and can perform well on a wide range of downstream language tasks with little or no fine-tuning. Large language models are generally composed of many transformer models, and their powerful capabilities mainly rely on their large parameter scale (usually billions of parameters). In order to speed up the training and reasoning of large language models and facilitate their deployment on resource-constrained devices, large language models are generally compressed. Common compression methods include matrix factorization, quantization, distillation, etc.

[0003] Among them, matrix decomposition refers to decomposing the weight matrix of the model and using two or more smaller matrices to approximate the original weight matrix to save storage space and compress the model. Matrix decomposition is mainly based on the assumption that the model weight matrix has a certain low rank, so it can be approximated by a low rank matrix. In the process of low rank decomposition, the general matrix decomposition method may cause some weights that have a greater impact on the inference results to be decomposed, which will lead to a significant decline in model performance. After the model is rebuilt, fine-tuning or even retraining is required to recover some of the losses caused by the decomposition. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a large language model weight matrix decomposition compression method, system and application that does not require fine-tuning after reconstruction. The method of the present invention proposes a new matrix low-rank decomposition framework, which can achieve a high compression rate of matrix low-rank decomposition by detecting decomposition sensitive values ​​in the original weight matrix and saving and reloading the detected decomposition sensitive values. The compressed model does not need fine-tuning and is suitable for compression of large language models; the decomposition sensitive value refers to the weight that has a greater impact on the functional performance of the model.

[0005] Specifically,

[0006] The present invention proposes a large language model weight matrix compression decomposition method without fine-tuning, the method comprising the following steps:

[0007] Step 1: Perform decomposition sensitive value detection on the large language model weight matrix to be decomposed to determine one or more decomposition sensitive values;

[0008] Step 2: performing low-rank decomposition on the weight matrix, and retaining the decomposition sensitive value during the decomposition process to obtain one or more decomposed low-rank matrices;

[0009] Step 3: Reconstruct the original weight matrix using the decomposed sensitive values ​​and the low-rank matrix saved in step 2.

[0010] In step 1, the decomposition sensitivity value is defined by a weighted absolute value method and / or an element-by-element relative error method;

[0011] The weight absolute value method calculates the sensitivity of the matrix elements in the weight matrix to be decomposed respectively, and presets the sensitivity threshold to perform decomposition sensitivity screening;

[0012] For the weight matrix to be decomposed W = (w ij ) m×n , the sensitivity of each element decomposition in the weight matrix G ij = abs(w ij );

[0013] The preset sensitivity threshold is t. If the sensitivity of the matrix element G ij >t, then the sensitivity G ij The corresponding matrix element w ij is the decomposition sensitivity value; retain all the decomposition sensitivity values, and record the decomposition sensitivity value w ij The coordinate c on the weight matrix is ​​obtained by decomposing the sensitive value set S = {s i =(w ij , c), i = 1, ..., N};

[0014] The element-by-element relative error method pre-decomposes and reconstructs the weight matrix to be decomposed, defines the relative error between the elements before and after the reconstruction as the sensitivity, and presets the sensitivity threshold to perform decomposition sensitivity screening;

[0015] The weight matrix is ​​pre-decomposed and reconstructed to obtain an approximate matrix of the original weight matrix, and the original weight matrix W is calculated as follows: ij ) m×n and the approximate matrix Each element w ij The relative error is taken as the sensitivity of the matrix elements and is expressed as follows:

[0016]

[0017] The preset sensitivity threshold is t. If the sensitivity of the matrix element G ij >t, then the sensitivity G ij The corresponding matrix element w ij is the decomposition sensitivity value; retain all the decomposition sensitivity values, and record the decomposition sensitivity value w ij The coordinate c on the weight matrix is ​​obtained by decomposing the sensitive value set S = {s i =(w ij , c), i=1,…,N}.

[0018] In step 2, the low-rank decomposition method includes singular value decomposition, non-negative matrix decomposition, block matrix decomposition, tensor decomposition and other methods, which can reduce storage and computing costs by simplifying data representation while retaining important information as much as possible;

[0019] In step three, the decomposed small matrix obtained in step two is reconstructed, and the decomposed sensitive value obtained in step one is reloaded into the reconstructed weight matrix according to the position in the original weight matrix; the loading means replacing the approximate value of the corresponding position in the reconstructed weight matrix with the decomposed sensitive value.

[0020] The present invention also proposes a system for implementing the above method, the system comprising: a data preprocessing module, a decomposition sensitive value detection module, a weight matrix low-rank decomposition module, a decomposition sensitive value reloading module, and a model reconstruction module;

[0021] The data preprocessing module is used to receive and prepare a weight matrix to be decomposed;

[0022] The decomposition sensitivity value detection module is used to detect and screen the decomposition sensitivity value based on the weight absolute value method and / or the element-by-element relative error method, and save the decomposition sensitivity value;

[0023] The weight matrix low-rank decomposition module uses a low-rank decomposition method including singular value decomposition to perform low-rank decomposition on the weight matrix;

[0024] The decomposition sensitive value reloading module is used to reload the stored decomposition sensitive value into the reconstructed approximate matrix;

[0025] The model reconstruction module is used to complete the final reconstruction of the model using the decomposed low-rank matrix and the overloaded sensitive values.

[0026] The present invention also provides applications of the above method or system in deploying complex deep learning models in a resource-constrained environment, etc.

[0027] Specifically, the specific scenarios in which the method or system of the present invention can be applied include but are not limited to the following:

[0028] 1) Mobile and embedded devices: smartphones, tablets, IoT devices, etc. These devices have limited processing power and storage space, but need to run advanced AI applications such as speech recognition and real-time translation. By compressing the size of the model, large language models can be run on these devices without connecting to a cloud server.

[0029] 2) Edge computing: In an edge computing environment, data is processed on-site at the point where it is generated, which can reduce data transmission time and improve response speed. The compressed model can be run directly on the edge device, reducing dependence on the central server and improving the speed and efficiency of data processing.

[0030] 3) Cloud service optimization: Cloud platforms provide a variety of machine learning and AI services, but resource allocation and computing cost control are key considerations; using compressed models can reduce cloud platform resource consumption (such as CPU and memory usage), reduce operating costs, and allow more customers to use services simultaneously;

[0031] 4) Real-time applications: applications that require fast response, such as real-time speech translation and video content analysis; the compressed model has a faster inference speed and can meet the needs of real-time processing;

[0032] 5) Data Center: Large-scale data centers need to process huge amounts of data and complex computing tasks. Through model compression, more service instances can be deployed with the same hardware resources, improving the service capabilities and economic benefits of the data center.

[0033] The present invention also provides a hardware system for implementing the above method, the hardware system comprising: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the above method is implemented.

[0034] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0035] The beneficial effects of the present invention include:

[0036] The method of the present invention effectively reduces the number of parameters of the model through low-rank decomposition, achieves a high compression rate, helps to reduce the model's demand for storage space, and reduces the computing resources required when executing the model; the present invention can maintain or only slightly affect the original performance of the model after model compression by identifying and reloading decomposition sensitive values, thereby eliminating the tedious fine-tuning process that is often required after traditional model compression; compared with other compression technologies, the method of the present invention better maintains the performance of the model by accurately retaining key weights (decomposition sensitive values), ensuring that the compressed model can still effectively perform model tasks. The use of the method and system of the present invention is not limited to a specific type of large language model, but can be widely used in various large deep learning models that need to be compressed, including but not limited to NLP, image recognition and speech processing models; the compressed model can be more easily deployed on resource-constrained devices, such as mobile devices and edge computing devices, thereby reducing deployment costs and improving the accessibility and practicality of the model; in addition, while compressing the model, the computational burden is also reduced, thereby accelerating the reasoning speed of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 It is a flow chart of the process of implementing the method of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0040] In view of the problem that the existing low-rank decomposition methods of model matrices generally require fine-tuning or retraining to make the model reach the performance before compression; the present invention proposes a new large language model weight matrix decomposition compression method that does not require fine-tuning, and realizes a high compression rate decomposition scheme that does not require fine-tuning by detecting special values ​​that are sensitive to decomposition. The present invention also provides a system for implementing the above method, and the application of the above method or system in deploying complex deep learning models in resource-constrained environments, which has a wide range of application scenarios.

[0041] The present invention provides a large language model weight matrix compression decomposition method without fine-tuning, the method comprising the following steps:

[0042] Step 1: Perform decomposition sensitive value detection on the large language model weight matrix to be decomposed to determine one or more decomposition sensitive values;

[0043] Step 2: performing low-rank decomposition on the weight matrix, and retaining the decomposition sensitive value during the decomposition process to obtain one or more decomposed low-rank matrices;

[0044] Step 3: Reconstruct the original weight matrix using the decomposed sensitive values ​​and the low-rank matrix saved in step 2.

[0045] Before the subsequent low-rank decomposition of the matrix, the weight matrix to be decomposed will first be subjected to a sensitive value detection. The so-called decomposition sensitive value refers to a special value that is sensitive to the low-rank decomposition and reconstruction of the matrix; by detecting the decomposition sensitive value, one or more decomposition sensitive values ​​are determined to facilitate the subsequent reconstruction of the matrix. According to different definitions of sensitivity, the decomposition sensitive values ​​in the matrix are different, and the detection method will also be different. It should be pointed out that the method of the present invention mainly proposes a new decomposition framework, and therefore does not limit the specific definition and way and method of detecting sensitive values. As long as the framework of the method of the present invention is adopted, it should belong to the protection scope of the method of the present invention.

[0046] Specifically, in step 1, the decomposition sensitivity value may be defined by a weighted absolute value method and / or an element-by-element relative error method;

[0047] The weight absolute value method calculates the sensitivity of the matrix elements in the weight matrix to be decomposed respectively, and presets the sensitivity threshold to perform decomposition sensitivity screening;

[0048] It is generally believed that weights with larger absolute values ​​will have a greater impact on the output. Therefore, it can be simply assumed that the absolute value of the weight can define its sensitivity to decomposition: the larger the absolute value, the higher its sensitivity. ij ) m×n , the sensitivity of each element decomposition in the weight matrix G ij = abs(w ij );

[0049] The preset sensitivity threshold is t. If the sensitivity of the matrix element G ij >t, then the sensitivity G ij The corresponding matrix element w ij is the decomposition sensitivity value; retain all the decomposition sensitivity values, and record the decomposition sensitivity value w ij The coordinate c on the weight matrix is ​​obtained by decomposing the sensitive value set S = {s i =(w ij,c), i=1,…,N}, for subsequent reconstruction and reloading;

[0050] The element-by-element relative error method pre-decomposes and reconstructs the weight matrix to be decomposed, defines the relative error between the elements before and after the reconstruction as the sensitivity, and presets the sensitivity threshold to perform decomposition sensitivity screening;

[0051] In a specific implementation, after the decomposition method is selected, the sensitivity to decomposition can be defined by calculating the relative error of each element in the matrix before and after decomposition and reconstruction. Specifically, the weight matrix is ​​pre-decomposed and reconstructed again to obtain an approximate matrix of the original weight matrix, and the original weight matrix W = (w ij ) m×n and the approximate matrix Each element w ij The relative error of

[0052] The relative error is used as the weight w ij Decomposition sensitivity G ij , the formula is as follows:

[0053]

[0054] The larger the relative error, the greater the difference between the weight value reconstructed using the decomposition method and the original value, and the more sensitive the weight is to the decomposition.

[0055] The preset sensitivity threshold is t. If the sensitivity of the matrix element G ij >t, then the sensitivity G ij The corresponding matrix element w ij is the decomposition sensitive value; retain all the decomposition sensitive values, and record the coordinate c of the decomposition sensitive value wij on the weight matrix, and obtain the decomposition sensitive value set S = {s i =(w ij ,c),i=1,…,N}, for subsequent reconstruction and reloading.

[0056] In step 2 of the present invention, the specific method of matrix decomposition is not limited, and any framework that adopts this method should belong to the protection scope of this method; the low-rank decomposition methods that can be used in the present invention include singular value decomposition, non-negative matrix decomposition, block matrix decomposition, tensor decomposition and other methods, which reduce storage and computing costs by simplifying data representation, while retaining important information as much as possible;

[0057] In a specific implementation, taking singular value decomposition as an example, the process of matrix low-rank decomposition and reconstruction is briefly described:

[0058] 1. Let the original weight matrix be right Perform singular value decomposition and get the matrix and Where k = min(m,n), Σ is a diagonal matrix, and the diagonal elements are the singular values ​​arranged from large to small.

[0059] 2. Assume that the number of singular values ​​retained by low-rank reconstruction is r, that is, retain the largest first r singular values This is equivalent to taking the first r rows and r columns of Σ to obtain Then take the first r columns of U to get Take the first r rows of V and we get

[0060] 3. The reconstructed matrix

[0061] 4. Set You can save the matrix and Instead of saving W, it can save storage space and achieve model compression.

[0062] Before decomposition, saving the original matrix W requires storing m×n parameters, while saving and The number of parameters required is r(m+n), and generally r is relatively small, so storage space can be saved. The compression ratio before and after reconstruction can be defined as m×n / r(m+n).

[0063] Generally, in order to evaluate the reconstruction loss, the relative error is used to calculate the matrix before and after reconstruction. The calculation formula is as follows:

[0064]

[0065] Among them, W and is the original matrix and the reconstructed matrix, ||·|| F is the Frobenius norm of the matrix.

[0066] After decomposition, the small matrix obtained by saving the decomposition can be and And decompose the sensitive value set S instead of saving the original matrix W.

[0067] In step three, the decomposed small matrix obtained in step two is reconstructed, and the decomposed sensitive value obtained in step one is reloaded into the reconstructed weight matrix according to the position in the original weight matrix; the loading means replacing the approximate value of the corresponding position in the reconstructed weight matrix with the decomposed sensitive value.

[0068] When training or inference is required, the saved decomposition matrix needs to be reconstructed. Generally, multiple decomposed small matrices are multiplied to obtain the reconstructed matrix. Next, the weight value s in the decomposition sensitive value set S is i Reload it into the reconstruction matrix according to its coordinate c The final reconstruction result is obtained

[0069] Example

[0070] Take the Llama-7B model as an example. The model consists of 32 transformer layers, each of which contains two modules: multi-head self-attention and feedforward network. The multi-head self-attention module contains 4 linear layers, and the feedforward network module contains 3 linear layers. The matrix decomposition of Llama-7B is mainly to decompose the weights of the linear layers mentioned above (a total of 32*(3+4)=224).

[0071] Taking the relative error method to define the decomposition sensitivity value as an example, the preset sensitivity threshold is t llama (For example, t llama =0.6), detect the decomposition sensitivity value according to the above step 1 and save it as S llama . Step 2 is to perform matrix decomposition, for example, singular value decomposition is used to compress the weight matrix by 4 times, that is, the compression rate is m×n / r(m+n)=4. Then for the 4 linear layers (m=n=4096) of the multi-head self-attention module of Llama-7B, its r=512, and for the 3 linear layers (m=4096,n=11008 or m=11008,n=4096) of the feedforward network module, its r=746. Save the small matrix after decomposition and During inference, step three is executed to reconstruct the decomposed small matrix obtained in step two, and the decomposed sensitive values ​​obtained in step one are reloaded into the reconstructed weight matrix according to their positions in the original weight matrix.

[0072] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0073] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0074] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0075] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the module, program segment or instruction part contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0076] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A method for compressing and decomposing a large language model weight matrix without fine-tuning, characterized in that: The method comprises the following steps: Step 1: Perform decomposition sensitive value detection on the large language model weight matrix to be decomposed to determine one or more decomposition sensitive values; Step 2: performing low-rank decomposition on the weight matrix, and retaining the decomposition sensitive value during the decomposition process to obtain one or more decomposed low-rank matrices; Step 3: Reconstruct the original weight matrix using the decomposed sensitive values ​​and the low-rank matrix saved in step 2.

2. The method according to claim 1, characterized in that In step 1, the decomposition sensitivity value is defined by a weighted absolute value method and / or an element-by-element relative error method; The weight absolute value method calculates the sensitivity of the matrix elements in the weight matrix to be decomposed respectively, and presets the sensitivity threshold to perform decomposition sensitivity screening; The element-by-element relative error method pre-decomposes and reconstructs the weight matrix to be decomposed, defines the relative error between the elements before and after the reconstruction as the sensitivity, and presets the sensitivity threshold to perform decomposition sensitivity screening.

3. The method according to claim 2, characterized in that In the weight absolute value method, for the weight matrix to be decomposed W=(w ij ) m×n , the sensitivity of each element decomposition in the weight matrix G ij = abs(w ij ); The preset sensitivity threshold is t. If the sensitivity of the matrix element G ii >t, then the sensitivity G ij The corresponding matrix element w ij To decompose sensitive values; Keep all the decomposition sensitive values ​​and record the decomposition sensitive value w ij The coordinate c on the weight matrix is ​​obtained by decomposing the sensitive value set S = {s i =(w ij , c), i=1,…,N}.

4. The method according to claim 2, characterized in that In the element-by-element relative error method, the weight matrix is ​​pre-decomposed and reconstructed to obtain an approximate matrix of the original weight matrix, and the original weight matrix W is calculated as (w ij ) m×n and the approximate matrix Each element w ij The relative error is taken as the sensitivity of the matrix elements and is expressed as follows: The preset sensitivity threshold is t. If the sensitivity of the matrix element G ij >t, then the sensitivity G ij The corresponding matrix element w ij To decompose sensitive values; Keep all the decomposition sensitive values ​​and record the decomposition sensitive value w ij The coordinate c on the weight matrix is ​​obtained by decomposing the sensitive value set S = {s i =(w ij , c), i=1,…,N}.

5. The method according to claim 1, characterized in that In step 2, the low-rank decomposition method includes a singular value decomposition method, which reduces storage and computing costs by simplifying data representation while retaining important information as much as possible.

6. The method according to claim 1, characterized in that In step three, the decomposed small matrix obtained in step two is reconstructed, and the decomposed sensitive value obtained in step one is reloaded into the reconstructed weight matrix according to the position in the original weight matrix; the loading means replacing the approximate value of the corresponding position in the reconstructed weight matrix with the decomposed sensitive value.

7. A system for implementing the method according to any one of claims 1 to 6, characterized in that: The system comprises: a data preprocessing module, a decomposition sensitive value detection module, a weight matrix low-rank decomposition module, a decomposition sensitive value reloading module, and a model reconstruction module; The data preprocessing module is used to receive and prepare a weight matrix to be decomposed; The decomposition sensitivity value detection module is used to detect and screen the decomposition sensitivity value based on the weight absolute value method and / or the element-by-element relative error method, and save the decomposition sensitivity value; The weight matrix low-rank decomposition module uses a low-rank decomposition method including singular value decomposition to perform low-rank decomposition on the weight matrix; The decomposition sensitive value reloading module is used to reload the stored decomposition sensitive value into the reconstructed approximate matrix; The model reconstruction module is used to complete the final reconstruction of the model using the decomposed low-rank matrix and the overloaded sensitive values.

8. Application of the method as described in any one of claims 1-6, or the system as described in claim 7, in deploying complex deep learning models in a resource-constrained environment.

9. A hardware system for implementing the method according to any one of claims 1 to 6, characterized in that: The hardware system comprises: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images