Method and device for compressing large language model through singular value decomposition based on gradient attribution

Abstract

The invention relates to the technical field of large model compression, and discloses a method, a device, equipment and a medium for compressing a large language model through singular value decomposition based on gradient attribution, and the method comprises the steps: calculating the gradient of an original weight matrix of each linear layer on an auxiliary calibration data set, during singular value decomposition of the original weight matrix, sparseness is distributed according to the gradient of the weight of each layer; collecting the gradient of each singular value when the original weight matrix passes through singular value decomposition on the auxiliary calibration data set; using an approximate loss item to represent the importance of each singular value when the original weight matrix passes through singular value decomposition, and filtering unimportant singular values according to a preset dynamic threshold value; and setting the filtered singular value as an index of the original weight matrix when the original weight matrix passes through singular value decomposition, and performing low-rank decomposition on the singular value related to the index in the original weight matrix layer by layer according to a gradient descent method of each singular value until the compression ratio of the large language model is met. And the compression ratio of the model can be obviously improved. The method is applied to application scenes such as financial science and technology large model compression.

Description

Technical Field

[0001] The present invention relates to the technical field of large model compression, and in particular to a method, device, equipment and medium for compressing a large language model through singular value decomposition based on gradient attribution. Background Art

[0002] The techniques for compressing large language models can be mainly divided into the following categories: pruning, quantization, distillation, and low-rank approximation. Among them, unstructured pruning and quantization can reduce the number of parameters or memory requirements by half or more, and will not significantly reduce model performance, but they require dedicated GPU cores to fully realize the acceleration potential. In contrast, structured pruning can generate lightweight models that do not rely on dedicated hardware, but its performance is still significantly behind the original model. The low-rank decomposition method decomposes the original weight matrix into the product of two low-rank matrices, discarding unimportant information during the decomposition process, effectively reducing the number of parameters while accelerating reasoning, and can be combined with other model compression methods to further compress the model size. However, there is currently little exploration of low-rank compression of large language models, and it is a potential direction for exploration. Due to the high-rank nature of the large model weights and the phenomenon of extreme outliers in the output activation values, the naive low-rank compression method based on weight singular value decomposition will produce large reconstruction errors, resulting in a sharp decline in the performance of the large model; singular value decomposition is an activation value-aware singular value decomposition method for large models, which takes into account the phenomenon of extreme outliers in the output activation values ​​of large models, but the screened singular values ​​do not establish a direct mapping relationship with the approximate loss caused by the low-rank compressed output activation values, that is, the existence of the main singular value direction will cause greater performance loss than the eliminated singular value direction. Summary of the invention

[0003] The present invention provides a method, device, computer equipment and medium for compressing a large language model based on singular value decomposition of gradient attribution, so as to solve the technical problem that the screened singular values ​​do not establish a direct mapping relationship with the approximate loss caused by the low-rank compressed output activation value, that is, the main singular value direction will cause greater performance loss than the eliminated singular value direction.

[0004] In a first aspect, a method for compressing a large language model based on singular value decomposition with gradient attribution is provided, comprising:

[0005] Calculate the gradient of the original weight matrix of each linear layer on the auxiliary calibration data set, wherein the original weight matrix is ​​assigned sparsity according to the gradient size of the weight of each layer when passing through singular value decomposition;

[0006] Collecting the gradients of each singular value of the original weight matrix on the auxiliary calibration data set when the original weight matrix is ​​decomposed by singular value, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term at the singular value 0 point;

[0007] Using the approximate loss term to represent the importance of each singular value of the original weight matrix during singular value decomposition, filtering unimportant singular values ​​according to a pre-set dynamic threshold;

[0008] The filtered singular values ​​are set as the indices of the original weight matrix during singular value decomposition, and the singular values ​​associated with the indexes in the original weight matrix are low-rank decomposed layer by layer according to the gradient descent method of each singular value until the compression ratio of the large language model is met.

[0009] In a second aspect, a device for compressing a large language model by singular value decomposition based on gradient attribution is provided, comprising:

[0010] A calculation module, used to calculate the gradient of the original weight matrix of each linear layer on the auxiliary calibration data set, wherein the original weight matrix is ​​allocated sparsity according to the gradient size of each layer weight when passing through singular value decomposition;

[0011] A collecting module, used for collecting the gradients of each singular value of the original weight matrix when the original weight matrix is ​​decomposed by singular value on the auxiliary calibration data set, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term at the singular value 0 point;

[0012] A filtering module, used to use the approximate loss term to represent the importance of each singular value of the original weight matrix when it is decomposed by singular value, and filter out unimportant singular values ​​according to a preset dynamic threshold;

[0013] The iterative module is used to set the filtered singular values ​​as the indices of the original weight matrix during singular value decomposition, and perform low-rank decomposition layer by layer on the singular values ​​related to the indexes in the original weight matrix according to the gradient descent method of each singular value until the compression ratio of the large language model is met.

[0014] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for compressing a large language model by singular value decomposition based on gradient attribution are implemented.

[0015] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the method for compressing a large language model based on singular value decomposition based on gradient attribution are implemented.

[0016] In the scheme implemented by the above-mentioned method, device, computer equipment and storage medium for compressing a large language model based on gradient attribution singular value decomposition, the gradient of the original weight matrix of each linear layer can be calculated on an auxiliary calibration data set, and the original weight matrix is ​​assigned sparsity according to the gradient size of the weight of each layer when passing through the singular value decomposition; the gradient of each singular value of the original weight matrix when passing through the singular value decomposition is collected on the auxiliary calibration data set, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term when the singular value is 0 point; the approximate loss term is used to represent the importance of each singular value of the original weight matrix when passing through the singular value decomposition, and unimportant singular values ​​are filtered according to a pre-set dynamic threshold; the filtered singular value is set as the index of the original weight matrix when passing through the singular value decomposition, and the singular values ​​related to the index in the original weight matrix are low-rank decomposed layer by layer according to the gradient descent method of each singular value until the compression ratio of the large language model is met. By filtering out the singular value directions that are important for the approximate loss term through gradient information, the redundant retention of non-critical directions in the traditional singular value decomposition method is avoided. This method does not rely on specific hardware cores and can be efficiently deployed on general devices. At the same time, by minimizing the retention of non-critical singular values ​​and without significantly losing model performance, the compression ratio of the model can be significantly improved, further improving the inference speed and reducing video memory usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 1 is a schematic diagram of an application environment of a method for compressing a large language model based on singular value decomposition with gradient attribution in one embodiment of the present invention;

[0019] Figure 2 It is a flow chart of a method for compressing a large language model based on singular value decomposition with gradient attribution in one embodiment of the present invention;

[0020] Figure 3 It is a structural schematic diagram of a device for compressing a large language model based on singular value decomposition of gradient attribution in one embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0022] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The method for compressing a large language model based on singular value decomposition with gradient attribution provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the client communicates with the server through a network. The server can calculate the gradient of the original weight matrix of each linear layer on the auxiliary calibration data set, and the original weight matrix allocates sparsity according to the gradient size of the weight of each layer when passing through the singular value decomposition; collect the gradients of each singular value of the original weight matrix when passing through the singular value decomposition on the auxiliary calibration data set, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term when the singular value is 0 point; use the approximate loss term to represent the importance of each singular value of the original weight matrix when passing through the singular value decomposition, and filter out unimportant singular values ​​according to a pre-set dynamic threshold; set the filtered singular value as the index of the original weight matrix when passing through the singular value decomposition, and perform low-rank decomposition layer by layer on the singular values ​​related to the index in the original weight matrix according to the gradient descent method of each singular value until the compression ratio of the large language model is met. By filtering out the singular value directions that are important for the approximate loss term through gradient information, the redundant retention of non-critical directions in the traditional singular value decomposition method is avoided. This method does not rely on specific hardware cores and can be efficiently deployed on general devices. At the same time, by minimizing the retention of non-critical singular values ​​and without significantly losing model performance, the compression ratio of the model can be significantly improved, further improving the inference speed and reducing video memory usage.

[0025] The present invention is applied to application scenarios such as large-scale model compression in financial technology. It uses gradient information to screen out singular value directions that are important for the approximate loss term, thereby avoiding redundant retention of non-critical directions in traditional singular value decomposition methods. This method does not rely on specific hardware cores and can be efficiently deployed on general-purpose devices. At the same time, by minimizing the retention of non-critical singular values ​​and without significantly losing model performance, it can significantly improve the compression ratio of the model, further improve the inference speed and reduce video memory usage.

[0026] The present invention is described in detail below through specific embodiments.

[0027] See also Figure 2 As shown, Figure 2A flow chart of a method for compressing a large language model by singular value decomposition based on gradient attribution provided by an embodiment of the present invention includes the following steps:

[0028] S10: Calculate the gradient of the original weight matrix of each linear layer on the auxiliary calibration data set, and allocate sparsity according to the gradient size of the weight of each layer when the original weight matrix is ​​decomposed by singular value; when the original weight matrix with a large gradient has low sparsity when performing singular value decomposition, retain the rank that accounts for a proportion greater than a preset value.

[0029] Large language models are built on the basis of massive data (corpus) training, and their parameters often number in the billions or even trillions.

[0030] Deep learning: the cornerstone of large language models. Large language models are based on deep learning technology and are deep learning models trained using large amounts of text data. Deep learning is a branch of machine learning that simulates the learning process of the human brain by building deep neural networks. Through deep learning, large language models can automatically learn the structure and rules of language from massive amounts of text data, thereby achieving accurate understanding and generation of natural language.

[0031] Transformer architecture: the core of large language models. Transformer is a deep learning model architecture, and its core is the self-attention mechanism. In the computer field, it is called "parallel" processing capability. The so-called parallelism means that there are multiple lanes on the road and vehicles can drive in parallel. Traditional recurrent neural networks (RNN) and convolutional neural networks (CNN) are "serial", and serial has only one lane. Therefore, Transformer can improve training efficiency.

[0032] Singular value decomposition is commonly used for image compression, and in this embodiment, singular value decomposition is used for compression of a large language model. The original image matrix is ​​decomposed into the product of three matrices using singular value decomposition, and then the original image is approximately reconstructed by retaining larger singular values, thereby achieving image compression. For an image, it is represented as a matrix, in which each element represents the pixel value of the image. The image matrix is ​​decomposed into the product of three matrices using singular value decomposition, and then the original image is approximately reconstructed by truncating the singular values. Retain larger singular values ​​and reduce smaller singular values, thereby reducing storage space and achieving image compression. The compression ratio refers to the ratio between the size of the compressed image and the size of the original image. Different compression ratios can be selected according to application requirements and image quality requirements.

[0033] Singular value decomposition decomposes a matrix into the product of three matrices. For an m×n matrix A, its singular value decomposition is expressed as A=UΣV^T, where U is an m×r unitary matrix, Σ is an r×r diagonal matrix, V is an n×r unitary matrix, and r is the rank of matrix A. The elements on the diagonal of Σ are called singular values ​​and are arranged in order from large to small. The mathematical principle of singular value decomposition is based on the concepts of eigenvalues ​​and eigenvectors of a matrix. The singular value decomposition is obtained by eigendecomposing the matrix.

[0034] Data dimensionality reduction: Singular value decomposition can reduce the dimension of a high-dimensional matrix to a low-dimensional space. By retaining larger singular values ​​and corresponding eigenvectors, the important information of the original data can be better preserved, while reducing the data dimension and simplifying calculation and storage. This is very important for processing large-scale data and improving algorithm efficiency.

[0035] Image compression: Singular value decomposition is also widely used in image compression. After performing singular value decomposition on the image matrix, you can choose to retain fewer singular values ​​and corresponding eigenvectors, and then reconstruct the matrix to achieve image compression. This can reduce the size of the image file to a certain extent while maintaining a good visual effect.

[0036] Recommendation system: Singular value decomposition is also widely used in recommendation systems. By performing singular value decomposition on the user-item rating matrix, low-dimensional representations of users and items can be obtained, thereby extracting user interests and item features. Recommendation algorithms based on singular value decomposition can make personalized recommendations based on user interests and item features.

[0037] Topic modeling: Singular value decomposition can also be used for topic modeling, that is, extracting topic information from a document collection. By performing singular value decomposition on the document-word matrix, a low-dimensional representation of documents and words can be obtained, thereby extracting the topic distribution of documents and the semantic information of words. This is very valuable for tasks such as text mining and information retrieval.

[0038] Therefore, in this embodiment, the gradient of the original weight matrix of each linear layer is calculated specifically as follows: weight features after singular value decomposition of the original weight matrix are collected on the auxiliary calibration data set; the weight features are transposed into a feature matrix; eigenvalues ​​are calculated based on the feature matrix, and singular values ​​are obtained according to the eigenvalues.

[0039] Singular value decomposition is specifically: extract the most important features in a matrix and use the gradient descent method to achieve decomposition. Singular value decomposition is to decompose an M*N matrix into M*k and k*N matrices, so that the multiplication of the two matrices will give the original M*N matrix. The gradient descent method calculates the average loss of all training samples each time the gradient descent is performed: the loss of a single sample is randomly selected to approximate the average loss term, and the method of decaying the learning rate is used. A larger learning rate is selected at the beginning, and the learning rate is reduced after the error curve enters the stable period for more precise adjustments. When the amount of training data is particularly large, it greatly speeds up the data calculation speed.

[0040] S20: collecting the gradients of each singular value of the original weight matrix when performing singular value decomposition on the auxiliary calibration data set, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term at the singular value 0 point;

[0041] An approximate loss term is used to represent the importance of each singular value of the original weight matrix when it is decomposed by singular value, and a preset number of singular values ​​that meet the required compression ratio are set.

[0042] Assuming A is an N*M matrix, then the resulting U is an N*N square matrix (the vectors in it are orthogonal, and the vectors in U are called left singular vectors), Σ is an N*M matrix (except for the elements on the diagonal, all elements are 0, and the elements on the diagonal are called singular values), V' (the transpose of V) is an N*N matrix, the vectors in it are also orthogonal, and the vectors in V are called right singular vectors), reflecting the size of several multiplied matrices.

[0043] A(N*M)=U(N*M)*Σ(N*M)*V'(N*N)

[0044] Transpose a matrix A*A, and you will get a square matrix. Use the square matrix to find the eigenvalue (A T A)v i =λ i v i

[0045] The obtained v is the right singular vector mentioned above. In addition, we can also get:

[0046]

[0047] σ is the singular value mentioned above, and u is the left singular vector mentioned above. The singular value σ is similar to the eigenvalue, and is also arranged from large to small in the matrix Σ, and σ decreases very quickly. In many cases, the sum of the first 10% or even 1% of the singular values ​​accounts for more than 99% of the sum of all the singular values. In other words, the first r large singular values ​​can be used to approximate the matrix. Here is a definition of partial singular value decomposition:

[0048]

[0049] r is a number much smaller than m and n, so the matrix multiplication is:

[0050]

[0051] The result of multiplying the three matrices on the right will be a matrix close to A. Here, the closer r is to n, the closer the result of the multiplication is to A. The sum of the areas of these three matrices (from a storage point of view, the smaller the matrix area, the smaller the storage volume) is much smaller than the original matrix A. If you want to compress the space to represent the original matrix A, save the three matrices here: U, Σ, and V. The physical meaning of singular value decomposition is decomposition - decomposing a complex transformation process into three simple basic transformations (rotation, scaling, and rotation). The size of the singular value represents the degree of scaling of the corresponding singular vector (which can be understood as a weight). The larger the singular value, the greater the influence of the corresponding singular vector on the final result. If the singular value matrix has several non-zero singular values, then the matrix A is a rank matrix.

[0052] S30: using the approximate loss term to represent the importance of each singular value of the original weight matrix during singular value decomposition, and filtering unimportant singular values ​​according to a preset dynamic threshold;

[0053] Use the approximate loss term to represent the importance of each singular value of the original weight matrix when it is decomposed by singular value. Filter unimportant singular values ​​according to the pre-set threshold, or select the top-k singular values ​​to meet the required compression ratio. By setting the dynamic threshold, the compression ratio can be flexibly adjusted according to the performance requirements of the specific task, while ensuring the task performance, to achieve the best compression effect. The dynamic threshold can be set according to the needs. Pre-set the dynamic threshold and adjust the compression ratio according to the performance requirements of the specific task.

[0054] The gradient attribution technique is used in this step. The gradient attribution measures the contribution of each input original weight matrix singular value to the output of the large language model by calculating the gradient of the large language model output to the input original weight matrix singular value. Specifically, for a given input sample (original weight matrix), the rate of change of the output of the large language model can be approximated by calculating the gradient of the input sample at each input original weight matrix singular value. Singular values ​​with large gradients are generally considered to be more important singular values ​​because they have a greater impact on changes in the output of the large language model, and the importance of each input singular value is evaluated by calculating the gradient of the large language model output to the input singular value. Gradient attribution makes the method of this embodiment more interpretable, and the importance of each singular value direction in the large language model to the final task is clearer, which helps enterprises optimize model deployment strategies.

[0055] S40: Setting the filtered singular value as the index of the original weight matrix during singular value decomposition, and performing low-rank decomposition layer by layer on the singular values ​​associated with the index in the original weight matrix according to the gradient descent method of each singular value until the compression ratio of the large language model is met.

[0056] Specifically, the filtered singular values ​​are set as the index of the original weight matrix when it is decomposed by singular value decomposition; the singular values ​​related to the index in the original weight matrix are decomposed into the product of two low-rank matrices, wherein the product of the two low-rank matrices constitutes a transposed weight matrix perpendicular to the original weight matrix. Among them, by obtaining the eigenvectors that appear in the cluster in the original weight matrix, and setting the eigenvectors to be retained; the filtered singular values ​​are used as the index of the original weight matrix when it is decomposed by singular value decomposition; when obtaining the index words in the index, the singular values ​​that match the index words are queried correspondingly, and the original weight matrix where the singular values ​​are located is obtained. The singular values ​​related to the index are decomposed layer by layer according to the gradient descent method of each singular value. The gradient descent method is to calculate the average loss of all singular values ​​related to the index each time the gradient descent is performed: the loss of a single singular value related to the index is randomly selected to approximate the average loss term, and the method of decaying the learning rate is adopted. A larger learning rate is selected at the beginning, and the learning rate is reduced after the error curve enters the stable period for more precise adjustment. When the amount of training data is particularly large, the data calculation speed is greatly accelerated.

[0057] In this embodiment, assuming that the filtered singular values ​​are indexed as indices in the original weight matrix through singular value decomposition, the original weight is decomposed into the product of two low-rank matrices

[0058]

[0059] Repeat steps S20-S40 to perform low-rank decomposition on the original weight matrix inside the large language model layer by layer until the corresponding compression ratio is met or the threshold of the minimum performance of the model is reached.

[0060] In another embodiment, each row in the first matrix U represents a class of words with related meanings, and each non-zero element thereof represents the importance (or relevance) of each word in this class of words, and the larger the value, the more relevant it is. Each column in the last matrix Σ represents a class of articles of the same topic, and each element thereof represents the relevance of each article in this class of articles. The middle matrix represents the relevance between the class words and the article. Therefore, as long as the association matrix A is subjected to a singular value decomposition, the classification of synonyms and the classification of articles can be completed at the same time. (At the same time, the relevance of each class of articles and each class of words is obtained). The left singular vector represents some characteristics of the word, and the right singular vector represents some characteristics of the document. The middle singular value matrix represents the important procedures of a row of the left singular vector and a column of the right singular vector, and the larger the number, the more important it is. The effect of obtaining the right singular vector and the clustering of the left singular vector can extract the synonyms in the document collection. When the user searches for a document, it uses the semantic level (synonym collection) to search, thereby reducing our search and storage volume, and the document collection that can be compressed can also improve the user experience. The user enters a word and can find it in the collection of synonyms of this word.

[0061] It can be seen that in the above scheme, for the iteration of intelligent risk control data, the gradient of the original weight matrix of each linear layer is first calculated on the auxiliary calibration data set, and the sparsity is allocated according to the gradient size of the weights of each layer when the original weight matrix is ​​singular value decomposed; the gradients of each singular value of the original weight matrix when passing through the singular value decomposition are collected on the auxiliary calibration data set, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term when the singular value is 0 point; the approximate loss term is used to represent the importance of each singular value of the original weight matrix when passing through the singular value decomposition, and unimportant singular values ​​are filtered according to a pre-set dynamic threshold; the filtered singular value is set as the index of the original weight matrix when passing through the singular value decomposition, and the singular values ​​related to the index in the original weight matrix are low-rank decomposed layer by layer according to the gradient descent method of each singular value until the compression ratio of the large language model is met. By filtering out the singular value directions that are important for the approximate loss term through gradient information, the redundant retention of non-critical directions in the traditional singular value decomposition method is avoided. This method does not rely on specific hardware cores and can be efficiently deployed on general devices. At the same time, by minimizing the retention of non-critical singular values, the compression ratio of the model can be significantly improved without significantly losing model performance, further improving the inference speed and reducing video memory usage.

[0062] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0063] In one embodiment, a device for compressing a large language model based on singular value decomposition of gradient attribution is provided, and the device for compressing a large language model based on singular value decomposition of gradient attribution corresponds to the method for compressing a large language model based on singular value decomposition of gradient attribution in the above embodiment. Figure 3 As shown, the device for compressing a large language model based on singular value decomposition of gradient attribution includes a calculation module 101, a collection module 102, a filtering module 103, and an iteration module 104. The functional modules are described in detail as follows:

[0064] The calculation module 101 is used to calculate the gradient of the original weight matrix of each linear layer on the auxiliary calibration data set, and allocate sparsity according to the gradient size of each layer weight during the singular value decomposition of the original weight matrix;

[0065] A collecting module 102 is used to collect the gradients of each singular value of the original weight matrix when the original weight matrix is ​​decomposed by singular value on the auxiliary calibration data set, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term at the singular value 0 point;

[0066] A filtering module 103 is used to use the approximate loss term to represent the importance of each singular value of the original weight matrix during singular value decomposition, and filter out unimportant singular values ​​according to a preset dynamic threshold;

[0067] The iterative module 104 is used to set the filtered singular value as the index of the original weight matrix when it passes through the singular value decomposition, decompose the singular value related to the index in the original weight matrix into the product of two low-rank matrices, and repeat the step of collecting the gradients of each singular value of the original weight matrix when it passes through the singular value decomposition on the auxiliary calibration data set, and perform low-rank decomposition on the original weight matrix layer by layer based on the gradient descent method of each singular value until the compression ratio of the large language model is met.

[0068] In one embodiment, the device is further configured to:

[0069] When the original weight matrix with large gradient has low sparsity during singular value decomposition, the rank with a proportion greater than the preset value is retained.

[0070] In one embodiment, the device is further configured to:

[0071] The approximate loss term is used to represent the importance of each singular value of the original weight matrix when it is decomposed by singular value, and the singular values ​​of the preset data satisfying the required compression ratio are set.

[0072] In one embodiment, the device is further configured to:

[0073] A dynamic threshold is preset, and the compression ratio is adjusted according to the performance requirements of a specific task.

[0074] In one embodiment, the device is further configured to:

[0075] Collect weight features after singular value decomposition of the original weight matrix on the auxiliary calibration dataset;

[0076] Transposing the weight features into a feature square matrix;

[0077] Eigenvalues ​​are calculated based on the characteristic matrix, and singular values ​​are obtained according to the eigenvalues.

[0078] In one embodiment, the iteration module 104 is further configured to:

[0079] Set the filtered singular values ​​to be the indices of the original weight matrix when it is decomposed by singular value;

[0080] The singular values ​​associated with the indices in the original weight matrix are decomposed into the product of two low-rank matrices, wherein the product of the two low-rank matrices constitutes a transposed weight matrix that is perpendicular to the original weight matrix.

[0081] In one embodiment, the iteration module 104 is further configured to:

[0082] Get the eigenvectors of clusters in the original weight matrix and keep the eigenvector settings;

[0083] The filtered singular values ​​are used as indices of the original weight matrix when passing through the singular value decomposition;

[0084] When obtaining the index words in the index, the corresponding query matches the singular value of the index word, and obtains the original weight matrix where the singular value is located.

[0085] The present invention provides a device for compressing a large language model by singular value decomposition based on gradient attribution. The device first calculates the gradient of the original weight matrix of each linear layer on an auxiliary calibration data set, and allocates sparsity according to the gradient size of the weight of each layer during singular value decomposition of the original weight matrix; collects the gradient of each singular value of the original weight matrix when it passes through singular value decomposition on the auxiliary calibration data set, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term when the singular value is 0 point; uses the approximate loss term to represent the importance of each singular value of the original weight matrix when it passes through singular value decomposition, and filters unimportant singular values ​​according to a preset dynamic threshold; sets the filtered singular value as the index of the original weight matrix when it passes through singular value decomposition, and performs low-rank decomposition layer by layer on the singular values ​​related to the index in the original weight matrix according to the gradient descent method of each singular value until the compression ratio of the large language model is met. By filtering out the singular value directions that are important for the approximate loss term through gradient information, the redundant retention of non-critical directions in the traditional singular value decomposition method is avoided. This method does not rely on specific hardware cores and can be efficiently deployed on general devices. At the same time, by minimizing the retention of non-critical singular values, this method can significantly improve the compression ratio of the model without significantly losing model performance, further improving the inference speed and reducing video memory usage.

[0086] For the specific limitations of the device for compressing a large language model based on singular value decomposition with gradient attribution, please refer to the limitations of the method for compressing a large language model based on singular value decomposition with gradient attribution mentioned above, which will not be repeated here. Each module in the above-mentioned device for compressing a large language model based on singular value decomposition with gradient attribution can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0087] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a method for compressing a large language model based on singular value decomposition of gradient attribution.

[0088] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the client-side functions or steps of a method for compressing a large language model based on singular value decomposition of gradient attribution are implemented

[0089] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0090] Select samples and clean the sample data. The cleaned samples use user ID as the primary key. The fields in the cleaned sample data contain variable features. The number of candidate questions is used as matching questions.

[0091] A matching answer corresponding to the matching question is selected to respond to the user question.

[0092] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0093] The gradient of the original weight matrix of each linear layer is calculated on the auxiliary calibration data set, and the sparsity is allocated according to the gradient size of the weight of each layer during the singular value decomposition of the original weight matrix;

[0094] Collecting the gradients of each singular value of the original weight matrix on the auxiliary calibration data set when the original weight matrix is ​​decomposed by singular value, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term at the singular value 0 point;

[0095] Using the approximate loss term to represent the importance of each singular value of the original weight matrix during singular value decomposition, filtering unimportant singular values ​​according to a pre-set dynamic threshold;

[0096] The filtered singular value is set as the index of the original weight matrix when it passes through singular value decomposition, the singular value associated with the index in the original weight matrix is ​​decomposed into the product of two low-rank matrices, and the step of collecting the gradient of each singular value of the original weight matrix when it passes through singular value decomposition is repeated on the auxiliary calibration data set, and the original weight matrix is ​​low-rank decomposed layer by layer based on the gradient descent method of each singular value until the compression ratio of the large language model is met.

[0097] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0099] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for compressing a large language model based on singular value decomposition with gradient attribution, characterized in that: include: Calculate the gradient of the original weight matrix of each linear layer on the auxiliary calibration data set, wherein the original weight matrix is ​​assigned sparsity according to the gradient size of the weight of each layer when passing through singular value decomposition; Collecting the gradients of each singular value of the original weight matrix on the auxiliary calibration data set when the original weight matrix is ​​decomposed by singular value, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term at the singular value 0 point; Using the approximate loss term to represent the importance of each singular value of the original weight matrix during singular value decomposition, filtering unimportant singular values ​​according to a pre-set dynamic threshold; The filtered singular values ​​are set as the indices of the original weight matrix during singular value decomposition, and the singular values ​​associated with the indexes in the original weight matrix are low-rank decomposed layer by layer according to the gradient descent method of each singular value until the compression ratio of the large language model is met.

2. The method for compressing a large language model based on singular value decomposition with gradient attribution according to claim 1, characterized in that: When the original weight matrix with large gradient has low sparsity during singular value decomposition, the rank with a proportion greater than the preset value is retained.

3. The method for compressing a large language model based on singular value decomposition with gradient attribution according to claim 2, characterized in that: After the step of collecting the gradients of each singular value of the original weight matrix on the auxiliary calibration data set when the original weight matrix is ​​decomposed by singular value, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term at the singular value 0 point, the method further comprises: The approximate loss term is used to represent the importance of each singular value of the original weight matrix when it is decomposed by singular value, and a preset number of singular values ​​that meet the required compression ratio are set.

4. The method for compressing a large language model based on singular value decomposition with gradient attribution according to claim 3, characterized in that: After the step of using the approximate loss term to represent the importance of each singular value of the original weight moment during singular value decomposition, and filtering unimportant singular values ​​according to a preset dynamic threshold, the method further comprises: A dynamic threshold is preset, and the compression ratio is adjusted according to the performance requirements of a specific task.

5. The method for compressing a large language model based on gradient-attributed singular value decomposition as claimed in claim 2, characterized in that: After the step of calculating the gradient of the original weight matrix of each linear layer on the auxiliary calibration data set and allocating sparsity of the original weight matrix according to the gradient size of each layer weight when performing singular value decomposition, the method further includes: Collect weight features of the original weight matrix after singular value decomposition on the auxiliary calibration dataset; Transposing the weight features into a feature square matrix; Eigenvalues ​​are calculated based on the characteristic matrix, and singular values ​​are obtained according to the eigenvalues.

6. The method for compressing a large language model based on singular value decomposition with gradient attribution according to claim 2, characterized in that: The step of performing low-rank decomposition layer by layer of the singular values ​​related to the index in the original weight matrix according to the gradient descent method of each singular value comprises: The singular values ​​associated with the index in the original weight matrix are decomposed into the product of two low-rank matrices, and the singular values ​​associated with the index are low-rank decomposed layer by layer according to the gradient descent method of each singular value.

7. The method for compressing a large language model based on gradient-attributed singular value decomposition as claimed in claim 6, characterized in that: The step of setting the filtered singular value as the index of the original weight matrix during singular value decomposition comprises: Get the eigenvectors of clusters in the original weight matrix and keep the eigenvector settings; The filtered singular values ​​are the indices of the original weight matrix when it is decomposed by singular value; When obtaining the index words in the index, the corresponding query matches the singular value of the index word, and obtains the original weight matrix where the singular value is located.

8. A device for compressing a large language model based on singular value decomposition of gradient attribution, characterized in that: include: A calculation module, used to calculate the gradient of the original weight matrix of each linear layer on the auxiliary calibration data set, wherein the original weight matrix is ​​allocated sparsity according to the gradient size of each layer weight when passing through singular value decomposition; A collecting module, used for collecting the gradients of each singular value of the original weight matrix when the original weight matrix is ​​decomposed by singular value on the auxiliary calibration data set, wherein the first-order Taylor expansion term at the singular value 0 point is used as the approximate loss term at the singular value 0 point; A filtering module, used to use the approximate loss term to represent the importance of each singular value of the original weight matrix when it is decomposed by singular value, and filter out unimportant singular values ​​according to a preset dynamic threshold; The iterative module is used to set the filtered singular values ​​as the indices of the original weight matrix during singular value decomposition, and perform low-rank decomposition layer by layer on the singular values ​​related to the indexes in the original weight matrix according to the gradient descent method of each singular value until the compression ratio of the large language model is met.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for compressing a large language model based on gradient attribution singular value decomposition as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for compressing a large language model based on singular value decomposition with gradient attribution as claimed in any one of claims 1 to 7 are implemented.

Images