Pruning method and device of large language model, storage medium and computer equipment
By pruning large language models, removing redundant layers and performing singular value decomposition, the problem of high storage and computing requirements caused by large amounts of model parameters is solved, and the model's efficiency and performance maintenance is achieved.
Patent Information
- Application Number
- CN202411727166.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-18
AI Technical Summary
Large language models (LLMs) are highly storage demand and computationally intensive due to the huge amount of parameters, making it difficult to effectively deploy in resource-constrained financial terminal devices or edge computing environments.
By pruning the large language model, the weight matrix of the redundant layer is identified and removed, and the singular value decomposition is performed, the performance impact of each singular value on the downstream task is evaluated, the target singular value is determined, and the low-rank decomposition is performed to replace the original weight matrix.
It reduces the amount of parameters of the model, reduces the computing resource requirements, speeds up the response speed of the model, while maintaining the performance of the model, suitable for deployment on resource-constrained devices.
Smart Images
Figure CN119647623A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of financial technology and large language model technology, and in particular to a large language model pruning method, device, storage medium and computer equipment. Background Art
[0002] In the field of financial technology, large language models (LLMs) often have a huge number of parameters, which has caused a series of challenges in practical applications. The first problem is that the storage requirements of the model are extremely high, making it difficult to effectively deploy it in resource-constrained financial terminal devices or edge computing environments. Secondly, the massive number of parameters increases the computational intensity of the reasoning process, making it difficult to meet the strict requirements of real-time response in financial technology applications. Implementing a compression strategy for LLMs can not only significantly reduce video memory usage, but also speed up reasoning, while only slightly affecting model performance, providing financial technology companies with a practical solution that can both cut costs and improve efficiency.
[0003] Recent studies have shown that large models contain a high degree of redundancy, and by removing redundant layers in the model, significant inference acceleration can be achieved in practice. However, compared with other model compression techniques, large language model compression methods that simply use layer removal still have certain disadvantages in performance. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a pruning method, apparatus, storage medium and computer device for a large language model, which implement pruning compensation for the large language model, while reducing the number of parameters of the large language model and ensuring the performance of the large language model.
[0005] According to one aspect of the present application, a pruning method for a large language model is provided, the method comprising:
[0006] Obtaining a large language model to be pruned, and determining redundant layers in the large language model;
[0007] Extracting a weight matrix of the redundant layer, and performing singular value decomposition on the weight matrix to determine a plurality of singular values corresponding to the weight matrix;
[0008] Based on the auxiliary calibration data set corresponding to the large language model, calculating the performance impact of each singular value on the downstream task, and determining a target singular value to be retained among the multiple singular values according to the performance impact of each singular value;
[0009] The weight matrix is subjected to low-rank decomposition according to the target singular value, and the weight matrix is replaced by a low-rank matrix obtained by the low-rank decomposition.
[0010] In the embodiment of the present application, optionally, determining the redundant layer in the large language model includes:
[0011] Performing a first round of calculation on the inter-layer input-output similarity of the large language model according to the auxiliary calibration data set corresponding to the large language model, and selecting the layer with the lowest inter-layer input-output similarity calculated as the first round of redundant layer;
[0012] A second round of calculation is performed on the inter-layer input-output similarity of the large language model with the first-round redundant layer removed according to the auxiliary calibration data set, and the layer with the lowest calculated inter-layer input-output similarity is selected as the second-round redundant layer. After removing the second-round redundant layer, a third round of calculation of the inter-layer input-output similarity and determination of the third-round redundant layer are continued until a preset redundant layer determination condition is met, wherein the preset redundant layer determination condition is that the number of redundant layers reaches a preset number of redundant layers or the inter-layer input-output similarity is greater than a preset similarity.
[0013] In the embodiment of the present application, optionally, performing singular value decomposition on the weight matrix to determine a plurality of singular values corresponding to the weight matrix includes:
[0014] Performing singular value decomposition on the weight matrix to obtain a decomposed left singular vector matrix, a singular value matrix and a right singular vector matrix;
[0015] A plurality of singular values corresponding to the weight matrix are determined based on the singular value matrix.
[0016] In the embodiment of the present application, optionally, the calculating the performance impact of each singular value on the downstream task based on the auxiliary calibration data set corresponding to the large language model includes:
[0017] Inputting the data in the auxiliary calibration data set into the large language model, passing the input data layer by layer through the large language model to obtain output data of the large language model, calculating the loss value of the loss function of the large language model based on the output data and the input data, and performing backpropagation on the large language model to calculate the gradient of the loss function with respect to the weight matrix;
[0018] Projecting the gradient of the weight matrix to the direction of each singular value of the weight matrix respectively, and determining the gradient of each singular value;
[0019] Based on the gradient of each singular value, the performance impact of each singular value on the downstream task is determined.
[0020] In the embodiment of the present application, optionally, determining the performance impact of each singular value on the downstream task according to the gradient of each singular value includes:
[0021] For each singular value, determine a change amount of the singular value from a current value to 0, perform a product calculation on the gradient of the singular value and the change amount, determine an approximate loss when the singular value is 0 based on the loss value and a result of the product calculation, and determine a change amount of the loss function of the singular value according to the loss value and the approximate loss;
[0022] The impact of the singular value on the performance of the downstream task is determined according to the change in the loss function.
[0023] In an embodiment of the present application, optionally, performing low-rank decomposition on the weight matrix according to the target singular value includes:
[0024] Truncating the singular value matrix based on the target singular values to obtain a target diagonal matrix;
[0025] Based on the number k of the target singular values, truncating the left singular vector matrix and the right singular vector matrix respectively, taking the first k columns of the left singular vector matrix as the target left singular vector matrix, and taking the first k columns of the right singular vector matrix as the target right singular vector matrix;
[0026] The target left singular vector matrix, the target diagonal matrix, and the transpose of the target right singular vector matrix are multiplied to obtain the low rank matrix.
[0027] In the embodiment of the present application, optionally, determining a target singular value to be retained among multiple singular values according to the performance impact of each singular value includes:
[0028] The singular values are sorted in descending order of performance impact, and the singular value with a preset rank before sorting is taken as the target singular value to be retained; or,
[0029] The singular value whose performance impact is greater than a preset performance threshold is taken as the target singular value to be retained.
[0030] According to another aspect of the present application, a pruning device for a large language model is provided, the device comprising:
[0031] A redundant layer determination module, used to obtain a large language model to be pruned and determine redundant layers in the large language model;
[0032] A singular value decomposition module, used to extract the weight matrix of the redundant layer, and perform singular value decomposition on the weight matrix to determine a plurality of singular values corresponding to the weight matrix;
[0033] A singular value calculation module, used to calculate the performance impact of each singular value on a downstream task based on the auxiliary calibration data set corresponding to the large language model, and determine a target singular value to be retained among multiple singular values according to the performance impact of each singular value;
[0034] A low-rank decomposition module is used to perform low-rank decomposition on the weight matrix according to the target singular value, and replace the weight matrix with a low-rank matrix obtained by the low-rank decomposition.
[0035] In the embodiment of the present application, optionally, the redundant layer determination module is further used to:
[0036] Performing a first round of calculation on the inter-layer input-output similarity of the large language model according to the auxiliary calibration data set corresponding to the large language model, and selecting the layer with the lowest inter-layer input-output similarity calculated as the first round of redundant layer;
[0037] A second round of calculation is performed on the inter-layer input-output similarity of the large language model with the first-round redundant layer removed according to the auxiliary calibration data set, and the layer with the lowest calculated inter-layer input-output similarity is selected as the second-round redundant layer. After removing the second-round redundant layer, a third round of calculation of the inter-layer input-output similarity and determination of the third-round redundant layer are continued until a preset redundant layer determination condition is met, wherein the preset redundant layer determination condition is that the number of redundant layers reaches a preset number of redundant layers or the inter-layer input-output similarity is greater than a preset similarity.
[0038] In the embodiment of the present application, optionally, the singular value decomposition module is further used to:
[0039] Performing singular value decomposition on the weight matrix to obtain a decomposed left singular vector matrix, a singular value matrix and a right singular vector matrix;
[0040] A plurality of singular values corresponding to the weight matrix are determined based on the singular value matrix.
[0041] In the embodiment of the present application, optionally, the singular value calculation module is further used to:
[0042] Inputting the data in the auxiliary calibration data set into the large language model, passing the input data layer by layer through the large language model to obtain output data of the large language model, calculating the loss value of the loss function of the large language model based on the output data and the input data, and performing backpropagation on the large language model to calculate the gradient of the loss function with respect to the weight matrix;
[0043] Projecting the gradient of the weight matrix to the direction of each singular value of the weight matrix respectively, and determining the gradient of each singular value;
[0044] Based on the gradient of each singular value, the performance impact of each singular value on the downstream task is determined.
[0045] In the embodiment of the present application, optionally, the singular value calculation module is further used to:
[0046] For each singular value, determine a change amount of the singular value from a current value to 0, perform a product calculation on the gradient of the singular value and the change amount, determine an approximate loss when the singular value is 0 based on the loss value and a result of the product calculation, and determine a change amount of the loss function of the singular value according to the loss value and the approximate loss;
[0047] The impact of the singular value on the performance of the downstream task is determined according to the change in the loss function.
[0048] In the embodiment of the present application, optionally, the low-rank decomposition module is further used to:
[0049] Truncating the singular value matrix based on the target singular values to obtain a target diagonal matrix;
[0050] Based on the number k of the target singular values, truncating the left singular vector matrix and the right singular vector matrix respectively, taking the first k columns of the left singular vector matrix as the target left singular vector matrix, and taking the first k columns of the right singular vector matrix as the target right singular vector matrix;
[0051] The target left singular vector matrix, the target diagonal matrix, and the transpose of the target right singular vector matrix are multiplied to obtain the low rank matrix.
[0052] In the embodiment of the present application, optionally, the singular value calculation module is further used to:
[0053] The singular values are sorted in descending order of performance impact, and the singular value with a preset rank before sorting is taken as the target singular value to be retained; or,
[0054] The singular value whose performance impact is greater than a preset performance threshold is taken as the target singular value to be retained.
[0055] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned pruning method of the large language model is implemented.
[0056] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned large language model pruning method when executing the program.
[0057] By means of the above technical scheme, a pruning method, device, storage medium and computer equipment of a large language model provided in an embodiment of the present application selects a large language model that needs to be optimized, identifies the redundant layers that contribute less to the model performance, extracts the weight matrices of these redundant layers, and performs singular value decomposition on them to obtain multiple singular values, uses an auxiliary calibration data set, evaluates the impact of each singular value on the performance of a specific task, determines the key singular values that need to be retained, performs low-rank decomposition on the weight matrix based on the key singular values, replaces the original weight matrix with a low-rank matrix, and reduces the number of parameters of the model. By reducing the number of parameters of the model, the embodiment of the present application can reduce the computing resources required for the model during reasoning and speed up the response speed of the model. At the same time, pruning can also reduce the noise of the model to a certain extent, provide an exploration direction for improving the performance of the model, and the pruned model takes up less storage space, which is particularly important for applications deployed on resource-constrained devices. In addition, due to the reduction in computing and storage requirements, the energy consumption of the model during operation will also be reduced accordingly, which is conducive to the development of environmentally friendly applications, making it more efficient and lightweight while maintaining good performance.
[0058] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0060] Figure 1 A schematic diagram of a process of pruning a large language model provided in an embodiment of the present application is shown;
[0061] Figure 2 A schematic diagram of a process flow of another large language model pruning method provided in an embodiment of the present application is shown;
[0062] Figure 3 A schematic diagram of the structure of a large language model pruning device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0063] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.
[0064] In this embodiment, a pruning method for a large language model is provided. Figure 1As shown, the method includes:
[0065] Step 101: Obtain a large language model to be pruned, and determine redundant layers in the large language model.
[0066] Step 102: extract the weight matrix of the redundant layer, and perform singular value decomposition on the weight matrix to determine a plurality of singular values corresponding to the weight matrix.
[0067] Step 103: Based on the auxiliary calibration data set corresponding to the large language model, calculate the performance impact of each singular value on the downstream task, and determine the target singular value to be retained among the multiple singular values according to the performance impact of each singular value.
[0068] Step 104: perform low-rank decomposition on the weight matrix according to the target singular value, and replace the weight matrix with a low-rank matrix obtained by the low-rank decomposition.
[0069] The embodiment of the present application provides a method for pruning a large language model (LLM). Pruning refers to removing unimportant parts of the model (such as connections or layers) to reduce the complexity and number of parameters of the model while trying to maintain the performance of the model, which helps to optimize the operating efficiency of the model and reduce the demand for computing resources. In the field of financial technology, large language models (LLMs) are widely used, such as in intelligent customer service, risk assessment, market analysis, and investment advice. The above pruning method can significantly improve the efficiency and performance of these application scenarios. First, a large language model that needs to be pruned is selected. The large language model is usually composed of a multi-layer neural network, and each layer contains a large number of parameters. Redundant layers refer to those layers that contribute less to the final output of the model or can be optimized by other means. Determining these redundant layers is the first step in the pruning process, which helps to perform pruning operations more accurately later. After determining the redundant layers, the weight matrices of these layers are extracted. The weight matrix represents the strength of the connection between the layers and is the core component of the knowledge learned by the model. By performing singular value decomposition (SVD) on the weight matrix, it can be decomposed into the product of three matrices: UΣV T. Among them, U and V are orthogonal matrices, Σ is a diagonal matrix, and the elements on its diagonal are singular values, which reflect the importance of the information in the original matrix. Then, by using auxiliary calibration data sets (such as historical conversation data, loan data, market data, etc.), the importance of each singular value can be quantified. Specifically, some smaller singular values can be removed one by one or in combination, and then the performance of the model on downstream tasks can be tested to determine which singular values can be safely removed and which must be retained. This process helps to accurately locate which information is critical to model performance. Finally, based on the determined target singular values, the original weight matrix is low-rank approximated. The low-rank decomposition only retains the most important singular values and their corresponding vectors, thereby forming a new, lower-rank matrix to replace the original weight matrix. The purpose of this is to reduce the number of parameters of the model, thereby reducing storage and computing costs, while maintaining the performance of the model as much as possible. For example, in risk assessment, large language models can be used to analyze unstructured data such as credit records and social media activities of loan applicants to predict default risks. First, determine the redundant layers in the model that have less impact on risk prediction; then perform singular value decomposition on the weight matrices of these layers to extract the main feature information; then use historical loan data as an auxiliary calibration data set to evaluate the impact of each singular value on the accuracy of risk prediction and determine the key singular values; finally, perform low-rank decomposition based on the key singular values to optimize the model and improve prediction speed and efficiency.
[0070] By applying the technical solution of this embodiment, a large language model to be optimized is selected, and redundant layers that contribute less to the performance of the model are identified, the weight matrices of these redundant layers are extracted, and singular value decomposition is performed on them to obtain multiple singular values. The auxiliary calibration data set is used to evaluate the impact of each singular value on the performance of a specific task, and the key singular values to be retained are determined. The weight matrix is low-rank decomposed based on the key singular values, and the original weight matrix is replaced with a low-rank matrix to reduce the number of parameters of the model. The embodiment of the present application can reduce the computing resources required for the model during reasoning and speed up the response speed of the model by reducing the number of parameters of the model. At the same time, pruning can also reduce the noise of the model to a certain extent, provide an exploration direction for improving the performance of the model, and the pruned model takes up less storage space, which is particularly important for applications deployed on resource-constrained devices. In addition, due to the reduction in computing and storage requirements, the energy consumption of the model during operation will also be reduced accordingly, which is conducive to the development of environmentally friendly applications. In summary, this pruning method based on singular value decomposition provides an effective way to optimize large language models, making them more efficient and lightweight while maintaining good performance.
[0071] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another pruning method for a large language model is provided, such as Figure 2 As shown, the method includes:
[0072] Step 201: Obtain a large language model to be pruned.
[0073] Step 202: Perform a first round of calculation on the inter-layer input-output similarity of the large language model according to the auxiliary calibration data set corresponding to the large language model, and select the layer with the lowest calculated inter-layer input-output similarity as the first round of redundant layer; perform a second round of calculation on the inter-layer input-output similarity of the large language model with the first round of redundant layers removed according to the auxiliary calibration data set, select the layer with the lowest calculated inter-layer input-output similarity as the second round of redundant layer, and continue to perform a third round of calculation on the inter-layer input-output similarity and determine the third round of redundant layers after removing the second round of redundant layers until a preset redundant layer determination condition is met, wherein the preset redundant layer determination condition is that one of the number of redundant layers reaches the preset number of redundant layers and the inter-layer input-output similarity is greater than the preset similarity.
[0074] In this embodiment, the redundant layers are gradually determined by calculating the inter-layer input-output similarity multiple times. The specific steps are as follows: First round of calculation: Use the auxiliary calibration data set corresponding to the large language model to calculate the input-output similarity between the layers of the model, and select the layer with the lowest inter-layer input-output similarity calculated as the first round of redundant layer. Among them, if the input-output similarity of a certain layer is low, it means that the layer contributes less to the overall output of the model and can be regarded as a redundant layer. By gradually removing these layers, the model can be gradually optimized. On the contrary, if the input-output similarity of a certain layer is high, it means that the layer has an important contribution to the output of the model and should not be removed. Second round of calculation: After removing the redundant layers determined in the first round, use the auxiliary calibration data set again to calculate the inter-layer input-output similarity of the remaining layers. Select the layer with the lowest inter-layer input-output similarity calculated as the second round of redundant layer. Subsequent rounds of calculation: Continue the above process, and after removing the redundant layers determined in one round each time, recalculate the inter-layer input-output similarity of the remaining layers, and select the layer with the lowest similarity as the new redundant layer. This process continues until the preset redundant layer determination condition is met. The preset conditions may be: the number of redundant layers reaches a preset number of redundant layers, and the similarity between input and output layers is greater than a preset similarity threshold. Through multiple rounds of calculation and removal of redundant layers, the model can be gradually optimized to ensure that each removed layer has the least impact on model performance. After each layer is removed, the similarity of the remaining layers is recalculated, and the selection of redundant layers can be dynamically adjusted to avoid removing too many important layers at one time, resulting in a significant decrease in model performance. By setting an upper limit on the number of redundant layers, the complexity of the model can be controlled to prevent excessive pruning, and by setting a similarity threshold, it can be ensured that the remaining layers contribute significantly to model performance and avoid removing important layers. The embodiment of the present application can significantly reduce the number of parameters and computational complexity of the model by gradually removing redundant layers, improve the operating efficiency of the model, and ensure that each removed layer has the least impact on model performance through multiple rounds of optimization based on input-output similarity, thereby maintaining high performance while reducing the number of parameters.
[0075] Step 203: extract the weight matrix of the redundant layer; perform singular value decomposition on the weight matrix to obtain a decomposed left singular vector matrix, a singular value matrix and a right singular vector matrix; determine multiple singular values corresponding to the weight matrix based on the singular value matrix. Wherein, the singular value can be represented by σ, σ = σ 1 , σ 2 ..., σ i , ... σ n .
[0076] In this embodiment, the weight matrix can be decomposed into the product of three matrices through singular value decomposition, so as to extract the main information components of the weight matrix. First, the weight matrix is subjected to singular value decomposition. Assuming that the weight matrix is W, the singular value decomposition thereof can be expressed as: W = UΣVT , where U is the left singular vector matrix, whose columns are the left singular vectors of W, Σ is the singular value matrix, which is a diagonal matrix, and the elements on the diagonal are singular values. V is the right singular vector matrix, whose columns are the right singular vectors of W. The singular value matrix Σ is a diagonal matrix, and the elements on the diagonal are singular values. These singular values are arranged in order from large to small, indicating the importance of each direction in the weight matrix. By extracting all the elements on the diagonal of Σ, multiple singular values corresponding to the weight matrix can be obtained. The elements on the diagonal of the singular value matrix Σ (singular values) indicate the importance of each direction in the weight matrix. Larger singular values correspond to more important information, while smaller singular values correspond to secondary or redundant information. By retaining larger singular values and removing smaller singular values, the complexity of the model can be reduced while maintaining model performance.
[0077] Step 204: input the data in the auxiliary calibration data set into the large language model, pass the input data layer by layer through the large language model to obtain the output data of the large language model, calculate the loss value of the loss function of the large language model based on the output data and the input data, and perform back propagation on the large language model to calculate the gradient of the loss function with respect to the weight matrix; project the gradient of the weight matrix to the direction of each singular value of the weight matrix, respectively, and determine the gradient of each singular value; and determine the performance impact of each singular value on the downstream task according to the gradient of each singular value.
[0078] In this embodiment, the data in the auxiliary calibration data set is input into the large language model, the input data is passed layer by layer through the model to obtain the output data of the model, the loss value of the loss function of the model is calculated based on the output data and the input data, the model is back-propagated, and the gradient of the loss function to the weight matrix is calculated. The gradient of the weight matrix is projected to the direction of each singular value of the weight matrix, and the gradient of each singular value is determined. According to the gradient of each singular value, the impact of each singular value on the performance of the downstream task is evaluated. Specifically, the data x in the auxiliary calibration data set is i Input into the large language model, pass the input data layer by layer through the model, and get the model's output data y i Based on the output data y i and the true label t i , calculate the loss value of the loss function L of the model. Where N is the size of the dataset, l(y i ,t i ) is the loss function of a single sample (such as cross entropy loss). Backpropagate the model and calculate the gradient of the loss function with respect to the weight matrix W Assume that the singular value decomposition result of the weight matrix W is W = UΣV T , the gradient Projected onto the direction of each singular value, specifically, for the i-th singular value σ i , and its corresponding left singular vector is u i , the right singular vector is v i , calculate the gradient of the i-th singular value Then the singular value σ i is the first-order Taylor expansion term at point 0 as σ i =0. Optionally, determining the performance impact of each singular value on the downstream task based on the gradient of each singular value includes: for each singular value, determining the change of the singular value from the current value to 0, performing a product calculation on the gradient of the singular value and the change, determining the approximate loss when the singular value is 0 based on the loss value and the result of the product calculation, determining the change of the loss function of the singular value according to the loss value and the approximate loss; and determining the performance impact of the singular value on the downstream task according to the change of the loss function. Wherein, for each singular value σ i , calculate the singular value σ i Approximate loss when it is 0 Among them, L(σ i ) represents the loss value, represents the singular value σ i Gradient; According to the approximate loss of each singular value, calculate the change ΔL of the loss function of each singular value i , ΔL i =L(σ i =0)-L(σ i ).
[0079] Step 205: Determine a target singular value to be retained among the multiple singular values according to the performance impact of each singular value.
[0080] In this embodiment, the influence of each singular value on the performance of downstream tasks can be quantified by the absolute value of the gradient. A larger absolute value of the gradient indicates that the singular value has a significant impact on the model performance, and vice versa.
[0081] In an embodiment of the present application, optionally, determining a target singular value to be retained among multiple singular values based on the performance impact of each singular value includes: sorting the singular values from large to small according to their performance impact, and taking the singular value with a preset position before sorting as the target singular value to be retained; or, taking the singular value whose performance impact is greater than a preset performance threshold as the target singular value to be retained.
[0082] In the above embodiment, all singular values are sorted from large to small according to their performance impact, and the singular values with preset positions before sorting are selected as the target singular values to be retained. By sorting and selecting, it is possible to ensure that the singular values with the greatest impact on model performance are retained, thereby reducing the complexity of the model while maintaining high performance. By presetting the positions, the degree of pruning can be flexibly controlled to adapt to different application scenarios and requirements. Alternatively, singular values with performance impact greater than a preset performance threshold are directly selected as the target singular values to be retained. By setting the performance threshold, it is possible to clarify which singular values have a significant impact on model performance, so that pruning can be performed more targeted. The performance threshold can prevent excessive pruning and ensure that the model still maintains good performance after pruning. Whether based on sorting or threshold, the embodiments of the present application can effectively reduce the number of parameters of the model, reduce the complexity of the model, and improve the operating efficiency of the model. The two methods provide different pruning strategies, and the most appropriate pruning method can be selected according to specific needs to flexibly respond to different application scenarios. Removing singular values that have little impact on performance can reduce the noise and redundant information of the model and improve the stability and robustness of the model.
[0083] Step 206: truncate the singular value matrix based on the target singular values to obtain a target diagonal matrix; based on the number k of the target singular values, truncate the left singular vector matrix and the right singular vector matrix respectively, take the first k columns of the left singular vector matrix as the target left singular vector matrix, and take the first k columns of the right singular vector matrix as the target right singular vector matrix; multiply the transpose of the target left singular vector matrix, the target diagonal matrix, and the target right singular vector matrix to obtain the low rank matrix.
[0084] In this embodiment, after determining the target singular values to be retained, the weight matrix is low-rank decomposed based on these target singular values. Low-rank decomposition can significantly reduce the number of model parameters while maintaining the performance of the model. Specifically, from the singular value matrix Σ, the first k target singular values are selected to form the target diagonal matrix Σ k From the left singular vector matrix U, select the first k columns to form the target left singular vector matrix U k From the right singular vector matrix V, select the first k columns to form the target right singular vector matrix V k . The target left singular vector matrix U k , target diagonal matrix Σ k and the target right singular vector matrix V k The transpose of V k TMultiply them together to get a low-rank matrix. By retaining the most important singular values, low-rank approximation can ensure that the main information of the model is retained, thereby maintaining the performance of the model while reducing the number of parameters. Low-rank approximation is an effective dimensionality reduction method. By retaining the most important singular values and their corresponding singular vectors, the main information of the original matrix can be approximated. The low-rank matrix can maintain the main features of the original matrix while reducing the number of parameters, thereby maintaining the performance of the model while reducing the complexity of the model. The dimension of the low-rank matrix is reduced from m×n to m×k and k×n, which reduces the number of parameters of the model, improves the operating efficiency of the model, and reduces the demand for computing resources and storage space.
[0085] Step 207: Replace the weight matrix with a low-rank matrix obtained by low-rank decomposition.
[0086] In this embodiment, after completing the low-rank decomposition, replacing the original weight matrix with a low-rank matrix is the last step in the pruning process. This replacement step ensures that the model maintains high performance while reducing the number of parameters.
[0087] By applying the technical solution of this embodiment, most compression methods for removing the LLMs layer need to compensate for performance by training a lightweight network module. Although these methods can reduce the model size to a certain extent, training lightweight modules often requires additional computing resources and time. In response to this problem, the embodiment of the present application proposes a compensation scheme that does not require additional training. This method makes full use of the weight matrix information of the original pre-trained large model, and combines the singular value decomposition technology based on Taylor expansion term truncation to effectively retain the most critical part of the task. Through the low-rank approximation of the weight matrix, this method can compensate the model after the layer is removed at the lowest cost without training, ensuring that the compressed model can still maintain efficient performance. Compared with the traditional method that requires training compensation modules, this solution can reduce the computational overhead while enabling the compression model to meet the dual requirements of real-time and performance in industrial production practice, providing an efficient and low-cost solution for the practical application of LLMs.
[0088] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a pruning device for a large language model, such as Figure 3 As shown, the device comprises:
[0089] A redundant layer determination module, used to obtain a large language model to be pruned and determine redundant layers in the large language model;
[0090] A singular value decomposition module, used to extract the weight matrix of the redundant layer, and perform singular value decomposition on the weight matrix to determine a plurality of singular values corresponding to the weight matrix;
[0091] A singular value calculation module, used to calculate the performance impact of each singular value on a downstream task based on the auxiliary calibration data set corresponding to the large language model, and determine a target singular value to be retained among multiple singular values according to the performance impact of each singular value;
[0092] A low-rank decomposition module is used to perform low-rank decomposition on the weight matrix according to the target singular value, and replace the weight matrix with a low-rank matrix obtained by the low-rank decomposition.
[0093] In the embodiment of the present application, optionally, the redundant layer determination module is further used to:
[0094] Performing a first round of calculation on the inter-layer input-output similarity of the large language model according to the auxiliary calibration data set corresponding to the large language model, and selecting the layer with the lowest inter-layer input-output similarity calculated as the first round of redundant layer;
[0095] A second round of calculation is performed on the inter-layer input-output similarity of the large language model with the first-round redundant layer removed according to the auxiliary calibration data set, and the layer with the lowest calculated inter-layer input-output similarity is selected as the second-round redundant layer. After removing the second-round redundant layer, a third round of calculation of the inter-layer input-output similarity and determination of the third-round redundant layer are continued until a preset redundant layer determination condition is met, wherein the preset redundant layer determination condition is that the number of redundant layers reaches a preset number of redundant layers or the inter-layer input-output similarity is greater than a preset similarity.
[0096] In the embodiment of the present application, optionally, the singular value decomposition module is further used to:
[0097] Performing singular value decomposition on the weight matrix to obtain a decomposed left singular vector matrix, a singular value matrix and a right singular vector matrix;
[0098] A plurality of singular values corresponding to the weight matrix are determined based on the singular value matrix.
[0099] In the embodiment of the present application, optionally, the singular value calculation module is further used to:
[0100] Inputting the data in the auxiliary calibration data set into the large language model, passing the input data layer by layer through the large language model to obtain output data of the large language model, calculating the loss value of the loss function of the large language model based on the output data and the input data, and performing backpropagation on the large language model to calculate the gradient of the loss function with respect to the weight matrix;
[0101] Projecting the gradient of the weight matrix to the direction of each singular value of the weight matrix respectively, and determining the gradient of each singular value;
[0102] Based on the gradient of each singular value, the performance impact of each singular value on the downstream task is determined.
[0103] In the embodiment of the present application, optionally, the singular value calculation module is further used to:
[0104] For each singular value, determine a change amount of the singular value from a current value to 0, perform a product calculation on the gradient of the singular value and the change amount, determine an approximate loss when the singular value is 0 based on the loss value and a result of the product calculation, and determine a change amount of the loss function of the singular value according to the loss value and the approximate loss;
[0105] The impact of the singular value on the performance of the downstream task is determined according to the change in the loss function.
[0106] In the embodiment of the present application, optionally, the low-rank decomposition module is further used to:
[0107] Truncating the singular value matrix based on the target singular values to obtain a target diagonal matrix;
[0108] Based on the number k of the target singular values, truncating the left singular vector matrix and the right singular vector matrix respectively, taking the first k columns of the left singular vector matrix as the target left singular vector matrix, and taking the first k columns of the right singular vector matrix as the target right singular vector matrix;
[0109] The target left singular vector matrix, the target diagonal matrix, and the transpose of the target right singular vector matrix are multiplied to obtain the low rank matrix.
[0110] In the embodiment of the present application, optionally, the singular value calculation module is further used to:
[0111] The singular values are sorted in descending order of performance impact, and the singular value with a preset rank before sorting is taken as the target singular value to be retained; or,
[0112] The singular value whose performance impact is greater than a preset performance threshold is taken as the target singular value to be retained.
[0113] It should be noted that for other corresponding descriptions of the functional units involved in the pruning device for a large language model provided in the embodiment of the present application, reference can be made to Figure 1 to Figure 2 The corresponding description in the method will not be repeated here.
[0114] The embodiment of the present application also provides a computer device, which can be a personal computer, a server, a network device, etc. The computer device includes a bus, a processor, a memory and a communication interface, and can also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.
[0115] Those skilled in the art will appreciate that the structure of the above-mentioned computer device is only a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components, or combine certain components, or have a different arrangement of components.
[0116] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0117] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0121] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A pruning method for a large language model, characterized in that: The method comprises: Obtaining a large language model to be pruned, and determining redundant layers in the large language model; Extracting a weight matrix of the redundant layer, and performing singular value decomposition on the weight matrix to determine a plurality of singular values corresponding to the weight matrix; Based on the auxiliary calibration data set corresponding to the large language model, calculating the performance impact of each singular value on the downstream task, and determining a target singular value to be retained among the multiple singular values according to the performance impact of each singular value; The weight matrix is subjected to low-rank decomposition according to the target singular value, and the weight matrix is replaced by a low-rank matrix obtained by the low-rank decomposition.
2. The method according to claim 1, characterized in that The determining of the redundant layers in the large language model comprises: Performing a first round of calculation on the inter-layer input-output similarity of the large language model according to the auxiliary calibration data set corresponding to the large language model, and selecting the layer with the lowest inter-layer input-output similarity calculated as the first round of redundant layer; A second round of calculation is performed on the inter-layer input-output similarity of the large language model with the first-round redundant layer removed according to the auxiliary calibration data set, and the layer with the lowest calculated inter-layer input-output similarity is selected as the second-round redundant layer. After removing the second-round redundant layer, a third round of calculation of the inter-layer input-output similarity and determination of the third-round redundant layer are continued until a preset redundant layer determination condition is met, wherein the preset redundant layer determination condition is that the number of redundant layers reaches a preset number of redundant layers or the inter-layer input-output similarity is greater than a preset similarity.
3. The method according to claim 1, characterized in that The performing singular value decomposition on the weight matrix to determine a plurality of singular values corresponding to the weight matrix includes: Performing singular value decomposition on the weight matrix to obtain a decomposed left singular vector matrix, a singular value matrix, and a right singular vector matrix; A plurality of singular values corresponding to the weight matrix are determined based on the singular value matrix.
4. The method according to claim 3, characterized in that The calculating, based on the auxiliary calibration data set corresponding to the large language model, the performance impact of each singular value on the downstream task includes: Inputting the data in the auxiliary calibration data set into the large language model, passing the input data layer by layer through the large language model to obtain output data of the large language model, calculating the loss value of the loss function of the large language model based on the output data and the input data, and performing backpropagation on the large language model to calculate the gradient of the loss function with respect to the weight matrix; Projecting the gradient of the weight matrix to the direction of each singular value of the weight matrix respectively, and determining the gradient of each singular value; Based on the gradient of each singular value, the performance impact of each singular value on the downstream task is determined.
5. The method according to claim 4, characterized in that Determining the performance impact of each singular value on the downstream task according to the gradient of each singular value includes: For each singular value, determine a change amount of the singular value from a current value to 0, perform a product calculation on the gradient of the singular value and the change amount, determine an approximate loss when the singular value is 0 based on the loss value and a result of the product calculation, and determine a change amount of the loss function of the singular value according to the loss value and the approximate loss; The impact of the singular value on the performance of the downstream task is determined according to the change in the loss function.
6. The method according to claim 5, characterized in that The step of performing low-rank decomposition on the weight matrix according to the target singular value comprises: Truncating the singular value matrix based on the target singular values to obtain a target diagonal matrix; Based on the number k of the target singular values, truncating the left singular vector matrix and the right singular vector matrix respectively, taking the first k columns of the left singular vector matrix as the target left singular vector matrix, and taking the first k columns of the right singular vector matrix as the target right singular vector matrix; The low-rank matrix is obtained by multiplying the target left singular vector matrix, the target diagonal matrix, and the transpose of the target right singular vector matrix.
7. The method according to any one of claims 1 to 6, characterized in that The step of determining a target singular value to be retained among a plurality of singular values according to the performance impact of each singular value comprises: The singular values are sorted in descending order of performance impact, and the singular value with a preset rank before sorting is taken as the target singular value to be retained; or, The singular value whose performance impact is greater than a preset performance threshold is taken as the target singular value to be retained.
8. A pruning device for a large language model, characterized in that: The device comprises: A redundant layer determination module, used to obtain a large language model to be pruned and determine redundant layers in the large language model; A singular value decomposition module, used to extract the weight matrix of the redundant layer, and perform singular value decomposition on the weight matrix to determine a plurality of singular values corresponding to the weight matrix; A singular value calculation module, used to calculate the performance impact of each singular value on a downstream task based on the auxiliary calibration data set corresponding to the large language model, and determine a target singular value to be retained among multiple singular values according to the performance impact of each singular value; A low-rank decomposition module is used to perform low-rank decomposition on the weight matrix according to the target singular value, and replace the weight matrix with a low-rank matrix obtained by the low-rank decomposition.
9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.