Inference optimization method for pre-training large language model based on cross-layer parameter low-rank decomposition
Through the cross-layer parameter low-rank decomposition method, Cholesky decomposition and SVD are used to optimize the parameter matrix of the large language model, and the core matrix is shared to reduce memory and computational complexity. This solves the application problem of large language models on resource-constrained devices and realizes an efficient and fast inference solution.
Patent Information
- Application Number
- CN202510751043.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-06
- Publication Date
- 2025-09-19
AI Technical Summary
Existing large language models have high memory requirements and computational complexity during inference, which limits their application on resource-constrained devices. In addition, existing SVD methods fail to fully utilize the weight similarity between different layers.
A cross-layer parameter low-rank decomposition method is adopted to generate a whitening matrix through Cholesky decomposition. The parameter matrix of each layer is whitened and then spliced horizontally or vertically before SVD decomposition. The core matrix is shared to optimize the model parameters, reduce memory overhead and improve computational efficiency.
It reduces the memory overhead and computational complexity during model inference while maintaining high accuracy, meeting the needs of low-cost and high-efficiency model deployment.
Smart Images
Figure CN120671829A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of pre-trained large language models, and specifically relates to an inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition. Background Art
[0002] Large language models (LLMs) have achieved significant breakthroughs in natural language processing (NLP), enabling them to perform a variety of complex language tasks such as text generation, machine translation, and question-answering systems. However, these models often have a very large number of parameters; for example, GPT-3 has 175 billion parameters. Such large models require extensive memory storage and computational resources during inference, limiting their practical application on resource-constrained devices such as mobile devices and embedded systems. Therefore, how to effectively compress these models and reduce their memory requirements and computational complexity during inference has become a hot research topic.
[0003] Singular Value Decomposition (SVD) is a commonly used matrix decomposition technique that can decompose a matrix into the product of three matrices: W = UΣV T , where U and V are orthogonal matrices, Σ is a diagonal matrix whose diagonal elements are called singular values. By retaining the first k largest singular values, the original matrix W can be approximated as a low-rank matrix This reduces the memory requirements during inference of large language models. However, existing SVD methods mainly compress single-layer weight matrices and fail to fully utilize the weight similarity between different layers.
[0004] Therefore, how to use the similarity optimization between the parameter matrices of different layers of pre-trained large language models to solve the problems of high memory and high computational overhead required for large language model inference; and providing users of pre-trained large language models with more efficient, fast, and high-precision model inference solutions are issues that researchers in this field urgently need to solve. Summary of the Invention
[0005] In view of this, the present invention provides an inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition, which meets the needs of users of pre-trained large language models for low-cost and high-efficiency deployment of large language models.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] A method for optimizing inference of a pre-trained large language model based on cross-layer parameter low-rank decomposition, comprising:
[0008] S1. Obtain a pre-trained large language model and a calibration dataset, and collect the feature embedding X of each layer parameter matrix during the calibration data inference process.
[0009] S2. Generate a feature embedding whitening matrix S based on the feature embedding X of each layer parameter matrix;
[0010] S3, using whitening matrix S and its inverse matrix S -1 , whiten the parameter matrix of each layer of the large language model;
[0011] S4. Splice the parameter matrix of the large language model after whitening horizontally or vertically into the parameter matrix W; perform SVD decomposition on the parameter matrix W to obtain U, Σ, V T Three sub-matrices; calculate the number of singular values k required to retain the low-rank decomposition, and perform the calculations on U, Σ, and V according to the k value. T The three sub-matrices are truncated and combined into a core matrix and a regulation matrix; the parameter matrices participating in the cross-layer low-rank decomposition are made to share the core matrix, while the coordination matrix is unique to each layer. The parameter matrix of the large language model of the i-th layer is updated to obtain the optimized large language model.
[0012] Optionally, step S2 uses Cholesky decomposition to generate a whitening matrix S for feature embedding.
[0013] Optionally, the generation process of the whitening matrix S of the present invention is:
[0014] Initialize the whitening matrix S to a lower triangular matrix of 0, and for each row i = 1, 2, ..., n, calculate the i-th row element of S
[0015]
[0016] For each column j=i+1,i+2,…,n, calculate:
[0017]
[0018] Optionally, the k value of the present invention is:
[0019]
[0020] Among them, d1 and d2 are the number of rows and columns of the parameter matrix, n is the number of layers participating in the cross-layer low-rank decomposition, and x is the given target compression ratio.
[0021] Optionally, the present invention adopts horizontal splicing if d1≤d2 of the parameter matrix, otherwise adopts vertical splicing.
[0022] Optionally, when the present invention adopts horizontal splicing,
[0023] The core matrix A=U :,:k Σ :k,:k , the coordination matrix Among them U :,:k Represents the matrix formed by keeping the number of rows of matrix U unchanged and taking the first k columns; Σ :k,:k Indicates that the first k rows and columns of the matrix Σ are intercepted to form a new matrix; V T is a matrix with n*d2 columns, Represents the matrix V T Cut out n matrices from the matrix to form the coordination matrix B1, B2, ..., B n , the number of rows of these n matrices is k, so the subscript: k means from V T The first k rows are intercepted, and then every d2 columns are intercepted to form a coordination matrix. When i=1, (i-1)d2:i·d2=0:d2, which means intercepting the first d2 columns. When i=2, (i-1)d2:i·d2=d2:2d2, which means intercepting from the d2+1 column to the 2*d2 column, and so on.
[0024] Optionally, when longitudinal splicing is adopted in the present invention,
[0025] The core matrix The coordination matrix Among them, Σ :k,:k and Denotes the pair of matrices Σ and V T The rows and columns of truncate the first k matrices; U is a matrix with n*d1 rows. It means that n matrices are cut out from the matrix U to form the coordination matrix A1, A2, ..., A n The number of columns of these n matrices is k. The subscript :k means that the first k columns are intercepted from U, and then every d1 rows are intercepted to form a coordination matrix. When i = 1, (i-1)d1:i·d1=0:d1, which means that the first d1 rows are intercepted. When i = 2, (i-1)d1:i·d1=d1:2d1, which means that the rows from d1+1 to 2*d1 are intercepted, and so on.
[0026] Optionally, the present invention allows the parameter matrices participating in the cross-layer low-rank decomposition to share the core matrix, while the coordination matrix is unique to each layer, and the parameter matrix of the large language model of the i-th layer is updated to W i ≈AB i .
[0027] Optionally, the layers participating in the cross-layer parameter low-rank decomposition in the present invention are adjacent layers in a large language model.
[0028] Optionally, the pre-trained large language model described in the present invention is a large language model with a decoder-only structure, including the Llama series, the GTP series, and the OPT series.
[0029] Beneficial effects:
[0030] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a large language model inference optimization method based on cross-layer parameter low-rank decomposition, which uses Cholesky decomposition to perform data whitening on the parameter matrix of the pre-trained large language model, and uses a cross-layer parameter low-rank decomposition algorithm to achieve low-rank decomposition of the parameter matrix while maintaining high precision, thereby reducing the memory overhead in the model inference process, and further optimizing the calculation speed by sharing the core matrix, thereby meeting the low-cost and high-efficiency deployment and inference requirements of users of pre-trained large language models. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0032] Figure 1 This is a schematic diagram of the overall process structure of the method provided by the present invention.
[0033] Figure 2 This is a flow chart of feature embedding extraction and whitening matrix calculation in the method provided by the present invention.
[0034] Figure 3 Schematic diagram of low-rank decomposition of cross-layer parameters in the method provided by the present invention.
[0035] Figure 4 This is a structural diagram of the pre-trained large language model ultimately generated by the method provided by the present invention when n=2. DETAILED DESCRIPTION
[0036] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0037] It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments may be combined with each other; and, based on the embodiments in this disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of this disclosure.
[0038] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.
[0039] like Figure 1 As shown, in this embodiment, a method for optimizing inference of a pre-trained large language model based on cross-layer parameter low-rank decomposition includes the following steps:
[0040] S1. Obtain a pre-trained large language model and a calibration dataset, and collect the feature embedding X of each layer parameter matrix during the calibration data inference process.
[0041] To obtain the whitening matrix, we first need to select a dataset suitable for the pre-trained large language model as the calibration dataset, perform inference on it, and record the feature embedding X output by each parameter matrix during the collection process.
[0042] S2, use the collected feature embedding X of each layer to perform Cholesky decomposition to generate the feature embedding whitening matrix S; the calculation process is as follows:
[0043] First, initialize the whitening matrix S to a lower triangular matrix of 0;
[0044] For each row i=1,2,…,n, calculate the i-th row element of the whitening matrix S
[0045]
[0046] For each column j=i+1,i+2,…,n, calculate:
[0047]
[0048] Figure 2 The process of using the rectified dataset to collect feature embeddings and calculate the corresponding whitening matrix in this embodiment is demonstrated.
[0049] S3, using whitening matrix S and its inverse matrix S -1 Whiten the model parameter matrix of each layer.
[0050] S4. Perform low-rank decomposition of cross-layer parameters based on the whitened model to obtain an optimized large language model.
[0051] The detailed process of using the horizontal splicing method to perform cross-layer parameter low-rank decomposition in this embodiment is as follows:
[0052] First, the parameter matrices of several layers of the whitened model are concatenated into the parameter matrix W, that is, Figure 3 in When splicing the model parameter matrix, there are two ways: horizontal splicing and vertical splicing according to the shape of the parameter matrix. If d1≤d2 of the parameter matrix, horizontal splicing is used, otherwise vertical splicing is used.
[0053] Secondly, perform SVD decomposition on the concatenated parameter matrix W to obtain U, Σ, V T Three sub-matrices;
[0054] Again, according to the k value, U, Σ, V T The three sub-matrices are truncated to obtain U k ,Σ k 、 And combined into core matrix and adjustment matrix, according to the different splicing methods, divided into:
[0055] Horizontal splicing: core matrix A = U :,:k Σ :k,:k , coordination matrix The notation “:k” means cutting off the first k rows / columns from the corresponding dimension of the matrix, where U :,:k Indicates that the number of rows of matrix U remains unchanged and the first k columns are taken to form a new matrix with shape (d1, k); Σ :k,:k It means that the first k rows and columns of the matrix Σ are intercepted to form a new matrix (k, k); the core matrix A = U :,:k Σ :k,:k , the coordination matrix Among them U :,:k Represents the matrix formed by keeping the number of rows of matrix U unchanged and taking the first k columns; Σ :k,:k Indicates that the first k rows and columns of the matrix Σ are intercepted to form a new matrix; V T is a matrix with n*d2 columns, Represents the matrix V T Cut out n matrices from the matrix to form the coordination matrix B1, B2, ..., B n , the number of rows of these n matrices is k, so the subscript: k means from V T The first k rows are intercepted, and then every d2 columns are intercepted to form a coordination matrix. When i=1, (i-1)d2:i·d2=0:d2, which means intercepting the first d2 columns. When i=2, (i-1)d2:i·d2=d2:2d2, which means intercepting from the d2+1 column to the 2*d2 column, and so on.
[0056] Vertical splicing: core matrix Coordination Matrix where Σ :k,:k and Denotes the pair of matrices Σ and V T The first k rows and columns of are intercepted to form a new matrix of (k, k); the core matrix The coordination matrix Among them, Σ :k,:k and Denotes the pair of matrices Σ and V T The rows and columns of truncate the first k matrices; U is a matrix with n*d1 rows. It means that n matrices are cut out from the matrix U to form the coordination matrix A1, A2, ..., A n The number of columns of these n matrices is k. The subscript :k means that the first k columns are intercepted from U, and then every d1 rows are intercepted to form a coordination matrix. When i = 1, (i-1)d1:i·d1=0:d1, which means that the first d1 rows are intercepted. When i = 2, (i-1)d1:i·d1=d1:2d1, which means that the rows from d1+1 to 2*d1 are intercepted, and so on.
[0057] The compression loss caused by truncating the smallest k singular values is also the smallest. The formula for calculating the number of singular values required to retain the low-rank decomposition is as follows:
[0058]
[0059] Among them, d1 and d2 are the number of rows and columns of the parameter matrix, n is the number of layers participating in the cross-layer low-rank decomposition, and x is the given target compression ratio.
[0060] Finally, the n parameter matrices originally involved in the cross-layer parameter low-rank decomposition are decomposed into a core matrix A and n coordination matrices B i After that, the parameter matrices participating in the cross-layer low-rank decomposition share the core matrix, while the coordination matrix is unique to each layer. For example, A is shared by the n layers, that is, the original parameter matrix can be expressed as W i ≈AB i , that is, multiply A by B i Calculate the parameter matrix. When n=2 is selected, the model structure is as follows: Figure 4 As shown, the original large parameter matrix is decomposed into two smaller parameter matrices, and the core matrix is shared across layers.
[0061] In this embodiment, the layers participating in the cross-layer parameter low-rank decomposition are adjacent layers in the large language model.
[0062] The pre-trained large language models supported by this embodiment are large language models with a decoder-only structure, including the Llama series, GTP series, and OPT series.
[0063] The present invention provides a large language model inference optimization method based on cross-layer parameter low-rank decomposition. It uses Cholesky decomposition to perform data whitening on the parameter matrix of a pre-trained large language model, and uses a cross-layer parameter low-rank decomposition algorithm to achieve low-rank decomposition of the parameter matrix while maintaining high precision, reducing memory overhead during model inference and further optimizing computing speed by sharing the core matrix. The present invention meets the low-cost and high-efficiency deployment and inference requirements of users of pre-trained large language models.
[0064] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.
[0065] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for optimizing inference of a pre-trained large language model based on cross-layer parameter low-rank decomposition, characterized in that: include: S1. Obtain a pre-trained large language model and a calibration dataset, and collect the feature embedding X of each layer parameter matrix during the calibration data inference process. S2. Generate a feature embedding whitening matrix S based on the feature embedding X of each layer parameter matrix; S3, using whitening matrix S and its inverse matrix S -1 , whiten the parameter matrix of each layer of the large language model; S4. Splice the parameter matrix of the large language model after whitening horizontally or vertically into the parameter matrix W; perform SVD decomposition on the parameter matrix W to obtain U, Σ, V T Three sub-matrices; calculate the number of singular values k required to retain the low-rank decomposition, and perform the calculations on U, Σ, and V according to the k value. T The three sub-matrices are truncated and combined into a core matrix and a regulation matrix; the parameter matrices participating in the cross-layer low-rank decomposition are made to share the core matrix, while the coordination matrix is unique to each layer. The parameter matrix of the large language model of the i-th layer is updated to obtain the optimized large language model.
2. The inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition according to claim 1 is characterized in that: In step S2, Cholesky decomposition is used to generate a whitening matrix S of feature embedding for the feature embedding X.
3. The inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition according to claim 2 is characterized in that: The generation process of the whitening matrix S is: Initialize the whitening matrix S to a lower triangular matrix of 0, and for each row i = 1, 2, ..., n, calculate the i-th row element of S For each column j=i+1,i+2,…,n, calculate:
4. The inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition according to claim 1, characterized in that: The k value is: Among them, d1 and d2 are the number of rows and columns of the parameter matrix, n is the number of layers participating in the cross-layer low-rank decomposition, and x is the given target compression ratio.
5. The inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition according to claim 1 or 4, characterized in that: If the parameter matrix d1≤d2, horizontal splicing is used, otherwise vertical splicing is performed.
6. The inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition according to claim 5, characterized in that: When horizontal splicing is used, The core matrix A=U :,:k Σ :k,:k , the coordination matrix Among them U :,:k Represents the matrix formed by keeping the number of rows of matrix U unchanged and taking the first k columns; Σ :k,:k Indicates that the first k trips of the rows and columns of the matrix Σ are intercepted into a new matrix; V T is a matrix with n*d2 columns, Represents the matrix V T Cut out n matrices from the matrix to form the coordination matrix B1, B2, ..., B n , the number of rows of these n matrices is k, so the subscript: k means from V T The first k rows are intercepted, and then every d2 columns are intercepted to form a coordination matrix. When i=1, (i-1)d2:i·d2=0:d2, which means intercepting the first d2 columns. When i=2, (i-1)d2:i·d2=d2:2d2, which means intercepting from the d2+1 column to the 2*d2 column, and so on.
7. The inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition according to claim 5, characterized in that: When longitudinal splicing is used, The core matrix The coordination matrix Among them, Σ :k,:k and Denotes the pair of matrices Σ and V T The rows and columns of truncate the first k matrices; U is a matrix with n*d1 rows. It means that n matrices are cut out from the matrix U to form the coordination matrix A1, A2, ..., A n The number of columns of these n matrices is k. The subscript :k means that the first k columns are intercepted from U, and then every d1 rows are intercepted to form a coordination matrix. When i = 1, (i-1)d1:i·d1=0:d1, which means that the first d1 rows are intercepted. When i = 2, (i-1)d1:i·d1=d1:2d1, which means that the rows from d1+1 to 2*d1 are intercepted, and so on.
8. The inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition according to claim 6 or 7, characterized in that: The parameter matrix of the updated large language model at layer i is matrix A multiplied by matrix B i . .
9. The inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition according to claim 1, characterized in that: The layers participating in the cross-layer parameter low-rank decomposition are adjacent layers in the large language model.
10. The inference optimization method for a pre-trained large language model based on cross-layer parameter low-rank decomposition according to claim 1, characterized in that: The pre-trained large language model is a large language model with a decoder-only structure, including the Llama series, GTP series, and OPT series.