Large language model fine tuning method and device based on shared gradient low-rank projection

By using the shared gradient low-rank projection method, the problem of high computational cost in fine-tuning large language models is solved, which reduces memory usage and shortens fine-tuning time, thereby improving fine-tuning efficiency.

CN121031666APending Publication Date: 2025-11-28GUANGDONG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511190511.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-08-25
Publication Date
2025-11-28

AI Technical Summary

Technical Problem

Large language models require a large amount of computation during fine-tuning, resulting in low fine-tuning efficiency.

Method used

The shared gradient low-rank projection method is adopted. The loss function value and the number of iterations are determined by fine-tuning the dataset. It is then determined whether the iteration stopping condition is met, the shared projection matrix is ​​updated, the Adam optimizer is used to optimize the subspace gradient, and the model weight matrix is ​​updated.

Benefits of technology

It reduces video memory usage, shortens fine-tuning time, and improves model fine-tuning efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121031666A_ABST
    Figure CN121031666A_ABST
Patent Text Reader

Abstract

The invention discloses a large language model fine tuning method and device based on shared gradient low-rank projection, and relates to the technical field of model fine tuning, and the method comprises the steps: inputting a fine tuning data set into a to-be-fine-tuned large language model to determine a loss function value, updating the number of iterations of the model, and judging whether a model iteration stop condition is satisfied or not; if not, determining a current gradient matrix of the model weight matrix according to the loss function value, and judging whether to update a shared projection matrix of the current gradient matrix; if the current gradient matrixes are updated, dividing the current gradient matrixes with the same matrix shape into gradient matrix groups with the same shape, and extracting gradient public information of each group to construct a corresponding new shared projection matrix; performing low-rank projection on the associated current gradient matrix by adopting the current shared projection matrix to determine a corresponding subspace gradient; and after optimizing each subspace gradient, carrying out re-projection and correspondingly updating a model weight matrix until a model iteration stopping condition is met. Based on the scheme, the fine adjustment efficiency of the model can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model fine-tuning, in particular to a large language model fine-tuning method and device based on shared gradient low-rank projection. BACKGROUND

[0002] In recent years, large language models (LLMs) have shown excellent performance in natural language processing, machine translation and other tasks. As the application scenarios continue to expand, the model size has jumped from millions to billions, and even to hundreds of billions. This scale expansion has brought huge training costs and memory pressure. The current mainstream optimizer for model training is the Adam optimizer, which needs to save a large amount of information such as model weights, current gradients and optimizer state parameters during training, resulting in much higher memory overhead than model parameters themselves.

[0003] To reduce the memory overhead during training, researchers have proposed various strategies such as activation checkpointing, optimizer quantization, gradient accumulation, parameter efficient fine-tuning (e.g., LoRA), and gradient low-rank projection. Among them, the gradient low-rank projection (GaLore) method uses a projection matrix to project the gradient matrix to a low-dimensional subspace to reduce the optimizer state parameters generated by the Adam optimizer by reducing the input gradient parameters. For example, the optimizer for low-dimensional gradient state adaptive optimization (LDAdam) is used to reduce the projection error. This method requires that the projection matrix can extract the information of the gradient matrix to the greatest extent. However, each iteration of model training causes the gradient to change, which requires the projection matrix to be updated regularly. However, the calculation required for updating the projection matrix is large, resulting in low efficiency of model fine-tuning. SUMMARY

[0004] The present application provides a large language model fine-tuning method and device based on shared gradient low-rank projection, which solves the technical problem of large language model requiring large amount of calculation during model fine-tuning, resulting in low efficiency of model fine-tuning.

[0005] The first aspect of the present application provides a large language model fine-tuning method based on shared gradient low-rank projection, comprising:

[0006] The fine-tuning data set is input into the large language model to be fine-tuned to determine the loss function value, update the model iteration number, and determine whether the model iteration stop condition is met based on the loss function value and the model iteration number;

[0007] When the model iteration stop condition is not met, the current gradient matrix of the model weight matrix is determined according to the loss function value, and the shared projection matrix of the current gradient matrix is determined according to the model iteration number;

[0008] If updated, the current gradient matrix of the same matrix shape is divided into the same shape gradient matrix group, and a new shared projection matrix corresponding to each of the same shape gradient matrix group is constructed by extracting gradient public information according to each of the same shape gradient matrix group;

[0009] The current shared projection matrix is used for low-rank projection on the associated current gradient matrix to determine the subspace gradient of each of the current gradient matrix;

[0010] After each of the subspace gradient is optimized by the Adam optimizer, the optimized gradient matrix is determined by re-projecting;

[0011] According to each of the current optimized gradient matrix, the model weight matrix of the large language model to be fine-tuned is updated until the model iteration stop condition is met.

[0012] Further, the new shared projection matrix corresponding to each of the same shape gradient matrix group is constructed by extracting gradient public information according to each of the same shape gradient matrix group, comprising:

[0013] Randomly sampling and information fusion are performed on the current gradient matrix of any of the same shape gradient matrix groups to determine the corresponding gradient fusion matrix;

[0014] Each of the gradient fusion matrix is used for Krylov iteration update to determine the corresponding matrix sequence;

[0015] Each of the matrix sequence is QR decomposed to construct the corresponding orthogonal basis sequence;

[0016] The correlation matrix of the first two orthogonal bases in each of the orthogonal basis sequence is calculated;

[0017] Each of the correlation matrix is subjected to truncated singular value decomposition to generate the corresponding truncated left singular matrix and truncated right singular matrix;

[0018] Based on the first two orthogonal bases in each of the orthogonal basis sequence and the associated truncated left singular matrix and truncated right singular matrix, the corresponding to-be-orthogonalized common basis is determined;

[0019] Each of the to-be-orthogonalized common basis is orthogonalized, and the output orthogonal basis is taken as the corresponding initial common basis;

[0020] Each of the initial common basis is iteratively optimized by a known common component general orthogonal basis extraction algorithm to determine the corresponding shared projection matrix.

[0021] Further, the to-be-orthogonalized common basis corresponding to each of the first two orthogonal bases in each of the orthogonal basis sequence and the associated truncated left singular matrix and truncated right singular matrix is determined, comprising:

[0022] The first two orthogonal bases in each of the orthogonal basis sequence are taken as the first orthogonal basis and the second orthogonal basis.

[0023] For any of the orthogonal basis sequences, the first product value of the first orthogonal basis and the truncated left singular matrix, and the second product value of the second orthogonal basis and the truncated right singular matrix are subjected to mean operation to determine the corresponding common basis to be orthogonalized.

[0024] Further, the iterative optimization of each of the initial common bases by the known common component general orthogonal basis extraction algorithm to determine the corresponding shared projection matrix comprises:

[0025] Determine the gradient matrix number of each of the same shape gradient matrix groups;

[0026] Extract the orthogonal bases from each of the orthogonal basis sequences in order of the associated gradient matrix number and construct the orthogonal projection operators one by one;

[0027] Respectively, the orthogonal projection operators of each of the same shape gradient matrix groups and the candidate common bases of the last round of projection optimization iteration are subjected to projection accumulation operation to construct the common basis projection matrix of each of the same shape gradient matrix groups in the current round of projection optimization iteration; wherein, the initial common basis is the candidate common basis of the first round of projection optimization iteration;

[0028] SVD decomposition is performed on each of the common basis projection matrices, and the product operation of the associated output left singular matrix and the transpose of the right singular matrix is performed to correspondingly construct the candidate common bases of the current round of projection optimization iteration;

[0029] The candidate common bases of the current round of projection optimization iteration with a change amplitude less than a change threshold are taken as the shared projection matrix.

[0030] Further, the determination of whether the model iteration stopping condition is met based on the loss function value and the model iteration number comprises:

[0031] Determine whether the loss function value converges;

[0032] If it converges, it is determined that the model iteration stopping condition is met;

[0033] If it does not converge, it is determined whether the model iteration number meets a number threshold;

[0034] When the number threshold is met, it is determined that the model iteration stopping condition is met;

[0035] When the number threshold is not met, it is determined that the model iteration stopping condition is not met.

[0036] Further, the determination of whether to update the shared projection matrix of the current gradient matrix according to the model iteration number comprises:

[0037] Determine whether the model iteration number can be divided by a preset hyperparameter update frequency.

[0038] if divisible, update the shared projection matrix of the current gradient matrix;

[0039] if not divisible, do not update the shared projection matrix of the current gradient matrix.

[0040] The second aspect of the present application provides a large language model fine-tuning device based on shared gradient low-rank projection, comprising:

[0041] The model operation module is configured to input a large language model to be fine-tuned using a fine-tuning data set to determine a loss function value, update the number of model iterations, and determine whether the model iteration stop condition is met based on the loss function value and the number of model iterations.

[0042] The update judgment module is configured to determine the current gradient matrix of the model weight matrix according to the loss function value when the model iteration stop condition is not met, and determine whether to update the shared projection matrix of the current gradient matrix according to the number of model iterations.

[0043] The matrix update module is configured to, if updated, divide the current gradient matrix with the same matrix shape into a same shape gradient matrix group, construct a corresponding new shared projection matrix according to the gradient public information extracted from each same shape gradient matrix group.

[0044] The gradient projection module is configured to perform low-rank projection on the associated current gradient matrix using the current shared projection matrix to determine the subspace gradient of each current gradient matrix.

[0045] The gradient optimization module is configured to optimize each subspace gradient through an Adam optimizer and then perform re-projection to determine an optimized gradient matrix.

[0046] The parameter update module is configured to update the model weight matrix of the large language model to be fine-tuned according to each current optimized gradient matrix until the model iteration stop condition is met.

[0047] The third aspect of the present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the large language model fine-tuning method based on shared gradient low-rank projection according to any one of the above aspects.

[0048] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to implement the large language model fine-tuning method based on shared gradient low-rank projection according to any one of the above aspects.

[0049] The fifth aspect of the present application provides a computer program product comprising computer programs / instructions which, when executed by a processor, implement the large language model fine-tuning method based on shared gradient low-rank projection as described in any of the above.

[0050] From the above technical solutions, the present application has the following advantages:

[0051] The above-mentioned scheme of the present application provides a large language model fine-tuning method based on shared gradient low-rank projection, comprising: inputting a fine-tuning data set into a large language model to be fine-tuned to determine a loss function value, updating the number of model iterations, and determining whether the model iteration stop condition is met based on the loss function value and the number of model iterations; when the model iteration stop condition is not met, determining the current gradient matrix of the model weight matrix according to the loss function value, and determining whether to update the shared projection matrix of the current gradient matrix according to the number of model iterations; if yes, the current gradient matrix with the same matrix shape is divided into a same shape gradient matrix group, a new shared projection matrix is constructed according to the gradient public information extracted from each same shape gradient matrix group; using the current shared projection matrix to perform low-rank projection on the associated current gradient matrix to determine the subspace gradient of each current gradient matrix; after optimizing each subspace gradient by an Adam optimizer, re-projecting to determine an optimized gradient matrix; updating the model weight matrix of the large language model to be fine-tuned according to each current optimized gradient matrix until the model iteration stop condition is met. Based on the above-mentioned scheme, the shared projection matrix is used instead of the independent projection matrix in the gradient low-rank projection, and there is no need to store the projection matrix for each gradient matrix separately, which reduces the occupation of the display memory, greatly shortens the fine-tuning time, and helps to improve the fine-tuning efficiency of the model. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0053] Figure 1 A step flowchart of a large language model fine-tuning method based on shared gradient low-rank projection is provided for the first embodiment of the present application.

[0054] Figure 2 A flowchart of a large language model fine-tuning method based on shared gradient low-rank projection is provided for the first embodiment of the present application.

[0055] Figure 3A structural block diagram of a large language model fine-tuning device based on shared gradient low-rank projection is provided for the second embodiment of the present application. DETAILED DESCRIPTION

[0056] The second embodiment of the present application provides a large language model fine-tuning method and device based on shared gradient low-rank projection, which is used to solve the technical problem that the large language model requires a large amount of calculation when performing model fine-tuning, resulting in low model fine-tuning efficiency.

[0057] In order to make the purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the following described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0058] Please refer to Figure 1 and Figure 2 The first embodiment of the present application provides a large language model fine-tuning method based on shared gradient low-rank projection, which comprises:

[0059] Step 101, inputting the fine-tuning data set into the large language model to be fine-tuned to determine the loss function value, updating the model iteration number, and judging whether the model iteration stop condition is met based on the loss function value and the model iteration number.

[0060] The fine-tuning data set refers to the data set for further training of the large language model for a specific task.

[0061] The loss function value refers to the index value of the difference between the predicted result and the true result of the model.

[0062] The model iteration number can represent the number of times of training using fine-tuning sample data in the model fine-tuning process.

[0063] The model iteration stop condition refers to the determination condition for stopping the model fine-tuning iteration.

[0064] It should be noted that when the model fine-tuning is carried out, the fine-tuning data set and the large language model to be fine-tuned are first obtained, which can be selected according to the specific task, for example, Figure 2As shown, the RoBERTa-base model is loaded as the large language model to be fine-tuned, including loading the weight file and configuration file of the RoBERTa-base, and loading the GLUE fine-tuning data according to the requirements of language understanding evaluation and other tasks, and obtaining the fine-tuning sample data after preprocessing the GLUE fine-tuning data to form a fine-tuning dataset, the preprocessing can include performing word segmentation, truncation, padding, generating input id and attention mask and the like; the fine-tuning dataset is usually divided into multiple fine-tuning batches, each fine-tuning batch contains a certain number of fine-tuning sample data, then a fine-tuning batch is taken from the fine-tuning dataset to input into the large language model to be fine-tuned to perform forward calculation to determine the prediction result of the model output, and the loss function value is calculated based on the prediction result and the corresponding true result.

[0065] In one specific embodiment of the present embodiment, whether the model iteration stop condition is met is determined based on the loss function value and the number of model iterations, including:

[0066] S11, determining whether the loss function value converges;

[0067] S12, if the loss function value converges, determining that the model iteration stop condition is met;

[0068] S13, if the loss function value does not converge, determining whether the number of model iterations meets a number threshold;

[0069] S14, when the number threshold is met, determining that the model iteration stop condition is met;

[0070] S15, when the number threshold is not met, determining that the model iteration stop condition is not met.

[0071] It should be noted that when determining the model iteration stop condition, the loss function value and the number of model iterations can be analyzed comprehensively, and other mechanisms can also be considered, which are not limited by the present embodiment. It can be understood that the number threshold can be the maximum number of iterations, etc.

[0072] Step 102, when the model iteration stop condition is not met, determining the current gradient matrix of the model weight matrix according to the loss function value, and determining whether to update the shared projection matrix of the current gradient matrix according to the number of model iterations.

[0073] The model weight matrix refers to the learnable parameter matrix stored in the model.

[0074] The current gradient matrix refers to the gradient matrix obtained in the current iteration.

[0075] The shared projection matrix refers to the matrix shared by multiple gradient matrices for low-rank projection.

[0076] It should be noted that if the model iteration stop condition is not met, the model parameter needs to be updated, and the parameter update design based on the gradient low-rank projection is adopted in the embodiment. Therefore, the current gradient matrix is first determined based on the partial derivative of the loss function value to the model weight matrix, and in order to reduce the occupation of the display memory, the current gradient matrix is divided into groups, and a shared projection matrix for low-rank projection is determined for each group, which can reduce the calculation amount of the updated projection matrix, and the periodic update of the shared projection matrix is determined by the number of model iterations.

[0077] In one specific embodiment of the present embodiment, whether to update the shared projection matrix of the current gradient matrix according to the number of model iterations, comprising:

[0078] S21, determining whether the number of model iterations can be divided by the preset hyperparameter update frequency;

[0079] S22, if it can be divided, updating the shared projection matrix of the current gradient matrix;

[0080] S23, if it cannot be divided, not updating the shared projection matrix of the current gradient matrix.

[0081] The hyperparameter update frequency refers to the update period of the hyperparameters of the model. The hyperparameters are understood as parameters set to control the fine-tuning training process of the model, such as batch size, learning rate, etc. Some hyperparameters will be adjusted according to the iteration process, such as learning rate, etc.

[0082] It should be noted that the present embodiment sets a hyperparameter update frequency T for the model fine-tuning process, for example, T = 100. When the number of model fine-tuning iterations can be divided by the hyperparameter update frequency T, the current shared projection matrix is updated, otherwise, the shared projection matrix of the last iteration is continued to be used.

[0083] Step 103, if it is updated, the current gradient matrix with the same matrix shape is divided into the same shape gradient matrix group, and the corresponding new shared projection matrix is constructed according to the gradient public information extracted from each same shape gradient matrix group.

[0084] The matrix shape refers to the shape formed by the number of rows and the number of columns of the matrix.

[0085] The same shape gradient matrix group refers to a combination of gradient matrices with the same matrix shape.

[0086] The gradient public information refers to the common information between different gradient matrices.

[0087] It should be noted that if the current shared projection matrix needs to be updated, first the current gradient matrix of the same shape is divided into the same isomorphic gradient matrix group, and for the gradient matrix of the same shape , is the gradient matrix index, is the first current gradient matrix, is the gradient matrix row size, is the gradient matrix column size, and the shared projection matrix is constructed by extracting gradient common information.

[0088] In one specific embodiment of the present embodiment, the gradient common information is extracted according to each isomorphic gradient matrix group to construct a corresponding new shared projection matrix, comprising:

[0089] S31, randomly sampling and information fusion are performed on the current gradient matrix of any isomorphic gradient matrix group to determine the corresponding gradient fusion matrix;

[0090] S32, Krylov iteration update is performed using each gradient fusion matrix to determine the corresponding matrix sequence;

[0091] S33, QR decomposition is performed on each matrix sequence to construct the corresponding orthogonal basis sequence;

[0092] S34, the correlation matrix of the first two orthogonal bases in each orthogonal basis sequence is calculated;

[0093] S35, truncated singular value decomposition is performed on each correlation matrix to generate the corresponding truncated left singular matrix and truncated right singular matrix;

[0094] S36, based on the first two orthogonal bases in each orthogonal basis sequence and the associated truncated left singular matrix and truncated right singular matrix, the corresponding to-be-orthogonalized common basis is determined;

[0095] S37, orthogonalization is performed on each to-be-orthogonalized common basis, and the output orthogonal basis is taken as the corresponding initial common basis;

[0096] S38, iterative optimization is performed on each initial common basis through the general orthogonal basis extraction algorithm of the known common component to determine the corresponding shared projection matrix.

[0097] In a more specific embodiment of the present embodiment, sub-step S36 comprises:

[0098] S361, the first two orthogonal bases in each orthogonal basis sequence are taken as the first orthogonal basis and the second orthogonal basis;

[0099] ​S362. For any orthogonal basis sequence, the first product of the first orthogonal basis and the truncated left singular matrix, and the second product of the second orthogonal basis and the truncated right singular matrix are used to perform mean operation to determine the corresponding common basis to be orthogonalized.

[0100] In a more specific embodiment of this example, sub-step S38 includes:

[0101] S381. Determine the number of gradient matrices in each group of gradient matrices with the same shape;

[0102] S382. Extract orthogonal bases from each orthogonal basis sequence in order of the number of associated gradient matrices and construct orthogonal projection operators one by one;

[0103] S383. The orthogonal projection operators of each gradient matrix group with the same shape are used to perform projection accumulation operations with the candidate common basis of the previous projection optimization iteration to construct the common basis projection matrix of each gradient matrix group with the same shape in the current projection optimization iteration; wherein, the initial common basis is the candidate common basis of the first projection optimization iteration.

[0104] S384. Perform SVD decomposition on the projection matrices of each common basis, and multiply the left singular matrix and the transpose of the right singular matrix of the associated output to construct the candidate common basis for the current round of projection optimization iteration.

[0105] S385. The candidate common basis of the current round of projection optimization iteration with a change magnitude less than the change threshold is used as the shared projection matrix.

[0106] It should be noted that the current gradient matrix of any group of gradient matrices of the same shape uses a general orthogonal basis extraction algorithm (rBKI-COBEC) containing a known common component c of random block Krylov iterations to extract common information of the gradients in order to update the shared projection matrix. The algorithm process is shown in Table 1:

[0107] Table 1 rBKI-COBEC Algorithm

[0108]

[0109] As shown in Table 1, information extraction is first performed using randomized Krylov iteration; then, a random projection matrix is ​​created that follows a distribution with a mean of 0 and a variance of 1. , For gradient matrix indexing, It is a random projection matrix. For public component hyperparameters, For oversampling hyperparameters; create an all-zero gradient fusion matrix. The process of randomly sampling and fusing information from the current gradient matrix is ​​expressed by the following formula:

[0110]

[0111] where, is the gradient fusion matrix, is the gradient matrix index, is the number of gradient matrices in the same shape gradient matrix group, is the current gradient matrix, is the random projection matrix; the matrix is the Kronecker subspace computation, and the resulting list is shown as follows:

[0112]

[0113] where, is the matrix sequence, is the transpose symbol, is the polynomial degree index; according to experience, is sufficient to achieve satisfactory approximation accuracy; for each term of , QR decomposition is performed to obtain a set of standard orthogonal bases: , where, is the orthogonal basis sequence, is the th orthogonal basis in the orthogonal basis sequence, and ,

[0114] the purpose is to make subsequent CCA initialization and subspace-based iterative update more numerically stable and efficient; The common information of the gradient matrix is extracted by the common orthogonal basis extraction method (COBEC) with known common components c; the first two terms of are taken for canonical correlation analysis initialization (CCA), the purpose of which is to find the most relevant dimensional subspace between the two terms as the initial common basis ; the steps of CCA initialization are as follows:

[0115] First, the correlation matrix of the two orthogonal bases is calculated: , is the correlation matrix, is the transpose symbol, ;

[0116] Truncated singular value decomposition is performed on , and the first principal direction is taken: , , is a left-singular matrix, is a singular value matrix, is a right-singular matrix, is a singular value decomposition, is a truncated left-singular matrix, is a truncated right-singular matrix; wherein, , These two matrices represent the directions that make the first orthogonal basis and the second orthogonal basis most relevant; project and onto the dimensional subspace respectively and average to balance the information of the two subspaces to get the common basis to be orthogonalized , the specific process is as follows: , ; orthogonalize to get the final initial common basis: , is the initial common basis, is an upper triangular matrix of the common basis to be orthogonalized, is a QR decomposition, wherein ;

[0117] Iteratively optimize the initial common basis using the known common component extraction method (COBEC) of the general orthogonal basis to make the final shared projection matrix be able to extract the common information of all gradient matrices; specifically, first construct a common basis projection matrix , The calculation is as follows:

[0118]

[0119] In the formula, is the index of the projection optimization iteration, is the common basis projection matrix of the th projection optimization iteration, is the candidate common basis of the th projection optimization iteration, is the gradient matrix index, is the number of gradient matrices in the same shape gradient matrix group, is an orthogonal projection operator; then perform SVD decomposition on : , is the left-singular matrix of the common basis projection matrix, is the singular value matrix of the common basis projection matrix, is a right singular matrix of the public basis projection matrix, is a singular value decomposition;

[0120] Let the first iteration of the projection optimization candidate public basis , when the change amplitude is less than the change threshold, that is , it is considered that has converged, and , and finally the shared projection matrix extracting all gradient public information is obtained .

[0121] Step 104, using the current shared projection matrix to perform low-rank projection on the associated current gradient matrix to determine the subspace gradient of each current gradient matrix.

[0122] It should be noted that the current shared projection matrix is the latest shared projection matrix retained in the iteration process. If the shared projection matrix needs to be updated in the foregoing steps, the updated shared projection matrix is used, and if it does not need to be updated, the shared projection matrix of the last iteration is continued to be used. The shared projection matrix is used to perform low-rank projection on each gradient matrix to obtain the subspace gradient: , wherein the subspace gradient , is a given matrix rank.

[0123] It should be noted that for the parameter update part of the shared projection matrix, the foregoing embodiment is realized by truncated singular value decomposition. In another implementation manner, a random parameter can also be used to update the projection matrix, but the selection requirement of the random seed is very strict, and only a small number of random seeds can achieve the same performance.

[0124] Step 105, re-projecting after optimizing each subspace gradient by an Adam optimizer to determine an optimized gradient matrix.

[0125] It should be noted that the standard Adam optimizer is used to update the subspace gradient: , is is the optimized subspace gradient, and the optimized subspace gradient is re-projected back to the original space: , is the optimized gradient matrix, wherein .

[0126] Step 106, updating the model weight matrix of the fine-tuning large language model according to each current optimized gradient matrix until the model iteration stop condition is met.

[0127] It should be noted that the model weight matrix is updated by using the optimized gradient matrix: , is an index of the number of model iterations, is the model weight matrix of the i-th iteration, is the learning rate; continue to select data from the fine-tuning data set Next fine-tuning batch, execute steps 101-106, until the model iteration stop condition is met, then complete the model fine-tuning.

[0128] In order to better illustrate the technical effect of the embodiment, an experiment is compared with the existing fine-tuning method, and the experimental results are as follows:

[0129] 1、The embodiment extracts the common information of all gradient matrices, projects the gradient matrix to the common subspace, avoids the huge calculation amount consumed by separately calculating the independent subspace of each gradient matrix, and greatly improves the calculation speed. The fine-tuning time of the prior art in multiple data sets is as shown in the following table:

[0130] Table 2 Fine-tuning time comparison

[0131]

[0132] In the table, RTE and SST2 are different fine-tuning data sets, Adam is the original optimizer, Galore, GaRare and LDAdam are existing gradient low-rank projection-based optimizers, and LoCo is the scheme of the embodiment. As can be seen from Table 2, compared with the existing gradient low-rank projection scheme, the embodiment is faster, close to the original optimizer;

[0133] 2、The embodiment does not need to store the projection matrix separately for each gradient matrix, further reducing the memory occupation, as shown in the following table:

[0134] Table 3 Memory occupation comparison

[0135]

[0136] Wherein, m and n are the rows and columns of the gradient matrix respectively, and r is the given matrix rank; Since the shared projection matrix is only called when needed, and not stored separately for each weight, the memory overhead brought by LoCo can be ignored compared to the size of the entire model, and the memory occupation is very close to GaRare, which is more memory-saving than other methods;

[0137] 3、Using RoBERTa-base model to compare the performance of different methods in multiple fine-tuning tasks, the performance is as shown in the following table:

[0138] Table 4 Task performance comparison ​

[0139]

[0140] Wherein, MRPC, STS-B, CoLA, SST2, QNLI, QQP, MNLI, RTE are fine-tuning data sets of different tasks, Avg is the average score of eight tasks, the higher the score represents the better performance, and the bold indicates the best result under the task; As can be seen from Table 4, the method of the embodiment can achieve similar or better performance with less calculation amount than other methods, and the average performance is optimal.

[0141] In the embodiment of the application, a shared projection matrix is used instead of an independent projection matrix in gradient low-rank projection, without the need to store a projection matrix for each gradient matrix, thereby reducing the occupation of the display memory; the common information of all gradient matrices in the same shape gradient matrix group is extracted using a common orthogonal basis extraction algorithm (COBEC) of a known public component c to update the shared projection matrix parameters, and the data preprocessing and information fusion steps of COBEC are improved using random block Krylov iteration to speed up the calculation speed of COBEC; therefore, an efficient model memory efficient fine-tuning method of gradient low-rank projection is provided as a whole, the fine-tuning time is greatly shortened, and it has great significance for improving the fine-tuning performance and efficiency of the model.

[0142] Please refer to Figure 3 The embodiment two of the application provides a large language model fine-tuning device based on shared gradient low-rank projection, which comprises:

[0143] The model operation module 301 is configured to input a fine-tuning data set into a large language model to be fine-tuned to determine a loss function value, update the number of model iterations, and determine whether a model iteration stop condition is met based on the loss function value and the number of model iterations;

[0144] The update judgment module 302 is configured to determine a current gradient matrix of a model weight matrix according to the loss function value when the model iteration stop condition is not met, and determine whether to update a shared projection matrix of the current gradient matrix according to the number of model iterations;

[0145] The matrix update module 303 is configured to, if the shared projection matrix is to be updated, divide the current gradient matrix of the same matrix shape into a same shape gradient matrix group, and construct a corresponding new shared projection matrix according to the gradient common information of each same shape gradient matrix group;

[0146] The gradient projection module 304 is configured to perform low-rank projection on the associated current gradient matrix by using the current shared projection matrix to determine the subspace gradient of each current gradient matrix;

[0147] The gradient optimization module 305 is configured to optimize each subspace gradient through an Adam optimizer and then perform re-projection to determine an optimized gradient matrix.

[0148] The parameter updating module 306 is configured to update the model weight matrix of the large language model to be fine-tuned according to each current optimization gradient matrix until a model iteration stop condition is met.

[0149] In one specific embodiment of the present embodiment, a new shared projection matrix corresponding to each same-shape gradient matrix group is constructed by extracting gradient common information, including:

[0150] Randomly sampling and fusing information of the current gradient matrix of any same-shape gradient matrix group to determine a corresponding gradient fusion matrix;

[0151] Performing Krylov iteration updating using each gradient fusion matrix to determine a corresponding matrix sequence;

[0152] Performing QR decomposition on each matrix sequence to construct a corresponding orthogonal basis sequence;

[0153] Calculating the correlation matrix of the first two orthogonal bases in each orthogonal basis sequence;

[0154] Performing truncated singular value decomposition on each correlation matrix to generate a corresponding truncated left singular matrix and a truncated right singular matrix;

[0155] Based on the first two orthogonal bases in each orthogonal basis sequence and the associated truncated left singular matrix and truncated right singular matrix, determining a corresponding common basis to be orthogonalized;

[0156] Orthogonalizing each common basis to be orthogonalized, and taking the output orthogonal basis as a corresponding initial common basis;

[0157] Iteratively optimizing each initial common basis through a general orthogonal basis extraction algorithm of a known common component to determine a corresponding shared projection matrix.

[0158] In a more specific embodiment of the present embodiment, based on the first two orthogonal bases in each orthogonal basis sequence and the associated truncated left singular matrix and truncated right singular matrix, determining a corresponding common basis to be orthogonalized, includes:

[0159] Taking the first two orthogonal bases in each orthogonal basis sequence as a first orthogonal basis and a second orthogonal basis;

[0160] For any orthogonal basis sequence, performing mean operation on a first product value of the first orthogonal basis and the truncated left singular matrix, and a second product value of the second orthogonal basis and the truncated right singular matrix to determine a corresponding common basis to be orthogonalized.

[0161] In a more specific embodiment of the present embodiment, iteratively optimizing each initial common basis through a general orthogonal basis extraction algorithm of a known common component to determine a corresponding shared projection matrix, includes:

[0162] determining the number of gradient matrices of each isomorphic gradient matrix group;

[0163] extracting the orthogonal bases from each orthogonal base sequence in order of the associated number of gradient matrices and constructing the orthogonal projection operators one by one;

[0164] respectively performing projection accumulation operation on each orthogonal projection operator of the isomorphic gradient matrix group and the candidate common base of the last round of projection optimization iteration to construct the common base projection matrix of each isomorphic gradient matrix group in the current round of projection optimization iteration; wherein, the initial common base is the candidate common base of the first round of projection optimization iteration;

[0165] performing SVD decomposition on each common base projection matrix, performing product operation on the left singular matrix and the transpose of the right singular matrix output correspondingly to construct the candidate common base of the current round of projection optimization iteration;

[0166] taking the candidate common base of the current round of projection optimization iteration with a change amplitude less than a change threshold as the shared projection matrix.

[0167] In one specific embodiment of the present embodiment, whether the model iteration stopping condition is met is determined based on the loss function value and the number of model iterations, including:

[0168] determining whether the loss function value converges;

[0169] if the loss function value converges, it is determined that the model iteration stopping condition is met;

[0170] if the loss function value does not converge, it is determined whether the number of model iterations meets a number threshold;

[0171] when the number threshold is met, it is determined that the model iteration stopping condition is met;

[0172] when the number threshold is not met, it is determined that the model iteration stopping condition is not met.

[0173] In one specific embodiment of the present embodiment, whether the shared projection matrix of the current gradient matrix is updated is determined according to the number of model iterations, including:

[0174] determining whether the number of model iterations can be divided by a preset hyperparameter update frequency;

[0175] if the number of model iterations can be divided by the preset hyperparameter update frequency, the shared projection matrix of the current gradient matrix is updated;

[0176] if the number of model iterations cannot be divided by the preset hyperparameter update frequency, the shared projection matrix of the current gradient matrix is not updated.

[0177] The embodiment three of the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program; the computer program is executed by the processor, so that the processor executes the steps of the large language model fine-tuning method based on shared gradient low-rank projection of any one of the above-mentioned embodiments.

[0178] The embodiment four of the present application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the large language model fine-tuning method based on shared gradient low-rank projection of any one of the above-mentioned embodiments.

[0179] The embodiment five of the present application further provides a computer program product, comprising a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the large language model fine-tuning method based on shared gradient low-rank projection of any one of the above-mentioned embodiments.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0181] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between the devices or modules, which can be electrical, mechanical or other forms.

[0182] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, that is, they can be located in one place, or can be distributed to multiple network modules. According to actual needs, some or all of the modules can be selected to achieve the purpose of the embodiment scheme.

[0183] In addition, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can exist physically, or two or more modules can be integrated in one module. The above integrated module can be realized in the form of hardware or in the form of software functional module.

[0184] The integrated module, if implemented in the form of a software function module and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0185] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for fine-tuning a large language model based on shared gradient low-rank projection, characterized in that, include: The loss function value is determined by inputting the fine-tuning dataset into the large language model to be fine-tuned, the model iteration number is updated, and it is determined whether the model iteration stopping condition is met based on the loss function value and the model iteration number. When the model iteration stopping condition is not met, the current gradient matrix of the model weight matrix is ​​determined based on the loss function value, and the shared projection matrix of the current gradient matrix is ​​determined based on the number of model iterations. If an update is needed, the current gradient matrices with the same shape are divided into groups of gradient matrices with the same shape, and the common gradient information is extracted according to each group of gradient matrices with the same shape to construct a new shared projection matrix. The associated current gradient matrix is ​​low-ranked by using the current shared projection matrix to determine the subspace gradient of each of the current gradient matrices; After optimizing the gradients of each subspace using the Adam optimizer, reprojection is performed to determine the optimized gradient matrix. Update the model weight matrix of the large language model to be fine-tuned according to the current optimized gradient matrix, until the model iteration stopping condition is met.

2. The method for fine-tuning a large language model based on shared gradient low-rank projection according to claim 1, characterized in that, The step of extracting common gradient information according to each of the same shape gradient matrix groups to construct a corresponding new shared projection matrix includes: Randomly sample and fuse the current gradient matrix of any of the gradient matrix groups of the same shape to determine the corresponding gradient fusion matrix. Krylov iteration update is performed using the gradient fusion matrices described above to determine the corresponding matrix sequence; Each of the matrix sequences is decomposed into QR sequences to construct the corresponding orthogonal basis sequences. Calculate the correlation matrix of the first two orthogonal bases in each of the orthogonal base sequences; Perform truncated singular value decomposition on each of the aforementioned correlation matrices to generate corresponding truncated left singular matrices and truncated right singular matrices; Based on the first two orthogonal bases in each of the orthogonal basis sequences and the associated truncated left singular matrix and truncated right singular matrix, the corresponding common basis to be orthogonalized is determined. Orthogonalize each of the aforementioned common bases to be orthogonalized, and use the output orthogonal bases as the corresponding initial common bases; The initial common basis is iteratively optimized using a general orthogonal basis extraction algorithm based on known common components to determine the corresponding shared projection matrix.

3. The method for fine-tuning a large language model based on shared gradient low-rank projection according to claim 2, characterized in that, The step of determining the corresponding common basis to be orthogonalized based on the first two orthogonal bases in each of the orthogonal basis sequences and the associated truncated left singular matrix and truncated right singular matrix includes: The first two orthogonal bases in each of the orthogonal base sequences are taken as the first orthogonal base and the second orthogonal base, respectively. For any of the orthogonal basis sequences, the first product of the first orthogonal basis and the truncated left singular matrix, and the second product of the second orthogonal basis and the truncated right singular matrix are used to perform mean operation to determine the corresponding common basis to be orthogonalized.

4. The method for fine-tuning a large language model based on shared gradient low-rank projection according to claim 2, characterized in that, The step of iteratively optimizing each initial common basis using a general orthogonal basis extraction algorithm based on known common components to determine the corresponding shared projection matrix includes: Determine the number of gradient matrices in each of the aforementioned gradient matrix groups of the same shape; Orthogonal bases are extracted from each of the orthogonal basis sequences in order of the number of associated gradient matrices, and orthogonal projection operators are constructed one by one; The orthogonal projection operators of each set of gradient matrices with the same shape are used to perform projection accumulation operations with the candidate common basis of the previous projection optimization iteration to construct the common basis projection matrix of each set of gradient matrices with the same shape in the current projection optimization iteration; wherein, the initial common basis is the candidate common basis of the first projection optimization iteration; The projection matrices of each common basis are decomposed by SVD, and the left singular matrix and the transpose of the right singular matrix of the associated output are multiplied to construct the candidate common basis for the current round of projection optimization iteration. The candidate common basis of the current round of projection optimization iterations with a change magnitude less than the change threshold is used as the shared projection matrix.

5. The method for fine-tuning a large language model based on shared gradient low-rank projection according to claim 1, characterized in that, The step of determining whether the model iteration stopping condition is met based on the loss function value and the number of model iterations includes: Determine whether the loss function value has converged; If convergence occurs, then the model iteration stopping condition is satisfied. If convergence is not achieved, determine whether the number of model iterations meets the threshold. When the threshold number of iterations is met, the model iteration stopping condition is determined to be met. If the threshold number of iterations is not met, the model iteration stopping condition is determined to be unmet.

6. The method for fine-tuning a large language model based on shared gradient low-rank projection according to claim 1, characterized in that, The step of determining whether to update the shared projection matrix of the current gradient matrix based on the number of model iterations includes: Determine whether the number of model iterations is divisible by the preset hyperparameter update frequency; If it is divisible, then update the shared projection matrix of the current gradient matrix; If it is not divisible, the shared projection matrix of the current gradient matrix is ​​not updated.

7. A fine-tuning device for a large language model based on shared gradient low-rank projection, characterized in that, include: The model computation module is used to input the large language model to be fine-tuned using the fine-tuning dataset to determine the loss function value, update the model iteration count, and determine whether the model iteration stopping condition is met based on the loss function value and the model iteration count. The update judgment module is used to determine the current gradient matrix of the model weight matrix based on the loss function value when the model iteration stopping condition is not met, and to determine whether to update the shared projection matrix of the current gradient matrix based on the number of model iterations. The matrix update module is used to divide the current gradient matrices with the same shape into groups of gradient matrices with the same shape if an update is required, and to extract the common gradient information of each group of gradient matrices with the same shape to construct a new shared projection matrix. The gradient projection module is used to perform low-rank projection of the associated current gradient matrix using the current shared projection matrix to determine the subspace gradient of each of the current gradient matrices. The gradient optimization module is used to optimize the gradients of each subspace using the Adam optimizer and then reproject them to determine the optimized gradient matrix. The parameter update module is used to update the model weight matrix of the large language model to be fine-tuned according to the current optimized gradient matrix, until the model iteration stopping condition is met.

8. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the large language model fine-tuning method based on shared gradient low-rank projection as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the large language model fine-tuning method based on shared gradient low-rank projection as described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the large language model fine-tuning method based on shared gradient low-rank projection as described in any one of claims 1-6.