Pre-training visual model parameter fine tuning method based on singular value

Abstract

The invention discloses a singular value-based pre-trained visual model parameter fine tuning method, which comprises the following steps of: for a pre-trained visual model, selecting levels which need to be subjected to parameter fine tuning from the visual model according to actual requirements, and respectively performing singular value decomposition on a weight matrix of each layer to be subjected to fine tuning; obtaining an optimal rank-r approximation matrix and a residual matrix used for reconstructing a weight matrix, forming a singular value vector by the first r singular values as a trainable parameter, freezing other model parameters, inputting an image sample in a training sample set into a visual model to update the trainable parameter, and obtaining an optimal rank-r approximation matrix and a residual matrix used for reconstructing the weight matrix; and reconstructing by adopting the updated trainable parameters to obtain a weight matrix. According to the method, singular value decomposition is adopted to initialize the low-rank matrix, learning of downstream tasks on new domain knowledge is enhanced while pre-training information is reserved, the number of trainable parameters is remarkably reduced, the knowledge and adaptability of the model are effectively reserved, and the fine adjustment effect of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and more specifically, relates to a method for fine-tuning parameters of a pre-trained visual model based on singular values. Background Art

[0002] With the development of large pre-trained vision models, their complexity and computational requirements make it increasingly challenging to adapt models with billions of parameters to specific downstream tasks. In recent years, parameter-efficient fine-tuning methods have attracted much attention for their ability to minimize the required parameters and memory requirements while maintaining efficiency and accuracy, thus approaching full fine-tuning in performance. The typical parameter-efficient fine-tuning method LoRA (Low-rank adaptation) approximates weight updates by introducing two low-rank matrices during fine-tuning, seamlessly integrating incremental updates into pre-trained weights without significantly increasing latency during inference. Based on LoRA, existing parameter-efficient fine-tuning methods are mainly divided into two categories. The first category of methods modifies the parameter update strategy, such as freezing part of the low-rank matrix in LoRA, or only propagating the incremental part of the low-rank matrix product to the pre-trained weights, thereby significantly reducing the trainable parameters and activation memory costs without increasing computational overhead. The second type of method adaptively adjusts the rank of LoRA, uses singular value decomposition to parameterize the incremental update, and adaptively adjusts the trainable parameters based on the importance score of each module during training, or controls the sparsity through gating of the proximal gradient method to expand the optimization space and improve parameter efficiency.

[0003] Although LoRA and subsequent methods have improved computational efficiency to a certain extent, the existing parameter fine-tuning methods still have some problems:

[0004] 1) Existing parameter fine-tuning methods usually use random initialization for low-rank matrices, which may result in the main components of the pre-trained weight matrix cannot be effectively updated in the early stage of fine-tuning, resulting in inefficient gradient descent and reduced model generalization ability due to suboptimal starting points.

[0005] 2) Existing methods usually focus on updating a limited subset of parameters, which may be part of the existing model parameters or a completely new set of parameters. The main purpose is to adapt the existing knowledge in the pre-trained large model to the specific task while preserving it, thereby minimizing the risk of catastrophic forgetting. However, there is still room for compression of the trainable parameters of these methods, and further improvement is needed. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for fine-tuning parameters of a pre-trained visual model based on singular values. Singular value decomposition is used to initialize a low-rank matrix. While retaining pre-training information, the learning of new domain knowledge by downstream tasks is enhanced. While significantly reducing the number of trainable parameters, the knowledge and adaptability of the model are effectively retained, thereby improving the model fine-tuning effect.

[0007] In order to achieve the above-mentioned object of the invention, the method for fine-tuning parameters of a pre-trained visual model based on singular values ​​of the present invention comprises the following steps:

[0008] S1: Obtain a training sample set for parameter fine-tuning according to actual needs, where each training sample includes an image sample and a label;

[0009] S2: For the pre-trained visual model, select the layer that needs to be fine-tuned from the visual model according to actual needs. The weight matrix of the layer to be fine-tuned is Where k = 1, 2, ..., K, K represents the number of layers to be fine-tuned, M k ×N k Represents the weight matrix size of the layer to be fine-tuned;

[0010] S3: The weight matrix W of each layer to be fine-tuned k Perform singular value decomposition and get the decomposition result:

[0011]

[0012] in, denote an orthogonal matrix, a diagonal matrix, and an orthogonal matrix, respectively, and the superscript T denotes transpose;

[0013] The number of singular values ​​obtained by singular value decomposition is R k =min(M k ,N k ), R k The singular values ​​are arranged in descending order, and the first r singular values ​​form a singular value vector Σ k,r =[λ k,1 ,λ k,2 ,…,λ k,r ], the value of r is set according to actual needs, and the residual singular value vector is obtained at the same time Let the orthogonal matrix U k The i-th eigenvector in is u k,i ,i=1,2,…,M k , let the orthogonal matrix V k The jth eigenvector in is v k,j ,j=1,2,…,N k ; Based on the singular value decomposition results, the best rank-r approximate matrix W is obtained k,r and the residual matrix Wk,e :

[0014]

[0015] in,

[0016] U k,r =[u k,1 ,u k,2 ,…,u k,r ]

[0017] V k,r =[v k,1 ,v k,2 ,…,v k,r ]

[0018]

[0019] U k,r 、V k,r Denote the orthogonal matrix U k 、V k The submatrix corresponding to the singular value vector in U k,e 、V k,r Denote the orthogonal matrix U k 、V k Divisor matrix U k,r 、V k,r The remaining submatrices other than ; diag() means generating a diagonal matrix, [,] indicates matrix concatenation;

[0020] S4: Replace the U in the weight matrix of the layer to be fine-tuned k,r 、V k,r , U k,e ,Σ k,e 、V k,e And all parameters of other layers in the visual model are frozen, and Σ k,r As a trainable parameter;

[0021] S5: Input the image samples in the training sample set into the visual model, calculate the loss function and adjust the trainable parameters Σ of each layer to be fine-tuned k,r Update to get the updated trainable parameter Σ′ k,r ;

[0022] S6: According to the updated trainable parameters Σ′ k,r Calculate the updated model parameters W of each layer to be fine-tuned k ′:

[0023]

[0024] S7: Determine whether the fine-tuning end condition is met, if not, proceed to step S8, otherwise the fine-tuning ends;

[0025] S8: Let the weight matrix W k =W k ′, trainable parameter Σ k,r =Σ′ k,r , return to step S5.

[0026] The present invention is a method for fine-tuning parameters of a pre-trained visual model based on singular values. For the pre-trained visual model, a layer that needs to be fine-tuned is selected from the visual model according to actual needs, and the weight matrix of each layer to be fine-tuned is decomposed by singular values ​​to obtain the best rank-r approximate matrix and residual matrix for reconstructing the weight matrix. The first r singular values ​​constitute a singular value vector as a trainable parameter, and other model parameters are frozen. Image samples in a training sample set are input into the visual model to update the trainable parameters, and the weight matrix is ​​reconstructed using the updated trainable parameters.

[0027] The present invention has the following beneficial effects:

[0028] 1) The present invention uses the key singular values ​​obtained by singular value decomposition as trainable parameters to scale the basic subspace of the matrix, thereby promoting rapid domain adaptation of the model;

[0029] 2) Compared with other LoRA variants, the present invention uniquely adopts the most critical singular values ​​in SVD initialization as trainable parameters, thereby more effectively learning new domain knowledge for downstream tasks while retaining pre-training information;

[0030] 3) After the training is completed, the trainable parameters of the present invention can be seamlessly integrated into the original matrix. Therefore, while the present invention significantly reduces the number of trainable parameters, it does not introduce additional computing overhead or delay in the reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of a specific implementation method of the method for fine-tuning parameters of a pre-trained visual model based on singular values ​​of the present invention;

[0032] Figure 2 is a schematic diagram of singular value decomposition in the present invention;

[0033] Figure 3 is a comparison diagram of the images before and after reconstruction under different singular values ​​in this embodiment;

[0034] Figure 4 It is a structural diagram of the ViT model in this embodiment. DETAILED DESCRIPTION

[0035] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0036] Example

[0037] Figure 1 1 is a flowchart of a specific implementation of the method for fine-tuning parameters of a pre-trained visual model based on singular values ​​of the present invention. Figure 1 As shown, the specific steps of the method for fine-tuning parameters of a pre-trained visual model based on singular values ​​of the present invention include:

[0038] S101: Obtain parameter fine-tuning training sample set:

[0039] A training sample set for parameter fine-tuning is obtained according to actual needs, and each training sample includes an image sample and a label.

[0040] S102: Select parameter fine-tuning layer:

[0041] For the pre-trained visual model, select the layer that needs to be fine-tuned from the visual model according to actual needs, and the weight matrix of the layer to be fine-tuned is Where k = 1, 2, ..., K, K represents the number of layers to be fine-tuned, M k ×N k Indicates the size of the weight matrix of the layer to be fine-tuned. Generally speaking, all layers are fine-tuned by default.

[0042] S103: Singular Value Decomposition:

[0043] Figure 2 Schematic diagram of singular value decomposition in the present invention. Figure 2 As shown, the weight matrix W of each layer to be fine-tuned is k Perform singular value decomposition and get the decomposition result:

[0044]

[0045] in, They represent orthogonal matrix, diagonal matrix and orthogonal matrix respectively, and the superscript T represents transpose.

[0046] The number of singular values ​​obtained by singular value decomposition is R k =min(M k ,N k ), R k The singular values ​​are arranged in descending order, and the first r singular values ​​form a singular value vector Σ k,r =[λ k,1 ,λ k,2,…,λ k,r ], the value of r is set according to actual needs, and the residual singular value vector is obtained at the same time Let the orthogonal matrix U k The i-th eigenvector in is u k,i ,i=1,2,…,M k , let the orthogonal matrix V k The jth eigenvector in is v k,j ,j=1,2,…,N k Based on the singular value decomposition results, the best rank-r approximate matrix W is obtained k,r and the residual matrix W k,e :

[0047]

[0048] in,

[0049] U k,r =[u k,1 ,u k,2 ,…,u k,r ]

[0050] V k,r =[v k,1 ,v k,2 ,…,v k,r ]

[0051]

[0052] U k,r 、V k,r Denote the orthogonal matrix U k 、V k The submatrix corresponding to the singular value vector in U k,e 、V k,r Denote the orthogonal matrix U k 、V k Divisor matrix U k,r 、V k,r The remaining submatrices other than diag(). diag() means generating a diagonal matrix, Q means a 0 matrix, that is [,] represents matrix concatenation. It can be seen that the concatenated matrix

[0053] According to the above analysis, the weight matrix W k It can be expressed as:

[0054]

[0055] According to the above process, it can be seen that the present invention uses a larger singular value to reconstruct the weight matrix. Figure 3is a comparison diagram of the images before and after reconstruction under different singular values ​​in this embodiment. Figure 3 As shown, the largest 8, 16, 32, 64, 128 and 256 singular values, and the smallest 8, 16, 32, 64, 128 and 256 singular values ​​are used. The results show that larger singular values ​​are crucial to maintaining image quality. It is worth noting that the top 10% or even 1% of the singular values ​​contribute more than 99% of the total matrix sum. This observation suggests that the pre-trained weight matrix can be effectively approximated by focusing on the most important singular values ​​above a given threshold, while smaller singular values ​​have minimal impact on the overall structure. In general, the rank r can be preferably set in the range of [0.01N k ,0.1N k ].

[0056] S104: Trainable parameter settings:

[0057] In the present invention, in order to reduce the number of trainable parameters in model fine-tuning, the optimal rank-r approximation matrix W for each layer to be fine-tuned is selected. k,r , and the frozen residual matrix W k,e Inspired by recent advances in image generation, such as learnable scaling factors to improve domain adaptation, we only train the most critical singular values ​​obtained from SVD as trainable parameters. Specifically, we only train the first r singular values ​​while scaling the basic subspace based on SVD to facilitate fast adaptation to new domains.

[0058] The U in the weight matrix of the layer to be fine-tuned k,r 、V k,r , U k,e ,Σ k,e 、V k,e And all parameters of other layers in the visual model are frozen, and Σ k,r as a trainable parameter.

[0059] S105: Singular value training:

[0060] Input the image samples in the training sample set into the visual model, calculate the loss function and adjust the trainable parameters Σ of each layer to be fine-tuned. k,r Update to get the updated trainable parameter Σ′ k,r .

[0061] The specific type of loss function can be set according to the task of the visual model. Conventional loss functions include L2 norm loss, cross entropy loss, IOU loss (Intersection over Union Loss), etc. The parameter update method can also be set according to actual needs, such as gradient descent, Adam (Adaptive Moment Estimation, adaptive learning rate) algorithm.

[0062] S106: Update visual model parameters:

[0063] According to the updated trainable parameters Σ′ k,r Calculate the updated model parameters W of each layer to be fine-tuned k ′:

[0064]

[0065] S107: Determine whether the fine-tuning end condition is met, if not, proceed to step S108, otherwise the fine-tuning ends. The fine-tuning end condition can be set according to actual needs, usually including the maximum number of iterations and model parameter convergence.

[0066] S107: Let the weight matrix W k =W k ′, trainable parameter Σ k,r =Σ′ k,r , return to step S105.

[0067] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention.

[0068] In this embodiment, the visual model adopts the Transformer-based visual model ViT (Vision Transformer) pre-trained on the ImageNet-21K dataset. Figure 4 is a structural diagram of the ViT model in this embodiment. Figure 4 As shown, the ViT model includes a linear projection of flattened patches, a Transformer encoder, and an MLP head (for classification). In this embodiment, the self-attention and multi-layer perceptron layers in the Transformer Encoder are selected as the layers to be fine-tuned, and the parameters are fine-tuned. For ease of representation, this method is recorded as SVFit.

[0069] This embodiment evaluates the method of the present invention on the GLUE benchmark. GLUE is a comprehensive natural language understanding evaluation framework that covers a variety of tasks, including sentence relationship recognition, sentiment analysis, and natural language reasoning. In order to evaluate systematically, this embodiment selects six tasks: CoLA, STS-B, RTE, MRPC, SST-2, and QNLI. SVFit is used to fine-tune the RoBERTa-base and RoBERTa-large models, respectively. RoBERTa-base has 12 layers, a hidden layer size of 768, and a total of 125 million parameters; RoBERTa-large has 24 layers, a hidden layer size of 1024, and a total of 356 million parameters.

[0070] In order to compare and verify with the present invention, five existing parameter fine-tuning methods are selected as comparison methods in this embodiment, which are:

[0071] FT: Full parameter fine-tuning involves updating the entire set of model parameters. Although this approach is simple and robust, it requires a lot of computational resources.

[0072] LoRA (Low-Rank Adaptation of Large Language Models): A low-rank adaptation technique for fine-tuning large language models. It uses two randomly initialized low-rank matrices to learn incremental updates of each layer's parameters, reducing GPU memory overhead.

[0073] DyLoRA: Dynamically set random rank for the original LoRA low-rank matrix module during training;

[0074] AdaLoRA: addresses the challenge of optimal rank selection in incremental updates by adaptively pruning singular values ​​according to their size, resulting in different ranks at different layers;

[0075] PiSSA: Similar in structure to LoRA, it initializes the low-rank matrix using the principal components of the original weight matrix, and the remaining components form a residual matrix that remains frozen during fine-tuning.

[0076] Table 1 is a performance comparison table of the present invention and the comparative method in this embodiment on the GLUE benchmark.

[0077]

[0078] Table 1

[0079] As shown in Table 1, the SVFit method shows excellent performance in multiple key areas. For RoBERTa-base, SVFit has the highest MCC value on the CoLA task and the highest PCC value on the STS-B task, indicating its excellent performance in classification and regression tasks. Despite a slight gap in accuracy on the MRPC and SST-2 tasks, SVFit remains competitive in all tasks with strong overall average performance. For RoBERTa-large, SVFit outperforms other methods on the CoLA, RTE, and MRPC tasks, while also showing comparable performance on other tasks. Notably, SVFit's significant improvement in MCC on the CoLA task shows its advantage in complex language understanding tasks. The overall results show that SVFit provides a balanced and effective alternative to existing methods with its unique parameterization strategy and efficient use of trainable parameters. In addition, SVFit significantly reduces the number of trainable parameters, showing the potential to achieve high performance at a lower computational cost.

[0080] In order to ensure a comprehensive evaluation, other data sets are selected for comparative verification in this embodiment, including OxfordPets, CIFAR10, DTD, EuroSAT, RESISC45, StanfordCars, FGVC and CIFAR100. Table 2 is a performance comparison table of the present invention and the comparative method on different data sets in this embodiment.

[0081]

[0082] Table 2

[0083] As shown in Table 2, SVFit shows excellent performance at both model sizes. For ViT-base, SVFit achieves the highest accuracy on OxfordPets (97.0%) and DTD (80.5%), and performs very competitively on other datasets, with an overall average accuracy of 84.3%. Although full parameter fine-tuning (FT) achieves the highest average accuracy (86.5%), it requires a large number of trainable parameters (85.8M) compared to SVFit’s 0.018M trainable parameters. For ViT-large, SVFit also performs well, achieving the highest accuracy on OxfordPets (97.8%), CIFAR10 (99.3%), DTD (83.4%), and performs well on other datasets, with an average accuracy of 88.7%. Although FT still has an advantage in average accuracy (90.2%), it comes at the cost of training 303.3M parameters, while SVFit only uses 0.036M parameters. These results highlight the advantages of SVFit in terms of computational resource and parameter efficiency. By significantly reducing the number of trainable parameters while maintaining competitive performance, SVFit provides a compelling alternative to traditional fine-tuning methods, especially in application scenarios where computational resource and parameter efficiency are crucial.

[0084] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A method for fine-tuning parameters of a pre-trained visual model based on singular values, characterized in that: The following steps are involved: S1: Obtain a training sample set for parameter fine-tuning according to actual needs, where each training sample includes an image sample and a label; S2: For the pre-trained visual model, select the layer that needs to be fine-tuned from the visual model according to actual needs. The weight matrix of the layer to be fine-tuned is Where k = 1, 2, ..., K, K represents the number of layers to be fine-tuned, M k ×N k Represents the weight matrix size of the layer to be fine-tuned; S3: The weight matrix W of each layer to be fine-tuned k Perform singular value decomposition and get the decomposition result: in, denote an orthogonal matrix, a diagonal matrix, and an orthogonal matrix, respectively, and the superscript T denotes transpose; The number of singular values ​​obtained by singular value decomposition is R k =min(M k ,N k ), R k The singular values ​​are arranged in descending order, and the first r singular values ​​form a singular value vector Σ k,r =[λ k,1 ,λ k,2 ,…,λ k,r ], the value of r is set according to actual needs, and the residual singular value vector Σ k,e =[λ k,r+1 ,λ k,r+2 ,…,λ k,Rk ]; let the orthogonal matrix U k The i-th eigenvector in is u k,i ,i=1,2,…,M k , let the orthogonal matrix V k The jth eigenvector in is v k,j ,j=1,2,…,N k ; Based on the singular value decomposition results, the best rank-r approximate matrix W is obtained k,r and the residual matrix W k,e : in, IN k,r =[in k,1 ,in k,2 ,…,in k,r ] V k,r =[v k,1 ,v k,2 ,…,v k,r ] U k,r 、V k,r Denote the orthogonal matrix U k 、V k The submatrix corresponding to the singular value vector in U k,e 、V k,r Denote the orthogonal matrix U k 、V k Divisor matrix U k,r 、V k,r The remaining submatrices other than ; diag() means generating a diagonal matrix, [,] indicates matrix concatenation; S4: Replace the U in the weight matrix of the layer to be fine-tuned k,r 、V k,r , U k,e ,Σ k,e 、V k,e And all parameters of other layers in the visual model are frozen, and Σ k,r As a trainable parameter; S5: Input the image samples in the training sample set into the visual model, calculate the loss function and adjust the trainable parameters Σ of each layer to be fine-tuned k,r Update to get the updated trainable parameter Σ′ k,r ; S6: According to the updated trainable parameters Σ′ k,r Calculate the updated model parameters W of each layer to be fine-tuned k ′: S7: Determine whether the fine-tuning end condition is met, if not, proceed to step S8, otherwise the fine-tuning ends; S8: Let the weight matrix W k =W k ′, trainable parameter Σ k,r =Σ′ k,r , return to step S5.

2. The method for fine-tuning parameters of a pre-trained visual model according to claim 1, characterized in that: The value range of the rank r in step S3 is [0.01N k ,0.1N k ].

Images