Visual large model training method and system based on visual query fine tuning

Abstract

The invention provides a visual query fine tuning-based visual large model training method and system. The system comprises the following steps of: performing block processing on an input image; performing position coding on the feature vectors in the previous step; finely adjusting the visual query encoder; the output of the visual query fine-tuning encoder is sent to a multi-layer perception machine head network for result prediction; and performing loss function calculation according to the truth value and the predicted value of the data set during training. The technical problems of high calculation cost, low parameter efficiency, poor model performance and information loss are solved.

Description

Technical Field

[0001] The present invention relates to the field of visual model optimization, and in particular to a large visual model training method and system based on visual query fine-tuning. Background Art

[0002] The large visual model has developed rapidly in the past year. When the data of downstream tasks is limited, it has become a good choice to improve performance by adapting the large pre-trained model to the downstream tasks through transfer learning. There are two methods in the prior art to adapt the pre-trained large model: updating the model backbone to obtain a new feature embedding (the output of the penultimate layer) or reorganizing the existing feature embedding, which correspond to two popular methods, fine-tuning and linear detection. For example, the existing invention patent application document "Face Recognition Model Training Method, Device, Electronic Device and Storage Medium" with publication number CN117496582A, in which the second adversarial perturbation information is determined based on the gradient value and discretized gradient value in each iterative update process, including: determining the iterative update adversarial perturbation according to the discretized gradient value in each iterative update process; determining the sum of the iterative update gradient values ​​according to the gradient value in each iterative update process, and calculating the iterative update discretized gradient value based on the iterative update gradient value sum; determining the second adversarial perturbation information based on the iterative update discretized gradient value and the iterative update adversarial perturbation. And the existing invention patent application document "Neural Network Model Training and Fatigue Detection Method for Image Quality Assessment" with publication number CN117541879A, the existing method includes: inputting the labeled / pseudo-labeled data into the feature extraction backbone network to obtain the labeled / pseudo-labeled features; wherein the labeled / pseudo-labeled data has classification labels on multiple distortion attributes; using the classification model to obtain the classification results of the labeled / pseudo-labeled data on each distortion attribute and the loss function of the labeled / pseudo-labeled data of the classification model; using the regression model to obtain the quality score results of the labeled data and the loss function of the regression model; according to the loss function of the labeled / pseudo-labeled data of the classification model and the loss function of the regression model, the loss function of the neural network model is determined; when the loss function of the neural network model is not greater than the target loss value, the training is terminated and the parameters of the feature extraction backbone network and the regression model are saved as the trained neural network model. The fine-tuning operation of the aforementioned prior art updates all model parameters end-to-end based on the new data set. Although the performance of fine-tuning on various tasks is always better than linear detection, the aforementioned prior art needs to run gradient descent on all parameters and store a separate fine-tuning model for each task, which makes the computational cost high and the parameter efficiency low. These issues become more prominent for transformer-based models whose parameters grow exponentially. In addition, linear probing only trains and stores new prediction heads to recombine features while keeping the backbone parameters fixed. Despite its computational and parameter efficiency, linear probing usually leaves much room for improvement in the transfer process due to its poor performance.

[0003] At the same time, there are some studies in the prior art that attempt to overcome this dilemma in fine-tuning large models. Evci et al. attributed the success of fine-tuning to the use of the "intermediate" features of the pre-trained model and proposed to directly allow linear probing to access the intermediate features. Some other studies have also demonstrated the effectiveness of this approach. However, given that there are many intermediate features in each layer, most of these methods require pooling to reduce the dimensionality, which may eliminate useful information before the prediction head accesses it.

[0004] Existing technologies have technical problems such as high computational cost, low parameter efficiency, poor model performance and information loss. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to solve the technical problems of high calculation cost, low parameter efficiency, poor model performance and information loss in the prior art.

[0006] The present invention adopts the following technical solutions to solve the above technical problems: A visual large model training method based on visual query fine-tuning includes:

[0007] S1, according to the preset block size and block number, the input two-dimensional image is processed into blocks to obtain an input sequence, and the input sequence is converted and processed to obtain an image feature vector;

[0008] S2, performing a position encoding operation on the image feature vector to restore the two-dimensional space lost information in the image feature vector;

[0009] S3. Construct and use a visual query fine-tuning encoder to perform a visual fine-tuning query operation, a transformation operation, and a self-attention preprocessing operation on the image feature vector to obtain a fine-tuned visual query feature, wherein the visual query encoder includes: a fine-tuning visual query module, no less than 2 transformation layers Transformer, and a self-attention mechanism network;

[0010] S4, input the fine-tuned visual query features into the multi-layer perceptron head network to make result predictions;

[0011] S5. Train the preset visual big model, calculate the loss function according to the true value and predicted value of the preset data set, and obtain the applicable visual big model accordingly.

[0012] The present invention can aggregate the intermediate features of large transformer-based models (such as Vision Transformers, ViT). It has good universality and efficiency, and can effectively alleviate the problems of high computational cost, low parameter efficiency, and poor fine-tuning effect of existing methods.

[0013] In a more specific technical solution, in step S1, each image block in the input sequence is embedded using a pre-projection layer.

[0014] In a more specific technical solution, a special character cls is added to the image block to convert the input two-dimensional image into an image feature vector.

[0015] In a more specific technical solution, step S3 includes:

[0016] S31, in the fine-tuning visual query module, performing random initialization to obtain learnable parameters, and performing position encoding operations based on the parameters to obtain fine-tuning visual features;

[0017] S32, using the self-attention mechanism network, performing multi-head attention calculation on the fine-tuned visual features to obtain the fine-tuned visual query features and correlation information;

[0018] S33. Merge the fine-tuned visual query features and the original visual query features to obtain a merged vector.

[0019] The present invention fine-tunes the visual query mechanism by adding trainable visual queries to fine-tune the large model, and pre-processes it through the self-attention layer before inputting it into the Transformer layer, which can better adapt to the downstream unseen data sets.

[0020] In a more specific technical solution, in step S32, the dimension adjustment operation is completed by matrix multiplication to adjust the dimension of the fine-tuned visual query feature and the visual query fine-tuning encoder to be consistent.

[0021] In a more specific technical solution, step S33 includes:

[0022] S331, input the merged vector to the conversion layer Transformer of the preset visual large model;

[0023] S332. Each time a transformation layer Transformer is passed, the current transformation vector is split into original visual query features and fine-tuned visual query features, and the fine-tuned visual query features and the original visual query features are re-merged using a self-attention mechanism to obtain a merged vector.

[0024] In a more specific technical solution, in step S3, the architecture of the transformation layer Transformer includes: a stacking method.

[0025] In a more specific technical solution, in step S5, parameter adjustment operations are performed on the fine-tuning visual query module and the self-attention mechanism network, and the remaining parameters of the preset visual large model are fixed.

[0026] The present invention introduces a small amount of parameters into the backbone network, fixes the original model parameters of the large model, and only trains the newly introduced parameters, so as to adapt to downstream tasks and better adapt to data sets that have not been seen by downstream. In addition, the training parameters are reduced, and the video memory and training time during training can be reduced, so that the training is more efficient. The present invention alleviates the problems of high computational cost and low parameter efficiency.

[0027] In a more specific technical solution, parameter adjustment operations are performed through gradient backpropagation, wherein the components performing the parameter adjustment operations include: a self-attention layer and a fine-tuning visual query component.

[0028] In a more specific technical solution, the visual large model training system based on visual query fine-tuning includes:

[0029] An image block processing module is used to perform block processing on the input two-dimensional image according to a preset block size and block number to obtain an input sequence, and convert and process the input sequence to obtain an image feature vector;

[0030] A position coding module is used to perform position coding operation on the image feature vector to restore the two-dimensional space lost information in the image feature vector, and the position coding module is connected to the image block processing module;

[0031] A fine-tuning encoder module is used to construct and use a visual query fine-tuning encoder to perform a visual fine-tuning query operation, a transformation operation, and a self-attention preprocessing operation on an image feature vector to obtain a fine-tuning visual query feature, wherein the visual query encoder includes: a fine-tuning visual query module, no less than 2 transformation layers Transformer, and a self-attention mechanism network, and the fine-tuning encoder module is connected to the position encoding module and the position encoding module;

[0032] A multi-layer perceptron module is used to input the fine-tuned visual query features into the multi-layer perceptron head network to perform result prediction, and the multi-layer perceptron module is connected to the fine-tuned encoder module;

[0033] The model training optimization module is used to train the preset visual large model, calculate the loss function according to the true value and predicted value of the preset data set, and obtain the applicable visual large model. The model training optimization module is connected to the multi-layer perceptron module.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] The present invention can aggregate the intermediate features of large transformer-based models (such as Vision Transformers, ViT). It has good universality and efficiency, and can effectively alleviate the problems of high computational cost, low parameter efficiency, and poor fine-tuning effect of existing methods.

[0036] The present invention fine-tunes the visual query mechanism by adding trainable visual queries to fine-tune the large model, and pre-processes it through the self-attention layer before inputting it into the Transformer layer, which can better adapt to the downstream unseen data sets.

[0037] The present invention introduces a small amount of parameters into the backbone network, fixes the original model parameters of the large model, and only trains the newly introduced parameters, so as to adapt to downstream tasks and better adapt to data sets that have not been seen by downstream. In addition, the training parameters are reduced, and the video memory and training time during training can be reduced, so that the training is more efficient. The present invention alleviates the problems of high computational cost and low parameter efficiency.

[0038] The present invention solves the technical problems existing in the prior art, namely, high computational cost, low parameter efficiency, poor model performance and information loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a data flow processing diagram of a visual large model training method based on visual query fine-tuning according to Embodiment 1 of the present invention;

[0040] Figure 2 Schematic diagram of the basic steps of the visual large model training method based on visual query fine-tuning according to Example 1 of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Example 1

[0043] like Figure 1 and Figure 2 As shown, the visual large model training method based on visual query fine-tuning provided by the present invention includes the following basic steps:

[0044] Step S1, input image for block processing;

[0045] In this embodiment, the input image size may be, for example, 224x224, and the image is divided into small blocks of a fixed size, the size of the small block may be, for example, 16x16, then each image will generate, for example, (224x224) / (16x16)=196 small blocks, that is, the input sequence length is 196; each small block is embedded through the advance projection layer, the dimension of each small block is 16x16x3=768, the dimension of the linear projection layer is 768xN (N=768), so the dimension of the input after passing through the linear projection layer is still 196x768, that is, there are a total of 196 tags, and the dimension of each tag is 768. In this embodiment, a special character cls is added, so the final dimension is 197x768. It has now been converted from a two-dimensional image to a feature vector;

[0046] Step S2, position encoding the feature vector in the above step S1;

[0047] In this embodiment, since the two-dimensional image is converted into a vector with a length of 197 and a dimension of 768, the two-dimensional spatial information is lost, so the feature vector needs to be positionally encoded. The position encoding is consistent with the original large model;

[0048] [Innovation] Step S3, fine-tune the visual query encoder;

[0049] In this embodiment, the visual query encoder includes but is not limited to: no less than 2 stacked Transformer layers of the same structure; in this embodiment, a fine-tuned visual query is added before inputting into the Transformer layer. The fine-tuned visual query is a fixed number of feature vectors whose dimensions are consistent with the dimensions of the original visual query, such as 100x768, and whose values ​​use randomly initialized learnable parameters. Its position encoding uses randomly initialized learnable parameters. In this embodiment, a self-attention mechanism is added to pre-process it;

[0050] In this embodiment, the self-attention mechanism network is calculated through a multi-head attention mechanism, and the calculation formula is as follows:

[0051]

[0052] Among them, the multi-head attention mechanism network is used to calculate the correlation. The feature vector dimension is 768, Q and K are equal to the sum of the fine-tuned visual query and the position encoding, and V is equal to the fine-tuned visual query; the dimension is adjusted by a matrix multiplication to be consistent with the dimension of the encoder input value;

[0053] In this embodiment, after the self-attention mechanism, the output fine-tuned visual query is merged with the original visual query to obtain a 297x768 vector; which is then input into the Transformer layer of the original large visual model; in this embodiment, since the Transformer layer is a stack of multiple identical structures, each time after passing through a Transformer layer, the output 297x768 vector needs to be split into the original visual query and the fine-tuned visual query, and the fine-tuned visual query is then merged with the original visual query after the self-attention mechanism; and then input into the next Transformer layer; after passing through the Transformer layer, the output 297x768 vector is directly input into the head network;

[0054] Step S4, giving the output of the visual query fine-tuning encoder to the multi-layer perception head network for result prediction;

[0055] Step S5: during training, the loss function is calculated according to the true value and the predicted value of the data set;

[0056] In this embodiment, when updating parameters, only the parameters of the fine-tuning visual query and its corresponding self-attention mechanism in the visual query fine-tuning encoder are updated, and the remaining parameters of other large models are fixed. Through the aforementioned fine-tuning method, it can better adapt to downstream unseen data sets, and reduce training parameters, which can reduce video memory and training time during training, thereby making training more efficient.

[0057] Specifically, the pre-trained large model is generally trained based on data from natural scenes. For some specific scenes, such as insulator detection in power grids, workers not wearing helmets, etc., the performance of the trained large model is poor when used directly. However, fine-tuning the small-scale data set of specific scenes can efficiently utilize the pre-trained parameters of the large model. However, due to the large number of parameters in the large model, fine-tuning all parameters requires high hardware and training time. Therefore, the patented method uses small-scale parameter fine-tuning, which can effectively reduce the video memory and training time during training, thereby making training more efficient. Depending on the parameters of the large model, this method can reduce at least 90% of the training parameters.

[0058] In summary, the present invention can aggregate the intermediate features of large transformer-based models (such as Vision Transformers, ViT), has good universality and efficiency, and can effectively alleviate the problems of high computational cost, low parameter efficiency, and poor fine-tuning effect of existing methods.

[0059] The present invention fine-tunes the visual query mechanism by adding trainable visual queries to fine-tune the large model, and pre-processes it through the self-attention layer before inputting it into the Transformer layer, which can better adapt to the downstream unseen data sets.

[0060] The present invention introduces a small amount of parameters into the backbone network, fixes the original model parameters of the large model, and only trains the newly introduced parameters, so as to adapt to downstream tasks and better adapt to data sets that have not been seen by downstream. In addition, the training parameters are reduced, and the video memory and training time during training can be reduced, so that the training is more efficient. The present invention alleviates the problems of high computational cost and low parameter efficiency.

[0061] The present invention solves the technical problems existing in the prior art, namely, high computational cost, low parameter efficiency, poor model performance and information loss.

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual large model training method based on visual query fine-tuning, characterized in that: The method comprises: S1, according to a preset block size and block number, the input two-dimensional image is processed into blocks to obtain an input sequence, and the input sequence is converted and processed to obtain an image feature vector; S2, performing a position encoding operation on the image feature vector to restore the two-dimensional space lost information in the image feature vector; S3, constructing and using a visual query fine-tuning encoder to perform a visual fine-tuning query operation, a transformation operation, and a self-attention preprocessing operation on the image feature vector to obtain a fine-tuned visual query feature, wherein the visual query encoder includes: a fine-tuning visual query module, no less than 2 transformation layers Transformer, and a self-attention mechanism network; S4, inputting the fine-tuned visual query features into a multi-layer perceptron head network to perform result prediction; S5. Train the preset visual big model, calculate the loss function according to the true value and predicted value of the preset data set, and obtain the applicable visual big model accordingly.

2. The visual large model training method based on visual query fine-tuning according to claim 1 is characterized in that: In the step S1, each image block in the input sequence is embedded using a previous projection layer.

3. The visual large model training method based on visual query fine-tuning according to claim 2 is characterized in that: Special characters cls are added to the image block to convert the input two-dimensional image into the image feature vector.

4. The visual large model training method based on visual query fine-tuning according to claim 1 is characterized in that: The step S3 comprises: S31, performing random initialization in the fine-tuning visual query module to obtain learnable parameters, and performing the position encoding operation based on the parameters to obtain fine-tuning visual features; S32, using the self-attention mechanism network to perform multi-head attention calculation on the fine-tuned visual features to obtain the fine-tuned visual query features and correlation information; S33. Merge the fine-tuned visual query feature and the original visual query feature to obtain a merged vector.

5. The visual large model training method based on visual query fine-tuning according to claim 4 is characterized in that: In the step S32, the dimension adjustment operation is completed by matrix multiplication, so that the dimension of the fine-tuning visual query feature is adjusted to be consistent with the dimension of the visual query fine-tuning encoder.

6. The visual large model training method based on visual query fine-tuning according to claim 4 is characterized in that: The step S33 comprises: S331, inputting the merged vector into the conversion layer Transformer of the preset visual large model; S332. Each time a layer of the transformation layer Transformer is passed, the current transformation vector is split into the original visual query feature and the fine-tuned visual query feature, and the fine-tuned visual query feature and the original visual query feature are re-merged using a self-attention mechanism to obtain the merged vector.

7. The visual large model training method based on visual query fine-tuning according to claim 1 is characterized in that: In the step S3, the architecture of the transformation layer Transformer includes: a stacking method.

8. The visual large model training method based on visual query fine-tuning according to claim 1 is characterized in that: In step S5, parameter adjustment operations are performed on the fine-tuning visual query module and the self-attention mechanism network, and the remaining parameters of the preset large visual model are fixed.

9. The visual large model training method based on visual query fine-tuning according to claim 8 is characterized in that: The parameter adjustment operation is performed through gradient feedback, wherein the components performing the parameter adjustment operation include: a self-attention layer and a fine-tuning visual query component.

10. A visual large model training system based on visual query fine-tuning, characterized in that: The system comprises: An image block processing module is used to perform block processing on the input two-dimensional image according to a preset block size and block number to obtain an input sequence, and convert and process the input sequence to obtain an image feature vector; A position coding module, used for performing a position coding operation on the image feature vector to restore the two-dimensional space lost information in the image feature vector, the position coding module being connected to the image block processing module; A fine-tuning encoder module is used to construct and use a visual query fine-tuning encoder to perform a visual fine-tuning query operation, a transformation operation, and a self-attention preprocessing operation on the image feature vector to obtain a fine-tuning visual query feature, wherein the visual query encoder includes: a fine-tuning visual query module, no less than 2 transformation layers Transformer and a self-attention mechanism network, and the fine-tuning encoder module is connected to the position encoding module and the position encoding module; A multi-layer perceptron module, used for inputting the fine-tuned visual query features into a multi-layer perceptron head network for result prediction, the multi-layer perceptron module being connected to the fine-tuned encoder module; The model training optimization module is used to train the preset visual large model, calculate the loss function according to the true value and predicted value of the preset data set, and obtain the applicable visual large model accordingly. The model training optimization module is connected to the multi-layer perceptron module.

Images