Efficient parameter fine tuning method for visual basis model of dense prediction task

Abstract

The invention relates to the technical field of model parameter fine tuning, and discloses an efficient parameter fine tuning method for a visual basic model of a dense prediction task, which comprises the following steps: introducing a dynamic parameter generation module for the visual basic model, and parallelly adding an HOM module on a multi-head attention module and a feedforward neural network of each transformer layer of the visual basic model; each super network in the dynamic parameter generation module generates a convolution kernel parameter for the corresponding convolution layer; when fine tuning training is carried out on the visual basic model, parameters of the visual basic model are kept frozen, and parameters of the HOM module and the dynamic parameter generation module are trained through dense prediction task training images. According to the method, resources consumed when the visual basic model is applied to the dense prediction task are effectively reduced, and the cost and the training time of a video memory are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of efficient parameter fine-tuning, and in particular to a method for efficient parameter fine-tuning of a visual base model for dense prediction tasks. Background Art

[0002] Parameter efficient fine-tuning is an optimization technique for deep learning models. In deep learning, large-scale datasets are often used for pre-training, such as using large datasets such as ImageNet for initial training. These pre-trained models are then fine-tuned to adapt to datasets for specific tasks or domains. With the development of deep learning technology, visual basis models (VFM) have shown superior performance on a variety of downstream tasks. However, fully fine-tuning these models is not only computationally expensive, but also requires a lot of storage resources. To address this problem, parameter efficient fine-tuning (PEFT) technology has emerged, which replicates the results of full fine-tuning by only updating a portion of the model's parameters or adding lightweight structures. Although PEFT has made significant progress in image classification tasks, its application in dense prediction tasks remains an underexplored area.

[0003] The main purpose of this invention is to solve the problem of efficiently using visual base models for dense prediction tasks. Although pre-trained visual base models have shown good performance in various downstream tasks, when they are directly used for dense prediction tasks, their performance often does not meet expectations; and if these visual base models are fully fine-tuned, it often consumes a lot of computing resources and storage resources. Summary of the invention

[0004] To solve the above technical problems, the present invention provides a method for efficiently fine-tuning parameters of a visual base model for intensive prediction tasks, which can efficiently fine-tune the pre-trained visual base model to adapt to various downstream visual tasks, especially intensive prediction tasks such as target detection, instance segmentation, and semantic segmentation, without the need to fine-tune all parameters of the entire visual base model.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A method for efficiently fine-tuning parameters of a visual base model for dense prediction tasks, which is used to achieve dense prediction tasks on images, including:

[0007] Introducing a dynamic parameter generation module for the visual base model, and adding a HOM module in parallel to the multi-head attention module and feedforward neural network of each transformer layer of the visual base model;

[0008] Each HOM module consists of a down-projection layer, a convolutional layer, and an up-projection layer;

[0009] The dynamic parameter generation module includes multiple hypernetworks, each of which generates convolution kernel parameters for the corresponding convolution layer:

[0010] Introduce multiple prompts, calculate the similarity between each prompt and the input feature map corresponding to the image in the lth transformer layer, obtain the weight of each pixel in the feature map, perform weighted average on the pixels to obtain the compressed input-related layer embedding, and then input the compressed input-related layer embedding into the globally shared hypernetwork to generate convolution kernel parameters for the convolution layer in the HOM module;

[0011] When the visual base model is fine-tuned, the parameters of the visual base model remain frozen, and the parameters of the HOM module and the dynamic parameter generation module are trained through dense prediction task training images;

[0012] The dense prediction task test image is input into the visual base model that has completed fine-tuning training to obtain the result of the dense prediction task.

[0013] Furthermore, the HOM module is formulated as follows:

[0014]

[0015] in Represents the input feature map corresponding to the image in the lth transformer layer, Represents the output feature map of the image at the lth transformer layer. represents the downward projection layer, represents the upward projection layer, C l represents the convolutional layer; s is a hyperparameter used to control the influence of the HOM module on the visual base model.

[0016] Furthermore, the method introduces multiple prompts, calculates the similarity between each prompt and the input feature map corresponding to the image in the lth transformer layer, obtains the weight of each pixel in the feature map, performs weighted average on the pixels to obtain the compressed input-related layer embedding, and then inputs the compressed input-related layer embedding into the globally shared hypernetwork to generate convolution kernel parameters for the convolution layer in the HOM module, specifically including:

[0017] The introduced T prompts form the prompt matrix Compute the similarity between each prompt and the feature map:

[0018]

[0019] in, Represents the input feature map of the image at the lth transformer layer, E t represents the tth element in the prompt matrix, H and W represent The length and width of C The number of channels, M l represent The weight of each pixel in the matrix is ​​calculated by weighted average to obtain the layer embedding related to the input after compression:

[0020]

[0021] in represents the layer embedding associated with the input after compression, express The similarity between the pixel in the i-th row and the j-th column and the t-th element of the prompt matrix, express The pixel value at the i-th row and j-th column, Z represents the dimension of the layer embedding associated with the input; H l Input into the globally shared hypernetwork to generate convolution kernel parameters for the convolution layer in the HOM module:

[0022]

[0023] in represents the convolution kernel parameter of the convolution layer in the HOM module corresponding to the lth transformer layer, k represents the core size of the convolution kernel, C in and C out Represents the input feature map corresponding to the lth transformer layer And the output feature map The number of channels.

[0024] If the dense prediction task is target detection, the result of the dense prediction task is the target detection result of the test image, that is, the position and category of the specified target in the test image are detected.

[0025] If the dense prediction task is instance segmentation, the result of the dense prediction task is the instance segmentation result of the test image, that is, the pixel-level segmentation of the instance in the test image is completed.

[0026] If the dense prediction task is semantic segmentation, the result of the dense prediction task is the semantic segmentation result of the test image, that is, semantic segmentation is performed on each pixel of the test image.

[0027] Compared with the prior art, the beneficial technical effects of the present invention are:

[0028] The efficient parameter fine-tuning method in the present invention can effectively reduce the resources consumed when applying the visual basic model to intensive prediction tasks. The present invention only uses about 3% of the total parameter amount to fine-tune, reducing the cost of video memory and training time.

[0029] The efficient parameter fine-tuning method in the present invention can effectively improve the performance when the visual base model is applied to dense prediction tasks. The present invention achieves the performance of full fine-tuning (100% parameter amount) by using only about 3% of the parameter amount.

[0030] The efficient parameter fine-tuning method of the present invention has high flexibility, and the dynamic parameter generation mechanism enables the VFM-Adapter structure of the present invention to flexibly adjust the feature extraction method according to different input images. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the principle of the efficient parameter fine-tuning method in an embodiment of the present invention;

[0032] Figure 2 Schematic diagram showing the difference between the efficient parameter fine-tuning method in the embodiment of the present invention and the prior art. DETAILED DESCRIPTION

[0033] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, the method for efficiently fine-tuning parameters of the visual basic model for dense prediction tasks in the present invention adopts the following technical solutions:

[0035] The present invention freezes the parameters of the visual base model, and adds a HOM (Hybird Operation Mapping) module in parallel to the Multi-Head Attention (MHA) module and the Feed-Forward Network (FFN) of each transformer layer of the visual base model. Each HOM module includes a down projection, a convolution, and an up projection, which can be expressed as follows:

[0036]

[0037] in represents the input features of the lth layer, represents downward projection, represents upward projection, C lRepresents a 3*3 convolution, and s is a hyperparameter used to control the influence of the HOM module on the original visual base model.

[0038] In addition, in order to further reduce the number of parameters and improve the performance of the model, the present invention further introduces a hypernet to dynamically generate the parameters of the convolution kernel. First, T prompt tokens are introduced, denoted as Calculate the similarity between each prompt token and the feature map:

[0039] M l =softmax(F l E t );

[0040] in Represents the feature map of layer l, M l The code represents the weight of each pixel. The present invention performs weighted averaging on the pixels to obtain the compressed input-aware layer embedding:

[0041]

[0042] in Represents the layer embedding related to the input, H l Input into the globally shared Hypernet to generate convolution kernel parameters for the convolution layer in the HOM module:

[0043]

[0044] in represents the parameters of the convolution kernel of the first layer, k represents the core size of the convolution kernel, C in and C out Represents the number of channels of input and output features respectively.

[0045] like Figure 2 As shown, during the training process, the parameters of the original visual basic model remain frozen, and the present invention only trains the parameters of the additionally added HOM module and the dynamic parameter generation module, thereby greatly reducing the occupancy of the video memory and improving the training speed, thereby achieving efficient utilization of the visual basic model.

[0046] Example

[0047] The present invention can efficiently apply the visual basic model to intensive prediction tasks, and can achieve better performance with lower resource consumption.

[0048] The specific implementation steps are as follows:

[0049] Step 1: Select the visual base model:

[0050] First, the present invention selects the currently commonly used pre-trained visual basic models (such as DIONv2, EVA, EVA02, SAM, etc.), and uses the visual basic model as the backbone network of dense prediction, while adapting multi-scale features so that the head structure of the dense prediction task can be connected.

[0051] Step 2: Add the VFM-Adapter structure to the visual base model:

[0052] Based on the original structure of the visual base model, the VFM-Adapter structure proposed in the present invention is applied to the visual base model. The VFM-Adapter structure is the dynamic parameter generation module introduced in the present invention, and the HOM module is added in parallel to the multi-head attention module and feedforward neural network of each transformer layer of the visual base model.

[0053] Step 3: Fine-tuning and evaluation of the visual base model:

[0054] Select specific intensive prediction tasks (such as target detection, instance segmentation, semantic segmentation, etc.) and corresponding data sets, freeze the parameters of the original visual base model, only train the parameters of the additionally added VFM-Adapter structure, and efficiently fine-tune the parameters of the visual base model. Specifically, the present invention uses 8 NVIDIA A100 GPU machines to fine-tune the visual base model of the present invention and trains for 12 rounds. The AdamW optimizer is used to optimize the entire network, the learning strategy uses cosine annealing, the initial learning rate is 3e-4, and the batch size is set to 2. The MMDetection toolkit is used as the code basis. When evaluating model performance, mAP or mIOU is used as an evaluation indicator. When the present invention selects SAM as the visual base model and trains on the COCO data set, the method of the present invention uses only 3% of the parameters to achieve the effect of full fine-tuning (100% parameters).

[0055] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0056] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for efficiently fine-tuning parameters of a visual base model for dense prediction tasks, for implementing dense prediction tasks of images, characterized in that: include: Introducing a dynamic parameter generation module for the visual base model, and adding a HOM module in parallel to the multi-head attention module and feedforward neural network of each transformer layer of the visual base model; Each HOM module consists of a down-projection layer, a convolutional layer, and an up-projection layer; The dynamic parameter generation module includes multiple hypernetworks, each of which generates convolution kernel parameters for the corresponding convolution layer: Introduce multiple prompts, calculate the similarity between each prompt and the input feature map corresponding to the image in the lth transformer layer, obtain the weight of each pixel in the feature map, perform weighted average on the pixels to obtain the compressed input-related layer embedding, and then input the compressed input-related layer embedding into the globally shared hypernetwork to generate convolution kernel parameters for the convolution layer in the HOM module; When the visual base model is fine-tuned, the parameters of the visual base model remain frozen, and the parameters of the HOM module and the dynamic parameter generation module are trained through dense prediction task training images; The dense prediction task test image is input into the visual base model that has completed fine-tuning training to obtain the result of the dense prediction task.

2. The method for efficiently fine-tuning parameters of a visual base model for dense prediction tasks according to claim 1, characterized in that: The HOM module is formulated as follows: in Represents the input feature map corresponding to the image in the lth transformer layer, Represents the output feature map of the image at the lth transformer layer. represents the downward projection layer, represents the upward projection layer, C l represents the convolutional layer; s is a hyperparameter used to control the influence of the HOM module on the visual base model.

3. The method for efficiently fine-tuning parameters of a visual base model for dense prediction tasks according to claim 1, characterized in that: The method introduces multiple prompts, calculates the similarity between each prompt and the input feature map corresponding to the image in the lth transformer layer, obtains the weight of each pixel in the feature map, performs weighted average on the pixels to obtain the compressed input-related layer embedding, and then inputs the compressed input-related layer embedding into the globally shared hypernetwork to generate convolution kernel parameters for the convolution layer in the HOM module, specifically including: The introduced T prompts form the prompt matrix Compute the similarity between each prompt and the feature map: in, Represents the input feature map of the image at the lth transformer layer, E t represents the tth element in the prompt matrix, H and W represent The length and width of C The number of channels, M l represent The weight of each pixel in the matrix is ​​calculated by weighted average to obtain the layer embedding related to the input after compression: in represents the layer embedding associated with the input after compression, express The similarity between the pixel in the i-th row and the j-th column and the t-th element of the prompt matrix, express The pixel value at the i-th row and j-th column, Z represents the dimension of the layer embedding associated with the input; H l Input into the globally shared hypernetwork to generate convolution kernel parameters for the convolution layer in the HOM module: in represents the convolution kernel parameter of the convolution layer in the HOM module corresponding to the lth transformer layer, k represents the core size of the convolution kernel, C in and C out Represents the input feature map corresponding to the lth transformer layer And the output feature map The number of channels.

Images