An adaptive model reuse method based on adapter fine-tuning

Abstract

This invention discloses an adaptive model reuse method based on adapter fine-tuning. Taking object detection in computer vision as an example, firstly, a model pre-trained on a large-scale image dataset is obtained as the original model, and candidate adapters are set for it according to the structure of the pre-trained model. Then, the Taylor expansion method is used to calculate the global gain of adding adapters at different positions in the network for images with localized bounding boxes and class labels in the target domain, thereby achieving adaptive adapter configuration. In the fine-tuning stage, the global gain of adapters at different positions for labeled image samples is recalculated. The original model parameters are frozen, and the adapter parameters are updated using gradient backpropagation with a learning rate weighted by the global gain softmax. Finally, the fine-tuned model is reused in downstream tasks to locate objects in images and provide class predictions. This invention achieves adaptive adapter configuration based on global gain, realizing more accurate object localization and class prediction in object detection tasks.

Description

Technical Field

[0001] This invention relates to an adaptive model reuse method based on adapter fine-tuning, belonging to the field of artificial intelligence technology. Background Technology

[0002] In the field of artificial intelligence, collecting sufficient high-quality labeled data for training neural network models is often extremely costly. Even with sufficient data, retraining and deploying models for each task requires significant time and computational resources. How to efficiently reuse models without accessing the original data has become a hot research topic. For example, models pre-trained on the large-scale image dataset ImageNet have already been applied to various computer vision tasks. Taking object detection as an example, object detection refers to identifying objects in a given image and providing bounding box localization and category prediction. Because the features and distribution of image data change in new domains, and machine learning models heavily rely on the assumption of independent and identically distributed (ICD) training and test sets, directly reusing pre-trained models for object detection is not ideal.

[0003] With the improvement of computing power and the continuous expansion of data scale, pre-training-fine-tuning has become a typical learning paradigm. In the pre-training stage, the model learns transferable general features from large-scale image data, such as image edges, textures, and shapes. In the fine-tuning stage, a small number of images with localization bounding boxes and category labels are used to fine-tune the pre-trained model, improving the model's object localization and category prediction accuracy. Compared to directly fine-tuning the pre-trained model, adapter fine-tuning, by adding a small number of trainable layers, can effectively generalize to new domains while maintaining the performance of the source model. However, existing fine-tuning-based model reuse methods usually require manually designing image feature transfer strategies, resulting in a huge number of parameters to be updated, which seriously affects the efficiency of model reuse. In some domains where data is difficult to collect, such as military surveying and disaster relief, the labeled images used for fine-tuning are small in scale, and the model needs to learn the personalized features of the target domain image samples, a factor that current model reuse methods do not consider. Therefore, lightweight adaptive model reuse techniques need to be researched. Summary of the Invention

[0004] Objective: To address the problems and shortcomings of existing technologies, this invention provides an adaptive model reuse method based on adapter fine-tuning. Taking object detection in computer vision as an example, firstly, a model pre-trained on a large-scale image dataset is obtained, and candidate adapters are set for it according to the pre-trained model structure. Then, the Taylor expansion method is used to calculate the global gain of adding adapters at different locations in the network for a small number of image samples in the target domain with localized bounding boxes and class labels, thereby achieving adaptive adapter configuration. In the fine-tuning stage, the global gain of adapters at different locations for labeled image samples is recalculated. The original model parameters are frozen, and the adapter parameters are updated using gradient backpropagation with a learning rate weighted by the global gain softmax. Finally, the fine-tuned model is reused in downstream tasks to locate objects in images and provide class predictions.

[0005] Technical Solution: An adaptive model reuse method based on adapter fine-tuning achieves lightweight and efficient reuse of pre-trained models through the following steps, suitable for object detection tasks:

[0006] Step (1) Obtain the model pre-trained on a large-scale image dataset as the original model;

[0007] Step (2) Determine whether the model can be directly reused on the downstream object detection dataset. If yes, proceed to step (12); otherwise, prepare a small number of images with localization bounding boxes and category labels for fine-tuning. The bounding boxes and category labels constitute the image annotations.

[0008] Step (3) Set up candidate adapters according to the pre-trained model structure;

[0009] Step (4) Independently add candidate adapters to the network, minimize the loss on each labeled image sample, and learn the gain factor;

[0010] Step (5) uses the Taylor expansion method to calculate the global gain of the candidate adapter for each labeled image sample;

[0011] Step (6) accumulate all the labeled image samples used for fine-tuning to obtain the global gain of the candidate adapter for the target domain image;

[0012] Step (7) Adaptively configure the adapter based on the calculated global gain to obtain a new network structure;

[0013] Step (8) Recalculate the global gain of the adapter for each labeled image sample under the new network structure;

[0014] Step (9) Calculate the softmax value of the global gain;

[0015] Step (11) Freeze the original model parameters, update the adapter parameters with the learning rate weighted by global gain softmax using gradient backpropagation, iterate training until the maximum number of iterations, and obtain the fine-tuned model;

[0016] Step (12) reuses the fine-tuned model on the downstream task to locate objects in the image and give a category prediction.

[0017] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the adaptive model reuse method based on adapter fine-tuning as described above.

[0018] A computer-readable storage medium storing a computer program that performs the adapter-based adaptive model reuse method as described above.

[0019] Beneficial effects: Existing fine-tuning-based model reuse techniques have the following main problems:

[0020] (1) Manual design of transfer strategies is required. Existing methods require manual design of transfer strategies based on the features of the target domain image. Manual debugging is inefficient and seriously affects the model's reusability. Adaptive transfer strategies need to be studied.

[0021] (2) Huge parameter size. Due to the shift in features and distribution of the target domain image, existing methods usually need to adjust a large number of parameters to adapt to the new domain, especially when the number of network layers is deep, the model parameter size will become huge;

[0022] (3) All samples use the same learning strategy. The labeled images used for fine-tuning are treated as the same, and uniform hyperparameters are used, which may lead to poor model performance under conditions where small sample sizes make it difficult to collect data.

[0023] Compared with existing technologies, the adaptive model reuse method based on adapter fine-tuning provided by this invention achieves adaptive adapter configuration based on global gain with high parameter sharing and fine-grained control. Simultaneously, by performing personalized learning on the features of image samples, it achieves more accurate object localization and category prediction in object detection tasks. It should be noted that the model reuse method provided by this invention is applicable to pre-trained models with various architectures and different downstream tasks in the fields of computer vision and natural language processing. The embodiments specifically illustrate this invention using object detection as a downstream task. Attached Figure Description

[0024] Figure 1 The flowchart shows the adaptive model reuse method based on adapter fine-tuning.

[0025] Figure 2 This is a flowchart illustrating the global gain calculation process according to an embodiment of the present invention.

[0026] Figure 3 This is a flowchart illustrating the adaptive configuration process of the adapter according to an embodiment of the present invention. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0028] The adaptive model reuse method based on adapter fine-tuning proceeds as follows: Figure 1 As shown. First, a pre-trained model to be reused is obtained as the original model. If the model cannot be directly reused on the downstream task, image data with localization bounding boxes and category labels is prepared for fine-tuning. In this embodiment, the pre-trained model is a ResNet neural network pre-trained on the large-scale image data ImageNet, and the downstream task is object detection on the KITTI autonomous driving dataset. Candidate adapters are set according to the pre-trained model structure. Candidate adapters are added independently to the pre-trained model to minimize the loss on each labeled image sample and learn the gain factor. The Taylor expansion method is used to calculate the global gain of the adapter for each labeled image sample. For each candidate adapter, the global gain of all samples on the KITTI autonomous driving training dataset is accumulated. The adapter is adaptively configured for the pre-trained model according to the accumulated global gain to obtain a new network structure. The global gain of the adapter for each labeled image sample in the training set is recalculated. The original model parameters are frozen, and the adapter parameters are updated with a learning rate weighted by global gain softmax using gradient backpropagation. Iterative training is performed until the maximum number of iterations to obtain the fine-tuned model. The fine-tuned model is reused on the KITTI autonomous driving test dataset to locate objects in the image and give a category prediction.

[0029] Global gain is a measure of the importance of adding candidate adapters at specified locations in a pre-trained model for learning downstream data. The global gain calculation process in this embodiment is as follows: Figure 2 As shown, candidate adapters are first added independently to the pre-trained model, and the loss is minimized to learn the gain factor. in Let x represent the loss function for the object detection task. i This represents the i-th image sample on the KITTI autonomous driving training dataset. This is a pre-trained model that adds an adapter to the k-th basic block of the pre-trained model. ⊙ represents element-wise multiplication, α represents a factor to be optimized, and the argmin function returns the variable value that minimizes the function. Then, the Taylor expansion method is used to calculate the candidate adapter for the image sample x. i global gain Summing the global gain over all image samples yields the adapter's global gain over the target domain.

[0030] Adaptive adapter configuration is a key step in efficient model reuse. This embodiment uses adapter configuration based on the global gain of candidate adapters to the target domain, avoiding the drawback of existing methods that require manual design of configuration methods for downstream tasks. The process is as follows: Figure 3 As shown: 1) First, receive the calculated global gain θ of the candidate adapter to the target domain. k 2) Sort the candidate adapters by global gain from highest to lowest; 3) Add the candidate adapter with the highest current global gain to the pre-trained model (denoted as the current adapter); 4) Replace the global gains of other candidate adapters within the basic block containing the current adapter with additional gains, calculated as θ. k′ | k =θ kk′ -θ k , where θ kk′ This indicates that the global gain of the current adapter and the other candidate adapters within the same basic block are added simultaneously to the pre-trained model, θ. k This indicates that only the global gain of the current adapter is added; 5) Repeat steps 2)-4) until the set adapter addition ratio is reached.

[0031] Obviously, those skilled in the art should understand that the steps of the adaptive model reuse method based on adapter fine-tuning in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. An adaptive model reuse method based on adapter fine-tuning, characterized in that, The following steps enable the reuse of pre-trained models, suitable for object detection tasks: Step (1) Obtain the model pre-trained on the image dataset as the original model; Step (2) Determine whether the model can be directly reused on the downstream object detection dataset. If yes, proceed to step (12); otherwise, prepare images with localization bounding boxes and category labels for fine-tuning. Step (3) Set up candidate adapters based on the pre-trained model structure; Step (4) Independently add candidate adapters to the network, minimize the loss on each labeled image sample, and learn the gain factor; Step (5) Calculate the global gain of the candidate adapter for each labeled image sample using the Taylor expansion method; Step (6) Accumulate all labeled image samples used for fine-tuning to obtain the global gain of the candidate adapter for the target domain image; Step (7) Adaptively configure the adapter based on the calculated global gain to obtain a new network structure; Step (8) Recalculate the global gain of the adapter for each labeled image sample under the new network structure; Step (9) Calculate the softmax value of the global gain; Step (11) Freeze the original model parameters, update the adapter parameters with the learning rate weighted by global gain softmax using gradient backpropagation, iterate training until the maximum number of iterations, and obtain the fine-tuned model; Step (12) Reuse the fine-tuned model on the downstream task to locate objects in the image and give a category prediction; Global gain is a measure of the importance of adding candidate adapters at specified locations in the pre-trained model for learning downstream data; first, candidate adapters are added independently to the pre-trained model, and the loss is minimized to learn the gain factor. = ,in This represents the loss function for the object detection task. Represents the first on the training dataset Image samples, For the first time in the pre-trained model Pre-trained models with adapters added within each basic block This indicates element-wise multiplication. A factor to be optimized; Then, the Taylor expansion method is used to calculate the candidate adapter pairs for the image samples. global gain ； Summing the global gain over all image samples yields the adapter's global gain over the target domain. ; The adapter configuration is based on the global gain of the candidate adapter to the target domain. The adapter configuration process is as follows: 1) First, receive the calculated global gain of the candidate adapter to the target domain. ; 2) Then sort the candidate adapters from highest to lowest global gain; 3) Add the candidate adapter with the highest current global gain to the pre-trained model, and denote it as the current adapter; 4) Replace the global gain of other candidate adapters within the current adapter's basic block with an additional gain. The additional gain is calculated as follows: ,in This means adding the global gain of the current adapter and the other candidate adapters within the same basic block to the pre-trained model simultaneously. This indicates that only the global gain of the current adapter is added; 5) Repeat steps 2)-4) until the set adapter addition ratio is reached.

2. The adaptive model reuse method based on adapter fine-tuning according to claim 1, characterized in that, The proposed model reuse method is applicable to pre-trained models with various architectures and different downstream tasks in the fields of computer vision and natural language processing.

3. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the adaptive model reuse method based on adapter fine-tuning as described in any one of claims 1-2.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that performs the adaptive model reuse method based on adapter fine-tuning as described in any one of claims 1-2.

Images