An Optimization Method for Visual Language Pre-training Models Based on Momentum Imitation Learning

Abstract

A method for optimizing a visual language pre-trained model using momentum imitation learning involves efficient parameter transfer learning. 1) Calculate the loss function for weighted imitation learning: 2) Update the weights of the model using momentum: 3) Calculate the mixture approximation function: (1) f t (X)=X+(XA d +a d B d +b d (2)f b (X)=XW0+XA b B d (3)f h (X)=f b (f t (X);4) Reparameterized approximation function. By mimicking the original model's weight learning process and optimizing the approximation error of the low-rank adapter, efficient optimization of the visual language pre-trained model is achieved.

Description

Technical Field

[0001] This invention belongs to the field of efficient parameter transfer learning technology, and relates to a method for transferring learning of a pre-trained model to a downstream task using a small number of adapter parameters, particularly a method for optimizing a visual language pre-trained model using momentum imitation learning. Background Technology

[0002] In recent years, large-scale pre-training based on millions of image-text datasets has become the default paradigm for visual-language learning. Leveraging vast amounts of pre-trained knowledge, Transformer-based visual-language pre-trained models can achieve better multimodal representations and cross-modal alignment than custom models, thus dominating many multimodal tasks such as visual question answering, image captioning, and image-text matching. However, these models typically require an excessive number of parameters to accommodate the massive amounts of pre-trained data, making their adaptation to downstream tasks prohibitively expensive in terms of storage costs.

[0003] Recent advances in natural language processing have offered a potential solution: efficient parameter transfer learning (PETL). The core principle of PETL is to update or insert only a small number of trainable parameters to adapt to downstream tasks, such as prompt tokens or lightweight networks (adapters), thereby avoiding the costly overhead of global parameter updates. This strategy has demonstrated competitive transfer learning performance and significant parameter efficiency across various natural language processing tasks. However, most existing PETL methods incur a considerable computational overhead during inference.

[0004] Currently, a feasible solution that better balances the efficiency and performance of pre-trained models is the reparameterization method proposed by Hu et al. (EJHu, Yelong Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in ICLR, 2022), namely Low-Rank Adaptation (LoRA). LoRA assumes that the fine-tuning parameters of large pre-trained models typically have low intrinsic dimensionality. In this case, LoRA can use rank decomposition matrices to approximate the parameter updates of dense layers in the pre-trained model. Ideally, if the approximation error of LoRA is small, it can achieve performance comparable to full tuning. Simultaneously, the low-rank weights of LoRA can be reparameterized into the model after training, achieving zero additional cost during the inference phase. While maintaining high computational efficiency, LoRA also achieves performance comparable to many PETL methods in various natural language processing tasks.

[0005] However, LoRA suffers from large approximation errors on vision-language pre-trained models, and its optimization is also very inefficient. In particular, the objective function of the pre-trained model, such as masked language modeling, often differs significantly from its downstream tasks (such as visual question answering). This discrepancy greatly increases the difficulty of LoRA's approximation. Meanwhile, LoRA's training objective is to minimize the difference between the prediction and the true value, which is also indirect and inefficient for optimizing the approximation error. Summary of the Invention

[0006] The purpose of this invention is to provide a method for optimizing visual language pre-trained models using momentum imitation learning (MoIL). By imitating the weight learning process of the original model and optimizing the approximation error of the low-rank adapter, a highly efficient optimization scheme for visual language pre-trained models can be achieved.

[0007] This invention directly fine-tunes a small number of weights in a pre-trained model and minimizes the distance between the adapter weights and the adjusted weights. The invention includes the following steps:

[0008] Step 1, calculate the loss function for weighted simulated learning:

[0009]

[0010] in This represents the objective function of the downstream task that updates the parameters of the original model. Let λ represent the objective function that minimizes the distance between the adapter parameters and the fine-tuned parameters of the original model, i.e., the objective function of momentum imitation learning, where λ represents the hyperparameters of the objective function. Simultaneously, the objective function... It can be defined as:

[0011]

[0012] Where, δ ′ δ and θ represent the parameters for low-rank global fine-tuning and partial fine-tuning, respectively. ′ φ(δ) represents the model parameters for global fine-tuning, and φ(δ) represents the approximation function.

[0013] In step 1, ViLT, VLMO, and VinVL are used as pre-trained models to compute the objective function in the downstream task. VQA2.0 and GQA are used to perform the visual question answering task, NLVR2 is used to perform the visual reasoning task, and the Flickr30K and MS-COCO datasets are used to perform the image-text matching task. At the same time, MS-COCO is used to perform the image captioning task.

[0014] In step 1, the hidden layer dimension of the adapter is set to 96, the hyperparameter λ is set to 1, and the rest of the training settings are the same as those for Prompt, LoRA, and Adapte.

[0015] Step 2, Momentum updates the model weights:

[0016]

[0017] in and Let θ′ represent the model weights for the k-th and (k-1)-th iterations, respectively. k The model weights are obtained from the backpropagation of the objective function in the k-th calculation, and α represents the momentum coefficient. Through momentum updates, the imitation learning loss function of the pre-trained model is as follows:

[0018]

[0019] The three terms added in the above equations are the distances between the original model's Q, K, and V weights and the low-rank adapter approximation function, calculated in the Transformer model.

[0020] Step 3, calculate the mixed approximation function:

[0021] (1)f t (X)=X+(XA d +a d B d +b d Where X is the input feature, A d and B d It is the projection parameter, a d and b d It is the bias, f t (X) represents the transformed adapter weight;

[0022] (2)f b (X)=XW0+XA b B d W o These are the frozen weights of the original model, f b (X) represents the bias;

[0023] (3)f h (X)=f b (f t (X) represents the weights after reparameterizing the adapter weights and model weights.

[0024] Step 4, reparameterize the approximation function:

[0025] f h (X)=f a (f d (X))

[0026] =X(W0+A) d B d W0+A b B b )+a d B d W0+b d W0

[0027] Ignoring the bias parameters, the approximation function can be simplified as:

[0028] φ h (S, ΔW) = (I + A) d B d W0+A b B b

[0029] =SW o +ΔW

[0030] Where I represents the identity matrix.

[0031] This invention provides a novel optimization method for visual language pre-trained models—Momentum Imitation Learning (MoIL). This method achieves efficient optimization of visual language pre-trained models by imitating the original model's weight learning process and optimizing the approximation error of the low-rank adapter. Experiments show that the approximation error of LoRA can be optimized through a new optimization objective: the weight distance between LoRA and the fine-tuned original model. In practice, a small number of weights in the pre-trained model are directly fine-tuned, minimizing the distance between the adapter weights and the adjusted weights. Furthermore, this invention proposes a momentum update strategy to improve the stability of imitation learning, using an exponential moving average of the adjusted weights as the imitation objective. In addition to this training scheme, a novel hybrid approximation function is proposed in MoIL for low-rank adaptation, further reducing the learning difficulty. Through these novel designs, MoIL can significantly improve optimization efficiency on downstream tasks. Attached Figure Description

[0032] Figure 1 It is the structure of LoRA and MoIL;

[0033] Figure 2 This is the momentum learning process of MoIL;

[0034] Figure 3 This is the deployment method for MoIL. Detailed Implementation

[0035] The technical solution and beneficial effects of the present invention will be described in detail below with reference to the accompanying drawings.

[0036] This invention provides a momentum imitation learning-based visual model adapter for efficient transfer learning, proposing a novel PETL scheme for pre-trained models, namely momentum imitation learning (MoIL). Figure 1 As shown, MoIL directly optimizes the weight distance by mimicking fine-tuning the weights, including the following process:

[0037] I. Model Implementation Process:

[0038] 1.1 For a specific task, calculate the loss function for weighted simulation learning (e.g., Figure 1 (As shown on the right):

[0039]

[0040] in, This represents the objective function of the downstream task that updates the parameters of the original model. Let λ represent the objective function that minimizes the distance between the adapter parameters and the fine-tuned parameters of the original model, i.e., the objective function of momentum imitation learning. λ represents the hyperparameters that adjust the objective function. Simultaneously, the objective function... It can be defined as:

[0041]

[0042] Where δ′ and δ represent the parameters of low-rank global fine-tuning and partial fine-tuning, respectively, θ′ represents the model parameters of global fine-tuning, and φ(δ) represents the approximation function.

[0043] 1.2 During the fine-tuning process and imitation learning process of downstream tasks, the momentum updates the model's weights (e.g., ... Figure 2 As shown):

[0044]

[0045] in, and Let θ′ represent the model weights for the k-th and (k-1)-th iterations, respectively. k These are the model weights obtained from the k-th backpropagation calculation of the objective function, where α represents the momentum coefficient. Through momentum updates, the imitation learning loss function of the pre-trained model is as follows:

[0046]

[0047] The three terms added in the above equations are the distances between the original model's Q, K, and V weights and the low-rank adapter approximation function, calculated in the Transformer model.

[0048] 1.3 Calculate the mixed approximation function, the module structure is as follows: Figure 3 As shown:

[0049] (1)f t(X)=X+(XA d +a d B d +b d Where X is the input feature, A d and B d It is the projection parameter, a d and b d It is the bias, f t (X) represents the transformed adapter weight;

[0050] (2)f b (X)=XW0+XA b B a Where W0 is the frozen original model weights, f b (X) represents the bias;

[0051] (3)f h (X)=f b (f t (X) represents the weights after reparameterizing the adapter weights and model weights.

[0052] The hidden layer dimension of the adapter was set to 96, the hyperparameter λ was set to 1, and the rest of the training settings were the same as those for Prompt (B. Lester, et al., “The power of scale for parameter-efficient prompt tuning,” arXiv preprint arXiv: 2104.08691, 2021; M. Jia, et al., “Visual prompt tuning,” in ECCV, 2022), LoRA (N. Houlsby, et al., “Parameter-efficient transfer learning for NLP,” in ICML, 2019) and Adapter (EJHu, et al., “LoRA: Low-rank adaptation of large language models,” in ICLR, 2022).

[0053] 1.4 Reparameterized approximation function, module structure as follows Figure 3 As shown:

[0054] f h (X)=f a (f d (X))

[0055] =X(W0+A) d B d W0+A bB b )+a d B d W0+b d W0

[0056] Ignoring the bias parameters, the approximation function can be simplified as:

[0057] φ h (S, ΔW) = (I + A) d B d W0+A b B b

[0058] =SW0+ΔW

[0059] Where I represents the identity matrix.

[0060] II. Model Training Process:

[0061] 2.1 Pre-trained model:

[0062] ViLT (W. Kim, et al., “Vilt: Vision-and-language transformer without convolution or region supervision,” in International Conference on Machine Learning, PMLR, 2021, pp. 5583-5594), VLMO (H. Bao, et al., “Vlmo: Unified vision-language pre-training with mixture-of-modality-experts,” arXiv preprint arXiv: 2111.02358, 2021), and VinVL (P. Zhang, et al., “Vinvl: Revisiting visual representations in vision-language models,” in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 2021, pp. 5579-5588) were used as pre-trained models to compute the objective function in the downstream task. ViLT is the first unified visual-language pre-trained model. It does not have a large visual skeleton and language encoder. The input image is directly input into a 12-layer Transformer block after block projection and tokenization, and the model dimension is 768. VLMO is also a unified model. Unlike ViLT, it adopts a hybrid expert mode to effectively handle different modalities. It can adopt a two-stream model like CLIP (A. Radford, et al., “Learning transferable visual models from natural language supervision,” in ICML, ser. Proceedings of Machine Leaming Research, 2021). In the experiment, the number of Transformer layers was set to 12 and the dimension was 768. VinVL uses a two-stage pipeline to process visual and linguistic features. In the experiment, the number of Transformer layers was set to 12 and the dimension was 768.

[0063] 2.2 Downstream Tasks:

[0064] Visual question answering tasks were performed using VQA2.0 (A. Radford, et al., “Learning transferable visual models from natural language supervision,” in ICML, ser. Proceedings of Machine Learning Research, 2021) and GQA (DA Hudson and CD Manning, “GQA: A new dataset for real-world visual reasoning and compositional question answering,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019, pp. 6700-6709). Visual reasoning tasks were performed using NLVR2 (A. Suhr, et al., “Acorpus for reasoning about natural language grounded in photographs,” arXiv preprint arXiv:1811.00491, 2018). Flickr30K (BAP Lumber, et al., “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence”) was used. The image-text matching task was performed using two datasets: "models," in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2641-2649, and MS-COCO (T.-Y. Lin, et al., "Microsoft coco: Common objects in context," in Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V13, Springer, 2014, pp. 740-755). Simultaneously, the image captioning task was performed using MS-COCO.Among them, VQA2.0 has 1.1 million manually annotated question-and-answer pairs, with 80,000, 40,000, and 80,000 images from MS-COCO in the training, validation, and test sets, respectively; NLVR2 has 107,000 question-and-answer pairs and 214,000 images, with 86,000, 7,000, and 7,000 samples in the training, validation, and test sets, respectively; GQA contains 113,000 images and 2.2 million question-and-answer pairs, with the training, validation, and test sets accounting for 70%, 10%, and 10%, respectively; Flicker30K has 31,000 images and 15,500 text descriptions, with 29,000, 1,000, and 1,000 images in the training, validation, and test sets, respectively; and MS-COCO has 8,000, 5,000, and 5,000 images in the training, validation, and test sets, respectively.

[0065] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A momentum imitation learning visual language pre-training model optimization method, characterized in that Includes the following steps: Step 1, calculate the loss function for weighted simulated learning: in, This represents the objective function of the downstream task that updates the parameters of the original model. The objective function represents minimizing the distance between the adapter parameters and the fine-tuned parameters of the original model; that is, the objective function of momentum imitation learning. This represents the hyperparameters used to adjust the objective function; Step 2, Momentum updates the model weights: in, and These represent the model weights for the k-th and (k-1)-th iterations, respectively. These are the model weights obtained from the k-th backpropagation calculation of the objective function. Indicates the momentum coefficient; Step 3, calculate the mixed approximation function: (1) Where X is the input feature. and These are projection parameters. and It is the bias value. Indicates the weight of the transformed adapter; (2) ,in, These are the frozen weights of the original model. This represents the features after bias-adaptive transformation; (3) The hybrid approximation function represents the specific calculation of the hybrid adaptive structure, where, Indicates the transformation adaptation module. Indicates the bias adaptation module; Step 4, reparameterize the approximation function: Ignoring the bias parameter, the approximate function can be simplified as follows: Where I represents the identity matrix.

2. The method for optimizing a visual language pre-training model based on momentum imitation learning as described in claim 1, characterized in that... In step 1, the objective function is computed in the downstream task using ViLT, VLMO and VinVL as pre-trained models, the visual question answering task is performed using VQA2.0 and GQA, the visual reasoning task is performed using NLVR2, the image-text matching task is performed using the Flickr30K and MS-COCO datasets, and the image captioning task is performed using MS-COCO.

3. The method for optimizing a visual language pre-training model based on momentum imitation learning as described in claim 1, characterized in that... In step 1, the objective function Defined as: in, and These represent the parameters for low-rank global fine-tuning and partial fine-tuning, respectively. This represents the model parameters for global fine-tuning. It represents an approximate function.

4. The method for optimizing a visual language pre-training model based on momentum imitation learning as described in claim 1, characterized in that... In step 1, the hidden layer dimension of the adapter is set to 96, and the hyperparameter is... Set it to 1, and the rest of the training settings are the same as Prompt, LoRA, and Adapter.

5. The method for optimizing a visual language pre-training model based on momentum imitation learning as described in claim 1, characterized in that... In step 2, through momentum update, the imitation learning loss function of the pre-trained model is as follows: The three terms added in the above equations are the distances between the original model's Q, K, and V weights and the low-rank adapter approximation function, calculated in the Transformer model.

Images