Visual language model asymmetric fine tuning method and system and storage medium

Abstract

The invention discloses an asymmetric fine tuning method and system for a visual language model, and a storage medium, and the method employs an asymmetric decoupling fine tuning strategy for the structural heterogeneity of a visual and language module of a large visual language model: unfreezing a visual encoder, carrying out the fine tuning of a layer normalized affine parameter in a Transform block, and calibrating the feature distribution offset caused by high-resolution input; for the language model backbone, a learnable Switch GLU-Adapter module is inserted through residual parallel connection, and the fine-grained reasoning capability is enhanced; only a small number of key parameters of the model are updated, and other pre-training parameters are frozen; in a high-resolution visual question and answer task, the method not only adapts to differentiated requirements caused by heterogeneity of modules, but also remarkably reduces training calculation and storage overhead, avoids disastrous forgetting, and meanwhile ensures the adaptation performance of efficient fine tuning of parameters.

Description

A method, system, and storage medium for asymmetric fine-tuning of a visual language model Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to multimodal data processing technology. It is a method, system and storage medium for asymmetric fine-tuning of visual language models for high-resolution images. Background Technology

[0002] In recent years, large vision-language models (LVLMs) such as GPT-4V, Gemini, Qwen-VL, and DeepSeek have rapidly emerged, demonstrating outstanding performance in multimodal understanding, reasoning, and generation tasks. As shown in Figure 1, LVLMs simultaneously process image / video data input and text input, generating text output, and can perform tasks such as visual question answering, image captioning, document understanding, object detection, and video analysis. As shown in Figure 2, these models typically employ a composite architecture of "pre-trained visual encoder + vision-language interface + large language model (LLM)". To adapt these general-purpose models to specific downstream tasks, fine-tuning is required. While the traditional "full fine-tuning" method is effective, its computational and storage costs are extremely high due to the massive number of model parameters, typically in the billions to hundreds of billions, and it is prone to catastrophic forgetting, destroying the general knowledge acquired during pre-training.

[0003] Therefore, parameter-efficient fine-tuning (PEFT) techniques have emerged, such as LoRA (Low-Rank Adaptation) and Adapter. Figure 3 illustrates three main categories of PEFT schemes: selective fine-tuning, which trains only a subset of parameters while freezing the rest; additive fine-tuning, which adds learnable parameters (such as adapters or low-rank matrix factorization) to the model weights and trains only these added parameters; and cue word fine-tuning, which inserts a learnable vector before the cue word without training the model's internal parameters and trains only the inserted vector. Most existing PEFT methods employ a homogenization strategy, applying the same fine-tuning mechanism to all modules of the model, or simply freezing the visual encoder and fine-tuning only the language model. However, when dealing with high-resolution visual question answering (VQA) tasks, such as document analysis, remote sensing monitoring, and fine-grained scene understanding, existing homogenization strategies face significant challenges.

[0004] Distribution shift in visual encoders: Pre-trained visual encoders are typically trained on datasets with large-scale, commonly used datasets and low resolution. However, in high-resolution visual question answering scenarios, inputting high-resolution images, such as images larger than 1500x1500, can easily lead to a significant covariate shift in the statistical distribution of feature maps. Simply freezing the visual encoder can result in a decrease in perceptual ability.

[0005] The inference pressure on language models: High-resolution input brings an exponentially increasing number of visual detail tokens, requiring language models to have stronger fine-grained structured reasoning capabilities. Ordinary fine-tuning schemes often struggle to capture complex non-linear interactions.

[0006] In summary, existing PEFT methods cannot address the mismatch between visual details and language reasoning capabilities, especially the asymmetric challenges faced by the vision and language modules in high-resolution scenes. A differentiated fine-tuning scheme for structure perception is urgently needed. Summary of the Invention

[0007] The technical problem to be solved by this invention is that in the visual question answering task of processing high-resolution images, the existing fine-tuning methods ignore the heterogeneity of the model structure, resulting in mismatch of visual feature distribution and insufficient fine-grained reasoning ability of the language model, which in turn affects the overall performance and parameter efficiency of the model.

[0008] The technical solution of this invention is: an asymmetric fine-tuning method for visual language models, which employs different fine-tuning strategies for the visual encoder and backbone network of a large visual language model to perform asymmetric fine-tuning to adapt to downstream tasks, including the following steps:

[0009] Step 1: Construct a dataset based on the downstream task. The dataset contains images and corresponding text instructions. Construct or load a pre-trained large-scale visual language model, which includes a visual encoder, a visual-language interface module, and a large-scale language model backbone network.

[0010] Step 2: Implement an asymmetric parameter configuration strategy for large-scale visual language models:

[0011] For large language models, all parameters are frozen, and at least one trainable SwiGLU adapter module is inserted into the backbone network. The SwiGLU adapter module receives the output of the normalization layer in the backbone network, inputs the input feature vector into a gated projection layer and a content projection layer for linear mapping, applies the SiLU nonlinear activation function to the mapping result of the gated projection layer to generate a gated signal, multiplies the gated signal with the mapping result of the content projection layer element-wise, and uses the gating mechanism to dynamically filter feature information. Finally, the result of the element-wise multiplication is input into an up-dimensional projection layer for linear mapping, and the mapping result is multiplied by a fixed scaling factor and used as the output of the SwiGLU adapter module to form a residual connection with the input features of the normalization layer.

[0012] For the visual encoder and visual-language interface module, all parameters except the layer normalization layer are frozen. The learnable affine parameters in the unfrozen layer normalization layer are used to recalibrate the statistical distribution of the image features to generate a visual embedding vector. The text segmenter generates a text embedding vector for the text instructions. The visual embedding vector is mapped to the language feature space through the visual-language interface module and concatenated with the text embedding vector to form a multimodal input sequence.

[0013] Multimodal input sequences are input into the backbone network of a large language model, and the output is predicted text. During the inter-layer processing of the backbone network, the SwiGLU adapter module performs fine-grained semantic enhancement on the multimodal input sequences.

[0014] Step 3: Train the large visual language model after implementing the asymmetric parameter configuration strategy using the constructed dataset. Calculate the loss function between the generated predicted text and the target text. Based on the backpropagation algorithm, jointly update the layer normalization layer parameters in the visual encoder and visual-language interface module, as well as the SwiGLU adapter module parameters in the backbone of the large language model, to obtain the final fine-tuned model.

[0015] Furthermore, SwiGLU adapter modules are inserted in parallel into the attention layers and / or feedforward network layers of the Transformer layers of large language models.

[0016] Furthermore, the fine-tuned large-scale visual language model is evaluated on a test set. If the evaluation metrics are not met, the following optimization strategies are used to improve the model's performance, including:

[0017] By adjusting the number of input tokens for a large visual language model within the model's acceptable upper limit through image cropping techniques, and by adjusting the prompt words to adapt the model to visual question answering questions, the input limitations of the visual language model can be accommodated.

[0018] Adjust the number of Transformer layers in the visual encoder or the parameter scale of the backbone of a large language model;

[0019] Adjust the insertion position of the SwiGLU adapter module, and adjust the number of SwiGLU adapter modules inserted and the hyperparameters;

[0020] Adjust training hyperparameters, including learning rate, batch size, or scaling factor for the SwiGLU adapter module.

[0021] Furthermore, the SwiGLU adapter module adopts a zero-initialization strategy: before training begins, the weights of the up-dimensional projection layer are initialized to zero.

[0022] This invention is particularly applicable to high-resolution image scenarios, where a high-resolution image is an image with a resolution greater than 1 megapixel.

[0023] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the above-described asymmetric fine-tuning method for visual language models.

[0024] The present invention also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein when the at least one instruction or the program is executed, the above-described asymmetric fine-tuning method for visual language models is implemented.

[0025] Compared with existing technologies, this invention improves the efficiency of structure perception. It proposes an asymmetric fine-tuning framework for the heterogeneity of visual language models, employing different fine-tuning strategies for the visual encoder and the backbone of the large language model. Specifically, it unfreezes the normalization layer in the visual encoder and inserts an adapter into the backbone of the large language model. Compared to the homogeneous fine-tuning of existing technologies, this invention can allocate more suitable parameters and fine-tuning methods according to the different needs of the visual and language ends. It achieves image feature distribution calibration by fine-tuning the normalization layer in the visual end and fine-grained inference enhancement by introducing a SwiGLU adapter module in the language end, achieving a better balance between performance and efficiency. Details are as follows:

[0026] 1) Alleviating the high-resolution adaptation problem: By fine-tuning the layer normalization layer of the visual encoder, the method of this invention effectively alleviates the feature distribution shift problem caused by high-resolution input at a very low cost, which is much more efficient than retraining the expensive visual encoder or other fine-tuning schemes.

[0027] 2) Enhanced Fine-Grained Inference Capability: By introducing the SwiGLU adapter, leveraging its self-gating mechanism and non-linear expression capabilities, the model's ability to process high-resolution detailed information is significantly improved, outperforming traditional linear adapters and other classic solutions. Furthermore, in high-resolution benchmark tests, the method of this invention maintains efficient parameter fine-tuning performance while significantly reducing the number of parameters. Figure 6 shows a comparison of the training parameter count and fine-tuned model accuracy between the method of this invention and other mainstream methods. AH-PEFT is the method of this invention, the dataset used in the experiment is MME-RealWorld-Lite, and the pre-trained large-scale visual language models are Qwen3VL-8B and Qwen3VL-2B. Parameter metrics include Params (M) – number of parameters (millions), Ratio (%) – parameter activation ratio, and Score (%) – accuracy. Data shows that the method of this invention achieves accuracy comparable to or even better than other mainstream methods while significantly reducing the number of parameters. Attached Figure Description

[0028] Figure 1 is a schematic diagram of the application of a large-scale visual language model.

[0029] Figure 2 is a schematic diagram of the model structure of a large-scale visual language model.

[0030] Figure 3 is a schematic diagram of the efficient parameter fine-tuning method.

[0031] Figure 4 is a schematic diagram of the asymmetric fine-tuning framework in this invention, showing the asymmetric configuration of the visual end's thawing layer and normalization layer and the language end's inserted SwiGLU adapter.

[0032] Figure 5 is a schematic diagram of the internal structure and forward propagation of the SwiGLU adapter module in this invention.

[0033] Figure 6 shows a comparison of the number of training parameters and the accuracy of the model after fine-tuning between the present invention and other mainstream fine-tuning methods, including Fixed fine-tuning, Full FT fine-tuning, Prompt Tuning, P-Tuning, Adapter fine-tuning, Parallel Adapter, Low-Rank Adaptive (LoRA), and Hierarchical Random Adaptive (HiRA). Detailed Implementation

[0034] This invention proposes an efficient parameter fine-tuning technique for large visual language models (LVLMs) for high-resolution image input scenarios, which optimizes the problems of visual feature distribution mismatch and insufficient fine-grained reasoning ability caused by traditional homogeneous fine-tuning strategies in high-resolution visual question answering scenarios.

[0035] As shown in Figure 4, the specific implementation process of the present invention is as follows:

[0036] Step 1: Data Acquisition and Preprocessing. Obtain a high-resolution visual question-answering dataset. The dataset contains high-resolution images and corresponding question-answer pairs; high resolution refers to images with a resolution greater than 1 megapixel. Divide the dataset into training and validation sets.

[0037] Step 2: Construct the base model. A general-purpose large-scale visual language model is selected as the base model, which includes: a visual encoder: usually a ViT architecture, responsible for extracting image features; a visual-language interface module: responsible for aligning visual features to the text space; and a large-scale language model (LLM): responsible for understanding instructions and generating responses.

[0038] Step 3: Asymmetric parameter configuration. This embodiment abandons the traditional full-scale fine-tuning or uniform, symmetrical fine-tuning strategies, and adopts asymmetric configuration:

[0039] On the visual side: All parameters of the visual encoder, except for the layer normalization layer, are frozen. In actual configuration, all parameters are frozen by default; this invention only unfreezes the normalization layers, that is, only sets the parameters of the layer normalization layers in all Transformer layers to a trainable state. Simultaneously, the parameters of the layer normalization layers in the vision-language interface module are also set to a trainable state. The layer normalization layer contains learnable scaling and translation factors. During the fine-tuning training cycle of this invention, the parameter updates of the layer normalization layer only involve the scaling and translation factors, without changing the convolutional kernel weights or attention weights used in the visual encoder to extract spatial features.

[0040] On the language side: All original parameters of the large language model are frozen, and trainable SwiGLU adapter modules are inserted into the backbone network. Specifically, the proposed SwiGLU adapter modules are inserted in parallel into the attention layer and / or feedforward network layer of each Transformer layer in the large language model. The SwiGLU adapter modules can be inserted next to the attention layer or the feedforward neural network layer, or both, adjusting the number of adapters and hyperparameters according to different situations. The SwiGLU adapter module contains three linear projection layers: a gated projection layer, a content projection layer, and an upscaling projection layer, as shown in Figure 5. For input features, the forward propagation process of the SwiGLU adapter is as follows:

[0041] Using the output of the layer normalization layer as input, the input features are mapped to a lower dimension through the gated projection layer and the content projection layer, respectively.

[0042] The SiLU activation function is applied to the output of the gated projection layer to generate a gated signal;

[0043] The gating signal is multiplied element-wise with the output of the content projection layer, and important features are dynamically filtered using the gating mechanism.

[0044] The result of element-wise multiplication is mapped back to the original dimension through a dimension-up projection layer, multiplied by a fixed scaling factor, and finally residually connected (added) with the input features of the layer normalization layer.

[0045] Step 4: Train the large visual language model with asymmetric parameter configuration to achieve the following fine-tuning.

[0046] Visual feature distribution calibration. When a high-resolution image is input into the visual encoder, the feature distribution shifts because the input resolution is much higher than the resolution used during model pre-training. At this point, the model dynamically adjusts the statistical distribution of the features by training the unfrozen layer normalization parameters. Specifically, the layer normalization operation first normalizes the input features by subtracting the mean and dividing by the standard deviation, then multiplies by a learnable scaling parameter and adds a learnable bias parameter. This step ensures that the visual features entering subsequent modules are statistically stable, resolving the covariate shift problem.

[0047] Multimodal feature fusion. After the calibrated visual features are processed by the interface module, they are concatenated with the text embedding vectors to form a multimodal input sequence, which is then input into a large language model.

[0048] Fine-grained inference based on the SwiGLU adapter. Within a large language model, data flows through parallel SwiGLU adapter modules. As shown in Figure 4, the SwiGLU adapter module consists of a gated projection layer, a content projection layer, and an upscaling projection layer. The specific computation process is as follows: input features are simultaneously input into the gated projection layer and the content projection layer; the output of the gated projection layer is processed by the SiLU activation function, which is defined as the input value multiplied by the Sigmoid function value of that input value; the activated gated signal is multiplied element-wise with the output of the content projection layer; the multiplication result is input into the upscaling projection layer, and the output result is multiplied by a fixed scaling factor, and finally added to the original input features. The SwiGLU adapter module utilizes a gating mechanism to dynamically "filter" key details from the high-resolution visual feature stream based on the current text context, while suppressing irrelevant background noise. This provides stronger information filtering and non-linear modeling capabilities than traditional adapters that only use the ReLU activation function.

[0049] Step 5: Joint Training and Optimization. The cross-entropy loss function is used to calculate the difference between the predicted results and the true labels. The backpropagation algorithm is used to simultaneously update the layer normalization parameters on the visual side and the SwiGLU adapter parameters on the language side. A zero-initialization strategy is adopted during training: the weights of the up-projection layer of the SwiGLU adapter are initialized to zero to ensure that the model performance does not degrade at the beginning of training.

[0050] The fine-tuned large-scale visual language model is evaluated on a test set. If the evaluation metrics are not met, the following optimization strategies are used to improve the model's performance, including:

[0051] By adjusting the number of input tokens in the model to within the acceptable upper limit through image cropping techniques, and by adjusting the prompt words to adapt the model to visual question answering questions, the input limitations of visual language models can be met.

[0052] Adjust the number of Transformer layers in the visual encoder or the parameter scale of the backbone of a large language model to build a pre-trained visual language model that is more suitable for downstream tasks.

[0053] Adjust the insertion position of the SwiGLU adapter module, including inserting it next to the attention layer or the feedforward neural network layer. Adjust the number of SwiGLU adapter modules and hyperparameters according to different situations to achieve good results.

[0054] Adjust training hyperparameters, including learning rate, batch size, or scaling factor of the SwiGLU adapter module, to achieve better adaptation.

[0055] This invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the asymmetric fine-tuning method for large visual language models described in the above embodiments, for use in high-resolution visual question answering scenarios.

[0056] This invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, implements the asymmetric fine-tuning method suitable for large visual language models described in the above embodiments, for use in high-resolution visual question-answering scenarios. The program code for implementing the method of this invention can be written in any combination of one or more programming languages. The program code can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

Claims

1. An asymmetric fine-tuning method for a visual language model, characterized by: Different fine-tuning strategies are employed for the visual encoder and backbone network of a large visual language model to perform asymmetric fine-tuning to adapt to downstream tasks. The steps include: Step 1: Constructing a dataset based on the downstream task, containing images and corresponding text instructions; constructing or loading a pre-trained large visual language model, which includes a visual encoder, a visual-language interface module, and a large language model backbone network; Step 2: Implementing an asymmetric parameter configuration strategy for the large visual language model: For the large language model, all parameters are frozen, and at least one trainable SwiGLU adapter module is inserted into the backbone network. The SwiGLU adapter module receives the output of the normalization layer in the backbone network, inputs the input feature vector into a gated projection layer and a content projection layer for linear mapping, applies the SiLU nonlinear activation function to the mapping result of the gated projection layer to generate a gated signal; multiplies the gated signal element-wise with the mapping result of the content projection layer, dynamically filtering feature information using the gating mechanism; finally, inputs the element-wise multiplication result into an up-dimensional projection layer for linear mapping, and multiplies the mapping result by a factor. After a fixed scaling factor, the output of the SwiGLU adapter module is residually connected to the input features of the layer normalization layer. For the visual encoder and visual-language interface module, all parameters except those of the layer normalization layer are frozen. The learnable affine parameters in the unfrozen layer normalization layer are used to recalibrate the statistical distribution of the image features to generate visual embedding vectors. The text segmenter generates text embedding vectors for the text instructions. The visual embedding vectors are mapped to the language feature space through the visual-language interface module and concatenated with the text embedding vectors to form a multimodal input sequence. The multimodal input sequence is input into the backbone network of the large language model to output predicted text. During the inter-layer processing of the backbone network, the SwiGLU adapter module performs fine-grained semantic enhancement on the multimodal input sequence. Step 3: The large visual-language model after implementing the asymmetric parameter configuration strategy is trained using the constructed dataset. The loss function is calculated between the generated predicted text and the target text. The parameters of the layer normalization layer in the visual encoder and visual-language interface module, as well as the parameters of the SwiGLU adapter module in the backbone of the large language model, are jointly updated based on the backpropagation algorithm to obtain the final fine-tuned model.

2. The asymmetric fine-tuning method for visual language models according to claim 1, characterized in that: The SwiGLU adapter module is inserted in parallel into the attention layer and / or feedforward network layer of the Transformer layer of a large language model.

3. The asymmetric fine-tuning method for visual language models according to claim 1, characterized in that: The fine-tuned large visual language model was evaluated on the test set. If the evaluation metrics were not met, the following optimization strategies were used to improve the model performance: adjusting the number of input tokens of the large visual language model to within the acceptable upper limit of the model through image cropping techniques, and adjusting the prompt words to adapt the model to visual question answering questions to adapt to the input constraints of the visual language model; adjusting the number of Transformer layers in the visual encoder or the parameter scale of the backbone of the large language model; adjusting the insertion position of the SwiGLU adapter module, adjusting the number of SwiGLU adapter module insertions and hyperparameters; and adjusting training hyperparameters, including learning rate, batch size, or scaling factor of the SwiGLU adapter module.

4. The asymmetric fine-tuning method for visual language models according to claim 1, characterized in that: The SwiGLU adapter module adopts a zero-initialization strategy: before training begins, the weights of the up-projection layer are initialized to zero.

5. The asymmetric fine-tuning method for visual language models according to claim 1, characterized in that: During the training and fine-tuning of the visual encoder and visual-language interface module, the normalization layer scaling factor and translation factor are updated.

6. The asymmetric fine-tuning method for visual language models according to claim 1, characterized in that: Fine-tuning of large visual language models in high-resolution image scenarios, where high-resolution images are those with a resolution greater than 1 million pixels.

7. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the asymmetric fine-tuning method for visual language models according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements the asymmetric fine-tuning method for the visual language model as described in any one of claims 1-6.

Images