Visual Mama model fine tuning method based on selective prompt

Abstract

The invention discloses a visual Mama model efficient fine tuning method based on selective prompt, and belongs to the technical field of image processing. According to the invention, by introducing the selective prompt module, the fine tuning process of the pre-trained visual Mamba model is converted into a prompt-based information propagation optimization method. According to the method, adaptive prompt information is generated according to the sample data set by using the selective prompt module, so that the response capability of the model to the input data is enhanced, and the performance of the classification task is effectively improved. Specifically, a double-path structure is adopted, two mechanisms of cross-layer prompting and intra-layer prompting are included, and optimization is carried out on propagation of inter-layer shared information and intra-layer specific information of the model. Information sharing among different layers is realized through cross-layer prompt, and feature consistency is ensured; secondly, in-layer prompt focuses on extraction and maintenance of specific features of each layer, so that the distinguishing capability of the model is improved; the method not only avoids the problem of disastrous forgetting, but also improves the training efficiency and performance of the visual Mama model in the downstream classification task.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a method for fine-tuning a visual Mamba model based on selective cues. Background Art

[0002] With the widespread application of large-scale pre-trained models, model fine-tuning in visual tasks has become one of the important technologies in the field of computer vision. In tasks such as image classification and object detection, pre-trained models provide good feature representation capabilities and can effectively improve the performance of downstream tasks. Therefore, how to efficiently fine-tune pre-trained models has become a hot topic in current research.

[0003] Existing visual model fine-tuning methods are usually based on models such as Vision Transformer (ViT) or Convolutional Neural Network (CNN), and rely on adjusting the entire model or some parameters to adapt to new downstream tasks. However, as the model size increases, directly fine-tuning all parameters not only requires a lot of computing resources, but also causes catastrophic forgetting in the model, that is, the model loses its memory of previous tasks when learning new tasks, affecting the performance of downstream tasks.

[0004] To solve this problem, the parameter efficient fine-tuning (PEFT) method came into being. The visual prompting technology introduces a small number of learnable parameters in the pre-trained model, which achieves effective adaptation of the model while saving computing resources. However, most of the existing visual prompting methods are based on the ViT model design, and do not fully consider the characteristics of the visual Mamba model in sequence order compression and propagation, resulting in poor performance of these methods in the application of the visual Mamba model. Therefore, there is an urgent need for a fine-tuning method specifically for the visual Mamba model to improve its information propagation ability and performance in downstream visual tasks. Summary of the invention

[0005] In order to improve the fine-tuning efficiency and performance of the large-scale pre-trained visual Mamba model in downstream image classification tasks, this paper proposes a method for fine-tuning the visual Mamba model based on selective cues. This method can not only generate cue information adapted to the input data, but also optimize the information propagation path by selectively activating the update and forget gates of the model, effectively improve the performance of the classification task, and achieve efficient fine-tuning of the model while avoiding catastrophic forgetting.

[0006] A method for fine-tuning a visual Mamba model based on selective cues, comprising:

[0007] Acquire a sample data set, wherein the sample data set includes a sample image and a classification label corresponding to the sample image;

[0008] A selective prompt module is added to each layer of the visual Mamba model, wherein the module includes a cross-layer prompt generator and an intra-layer prompt generator; the cross-layer prompt generator shares parameters between layers to generate cross-layer prompt information; the intra-layer prompter extracts and retains features of each layer to generate intra-layer prompt information;

[0009] The sample image is converted into an image token, and is input into the visual Mamba model together with a randomly initialized category token, wherein the image token is propagated forward to the last layer of the model through each layer by superimposing the cross-layer prompt information and the intra-layer prompt information, and the category token is propagated forward to the last layer of the model through each layer;

[0010] The resulting category token output by the last layer of the model is passed to the classification head to generate a probability distribution of classification predictions;

[0011] Calculate a loss function based on the probability distribution and the classification label;

[0012] The pre-trained parameters of the visual Mamba model are fixed, and the cross-layer prompt generator, the intra-layer prompt generator and the classification head are trained based on the loss function and the sample data set until convergence, so as to obtain a visual Mamba model with selective prompts.

[0013] Furthermore, the cross-layer prompt generator and the intra-layer prompt generator each include a dynamic adjustment factor for scaling the respective prompt information, and the scaled prompt information is superimposed on the input of the current layer.

[0014] Furthermore, each of the intra-layer prompt generators includes a linear dimensionality reduction layer, a linear dimensionality increase layer and a SiLU activation function.

[0015] Furthermore, each of the cross-layer hint generators includes a fully connected generator and a SiLU activation function.

[0016] Furthermore, the activation of the update gate is analyzed through visualization tools.

[0017] Furthermore, the sample data set contains diverse sample images and annotation information.

[0018] Furthermore, the image token is converted from the sample image in the following manner: the sample image is divided into image blocks, and the image token is obtained by block coding.

[0019] A method for image classification using any of the fine-tuned selectively cued visual Mamba models described above for image classification.

[0020] The beneficial effects of the present invention are as follows:

[0021] 1) The present invention proposes a visual Mamba model fine-tuning method based on selective cues. By introducing a selective cue module, it can effectively generate cue information that adapts to the input data, improve the model's responsiveness to the input data, and thus optimize the performance of the classification task.

[0022] 2) The present invention adopts a dual-path structure, including cross-cues and internal cues, which respectively optimize inter-layer shared information and intra-layer specific information, thereby improving the efficiency of information propagation, ensuring the consistency and distinguishability of features, and avoiding the problem of catastrophic forgetting.

[0023] 3) By fixing the parameters of the pre-trained model and only fine-tuning the selective prompt module and the classification head, the present invention greatly reduces the computing and storage resource requirements, while improving the training efficiency and overall performance of the model in downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of an embodiment of the present invention.

[0025] Figure 2 Schematic diagram of the visual Mamba model after selective cues of the present invention.

[0026] Figure 3 This is a visualization result diagram of the information propagation process in the SVMamba model of an embodiment of the present invention.

[0027] Figure 4 This is a visualization result diagram of updating the gate in the SVMamba model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0029] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the present invention are further described in detail below in conjunction with the accompanying drawings:

[0030] The specific implementation flow chart of the method of the present invention is as follows Figure 1 As shown, the process is as follows:

[0031] Step 1) Obtain a dataset, where the dataset contains image samples x and corresponding classification labels y for a classification task.

[0032] The present invention uses two standard datasets, HTA and VTAB-1K, which are widely used in image classification tasks for training and verifying the performance of the model. HTA (Head Tuning Adaptation) is a large-scale benchmark dataset containing 10 image classification subtasks, each representing a different visual category and scene. The dataset contains CIFAR-10, CIFAR-100, DTD, CUB-200, Stanford-Dogs, Oxford-Flowers, Food101, GTSRB, SVHN, and NABIRDS datasets. The image categories cover natural, artificial objects, and signs, and have a wide range of application scenarios, as shown in Table 1.

[0033]

[0034] Table 1

[0035] VTAB-1K (Visual Task Adaptation Benchmark) is a visual task adaptation benchmark consisting of 19 tasks divided into three categories: natural (such as CIFAR-100), specialized tasks (such as EuroSAT), and structured tasks (such as Clevr-Count), each with 1,000 training samples. These datasets are used to evaluate the adaptation performance of models in a variety of downstream tasks.

[0036] Step 2) Add a selective prompt module to the visual Mamba model. Construct a pre-trained model framework containing the visual Mamba model backbone, and introduce a dual-path selective prompt module; the selective prompt module is used to generate adaptive prompt information and enhance the information dissemination effect, such as Figure 2 shown.

[0037] Mamba and Visual Mamba models are designed based on the Selective State Space Model (SSM), which has significant linear computational efficiency and excellent performance, and is particularly suitable for processing long sequence data. The state space model converts a one-dimensional function or sequence into Map to Through N-dimensional hidden state The hidden state evolves over time, following a linear ordinary differential equation with parameters and

[0038] h′(t)=Ah(t)+Bx(t)

[0039] y(t)=Ch(t)

[0040] In order to apply the state space model to deep learning, the zero-order hold is used to discretize it. The continuous parameters A and B are discretized by the time scale parameter Convert to the corresponding discrete form: and

[0041]

[0042] Therefore, the discretized state space model can be written as:

[0043]

[0044] y i =Ch i

[0045] in Represents the N-dimensional hidden state.

[0046] The present invention constructs a pre-trained model framework including the visual Mamba model backbone and introduces a dual-path selective prompt module. The core idea of ​​the model is to dynamically adjust the model's processing of input information through an input-dependent modeling mechanism, thereby improving the effect of feature extraction.

[0047] During the construction process, the visual Mamba model pre-trained by a large-scale dataset (ImageNet-1K) is first loaded as the backbone. This model not only has deep feature extraction capabilities, but also can effectively compress input sequences and achieve efficient information dissemination. Based on the visual Mamba model, the present invention introduces a dual-path selective prompt module, which consists of a cross-layer prompt module and an intra-layer prompt module, to obtain a visual Mamba optimization model (SVMamba model) after selective prompting.

[0048] The Cross-Prompting Module is used to capture and propagate shared information between layers to ensure feature consistency. In this module, the present invention uses a fully connected cross-layer prompt generator for sharing information between layers. The generator shares parameters between layers and is fed into i Tips for generating cross-layer information in represents the input features. The number of layers sharing parameters is a hyperparameter and can be set to 6, 8, or 12. The cross-layer hint generator is a 384D to 384D fully connected generator followed by a SiLU activation function.

[0049] The Inner-Prompting Module focuses on extracting and preserving specific features of each layer to enhance the model's ability to distinguish. In order to minimize adjustable parameters while ensuring performance, this paper designs a lightweight Inner-Prompting Generator for each layer of the Visual Mamba model. The generator is able to generate different intra-layer cues The intra-layer cue generator consists of a linear dimensionality reduction layer (L down ), linear dimensionality layer (L up ) and a SiLU activation function, and the intermediate hidden dimension is set to 64.

[0050] Step 3) Train the model based on the dataset. According to the sample dataset, the selective prompt module generates prompt information to activate the update and forget gates in the visual Mamba model, thereby promoting the propagation of effective information and preventing irrelevant information from interfering with the classification process of the model. The cross entropy loss function is used to calculate the error between the model output prediction category and the actual label;

[0051] Get the input image x∈R H×W×C , where (H, W) represents the size of image x, C is the number of channels, and the image is randomly cropped to a size of (224, 224), and then the image is divided into (16, 16) image blocks. After this processing, multiple small blocks are obtained, which are embedded in the d-dimensional latent space and converted into image tokens through block coding, expressed as

[0052] The image token is input into the SVMamba model, and propagated forward to the last layer of the optimization model after each layer superimposes cross-layer prompts and intra-layer prompts. The selective prompt module of each layer in the model is based on the input x of the current layer. i Generate corresponding cross-layer prompt information And layer prompt information It can be described by the formula:

[0053]

[0054] In addition, considering the different importance of intra-layer hint information and cross-layer hint information, the present invention designs two element-wise dynamic scaling factors (α, β) to balance the impact of these hints on the input distribution.

[0055]

[0056] in and is a learnable parameter, initialized to zero, and the symbol ⊙ represents the Hadamard product.

[0057] Generated Tips Then add it to the input x of the current layer. i superior.

[0058]

[0059] Output image token of the selective prompt module in the current layer and category token Input the pre-trained Mamba block in the current layer together To extract features, the category token is randomly obtained at the beginning of the model. According to the above process, the image token and category token go through each layer of the model, and the selective hints are integrated into the input of each layer. Finally, the c output from the last Mamba block is N+1 is passed to the classification header To generate the predicted probability distribution y pred .

[0060] The pre-trained parameters of the visual Mamba model backbone are fixed, and only the selective prompt module and the classification head are trained; based on the classification loss function and the sample dataset, the model is trained until convergence to obtain the optimized image classification model.

[0061] The present invention calculates the classification loss function based on the probability distribution predicted by the model and the classification label y

[0062] The selectively cued visual Mamba model proposed in this invention introduces only a small number of learnable parameters, including the cue generator in the selective cue module. and dynamic scaling factors α, β.

[0063]

[0064] During the training process, the present invention keeps the encoder parameters of the pre-trained visual Mamba model unchanged and only allows the classification head and newly added components Based on the classification loss function and the data set, the selective prompt module and the classification head of the training model are trained until convergence to obtain a trained SVMamba model.

[0065]

[0066] Step 4) The performance of the trained SVMamba model is verified by using benchmark datasets of multiple classification tasks to test the classification accuracy of the model; at the same time, the activation of the update gate is analyzed through visualization tools to verify the optimization effect of the selective prompt module on information propagation and the improvement of the model classification performance.

[0067] Finally, the present invention verifies the performance of the trained SVMamba model to ensure its effectiveness in downstream classification tasks. The present invention uses benchmark datasets of multiple classification tasks to comprehensively test the classification accuracy of the model, and compares the performance with the existing visual cueing method in image classification tasks, as shown in Table 2. As described in step 1, these benchmark datasets include HTA and VTAB-1K, covering a variety of visual tasks, and can effectively evaluate the performance of the model in different scenarios.

[0068] During the model evaluation process, the classification accuracy of the model in the present invention is calculated on the test set and compared with the existing state-of-the-art model to verify the effectiveness of the method proposed in the present invention. The calculation formula for the classification accuracy is:

[0069]

[0070] Table 2

[0071] In addition, the present invention uses visualization tools to analyze the activation of the update gate in the SVMamba model to gain a deeper understanding of the optimization effect of the selective prompt module on information propagation. Figure 3 As shown in the figure, the present invention first visualizes the changes of the normalized update gate in the SVMamba model during the sequence propagation process, verifying the effectiveness of the present invention in promoting the propagation of model information. This visualization reveals the information retained in the hidden state of the SVMamba model as the sequence progresses, preventing the loss of discriminative information during the state space propagation process. The SVMamba model uses a lightweight selective prompter to generate token-level prompts on the entire sequence, ensuring adaptive activation of the update and forget gates in the Mamba block to promote the propagation of discriminative information, highlighting the improvement of the present method in extracting and transmitting discriminative features.

[0072] Figure 4 More examples of the update gate in the SVMamba model are shown. The method of the present invention is able to retain more discriminative information related to key elements (such as flowers and food), while less discriminative elements such as background (such as tableware) are retained to a lower degree. These visualization results show that the method of the present invention effectively activates the update gate in downstream tasks, allowing the SVMamba model to focus more on discriminative areas, thereby enhancing feature extraction and propagation.

[0073] Finally, through the visualization analysis of the model classification performance and information propagation, the present invention can comprehensively evaluate the optimization effect of the selective prompt module on improving the model classification performance. This step provides a valuable reference for further improvement and practical application of the model.

[0074] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A method for fine-tuning a visual Mamba model based on selective cues, comprising: Acquire a sample data set, wherein the sample data set includes a sample image and a classification label corresponding to the sample image; Adding a selective cue module in each layer of the visual Mamba model, wherein the module includes a cross-layer cue generator and an intra-layer cue generator; The cross-layer hint generator shares parameters between layers to generate cross-layer hint information; the intra-layer hint generator extracts and retains the features of each layer to generate intra-layer hint information; The sample image is converted into an image token, and is input into the visual Mamba model together with a randomly initialized category token, wherein the image token is propagated forward to the last layer of the model through each layer by superimposing the cross-layer prompt information and the intra-layer prompt information, and the category token is propagated forward to the last layer of the model through each layer; The result category token output by the last layer of the model is passed to the classification head to generate a probability distribution of classification prediction; the loss function is calculated based on the probability distribution and the classification label; The pre-trained parameters of the visual Mamba model are fixed, and the cross-layer prompt generator, the intra-layer prompt generator and the classification head are trained based on the loss function and the sample data set until convergence, so as to obtain a visual Mamba model with selective prompts.

2. The method according to claim 1, characterized in that: The cross-layer hint generator and the intra-layer hint generator each include a dynamic adjustment factor for scaling the respective hint information, and the scaled hint information is superimposed on the input of the current layer.

3. The method according to claim 1 or 2, characterized in that: Each of the intra-layer prompt generators includes a linear dimensionality reduction layer, a linear dimensionality increase layer and a SiLU activation function.

4. The method according to claim 1 or 2, characterized in that: Each of the cross-layer hint generators comprises a fully connected generator and a SiLU activation function.

5. The method according to claim 1 or 2, characterized in that: Analyze update gate activations with visualization tools.

6. The method according to claim 1 or 2, characterized in that: The sample data set contains diverse sample images and annotation information.

7. The method according to claim 1 or 2, characterized in that: The image token is obtained by converting the sample image in the following way: the sample image is divided into image blocks, and the image token is obtained by block coding.

8. An image classification method, characterized in that: Image classification is performed using the fine-tuned selectively cued visual Mamba model described in claims 1-7.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 8.

10. A storage medium storing a computer program, wherein when the computer program is executed by a computer, the method according to any one of claims 1 to 8 is implemented.

Images