Training-free visual prompt method of multi-modal large language model, electronic equipment and storage medium

By introducing learnable latent variables and energy functions into multimodal large language models to optimize attention weights, the limitations of traditional models when dealing with complex visual details and cross-domain generalization are solved, and visual cues without additional training are achieved Integration improves the flexibility and practicality of the model.

CN119090003AActive Publication Date: 2024-12-06XIAMEN UNIV

Patent Information

Application Number
CN202411002420.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-12-06
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

Traditional multimodal large language models rely on rough image-level alignment methods, text prompts that it is difficult to accurately capture complex visual differences, and a large amount of training data and computing resources are required to improve model capabilities.

Method used

By introducing learnable latent variables in the inference process, combining them with visual features, optimizing latent variables using specific energy functions, adjusting attention weight matrix, supporting multiple visual cues such as dots, graffiti, boxes and masks, avoiding additional train.

Benefits of technology

Significantly reduce development and deployment time and cost, improve the model's description and reasoning ability of designated areas, enhance the detailed description and reasoning ability, support a variety of visual input forms, and improve extraterritorial generalization ability and interpretability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090003A_ABST
    Figure CN119090003A_ABST
Patent Text Reader

Abstract

The invention discloses a training-free visual prompt method for a multi-modal large language model, electronic equipment and a storage medium. Visual prompts can be integrated into the multi-modal large language model without training. Firstly, for a given image and a given text, visual features of the image are obtained through a visual encoder and a multi-layer perceptron, text features of the text are obtained through a text encoder, and meanwhile a learnable potential variable with the same size as the visual features is defined; thirdly, combining the potential variables with the visual features, calculating the potential variables and the visual features together with the text features through a large language model part to obtain a plurality of attention weight matrixes, and performing average pooling processing on the matrixes to form a final attention weight matrix; according to the input visual prompt, calculating and updating the value of the potential variable by using a corresponding energy function, and repeating the process for multiple times to optimize the potential variable; and finally, combining the updated potential variables with visual features, and generating a model response together with text features through a large language model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training-free visual prompting method, electronic device and storage medium for a multimodal large language model. Background Art

[0002] Large Language Models (LLMs) are deep learning models trained on large amounts of text data that can generate natural language text or understand the meaning of language text. These models typically use a Transformer architecture, have billions or even hundreds of billions of parameters, and can handle a variety of natural language tasks such as text classification, question-answering, and conversations. The LLM training process involves training on a large text corpus (billions of pages), enabling the model to learn grammatical, semantic, and conceptual relationships to generate coherent and contextually relevant language. On this basis, researchers have explored ways to integrate large language models with visual input, which has led to the birth of multimodal large language models (MLLMs), which combine language and visual information and expand the application areas and capabilities of the model.

[0003] Traditional multimodal large language models often rely on coarse image-level alignment methods, which limit users to guiding the model through textual cues for region description and reasoning, but textual cues often fail to accurately capture the complex visual differences in images. To address this problem, researchers have begun to integrate visual cues into multimodal large language models, allowing users to specify regions by providing inputs with specific coordinates. Although these methods are effective, they still require a large amount of training data and computing resources to improve the capabilities of the expanded multimodal large language models.

[0004] Therefore, how to provide a method to integrate visual cues into a large multimodal language model without training, and improve the regional description and reasoning capabilities of the large multimodal language model, has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a training-free visual prompt method, electronic device and storage medium for a multimodal large language model to solve the problems existing in the prior art. Visual prompts can be integrated into the multimodal large language model without training, thereby improving the regional description and reasoning capabilities of the multimodal large language model.

[0006] In order to achieve the above object, one of the solutions of the present invention is:

[0007] A training-free visual cueing method for a multimodal large language model comprises the following steps:

[0008] Step 1: During the inference process, for a given image and text, the image is passed through the visual encoder of the multimodal large language model LLaVA and the multi-layer perceptron for image and text alignment to obtain visual features, and the text is passed through the text encoder of LLaVA to obtain text features; define a learnable latent variable with the same width, height and depth as the visual feature;

[0009] Step 2: Combined with the strategy to prevent overfitting, four rounds of optimization of latent variables are repeated. The specific steps are:

[0010] Step 2-1, add the current latent variable to the visual feature to obtain a new visual feature. The new visual feature and the text feature are passed through the large language model part of LLaVA to obtain N attention weight matrices of the attention layer. The N attention weight matrices are average-pooled to obtain the final attention weight matrix.

[0011] Step 2-2: Based on the visual cues in the text, use the corresponding energy function to calculate the gradient of the latent variable and update the value of the latent variable;

[0012] Step 2-3: Add the optimized latent variables to the visual features, and then pass the result of the addition with the text features through the large language model part of LLaVA to obtain the final response of the model.

[0013] In step 1, the learnable latent variables are initialized to 0.

[0014] In step 2-1, the N attention weight matrices represent the attention weight matrices of each token in the text feature and the new visual feature.

[0015] In step 2-2, the visual cues include four forms: dots, scribbles, boxes, and masks. The energy function is divided into the following two forms according to the form of the visual cues:

[0016] (1) When the visual cue is in the form of dots or scribbles, the energy function at step t is The calculation formula is:

[0017]

[0018] in, represents the attention weight matrix extracted from the model during the optimization process of step t; i represents the subscript of the attention weight matrix list; r represents the specified area of ​​the visual cue; e represents the base of the natural logarithm; D i Represents the value of the distance matrix at the i-th position calculated by the distance change function in OpenCV during the optimization process; σ represents the standard deviation of the Gaussian function, set to 0.1;

[0019] (2) When the visual cue is in the form of a box or mask, the energy function at step t is The calculation formula is:

[0020]

[0021] Preferably, in step 2-2, the calculation formula for updating the latent variables is:

[0022]

[0023] Among them, p t -1, p t They represent the updated latent variables in the t-1th round and the tth round respectively; α represents the hyperparameter and is set to 400.

[0024] Preferably, in step 2, the overfitting prevention strategy includes an early stopping strategy and an exponential moving average strategy:

[0025] The early stopping strategy specifically tracks the loss during the optimization of the latent variable. If there is abs(L(t)-L( 0 )) / L( 0 )≥Δ, then stop updating the latent variables and end step 2;

[0026] Where abs(·) represents the operation of taking the absolute value; L(t) represents the loss value of the t-th step, corresponding to the energy function of the t-th step; L(0) represents the loss value of the 0-th step, corresponding to the energy function of the 0-th step; Δ represents the preset loss threshold, which is set to 0.25;

[0027] The exponential moving average strategy specifically calculates the moving average of the latent variable in the process of optimizing the latent variable, thereby stabilizing the optimization process. The calculation formula is as follows:

[0028]

[0029] in, They represent the exponential moving average of the latent variable at the tth step and the t-1th step respectively; β represents the smoothing factor, which is set to 0.5; θ(t) represents the latent variable at the tth step.

[0030] The optimization of the latent variables is only performed at step 0 of the inference process of the multimodal large language model.

[0031] The second solution of the present invention is:

[0032] An electronic device comprises a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the training-free visual prompt method of the multimodal large language model.

[0033] The third solution of the present invention is:

[0034] A computer-readable storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the training-free visual prompt method of the multimodal large language model.

[0035] After adopting the above technical solution, the present invention has the following technical effects:

[0036] (1) No additional training required: The present invention adjusts the attention weight matrix during the inference process without the need for additional training or fine-tuning of the model, which can significantly reduce the time and computational cost of development and deployment, making it more feasible and practical in resource-limited environments.

[0037] (2) Enhanced detail description and reasoning capabilities: By adding visual cues to the reasoning process, the present invention enables the multimodal large language model to better focus on the specified area and achieve more detailed and accurate description and reasoning, which is particularly important for processing tasks involving complex visual details.

[0038] (3) Support for multiple reference forms: The present invention flexibly supports multiple reference forms, including boxes, masks, scribbles, and dots. This diversity enables it to adapt to different types of visual input and user needs, enhancing the scope of application and practicality of the model.

[0039] (4) Out-of-domain generalization capability: Although no retraining is performed on new data domains or new basic multimodal large language models, the present invention also shows stronger generalization capability on different data domains. Experiments show that good performance and accuracy can be maintained even on unseen data domains.

[0040] (5) Explainability: This invention provides a more transparent reasoning process by adjusting visual cues, allowing users to more intuitively understand the working mechanism of the model and the basis for decision-making. This explainability is crucial in many practical applications, especially in areas that require high credibility and transparency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a specific embodiment of the present invention;

[0042] Figure 2 is a structural diagram of a specific embodiment of the present invention;

[0043] Figure 3 Schematic diagram of different reference forms of specific embodiments of the present invention. DETAILED DESCRIPTION

[0044] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0045] The overall idea of ​​the technical solution in this specific embodiment is as follows: an innovative method is proposed to integrate visual cues into a multimodal large language model without additional training. The method introduces a learnable latent variable and combines it with the visual features extracted from the image by the large language model, and optimizes this latent variable using a specific energy function. This optimization is based on visual cues such as points, graffiti, boxes, and masks, and aims to adjust the attention weight matrix in the large language model. By adjusting the attention weight matrix, the large language model can enhance attention to the area specified by the visual cues, thereby improving the description and reasoning capabilities of the area. The entire process does not require additional model training, which saves resources and improves the model's sensitivity and accuracy to specific visual content. The flexibility and efficiency of this method give it significant advantages in dealing with complex visual-text tasks, especially in application scenarios that require precise regional description and reasoning.

[0046] refer to Figure 1-3 As shown, the present invention discloses a training-free visual prompt method for a multimodal large language model, comprising the following steps:

[0047] Step 1: During the inference process, for a given image and text, the image is processed through the visual encoder of the multimodal large language model LLaVA and the multi-layer perceptron for image and text alignment to obtain visual features, and the text is processed through the text encoder (Tokenizer & Embedding) of LLaVA to obtain text features; define a learnable latent variable with the same width, height and depth as the visual feature;

[0048] Step 2: Combined with the strategy to prevent overfitting, four rounds of optimization of latent variables are repeated. The specific steps are:

[0049] Step 2-1: Add the current latent variables to the visual features to obtain new visual features. The new visual features and text features are passed through the large language model (LLM) in LLaVA to obtain N attention weight matrices of the attention layer. The N attention weight matrices are average-pooled to obtain the final attention weight matrix.

[0050] Step 2-2: Based on the visual cues in the text, use the corresponding energy function to calculate the gradient of the latent variable and update the value of the latent variable;

[0051] Step 2-3: Add the optimized latent variables to the visual features, and then pass the result of the addition with the text features through the large language model part of LLaVA to obtain the final response of the model.

[0052] Specific embodiments of the present invention are shown below.

[0053] In step 1 above, the learnable latent variables are initialized to 0.

[0054] In the above step 2-1, the N attention weight matrices represent the attention weight matrices of each token in the text feature and the new visual feature.

[0055] In the above step 2-2, the visual cues include four forms: dots, scribbles, boxes, and masks. The energy function is divided into the following two types according to the form of the visual cues:

[0056] (1) When the visual cue is in the form of dots or scribbles, the energy function at step t is The calculation formula is:

[0057]

[0058] in, represents the attention weight matrix extracted from the model during the optimization process of step t; i represents the subscript of the attention weight matrix list; r represents the specified area of ​​the visual cue; e represents the base of the natural logarithm; D i It represents the value of the distance matrix at the i-th position calculated by the distance transformation function (distanceTransform) in OpenCV during the optimization process; σ represents the standard deviation of the Gaussian function, which is set to 0.1.

[0059] (2) When the visual cue is in the form of a box or mask, the energy function at step t is The calculation formula is:

[0060]

[0061] Furthermore, the calculation formula for updating the latent variables is:

[0062]

[0063] Among them, p t -1, p t They represent the updated latent variables in the t-1th round and the tth round respectively; α represents the hyperparameter and is set to 400.

[0064] In step 2 above, the strategies to prevent overfitting include the Early Stop strategy and the Exponential Moving Average strategy:

[0065] The early stopping strategy specifically tracks the loss during the optimization of the latent variable. If there is abs(L(t)-L( 0 )) / L(0 )≥Δ, then stop updating the latent variables and end step 2;

[0066] Where abs(·) represents the operation of taking the absolute value; L(t) represents the loss value of the t-th step, corresponding to the energy function of the t-th step; L(0) represents the loss value of the 0-th step, corresponding to the energy function of the 0-th step; Δ represents the preset loss threshold, which is set to 0.25;

[0067] The exponential moving average strategy specifically calculates the moving average of the latent variable in the process of optimizing the latent variable, thereby stabilizing the optimization process. The calculation formula is as follows:

[0068]

[0069] in, They represent the exponential moving average of the latent variable at the tth step and the t-1th step respectively; β represents the smoothing factor, which is set to 0.5; θ(t) represents the latent variable at the tth step.

[0070] In the specific implementation of the present invention, the optimization of latent variables is only performed in step 0 of the reasoning process of the multimodal large language model.

[0071] In addition, the present invention also discloses an electronic device, including a processor, a memory and an application program; the application program is stored in the memory and is configured to execute the above method by the processor.

[0072] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0073] The following are experimental data of specific embodiments of the present invention:

[0074] This paper mainly evaluates whether the method can accurately locate and understand the semantics of the referred area on the reference target classification (ROC) task and the reference text classification (RTC) task. The experimental results are shown in the following table:

[0075]

[0076] It can be seen from the above table that the present invention shows better performance than the training methods GPT4-ROI and Shikra-7b in the reference target classification task, and the method of the present invention also shows superiority compared with the untrained color cue-based method; in the reference text classification task, all the evaluated training methods show poor out-of-domain generalization performance, while the present invention still shows the best out-of-domain generalization performance.

[0077] The ablation experiment of the relevant parameters of the present invention on the reference target classification (ROC) task is as follows:

[0078]

[0079] In summary, the advantages of the present invention are:

[0080] (1) No additional training required: The present invention adjusts the attention weight matrix during the inference process without the need for additional training or fine-tuning of the model, which can significantly reduce the time and computational cost of development and deployment, making it more feasible and practical in resource-limited environments.

[0081] (2) Enhanced detail description and reasoning capabilities: By adding visual cues to the reasoning process, the present invention enables the multimodal large language model to better focus on the specified area and achieve more detailed and accurate description and reasoning, which is particularly important for processing tasks involving complex visual details.

[0082] (3) Support for multiple reference forms: The present invention flexibly supports multiple reference forms, including boxes, masks, scribbles, and dots. This diversity enables it to adapt to different types of visual input and user needs, enhancing the scope of application and practicality of the model.

[0083] (4) Out-of-domain generalization capability: Although no retraining is performed on new data domains or new basic multimodal large language models, the present invention also shows stronger generalization capability on different data domains. Experiments show that good performance and accuracy can be maintained even on unseen data domains.

[0084] (5) Explainability: This invention provides a more transparent reasoning process by adjusting visual cues, allowing users to more intuitively understand the working mechanism of the model and the basis for decision-making. This explainability is crucial in many practical applications, especially in areas that require high credibility and transparency.

[0085] This paper overcomes the limitations of traditional large multimodal language models in processing complex visual details and cross-domain generalization through an innovative method. No additional training and fine-tuning is required, which not only reduces the computational and time costs, but also improves the flexibility and practicality of the model, greatly expanding the application scope and performance of large multimodal language models.

[0086] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A training-free visual prompting method for a multimodal large language model, characterized in that The following steps are involved: Step 1: During the inference process, for a given image and text, the image is passed through the visual encoder of the multimodal large language model LLaVA and the multi-layer perceptron for image and text alignment to obtain visual features, and the text is passed through the text encoder of LLaVA to obtain text features; define a learnable latent variable with the same width, height and depth as the visual feature; Step 2: Combined with the strategy to prevent overfitting, four rounds of optimization of latent variables are repeated. The specific steps are: Step 2-1, add the current latent variable to the visual feature to obtain a new visual feature. The new visual feature and the text feature are passed through the large language model part of LLaVA to obtain N attention weight matrices of the attention layer. The N attention weight matrices are average-pooled to obtain the final attention weight matrix. Step 2-2: Based on the visual cues in the text, use the corresponding energy function to calculate the gradient of the latent variable and update the value of the latent variable; Step 2-3: Add the optimized latent variables to the visual features, and then pass the result of the addition with the text features through the large language model part of LLaVA to obtain the final response of the model.

2. The training-free visual prompting method for a multimodal large language model as claimed in claim 1, characterized in that: In step 1, the learnable latent variables are initialized to 0.

3. The training-free visual prompting method for a multimodal large language model as claimed in claim 1, characterized in that: In step 2-1, the N attention weight matrices represent the attention weight matrices of each token in the text feature and the new visual feature.

4. The training-free visual prompting method for a multimodal large language model as claimed in claim 1, characterized in that: In step 2-2, the visual cues include four forms: dots, scribbles, boxes, and masks. The energy function is divided into the following two forms according to the form of the visual cues: (1) When the visual cue is in the form of dots or scribbles, the energy function at step t is The calculation formula is: in, represents the attention weight matrix extracted from the model during the optimization process of step t; i represents the subscript of the attention weight matrix list; r represents the specified area of ​​the visual cue; e represents the base of the natural logarithm; D i represents the value of the distance matrix at the i-th position calculated by the distance change function in OpenCV during the optimization process; σ represents the standard deviation of the Gaussian function, which is set to 0.1; (2) When the visual cue is in the form of a box or mask, the energy function at step t is The calculation formula is:

5. The training-free visual prompting method for a multimodal large language model as claimed in claim 4, characterized in that: In step 2-2, the calculation formula for updating the latent variables is: Among them, p t-1 、p t They represent the updated latent variables in the t-1th round and the tth round respectively; α represents the hyperparameter and is set to 400.

6. The training-free visual prompting method for a multimodal large language model according to any one of claims 1 to 5, characterized in that: In step 2, the overfitting prevention strategy includes an early stopping strategy and an exponential moving average strategy: The early stopping strategy specifically tracks the loss during the optimization of the latent variable. If there is abs(L (t) -L (0) ) / M (0) ≥Δ, then stop updating the latent variables and end step 2; Where abs(·) represents the operation of taking the absolute value; L (t) represents the loss value of the tth step, corresponding to the energy function of the tth step; L (0) represents the loss value of step 0, corresponding to the energy function of step 0; Δ represents the preset loss threshold, which is set to 0.25; The exponential moving average strategy specifically calculates the moving average of the latent variable in the process of optimizing the latent variable, thereby stabilizing the optimization process. The calculation formula is as follows: in, denotes the exponential sliding average of the latent variable at step t and step t-1 respectively; β denotes the smoothing factor, which is set to 0.5; θ (t) represents the latent variable at step t.

7. The training-free visual prompting method for a multimodal large language model as claimed in claim 1, characterized in that: The optimization of the latent variables is only performed at step 0 of the inference process of the multimodal large language model.

8. An electronic device, characterized in that The invention comprises a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the training-free visual prompt method for a multimodal large language model as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the training-free visual prompt method for a multimodal large language model as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Document visual language reasoning method based on layout perception prompt

    CN116822634A

Cited By

  • Visual language large model hybrid quantification method and device and electronic equipment

    CN119441513A