Interactive generation content control method of multi-modal large model
By highlighting specific parts in the input context of the multimodal large model, adjusting attention weights and performing attention activation, the controllability and interpretability problems when generating text are solved, more relevant and accurate generation results are achieved, and the complexity of input prompt design is reduced.
Patent Information
- Application Number
- CN202411506353.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-25
AI Technical Summary
Existing multimodal models lack controllability and interpretability in generating text, especially for non-expert users, designing accurate input prompts is difficult, and the generation results are unstable and difficult to predict.
By highlighting specific prompt parts in the input context, adjusting attention weights and performing attention activation, interactive control of generated content is achieved, and the relevance and accuracy of generated results are improved.
It improves the relevance and accuracy of the generated results, enhances the user's control flexibility and accuracy of the generated content, reduces the complexity of input prompt design, reduces the illusion of model generation, and is compatible with existing multimodal large models.
Smart Images
Figure FDA0005104078170000011 
Figure FDA0005104078170000013
Abstract
Description
Technical Field
[0001] The present invention relates to an intersectional field of natural language processing and computer vision, and more specifically to a method for interactively generating content control of a multimodal large model. Background Art
[0002] In recent years, large-scale language models (LLMs) and visual language models (VLMs) have made remarkable progress in the fields of natural language processing and computer vision. These models have shown strong capabilities in tasks such as image description and question answering by processing multimodal inputs (such as images and text). However, due to their autoregressive generation characteristics, these models often rely on the content of the input prompts when generating text, resulting in a lack of controllability and interpretability of the generated results. Existing multimodal large model generation methods mainly rely on pre-designed input prompts, which requires users to have certain expertise and experience. However, for non-expert users, it is challenging to design specific and precise prompts, and the generated results are often unstable and difficult to predict. In addition, existing methods lack fine-grained control over the generation process and cannot meet users' needs for dynamic adjustment of generated content. Summary of the invention
[0003] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for interactively generating content control for a multimodal large model, which enables users to interactively control the generated content by highlighting specific prompt parts in the input context, thereby improving the relevance and accuracy of the generated results.
[0004] To achieve the above object, the present invention provides the following technical solution: a method for interactively generating content control of a multimodal large model, characterized in that it comprises the following steps:
[0005] Step 1: Input the multimodal token X, binary highlight mask m, guidance strength γ and scaling factor a, and then initialize;
[0006] Step 2: Generate regular and unconditional input context pairs;
[0007] Step 3: Adjust the attention weight to control the generated content by adjusting the attention weight of the highlighted part during the reasoning process;
[0008] Step 4: Perform attention activation and use the adjusted attention mask to reweight the relevant attention scores, so that the generation process is more focused on the highlighted parts, thereby completing the control of the interactively generated content.
[0009] As a further improvement of the present invention, the specific method of initialization in step 1 is: initialization token sequence x = {x1, ..., x N}.
[0010] As a further improvement of the present invention, the specific method of generating the regular and unconditional input context pairs in step 2 is: calculating the embedding s of the context and the embedding s of the unconditional context The specific formula is: s = f(x) and As a further improvement of the present invention, the specific formula for adjusting the attention weight in step 3 is as follows:
[0011] In the formula, represents the conditional probability, and γ represents the guidance strength.
[0012] As a further improvement of the present invention, the specific formula for attention activation in step 4 is as follows:
[0013] k=Q@K T +m attn
[0014] h=log(β)w+k
[0015] Among them, Q, K, V are the query, key and value in the self-attention layer, m attn is the attention mask, β is the scaling factor, and w is the highlight mask.
[0016] The beneficial effect of the present invention is that compared with the traditional large model prompt word interaction method,
[0017] 1. Improved relevance and accuracy of generated results: By highlighting specific parts in the input context, the model is guided to generate content that is more in line with expectations. This method has achieved excellent results in multimodal benchmarks such as MMBench, MSCOCO, etc.
[0018] 2. Improved flexibility and precision of user control: Dynamic user control over generated content allows users to adjust highlights in real time during the generation process to meet the specific needs of different tasks.
[0019] 3. Reduce the design complexity of input prompts: This method simplifies the design requirements for input prompts, allowing non-expert users to easily use multimodal large models for text generation without the need for complex prompt engineering.
[0020] 4. Reduce model generation hallucinations: By re-weighting the attention on the highlighted parts, the model hallucinations that occur during the generation process can be effectively reduced, thereby improving the reliability of the generated content.
[0021] 5. Strong versatility and compatibility: This method does not require additional training, is compatible with existing large multimodal models (such as LLaVA, InstmctBLIP, Omchat, etc.), and is easy to integrate and apply. DETAILED DESCRIPTION
[0022] The present invention will be further described in detail with reference to the given embodiments below.
[0023] The interactive content generation control method of a multimodal large model in this embodiment specifically includes the following steps:
[0024] initialization
[0025] Input is a multimodal token x, a binary highlight mask m, a guidance strength γ, and a scaling factor a.
[0026] Initialize the token sequence x = {x1, ..., x N}, conditional embedding s = f(x) and unconditional embedding
[0027]
[0028] Generate regular and unconditional input context pairs:
[0029] Compute the embedding s of the regular context and the embedding of the unconditional context The formula is as follows:
[0030]
[0031] Adjust attention weights:
[0032] During the inference process, the generated content is controlled by adjusting the attention weight of the highlighted part. The specific formula is as follows: in, represents the conditional probability, and γ represents the guidance strength.
[0033] Attention Activation:
[0034] The relevant attention scores are re-weighted with the adjusted attention mask to make the generation process more focused on the highlights. The formula is as follows:
[0035] k=Q@K T +m attn
[0036] h=log(β)w+k
[0037] Among them, Q, K, V are the query, key and value in the self-attention layer, m attn is the attention mask, β is the scaling factor, and w is the highlight mask.
[0038] The following three specific examples are provided in this embodiment:
[0039] Example 1: Image description task
[0040] In the image description task, the cue word highlighting method is used to highlight specific parts of the image to guide the generation of more accurate descriptions. For example, when describing an image containing a person and a background, the person part can be highlighted so that the generated description pays more attention to the details of the person.
[0041] Example 2: Interactive Dialogue Generation Task
[0042] In interactive dialogue generation tasks, the generated content of the dialogue can be controlled by highlighting the keywords in the user input. For example, in a travel recommendation system, the destination keywords entered by the user can be highlighted to make the generated recommendation content more in line with the user's needs.
[0043] Example 3: Complex document generation
[0044] In complex document generation tasks, by highlighting specific paragraphs or sentences, the model is guided to generate content related to the highlighted parts. For example, when generating a technical report, key data paragraphs can be highlighted to make the generated report more targeted and accurate.
[0045] In summary, the interactive generation content control method of the multimodal large model of this embodiment guides the model to generate more expected content by highlighting specific parts in the input context. This method has achieved excellent results in multimodal benchmark tests (such as MMBench, MSCOCO, etc.).
[0046] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A method for interactively generating content control of a multimodal large model, characterized in that: The steps include: Step 1: Input the multimodal token x, binary highlight mask m, guidance strength γ and scaling factor a, and then initialize; Step 2: Generate regular and unconditional input context pairs; Step 3: Adjust the attention weight to control the generated content by adjusting the attention weight of the highlighted part during the reasoning process; Step 4: Perform attention activation and use the adjusted attention mask to reweight the relevant attention scores so that the generation process is more focused on the highlighted parts, thereby completing the control of the interactively generated content.
2. The interactive content generation control method of a multimodal large model according to claim 1 is characterized in that: The specific method of initialization in step 1 is: initializing token sequence x={x1,...,x N }.
3. The interactive content generation control method of a multimodal large model according to claim 1 or 2, characterized in that: The specific method of generating the regular and unconditional input context pairs in step 2 is: computing the embedding s of the context and the embedding s of the unconditional context The specific formula is: s = f(x) and 4. The interactive content generation control method of a multimodal large model according to claim 3 is characterized in that: The specific formula for adjusting the attention weight in step 3 is as follows: Where P θ represents the conditional probability, and γ represents the guidance strength.
5. The interactive content generation control method of a multimodal large model according to claim 4 is characterized in that: The specific formula for attention activation in step 4 is as follows: k=Q@K T +m attn h=log(β)w+k Among them, Q, K, V are the query, key and value in the self-attention layer, m attn is the attention mask, β is the scaling factor, and w is the highlight mask.