A multi-modal large model self-enhancement learning method based on thought chain guidance

Abstract

The present application relates to a kind of multi-modal large model self-enhancement learning method based on thought chain guide, the core of the present application is to iteratively fine-tune model by the thought chain generated by model, low-costly solve the problem of complex reasoning ability of multi-modal model is poor.This method first proposes a multi-modal large model self-enhancement learning method based on thought chain guide, through existing visual reasoning question and answer data set, guide multi-modal model to generate accurate thought chain, to build high-quality complex reasoning fine-tuning data set, iteratively enhance the complex reasoning ability of multi-modal model.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-modal large model self-enhancement learning method based on thought chain guidance, and belongs to the technical field of visual reasoning question answering. BACKGROUND

[0002] A visual question answering task is a question answering task involving natural language processing and computer vision, which usually takes an image and a natural language text question about the visual content of the image as input, and outputs an answer in the form of natural language through a visual question answering model.

[0003] Visual reasoning question answering is a subtask of visual question answering, which not only requires the model to accept and understand visual content and natural language text, but also requires the visual question answering model to have certain reasoning ability. For example, input a picture of an apple to the visual question answering model and ask it "Who is the seventeenth-century scientist most relevant to the fruit in the picture?", the model first needs to determine that the fruit in the picture is an apple, and needs to know that Newton discovered the law of universal gravitation because of the apple among the seventeenth-century scientists. The visual question answering model with reasoning ability has great potential in practical applications, such as visual search engines, etc.

[0004] With the development of large language models, multi-modal large language models have shown amazing performance on various open multi-modal understanding tasks in recent years. However, in visual-related long text generation tasks and visual reasoning question answering tasks involving complex reasoning, as the length of the generated text increases, the attention of the model to the visual input decreases, and the probability of generating incorrect content increases.

[0005] In order to enhance the explainability of large language models, a reasoning chain generation strategy simulating human slow thinking mode is proposed. The reasoning chain generation strategy specifically refers to a generation strategy that guides the model to first give a reasoning step in the decoding process and then get the final answer. For example, in the question mentioned above, the fruit in the picture is identified as an apple, and through the apple, the story of Newton discovering the law of universal gravitation because of the apple is associated, so the whole process of answering Newton is the thought chain for answering the question.

[0006] However, when the thought chain is directly applied to the multi-modal language model, as the length of the generated reasoning chain increases, the attention of the visual content decreases, and the multi-modal model is more likely to generate an output containing incorrect content. There are methods in the academic field to use artificial annotation or use commercial multi-modal models to generate complex reasoning datasets to fine-tune multi-modal models to improve the reasoning ability of the model, but the cost of artificial annotation and commercial model generation methods is high, and the effectiveness of the generated reasoning dataset cannot be guaranteed by the performance of the commercial multi-modal model.

[0007] In summary, how to obtain high-quality complex reasoning data sets at low cost is a topic worthy of in-depth study. This topic explores the direction of thought chains and model self-enhancement to solve the difficulties and key points of current methods and form a complete multi-modal large model self-enhancement learning method based on thought chain guidance. SUMMARY

[0008] To overcome the shortcomings of existing research, the present application provides a multi-modal large model self-enhancement learning method based on thought chain guidance, which is characterized by iteratively fine-tuning the model through the self-generated thought chain, and low-costly solving the problem of poor complex reasoning ability of multi-modal models. The present application first proposes a multi-modal large model self-enhancement method based on thought chain guidance, which generates accurate thought chains through existing visual reasoning question and answer data sets, thereby constructing a high-quality complex reasoning fine-tuning data set and iteratively enhancing the complex reasoning ability of multi-modal models.

[0009] A multi-modal large model self-enhancement learning method based on thought chain guidance, comprising the following steps:

[0010] Step one: collect multi-modal question and answer data sets;

[0011] Step two: build target model and verification model;

[0012] Step three: use prompt words to guide the target model to generate thought chain fine-tuning data set;

[0013] Step four: fine-tune the target model;

[0014] Step five: repeat steps three-four on the fine-tuned target model until the target model performance converges.

[0015] As a preferred embodiment, the step one is specifically as follows:

[0016] Collect existing multi-modal question and answer data sets Wherein v i represents the picture, q i represents the natural language question related to the picture content, a i represents the answer to the question, N D represents the number of samples in the data set.

[0017] As a preferred embodiment, the step two is specifically as follows:

[0018] Use a pre-trained multi-modal large model as the target model M and the verification model Y. The initial structure and model parameters of the target model M and the verification model Y should be consistent.

[0019] As a preferred embodiment, the step three is specifically as follows:

[0020] 3.1: For each multi-modal question answering data instance (q i ,v i ,a i ) in the multi-modal question answering data set D, construct the prompt word of the thinking chain generated by the guide model. The prompt word of the thinking chain generated by the guide model is composed of three parts: task definition T, example template E and input data I i .

[0021] The task definition T is used to prompt the model to generate the reasoning process according to the output template; the example template E is used as the specific content of the model output; and the input data I i is obtained by fitting the sample instance (q i ,v i ,a i ) in the data set D into the example template E. The task definition T, the example template E and the input data I i are spliced to obtain the thinking chain prompt word P i

[0022] 3.2: The thinking chain prompt word P i is input into the target model M to obtain N different thinking chains . The verification model Y will filter out the incorrect content in the thinking chain set C and integrate the information of the N thinking chains into a final thinking chain f i .

[0023] The final thinking chain f i generated by all samples in the multi-modal question answering data set D constitutes the thinking chain fine-tuning data set

[0024] As a preferred, the step four is specifically as follows:

[0025] The target model M' is fine-tuned using the thinking chain fine-tuning data set .

[0026] As a preferred, the step five is specifically as follows:

[0027] The performance of the updated target model M' is verified on the complex reasoning data set. If the performance of the target model M has not converged, the target model M is updated by M' and steps three-four are repeated until the performance of the target model M converges.

[0028] Compared with the prior art, the present application has the following beneficial effects:

[0029] The present application constructs a complex reasoning data set by guiding a multi-modal model to generate high-quality thinking chains through existing human annotations on existing visual reasoning question and answer data. The constructed complex reasoning data set is used to fine-tune the model itself, and the above process is selectively repeated multiple times to iteratively enhance the ability of the multi-modal model in the visual question and answer reasoning task without consuming additional manpower and financial resources.

[0030] The multi-modal large model self-enhancement learning method based on thinking chain guidance provided by the present application uses existing human annotations in a visual question and answer data set to guide the model to generate an accurate thinking chain fine-tuning data set at low cost and effectively, so that the model learns to generate its own thinking chain and improves the complex reasoning task of the model. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0032] Figure 1 The architecture diagram of the multi-modal large model self-enhancement learning method based on thinking chain guidance of the present application;

[0033] Figure 2 The flowchart of the multi-modal large model self-enhancement learning method based on thinking chain guidance of the present application;

[0034] Figure 3 The thinking chain generation prompt word schematic diagram of the present application;

[0035] Figure 4 The thinking chain integration prompt word schematic diagram of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0037] A multi-modal large model self-enhancement learning method based on thinking chain guidance, comprising the following steps:

[0038] Step 1: Collect multi-modal question and answer data set;

[0039] Step 2: Build target model and verification model;

[0040] Step 3: Use prompts to guide the target model to generate a mind chain fine-tuning dataset;

[0041] Step 4: Fine-tune the target model;

[0042] Step 5: Repeat steps 3 and 4 on the fine-tuned target model until the target model performance converges.

[0043] Preferably, step one is as follows:

[0044] Collect existing multimodal question answering datasets Where v i Represents an image, q i This refers to natural language questions related to the content of an image, a i N represents the answer to the question. D This indicates the number of samples in the dataset.

[0045] Preferably, step two is as follows:

[0046] Use a pre-trained multimodal large model as the target model M and the validation model Y. The initial structure and model parameters of the target model M and the validation model Y should be consistent.

[0047] As a preferred embodiment, step three is as follows:

[0048] 3.1: For each multimodal question-answering data instance (q) in the multimodal question-answering dataset D i ,v i ,a i ), constructing a guided model to generate thought chain prompts P i .like Figure 3 As shown, P i Defined by task T, example template E, and input data I. i It consists of three parts.

[0049] P i =T+E+I i #(Formula 1)

[0050] The task definition T prompts the model to generate the reasoning process according to the output template. The target model is required to generate a thought process chain c that can deduce the human-annotated answer based on the image, the question, and the corresponding human-annotated answer. The example template E serves as a prompt for the specific content of the model's output, ensuring that the target model M outputs only the thought process chain as expected. Input data I... i A sample instance (q) from dataset D i ,v i ,a i By applying the example template E, we obtain the answer a, where the human-annotated answer is a. iThe addition of the prompt word makes the model pay more attention to the visual information related to the question in the image, thereby generating more accurate content.

[0051] 3.2: As shown in Figure 2 each multimodal question and answer data instance (q i ,v i ,a i ) corresponds to a prompt word P i The input model generates N different thought chains

[0052] c ij =M(P i ) (Formula 2)

[0053] The verification model Y will filter out the wrong content in the thought chain set C and integrate the information of the N thought chains into a final thought chain f i The final thought chain f i generated by all samples of the multimodal question and answer data set D constitutes a complex reasoning fine-tuning data set The prompt word used by the verification model Y is shown in Figure 4 .

[0054] f i =Y(q i ,v i ,C i ) (Formula 3

[0055] As a preferred, the step four is specifically as follows:

[0056] The thought chain fine-tuning data set is used to fine-tune the target model M', and the fine-tuned model is used to update the target model M.

[0057] As a preferred, the step five is specifically as follows:

[0058] The performance of the updated target model M' is verified on the complex reasoning data set, and if the performance of the target model M has not converged, the target model M is updated with M' and steps three-four are repeated until the performance of the target model M converges, which is specifically as follows:

[0059] After the model is fine-tuned in step four, the model performance is verified on multiple complex reasoning data sets, and if the model accuracy increases, steps three and four are repeated to generate a thought chain data set and fine-tune the model on the updated target model; otherwise, the iteration is stopped and the final target model M is obtained.

[0060] The detailed parameters of the present application are further specifically described below.

[0061] As shown in Figures 1-4As shown, the present application provides a multi-modal large model self-enhancement learning method based on thought chain guidance.

[0062] Step one: The described collection of multi-modal question and answer data set is as follows:

[0063] The data set D is a complex reasoning data set composed of four data sets OK-VQA, A-OKVQA, GQA and OCRVQA with a scale of 40000 human annotation samples. OK-VQA and A-OKVQA are complex common sense reasoning data sets, which require the model to recognize the image content while having certain common sense reasoning ability. OCRVQA is a data set focusing on image text content, which requires the model to answer questions about the text in the image. GQA is a visual question and answer data set focusing on visual content in the image, which requires the model to infer the relative relationship of the visual content in the image.

[0064] Step two: The construction of the target model and the verification model is as follows:

[0065] The target model M and the verification model Y are both initialized using the trained LLaVA-v1.5-7b model. It needs to be emphasized that the target model will gradually update the parameters in the iterative training process, and the parameters of the verification model Y remain unchanged throughout the process. The maximum output length of the target model M and the verification model Y is set to 500.

[0066] In the target model M generation parameters, do_sampe is set to True, num_return_sequences is set to 5, and temperature is set to 1.0.

[0067] In the verification model Y generation parameters, do_sampe is set to False, num_return_sequences is set to 1, and temperature is set to 0.

[0068] Step three: The use of prompt words to guide the target model to generate thought chain fine-tuning data set is as follows:

[0069] 3.1. Take each multi-modal question and answer data instance (q i ,v i ,a i ) input Figure 2 The prompt word template is shown, and the corresponding prompt word p i is obtained.

[0070] 3.2. Thought chain generation prompt word p i Input model, get thought chain set C composed of 5 thought chains is input Figure 4The corresponding prompt word f is obtained in the prompt word template shown i .

[0071] 3.3. Prompt word f i In the input model, the data instance (q i , v i , a i ) corresponding to the final thinking chain f i is obtained. The final thinking chain f i of all instances constitutes the fine-tuning data set

[0072] Step four: fine-tune the target model, specifically as follows:

[0073] Take the fine-tuning data set O as the training data, fine-tune the model on the target model M using the default lora fine-tuning settings of the LLaVA-v1.5-7b model until the training is completed to obtain the trained model M'

[0074] Step five: repeat steps three to four on the fine-tuned target model, until the target model performance converges, specifically as follows:

[0075] 5.1. Verify the new target model M' obtained in step four on OK-VQA and A-OKVQA visual question answering data sets. If the model performance is improved, use the trained model M' to update the target model M and continue to perform steps three to four; otherwise, the current target model M is the final model.

[0076] The embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.

Claims

1. A self-reinforcement learning method for multimodal large models based on thought chain guidance, characterized in that: Includes the following steps: Step 1: Collect multimodal question answering datasets, including existing multimodal question answering datasets. ,in Represents an image. This refers to natural language questions related to the content of an image. This indicates the answer to the question. This represents the number of samples in the dataset; Step 2: Construct the target model and validation model, using a pre-trained multimodal large model as the target model. With validation model Target model With validation model The initial structure and model parameters should remain consistent; Step 3: Use prompts to guide the target model to generate a mind chain fine-tuning dataset; Step three is described in detail below: 3.1: On a multimodal question-answering dataset Each multimodal question-answering data instance in Constructing prompts for generating thought chains using guided models , Defined by task Example template and input data It consists of three parts: ; Task definition This tool is used to prompt the model to generate a reasoning process based on an output template. The target model is required to generate a thought process chain that can deduce the human-annotated answer based on an image, a question, and the corresponding human-annotated answer. Example template This serves as a template to indicate the specific content output by the model; input data From the dataset One sample instance Insert the example template Obtained, including human-annotated answers The addition of this feature allows the model to focus more on the visual information in the image that is relevant to the question, thereby generating more accurate content. 3.2: Each Multimodal Question Answering Data Instance Corresponding prompt words The input model will generate N different thought chains. ; ； Validation Model Filtering Mindset Collection Identify and correct errors in the content, and integrate information from N thought chains into a single final thought chain. In the multimodal question answering dataset The final thought chain generated from all samples Constructing a complex inference fine-tuning dataset ; ； Step 4: Fine-tune the target model; Step 5: Repeat steps 3 and 4 on the fine-tuned target model until the target model performance converges.

2. The self-reinforcement learning method for multimodal large models based on thought chain guidance according to claim 1, characterized in that: In step one, dataset D is a complex inference dataset consisting of four datasets: OK-VQA, A-OKVQA, GQA, and OCRVQA, with 40,000 human-annotated samples.

3. The self-reinforcement learning method for multimodal large models based on thought chain guidance according to claim 1, characterized in that: Step two specifically includes: Target Model With validation model All models were initialized using the pre-trained LLaVA-v1.5-7b model, and the maximum output length of both the target model M and the validation model Y was set to 500. In the parameters generated for the target model M, do_sample is set to True, num_return_sequences is set to 5, and temperature is set to 1.

0. In the parameters generated for validating model Y, do_sample is set to False, num_return_sequences is set to 1, and temperature is set to 0.

4. The self-reinforcement learning method for multimodal large models based on thought chain guidance according to claim 1, characterized in that: Step four specifically includes: To fine-tune the dataset For training data, in the target model The model was fine-tuned using the default LoRa tuning settings of the LLaVA-v1.5-7b model until training was complete, resulting in the trained model. .

5. The self-reinforcement learning method for multimodal large models based on thought chain guidance according to claim 1, characterized in that: Step five specifically includes: The new target model obtained in step four Validate the model on the OK-VQA and A-OKVQA visual question answering datasets. If the model performance is improved, use the trained model. Update target model Continue with steps three and four; otherwise, the current target model M is the final model.

Images