Multi-modal large model self-reinforcement learning method based on thinking chain guidance

Abstract

The invention relates to a multi-modal large model self-reinforcement learning method based on thinking chain guidance, and the core of the method is to solve the problem that a multi-modal model is complex and poor in reasoning capability at low cost through a model self-generated thinking chain iteration fine tuning model. According to the method, the multi-modal large model self-reinforcement learning method based on thinking chain guidance is proposed for the first time, the multi-modal model is guided to generate an accurate thinking chain through an existing visual reasoning question and answer data set, so that a high-quality complex reasoning fine adjustment data set is constructed, and the complex reasoning ability of the multi-modal model is iteratively enhanced.

Description

Technical Field

[0001] The present invention relates to a multimodal large model self-enhancement learning method based on thought chain guidance, belonging to the technical field of visual reasoning question answering. Background Art

[0002] The visual question answering task is a question answering task involving natural language processing and computer vision. It usually takes an image and a natural language text question about the visual content of the image as input, and outputs an answer in natural language form through the visual question answering model.

[0003] Visual reasoning question answering is a subtask in the field of visual question answering. It not only requires the model to accept and understand visual content and natural language text, but also requires the visual question answering model to have certain reasoning capabilities. For example, if you input a photo of an apple to a visual question answering model and ask it, "Who is the 17th century scientist most related to the fruit in the picture?", the model must first determine that the fruit in the picture is an apple, and need to know the story of Newton, one of the 17th century scientists, who discovered gravity because of an apple. Visual question answering models with reasoning capabilities have great potential in practical applications, such as visual search engines.

[0004] With the development of large language models, multimodal large language models have shown amazing performance in various open multimodal understanding tasks in recent years. However, in vision-related long text generation tasks and visual reasoning question-answering tasks involving complex reasoning, as the length of the generated text increases, the model's attention to visual input decreases, and the probability of generating incorrect content increases.

[0005] In order to enhance the interpretability of large language models, a reasoning chain generation strategy that simulates the slow thinking mode of humans has been proposed. The reasoning chain generation strategy specifically refers to a generation strategy that guides the model to first give reasoning steps in the decoding process and then get the final answer during the model generation process. For example, in the question mentioned above, the fruit in the picture is identified as an apple, and the apple is associated with the story of Newton discovering gravity because of the apple. Therefore, the whole process of answering Newton is the thinking chain to answer the question.

[0006] However, when the chain of thoughts is directly applied to the multimodal language model, as the length of the generated reasoning chain increases, the attention related to the visual content weakens, and the multimodal model is more likely to generate output containing incorrect content. The academic community has used manual annotation or commercial multimodal models to generate complex reasoning data sets to fine-tune the multimodal model to improve the model's reasoning ability, but the cost of manual annotation and commercial model generation methods is expensive, and the commercial multimodal model generation method is affected by the performance of the commercial multimodal model itself, and cannot guarantee the effectiveness of the generated reasoning data set.

[0007] In summary, how to obtain high-quality complex reasoning data sets at low cost is a topic worthy of in-depth research. This topic discusses the thinking chain and model self-enhancement, solves the difficulties and key points of the current methods, and forms a complete multimodal large model self-enhancement learning method based on the guidance of the thinking chain. Summary of the invention

[0008] In order to overcome the shortcomings of existing research, the present invention provides a multimodal large model self-enhanced learning method based on thought chain guidance. The core of the method is to solve the problem of poor complex reasoning ability of multimodal models at a low cost by fine-tuning the model through the iterative thinking chain generated by the model. This method is the first to propose a multimodal large model self-enhanced method based on thought chain guidance. Through the existing visual reasoning question and answer dataset, the multimodal model is guided to generate accurate thought chains, thereby constructing a high-quality complex reasoning fine-tuning dataset and iteratively enhancing the complex reasoning ability of the multimodal model.

[0009] A multimodal large model self-enhancement learning method based on thought chain guidance includes the following steps:

[0010] Step 1: Collect multimodal question-answering datasets;

[0011] Step 2: Build target model and verification model;

[0012] Step 3: Use prompt words to guide the target model to generate a thought chain fine-tuning dataset;

[0013] Step 4: Fine-tune the target model;

[0014] Step 5: Repeat steps 3-4 on the fine-tuned target model until the performance of the target model converges.

[0015] Preferably, the step 1 is as follows:

[0016] Collect existing multimodal question answering datasets where v i Indicates picture, q i represents a natural language question related to the image content, a i Indicates the answer to the question, N D Represents the number of samples in the dataset.

[0017] Preferably, the step 2 is as follows:

[0018] Use the pre-trained multimodal large model as the target model M and the verification model Y. The initial structure and model parameters of the target model M and the verification model Y should be consistent.

[0019] Preferably, the step three is as follows:

[0020] 3.1: Take each multimodal question answering data instance (q i ,v i ,a i ), construct the prompt words that guide the model to generate thought chains. The prompt words that guide the model to generate thought chains are composed of task definition T, example template E and input data I i It consists of three parts.

[0021] The task definition T is used to prompt the model to generate the reasoning process according to the output template; the example template E is used as a template to prompt the model to output specific content; the input data I i A sample instance (q i ,v i ,a i ) is inserted into the example template E to obtain the task definition T, the example template E and the input data I i The three parts are spliced ​​together to obtain the thinking chain prompt word P of this sample. i

[0022] 3.2: Thinking chain prompt word P i Input the target model M to obtain N different chains of thought Verification model Y will filter out the wrong content in the thought chain collection C and integrate the information of N thought chains into a final thought chain f i middle.

[0023] The final thought chain f generated by all samples of the multimodal question-answering dataset D i Constructing the Mind Chain Fine-tuning Dataset

[0024] Preferably, the step 4 is as follows:

[0025] Fine-tune the dataset using Mindchain Fine-tune the target model M'.

[0026] Preferably, the step five is as follows:

[0027] Verify the performance of the updated target model M' on a complex reasoning dataset. If the performance of the target model M has not converged, use M' to update the target model M and repeat steps three to four until the performance of the target model M converges.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention constructs a complex reasoning dataset based on existing visual reasoning question-answering data by using existing human annotations to guide the multimodal model to generate high-quality thought chains. The constructed complex reasoning dataset is used to fine-tune the model itself, and the above process is selectively repeated multiple times to iteratively enhance the ability of the multimodal model in visual question-answering reasoning tasks without consuming other additional manpower and financial resources.

[0030] The present invention proposes a multimodal large model self-enhanced learning method based on thought chain guidance. By using human annotations in existing visual question-answering datasets, the model is guided to generate accurate thought chain fine-tuning datasets in a low-cost and effective manner, so that the model can learn the thought chains generated by itself and improve the model's complex reasoning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 This is an architecture diagram of the multi-modal large model self-enhancement learning method guided by thought chain of the present invention;

[0033] Figure 2 It is a flow chart of the multi-modal large model self-enhancement learning method guided by thought chain of the present invention;

[0034] Figure 3 Generate a prompt word schematic diagram for the thought chain of the present invention;

[0035] Figure 4 This is a schematic diagram of the thought chain integration prompt words of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] A multimodal large model self-enhancement learning method based on thought chain guidance includes the following steps:

[0038] Step 1: Collect multimodal question-answering datasets;

[0039] Step 2: Build target model and verification model;

[0040] Step 3: Use prompt words to guide the target model to generate a thought chain fine-tuning dataset;

[0041] Step 4: Fine-tune the target model;

[0042] Step 5: Repeat steps 3-4 on the fine-tuned target model until the performance of the target model converges.

[0043] Preferably, the step 1 is as follows:

[0044] Collect existing multimodal question answering datasets where v i Indicates picture, q i represents a natural language question related to the image content, a i Indicates the answer to the question, N D Represents the number of samples in the dataset.

[0045] Preferably, the step 2 is as follows:

[0046] Use the pre-trained multimodal large model as the target model M and the verification model Y. The initial structure and model parameters of the target model M and the verification model Y should be consistent.

[0047] Preferably, the step three is as follows:

[0048] 3.1: For each multimodal question answering data instance (q i ,v i ,a i ), construct the prompt word P that guides the model to generate the thinking chain i .like Figure 3 As shown, P i The task definition T, the example template E and the input data I i It consists of three parts.

[0049] P i =T+E+I i #(Formula 1)

[0050] The task definition T is used to prompt the model to generate the reasoning process according to the output template, requiring the target model to generate a thought chain c that can infer the human-annotated answer based on the picture, question and corresponding human-annotated answer. The example template E is used as a template to prompt the model to output specific content, ensuring that the target model M can only output the thought chain as expected; the input data I i A sample instance (q i ,v i ,a i ) is inserted into the example template E to obtain the human-annotated answer a iThe addition of enables the model to pay more attention to the visual information in the image that is relevant to the question, thereby generating more accurate content.

[0051] 3.2: If Figure 2 As shown, each multimodal question-answering data instance (q i ,v i ,a i ) corresponding to the prompt word P i Inputting the model will generate N different chains of thought

[0052] c ij =M(P i )#(Formula 2)

[0053] Verification model Y will filter out the wrong content in the thought chain collection C and integrate the information of N thought chains into a final thought chain f i The final thought chain f generated by all samples of the multimodal question answering dataset D i Constructing complex inference fine-tuning datasets The prompt words used to verify model Y are shown in Figure 4 .

[0054] f i =Y(q i ,v i ,C i )#(Formula 3

[0055] Preferably, the step 4 is as follows:

[0056] Fine-tune the dataset using Mindchain Fine-tune the target model M' and update the target model M with the fine-tuned model.

[0057] Preferably, the step five is as follows:

[0058] Verify the performance of the updated target model M' on a complex reasoning dataset. If the performance of the target model M has not converged, use M' to update the target model M and repeat steps 3-4 until the performance of the target model M converges, as follows:

[0059] After fine-tuning the model in step 4, verify the model performance on multiple complex reasoning data sets. If the model accuracy improves, repeat step 3 to generate the thinking chain data set and step 4 to fine-tune the model on the updated target model; otherwise, stop the iteration and obtain the final target model M.

[0060] The detailed parameters of the present invention are further described in detail below.

[0061] like Figure 1-4As shown, the present invention provides a multimodal large model self-enhancement learning method based on thought chain guidance.

[0062] Step 1: Collect a multimodal question-answering dataset as described below:

[0063] Dataset D is a complex reasoning dataset with a scale of 40,000 human-annotated samples, consisting of four datasets: OK-VQA, A-OKVQA, GQA, and OCRVQA. OK-VQA and A-OKVQA are complex common sense reasoning datasets, which require the model to have certain common sense reasoning capabilities while being able to recognize the image content. OCRVQA is a dataset that focuses on the text content in the image, requiring the model to be able to answer questions about the text that appears in the image. GQA is a visual question-answering dataset that focuses on the visual content in the image, requiring the model to be able to infer the relative relationship of the visual content in the image.

[0064] Step 2: The target model and the verification model are constructed as follows:

[0065] Both the target model M and the validation model Y are initialized using the trained LLaVA-v1.5-7b model. It should be emphasized that the parameters of the target model will be gradually updated during the iterative training process, while the parameters of the validation model Y remain unchanged throughout the process. The maximum output length of both the target model M and the validation model Y is set to 500.

[0066] In the target model M generation parameters, do_sampe is set to True, num_return_sequences is set to 5, and temperature is set to 1.0.

[0067] In the verification model Y generation parameters, do_sampe is set to False, num_return_sequences is set to 1, and temperature is set to 0.

[0068] Step 3: The use of prompt words to guide the target model to generate a thought chain fine-tuning dataset is as follows:

[0069] 3.1. Take each multimodal question-answering data instance (q i ,v i ,a i )enter Figure 2 In the prompt word template shown, the corresponding prompt word p is obtained i .

[0070] 3.2. Thinking chain generation prompt word p i In the input model, a set of thinking chains consisting of 5 thinking chains is obtained C is entered Figure 4Obtain the corresponding prompt word f from the prompt word template shown i .

[0071] 3.3. Prompt word f i Input the model and obtain the data instance (q i ,v i ,a i ) corresponds to the final thought chain f i The final chain of thoughts for all instances i Constructing a fine-tuning dataset

[0072] Step 4: Fine-tune the target model, as follows:

[0073] The fine-tuning dataset O is used as the training data. The target model M is fine-tuned using the default lora fine-tuning settings of the LLaVA-v1.5-7b model until the training is completed and the trained model M' is obtained.

[0074] Step 5: Repeat steps 3 to 4 on the fine-tuned target model until the target model performance converges, as follows:

[0075] 5.1. Verify the new target model M' obtained in step 4 on visual question answering datasets such as OK-VQA and A-OKVQA. If the model performance is improved, use the trained model M' to update the target model M and continue to perform steps 3 to 4; otherwise, the current target model M is the final model.

[0076] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A multimodal large model self-enhancement learning method based on thought chain guidance, characterized by: The following steps are involved: Step 1: Collect multimodal question-answering datasets. Collect existing multimodal question-answering datasets. where v i Indicates picture, q i represents a natural language question related to the image content, a i Indicates the answer to the question, N D Indicates the number of samples in the data set; Step 2: Build the target model and the verification model. Use the pre-trained multimodal large model as the target model M and the verification model Y. The initial structure and model parameters of the target model M and the verification model Y should be consistent. Step 3: Use prompt words to guide the target model to generate a thought chain fine-tuning dataset; Step 4: Fine-tune the target model; Step 5: Repeat steps 3-4 on the fine-tuned target model until the performance of the target model converges.

2. According to claim 1, a multi-modal large model self-enhancement learning method based on thought chain guidance is characterized by: The dataset D in step 1 is a complex reasoning dataset with a scale of 40,000 human-annotated samples, consisting of four datasets: OK-VQA, A-OKVQA, GQA, and OCRVQA.

3. According to claim 1, a multi-modal large model self-enhancement learning method based on thought chain guidance is characterized by: The step 2 specifically includes: The target model M and the validation model Y are both initialized using the trained LLaVA-v1.5-7b model, and the maximum output length of the target model M and the validation model Y are both set to 500; In the target model M generation parameters, do sampe is set to True, num_retum_sequences is set to 5, and temperature is set to 1.

0. In the verification model Y generation parameters, do_sampe is set to False, num_return_sequences is set to 1, and temperature is set to 0.

4. According to claim 1, a multi-modal large model self-enhancement learning method based on thought chain guidance is characterized by: The step three specifically includes: 3.1: For each multimodal question answering data instance (q i , v i , a i ), construct the prompt word P that guides the model to generate the thinking chain i , P i The task definition T, the example template E and the input data I i It consists of three parts: P i =T+E+I i #(Formula 1) The task definition T is used to prompt the model to generate the reasoning process according to the output template. The target model is required to generate a thought chain c that can infer the human-annotated answer based on the picture, question and corresponding human-annotated answer; the example template E is used as a template to prompt the model to output the specific content; the input data I i is a sample instance (q i , v i , a i ) is inserted into the example template E to obtain the human-annotated answer a i The addition of enables the model to pay more attention to the visual information in the image that is relevant to the question and generate more accurate content; 3.2: Each multimodal question answering data instance (q i , v i , a i ) corresponding to the prompt word Pi input model, will generate N different thought chains c ij =M(P i )#(Formula 2) Verification model Y will filter out the wrong content in the thought chain collection C and integrate the information of N thought chains into a final thought chain f i The final thought chain f generated by all samples of the multimodal question answering dataset D i Constructing complex inference fine-tuning datasets f i =Y(q i , v i , C i )#(Formula 3).

5. According to claim 1, a multi-modal large model self-enhancement learning method based on thought chain guidance is characterized by: The step 4 specifically includes: The fine-tuning dataset O is used as the training data, and the target model M is fine-tuned using the default lora fine-tuning settings of the LLaVA-v1.5-7b model until the training is completed to obtain the trained model M'.

6. According to claim 1, a multi-modal large model self-enhancement learning method based on thought chain guidance is characterized by: The step five specifically includes: The new target model M' obtained in step 4 is verified on the OK-VQA and A-OKVQA visual question answering datasets. If the model performance is improved, the trained model M' is used to update the target model M and continue to perform steps 3 to 4; otherwise, the current target model M is the final model.

Images