Method and device for establishing a large multimodal model for visual question answering

Abstract

The embodiment of the present invention proposes a method and device for establishing a large multimodal model for visual question answering. The method includes: A1, obtaining a first training data set, each training data includes: at least one training image, a complex prompt word and a best answer; the complex prompt word includes a question, and also includes at least one of background text and constraint instructions; A2, extracting a training data from the first training data set, inputting the image and complex prompt word in the training data into the large multimodal model for visual question answering to be trained, and the large multimodal model for visual question answering outputs a predicted answer; calculating a loss value based on the predicted answer and the best answer in the training data; using the loss value to adjust the parameters of the large multimodal model for visual question answering; returning to step A2 until the training end condition is met. The embodiment of the present invention improves the accuracy of visual question answering.

Description

Technical Field

[0001] The present invention relates to the field of multimodal information processing technology, and in particular to a method and device for establishing a large multimodal model for visual question answering. Background Art

[0002] Currently, multimodal big models refer to models that combine multimodal information, such as text, images, video, and audio, for training. These models are used in many scenarios, such as visual question answering (VQA), intelligent customer service applications for online shopping, analysis of specific behaviors in specific scenarios, and identification of illegal operations in specific scenarios. In these scenarios, images or videos are fed into the multimodal big model along with questions, and the model generates answers.

[0003] Current approaches to visual question answering using large multimodal models involve a self-evolutionary approach mediated by instruction-based fine-tuning data. The approach uses direct preference optimization learning by asking the large multimodal model both questions with and without the image visible. The approach assumes that the answers obtained with the former are better than those obtained with the latter. This approach focuses on the match between the answer and the image content and lacks instruction-based capabilities. This means that the approach fails to understand the human intent behind the question, and therefore the answers it provides may not be ideal. Summary of the Invention

[0004] One embodiment of the present invention provides a visual question answering method and apparatus to improve the accuracy of visual question answering; another embodiment of the present invention provides a non-transitory computer-readable storage medium to improve the accuracy of visual question answering.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] A method for establishing a large multimodal model for visual question answering, the method comprising:

[0007] A1. Obtain a first training data set, where each piece of training data includes: at least one training image, a complex prompt word, and a best answer; the complex prompt word includes a question and at least one of background text and a constraint instruction; the background text is background information about the training image that needs to be incorporated or referenced when answering the question; the constraint instruction is the questioner's description of the answer;

[0008] A2. Extract a piece of training data from the first training dataset, input the image and complex prompt word in the training data into the multimodal visual question answering model to be trained, and have the multimodal visual question answering model output a predicted answer; calculate a loss value based on the predicted answer and the best answer in the training data; and use the loss value to adjust the parameters of the multimodal visual question answering model;

[0009] Return to step A2 until the training end condition is reached. The visual question answering multimodal large model at this time is the visual question answering multimodal large model finally used.

[0010] Each piece of training data in the first training data set described in step A1 further includes: a non-optimal answer;

[0011] Furthermore, the calculation of the loss value according to the predicted answer and the best answer in the training data in step A2 includes:

[0012] A direct preference optimization algorithm is used to compare the predicted answer with the best answer and the non-best answer, and the loss value is calculated.

[0013] The non-optimal answer is: an intermediate answer or a worst answer.

[0014] The step A1 further includes:

[0015] B1. Obtain a first training dataset, where each piece of training data includes at least one training image and a complex prompt word; and obtain an answer pool, where the answer pool includes multiple answers corresponding to each complex prompt word in the first training dataset;

[0016] B2. Extract a piece of training data from the first training data set in sequence, and simultaneously extract an answer corresponding to the complex prompt word in the training data from the answer pool in sequence; when an answer is extracted, the image, complex prompt word, and answer in the training data are sequentially input into the trained reward model, and the reward model outputs a reward value; when the reward value for each answer to the complex prompt word in the training data is obtained, the answer with the highest reward value is selected as the best answer, and a non-best answer is selected, and the best answer and the non-best answer are added to the training data in the first training data set;

[0017] Return to step B2 until the best answer and non-best answer are obtained for each training data in the first training data set.

[0018] The reward model is trained as follows:

[0019] C1. Extract a piece of training data from the first training data set, input the complex prompt word in the training data into multiple pre-selected answer multimodal large models, or input the same pre-selected answer multimodal large model trained using kernel sampling with different temperature coefficients, to obtain multiple answers; provide the obtained multiple answers to an operator, who annotates each answer with a score; and place the complex prompt word, the multiple answers, and their annotated scores into an answer pool;

[0020] Return to step C1, and execute step C2 until step C1 is completed for all complex prompt words in the first training data set;

[0021] C2. Extract one piece of training data from the first training data set, and extract multiple answers and their annotation scores corresponding to the complex prompt words in the training data from the answer pool;

[0022] C3. Extract one answer from the multiple answers to the complex prompt word in sequence, input the image, complex prompt word, and extracted answer in the training data into the reward model to be trained in sequence, and the reward model outputs a prediction score. The predicted score and the labeled score of the answer are calculated using a preset loss function to obtain a loss value, and the parameters of the reward model are adjusted according to the loss value;

[0023] Return to step C3 until step C3 is executed for each answer to the complex prompt word, then return to step C2 until the training end condition is met, at which time a trained reward model is obtained.

[0024] Before step A1, the method further comprises:

[0025] D1. Obtain a second training data set, where each piece of training data includes: at least one training image, a question, and a target answer;

[0026] D2. Extract a piece of training data from the second training dataset, input the image and question in the training data into the multimodal visual question answering model to be trained, and the multimodal visual question answering model outputs a predicted answer; calculate a loss value based on the predicted answer and the target answer in the training data; and use the loss value to adjust the parameters of the multimodal visual question answering model;

[0027] Return to step D2 until the training end condition is met. At this time, the visual question answering multimodal large model is the pre-trained visual question answering multimodal large model.

[0028] Furthermore, step A2 of inputting the images and complex prompt words in the training data into the visual question answering multimodal large model to be trained is: inputting the images and complex prompt words in the training data into the pre-trained visual question answering multimodal large model.

[0029] When the complex prompt word includes a question and background text, the complex prompt word is obtained in the following manner:

[0030] Extracting one piece of training data from the first training data set in sequence, wherein each piece of training data in the first training data set includes at least one training image;

[0031] If the complex prompt word defined in the extracted training data is composed of a question and background text, then the image in the training data is input into the trained image-text description multimodal model, and the model outputs a text description of the image;

[0032] Randomly select a background text example under a background type from the background text pool; input the text description of the image, the background type, and the background text example into a trained large language model that can be used for background text synthesis, and the model outputs the predicted background text;

[0033] Randomly select a capability type from a pool of question capability types; input the text description of the image, the capability type, and the predicted background text into a trained large language model for question synthesis, and the model outputs the predicted question; the question capability type pool contains all capability types covered by all questions;

[0034] The predicted question and the predicted background text are used as complex prompt words for the piece of training data, and the complex prompt words are put into the piece of training data in the first training data set.

[0035] When the complex prompt word includes a question and a constraint instruction, the complex prompt word is obtained in the following manner:

[0036] Extracting one piece of training data from the first training data set in sequence, wherein each piece of training data in the first training data set includes at least one training image;

[0037] If the complex prompt word defined in the extracted training data consists of a question and a constraint instruction, then the image in the training data is input into the trained image-text description multimodal large model, and the model outputs a text description of the image;

[0038] Randomly select a capability type from a pool of question capability types; input the text description of the image and the capability type into a large language model trained for question synthesis, and the model outputs a predicted question; the question capability type pool contains all capability types covered by all questions;

[0039] Randomly select one or more constraint instruction examples from the constraint instruction pool, input each selected constraint instruction example along with the question into a trained large language model that can be used for constraint instruction synthesis, and the model sequentially outputs one or more predicted constraint instructions; the constraint instruction pool contains constraint instruction examples under different constraint forms;

[0040] The predicted question and the predicted one or more constraint instructions are used as complex prompt words for the piece of training data, and the complex prompt words are put into the piece of training data in the first training data set.

[0041] When the complex prompt word includes a question, background text, and constraint instructions, the complex prompt word is obtained in the following manner:

[0042] Extracting one piece of training data from the first training data set in sequence, wherein each piece of training data in the first training data set includes at least one training image;

[0043] If the complex prompt word defined in the extracted training data consists of a question, background text, and constraint instructions, then the image in the training data is input into the trained image-text description multimodal large model, and the model outputs a text description of the image;

[0044] Randomly select a background text example under a background type from the background text pool; input the text description of the image, the background type, and the background text example into a trained large language model that can be used for background text synthesis, and the model outputs the predicted background text;

[0045] Randomly select a capability type from a pool of question capability types; input the text description of the image, the capability type, and the predicted background text into a trained large language model for question synthesis, and the model outputs the predicted question; the question capability type pool contains all capability types covered by all questions;

[0046] Randomly select one or more constraint instruction examples from the constraint instruction pool, input each selected constraint instruction example together with the predicted question into a trained large language model that can be used for constraint instruction synthesis, and the model sequentially outputs one or more predicted constraint instructions; the constraint instruction pool contains constraint instruction examples under different constraint forms;

[0047] The predicted question, the predicted background text, and the predicted one or more constraint instructions are used as complex prompt words for the piece of training data, and the complex prompt words are put into the piece of training data in the first training data set.

[0048] After the model outputs the predicted background text and before randomly selecting a capability type from the question capability type pool, the method further includes:

[0049] The text description of the image and the predicted background text are input into a trained large language model that can be used for background text verification. The model outputs a prompt as to whether the predicted background text and the text description of the image match. If the prompt matches, the action of randomly selecting an ability type from the question ability type pool is executed; otherwise, the action of randomly selecting a background text example under a background type from the background text pool is returned.

[0050] After the model outputs the predicted problem and before randomly selecting one or more constraint instruction examples from the constraint instruction pool, the method further includes:

[0051] The text description of the image, the predicted background text and the predicted question are input into a trained large language model that can be used for question verification. The model outputs a prompt as to whether the predicted question matches the text description of the image and the predicted background text. If the prompt matches, the action of randomly selecting one or more constraint instruction examples from the constraint instruction pool is executed; otherwise, the action of randomly selecting a capability type from the question capability type pool is returned.

[0052] When the model outputs a predicted constraint instruction, it further includes:

[0053] The predicted constraint instruction and the predicted question are input into a large language model that has been trained and can be used for constraint instruction verification. The model outputs a prompt as to whether the predicted constraint instruction and the predicted question match. If the prompt matches, the subsequent process continues; otherwise, the process returns to the action of randomly selecting a constraint instruction example from the constraint instruction pool.

[0054] The visual question answering multimodal large model at this time is the visual question answering multimodal large model finally used, and further includes:

[0055] An image of any scene and complex prompt words are input into the visual question answering multimodal large model, and the model outputs an answer.

[0056] A device for establishing a large multimodal model for visual question answering, comprising:

[0057] A first training data set acquisition module is configured to acquire a first training data set, wherein each piece of training data includes at least one training image, a complex prompt word, and a best answer; wherein the complex prompt word includes a question and at least one of background text and a constraint instruction; the background text is background information that needs to be combined or referenced with respect to the training image when answering the question; and the constraint instruction is a description of the answer required by the questioner;

[0058] The training module is used to: extract a piece of training data from the first training data set, input the image and complex prompt words in the training data into the visual question answering multimodal large model to be trained, and the visual question answering multimodal large model outputs a predicted answer; calculate the loss value based on the predicted answer and the best answer in the training data; use the loss value to adjust the parameters of the visual question answering multimodal large model until the training end conditions are met. The visual question answering multimodal large model at this time is the visual question answering multimodal large model finally used.

[0059] A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to execute any of the above methods for establishing a multimodal large model for visual question answering.

[0060] In the above embodiment, the training data not only includes questions, but also includes at least one of background text and constraint instructions, so that the predicted answers output by the visual question answering multimodal large model are more consistent with the content of the image and more consistent with the questioner's question intention, and the answers also meet the questioner's description requirements, thereby improving the instruction following ability of the visual question answering multimodal large model, that is, improving the accuracy of visual question answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 A flowchart of a method for establishing a large multimodal model for visual question answering provided by one embodiment of the present invention;

[0063] Figure 2 A flow chart of a method for obtaining the best answer and non-best answer in training data provided by one embodiment of the present invention;

[0064] Figure 3 A flowchart of a method for training a reward model according to an embodiment of the present invention;

[0065] Figure 4 A flowchart of a method for pre-training a large multimodal model for visual question answering provided by one embodiment of the present invention;

[0066] Figure 5 A flowchart of a method for synthesizing a complex prompt word when the complex prompt word includes a question and background text, provided by an embodiment of the present invention;

[0067] Figure 6 A flowchart of a method for synthesizing complex prompt words when the complex prompt words include questions and constraint instructions provided in one embodiment of the present invention;

[0068] Figure 7 A flowchart of a method for synthesizing a complex prompt word when the complex prompt word includes a question, background text, and constraint instructions, provided in one embodiment of the present invention;

[0069] Figure 8 A schematic diagram of the structure of a device for establishing a large multimodal model for visual question answering provided by one embodiment of the present invention;

[0070] Figure 9 This is an image of a kitchen in an application example of the present invention. DETAILED DESCRIPTION

[0071] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0072] The terms "first," "second," "third," "fourth," and so on (if any) in the description and claims of the present invention and in the drawings described above are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present invention described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0073] It can be seen that the existing schemes for visual question answering through multimodal large models focus more on the matching degree between the image content and the answer when answering, and fail to correctly understand the intention of the human question contained in the question. Moreover, the inventors have found through observation and analysis that in many visual question answering scenarios, it is not only necessary to ask questions only about images or videos, but there will also be other additional requirements, such as: in addition to images or videos, it is also necessary to ask questions in combination with background text and constraints. For example: in the application scenario of illegal operation identification in a specific scene, if the specific scene is the back kitchen, the illegal operation identification is: the identification of operations that do not comply with hygiene standards. At this time, in addition to providing the back kitchen picture, the back kitchen operation hygiene standard text will also be provided. At this time, the existing multimodal large model for visual question answering will not be able to combine the back kitchen picture and the back kitchen operation hygiene standard text to give a correct answer to the question asked. For example, in a specific behavior analysis scenario in a specific scenario, a picture of an intersection is provided and the answer is required to analyze the road traffic conditions, but the answer is limited to three sentences. Existing multimodal models for visual question answering are difficult to accurately analyze the question using three sentences as required, or they may fail to provide a good answer due to excessive adherence to the constraints.

[0074] Figure 1 This is a flow chart of a method for establishing a large multimodal model for visual question answering provided by one embodiment of the present invention. Figure 1 As shown, the specific steps are as follows:

[0075] Step 101: Obtain a first training data set, where each piece of training data includes at least one training image, a complex prompt word, and a best answer. The complex prompt word includes a question and at least one of background text and constraint instructions.

[0076] That is, there are three types of complex prompts: 1. Question + background text, 2. Question + constraint instructions, and 3. Question + background text + constraint instructions. When a complex prompt includes background text, there is usually only one background text; when a complex prompt includes constraint instructions, the number of constraint instructions is usually one to three.

[0077] The question is: the specific problem that the questioner expects the large multimodal model of visual question answering to solve, which directly reflects the needs of the questioner.

[0078] Background text refers to contextual information that needs to be incorporated or referenced when answering questions, specifically for the training image. This uniquely defined context is unlikely to exist in the training corpus and knowledge of the large multimodal visual question answering model. Examples of background text include safety regulations, hygiene standards, or product manuals.

[0079] Constraint instructions are the description requirements of the answer by the questioner, such as format constraints, expression constraints, grammatical constraints, and / or conditional constraints. For example, a constraint instruction can be one of the following or any combination:

[0080] 1) The answer must begin with "Summary:" and end with "Conclusion."

[0081] 2) The answer cannot contain any spaces;

[0082] 3) Answer using JSON format, including three keys: "title", "content" and "date";

[0083] 4) The answer must be more than 150 Chinese characters and divided into 3 paragraphs.

[0084] Step 102: Extract a piece of training data from the first training data set, input the image and complex prompt words in the training data into the visual question answering multimodal large model to be trained, and the visual question answering multimodal large model outputs a predicted answer; calculate the loss value based on the predicted answer and the best answer in the training data; and use the loss value to adjust the parameters of the visual question answering multimodal large model.

[0085] In step 102, extracting a piece of training data from the first training data set may include sequentially or randomly extracting a piece of training data from the first training data set.

[0086] Step 103: Return to step 102 until the training end condition is reached. The visual question answering multimodal large model at this time is the visual question answering multimodal large model finally used.

[0087] After that, for any image in any scene, the image and complex prompt words are input into the visual question answering multimodal large model, and the model outputs the corresponding answer. Among them, the complex prompt words can be questions + background text, or questions + constraint instructions, or questions + background text + constraint instructions. Among them, the image can be: a product image in an online shopping scene, and the background text can be an introduction to the product; or, the image can be an image in a traffic scene, and the background text can be traffic behavior regulations; or, the image can be a kitchen image, and the background text can be kitchen operation regulations; or, the image can be an operation screen of an operator on the production line, and the background text can be text related to the production line operation regulations; or, the image can be a screen of an area suspected of having safety hazards, and the background text can be a list of safety inspection items for the factory; or, the image can be a water surface screen, and the background text can be a description of the types of floating objects that need to be reported and the types of floating objects that do not need to be paid attention to; and so on.

[0088] In the above embodiment, the training data not only includes questions, but also includes at least one of background text and constraint instructions, so that the predicted answers output by the visual question answering multimodal large model are more consistent with the content of the image and more consistent with the questioner's questioning intention, and the answers also meet the questioner's description requirements, thereby improving the instruction following ability of the visual question answering multimodal large model, that is, improving the accuracy of visual question answering.

[0089] To further improve the command-following capability of the large multimodal model for visual question answering, the following optimization scheme is proposed:

[0090] Each piece of training data in the first training data set in step 101 further includes: a non-optimal answer;

[0091] Furthermore, in step 102 , a loss value is calculated based on the predicted answer and the best answer in the training data, including: using a DPO (Direct Preference Optimization) algorithm to compare the predicted answer with the best answer and non-best answers respectively to calculate the loss value.

[0092] The DPO algorithm is a mature algorithm, and its specific calculation process will not be described in detail in the embodiment of the present invention.

[0093] Among them, the non-best answer can be an intermediate answer or a worst answer.

[0094] In the above embodiment, by adding the best answer and the non-best answer to the training data, the predicted answer output by the visual question answering multimodal large model can be more and more inclined to the best answer and more and more away from the non-best answer during the training process, thereby further improving the instruction following ability of the visual question answering multimodal large model, that is, further improving the accuracy of visual question answering.

[0095] The following is a process of obtaining the best answer and the non-best answer for each training data in the first training data set. Obviously, this process is performed before step 101.

[0096] Figure 2 A flow chart of a method for obtaining the best answer and non-best answer in training data provided by an embodiment of the present invention, such as Figure 2 As shown, the specific steps are as follows:

[0097] Step 201: Obtain a first training data set, where each piece of training data includes at least one training image and a complex prompt word; and obtain an answer pool, where the answer pool includes multiple answers corresponding to each complex prompt word in the first training data set.

[0098] Step 201 is performed before step 101 , and the training data in the first training data set obtained at this time only contains training images and complex prompt words.

[0099] Step 202: Extract one piece of training data from the first training data set obtained in step 201 in sequence, and extract one answer corresponding to the complex prompt word in the training data from the answer pool in sequence; when an answer is extracted, input the image, complex prompt word and answer in the training data into the trained reward model in sequence, and the reward model outputs a reward value; when the reward value of each answer to the complex prompt word in the training data is obtained, take the answer with the highest reward value as the best answer, and select a non-best answer at the same time, and put the best answer and the non-best answer into the training data of the first training data set.

[0100] Among them, the non-optimal answer can be the answer with a middle reward value (i.e., the middle answer) or the answer with the lowest reward value (i.e., the worst answer).

[0101] Through step 202, the best answer and the non-best answer are obtained for a piece of training data in the first training data set.

[0102] Step 203: Return to step 202 until the best answer and non-best answer are obtained for each training data in the first training data set.

[0103] In the above embodiment, the reward value is calculated for each of the multiple answers corresponding to each complex prompt word through the trained reward model, wherein the higher the reward value, the more accurate the answer, and the answer with the highest reward value is taken as the best answer. At the same time, a non-best answer is selected, and the best answer and the non-best answer are put into the training data of the first training data set, thereby achieving the acquisition of the best answer and the non-best answer in the first training data set, laying a good foundation for using the first training data set for DPO training.

[0104] The training process of the reward model in step 202 is given below:

[0105] Figure 3 Flowchart of a method for training a reward model according to an embodiment of the present invention. Figure 3 As shown, the specific steps are as follows:

[0106] Step 301: Extract one piece of training data from the first training data set in sequence, input the complex prompt words in the training data into multiple pre-selected answer multimodal large models respectively, or input the same pre-selected answer multimodal large model trained using kernel sampling with different temperature coefficients in sequence to obtain multiple answers; provide the obtained multiple answers to the operator, who annotates the score of each answer; and put the complex prompt word and the multiple answers and their annotated scores into the answer pool.

[0107] Because the complex prompts in the first training dataset contain a wide variety of questions and constraints, manually labeling answers (including optimal and suboptimal answers) for each complex prompt is extremely difficult. Even for some complex prompts, manual labeling is difficult. Therefore, answer synthesis requires the use of a large multimodal model. To increase answer diversity and ultimately improve the accuracy of the large multimodal model for visual question answering, multiple pre-selected answer multimodal models can be used to generate different answers for the same complex prompt. Alternatively, the same pre-selected answer multimodal model can be used but with kernel sampling using different temperature coefficients to generate different answers.

[0108] Here, using multiple pre-selected answer multimodal models, or using the same pre-selected answer multimodal model but sampling with kernels of different temperature coefficients, to generate multiple responses for the same complex prompt word aims to increase response diversity and ultimately improve the command-following capabilities of the visual question-answering multimodal model. This is because: different pre-selected answer multimodal models, or the same multimodal model using kernels of different temperature coefficients, focus differently when synthesizing responses. For example, a question that model A cannot answer might be answered by model B. Furthermore, the responses synthesized by a single model are not necessarily correct, but using multiple models or kernels of different temperature coefficients increases the probability of obtaining accurate responses.

[0109] Among them, the multimodal large models of each pre-selected answer have been pre-trained and have been verified to be models with better answering effects. The training data set contains a large number of complex prompt words and corresponding annotated answers.

[0110] Step 302: Return to step 301 until step 301 is executed for all complex prompt words in the first training data set, and then execute step 303.

[0111] Step 303: extract one piece of training data from the first training data set in sequence, and simultaneously extract multiple answers and their annotation scores corresponding to the complex prompt words in the training data from the answer pool.

[0112] Step 304: Extract one answer from the multiple answers to the complex prompt word in turn, input the image, complex prompt word and extracted answer in the training data into the reward model to be trained in turn, and the reward model outputs a prediction score. A preset loss function is used to calculate the prediction score and the labeled score of the answer to obtain a loss value, and the parameters of the reward model are adjusted according to the loss value.

[0113] Step 305: Return to step 304 until step 304 is executed for each answer to the complex prompt word, then return to step 303 until the training end condition is met, at which time a trained reward model is obtained.

[0114] In practical applications, before using the first training dataset to train the visual question answering multimodal large model, especially before using the DPO method to train the visual question answering multimodal large model, in order to improve the accuracy of the trained visual question answering multimodal large model, the visual question answering multimodal large model can be pre-trained first, and then the pre-trained visual question answering multimodal large model can be further trained using the second training dataset to obtain the final visual question answering multimodal large model. In particular, during pre-training, complex prompt words are not used, and only simpler questions are used for training.

[0115] Figure 4 This is a flowchart of a method for pre-training a large multi-modal model for visual question answering provided by an embodiment of the present invention. Obviously, the pre-training process of the large multi-modal model for visual question answering is performed before step 101. Figure 4 As shown, the specific steps are as follows:

[0116] Step 401: Obtain a second training data set, where each piece of training data includes at least one training image, a question, and a target answer.

[0117] Step 402: Extract a piece of training data from the second training data set, input the image and question in the training data into the visual question answering multimodal large model to be trained, and the visual question answering multimodal large model outputs a predicted answer; calculate the loss value based on the predicted answer and the target answer in the training data; and use the loss value to adjust the parameters of the visual question answering multimodal large model.

[0118] Step 403: Return to step 402 until the training end condition is reached. At this time, the visual question answering multimodal large model is a pre-trained visual question answering multimodal large model.

[0119] Thereafter, in step 102, the image and complex prompt words in the training data are input into the visual question answering multimodal model to be trained: the image and complex prompt words in the training data are input into the pre-trained visual question answering multimodal model.

[0120] Since the complex prompt words in the second training dataset are diverse, specifically, the questions, background text, and constraint instructions in the complex prompt words are all diverse, it is unrealistic to use manual annotation for complex prompt words. The following gives the corresponding synthesis methods based on the three composition methods of complex prompt words:

[0121] Figure 5 A flowchart of a method for synthesizing complex prompt words when the complex prompt words include questions and background text is provided in one embodiment of the present invention. Figure 5 As shown, the specific steps are as follows:

[0122] Step 501: Extract one piece of training data from the first training data set, wherein each piece of training data in the first training data set includes at least one training image. If the complex prompt word defined in the extracted training data piece consists of a question and background text, execute the following steps:

[0123] Step 501 is performed before step 101 , and the purpose of this embodiment is to synthesize complex prompt words. Therefore, the first training data set at this time only includes training images.

[0124] Step 502: Input the image in the training data into the trained image-text description multimodal model, and the model outputs the text description of the image.

[0125] The text description of an image includes the scene, objects, and the relationships between them. These relationships can include spatial relationships and / or identity relationships between objects. Sufficiently detailed information in the text description of an image can be used as material for question synthesis.

[0126] Among them, the image-text description multimodal large model has been pre-trained, and the training dataset used contains a large number of images and corresponding annotated text descriptions. The annotated text descriptions include: text descriptions of the image scene, targets, and the relationship between targets.

[0127] Step 503: Randomly select a background text example under a background type from the background text pool; input the text description of the image, the background type, and the background text example into a large language model trained for background text synthesis, which outputs the predicted background text. The background text pool contains background text examples under each background type, and each background type contains one or more background text examples.

[0128] In practical applications, we first count the possible background types covered by the background text. Then, we can generate a multimodal large model using pre-trained background text examples to produce several background text examples for each background type. The background text examples under each background type constitute the background text pool. Background types include: knowledge-based background, standard-defined background, process-based background, social and application-based background, document-based background, and restricted background. The following describes each background type:

[0129] 1) Knowledge background: It can be knowledge content in any field related to the image.

[0130] 2) Normative and standard definition context: This can be the definition of behavioral norms, visual descriptions of behaviors, management standards, definitions of counterfactuals, or rules for counterfactuals, etc.

[0131] 3) Process background: Give the process steps required to solve certain problems and achieve certain goals.

[0132] 4) Social and application background: information related to social production, life, or industry.

[0133] 5) Document-based background: For example, the ingredient list or nutritional composition table of the food in the picture, generating the manual, usage document, or requirement document of the equipment or product in the picture.

[0134] 6) Restrictive context: Provide some constraints in the context to limit the scope or options of the problem that the large multimodal visual question answering model can solve.

[0135] The number of background text examples in each background type does not need to be large, about 10 is enough, mainly serving as example samples for the large language model that can be used for background text synthesis to refer to when generating background text.

[0136] In an optional embodiment, in order to facilitate the large language model that can be used for background text synthesis to identify which information in the input information is the text description of the image, which information is the background type, and which information is the background text example, a background text synthesis prompt word template is pre-defined. The background text synthesis prompt word template defines the format of the background text synthesis prompt word, and the format specifies the location of the text description of the image, the background type, and the background text example. For example, the background text synthesis prompt word template can be: "Based on the following image text description: '...', the following background type: '...' and the following background text example: '...' synthesis problem", where the first place... should be filled with the text description of the image, the second place... should be filled with the selected background type, and the third place... should be filled with the selected background text example. Furthermore, in this step 503, the text description of the image, the background type, and the background text example are input into a trained large language model that can be used for background text synthesis, specifically including: inserting the text description of the image, the background type, and the background text example into a predefined background text synthesis prompt word template to obtain background text synthesis prompt words, and inputting the background text synthesis prompt words into the trained large language model that can be used for background text synthesis.

[0137] Among them, the large language model that can be used for background text synthesis has been pre-trained, and the training dataset used contains at least: a large number of image text descriptions and corresponding background types and background text examples, as well as corresponding annotated background text.

[0138] Step 504: Randomly select a capability type from the question capability type pool; input the text description of the image, the capability type and the predicted background text into a trained large language model that can be used for question synthesis, and the model outputs the predicted question.

[0139] In an optional embodiment, to facilitate the large language model used for question synthesis in identifying which information in the input information is the text description of the image, which is the ability type, and which is the background text, a question synthesis prompt word template is predefined. The question synthesis prompt word template defines the format of the question synthesis prompt word, which specifies the location of the image text description, ability type, and background text. For example, the question synthesis prompt word template may be: "Based on the following image text description: '...' and the following background text: '...', give a synthetic question under the ability type: '...'", where the first location '...' should be filled with the text description of the image, the second location '...' should be filled with the predicted background text, and the third location '...' should be filled with the selected ability type. Furthermore, in step 504, the image text description, the ability type, and the predicted background text are input into the trained large language model used for question synthesis. Specifically, this includes inserting the image text description, the ability type, and the predicted background text into the predefined question synthesis prompt word template to obtain the question synthesis prompt word, and then inputting the question synthesis prompt word into the trained large language model used for question synthesis.

[0140] The question capability type pool contains all capability types covered by all questions. Capability types are organized into two layers: the first layer is the broad capability categories, which include, but are not limited to, recognition, transformation, analysis, positioning, reasoning, evaluation, and security. The second layer is the sub-capability categories, each of which is further subdivided into one or more sub-categories. For example, the recognition category might include image description, style recognition, animal recognition, food recognition, behavior recognition, person recognition, and license plate recognition.

[0141] After selecting a capability type from the pool of question capability types for the synthesized question, the large language model for question synthesis will use that capability type as the capability type that the question should reflect or focus on. For example, if the capability type selected is the behavior recognition subcategory within the recognition subcategory, for an image of a classroom, the text description of the image includes: scene: classroom, objects: table and students, relationship between objects: student to the left of the table. Based on the image text description, the behavior recognition subcategory within the recognition subcategory, and the background text, the large language model for question synthesis might construct a question such as "What is the student on the left side of the table doing?"

[0142] Among them, the large language model that can be used for question synthesis has been pre-trained, and the training dataset used contains at least: a large number of image text descriptions and corresponding ability types and background text, as well as corresponding annotation questions.

[0143] Step 505: Use the predicted question and the predicted background text as complex prompt words for the piece of training data, and put the complex prompt words into the piece of training data in the first training data set.

[0144] After the background text and question are synthesized, a complex prompt word can be synthesized according to a predefined complex prompt word template. The template includes but is not limited to using two line breaks to concatenate the question and background text.

[0145] Step 506: Return to step 501 until all the training data consisting of question and background text for the complex prompt words defined in the first training data set are synthesized into complex prompt words.

[0146] In the above embodiment, questions are synthesized based on the text description of the image and the background text, so that the questions are consistent with both the content of the image and the background text.

[0147] In an optional embodiment, in order to improve the accuracy of the synthesized background text, in step 503, after the model outputs the predicted background text and before step 504, it further includes: inputting the text description of the image and the predicted background text into a trained large language model that can be used for background text verification, and the model outputs a prompt as to whether the predicted background text and the text description of the image match. If the prompt matches, step 504 is executed; otherwise, return to step 503, that is, reselect the background type and background text example to re-predict a new background text until the predicted background text matches the text description of the image.

[0148] In an optional embodiment, the text description of the image and the predicted background text are input into a trained large language model that can be used for background text verification, specifically including: inserting the text description of the image and the predicted background text into a predefined background text verification prompt word template to obtain background text verification prompt words, and inputting the background text verification prompt words into the trained large language model that can be used for background text verification.

[0149] Among them, the background text verification prompt word template defines the format of the background text verification prompt word, for example: the background text verification prompt word template can be: "Judge whether the following background text: '...' and the following image text description: '...' match", where the first... should be filled in with the predicted background text, and the second... should be filled in with the text description of the image.

[0150] Among them, the large language model that can be used for background text verification has been pre-trained, and the training data set used at least contains: a large number of image text descriptions and corresponding background texts, as well as corresponding matching annotations.

[0151] In an optional embodiment, in order to improve the accuracy of the synthesized question, in step 504, after the model outputs the predicted question and before step 505, it further includes: inputting the text description of the image, the background text predicted in step 503 and the question predicted in step 504 into a trained large language model that can be used for question verification, and the model outputs a prompt as to whether the predicted question matches the text description of the image and the predicted background text. If the prompt matches, step 505 is executed; otherwise, return to step 504, that is, reselect the ability type to re-predict a new question until the predicted question matches the text description of the image and the predicted background text.

[0152] In an optional embodiment, the text description of the image, the background text predicted in step 503, and the question predicted in step 504 are input into a trained large language model that can be used for question verification, specifically including: inserting the text description of the image, the background text predicted in step 503, and the question predicted in step 504 into a predefined question verification prompt word template to obtain question verification prompt words, and inputting the question verification prompt words into the trained large language model that can be used for question verification.

[0153] Among them, the question verification prompt word template defines the format of the question verification prompt word, such as: the question verification prompt word template can be: "Judge whether the following question: '...' matches the following image text description: '...' and the following background text: '...'?", wherein the first place... should be filled in with the question predicted in step 504, the second place... should be filled in with the text description of the image, and the third place... should be filled in with the background text predicted in step 503.

[0154] Among them, the large language model that can be used for question verification has been pre-trained, and the training dataset used contains at least: a large number of image text descriptions and corresponding background text and questions, as well as corresponding matching annotations.

[0155] Figure 6 A flowchart of a method for synthesizing complex prompt words when the complex prompt words include questions and constraint instructions is provided for one embodiment of the present invention. Figure 6 As shown, the specific steps are as follows:

[0156] Step 601: Extract one piece of training data from the first training data set, wherein each piece of training data in the first training data set includes at least one training image; if the complex prompt word defined in the extracted training data is composed of a question and a constraint instruction, execute the following steps:

[0157] Step 601 is performed before step 101 , and the purpose of this embodiment is to synthesize complex prompt words. Therefore, the first training data set at this time only includes training images.

[0158] Step 602: Input the image in the training data into the trained image-text description multimodal model, and the model outputs the text description of the image.

[0159] Step 603: Randomly select a capability type from the question capability type pool; input the text description of the image and the capability type into a trained large language model that can be used for question synthesis, and the model outputs the predicted question; wherein the question capability type pool contains all capability types covered by all questions.

[0160] Among them, the large language model that can be used for question synthesis has been pre-trained, and the training dataset used contains at least: a large number of image text descriptions and corresponding ability types, as well as corresponding annotation questions.

[0161] Step 604: Randomly select one or more constraint instruction examples from the constraint instruction pool, and input each selected constraint instruction example together with the question into a trained large language model that can be used for constraint instruction synthesis. The model sequentially outputs one or more predicted constraint instructions; wherein the constraint instruction pool contains constraint instruction examples under different constraint forms.

[0162] When synthesizing multiple constraint instructions in the same complex prompt word, the selected constraint instruction examples are different from each other to avoid conflicts between the synthesized constraint instructions.

[0163] In actual applications, the number of constraint instructions contained in each complex prompt word is predefined, and the number of constraint instructions is generally 1 to 3.

[0164] The constraint instruction examples in the constraint instruction pool can be manually written or generated by the model. Constraint forms include: format constraints, expression constraints, syntax constraints, and / or conditional constraints. For example, the constraint instruction examples under format constraints can be as follows:

[0165] 1) The answer must begin with "Summary:" and end with "Conclusion."

[0166] 2) The answer cannot contain any spaces;

[0167] 3) Answer using JSON format, including three keys: "title", "content" and "date";

[0168] 4) The answer must be more than 150 Chinese characters and divided into 3 paragraphs.

[0169] Individual constraints generally do not contain specific questions; they are simply requirements for the format and / or expression style of the answer. Constraints cannot exist independently of the question, especially for multimodal tasks. For example, if the question is to recognize text in an image, a constraint that constrains the number of words will conflict with the question, so the question must be involved in the synthesis of the constraint. In addition, if there are multiple constraints, the constraints must be orthogonal to each other, and different constraints must come from different constraint forms to avoid conflicts between them.

[0170] Constraint instructions are the requirements that users expect the visual question answering multimodal large model to answer questions. They most directly reflect the instruction-following ability of the visual question answering multimodal large model and directly affect the user experience.

[0171] Among them, the large language model that can be used for constraint instruction synthesis has been pre-trained, and the training data set used at least contains: a large number of constraint instructions and corresponding questions, as well as corresponding labeled constraint instructions.

[0172] Step 605: Use the predicted question and the predicted one or more constraint instructions as complex prompt words for the piece of training data, and put the complex prompt words into the piece of training data in the first training data set.

[0173] After the question and all the constraint instructions are synthesized, the synthesized question and all the constraint instructions can be combined into a complex prompt word according to a preset complex prompt word template. The complex prompt word template includes but is not limited to: using two line breaks to splice the question and each constraint instruction.

[0174] Step 606: Return to step 601 until all the training data consisting of questions and constraint instructions for the complex prompt words defined in the first training data set are synthesized into complex prompt words.

[0175] In the above embodiment, a question is synthesized based on the text description of the image so that the question is consistent with the content of the image, and then a constraint instruction is synthesized based on the question, so that the constraint instruction does not conflict with the question, avoiding the situation where the final synthesized complex prompt word cannot answer the question, thereby improving the practicality of the synthesized complex prompt word.

[0176] In an optional embodiment, in order to improve the accuracy of the synthesized questions, in step 603, after the model outputs the predicted question and before step 604, the method further includes: inputting the text description of the image and the predicted question into a trained large language model that can be used for question verification, and the model outputs a prompt as to whether the predicted question matches the text description of the image. If the prompt matches, step 604 is executed; otherwise, the method returns to step 603, i.e., reselecting the capability type to re-predict a new question until the predicted question matches the text description of the image.

[0177] In an optional embodiment, the text description of the image and the predicted question are input into a trained large language model that can be used for question verification, specifically including: inserting the text description of the image and the predicted question into a predefined question verification prompt word template to obtain question verification prompt words, and inputting the question verification prompt words into the trained large language model that can be used for question verification.

[0178] Among them, the question verification prompt word template defines the format of the question verification prompt word, for example: the question verification prompt word template can be: "Judge whether the following question: '...' matches the text description of the following image: '...'?", where the first part... should be filled in with the predicted question, and the second part... should be filled in with the text description of the image.

[0179] Among them, the large language model that can be used for question verification has been pre-trained, and the training data set used contains at least: a large number of image text descriptions and corresponding questions, as well as corresponding matching annotations.

[0180] In an optional embodiment, in order to improve the accuracy of the synthesized constraint instructions, in step 604, after the model outputs a predicted constraint instruction, it further includes: inputting the predicted constraint instruction and the question predicted in step 603 into a trained large language model that can be used for constraint instruction verification, and the model outputs a prompt as to whether the predicted constraint instruction and the predicted question match; if the prompt matches, the subsequent process continues (i.e., continuing to predict the next constraint instruction for the current complex prompt word, or executing step 605 if all constraint instructions have been predicted for the current complex prompt word); otherwise, returning to step 604, i.e., reselecting an instruction constraint example to re-predict a new constraint instruction, until the predicted constraint instruction and the predicted question match.

[0181] In an optional embodiment, the predicted constraint instruction and the question predicted in step 603 are input into a trained large language model that can be used for constraint instruction verification, specifically including: inserting the predicted constraint instruction and the question predicted in step 603 into a pre-defined constraint instruction verification prompt word template to obtain a constraint instruction verification prompt word, and inputting the constraint instruction verification prompt word into the trained large language model that can be used for constraint instruction verification.

[0182] Among them, the constraint instruction verification prompt word template defines the format of the constraint instruction verification prompt word, such as: the constraint instruction verification prompt word template can be: "Judge whether the following constraint instruction: '...' and the following question: '...' match", where the first... should be filled in with the constraint instruction predicted in step 604, and the second... should be filled in with the question predicted in step 603.

[0183] Among them, the large language model that can be used for constraint instruction verification has been pre-trained, and the training data set used at least contains: a large number of constraint instructions and corresponding questions, as well as corresponding matching annotations.

[0184] Figure 7 A flowchart of a method for synthesizing complex prompt words when the complex prompt words include questions, background text, and constraint instructions is provided for one embodiment of the present invention. Figure 7 As shown, the specific steps are as follows:

[0185] Step 701: Extract one piece of training data from the first training data set, wherein each piece of training data in the first training data set includes at least one training image; if the extracted piece of training data defines a complex prompt word consisting of a question, background text, and constraint instructions, execute the following steps:

[0186] Step 701 is performed before step 101 , and the purpose of this embodiment is to synthesize complex prompt words. Therefore, the first training data set at this time only includes training images.

[0187] Step 702: Input the image in the training data into the trained image-text description multimodal model, and the model outputs the text description of the image.

[0188] Step 703: Randomly select a background text example under a background type from the background text pool; input the text description of the image, the background type and the background text example into a trained large language model that can be used for background text synthesis, and the model outputs the predicted background text.

[0189] Step 704: Randomly select a capability type from the question capability type pool; input the text description of the image, the capability type, and the predicted background text into a trained large language model that can be used for question synthesis, and the model outputs the predicted question; wherein the question capability type pool contains all capability types covered by all questions.

[0190] Step 705: Randomly select one or more constraint instruction examples from the constraint instruction pool, and input each selected constraint instruction example together with the predicted question into a trained large language model that can be used for constraint instruction synthesis. The model sequentially outputs one or more predicted constraint instructions; wherein the constraint instruction pool contains constraint instruction examples under different constraint forms.

[0191] Step 706: Use the predicted question, the predicted background text, and the predicted one or more constraint instructions as complex prompt words for the piece of training data, and put the complex prompt words into the piece of training data in the first training data set.

[0192] After the background text, question, and all constraint instructions are synthesized, the synthesized background text, question, and all constraint instructions can be combined into a complex prompt word according to a preset complex prompt word template. The complex prompt word template includes but is not limited to: using two line breaks to splice the question, background text, and each constraint instruction.

[0193] Step 707: Return to step 701 until all the training data consisting of questions, background texts and constraint instructions for the complex prompt words defined in the first training data set are synthesized into complex prompt words.

[0194] In an optional embodiment, in order to improve the accuracy of the synthesized background text, in step 703, after the model outputs the predicted background text and before step 704, it further includes: inputting the text description of the image and the predicted background text into a trained large language model that can be used for background text verification, and the model outputs a prompt as to whether the predicted background text and the text description of the image match. If the prompt matches, step 704 is executed; otherwise, return to step 703, that is, reselect the background type and background text example to re-predict a new background text until the predicted background text matches the text description of the image.

[0195] In an optional embodiment, the text description of the image and the predicted background text are input into a trained large language model that can be used for background text verification, specifically including: inserting the text description of the image and the predicted background text into a predefined background text verification prompt word template to obtain background text verification prompt words, and inputting the background text verification prompt words into the trained large language model that can be used for background text verification.

[0196] Among them, the background text verification prompt word template defines the format of the background text verification prompt word, for example: the background text verification prompt word template can be: "Judge whether the following background text: '...' and the following image text description: '...' match", where the first... should be filled in with the predicted background text, and the second... should be filled in with the text description of the image.

[0197] In an optional embodiment, in order to improve the accuracy of the synthesized question, in step 704, after the model outputs the predicted question and before step 705, it further includes: inputting the text description of the image, the background text predicted in step 703 and the question predicted in step 704 into a trained large language model that can be used for question verification, and the model outputs a prompt as to whether the predicted question matches the text description of the image and the predicted background text. If the prompt matches, step 705 is executed; otherwise, return to step 704, that is, reselect the ability type to re-predict a new question until the predicted question matches the text description of the image and the predicted background text.

[0198] In an optional embodiment, the text description of the image, the background text predicted in step 703, and the question predicted in step 704 are input into a trained large language model that can be used for question verification, specifically including: inserting the text description of the image, the background text predicted in step 703, and the question predicted in step 704 into a predefined question verification prompt word template to obtain question verification prompt words, and inputting the question verification prompt words into the trained large language model that can be used for question verification.

[0199] Among them, the question verification prompt word template defines the format of the question verification prompt word, such as: the question verification prompt word template can be: "Judge whether the following question: '...' matches the following image text description: '...' and the following background text: '...'?", wherein the first place... should be filled in with the question predicted in step 704, the second place... should be filled in with the text description of the image, and the third place... should be filled in with the background text predicted in step 703.

[0200] In an optional embodiment, in order to improve the accuracy of the synthesized constraint instructions, in step 705, after the model outputs a predicted constraint instruction, it further includes: inputting the predicted constraint instruction and the question predicted in step 704 into a trained large language model that can be used for constraint instruction verification, and the model outputs a prompt as to whether the predicted constraint instruction and the predicted question match; if the prompt matches, the subsequent process continues (i.e., continuing to predict the next constraint instruction for the current complex prompt word, or executing step 706 if all constraint instructions have been predicted for the current complex prompt word); otherwise, returning to step 705, i.e., reselecting an instruction constraint example to re-predict a new constraint instruction, until the predicted constraint instruction and the predicted question match.

[0201] In an optional embodiment, the predicted constraint instruction and the question predicted in step 704 are input into a trained large language model that can be used for constraint instruction verification, specifically including: inserting the predicted constraint instruction and the question predicted in step 704 into a pre-defined constraint instruction verification prompt word template to obtain a constraint instruction verification prompt word, and inputting the constraint instruction verification prompt word into the trained large language model that can be used for constraint instruction verification.

[0202] Among them, the constraint instruction verification prompt word template defines the format of the constraint instruction verification prompt word, such as: the constraint instruction verification prompt word template can be: "Judge whether the following constraint instruction: '...' and the following question: '...' match", where the first... should be filled in with the constraint instruction predicted in step 705, and the second... should be filled in with the question predicted in step 704.

[0203] It should be noted that the major language models involved in the various embodiments of the present invention include: a large language model that can be used for background text synthesis, a large language model that can be used for question synthesis, a large language model that can be used for constraint instruction synthesis, a large language model that can be used for background text verification, a large language model that can be used for question verification, and a large language model that can be used for constraint instruction verification. These six large language models can be trained independently and the training data required by each can be used during training; these six large language models can also use the same large language model, that is, the large language model has six functions at the same time: background text synthesis, question synthesis, constraint instruction synthesis, background text verification, question verification, and constraint instruction verification, and the training data required by these six functions can be used during training.

[0204] It should be noted that the reason why the embodiment of the present invention uses a large language model to generate the problem using the text description of the image as input instead of using a large multimodal model to generate the problem directly using the image as input is as follows:

[0205] On the one hand, the instruction-following ability of a large multimodal model with the same parameter scale is weaker than that of a large language model. In the tasks of background text synthesis, question synthesis, and constraint instruction synthesis, the large multimodal model cannot accurately understand the intention of the task, resulting in poor quality of the generated background text, questions, and constraint instructions.

[0206] On the other hand, using images as input causes the synthesized background text, questions, and constraints to focus more on the subject in the image and less on details or small objects in the image, reducing their diversity and complexity. For example, an image may contain a person as the subject, but the distant view also includes a puppy running towards the person. If the question is synthesized using the image as input, the synthesized question may only focus on the person and not the puppy. However, if the question is synthesized using the text description of the image as input, the objects in the text description will include both the person and the puppy, as well as the relationship between the person and the puppy: if the puppy is behind the person, the synthesized question may include information about the puppy.

[0207] It should be noted that the training data in the first training data set and the second training data set in the embodiment of the present invention need to be sufficiently large to improve the applicability and accuracy of the final multimodal large model of visual answers.

[0208] Figure 8 This is a schematic diagram of the structure of a device for building a large multimodal model for visual question answering provided by one embodiment of the present invention. Figure 8 As shown, it mainly includes: a first training data set acquisition module 81 and a training module 82, wherein:

[0209] The first training data set acquisition module 81 is used to: acquire a first training data set, where each training data includes: at least one training image, a complex prompt word, and a best answer; wherein the complex prompt word includes a question and at least one of background text and constraint instructions; the background text is: background information that needs to be combined or referenced with respect to the training image when answering the question; the constraint instruction is: the description requirements of the answer by the questioner.

[0210] The training module 82 is used to: extract a piece of training data from the first training data set, input the image and complex prompt words in the training data into the visual question answering multimodal large model to be trained, and the visual question answering multimodal large model outputs a predicted answer; calculate the loss value based on the predicted answer and the best answer in the training data; use the loss value to adjust the parameters of the visual question answering multimodal large model until the training end condition is met. The visual question answering multimodal large model at this time is the visual question answering multimodal large model finally used.

[0211] In an optional embodiment, each piece of training data in the first training data set acquired by the first training data set acquisition module 81 further includes: a non-optimal answer; and the training module 82 calculates a loss value based on the predicted answer and the best answer in the training data, including: using a direct preference optimization algorithm to compare the predicted answer with the best answer and the non-optimal answer, respectively, to calculate the loss value.

[0212] In an optional embodiment, the first training data set acquisition module 81 acquires a first training data set, each piece of training data including at least one training image, a complex prompt word, and a best answer, and is further configured to:

[0213] B1. Obtain a first training dataset, where each piece of training data includes at least one training image and a complex prompt word; and obtain an answer pool, where the answer pool includes multiple answers corresponding to each complex prompt word in the first training dataset;

[0214] B2. Extract a piece of training data from the first training data set in step B1, and extract an answer corresponding to the complex prompt word in the training data from the answer pool in step B1. When an answer is extracted, the image, complex prompt word, and answer in the training data are sequentially input into the trained reward model, and the reward model outputs a reward value. When the reward value of each answer to the complex prompt word in the training data is obtained, the answer with the highest reward value is selected as the best answer, and a non-best answer is selected. The best answer and the non-best answer are added to the training data in the first training data set.

[0215] Return to step B2 until the best answer and non-best answer are obtained for each training data in the first training data set.

[0216] In an optional embodiment, the apparatus further includes a reward model training module, configured to:

[0217] C1. Extract a piece of training data from the first training data set, input the complex prompt word in the training data into multiple pre-selected answer multimodal large models, or input the same pre-selected answer multimodal large model trained using kernel sampling with different temperature coefficients, to obtain multiple answers; provide the obtained multiple answers to an operator, who annotates each answer with a score; and place the complex prompt word, the multiple answers, and their annotated scores into an answer pool;

[0218] Return to step C1, and execute step C2 until step C1 is completed for all complex prompt words in the first training data set;

[0219] C2. Extract one piece of training data from the first training data set, and extract multiple answers and their annotation scores corresponding to the complex prompt words in the training data from the answer pool;

[0220] C3. Extract one answer from the multiple answers to the complex prompt word in sequence, input the image, complex prompt word, and extracted answer in the training data into the reward model to be trained in sequence, and the reward model outputs a prediction score. The predicted score and the labeled score of the answer are calculated using a preset loss function to obtain a loss value, and the parameters of the reward model are adjusted according to the loss value;

[0221] Return to step C3 until step C3 is executed for each answer to the complex prompt word, then return to step C2 until the training end condition is met, at which time a trained reward model is obtained.

[0222] In an optional embodiment, the above apparatus further includes a pre-training module for:

[0223] D1. Obtain a second training data set, where each piece of training data includes: at least one training image, a question, and a target answer;

[0224] D2. Extract a piece of training data from the second training dataset, input the image and question in the training data into the multimodal visual question answering model to be trained, and the multimodal visual question answering model outputs a predicted answer; calculate a loss value based on the predicted answer and the target answer in the training data; and use the loss value to adjust the parameters of the multimodal visual question answering model;

[0225] Return to step D2 until the training end condition is met. At this time, the visual question answering multimodal large model is the pre-trained visual question answering multimodal large model.

[0226] Furthermore, the training module 82 inputs the image and complex prompt words in the training data into the visual question answering multimodal model to be trained as follows: inputs the image and complex prompt words in the training data into the pre-trained visual question answering multimodal model.

[0227] In an optional embodiment, the above device further includes a first complex prompt word synthesis module, which is used to:

[0228] A piece of training data is extracted from the first training data set in sequence, wherein each piece of training data in the first training data set includes at least one training image; if the complex prompt word defined in the extracted training data consists of a question and background text, the image in the training data is input into a trained image-text description multimodal large model, and the model outputs a text description of the image; a background text example under a background type is randomly selected from a background text pool; the text description of the image, the background type and the background text example are input into a trained large language model that can be used for background text synthesis, and the model outputs a predicted background text; an ability type is randomly selected from a question ability type pool; the text description of the image, the ability type and the predicted background text are input into a trained large language model that can be used for question synthesis, and the model outputs a predicted question; wherein the question ability type pool contains all ability types covered by all questions; the predicted question and the predicted background text are used as the complex prompt word for the training data, and the complex prompt word is placed into the training data in the first training data set.

[0229] In an optional embodiment, the above device further includes a second complex prompt word synthesis module, which is used to:

[0230] A training data piece is sequentially extracted from a first training data set, wherein each training data piece in the first training data set includes at least one training image; if a complex prompt word defined in the extracted training data piece consists of a question and a constraint instruction, the image in the training data piece is input into a trained image-text description multimodal large model, and the model outputs a text description of the image; a capability type is randomly selected from a question capability type pool; the text description of the image and the capability type are input into a trained large language model that can be used for question synthesis, and the model outputs a predicted question; wherein the question capability type pool includes all capability types covered by all questions; one or more constraint instruction examples are randomly selected from a constraint instruction pool, and each selected constraint instruction example is input into the trained large language model that can be used for constraint instruction synthesis together with the question, and the model sequentially outputs one or more predicted constraint instructions; wherein the constraint instruction pool includes constraint instruction examples under different constraint forms; the predicted question and the predicted one or more constraint instructions are used as the complex prompt word for the training data piece, and the complex prompt word is placed into the training data piece in the first training data set.

[0231] In an optional embodiment, the above device further includes a third complex prompt word synthesis module, which is used to:

[0232] Extract one piece of training data from the first training data set in sequence, wherein each piece of training data in the first training data set includes at least one training image; if the complex prompt word defined in the extracted training data is composed of a question, background text and a constraint instruction, then input the image in the training data into the trained image-text description multimodal large model, and the model outputs the text description of the image; randomly select a background text example under a background type from the background text pool; input the text description of the image, the background type and the background text example into the trained large language model that can be used for background text synthesis, and the model outputs the predicted background text; randomly select an ability type from the question ability type pool; input the text description of the image, the ability type and the background text example into the trained large language model that can be used for background text synthesis, and the model outputs the predicted background text; A trained large language model for question synthesis is input together with the predicted background text, and the model outputs the predicted question; wherein the question ability type pool contains all ability types covered by all questions; one or more constraint instruction examples are randomly selected from the constraint instruction pool, and each selected constraint instruction example is input together with the predicted question into the trained large language model for constraint instruction synthesis, and the model sequentially outputs the predicted one or more constraint instructions; wherein the constraint instruction pool contains constraint instruction examples under different constraint forms; the predicted question, the predicted background text and the predicted one or more constraint instructions are used as complex prompt words for the training data, and the complex prompt words are placed in the training data of the first training data set.

[0233] In an optional embodiment, after the model outputs the predicted background text and before randomly selecting an ability type from the question ability type pool, the third complex prompt word synthesis module further includes: inputting the text description of the image and the predicted background text into a trained large language model that can be used for background text verification, and the model outputs a prompt as to whether the predicted background text and the text description of the image match; if the prompt matches, executing the action of randomly selecting an ability type from the question ability type pool; otherwise, returning to the action of randomly selecting a background text example under a background type from the background text pool.

[0234] In an optional embodiment, after the model outputs the predicted question and before randomly selecting one or more constraint instruction examples from the constraint instruction pool, the third complex prompt word synthesis module further includes: inputting the text description of the image, the predicted background text and the predicted question into a trained large language model that can be used for question verification, and the model outputs a prompt as to whether the predicted question matches the text description of the image and the predicted background text; if the prompt matches, executing the action of randomly selecting one or more constraint instruction examples from the constraint instruction pool; otherwise, returning to the action of randomly selecting a capability type from the question capability type pool.

[0235] In an optional embodiment, after the model outputs a predicted constraint instruction, the third complex prompt word synthesis module further includes: inputting the predicted constraint instruction and the predicted question into a trained large language model that can be used to verify the constraint instruction, and the model outputs a prompt as to whether the predicted constraint instruction and the predicted question match. If the prompt matches, the subsequent process continues; otherwise, the process returns to the action of randomly selecting a constraint instruction example from the constraint instruction pool.

[0236] In an optional embodiment, the above-mentioned device further includes an answer module, which is used to: input the image and complex prompt words of any scene into the visual question answering multimodal large model to be finally used, and the model outputs an answer.

[0237] An application example of the present invention is given below:

[0238] Figure 9 This is an image of a kitchen. The compound prompt words given by the questioner are as follows:

[0239] The background text reads: "Commercial Kitchen Hygiene Standards": Countertops must be kept clean and free of food debris; scissors must be washed and hung back on the rack after use; they must not be left alone on the countertop; greasy stains on the walls must be cleaned promptly, and yellow or brown greasy spots must not appear; ... (The standard contains a lot of content, so the rest is omitted);

[0240] The question is: Based on the above specifications, please determine whether there are any violations in the diagram;

[0241] The constraint instruction is: only explain the violations. If there are no violations, directly answer "there are no violations."

[0242] After the image and the complex prompt words consisting of background text, questions, and constraint instructions are input into the large multimodal visual question answering model obtained using the embodiment of the present invention, the model outputs the following answer:

[0243] There are violations. The picture clearly shows a pair of red scissors placed alone on the countertop, violating Article 2 of the "Commercial Kitchen Hygiene Code": "Scissors must be washed and hung back on the shelf after use and cannot be left alone on the countertop."

[0244] It can be seen that the answer is consistent with the image content, the background text and the constraint instructions, and fully responds to the questioner's intention to ask the question.

[0245] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores instructions that, when executed by a processor, enable the processor to execute the method for establishing a large multimodal model for visual question answering as described in any of the aforementioned embodiments.

[0246] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this application may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application may be combined and / or coupled in various ways, and all such combinations and / or couplings fall within the scope disclosed in this application.

[0247] The principles and implementation methods of the present invention are described herein using specific embodiments. The description of the above embodiments is only used to help understand the core ideas of the present invention and is not intended to limit this application. For those skilled in the art, changes can be made in the specific implementation methods and application scope based on the ideas, spirit and principles of the present invention. Any modifications, equivalent replacements, improvements, etc. made therein should be included within the scope of protection of this application.

Claims

1. A method for establishing a large multimodal model for visual question answering, characterized in that: The method includes: A1. Obtain a first training data set, where each piece of training data includes: at least one training image, a complex prompt word, and a best answer; wherein the complex prompt word includes a question and a constraint instruction; the constraint instruction is: the questioner's description requirement for the answer; A2. Extract a piece of training data from the first training dataset, input the image and complex prompt word in the training data into the multimodal visual question answering model to be trained, and have the multimodal visual question answering model output a predicted answer; calculate a loss value based on the predicted answer and the best answer in the training data; and use the loss value to adjust the parameters of the multimodal visual question answering model; Return to step A2 until the training end condition is met. The visual question answering multimodal large model at this time is the final visual question answering multimodal large model; The complex prompt word is obtained in the following way: Extracting one piece of training data from the first training data set in sequence, wherein each piece of training data in the first training data set includes at least one training image; Input the image in the training data into the trained image-text description multimodal model, and the model outputs the text description of the image; Randomly select a capability type from the question capability type pool; input the text description of the training image and the capability type into a trained large language model for question synthesis, and the model outputs the predicted question; the question capability type pool contains all capability types covered by all questions; Randomly select one or more constraint instruction examples from the constraint instruction pool, input each selected constraint instruction example together with the predicted question into a trained large language model that can be used for constraint instruction synthesis, and the model sequentially outputs one or more predicted constraint instructions; the constraint instruction pool contains constraint instruction examples under different constraint forms; The predicted question and the predicted one or more constraint instructions are used as complex prompt words for the piece of training data, and the complex prompt words are put into the piece of training data in the first training data set.

2. The method according to claim 1, characterized in that Each piece of training data in the first training data set described in step A1 further includes: a non-optimal answer; Furthermore, the calculation of the loss value according to the predicted answer and the best answer in the training data in step A2 includes: A direct preference optimization algorithm is used to compare the predicted answer with the best answer and the non-best answer, and the loss value is calculated.

3. The method according to claim 2, characterized in that The non-optimal answer is: an intermediate answer or a worst answer.

4. The method according to claim 2 or 3, characterized in that Before step A1, the method further comprises: B1. Obtain a first training dataset, where each piece of training data includes at least one training image and a complex prompt word; and obtain an answer pool, where the answer pool includes multiple answers corresponding to each complex prompt word in the first training dataset; B2. Extract a piece of training data from the first training data set in sequence, and simultaneously extract an answer corresponding to the complex prompt word in the training data from the answer pool in sequence; when an answer is extracted, the image, complex prompt word, and extracted answer in the training data are sequentially input into the trained reward model, and the reward model outputs a reward value; when the reward value for each answer to the complex prompt word in the training data is obtained, the answer with the highest reward value is selected as the best answer, and a non-best answer is selected, and the best answer and the non-best answer are added to the training data in the first training data set; Return to step B2 until the best answer and non-best answer are obtained for each training data in the first training data set.

5. The method according to claim 4, characterized in that The reward model is trained as follows: C1. Extracting a piece of training data from the first training data set, inputting the complex prompt words in the training data into multiple pre-selected answer multimodal large models, or sequentially inputting the same pre-selected answer multimodal large model trained using kernel sampling with different temperature coefficients, to obtain multiple answers; providing the obtained multiple answers to an operator, who then annotates a score for each answer; Putting the complex prompt word, the multiple answers and their annotated scores into an answer pool; Return to step C1, and execute step C2 until step C1 is completed for all complex prompt words in the first training data set; C2. Extract one piece of training data from the first training data set, and extract multiple answers and their annotation scores corresponding to the complex prompt words in the training data from the answer pool; C3. Extract one answer from the multiple answers to the complex prompt word in sequence, input the image, complex prompt word, and extracted answer in the training data into the reward model to be trained in sequence, and the reward model outputs a prediction score. The predicted score and the labeled score of the answer are calculated using a preset loss function to obtain a loss value, and the parameters of the reward model are adjusted according to the loss value; Return to step C3 until step C3 is executed for each answer to the complex prompt word, then return to step C2 until the training end condition is met, at which time a trained reward model is obtained.

6. The method according to any one of claims 1 to 3, characterized in that: Before step A1, the method further comprises: D1. Obtain a second training data set, where each piece of training data includes: at least one training image, a question, and a target answer; D2. Extract a piece of training data from the second training dataset, input the image and question in the training data into the multimodal visual question answering model to be trained, and the multimodal visual question answering model outputs a predicted answer; calculate a loss value based on the predicted answer and the target answer in the training data; and use the loss value to adjust the parameters of the multimodal visual question answering model; Return to step D2 until the training end condition is met. At this time, the visual question answering multimodal large model is the pre-trained visual question answering multimodal large model. Furthermore, step A2 of inputting the images and complex prompt words in the training data into the visual question answering multimodal large model to be trained is: inputting the images and complex prompt words in the training data into the pre-trained visual question answering multimodal large model.

7. The method according to claim 1, characterized in that The complex prompt words further include background text, which is background information that needs to be combined or referenced with respect to the training image when answering questions; After the model outputs a text description of the image and before randomly selecting a capability type from the problem capability type pool, the method further includes: Randomly select a background text example under a background type from the background text pool; input the text description of the image, the background type, and the background text example into a trained large language model that can be used for background text synthesis, and the model outputs the predicted background text; Furthermore, inputting the text description of the image and the capability type into a trained large language model that can be used for question synthesis includes: The text description of the image, the capability type, and the predicted background text are fed into a large language model trained for question synthesis. Furthermore, the predicted question and the predicted one or more constraint instructions are used as complex prompt words for the training data, including: The predicted question, the predicted background text, and the predicted one or more constraint instructions are used as complex prompt words for the training data.

8. The method according to claim 7, characterized in that After the model outputs the predicted background text and before randomly selecting a capability type from the question capability type pool, the method further includes: The text description of the image and the predicted background text are input into a trained large language model that can be used for background text verification. The model outputs a prompt as to whether the predicted background text and the text description of the image match. If the prompt matches, the action of randomly selecting an ability type from the question ability type pool is executed; otherwise, the action of randomly selecting a background text example under a background type from the background text pool is returned.

9. The method according to claim 7, characterized in that After the model outputs the predicted problem and before randomly selecting one or more constraint instruction examples from the constraint instruction pool, the method further includes: The text description of the image, the predicted background text and the predicted question are input into a trained large language model that can be used for question verification. The model outputs a prompt as to whether the predicted question matches the text description of the image and the predicted background text. If the prompt matches, the action of randomly selecting one or more constraint instruction examples from the constraint instruction pool is executed; otherwise, the action of randomly selecting a capability type from the question capability type pool is returned.

10. The method according to claim 7, characterized in that When the model outputs a predicted constraint instruction, it further includes: The predicted constraint instruction and the predicted question are input into a large language model that has been trained and can be used for constraint instruction verification. The model outputs a prompt as to whether the predicted constraint instruction and the predicted question match. If the prompt matches, the subsequent process continues; otherwise, the process returns to the action of randomly selecting a constraint instruction example from the constraint instruction pool.

11. The method according to claim 1, wherein The visual question answering multimodal large model at this time is the visual question answering multimodal large model finally used, and further includes: An image of any scene and complex prompt words are input into the visual question answering multimodal large model, and the model outputs an answer.

12. A device for building a large multimodal model for visual question answering, characterized in that: The device includes: A first training data set acquisition module is configured to acquire a first training data set, wherein each piece of training data includes at least one training image, a complex prompt word, and a best answer; wherein the complex prompt word includes a question and a constraint instruction; and the constraint instruction is a description requirement of the answer by the questioner; The training module is configured to: extract a piece of training data from a first training data set, input the image and complex prompt words in the training data into the visual question answering multimodal large model to be trained, and have the visual question answering multimodal large model output a predicted answer; calculate a loss value based on the predicted answer and the best answer in the training data; and use the loss value to adjust the parameters of the visual question answering multimodal large model until the training end condition is met. The visual question answering multimodal large model at this time is the visual question answering multimodal large model finally used; The second complex prompt word synthesis module is used to: extract one piece of training data from the first training data set in sequence, wherein each piece of training data in the first training data set includes at least one training image; input the image in the training data into a trained image-text description multimodal large model, and the model outputs a text description of the image; randomly select a capability type from a question capability type pool; input the text description of the training image and the capability type into a trained large language model that can be used for question synthesis, and the model outputs a predicted question; wherein the question capability type pool contains all capability types covered by all questions; randomly select one or more constraint instruction examples from a constraint instruction pool, input each selected constraint instruction example together with the predicted question into the trained large language model that can be used for constraint instruction synthesis, and the model sequentially outputs the predicted one or more constraint instructions; wherein the constraint instruction pool contains constraint instruction examples under different constraint forms; use the predicted question and the predicted one or more constraint instructions as the complex prompt word for the training data, and place the complex prompt word into the training data in the first training data set.

13. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores instructions, which, when executed by a processor, cause the processor to execute the method for establishing a visual question answering multimodal large model as described in any one of claims 1 to 11.

Images