Training sample construction method and device, electronic equipment and readable storage medium

Abstract

The application provides a training sample construction method and device, electronic equipment and a readable storage medium, and applies to the technical field of computers. The method comprises the following steps: obtaining a plurality of key-value pairs of a sample image, generating a plurality of questions about the sample image according to the plurality of key-value pairs, determining an answer corresponding to each question based on the sample image and the question, and constructing a training sample according to the sample image, the plurality of questions and the answer corresponding to each question. The method provided by the application generates related questions of the sample image by using a plurality of key information included in the sample image and attribute values of each key information. There are usually a large number of key information in the sample image, and different questions can be obtained for each key information when the questions are generated based on the plurality of key information. Therefore, a large number of questions can be obtained, thereby improving the question diversity of the training sample.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a training sample construction method, apparatus, electronic device, and readable storage medium in the field of computer technology. Background Technology

[0002] Visual Question Answering (VQA) is a cross-modal artificial intelligence task that aims to answer natural language questions related to images by understanding their content. This includes visual recognition, natural language processing, and reasoning. Current methods primarily rely on supervised machine learning to train VQA models, using a large number of images and examples of questions and answers around those images as training samples. The VQA model learns how to answer questions based on the image content.

[0003] In related technologies, large language models (LLMs) are mainly used to generate questions to construct training samples. However, LLMs lack sufficient visual data and can only rely on a limited number of patterns when generating questions, which lacks diversity and may result in insufficient question diversity in the final training samples. Summary of the Invention

[0004] This application provides a training sample construction method, apparatus, electronic device, and readable storage medium, which can improve the question diversity of training samples in visual question answering scenarios.

[0005] Firstly, a method for constructing training samples is provided, the method comprising:

[0006] Obtain multiple key-value pairs from a sample image, each key-value pair including a key piece of information from the sample image and an attribute value of the key piece of information;

[0007] Based on the multiple key-value pairs, generate multiple questions about the sample image;

[0008] For each of the questions, the answer to the question is determined based on the sample image and the question.

[0009] A first training sample is constructed based on the sample image, the plurality of questions, and the answer corresponding to each question.

[0010] In this embodiment, when constructing training samples, multiple key-value pairs of the sample image are first obtained. Then, multiple questions about the sample image are generated around these key-value pairs, and the answer to each question is determined. Finally, training samples are constructed based on the sample image, the multiple questions, and the answer to each question. This method utilizes multiple key information included in the sample image and the attribute values ​​of each key information to generate relevant questions about the sample image. Since there is usually a lot of key information in the sample image, generating questions based on multiple key information can yield different questions for each key information, thus resulting in a greater variety of questions and improving the question diversity of the training samples.

[0011] Furthermore, when training a visual question answering model based on multiple training samples, the types of questions that the visual question answering model can handle can be expanded, thereby improving the performance of the visual question answering model.

[0012] Optionally, generating multiple questions about the sample image based on the multiple key-value pairs includes: determining multiple question attributes; and for each key-value pair, generating one or more questions corresponding to each question attribute to obtain the multiple questions.

[0013] In this application example, during the process of generating questions based on key-value pairs in a sample image, multiple question attributes are first obtained. Then, for each key-value pair in the sample image, one or more questions are generated for each question attribute. By using multiple pre-determined question attributes, the generated questions can cover multiple question attributes, reducing the probability of multiple generated questions being identical, thereby improving the diversity of the final questions.

[0014] Optionally, determining multiple problem attributes includes: controlling a first model to generate multiple problem categories; controlling the first model to generate the problem attributes included in each of the problem categories, so as to obtain the multiple problem attributes.

[0015] In this embodiment, during the process of determining problem attributes, multiple problem categories are first determined using a first model. Then, the first model is controlled to further subdivide each problem category to determine the problem attributes included in each category. The problem attributes included in multiple problem categories are combined to obtain multiple problem attributes. By running the model twice, problem attributes covering a wider range of problems can be obtained. Furthermore, with more problem attributes, a wider variety of problem types can be identified, thereby improving the diversity of the final problem obtained.

[0016] Optionally, determining the answer to the question based on the sample image and the question includes: performing secondary training on a pre-trained second model using at least one second training sample, wherein the second training sample is an unqualified training sample; and controlling the trained second model to generate the answer to the question based on the sample image and the question.

[0017] In this embodiment, the second model is trained using unqualified second training samples, and then the trained second model generates corresponding answers based on the sample images and questions. This can improve the accuracy of the answers output by the second model, thereby improving the accuracy of the answers included in the first training samples.

[0018] Optionally, the control of the trained second model to generate the answer to the question based on the sample image and the question includes: controlling the trained second model to generate the answer to the question based on the sample image, the question, and the plurality of key-value pairs.

[0019] In this embodiment of the application, during the process of generating an answer based on the second model, a sample image and a question are provided to the second model, along with multiple key-value pairs from the sample image. These multiple key-value pairs can provide the second model with key information from the sample image, enabling the second model to output a more accurate answer, thereby improving the accuracy of the final answer.

[0020] Optionally, before performing secondary training on the pre-trained second model using at least one second training sample, the method further includes: acquiring a plurality of third training samples, wherein the third training samples are training samples constructed based on the questions generated by the second model; and determining the at least one second training sample from the plurality of third training samples.

[0021] In this embodiment of the application, the second model is retrained using multiple unqualified third training samples. This allows the second model to summarize the errors and experiences gained during the construction of the training samples, thereby improving the accuracy of the questions when generating questions based on the trained second model.

[0022] Optionally, obtaining multiple key-value pairs of a sample image includes: obtaining an image description of the sample image; and controlling a third model to generate the multiple key-value pairs based on the image description.

[0023] In this application example, the image description of the sample image is first obtained, and then the key information and attribute values ​​in the sample image are extracted from the image description, which can reduce the difficulty of extracting key-value pairs.

[0024] Secondly, a training sample construction apparatus is provided, the apparatus comprising:

[0025] The acquisition module is used to acquire multiple key-value pairs of a sample image, each key-value pair including a key piece of information in the sample image and an attribute value of the key piece of information;

[0026] A first model is used to generate multiple questions about the sample image based on the multiple key-value pairs;

[0027] The second model is used to determine the answer to each question based on the sample image and the question.

[0028] A construction module is used to construct a first training sample based on the sample image, the plurality of questions, and the answer corresponding to each question.

[0029] Thirdly, an electronic device is provided, the electronic device comprising:

[0030] Memory, used to store program code;

[0031] A processor is configured to call and run the program code from the memory, causing the electronic device to perform the method in any possible implementation of the first aspect described above.

[0032] Fourthly, a program product is provided, comprising: program code that, when run on an electronic device, causes the electronic device to execute the method in any possible implementation of the first aspect.

[0033] Fifthly, a readable storage medium is provided that stores program code, which, when run on an electronic device, causes the electronic device to perform the method in any possible implementation of the first aspect described above.

[0034] The technical solutions provided by this invention can be applied to the transaction and delivery services of instant e-commerce platforms, such as Taobao Flash Sale, Taoxianda, Ele.me takeaway and retail, but are not limited thereto. Attached Figure Description

[0035] Figure 1 This is an exemplary application scenario of the training sample construction method provided in this application;

[0036] Figure 2 This is a flowchart illustrating the steps of a training sample construction method provided in an embodiment of this application;

[0037] Figure 3 This illustration shows a problem classification diagram provided by an embodiment of this application;

[0038] Figure 4This is a flowchart illustrating a training sample construction method provided in an embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the structure of a training sample construction device provided in an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0042] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0043] Visual question answering is a cross-modal artificial intelligence task that aims to answer natural language questions related to images by understanding their content. It encompasses visual recognition, natural language processing, and reasoning, emphasizing the model's comprehensive capabilities in complex scenarios. The performance of a visual question answering model is closely related to the training samples. When the training samples contain a wide variety of questions, the trained visual question answering model can handle a greater number of question types. Conversely, when the training samples lack sufficient question diversity, the visual question answering model can only handle a limited number of question types.

[0044] For example, on food delivery platforms, restaurant information often goes beyond textual data describing product features, details, and prices; it also heavily relies on visual elements such as appearance, color, and details. These two aspects complement each other, providing comprehensive information about the food products. Therefore, leveraging this multimodal data to fully understand product information and create a more convenient and intelligent platform for merchants and customers has become a key task, and large-scale models are a method aligned with current technological trends. However, experiments show that general multimodal large-scale models perform poorly in downstream tasks within the food service sector, demonstrating insufficient understanding of specific task-related knowledge. The main reason for this is the insufficient diversity of problems in the training samples.

[0045] In related technologies, large language models (LLMs) are primarily used to construct training samples for visual question-answering models using self-instruction or evol-instruction methods. However, LLMs lack sufficient visually relevant data. When generating questions using self-instruction or evol-instruction methods, they can only rely on limited templates, lacking diversity. This results in insufficient question diversity in the final training samples. Insufficient question diversity in the training samples leads to poor performance of the ultimately trained visual question-answering model, limiting its ability to handle only a limited number of questions.

[0046] To address the aforementioned technical problems, this application provides a training sample construction method. This method utilizes multiple key information elements included in a sample image and the attribute values ​​of each key information element to generate related questions for the sample image. Since sample images typically contain a large amount of key information, generating questions based on multiple key information elements can yield different questions for each key information element, thus resulting in a greater variety of questions and improving the question diversity of the training samples. Furthermore, when training a visual question answering model based on multiple training samples, the types of questions the visual question answering model can handle can be expanded, improving the performance of the visual question answering model.

[0047] See Figure 1 , Figure 1 This is an exemplary application scenario of the training sample construction method provided in this application. For example... Figure 1 As shown, the scenario includes a storage device 11 and a computing device 12, both of which are electronic devices. The storage device 11 stores a training sample set (also known as a VQA dataset) consisting of multiple training samples 13. The storage device 11 can be any electronic device known in the art or developed in the future that can be used for data storage. The computing device 12 can be an electronic device supporting a visual question-answering model, such as a server or computer cluster. Each training sample 13 can include an image and multiple question-answer pairs, each question-answer pair including a question for the image and the answer to the question.

[0048] The computing device 12 can construct a training sample set and then provide the training sample set to the storage device 11, where the storage device 11 stores the training sample set. The computing device 12 or other devices can obtain the training sample set from the storage device 11 and train the visual question answering model using multiple training samples 13 in the training sample set to obtain an applicable visual question answering model.

[0049] It should be understood that Figure 1This is merely an example, and the application scenarios of the method provided in this application are not limited to this. In fact, the method provided in this application is applicable to any scenario of training a visual question answering model.

[0050] See Figure 2 , Figure 2 This is a flowchart illustrating the steps of a training sample construction method provided in an embodiment of this application. This method can be applied to... Figure 1 The scenario shown is executed by computing device 12, and includes the following steps:

[0051] Step 201: Obtain multiple key-value pairs from the sample image.

[0052] Each key-value pair (KV) includes a key piece of information from the sample image and the attribute value of that key piece of information.

[0053] In this embodiment, the sample image refers to the image used to construct the training samples. Relevant sample images can be obtained based on the domain in which the visual question answering model is applied (hereinafter referred to as the target domain). The sample image can be called the first sample image. For example, if the visual question answering model is mainly used to handle image-related questions in the catering field (i.e., the target domain), then the sample image can be a food image; if the visual question answering model is mainly used to handle image-related questions in the text writing field (i.e., the target domain), then the sample image can be a text image.

[0054] In one implementation, during the construction of the training sample set, the computing device can first acquire a sample image set, which includes multiple sample images related to the target domain. For example... Figure 1 As shown, the set of sample images can be stored in a storage device, and the computing device can retrieve the set of sample images from the storage device when constructing training samples. Alternatively, the set of sample images can be stored directly in the computing device, and the computing device can retrieve the locally stored set of sample images when constructing training samples.

[0055] After acquiring the set of sample images, for each sample image in the set, the computing device can extract multiple key-value pairs from the sample image using a fourth model. The fourth model can be a pre-trained multimodal large model, such as GPT-4o, Qwen-VL-Chat, and Llama4, but is not limited to these.

[0056] For example, a user can construct an extraction template and prompts in a computing device. These templates and prompts instruct the fourth model which key information to extract from the sample image, the specific content of the extracted key information, and the return format of the key information and attribute values. After receiving the user-constructed extraction template and prompts, the computing device constructs input data based on the template, prompts, and sample image, and inputs this input data to the fourth model, enabling the fourth model to extract multiple key-value pairs from the sample image based on this input data.

[0057] For example, the extracted template can be a JSON schema in a format that the model can recognize:

[0058] JSON Schema: {

[0059] "$schema": "http: / / json-schema.org / draft-07 / schema#",

[0060] "type": "object",

[0061] "title": "Product Image Description",

[0062] "description": "Based on the product name, product description, and given product image, describe each attribute of the following product image according to the corresponding rules. The description must be realistic, detailed, and vivid, and cannot be empty, thus creating a structured depiction of the food image's pattern."

[0063] "properties": {

[0064] "recommendUser": {

[0065] "type": "string",

[0066] "description": "Target audience for this product"

[0067] ...

[0068] "required": ["name", "description", "background", "recommendUser", "recommendSituation", "subjects", "containers"]

[0069] }

[0070] The prompt could be: Based on the image provided below, extract the various attributes of the dishes in the image to structure the dish information, and return it as a JSON object. Please ensure that the generated JSON object strictly conforms to the following JSON Schema rules. An example JSON Schema is shown below: {…}.

[0071] It should be noted that after obtaining the sample image set, the computing device can extract multiple key-value pairs from each sample image in the sample image set.

[0072] It should be understood that the above are merely illustrative examples, and the methods for obtaining multiple key-value pairs of a sample image may include, but are not limited to, the examples above.

[0073] Step 202: Generate multiple questions about the sample image based on multiple key-value pairs.

[0074] Among these, the questions generated based on multiple key values ​​of the sample image represent the questions users might ask about the key information contained in the sample image. For example, when the sample image is related to food, users might ask questions such as, "Who is the food in the picture suitable for?" or "In which scenarios is the food in the picture suitable for use?"—questions related to key information such as the target audience and usage scenarios of the food.

[0075] In one implementation, a fifth model can be used to generate multiple questions associated with each key-value pair. The fifth model can be a pre-trained multimodal large model capable of processing data such as images and text, such as GPT-4o, Qwen-VL-Chat, and Llama4, but is not limited to these.

[0076] Based on the above examples, for each sample image, after obtaining multiple key-value pairs of the sample image, the computing device can construct input data based on the multiple key-value pairs, input the input data into the fifth model, and enable the fifth model to generate one or more questions corresponding to each key-value pair based on the input data.

[0077] Input data constructed based on multiple key-value pairs is as follows: "For each attribute {att_prompt_set} related to the following food image, ask questions about the {name}{att_name} attribute, generating 10 questions for each attribute and returning them as a JSON object. When asking questions involving multiple topics in a location relationship, specify the exact names, not letters. Ensure that the generated JSON object strictly conforms to the following JSON Schema rules. An example of a question is shown below, JSON Schema {}; {ans_format}. Example: For each attribute related to the following food image, ask questions about the background color, generating 10 questions for each attribute. {……} Answer {exampl}."

[0078] Alternatively, for each sample image, after obtaining multiple key-value pairs of the sample image, input data is constructed based on the sample image and the multiple key-value pairs, and then the input data is input into the fifth model, so that the fifth model generates multiple questions based on the sample image and the multiple key-value pairs.

[0079] It should be understood that the above are merely illustrative examples, and methods for generating problems based on multiple key-value pairs may include, but are not limited to, the examples above.

[0080] Step 203: For each question, determine the answer corresponding to the question based on the sample image and the question.

[0081] In one implementation, after generating multiple questions related to a sample image based on multiple key-value pairs, a second model can generate an answer for each question based on the sample image. Referring to the example above, after obtaining multiple questions returned by the fifth model in JSON format, input data can be constructed based on the sample image and the multiple questions. This input data is then fed into the second model, causing the second model to generate an answer for each question based on the sample image, thus obtaining the answer for each question.

[0082] Alternatively, after receiving multiple questions returned by the fifth model in JSON format, input data can be constructed based on sample images, multiple key-value pairs, and multiple questions. This input data can then be fed into the second model, allowing the second model to generate an answer for each question based on the sample images and multiple key-value pairs. This way, the answer for each question can be obtained.

[0083] For example, input data constructed based on a sample image, multiple key-value pairs, and multiple questions might be as follows: Based on the question-and-answer rules {rules}, please answer the following question {question} using the given product image and the given known image information {att_info}. Ensure your answer reflects the actual situation of the given product image, incorporating knowledge of the food and beverage industry. Avoid simply affirming or denying; consider the question critically and address the specific issue. Furthermore, the answer must be based on the rules and consistent with the product. If the question pertains to a provided image, answer it in conjunction with the image content. If the context of the question is incorrect, correct it before answering. The answer is returned as a JSON object. Ensure the generated JSON object strictly conforms to the following JSON Schema rules. An example answer is shown below: "...". Here, {att_info} includes multiple key-value pairs of the sample image, and {question} includes multiple questions about the sample image.

[0084] In practical applications, when controlling the second model to generate corresponding answers for each question, multiple key-value pairs can provide the second model with key information from the sample images, which can improve the second model's understanding of the sample images and thus improve the accuracy of the final answer.

[0085] The second model can be a pre-trained multimodal large model, such as GPT-4o, Qwen-VL-Chat, and Llama4, but is not limited to these.

[0086] Step 204: Construct the first training sample based on the sample images, multiple questions, and the answers to each question.

[0087] In this embodiment, after obtaining the sample image, multiple questions about the sample image, and the answer to each question, the computing device can use the sample image as an image in the training sample, and use each question and its corresponding answer as a question-answer pair to obtain multiple question-answer pairs. By combining the sample image and multiple question-answer pairs, a first training sample can be obtained.

[0088] As mentioned above, during the construction of the training sample set, the computing device can acquire multiple sample images. For each sample image, the method shown in steps 201-204 can be used to obtain a corresponding first training sample. In this way, multiple first training samples can be obtained, and multiple first training samples can be combined to obtain the training sample set.

[0089] In this embodiment, when constructing training samples, multiple key-value pairs of the sample image are first obtained. Then, multiple questions about the sample image are generated around these key-value pairs, and the answer to each question is determined. Finally, training samples are constructed based on the sample image, the multiple questions, and the answer to each question. This method utilizes multiple key information included in the sample image and the attribute values ​​of each key information to generate relevant questions about the sample image. Since there is usually a lot of key information in the sample image, generating questions based on multiple key information can yield different questions for each key information, thus resulting in a greater variety of questions and improving the question diversity of the training samples.

[0090] Furthermore, when training a visual question answering model based on multiple training samples, the types of questions that the visual question answering model can handle can be expanded, thereby improving the performance of the visual question answering model.

[0091] Optionally, based on multiple key-value pairs, multiple questions about the sample image are generated, including:

[0092] Identify multiple problem attributes;

[0093] For each key-value pair, generate one or more questions corresponding to each question attribute based on the key-value pair, thus obtaining multiple questions.

[0094] Among them, the problem attribute can also be called the problem type, and different problem attributes result in different problems.

[0095] In one implementation, multiple question attributes can be defined by the user. For example, the user can predefine multiple question attributes based on experience. These attributes may include, but are not limited to, general questions, situation-specific questions, technical questions, non-technical questions, independent questions, dependent questions, direct effect questions, and indirect effect questions. The user-defined question attributes can be stored in the computing device in JSON format. When generating multiple questions for a sample image, the computing device can retrieve these locally stored question attributes.

[0096] In another implementation, users can predefine multiple basic problem attributes (hereinafter referred to as problem categories) based on experience. These problem categories cover common features and information in the target domain. Then, the first model refines each problem category to determine the problem attributes included in each problem category, thus obtaining multiple problem attributes.

[0097] See Figure 3 , Figure 3This illustration shows a problem classification diagram provided by an embodiment of this application. Users can predefine multiple problem categories based on experience and set a description for each category. The multiple problem categories and their corresponding descriptions can be stored in a computing device in JSON format. The multiple problem categories may include... Figure 3 The problems are categorized into several major types, including situational problems, technical problems, context-dependent problems, visual problems, length problems, type problems, goal-oriented problems, and complexity problems.

[0098] When acquiring multiple problem attributes, the computing device can acquire multiple problem categories stored locally, as well as a description of each problem category. Then, it can construct input data based on the multiple problem categories and their corresponding descriptions, and input this input data into the first model, so that the first model can generate multiple problem attributes included in each problem category based on the input data.

[0099] The input data for constructing the first model based on multiple question categories and their corresponding descriptions is, for example: "Generating more question attribute types about food images based on the following question attributes, and outputting them in JSON format. 1. Contextual question: Description of the contextual question. 2. Technical question: Description of the technical question. 3. Context-dependent question: Description of the context-dependent question. 4. Visual effect question: Description of the visual effect question. 5. Length question: Description of the length question. 6. Type question: Description of the type question. 7. Goal question: Goal question. 8. Complexity question: Description of the complexity question." This input data is used to instruct the first model to generate... Figure 3 Each major question category shown includes multiple question attributes, such as... Figure 3 The two problem attributes of the medium-sized problem category, situational problems, are general problems and situation-specific problems.

[0100] Based on the above examples, after obtaining multiple question attributes output by the first model, for each key-value pair, one or more questions corresponding to each question attribute can be generated to obtain multiple questions. Alternatively, input data can be constructed based on the sample image, multiple key-value pairs, and question attributes, and then this input data can be fed into the first model to obtain multiple questions output by the first model based on the sample image and multiple key-value pairs.

[0101] The input data, constructed based on sample images, multiple key-value pairs, and question attributes, is as follows: "Based on the images provided below, generate one question corresponding to each question attribute in the following list for each key-value pair provided below. Question attributes include: 'General question, Context-specific question, Technical question, Non-technical question,' etc." This input data can instruct the first model to output one or more questions under each question attribute for each key-value pair.

[0102] Alternatively, for each key-value pair, input data can be constructed based on the sample image, the key-value pair, and multiple question attributes. This input data is then fed into the first model to obtain a set of questions output by the first model. Combining the final set of questions corresponding to each key-value pair yields multiple questions about the sample image. For example, for one of the question attributes, the constructed input data could be: "Based on the image provided below, generate 10 questions under the question attribute 'General Questions' for the following key-value pairs." This input data can instruct the first model to output one or more questions under each question attribute for each key-value pair.

[0103] The first model can be a pre-trained multimodal large model, such as GPT-4o, Qwen-VL-Chat, and Llama4, but is not limited to these.

[0104] It should be understood that the above are merely illustrative examples, and the methods for generating one or more questions corresponding to each question attribute based on key-value pairs may include, but are not limited to, the examples above.

[0105] In this application example, during the process of generating questions based on key-value pairs in a sample image, multiple question attributes are first obtained. Then, for each key-value pair in the sample image, one or more questions are generated for each question attribute. By using multiple pre-determined question attributes, the generated questions can cover multiple question attributes, reducing the probability of multiple generated questions being identical, thereby improving the diversity of the final questions.

[0106] In related technologies, the common practice is to first design a question template, and then use a model to extract image content to fill the question template to generate questions. The problem with this method is that the generated questions are heavily dependent on the question template. Since the question template is fixed, the types of questions generated are limited and lack diversity.

[0107] The method provided in this application generates one or more questions corresponding to each question attribute for each key-value pair extracted from the sample image. It allows control over the generation of questions corresponding to each question attribute and each key-value pair. For example, if N key-value pairs are extracted from the sample image and M question attributes are simultaneously determined, the method provided in this application generates H questions corresponding to each question attribute for each key-value pair, resulting in N×M×H questions. These questions not only cover each key-value pair in the sample image but also each question attribute. Compared to methods in related technologies, the generated questions are more diverse and open-ended.

[0108] Optionally, multiple problem attributes can be defined, including:

[0109] The first model is controlled to generate multiple problem categories;

[0110] The first model is controlled to generate the problem attributes included in each problem category, resulting in multiple problem attributes.

[0111] In one implementation, multiple problem categories can be generated first through a first model, and then each problem category can be refined through the first model to generate multiple problem attributes included in each problem category. The problem attributes included in multiple problem categories can be combined to obtain multiple problem attributes.

[0112] For example, a user can pre-construct input data to instruct a first model to generate multiple question categories. This input data can be pre-stored in a computing device in JSON format. During the process of determining the multiple question categories, the computing device retrieves the locally stored input data and then inputs it into the first model, causing the first model to generate and output the multiple question categories based on the input data.

[0113] The user-constructed input data is, for example: "You are helping to build a dataset for training a visual question-answering model. The goal of this model is to generate relevant text based on food images. To build this dataset, we need to define some basic question attributes that should cover different aspects of food images, such as the appearance, type, ingredients, and context. Please determine these basic question attributes and output them in JSON format." This input data instructs the first model to generate multiple categories of questions related to the food and beverage industry.

[0114] After obtaining multiple problem categories output by the first model, the computing device can construct another set of input data based on these problem categories. This input data is used to instruct the first model to refine each problem category and output multiple problem attributes included in each problem category.

[0115] The input data constructed based on the multiple question categories output by the first model is, for example, "further subdividing each question attribute of the previous output to generate more question attributes about food images and outputting them in JSON format." Then, the computing device inputs this input data into the first model, so that the first model refines each question category and outputs multiple question attributes included in each question category. By combining each question attribute output by the first model, multiple question attributes can be obtained.

[0116] Optionally, after obtaining multiple issue attributes, the computing device can display these attributes, allowing the user to filter them and remove unqualified attributes. Then, based on the final obtained issue attributes, the computing device can generate multiple issues corresponding to each key-value pair.

[0117] In this embodiment, during the process of determining problem attributes, multiple problem categories are first determined using a first model. Then, the first model is controlled to further subdivide each problem category to determine the problem attributes included in each category. The problem attributes included in multiple problem categories are combined to obtain multiple problem attributes. By running the model twice, problem attributes covering a wider range of problems can be obtained. Furthermore, with more problem attributes, a wider variety of problem types can be identified, thereby improving the diversity of the final problem obtained.

[0118] As mentioned above, related technologies generate questions based on fixed question templates, resulting in a limited number of question types and a lack of diversity. In contrast, the method provided in this application first controls a first model to generate multiple question categories, and then controls the first model to refine each question category to obtain multiple question attributes. This allows for the subdivision and expansion of questions within each question category, generating richer and more detailed question attributes compared to related technologies.

[0119] Optionally, the answer to the question is determined based on the sample image and the question, including:

[0120] The pre-trained second model is trained a second time using at least one second training sample, where the second training sample is an unqualified training sample.

[0121] The second model, after training, generates answers to questions based on sample images and questions.

[0122] In one implementation, during the process of determining the answer to each question using a second model, one or more unqualified training samples, i.e. negative samples, can first be obtained, including a second sample image, multiple questions about the second sample image, and the answer to each question.

[0123] Then, the pre-trained second model is trained using multiple second training samples, allowing the second model to adjust its parameters based on the training results. After secondary training of the second model, sample images and multiple questions can be input into it. The second model then generates an answer for each question based on the sample images and multiple questions, thus obtaining the answer for each question.

[0124] It is understandable that unqualified second training samples can prompt the second model to check for unreasonable aspects in the question output process, and summarize them into rules (i.e., adjust the parameters of the second model), and then output more accurate questions according to the rules.

[0125] The pre-trained second model can be a large multimodal model that has already been put into use, such as GPT-4, Tongyi Qianwen, and Llama4 mentioned above, but it is not limited to these.

[0126] In this embodiment, the second model is trained using unqualified second training samples, and then the trained second model generates corresponding answers based on the sample images and questions. This can improve the accuracy of the answers output by the second model, thereby improving the accuracy of the answers included in the first training samples.

[0127] Optionally, before training the second model using multiple second training samples to obtain the trained second model, the method further includes:

[0128] Obtain multiple third training samples, which are training samples constructed based on the questions generated by the second model;

[0129] The at least one second training sample is determined from a plurality of third training samples.

[0130] For example, after obtaining the sample image set, the multiple sample images included in the multiple sample image set can be divided into a first part of sample images and a second part of sample images, each of which includes multiple sample images. For each sample image included in the first part of sample images, steps 201-204 are performed as described above. During the execution of step 203, a pre-trained second model generates the answer corresponding to each question based on the sample image and multiple questions, thereby obtaining multiple training samples. These training samples are then used as third training samples. The user can filter the multiple third training samples and determine one or more unqualified training samples as second training samples. Subsequently, the computing device can perform secondary training on the pre-trained second model based on one or more second training samples to obtain a second model after secondary training.

[0131] Next, for each sample image in the second part of the sample images, steps 201-202 are executed as described above. During step 203, the second model after secondary training is controlled to generate the answer corresponding to each question based on the sample images and multiple questions, thus obtaining multiple first training samples. The qualified training samples obtained based on the first part of the sample images are merged with this part of the first training samples to obtain the final first training sample set.

[0132] The second model is trained using substandard training samples, allowing it to act as an inspector, summarizing inconsistencies and categorizing them into rules. These rules, along with textual and graphical information, are then used by the second model to generate answers to questions, forming the final VQA dataset. This approach introduces additional high-quality key-value (KV) attribute values ​​during automated question-answering rule summarization and answer generation, improving the reasonableness and accuracy of the answers.

[0133] In this embodiment of the application, the second model is retrained using multiple unqualified third training samples. This allows the second model to summarize the errors and experiences gained during the construction of the training samples, thereby improving the accuracy of the questions when generating questions based on the trained second model.

[0134] Optionally, the trained second model is controlled to generate answers to questions based on sample images and questions, including:

[0135] The second model, after training, generates the answer to the question based on the sample image, the question, and multiple key-value pairs.

[0136] In one implementation, in determining the answer to each question, the computing device can construct input data for a second model based on a sample image and a question, as well as multiple key-value pairs extracted from the sample image, and then input the input data into the second model so that the second model generates the answer to each question based on the input data.

[0137] For example, after obtaining multiple questions about a sample image, the computing device can construct prompts based on multiple key-value pairs, and then construct input data based on the sample image, the multiple questions about the sample image, and the prompts. The input data is then input into a second model, which generates an answer to each question based on the input data.

[0138] It is understandable that when the question includes a sample image, multiple questions about the sample image, and prompts constructed based on multiple key-value pairs, the prompts can provide the second model with more key information about the sample image, thereby improving the accuracy of the answer output by the second model.

[0139] In this embodiment of the application, during the process of generating an answer based on the second model, a sample image and a question are provided to the second model, along with multiple key-value pairs from the sample image. These multiple key-value pairs can provide the second model with key information from the sample image, enabling the second model to output a more accurate answer, thereby improving the accuracy of the final answer.

[0140] Optionally, before generating multiple questions corresponding to key information and attribute values, the method may further include:

[0141] Obtain the image description of the sample image;

[0142] The third model is controlled to extract key information and attribute values ​​from image descriptions.

[0143] In one implementation, for each sample image, the computing device first extracts an image caption from the sample image, and then extracts multiple key-value pairs from the image caption. For example... Figure 1 As shown, for the sample images included in training sample 13, the sample images can first be input into the sixth model to obtain the image description output by the sixth model. The image description is as follows: "This is a dish called spicy lobster tail, spicy and sweet... which can bring a great taste experience."

[0144] After obtaining the image description, input data can be constructed based on the prompts, extraction templates, and sample images described above. This input data is then fed into the fourth model, causing the fourth model to output multiple key-value pairs from the sample image based on the input data. For example, the key-value pairs could be: {"recommendUser": "Customers who like spicy food and seafood", "recommendSituation": "Gathering, dinner, late-night snack", "subjectList": ["Spicy lobster tail"], "imageQuality":……"material": "Plastic", "shape": "Rectangle", "contents": [{"dishName": "Spicy lobster tail"}]}]}.

[0145] Next, the first model can be controlled to generate multiple question categories. After obtaining multiple question categories, input data is constructed based on these categories and input into the first model, causing the first model to generate multiple question attributes based on this input data. Furthermore, output data is constructed based on multiple key-value pairs and multiple question outputs, and this input data is input into the fifth model, causing the fifth model to generate multiple questions corresponding to each key-value pair based on this output input.

[0146] As described above, when obtaining multiple key-value pairs from a sample image, a json-schema template for the KV attribute set (i.e., multiple key-value pairs) and a prompt for extracting the KV attribute set can be constructed first. Then, the input data of the model can be constructed based on the caption of the sample image, the json-schema template, and the prompt. By inputting this input data into the fourth model, the KV attribute set output by the fourth model can be obtained, and the extraction effect is as described above.

[0147] The fourth and sixth models can be the same model or different models, both of which can be large language models or multimodal models. By utilizing the image descriptions of sample images, a large model extracts the set of key-value (KV) attributes from the image captions of the sample images, resulting in higher-quality image descriptions. Then, using the above method, multiple questions are derived based on the KV attribute sets, and training samples are constructed. This transforms the KV attribute sets into a VQA dataset; the technique for transforming KV attribute sets into VQA datasets can be called the caption2vqa technique.

[0148] In this application example, the image description of the sample image is first obtained, and then the key information and attribute values ​​in the sample image are extracted from the image description, which can reduce the difficulty of extracting key-value pairs.

[0149] join Figure 4 , Figure 4 This is a flowchart illustrating a training sample construction method provided in an embodiment of this application. Figure 4 As shown, the method includes the following steps:

[0150] Step 401: Obtain the sample image set.

[0151] Step 402: Obtain a sample image from the sample image set.

[0152] For example, in constructing a training sample set, the computing device can first obtain a set of sample images from a storage device, which includes multiple sample images. Then, a single sample image can be obtained from the set of sample images.

[0153] Step 403: Obtain the image description of the sample image.

[0154] Step 404: Extract multiple key-value pairs from the image description of the sample image.

[0155] For example, after obtaining a sample image from a set of sample images, the computing device can construct input data based on the sample image and input this input data into a sixth model to obtain an image description generated by the sixth model based on the input data. Then, input data can be constructed based on the prompts, extraction templates, and sample images as described above and input into a fourth model to obtain multiple key-value pairs generated by the fourth model.

[0156] Step 405: Identify multiple problem categories.

[0157] Step 406: Determine the problem attributes included in each problem category to obtain multiple problem attributes.

[0158] For example, the computing device can first control the first model to output multiple problem categories, and then control the first model to output the problem attributes included in each problem category, so as to obtain multiple problem attributes. Alternatively, the computing device can obtain multiple problem categories pre-constructed by the user, then construct input data based on the multiple problem categories, input the input data into the first model, and obtain multiple problem attributes output by the first model based on the input data.

[0159] Step 407: For each key-value pair, determine the question corresponding to each question attribute to obtain multiple questions.

[0160] Step 408: Determine the answer to each question based on the sample image, key-value pairs, and multiple questions.

[0161] For example, the computing device can construct input data based on multiple key-value pairs and multiple question attributes, and then input this input data into a fifth model to obtain multiple questions output by the fifth model based on the input data. Subsequently, the computing device can construct input data based on sample images, key-value pairs, and multiple questions, and input this input data into a second model, causing the second model to output the answer corresponding to each question based on the input data.

[0162] In this process, the model first generates multiple question categories, then refines each question category to obtain multiple question attributes, and combines multiple question attributes and image attributes (i.e., KV attribute set) to generate a question set. This increases the question dimension of the VAQ dataset from both question attributes and image attributes, thereby improving the diversity of question answering.

[0163] Step 409: Construct the first training sample based on the sample images, multiple questions, and the answers to each question.

[0164] For example, after obtaining multiple questions corresponding to a sample image and the answer to each question, the computing device can combine the sample image, the multiple questions, and the answer to each question to obtain a first training sample.

[0165] Step 410: Determine whether the sample image has been processed.

[0166] Step 411: Combine multiple first training samples to obtain a training sample set.

[0167] For example, after obtaining a first training sample corresponding to each sample image, the computing device can determine whether the sample image set includes unprocessed sample images. If it does not include unprocessed sample images, it combines the multiple first training samples already obtained to obtain a training sample set. Otherwise, if it is determined that the sample image set also includes unprocessed sample images, it returns to continue executing step 402.

[0168] Based on the above examples, the construction of the VQA dataset can be carried out in two stages: a unimodal question construction stage and a multimodal question-answer generation stage. In the unimodal question construction stage: to increase the diversity of question types, a two-step strategy is adopted: first, multiple question categories are generated using a large model; after filtering, each question category is further refined using the large model, forming multiple question attributes, and then converted into JSON format that is easy for the model to understand. Next, to make the generated questions more open-ended, image constraints are discarded, and a unimodal approach is adopted, allowing the large model to ask questions based on various question attributes and product image attributes (i.e., the KV attribute set), thereby obtaining a diverse set of questions (i.e., the final multiple questions). In the multimodal question-answer generation stage: to ensure the reasonableness and accuracy of the answers, a rule induction step is added: negative examples (i.e., the second training samples) are filtered out, and the large model acts as an inspector, summarizing the unreasonable aspects and incorporating them into rules. Then, combining these rules and image / text information (i.e., the KV attribute set and sample images), the large model generates answers based on the questions, forming the final VQA dataset.

[0169] For the VQA dataset obtained using the above method, the reasonableness (human evaluation) is >95%, while the human evaluation pass rate for the visual semantic evol-instruct (vsevol-instruct) method is 40%. Clearly, the method provided in this application embodiment is superior to the vsevol-instruct method in terms of reasonableness.

[0170] Furthermore, in experiments with 12,000 images, the VAQ dataset generated using the evol-instruct method showed that after the diversity of the dataset converged, the VAQ dataset generated using the method provided in this application showed a 50% increase in diversity, and the number of cluster centers increased from approximately 1,000 to approximately 1,500.

[0171] Clearly, the VQA data obtained through the method provided in this application has a reasonableness of over 95%, and can automatically and continuously generate data for direct use in pre-training and supervised fine-tuning. This can significantly improve the diversity of the original dataset and help large models learn more diverse information and knowledge.

[0172] In layman's terms, with the widespread application of multimodal large-scale models across various industries, the scarcity of training data for vertical domain models is becoming increasingly prominent, especially regarding VQA datasets. To address this issue, this solution proposes a caption-based method for generating multimodal question-answering data (i.e., VQA datasets). This method automatically constructs high-quality text and image data from large-scale models, significantly improving the training performance of vertical domain multimodal large-scale models and optimizing the user experience. The core innovations of this solution include:

[0173] 1) Use captions to extract key attribute information from images and generate accurate image descriptions;

[0174] 2) Utilizing caption2vqa technology: Through high-quality prompt engineering, multimodal question-answering data with an accuracy of over 95% is automatically generated, providing continuous high-quality support for model fine-tuning.

[0175] It should be noted that the input data mentioned in the embodiments of this application can be constructed in any form that the model can recognize. The input data only needs to include the data required by the model to perform the task, as well as prompt words that can instruct the model to complete the task accurately.

[0176] The following will combine Figure 5 and Figure 6 The apparatus embodiments of this application are described in detail below. It should be understood that the apparatus in the embodiments of this application can perform the various methods described in the foregoing embodiments of this application, that is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0177] See Figure 5 , Figure 5 This is a schematic diagram of a training sample construction device provided in an embodiment of this application. Figure 5 As shown, the training sample construction device 500 may include:

[0178] The acquisition module 501 is used to acquire multiple key-value pairs of a sample image, each key-value pair including a key piece of information in the sample image and an attribute value of the key piece of information;

[0179] The first model 502 is used to generate multiple questions about the sample image based on the multiple key-value pairs;

[0180] The second model 503 is used to determine the answer to each question based on the sample image and the question.

[0181] The construction module 504 is used to construct a first training sample based on the sample image, the plurality of questions and the answer corresponding to each question.

[0182] Optionally, the first model 502 is specifically used to determine multiple problem attributes; for each key-value pair, generate one or more problems corresponding to each problem attribute based on the key-value pair to obtain the multiple problems.

[0183] Optionally, the first model 502 is specifically used to generate multiple problem categories and generate the problem attributes included in each problem category to obtain the multiple problem attributes.

[0184] Optionally, the training sample construction device 500 further includes a trainer for performing secondary training on the pre-trained second model using at least one second training sample, wherein the second training sample is an unqualified training sample.

[0185] Optionally, the second model 503 is specifically used to generate the answer corresponding to the question based on the sample image, the question, and the plurality of key-value pairs.

[0186] Optionally, the acquisition module 501 is further configured to acquire a plurality of third training samples before performing secondary training on the pre-trained second model using at least one second training sample, wherein the third training samples are training samples constructed based on the questions generated by the second model; and to determine the at least one second training sample from the plurality of third training samples.

[0187] Optionally, the training sample construction device 500 further includes a third model for obtaining image descriptions of the sample images and generating the plurality of key-value pairs based on the image descriptions.

[0188] See Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device 600 includes a memory 601 and a processor 602. The memory 601 stores program code 6011, and the processor 602 is used to call and execute the program code 6011 to perform a training sample construction method.

[0189] Furthermore, embodiments of this application also protect a training sample construction apparatus, which may include a memory and a processor, wherein the memory stores program code, and the processor is used to call and execute the program code to execute a training sample construction method provided in embodiments of this application.

[0190] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0191] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0192] It should be understood that the apparatus provided in this embodiment is used to execute the above-described training sample construction method, and therefore can achieve the same effect as the above-described implementation method.

[0193] When this device is applied to electronic devices, the processing module can be used to control and manage the actions of the electronic devices. The storage module can be used to support the execution of relevant program code by the electronic devices.

[0194] The processing module may be a processor, which can implement or execute various exemplary logic blocks, modules, and circuits shown in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0195] This embodiment also provides a readable storage medium storing program code. When the program code is run on an electronic device, the electronic device executes the above-described related method steps to implement a training sample construction method provided in the above embodiment.

[0196] This embodiment also provides a program product that, when run on an electronic device, causes the electronic device to perform the aforementioned steps to implement a training sample construction method provided in the above embodiment.

[0197] In this embodiment, the device, readable storage medium, program product, or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0198] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0199] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0200] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing training samples, characterized in that, The method includes: Extract multiple key-value pairs from the sample image, each key-value pair including a key piece of information in the sample image and an attribute value of the key piece of information; Determine multiple question attributes; for each key-value pair, generate one or more questions corresponding to each question attribute based on the key-value pair, to obtain multiple questions; For each of the aforementioned questions, the pre-trained second model is controlled to generate the corresponding answer to the question based on the sample image and the question. A first training sample is constructed based on the sample image, the plurality of questions, and the answer corresponding to each question.

2. The method as described in claim 1, characterized in that, The determination of multiple problem attributes includes: The first model is controlled to generate multiple problem categories; The first model is controlled to generate the problem attributes included in each of the problem categories, so as to obtain the plurality of problem attributes.

3. The method as described in claim 1, characterized in that, Before the second model, which is pre-trained under control, generates the answer to the question based on the sample image and the question, the method further includes: The second model, which was previously trained, is retrained using at least one second training sample, which is an unqualified training sample.

4. The method as described in claim 1, characterized in that, The second model, pre-trained to control the system, generates the answer to the question based on the sample image and the question, including: The second model is controlled to generate the answer to the question based on the sample image, the question, and the multiple key-value pairs.

5. The method as described in claim 3, characterized in that, Before performing secondary training on the pre-trained second model using at least one second training sample, the method further includes: Obtain multiple third training samples, which are training samples constructed based on the questions generated by the second model; The at least one second training sample is determined from the plurality of third training samples.

6. The method according to any one of claims 1-5, characterized in that, The extraction of multiple key-value pairs from the sample image includes: Obtain the image description of the sample image; The third control model generates the multiple key-value pairs based on the image description.

7. A training sample construction apparatus, characterized in that, include: The acquisition module is used to extract multiple key-value pairs from a sample image, each key-value pair including a key piece of information in the sample image and an attribute value of the key piece of information; A first model is used to determine multiple question attributes; for each key-value pair, one or more questions corresponding to each question attribute are generated based on the key-value pair to obtain multiple questions; The second model is used to control the pre-trained second model to generate the answer to each question based on the sample image and the question. A construction module is used to construct a first training sample based on the sample image, the plurality of questions, and the answer corresponding to each question.

8. An electronic device, characterized in that, The electronic device includes: Memory, used to store program code; A processor for calling and running the program code from the memory, causing the electronic device to perform the method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 6.

Images