Electronic data image text question and answer instruction generation method based on multi-modal large model

Abstract

The invention relates to the technical field of natural language processing, and provides an electronic data image text question and answer instruction generation method based on a multi-modal large model. The method comprises the following steps: extracting characters of an image by using a multi-modal large model to obtain character contents; enabling the multi-modal large model to perform self-questioning and answering on the character information to generate questions and answers; enabling the multi-modal large model to answer the question for the second time, and generating an answer basis; enabling the multi-modal large model to judge whether the question and answer pairs are related to the character information, and recording the irrelated question and answer pairs as invalid; performing consistency check on the question and answer pairs; and if the answers of the questions are not consistent before and after, marking the question-answer pairs as invalid, and deleting the invalid question-answer pairs. According to the method, the large-scale and high-quality image character question and answer data is automatically generated by using the multi-modal large model, so that the current situation that the quantity of the current character-centered image question and answer instruction fine adjustment data is insufficient and the quality is relatively low is made up, and the whole process does not need manual annotation, so that a large amount of manpower and time cost is saved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for generating electronic data image text question-answering instructions based on a multimodal large model. Background Art

[0002] In recent years, multimodal large models have made significant progress in the field of artificial intelligence and have shown broad application prospects. These models are able to process and fuse information from different modalities, such as text, images, and sounds, thereby demonstrating more advanced intelligence in various tasks. For example, these models can be used for scene understanding in autonomous driving systems, voice and image command responses in smart assistants, and so on. These applications all benefit greatly from the model's in-depth understanding of image content. Among them, text information in images, such as street signs, product labels, document content, etc., is often a key part of scene understanding. However, despite the great success of multimodal large models in image content understanding and recognition, they still have certain limitations in extracting and understanding text information in images. The ability of current multimodal large models to extract text from images is usually weaker than that of specialized text detection and recognition models, and their ability to understand the meaning and context of text information also has room for improvement.

[0003] In order to further improve the performance of multimodal large models, researchers have begun to focus on instruction fine-tuning technology. Instruction fine-tuning is a technique for fine-tuning a model after the pre-training stage. It usually involves providing specific questions or instructions to the model and guiding the model to generate corresponding answers or perform specific tasks. This fine-tuning can help the model better understand and execute human instructions and improve the performance of the model on specific tasks. However, the current quantity and quality of text-centric instruction fine-tuning question and answer data need to be improved. Existing datasets are often limited in size and of varying quality, and cannot meet the needs of large-scale fine-tuning. In addition, existing datasets are mainly focused on certain specific scenes, the corresponding image content is not extensive enough, and the instruction diversity is also defective, which makes the effect of fine-tuning very limited.

[0004] Manually obtaining large-scale, high-quality text-centric instruction fine-tuning question answering data is a difficult and time-consuming task. Manually collecting and annotating data is not only costly, but also difficult to meet the needs of rapidly developing multimodal large models.

[0005] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for generating electronic data image text question and answer instructions based on a multimodal large model.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for generating electronic data image text question-answering instructions based on a multimodal large model, comprising:

[0009] In step 201, a detailed description of the target image content is generated using a multimodal macro model;

[0010] In step 202, using the detailed description and image content generated in step 201, all text in the image is extracted using a multimodal large model to obtain text content in the target image;

[0011] In step 203, based on the image content, the detailed description generated in step 201, and the text content extracted in step 202, the multimodal large model is made to perform self-questioning and answering on the text information, and a preset number of questions about the text information and corresponding answers are generated;

[0012] In step 204, based on the image content and the question generated in step 203, the multimodal large model is made to answer the question twice, and at the same time, a basis or reason for the answer is generated;

[0013] In step 205, based on the image content, the text content extracted in step 202, and the question and answer generated in step 203, the multimodal large model determines whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid.

[0014] In step 206, based on the image content, the text content extracted in step 202, the question and answer generated in step 203, and the secondary answer and reason generated in step 204, the question and answer pair is checked for consistency; if the answer corresponding to the question is inconsistent, the question and answer pair is marked as invalid;

[0015] In step 207, invalid question-answer pairs are deleted based on the question-answer pair tags obtained in steps 205 and 206.

[0016] Preferably, the generating a detailed description of the target image content by using the multimodal large model specifically includes:

[0017] Control the length of the total prompt instruction, and generate a detailed description between 50 and 100 words;

[0018] The instruction prompt of step 201 is randomly selected from three instructions with the same semantics but different expressions. If a single generation fails, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped.

[0019] Let the input image be I, the predefined instruction prompt be p1, and the multimodal large model used be M, then the detailed description is d=M(I,p1).

[0020] Preferably, the detailed description and image content generated in step 201 are used to extract all the text in the image using a multimodal large model to obtain the text content in the target image, specifically including:

[0021] The instruction prompt of step 202 is obtained by combining the predefined prompt trunk part p2 and the detailed description d generated in step 201;

[0022] The predefined hint trunk is randomly selected from three instructions with the same semantics but different expressions. If a single extraction fails, another hint is used. If all three hints fail to generate, the image is skipped.

[0023] In the instruction prompt of step 202, it is required to output the text:<extracted text> The extracted text information is output one by one in the format of , and then the extracted text is extracted one by one using regular expressions and saved as a list T; T = {t n |n=1,2,…,k1}=extract(R t ,o2), where o2=M(I,p2+d), R t is the preset regular expression, and k1 is the total number of text messages.

[0024] Preferably, based on the image content, the detailed description generated in step 201, and the text content extracted in step 202, the multimodal large model is made to perform self-questioning and answering of the text information, and a preset number of questions about the text information and corresponding answers are generated, specifically including:

[0025] The instruction prompt of step 203 is obtained by combining the predefined prompt trunk p3, the detailed description d of the image, and the extracted text information T;

[0026] The predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions. If a single self-questioning fails or the output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to be generated, the image is skipped.

[0027] In the instruction prompt of step 203, it is required to output Question:<question corpus> \nAnswer:<anser corpus> The generated question-answer pairs are output one by one in the format of , and then the question-answer pairs in the output are extracted one by one using regular expressions and saved as two lists Q and A respectively; Q = {q n|n=1,2,…,k2}=extract(R q ,o3), A={a n |n=1,2,…,k2}=extract(R a ,o3), where o3=M(I,p3+d+T), R1, R a is the regular expression of this step, and k2 is the number of question-answer pairs.

[0028] Preferably, based on the image content and the question generated in step 203, the multimodal large model is made to answer the question twice, and at the same time, a basis or reason for the answer is generated, specifically including:

[0029] The second question-answering and reasoning generation stage answers the questions generated in step 203 in the form of a thought chain, generating a reasoning basis and a new answer for each question;

[0030] The instruction prompt of step 204 is obtained by combining the predefined prompt trunk p4, the detailed description d of the image, the extracted text information T and the generated question Q; the predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions. If a single self-question and answer fails or the output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to be generated, the image is skipped;

[0031] The reasoning results and secondary answers generated in step 204 are used for consistency detection in subsequent stages to evaluate the quality of the question-answer pair;

[0032] In the instruction prompt of step 204, the output is required to be Reason:<reason corpus> \nAnswer: <ansercorpus>The generated thought chains are output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as two lists C and A′ respectively; C={c n |n=1,2,…,k2}=extract(R c ,o4), A′={a′ n |n=1,2,…,k2}=extract(R a ,o4), where o4=M(I,p4+d+T+Q), R c , R a is the default regular expression.

[0033] Preferably, based on the image content, the text content extracted in step 202, and the question and answer generated in step 203, the multimodal large model is allowed to determine whether the question-answer pair is related to the text information, and if not, the question-answer pair is marked as invalid, specifically including:

[0034] The instruction prompt of step 205 is obtained by combining the predefined prompt trunk p5 and the extracted text information T with the generated question-answer pair Q and A;

[0035] The multimodal large model gives a binary judgment of "yes" or "no" for each question-answer pair. Questions judged as "no" are retained with a probability of 30% as a supplement to the diverse question-answer data.

[0036] In the command prompt of step 205, it is required to output the content "Judgement":<True or False> The generated thought chain is output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as a list J T ; J T ={j t,n ∈{True,False}|n=1,2,…,k2}=extract(R jt ,o5), where o5=M(I,p5+T+Q+A), R jt is the default regular expression.

[0037] Preferably, the consistency check of the question-answer pair is performed based on the image content, the text content extracted in step 202, the question and answer generated in step 203, and the secondary answer and reason generated in step 204; if the answer corresponding to the question is inconsistent, the question-answer pair is marked as invalid, specifically including:

[0038] The multimodal large model determines whether the initial answer generated in step 203 and the secondary answer generated in step 204 express the same meaning based on the reasoning evidence generated in step 204. The multimodal large model gives a binary judgment of "yes" or "no" to each question-answer pair, and deletes the questions judged as "no";

[0039] The instruction prompt of step 206 is composed of the predefined prompt trunk p6, the question Q generated in step 203 and step 204, the answer A, A′ and the reasoning basis C;

[0040] In the instruction prompt of step 206, it is required to output the content "Judgement":<True or False> The generated thought chain is output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as a list J C ; J C ={j c,n ∈{True,False}|n=1,2,…,k2}=extract(R jc ,o6), where o6=M(I,p6+Q+A+A ′ +C), R jc is the default regular expression.

[0041] Preferably, the step of deleting invalid question-answer pairs based on the question-answer pair tags obtained in step 205 and step 206 specifically includes:

[0042] In step 205, a judgment J is obtained as to whether the question-answer pair is related to the text in the image. T ={j t,n ∈{True,False}|n=1,2,…,k2}, set False to True with a probability of 30% to increase the diversity of question-answer pairs to a certain extent;

[0043] J ′ T ={j ′ t,n ={True,False}|n=1,2,…,k2}

[0044] p(j ′ t,n =True|j t,n =True)=1

[0045] p(j ′ t,n =True|j t,n = False) = 0.3

[0046] p(j ′ t,n =False|j t,n = False) = 0.7

[0047] Afterwards, the final selection of the question and answer pairs is made by J ′ T and J C Ask and get: J final ={j f,n =and(j ′ t,n ,j c,n )|n=1,2,…,k2}, that is, according to the final judgment result, delete the question-answer pairs judged as False, and retain the question-answer pairs judged as True.

[0048] Preferably, the multimodal big model is a closed-source multimodal big model and a corresponding API interface, or the multimodal big model is an open-source multimodal big model.

[0049] In a second aspect, the present invention further provides an apparatus for generating electronic data image text question and answer instructions based on a multimodal large model, which is used to implement the method for generating electronic data image text question and answer instructions based on a multimodal large model described in the first aspect, and the apparatus comprises:

[0050] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for generating electronic data image text question and answer instructions based on a multimodal large model as described in the first aspect.

[0051] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0052] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory to execute the method of the first aspect.

[0053] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute the method of the first aspect.

[0054] The present invention uses a multimodal large model to automatically generate large-scale, high-quality image-text question-answering data, thereby making up for the current situation of insufficient and low-quality fine-tuning data for text-centric image question-answering instructions. The entire process of data generation does not require manual annotation, saving a lot of manpower and time costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a flowchart of a method for generating electronic data image text question-answering instructions based on a multimodal large model provided by an embodiment of the present invention;

[0057] Figure 2 It is a flowchart of a method for generating electronic data image text question-answering instructions based on a multimodal large model provided by an embodiment of the present invention;

[0058] Figure 3 It is a schematic diagram of a method for generating electronic data image text question-answering instructions based on a multimodal large model provided by an embodiment of the present invention;

[0059] Figure 4 It is a schematic diagram of a method for generating electronic data image text question-answering instructions based on a multimodal large model provided by an embodiment of the present invention;

[0060] Figure 5 It is a schematic diagram of a method for generating electronic data image text question-answering instructions based on a multimodal large model provided by an embodiment of the present invention;

[0061] Figure 6 It is a schematic diagram of the architecture of an electronic data image text question and answer instruction generation device based on a multimodal large model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as meaning open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0064] In the description of the present invention, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0065] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0066] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0067] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0068] Embodiment 1:

[0069] The present invention uses an existing multimodal large model to design an automated data generation pipeline, which completes the process of self-extraction, self-questioning, self-reasoning, and self-screening of text information through a series of carefully designed instructions and prompts, thereby realizing the automated generation of high-quality, large-scale, and diverse image text information question and answer data. This method is mainly aimed at electronic data images, including but not limited to electronic posters, web page screenshots, email screenshots, e-commerce platform product promotional images, and other images that include complex visual information and high-density, multi-form text. Example 1 of the present invention provides a method for generating electronic data image text question and answer instructions based on a multimodal large model, such as Figure 1 As shown, including:

[0070] In step 201, a detailed description of the target image content is generated using a multimodal macro model, with particular attention paid to text information in the target image; the target image is an electronic data image.

[0071] In step 202, using the detailed description and image content generated in step 201, a multimodal large model is used to extract all text in the image to obtain the text content in the target image.

[0072] In step 203, based on the image content, the detailed description generated in step 201, and the text content extracted in step 202, the multimodal large model is used to perform self-questioning and answering of the text information, and generate a preset number of questions about the text information and corresponding answers, which are also referred to as question-answer pairs in subsequent implementations; the preset number is obtained by technical personnel in this field based on empirical analysis.

[0073] In step 204, based on the image content and the question generated in step 203, the multimodal large model is instructed to give a second answer to the question and generate a basis or reason for the answer at the same time.

[0074] In step 205, based on the image content, the text content extracted in step 202, and the questions and answers generated in step 203, the multimodal large model determines whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid.

[0075] In step 206, the question-answer pair is checked for consistency based on the image content, the text content extracted in step 202, the question and answer generated in step 203, and the secondary answer and reason generated in step 204; if the answer corresponding to the question is inconsistent (i.e., the answer obtained in step 203 is inconsistent with the answer obtained in the secondary answer in step 204), the question-answer pair is marked as invalid.

[0076] In step 207, invalid question-answer pairs are deleted based on the question-answer pair tags obtained in steps 205 and 206.

[0077] In some embodiments, the multimodal large model is used to generate a detailed description of the target image content, such as Figure 2 As shown, specifically including:

[0078] In step 301, the length of the total prompt instruction is controlled, and the generated detailed description is between 50 and 100 words; this is to maintain the detailed description of the image content and control the length of the total prompt instruction. Considering the randomness in the generation of multimodal large models and the requirements for diversity of training data, the instruction prompt of step 201 is randomly selected from three instructions with the same semantics but different expressions.

[0079] In step 302, if a single generation fails, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped. Let the input image be I, the predefined instruction prompt be p1, and the multimodal large model used be M, then d = M (I, p1) is described in detail. Such a design not only greatly improves the success rate of the model generating usable results, but also ensures the diversity of training data by expressing different prompt instructions. This idea and method are implemented in all steps of the entire process involving the generation of large models. In the subsequent steps, the number, design method, and usage of prompt instructions are the same as this step, so the design idea and usage of prompt instructions will not be explained in detail in each step.

[0080] The instruction prompt is specifically: "You are an image content analysis expert. Please generate a brief but accurate description of the input image. The length of the description is about 50 to 100 words. Please pay special attention to the text information in the image."

[0081] In an optional implementation, the detailed description and image content generated in step 201 are used to extract all text in the image using a multimodal large model to obtain text content in the target image, specifically including:

[0082] The instruction prompt of step 202 is obtained by combining the predefined prompt trunk part p2 and the detailed description d generated in step 201; under ideal conditions, the model can extract all recognizable text in the image.

[0083] The predefined hint trunk is randomly selected from three instructions with the same semantics but different expressions. If a single extraction fails, another hint is used. If all three hints fail to generate, the image is skipped.

[0084] In order to control the output format of the model, the command prompt in step 202 requires the output to be in the form of Text: <extractedtext>The extracted text information is output one by one in the format of , and then the extracted text is extracted one by one using regular expressions and saved as a list T; T = {t n |n=1,2,…,k1}=extract(R t ,o2), where o2=M(I,p2+d), R t is the preset regular expression, and k1 is the total number of text messages.

[0085] The instruction prompt is specifically: "Based on the image information and detailed description provided, comprehensively analyze all text and other visual information in the image. The detailed description of the image is <description>; The output should conform to the following format:

[0086] Text: [First paragraph of text extracted]

[0087] Text: [Second paragraph of text extracted]

[0088] And so on."

[0089] In an actual application scenario, based on the image content, the detailed description generated in step 201, and the text content extracted in step 202, the multimodal large model is enabled to perform self-questioning and answering of the text information, and generate a preset number of questions about the text information and corresponding answers, specifically including:

[0090] During the question-answering stage, several question-answer pairs will be generated. Each question should be clear and meaningful and related to the text information in the image. Each question corresponds to an answer, and each answer should be a few short words or phrases. The instruction prompt of step 203 is obtained by combining the predefined prompt trunk p3, the detailed description d of the image, and the extracted text information T.

[0091] The predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions. If a single self-question and answer fails or the output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to be generated, the image is skipped.

[0092] In order to control the output format of the model, the instruction prompt in step 203 requires the output to be Question:<question corpus> \nAnswer:<anser corpus> The generated question-answer pairs are output one by one in the format of , and then the question-answer pairs in the output are extracted one by one using regular expressions and saved as two lists Q and A respectively; Q = {q n |n=1,2,…,k2}=extract(R q ,o3), A={a n |n=1,2,…,k2}=extract(R a ,o3), where o3=M(I,p3+d+T), R q , R a is the regular expression of this step, and k2 is the number of question-answer pairs.

[0093] The instruction prompt is specifically: "Based on the image information, detailed description and text information, generate some questions and corresponding answers as required; the questions should be straightforward and meaningful, asking about the text information in the image. You can only ask questions about the text information in the image. The answer to the question must be related to the text information. The detailed description of the image is <description>, the text information is<text content> ; The output should conform to the following format:

[0094] Question: [First question]

[0095] Answer: [Answer to the first question]

[0096] Question: [Second question]

[0097] Answer: [Answer to the second question]

[0098] And so on."

[0099] The multimodal large model is made to answer the question twice based on the image content and the question generated in step 203, and at the same time, a basis or reason for the answer is generated, specifically including:

[0100] The secondary question-answering and reasoning generation stage answers the questions generated in step 203 in the form of a thought chain, and generates a reasoning basis and a new answer for each question.

[0101] The instruction prompt of step 204 is obtained by combining the predefined prompt trunk p4, the detailed description d of the image, the extracted text information T and the generated question Q; the predefined prompt trunk part is randomly selected from three instructions with the same semantics but different expressions. If a single self-question and answer fails or the output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to be generated, the image is skipped.

[0102] The purpose of this step is, first of all, to generate reasoning information data for the answer to the question-answer pair, which not only helps to expand the training data, but also can generate higher quality answers by using the thought chain feature. The reasoning results and secondary answers generated in step 204 are used for consistency detection in subsequent stages to evaluate the quality of the question-answer pair.

[0103] In order to control the output format of the model, the command prompt in step 204 requires the output to be in Reason: <reasoncorpus>\nAnswer:<anser corpus> The generated thought chains are output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as two lists C and A′ respectively. C={c n |n=1,2,…,k2}=extract(R c ,o4), A′={a′ n |n=1,2,…,k2}=extract(R a ,o4), where o4=M(I,p4+d+T+Q), R c , R a is the preset regular expression;

[0104] The instruction prompt of step 204 is specifically: "Based on the image information, detailed description and text information, answer the following questions. The answers to the questions must be related to the text information. Please think step by step, first give a short reasoning to explain the basis for answering the question, and then give a specific answer; the detailed description of the image is <description>, the text information is<text content> The questions that need to be answered are <questions>The output should be in the following format:

[0105] Question: [First question]

[0106] Answer: [Answer to the first question]

[0107] Basis: [reasoning basis corresponding to the first question]

[0108] Question: [Second question]

[0109] Answer: [Answer to the second question]

[0110] Basis: [reasoning basis for the second question]

[0111] And so on."

[0112] In an optional implementation, based on the image content, the text content extracted in step 202, and the question and answer generated in step 203, the multimodal large model is allowed to determine whether the question-answer pair is related to the text information, and if not, the question-answer pair is marked as invalid, specifically including:

[0113] The instruction prompt of step 205 is obtained by combining the predefined prompt trunk p5 and the extracted text information T with the generated question-answer pair Q and A.

[0114] The multimodal large model gives a binary judgment of "yes" or "no" for each question and answer pair. Questions judged as "no" are retained with a probability of 30% to supplement the diverse question and answer data. This is due to the diversity of the data set. When training the large model, not only the OCR and text reasoning capabilities of the model should be considered, but also the basic image understanding capabilities must be covered and guaranteed to a certain extent.

[0115] In the command prompt of step 205, it is required to output the content "Judgement":<True or False> The generated thought chain is output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as a list J T .J T ={j t,n ∈{True,False}|n=1,2,…,k2}=extract(R jt ,o5), where o5=M(I,p5+T+Q+A), R jt is the default regular expression.

[0116] The instruction prompt of step 205 is specifically: "Below is a series of questions and corresponding answers; the questions are <questions>, the corresponding answer is <answers>; As a data verification expert, please judge whether the answers to these questions can be directly obtained from<text content> , or infer from them; output "yes" or "no" judgments for these questions in turn. The output should conform to the following format:

[0117] Judgment: [Judgment corresponding to the first question and answer]

[0118] Judgment: [Judgment corresponding to the second question and answer]

[0119] And so on."

[0120] In actual use, the consistency check is performed on the question-answer pair based on the image content, the text content extracted in step 202, the question and answer generated in step 203, and the secondary answer and reason generated in step 204; if the answer corresponding to the question is inconsistent, the question-answer pair is marked as invalid, specifically including:

[0121] The multimodal large model determines whether the initial answer generated in step 203 and the secondary answer generated in step 204 express the same meaning based on the reasoning basis generated in step 204. The multimodal large model gives a binary judgment of "yes" or "no" to each question and answer pair, and deletes questions judged as "no".

[0122] The instruction prompt of step 206 is composed of the predefined prompt trunk p6, the question Q generated in step 203 and step 204, the answer A, A' and the reasoning basis C; this step will delete the question judged as "no". The purpose of consistency detection is mainly to delete the question-answer pairs that are confusing, incorrect, require out-of-distribution knowledge, and have no answers. The questions and answers of these question-answer pairs often do not correspond strictly, so they will introduce noise to the training of the model and damage the performance of the model.

[0123] In the instruction prompt of step 206, it is required to output the content "Judgement":<True or False> The generated thought chain is output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as a list J C .J C ={j c,n ∈{True,False}|n=1,2,…,k2}=extract(R j3 ,o6), where o6=M(I,p6+Q+A+A ′ +C), R jc is the default regular expression.

[0124] The instruction prompt of step 206 is specifically: "You are an expert in image content analysis. Here are some questions and two sets of answers. You need to judge whether the two sets of answers are consistent and correct based on the content of the image and the given reasoning basis. If the content of the first set of answers is different from the content of the second set of answers, the output judgment is "no"; the question to be judged is <questions>, the reasoning needed to answer the question is <reasons>, the first set of answers is <answers>, the second set of answers is<new answers> Output "yes" or "no" answers to these questions in turn; the output should conform to the following format:

[0125] Judgment: [Judgment corresponding to the first question and answer]

[0126] Judgment: [Judgment corresponding to the second question and answer]

[0127] And so on."

[0128] In some embodiments, deleting invalid question-answer pairs based on the question-answer pair tags obtained in step 205 and step 206 specifically includes:

[0129] In step 205, a judgment J is obtained as to whether the question-answer pair is related to the text in the image. T ={j t,n ∈{True,False}|n=1,2,…,k2}, False is set to True with a probability of 30%. That is, by changing the judgment result, the question-answer pairs originally judged as False have a 30% probability of being retained, thereby increasing the diversity of question-answer pairs to a certain extent.

[0130] J ′ T ={j ′ t,n ={True,False}|n=1,2,…,k2}

[0131] p(j ′ t,n =True|j t,n =True)=1

[0132] p9j ′ t,n =True|j t,n = False) = 0.3

[0133] p(j ′ t,n =False|j t,n = False) = 0.7

[0134] Afterwards, the final selection of the question and answer pairs is made by J ′ T and J C Ask and get: J final ={j f,n =and(j ′ t,n ,j c,n )|n=1,2,…,k2}, that is, according to the final judgment result, delete the question-answer pairs judged as False, and retain the question-answer pairs judged as True.

[0135] Each step in this embodiment can be understood as: by inputting corresponding instruction prompts into the multimodal big data, the results of the corresponding stage (i.e., the output results of the multimodal large model) are obtained.

[0136] This embodiment uses a multimodal large model to automatically generate large-scale, high-quality image-text question-and-answer data, thereby making up for the current situation of insufficient quantity and low quality of text-centric image question-and-answer instruction fine-tuning data. The entire process of data generation does not require manual annotation, saving a lot of manpower and time costs. This embodiment implements a method for generating electronic data image graphic understanding question-and-answer data, and its workflow and instruction prompts can process various types of generalized electronic data images, including but not limited to electronic posters, web page screenshots, email screenshots, e-commerce platform product promotional images, etc. At the same time, it has the potential to be extended to more image types, such as natural scene images, document images, etc.

[0137] Embodiment 2:

[0138] In view of the defects and deficiencies in the current research field, the present invention provides a method for generating image text understanding question and answer data based on a multimodal large model. The purpose is to use the existing multimodal large model to realize large-scale batch automatic generation of various types of electronic data image text question and answer instructions. This solves the shortcomings in quantity and quality of large-scale multimodal instruction fine-tuning data centered on text, and overcomes the high time and labor costs of obtaining large-scale image and text understanding question and answer data annotation.

[0139] To achieve the above purpose, Figure 3 As shown, the method comprises the following steps:

[0140] Step 1: Use the multimodal large model to generate a detailed description of the target image content, paying special attention to the text information in the target image.

[0141] Step 2: Using the detailed description and image content generated in step 1, use the multimodal large model to extract all the text in the image and obtain the text content in the target image.

[0142] Step 3: Based on the image content, the detailed description generated in step 1, and the text content extracted in step 2, the multimodal large model is used to perform self-questioning and answering of the text information, generating several questions about the text information and corresponding answers.

[0143] Step 4: Based on the image content and the question generated in step 3, the multimodal large model is used to give a second answer to the question and generate the basis or reason for the answer.

[0144] Step 5: Based on the image content, the text content extracted in step 2, and the questions and answers generated in step 3, the multimodal large model determines whether the question-answer pair is related to the text information. If the content of the question-answer pair is not related to the text information, the corresponding judgment result is "no". To ensure the diversity of questions, the question-answer pairs that are not related to the text information are retained with a probability of 30%.

[0145] Step 6: Based on the image content, the text content extracted in step 2, the question and answer generated in step 3, and the secondary answer and reason generated in step 4, the question and answer pair are checked for consistency. If the answer to the question is inconsistent, the corresponding judgment result is "no".

[0146] Step 7: Delete invalid question-answer pairs based on the question-answer pair tags obtained in steps 5 and 6.

[0147] In one embodiment of the present invention, in step 1: in order to maintain the detailed description of the image content and control the length of the total prompt instruction, the generated detailed description is about 50 to 100 words. Taking into account the randomness in the generation of the multimodal large model and the requirements of the diversity of the training data, the instruction prompts for the detailed description generation stage are randomly selected from three instructions with the same semantics but different expressions. If a single generation fails, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped. Let the input image be I, the predefined instruction prompt be p1, and the multimodal large model used be M, then the detailed description d=M(I,p1). Such a design not only greatly improves the success rate of the model generating usable results, but also ensures the diversity of the training data under the effect of expressing different prompt instructions. This idea and method are implemented in all steps involving the generation of the large model in the entire process. In the subsequent steps, the number, design method, and usage of the prompt instructions are the same as this step, so the design idea and usage of the prompt instructions will not be explained in detail in each step.

[0148] In one embodiment of the present invention, in step 2: the instruction prompt for extracting text is obtained by combining the predefined prompt trunk part p2 and the detailed description d generated in step 1; under ideal conditions, the model can extract all recognizable text in the image; the predefined prompt trunk part is randomly selected from three instructions with the same semantics but different expressions. If a single extraction fails, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped. In order to control the output format of the model, the instruction prompt requires the output to be in the form of Text:<extracted text> The extracted text information is output one by one in the format of , and then the extracted text is extracted one by one using regular expressions and saved as a list T. T={t n |n=1,2,…,k1}=extract(R t ,o2), where o2=M(I,p2+d), R t is the regularization of this step, and k1 is the total number of text messages.

[0149] In one embodiment of the present invention, in step three: the self-questioning stage will generate several question-answer pairs, each question should be clear and meaningful, and related to the text information in the image, each question corresponds to an answer, and each answer should be a few short words or phrases; the instruction prompts at this stage are obtained by combining the predefined prompt trunk p3, the detailed description of the image d, and the extracted text information T; the predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions. If a single self-questioning fails or the output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped. In order to control the output format of the model, the instruction prompt requires the output to be Question:<question corpus> \nAnswer:<anser corpus> The generated question-answer pairs are output one by one in the format of , and then the question-answer pairs in the output are extracted one by one using regular expressions and saved as two lists Q and A respectively. n |n=1,2,…,k2}=extract(R q ,o3), A={a n |n=1,2,…,k2}=extract(R a ,o3), where o3=M(I,p3+d+T), R q , R a is the regularization of this step, and k2 is the number of question-answer pairs.

[0150] In one embodiment of the present invention, in step 4: the secondary question-answering and reasoning generation stage will answer the questions generated in step 3 in the form of a chain of thoughts, and generate a reasoning basis and a new answer for each question. The instruction prompts in this stage are obtained by combining the predefined prompt trunk p4, the detailed description d of the image, the extracted text information T and the generated question Q; the predefined prompt trunk part is randomly selected from three instructions with the same semantics but different expressions. If a single self-question-answering fails or the output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped. The purpose of this step is, first of all, to generate reasoning information data for the answer to the question and answer pair, which not only helps to expand the training data, but also can generate higher quality answers by using the characteristics of the chain of thoughts. In addition, the reasoning results and secondary answers generated in this stage will be used for consistency detection in subsequent stages to evaluate the quality of the question and answer pair. In order to control the output format of the model, the instruction prompt requires the output to be Reason:<reason corpus> \nAnswer: <ansercorpus>The generated thought chains are output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as two lists C and A′ respectively. C={c n |n=1,2,…,k2}=extract(R c ,o4), A′={a′ n |n=1,2,…,k2}=extract(R a ,o4), where o4=M(I,p4+d+T+Q), R c , R a is the regularization for this step.

[0151] In one embodiment of the present invention, in step five: in the stage of judging whether the question-answer pair is related to the text information, the multimodal large model will give a binary judgment of "yes" or "no" for each question-answer pair. The instruction prompt at this stage is obtained by combining the predefined prompt trunk p5 and the extracted text information T with the generated question-answer pairs Q and A. Questions judged as "no" are retained with a probability of 30% as a supplement to the diverse question-answer data. This is due to the diversity of the data set. When training the large model, not only the OCR and text reasoning capabilities of the model should be considered, but also the basic image understanding capabilities should be covered and guaranteed to a certain extent. In order to control the output format of the model, the instruction prompt requires the output to be in the form of Judgement:<True orFalse> The generated thought chain is output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as a list J T .J T ={j t,n ∈{True,False}|n=1,2,…,k2}=extract(R jt ,o5), where o5=M(I,p5+T+Q+A), R jt is the regularization for this step.

[0152] In one embodiment of the present invention, in step six: the consistency detection stage, the multimodal large model determines whether the initial answer generated in step three and the secondary answer generated in step four express the same meaning based on the reasoning basis generated in step four. The multimodal large model will give a binary judgment of "yes" or "no" for each question and answer pair. The instruction prompts at this stage are composed of a predefined prompt trunk p6, the questions Q generated in steps three and four, the answers A, A' and the reasoning basis C. This step will delete the questions that are judged as "no". The purpose of consistency detection is mainly to delete question and answer pairs that are confusing, incorrect, require knowledge outside the distribution, and have no answers. The questions and answers of these question and answer pairs often cannot be strictly corresponding, so they will introduce noise into the training of the model and destroy the performance of the model. In order to control the output format of the model, the instruction prompt requires the output to be in the form of Judgement:<True or False> The generated thought chain is output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as a list J C .J C ={j c,n ∈{True,False}|n=1,2,…,k2}=extract(R jc ,o6), where o6=M(I,p6+Q+A+A ′ +C), R jc is the regularization for this step.

[0153] In one embodiment of the present invention, the screening process of the question-answer data is as follows: In step 5, a judgment J is obtained as to whether the question-answer pair is related to the text in the image. T ={j t,n ∈{True,False}|n=1,2,…,k2}, False is set to True with a probability of 30% to increase the diversity of question-answer pairs to a certain extent.

[0154] J ′ T ={j ′ t,n ={True,False}|n=1,2,…,k2}

[0155] p(j ′ t,n =True|j t,n =True)=1

[0156] p(j ′ t,n =True|j t,n =False0=0.3

[0157] p(j ′ t,n =False|j t,n = False) = 0.7

[0158] Afterwards, the final selection of the question and answer pairs is made by J ′ T and J C Ask and get:

[0159] J final ={j f,n =and(j ′ t,n ,j c,n )|n=1,2,…,k2}

[0160] The present invention is based on Example 1 and the above method, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0161] This embodiment proposes a method for generating electronic data image text understanding question and answer data based on a multimodal large model, such as Figure 3 As shown, the implementation steps include the following:

[0162] Step 1: Use the multimodal large model to generate a detailed description of the target image content, paying special attention to the text information in the target image.

[0163] In step 1, a multimodal large model is used to generate a detailed description of the target image, and the text information in the image is emphasized in the instruction prompt, such as Figure 3 In order to maintain the detailed description of the image content and control the length of the total prompt instruction, the generated detailed description is about 50 to 100 words. A reference instruction prompt is as follows:

[0164] "You are an expert in image content analysis. Please generate a brief but accurate description of the input image. The length of the description is about 50 to 100 words. Please pay special attention to the text information in the image."

[0165] Using command prompts similar to the reference commands above, the multimodal large model can generate a more detailed image description. The image description obtained in this step will be integrated into the command prompts of subsequent steps as auxiliary information, helping the multimodal large model to produce high-quality output in the extraction of text information and the generation of question-answer pairs.

[0166] Step 2: Using the detailed description and image content generated in step 1, use the multimodal large model to extract all the text in the image and obtain the text content in the target image.

[0167] like Figure 3 As shown in the second part of the stage, step 2 is mainly responsible for extracting all the text information in the image. This step can also be implemented using traditional OCR text recognition algorithms. When using a multimodal large model to extract text information, the command prompt is composed of a predefined prompt trunk and a detailed description obtained in step 1. A reference command prompt is as follows:

[0168] "Based on the image information and detailed description provided, a comprehensive analysis of all text and other visual information in the image. The detailed description of the image is <description>The output should be in the following format:

[0169] Text: [First paragraph of text extracted]

[0170] Text: [Second paragraph of text extracted]

[0171] And so on. ”

[0172] Using command prompts similar to the above reference commands, the multimodal large model can more completely extract the text information in the image, as well as some other important visual information. At the same time, this section of command prompts also plays a role in controlling the output of the large model. Based on regular expressions, text information can be easily extracted from the output of the large model.

[0173] The schematic diagram of regular extraction is as follows Figure 4 As shown. Through the control of the instructions, the output of the large model has been divided into several columns by row. At this time, by detecting the position of the prefix "text:", the following valid information can be extracted and stored in a list, which is convenient for subsequent processing and use in the following steps.

[0174] Step 3: Based on the image content, the detailed description generated in step 1, and the text content extracted in step 2, the multimodal large model is used to perform self-questioning and answering of the text information, generating several questions about the text information and corresponding answers.

[0175] like Figure 3 As shown in the third part of the stage, step three is mainly responsible for generating self-questioning data based on image information. The self-questioning stage will generate several question-answer pairs. Each question should be clear and meaningful and related to the text information in the image. Each question corresponds to an answer, and each answer should be a few short words or phrases. The instruction prompts in this stage are a combination of a predefined prompt trunk, a detailed description of the image, and the extracted text information. A reference instruction prompt is as follows:

[0176] "Based on the image information, detailed description and text information, generate some questions and corresponding answers as required. The questions should be straightforward and meaningful, asking about the text information in the image. You can only ask questions about the text information in the image. The answers to the questions must be related to the text information. The detailed description of the image is <description>, the text information is <textcontent>The output should be in the following format:

[0177] Question: [First question]

[0178] Answer: [Answer to the first question]

[0179] Question: [Second question]

[0180] Answer: [Answer to the second question]

[0181] And so on. ”

[0182] Using command prompts similar to the reference commands above, the multimodal large model can generate a series of question-answer pairs. At the same time, it can ensure high-quality output. These question-answer pairs are usually related to text, but there will also be some questions that are not related to text and ask about other visual information in the image. Therefore, these questions need to be post-processed and screened. Note that the command promotion in this step also controls the output format of the large model. The output of this stage also obtains question-answer pair information through regular extraction, and the calculation process of regular matching will not be repeated here.

[0183] Step 4: Based on the image content and the question generated in step 3, the multimodal large model is used to give a second answer to the question and generate the basis or reason for the answer.

[0184] like Figure 3 As shown in the fourth part of the figure, the main function of step four is to generate the reasoning basis of the question-answer pair and the secondary answer required in the subsequent consistency detection stage. The secondary question-answer and reasoning generation stage will answer the questions generated in step three in the form of a thought chain, and generate a reasoning basis and a new answer for each question. The instruction prompts at this stage are obtained by combining the predefined prompt trunk, the detailed description of the image, the extracted text information, and the generated questions. A reference instruction prompt is as follows:

[0185] "Based on the image information, detailed description and text information, answer the following questions. The answers to the questions must be related to the text information. Please think step by step, first give a short reasoning, explain the basis for answering the question, and then give a specific answer. The detailed description of the image is <description>, the text information is<text content> The questions that need to be answered are <questions>The output should be in the following format:

[0186] Question: [First question]

[0187] Answer: [Answer to the first question]

[0188] Basis: [reasoning basis corresponding to the first question]

[0189] Question: [Second question]

[0190] Answer: [Answer to the second question]

[0191] Basis: [reasoning basis for the second question]

[0192] And so on. ”

[0193] Using instruction prompts similar to the above reference instructions can enable the multimodal large model to answer questions twice in the form of thought chains. The thought chain answering process can not only improve the accuracy of the model's answer to questions, but also generate relevant reasoning evidence. The reasoning evidence can not only enrich the information of the data set, but also provide relevant information for the subsequent consistency verification steps.

[0194] Step 5: Based on the image content, the text content extracted in step 2, and the questions and answers generated in step 3, the multimodal large model determines whether the question-answer pair is related to the text information. If the content of the question-answer pair is not related to the text information, the corresponding judgment result is "no". To ensure the diversity of questions, the question-answer pairs that are not related to the text information are retained with a probability of 30%.

[0195] like Figure 3 As shown in the stage 5 part of , the main function of step 5 is to generate a judgment on whether it is related to the text. Based on the generated judgment, the question-answer pair data generated in step 3 can be screened. In actual operation, it is observed that there is certain irrelevant information in the question-answer pair data in step 3. These question-answer pairs do not focus on the text information in the image, but on some other visual information, or raise divergent questions that require external knowledge. The answers to these questions are often vague, and the corresponding answers generated in step 3 may not be suitable. Therefore, this type of question needs to be screened to a certain extent to avoid damaging the performance of the large model. In the stage of judging whether the question-answer pair is related to the text information, the multimodal large model will give a binary judgment of "yes" or "no" for each question-answer pair. The instruction prompts at this stage are obtained by combining the predefined prompt trunk and the extracted text information with the generated question-answer pairs. A reference instruction prompt is as follows:

[0196] "Below is a series of questions and their corresponding answers. The questions are <questions>, the corresponding answer is <answers>As a data validation expert, please judge whether the answers to these questions can be directly obtained from<text content> , or inferred from it. Output "yes" or "no" judgments for these questions in turn. The output should conform to the following format:

[0197] Judgment: [Judgment corresponding to the first question and answer]

[0198] Judgment: [Judgment corresponding to the second question and answer]

[0199] And so on. ”

[0200] Using command prompts similar to the reference commands above, the multimodal large model can output judgments on a series of question-answer pairs. Considering the possibility of misjudgment and the need to ensure data diversity, questions that are judged as "no" are retained with a probability of 30% as a supplement to the diverse question-answer data. The judgments generated in this stage will work together with the judgments generated in the consistency detection stage for screening.

[0201] Step 6: Based on the image content, the text content extracted in step 2, the question and answer generated in step 3, and the secondary answer and reason generated in step 4, the question and answer pair are checked for consistency. If the answer to the question is inconsistent, the corresponding judgment result is "no".

[0202] like Figure 3 As shown in the stage 6 section of , the main function of step 6 is to perform consistency check on the question-answer pair data. The questions and answers of the instruction data must be highly relevant and minimize ambiguity. Questions that cannot be answered or ambiguous are not suitable as instruction fine-tuning data. The role of consistency check is to let the large model answer the same question based on different prefix conditions, and then check whether the content of multiple answers is consistent. In the consistency detection stage, the multimodal large model determines whether the initial answer generated in step three and the secondary answer generated in step four express the same meaning based on the reasoning basis generated in step four. The multimodal large model will give a binary judgment of "yes" or "no" for each question and answer pair. The instruction prompts at this stage consist of a predefined prompt trunk, the questions generated in steps three and four, two sets of answers, and reasoning basis. A reference instruction prompt is as follows:

[0203] "You are an expert in image content analysis. Below are some questions and two sets of answers. You need to judge whether the two sets of answers are consistent and correct based on the content of the image and the reasoning given. If the content of the first set of answers is different from the second set of answers, the output judgment is "no". The question to be judged is <questions>, the reasoning needed to answer the question is <reasons>, the first set of answers is <answers>, the second set of answers is<new answers> Output "yes" or "no" for each of these questions. The output should be in the following format:

[0204] Judgment: [Judgment corresponding to the first question and answer]

[0205] Judgment: [Judgment corresponding to the second question and answer]

[0206] And so on. ”

[0207] Using command prompts similar to the above reference commands, the multimodal large model can output judgments on a series of question-answer pairs. The judgments at this stage are used together with the judgments at the previous stage to perform the final screening operation.

[0208] Step 7: Delete invalid question-answer pairs based on the question-answer pair tags obtained in steps 5 and 6.

[0209] like Figure 3 As shown in stage 7 of the above, this stage uses the judgments obtained in the first two steps to perform the final screening. First, according to the requirements of step 5, the "no" judgment in the text-related judgment is flipped to "yes" with a probability of 30%. Then the text-related judgment and the consistency test judgment are summed in turn to obtain the final judgment result. According to the final judgment result, the question-answer pairs judged as "no" are deleted, and the question-answer pairs judged as "yes" are retained, as shown in Figure 5 shown.

[0210] Embodiment 3:

[0211] like Figure 6 , is a schematic diagram of the architecture of a device for generating electronic data image text question and answer instructions based on a multimodal large model according to an embodiment of the present invention. The device for generating electronic data image text question and answer instructions based on a multimodal large model according to this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0212] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0213] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for generating electronic data image text question and answer instructions based on a multimodal large model in Example 1. The processor 21 executes the method for generating electronic data image text question and answer instructions based on a multimodal large model by running the non-volatile software programs and instructions stored in the memory 22.

[0214] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0215] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the method for generating electronic data image text question and answer instructions based on a multimodal large model in the above-mentioned embodiment 1 is executed.

[0216] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0217] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0218] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.< / answers> < / reasons> < / questions> < / answers> < / questions> < / questions> < / description> < / textcontent> < / description> < / description> < / ansercorpus> < / answers> < / reasons> < / questions> < / answers> < / questions> < / questions> < / description> < / reasoncorpus> < / description> < / description> < / extractedtext> < / ansercorpus>

Claims

1. A method for generating electronic data image text question-answering instructions based on a multimodal large model, characterized in that: include: In step 201, a detailed description of the target image content is generated using a multimodal macro model; In step 202, using the detailed description and image content generated in step 201, all text in the image is extracted using a multimodal large model to obtain text content in the target image; In step 203, based on the image content, the detailed description generated in step 201, and the text content extracted in step 202, the multimodal large model is made to perform self-questioning and answering on the text information, and a preset number of questions about the text information and corresponding answers are generated; In step 204, based on the image content and the question generated in step 203, the multimodal large model is made to answer the question twice, and at the same time, a basis or reason for the answer is generated; In step 205, based on the image content, the text content extracted in step 202, and the question and answer generated in step 203, the multimodal large model determines whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid. In step 206, based on the image content, the text content extracted in step 202, the question and answer generated in step 203, and the secondary answer and reason generated in step 204, the question and answer pair is checked for consistency; if the answer corresponding to the question is inconsistent, the question and answer pair is marked as invalid; In step 207, invalid question-answer pairs are deleted based on the question-answer pair tags obtained in steps 205 and 206.

2. The method for generating electronic data image text question-answering instructions based on a multimodal large model according to claim 1, characterized in that: The method of using the multimodal large model to generate a detailed description of the target image content specifically includes: Control the length of the total prompt instruction, and generate a detailed description between 50 and 100 words; The instruction prompt of step 201 is randomly selected from three instructions with the same semantics but different expressions. If a single generation fails, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped. Let the input image be I, the predefined instruction prompt be p1, and the multimodal large model used be M, then the detailed description is d=M(I,p1).

3. The method for generating electronic data image text question-answering instructions based on a multimodal large model according to claim 1, characterized in that: The detailed description and image content generated in step 201 are used to extract all the text in the image using a multimodal large model to obtain the text content in the target image, specifically including: The instruction prompt of step 202 is obtained by combining the predefined prompt trunk part p2 and the detailed description d generated in step 201; The predefined hint trunk is randomly selected from three instructions with the same semantics but different expressions. If a single extraction fails, another hint is used. If all three hints fail to generate, the image is skipped. In the instruction prompt of step 202, it is required to output the text:<extracted text> The extracted text information is output one by one in the format of , and then the extracted text is extracted one by one using regular expressions and saved as a list T; T = {t n |n=1,2,…,k1}=extract(R t ,o2), where o2=M(I,p2+d), R t is the preset regular expression, and k1 is the total number of text messages.

4. The method for generating electronic data image text question-answering instructions based on a multimodal large model according to claim 1, characterized in that: Based on the image content, the detailed description generated in step 201, and the text content extracted in step 202, the multimodal large model is made to perform self-questioning and answering on the text information, and a preset number of questions about the text information and corresponding answers are generated, specifically including: The instruction prompt of step 203 is obtained by combining the predefined prompt trunk p3, the detailed description d of the image, and the extracted text information T; The predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions. If a single self-questioning fails or the output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to be generated, the image is skipped. In the instruction prompt of step 203, it is required to output Question:<question corpus> \nAnswer: <ansercorpus>The generated question-answer pairs are output one by one in the format of , and then the question-answer pairs in the output are extracted one by one using regular expressions and saved as two lists Q and A respectively; Q = {q n |n=1,2,…,k2}=extract(R q ,o3), A={a n |n=1,2,…,k2}=extract(R a ,o3), where o3=M(I,p3+d+T), R q , R a is the regular expression of this step, and k2 is the number of question-answer pairs.< / ansercorpus> 5. The method for generating electronic data image text question-answering instructions based on a multimodal large model according to claim 1, characterized in that: Based on the image content and the question generated in step 203, the multimodal large model is made to answer the question twice, and at the same time, a basis or reason for the answer is generated, specifically including: The second question-answering and reasoning generation stage answers the questions generated in step 203 in the form of a thought chain, generating a reasoning basis and a new answer for each question; The instruction prompt of step 204 is obtained by combining the predefined prompt trunk p4, the detailed description d of the image, the extracted text information T and the generated question Q; the predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions. If a single self-question and answer fails or the output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to be generated, the image is skipped; The reasoning results and secondary answers generated in step 204 are used for consistency detection in subsequent stages to evaluate the quality of the question-answer pair; In the instruction prompt of step 204, the output is required to be Reason:<reason corpus> \nAnswer: <ansercorpus>The generated thought chains are output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as two lists C and A′ respectively; C={c n |n=1,2,…,k2}=extract(R c ,o4), A′={a′ n |n=1,2,…,k2}=extract(R a ,o4), where o4=M(I,p4+d+T+Q), R c , R a is the default regular expression.< / ansercorpus> 6. The method for generating electronic data image text question-answering instructions based on a multimodal large model according to claim 1, characterized in that: Based on the image content, the text content extracted in step 202, and the questions and answers generated in step 203, the multimodal large model is made to determine whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid, specifically including: The instruction prompt of step 205 is obtained by combining the predefined prompt trunk p5 and the extracted text information T with the generated question-answer pair Q and A; The multimodal large model gives a binary judgment of "yes" or "no" for each question-answer pair. Questions judged as "no" are retained with a probability of 30% as a supplement to the diverse question-answer data. In the command prompt of step 205, it is required to output the content "Judgement":<True or False> The generated thought chain is output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as a list J T ; J T ={j t,n ∈{True,False}|n=1,2,…,k2}=extract(R jt ,o5), where o5=M(I,p5+T+Q+A), R jt is the default regular expression.

7. The method for generating electronic data image text question-answering instructions based on a multimodal large model according to claim 1, characterized in that: The consistency check of the question-answer pair is performed based on the image content, the text content extracted in step 202, the question and answer generated in step 203, and the secondary answer and reason generated in step 204; if the answer corresponding to the question is inconsistent, the question-answer pair is marked as invalid, specifically including: The multimodal large model determines whether the initial answer generated in step 203 and the secondary answer generated in step 204 express the same meaning based on the reasoning evidence generated in step 204. The multimodal large model gives a binary judgment of "yes" or "no" to each question-answer pair, and deletes the questions judged as "no"; The instruction prompt of step 206 is composed of the predefined prompt trunk p6, the question Q generated in step 203 and step 204, the answer A, A′ and the reasoning basis C; In the instruction prompt of step 206, it is required to output the content "Judgement":<True or False> The generated thought chain is output one by one in the format of , and then the information in the output is extracted one by one using regular expressions and saved as a list J C ; J C ={j c,n ∈{True,False}|n=1,2,…,k2}=extract(R jc ,o6), where o6=M(I,p6+Q+A+A ′ +C), R jc is the default regular expression.

8. The method for generating electronic data image text question-answering instructions based on a multimodal large model according to claim 1, characterized in that: The step of deleting invalid question-answer pairs according to the question-answer pair marks obtained in step 205 and step 206 specifically includes: In step 205, a judgment J is obtained as to whether the question-answer pair is related to the text in the image. T ={j t,n ∈{True,False}|n=1,2,…,k2}, set False to True with a probability of 30% to increase the diversity of question-answer pairs to a certain extent; J ′ T ={j ′ t,n ={True,False}|n=1,2,…,k2} p(j ′ t,n =True|j t,n =True)=1 p(j ′ t,n =True|j t,n =False)=0.3 p(j ′ t,n =False|j t,n (False)=0.7 Afterwards, the final selection of the question and answer pairs is made by J ′ T and J C Ask and get: J final ={j f,n =and(j ′ t,n ,j c,n )|n=1,2,…,k2}, that is, according to the final judgment result, delete the question-answer pairs judged as False, and retain the question-answer pairs judged as True.

9. The method for generating electronic data image text question-answering instructions based on a multimodal large model according to any one of claims 1 to 8, characterized in that: The multimodal big model is a closed-source multimodal big model and a corresponding API interface, or the multimodal big model is an open-source multimodal big model.

10. The method for generating electronic data image text question-answering instructions based on a multimodal large model according to any one of claims 1 to 8, characterized in that: Electronic data images include one or more of electronic posters, web page screenshots, email screenshots, and product promotional images on e-commerce platforms.

Images