Multi-modal large model-based sequence character bill image question and answer data generation method

Abstract

The invention relates to the technical field of natural language processing technologies, and provides a sequence character bill image question and answer data generation method based on a multi-modal large model. The method comprises the following steps: detailedly describing the content of a target image; enabling the multi-modal large model to perform self-questioning and answering on the character information, and generating questions and answers of the character information; enabling the multi-modal large model to perform secondary answering on the proposed question and reasoning to generate an answering basis or reason at the same time; enabling the multi-modal large model to judge whether the question-answer pairs are related to the character information or not, and performing consistency check on the question-answer pairs; and deleting irrelevant and inconsistent question and answer pairs. According to the method, a multi-mode large model is used, and large-scale batch automatic generation of image text information question and answer data in a receipt scene is realized. Therefore, the defects of large-scale multi-mode instruction fine adjustment data taking characters as the center for receipts in quantity and quality are overcome, and the high time and labor cost for obtaining large-scale image-text understanding question and answer data annotations is overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for generating sequence text bill image question-answering data based on a multimodal large model. Background Art

[0002] In recent years, multimodal large models have made significant progress in the field of artificial intelligence and have shown broad application prospects. Multimodal large models are artificial intelligence models that can process multiple types of data including text, images, and audio, and show advanced intelligence in various tasks. For example, multimodal large models can be used for scene understanding in autonomous driving systems, voice and image command responses in smart assistants, generating natural language descriptions for images, combining text and images for translation, and answering questions about specified images. These applications all benefit greatly from the multimodal model's in-depth understanding of multiple types of data, especially image data. One factor that makes multimodal large models have a strong understanding of image data is that they can recognize and understand text information in images, such as road signs, trademarks, document content, etc.

[0003] However, despite breakthroughs in image content understanding and recognition, multimodal large models still face challenges in processing text information in images. Current multimodal models are generally not as accurate as specialized optical character recognition systems in extracting text information from images, and they also lack in understanding the semantics and context of these text information. To overcome these limitations, researchers are actively developing new training strategies and techniques, one of which is instruction fine-tuning. Instruction fine-tuning is a method of performing additional training on a pre-trained model so that the model can better understand and execute given task instructions. This method usually involves using a set of carefully designed instruction-output pairs as training data, which is designed to teach the model how to correctly generate the expected output. It can significantly improve the performance of the pre-trained model in performing specific tasks. By training with an appropriate instruction dataset, the model can be more adapted to specific application scenarios, thereby producing more accurate and useful results. How to generate a high-quality instruction dataset is a key issue.

[0004] However, the current public fine-tuning Q&A data for receipt images is small in quantity and poor in quality, making this part of the data a defect of large-model fine-tuning Q&A datasets, affecting the performance of large models in related tasks. Relying on manual acquisition of large-scale, high-quality text-centric fine-tuning Q&A data is a difficult and time-consuming task. Manually collecting and annotating data is not only costly, but also difficult to meet the needs of rapidly developing multimodal large models.

[0005] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for generating sequence text bill image question and answer data based on a multimodal large model.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for generating sequential text bill image question-answering data based on a multimodal large model, comprising:

[0009] In step 201, the public receipt dataset is used as image data to generate corresponding text comprehension question and answer data, and the annotations of the dataset are sorted to obtain information of the image data;

[0010] In step 202, the target image content is described in detail using the multimodal large model and the information of the image data obtained in step 201;

[0011] In step 203, using the information of the picture data obtained in step 201, determine the number of questions related to the text information generated for a picture and the corresponding answers;

[0012] In step 204, using the information of the picture data obtained in step 201 and the image content description obtained in step 202, the multimodal large model is made to perform self-questioning and answering on the text information, and generate questions related to the text information and corresponding answers, which are also referred to as question-answer pairs in subsequent embodiments; wherein the number of questions and answers generated is equal to the number of questions related to the text information and corresponding answers generated determined in step 202;

[0013] In step 205, using the information of the image data obtained in step 201 and the image content description obtained in step 202, the multimodal large model is made to give a second answer to the question raised in step 204 and at the same time infer the basis or reason for generating the answer;

[0014] In step 206, based on the information of the image data sorted in step 201 and the questions and answers generated in step 204, the multimodal large model is used to determine whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid.

[0015] In step 207, the question-answer pair is checked for consistency using the information of the image data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer to the question and the reason generated in step 205; if the two answers corresponding to the same question are inconsistent (i.e., the answer generated in step 204 and the answer generated by the secondary answer in step 205 are inconsistent), the question-answer pair is marked as invalid;

[0016] In step 208, invalid question-answer pairs are deleted based on the question-answer pair tags obtained in steps 206 and 207.

[0017] Preferably, the use of a public receipt dataset as image data to generate corresponding text comprehension question and answer data, and collating the annotations of the dataset to obtain information about the image data, specifically includes:

[0018] Arrange the annotations of the dataset to obtain information of the image data, extract key information from the annotations and simplify them; the specific extracted information includes one or more of the text content, the location of the text content, the type of information represented by the text content, and the brief information of the receipt;

[0019] The position of the text content is represented by the coordinates of the lower left corner and the lower right corner of the text content. The types of information represented by the text content include store name, store address, product name, unit price of product and total price of product. The brief information of the receipt includes store name, store address and amount spent. The specific saving format is that for the text content Text, the text content position Coordinate and the type of information Category represented by the text content are saved together as a tuple (Text, Category, Corrdinate). The brief information of the receipt is saved as a separate dictionary Info_dict: {"store name":,"store address":,"total price of item":,}.

[0020] Preferably, the use of the multimodal large model and the information of the image data obtained in step 201 to describe the target image content in detail specifically includes:

[0021] Use a multimodal large model to generate a detailed description of the target image, and emphasize the text information in the image in the instruction prompts during the detailed description generation stage. At the same time, control the length of the total prompt instructions so that the generated detailed description is between 70 and 100 words.

[0022] The instruction prompts of the detailed description generation stage are randomly selected from three instructions with the same semantics but different expressions, and the generated question-answer pairs are extracted from the answers of the multimodal large model by using a pre-set regular expression matching method. If the matching fails, the regular expression is replaced to continue matching. If all failures occur, it means that the self-question-answer output does not meet the format requirements, and another instruction prompt is used. If the outputs of the three instruction prompts do not meet the format requirements, the image is skipped;

[0023] The instruction prompts for generating detailed descriptions are generated based on the following template: "You are an image content analysis expert for receipt images; input text information of the receipt image: (Text, Category, Corrdinate); brief information: Info_dict, please generate a short but accurate description of the input receipt image based on the provided image information. The length of the description is about 70 to 100 words. Please pay special attention to the text information in the image."

[0024] Preferably, the method of using the information of the picture data obtained in step 201 to determine the number of questions about text information generated for a picture and the corresponding answers specifically includes:

[0025] For the input image I, the corresponding image annotation information in the dataset is L, and the sorted image information is Info. Then the number of question-answer pairs generated for the input image I is Q num =Num(Info); wherein Num(Info) is a preset function.

[0026] Preferably, the information of the picture data obtained in step 201 and the image content description obtained in step 202 are used to enable the multimodal large model to perform self-questioning and answering of text information, and generate questions about the text information and corresponding answers, specifically including:

[0027] The instruction prompt used by the multimodal large model to perform self-questioning and answering of text information is obtained by combining a predefined prompt trunk p2, a detailed description d of the image, and information Info of the image data obtained from the annotation; the predefined prompt trunk part is randomly selected from three instructions with the same semantics but different expressions;

[0028] The instruction prompt in the self-questioning and answering stage requires the output of question and answer pairs one by one in the format of "Question: [Your first question] Answer: [Answer according to the text in the picture]";

[0029] The results generated by the multimodal large model are matched with a preset regular expression to extract the question Q and the corresponding answer A. If the number of questions or answers extracted is 0, or the number of questions is not equal to the number of answers, it means that the match is wrong, and the regular expression is changed to continue matching;

[0030] If all fail, it means that the output format of the multimodal large model in this self-question and answering does not meet the requirements. Change to another instruction prompt. If the output formats of all three instruction prompts do not meet the requirements, the generation of question and answer pairs for the image fails and the image is skipped.

[0031] Preferably, the information of the picture data obtained in step 201 and the image content description obtained in step 202 are used to enable the multimodal large model to give a secondary answer to the question raised in step 204 and to infer the basis or reason for generating the answer, specifically including:

[0032] Answer the questions generated in step 204 in the form of a thought chain, generating a reasoning basis and a new answer for each question;

[0033] The instruction prompts in the secondary question answering and reasoning generation stage are obtained by combining the predefined prompt trunk p3, the detailed description d of the image, the information Info of the image data sorted out from the annotation, and the question Q generated in step 204. The predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions;

[0034] If a single self-question and answer output does not meet the format requirements, another command prompt is used. If all three command prompts fail to generate, the image is skipped;

[0035] Preferably, the instruction prompts in the secondary question-answering and reasoning generation stage require the output of the generated thought chain one by one in the format of "question: [first question] answer: [answer corresponding to the first question] basis: [reasoning basis corresponding to the first question]", and then use regular expressions to extract the information in the output one by one and save them separately as two lists, reasoning reason Reason and secondary answer A'.

[0036] Preferably, based on the information of the image data collated in step 201 and the questions and answers generated in step 204, the multimodal large model is made to determine whether the question-answer pair is related to the text information, and if not, the question-answer pair is marked as invalid, specifically including:

[0037] The instruction prompt used by the multimodal large model to determine whether the question-answer pair is related to the text information is obtained by combining the predefined prompt trunk p4 and the information Info of the image data obtained from the annotation with the questions Q and A generated in step 204;

[0038] For questions that are judged as "no", the answer pair is discarded and the questions that are judged as "yes" are retained;

[0039] Among them, the instruction prompts in the judgment stage require the output of the generated thinking chain one by one in the format of "Judgment: [the judgment corresponding to the i-th question and answer]", and then use regular expressions to extract the information in the output one by one and save it as a list Judge.

[0040] Preferably, the consistency check of the question-answer pair using the information of the picture data obtained in step 201 and the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer and reason to the question generated in step 205 specifically includes:

[0041] The instruction prompt used for consistency check of the question-answer pair is composed of the predefined prompt trunk p5, the question Q generated in step 203 and step 204, the answer A, the secondary answer A' generated in step 205 and the reasoning basis Reason;

[0042] Delete the questions that are judged as "no";

[0043] Among them, the instruction prompt in the consistency check phase requires the output to be "Judge:<True or False> " format to output the generated thought chain one by one, and then use regular expressions to extract the information in the output one by one and save it as a consistency judgment list Judge; where Judge i ={j i,k ∈{True,False}k=1,2,...,Q num}.

[0044] Preferably, the multimodal big model is a closed-source multimodal big model and a corresponding API interface, or the multimodal big model is an open-source multimodal big model.

[0045] In a second aspect, the present invention further provides a device for generating sequential text bill image question and answer data based on a multimodal large model, which is used to implement the method for generating sequential text bill image question and answer data based on a multimodal large model described in the first aspect, and the device comprises:

[0046] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for generating sequential text bill image question and answer data based on a multimodal large model as described in the first aspect.

[0047] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the method for generating sequence text bill image question and answer data based on a multimodal large model as described in the first aspect.

[0048] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory to execute the method of the first aspect.

[0049] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute the method of the first aspect.

[0050] The present invention uses a multimodal large model to achieve large-scale batch automatic generation of image text information question and answer data in the ticket scenario. This solves the lack of quantity and quality of large-scale multimodal instruction fine-tuning data centered on text for tickets, and overcomes the high time and labor costs of obtaining large-scale image and text understanding question and answer data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a flowchart of a method for generating sequential text bill image question-answering data based on a multimodal large model provided by an embodiment of the present invention;

[0053] Figure 2 It is a flowchart of a method for generating sequential text bill image question-answering data based on a multimodal large model provided by an embodiment of the present invention;

[0054] Figure 3 It is a flowchart of a method for generating sequential text bill image question-answering data based on a multimodal large model provided by an embodiment of the present invention;

[0055] Figure 4 It is a schematic diagram of a method for generating sequential text bill image question-answering data based on a multimodal large model provided by an embodiment of the present invention;

[0056] Figure 5 It is a schematic diagram of the architecture of a device for generating sequence text bill image question and answer data based on a multimodal large model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0059] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0060] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0061] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0062] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] Embodiment 1:

[0064] In order to alleviate the current situation of insufficient quantity and low quality of fine-tuning data for text-oriented receipt image question-answering instructions. The present invention proposes an innovative solution, specifically: using the existing multimodal large model and the public receipt (i.e., serial text receipt) data set, an automatic data generation pipeline is designed, and a series of carefully designed instructions are used to complete the information collation, self-question-answering, self-reasoning, and self-screening processes of the annotation of images in the receipt data set, so as to realize the automatic generation of high-quality and large-scale receipt image text information question-answering data. Example 1 of the present invention provides a method for generating serial text receipt image question-answering data based on a multimodal large model, such as Figure 1 As shown, including:

[0065] In step 201, the public receipt dataset is used as the image data to generate the corresponding text comprehension question and answer data, and the annotations of the dataset are sorted to obtain the information of the image data.

[0066] In step 202, the target image content is described in detail using the multimodal big model and the information of the image data collated in step 201; wherein the multimodal big model is a closed-source multimodal big model and a corresponding API interface, or the multimodal big model is an open-source multimodal big model.

[0067] In step 203, the information of the picture data obtained in step 201 is used to determine the number of questions related to the text information generated for a picture and the corresponding answers.

[0068] In step 204, using the information of the picture data obtained in step 201 and the image content description obtained in step 202, the multimodal large model is allowed to perform self-questioning and answering on the text information, generating questions about the text information and corresponding answers; wherein the number of generated questions and answers is equal to the number of generated questions about the text information and corresponding answers determined in step 202.

[0069] In step 205, using the information of the image data obtained in step 201 and the image content description obtained in step 202, the multimodal large model is made to give a second answer to the question raised in step 204 and at the same time infer the basis or reason for the answer.

[0070] In step 206, based on the information of the image data organized in step 201 and the questions and answers generated in step 204, the multimodal large model determines whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid.

[0071] In step 207, the question and answer pairs are checked for consistency using the information of the image data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer to the question and reason generated in step 205; if the two answers corresponding to the same question are inconsistent, the question and answer pair is marked as invalid.

[0072] In step 208, invalid question-answer pairs are deleted based on the question-answer pair tags obtained in steps 206 and 207.

[0073] This embodiment uses a multimodal large model to achieve large-scale batch automatic generation of image text information question and answer data in the ticket scenario. This solves the lack of quantity and quality of large-scale multimodal instruction fine-tuning data centered on text for tickets, and overcomes the high time and labor costs of obtaining large-scale image and text understanding question and answer data annotation.

[0074] The use of the public receipt dataset as image data to generate corresponding text comprehension question and answer data, and the information of the image data obtained by sorting out the annotations of the dataset, specifically includes:

[0075] The annotations that come with the dataset are sorted out to obtain information about the image data, and the key information in the annotations is extracted and simplified; In order to sort out the annotations that come with the dataset, obtain information about the image data, extract and simplify the key information in the annotations, carry as much information as possible with as few words as possible, and provide reliable text information for the multimodal large model's self-questioning and answering of images, which effectively improves the multimodal large model's ability to understand image content and the accuracy of the generated question and answer pairs about text information. For the input image I, the annotation information of the image in the corresponding dataset is L, and the sorted image information is Info=extract(L). Among them, the specific extracted information includes one or more of the text content, the location of the text content, the type of information represented by the text content, and the brief information of the receipt.

[0076] The position of the text content is represented by the coordinates of the lower left corner and the lower right corner of the text content. The types of information represented by the text content include store name, store address, product name, unit price of product and total price of product, etc. The brief information of the receipt includes store name, store address and amount spent. The specific saving format is that for the text content Text, the text content position Coordinate and the type of information Category represented by the text content are saved together as a tuple (Text, Category, Corrdinate), and the brief information of the receipt is saved as a separate dictionary Info_dict: {"store name":,"store address":,"total price of item":,}.

[0077] In one embodiment, the target image content is described in detail using the multimodal large model and the information of the image data obtained in step 201, such as Figure 2 As shown, specifically including:

[0078] In step 301, a multimodal large model is used to generate a detailed description of the target image, and the instruction prompts in the detailed description generation stage emphasize the text information in the image, while controlling the length of the total prompt instruction so that the generated detailed description is between 70 and 100 words; this is to strike a balance between controlling the length of the total prompt instruction and ensuring the detailed description of the image content. Considering the randomness of the large model and the requirements of the diversity of the training data, the instruction prompts in the detailed description generation stage are randomly selected from three instructions with the same semantics but different expressions.

[0079] In step 302, the generated question-answer pairs are extracted from the answers of the multimodal large model by using a pre-set regular expression matching method. If the match fails, the regular expression is replaced to continue matching. If all failures occur, it means that the self-question-answer output does not meet the format requirements, then another instruction prompt is used. If the output of the three instruction prompts does not meet the format requirements, then the image is skipped; let the input image be I, the predefined instruction prompt be p1, and the multimodal large model used be M, then d=M(I,p1) is described in detail. This design effectively improves the success rate of model generation and the diversity of the data obtained. This idea and method are implemented in all steps involving the generation of the large model in the entire process. In the subsequent steps, the number, design method, and usage of the prompt instructions are the same as in this step, so the design idea and usage of the prompt instructions will not be explained in detail in each step.

[0080] The instruction prompts for generating the detailed description stage (i.e., step 202) are generated based on the following template: "You are an image content analysis expert for receipt images; input text information of the receipt image: (Text, Category, Corrdinate); brief information: Info_dict, please generate a brief but accurate description of the input receipt image based on the provided image information. The length of the description is about 70 to 100 words. Please pay special attention to the text information in the image."

[0081] In an optional implementation, the information of the picture data obtained in step 201 is used to determine the number of questions about text information generated for a picture and the corresponding answers, specifically including:

[0082] In order to balance the number of questions and answers about text information generated for a picture, whether the questions and answers are relevant to the text in the image, and the accuracy of the generated question-answer pairs, we use the information of the image data obtained by sorting the annotations of the dataset in step 201 to limit the number of question-answer pairs generated by the multimodal large model for a specific image. For an input image I, the annotation information of the image in the corresponding dataset is L, and the sorted image information is Info. Then the number of question-answer pairs generated for the input image I is Q num = Num(Info); where Num(Info) is a preset function obtained by a technician in this field based on demand analysis. For different receipt data sets, the annotations of the data sets may be different, and the sorted image information may also be different. Through this design, as many question-answer pairs as possible are generated while effectively improving the relevance of the question-answer pairs generated by the model to the text and the accuracy of the question-answer pairs.

[0083] In the actual application scenario, the information of the picture data obtained in step 201 and the image content description obtained in step 202 are used to enable the multimodal large model to perform self-questioning and answering of the text information, and generate questions about the text information and corresponding answers, such as Figure 3 As shown, specifically including:

[0084] In step 401, the self-questioning and answering stage will generate several question-answer pairs, each question should be clear and meaningful, and related to the text information in the image, each question corresponds to an answer, and the answer to each question should be a few short words or phrases related to the text in the image; the instruction prompt used by the multimodal large model to self-question and answer the text information is obtained by combining a predefined prompt trunk p2, a detailed description d of the image, and information Info of the image data sorted out from the annotation; the predefined prompt trunk part is randomly selected from three instructions with the same semantics but different expressions; in the instruction prompt of the self-questioning and answering stage (i.e., step 204), it is required to output the question-answer pairs one by one in the format of "Question: [Your first question] Answer: [Answer according to the text in the picture]".

[0085] In step 402, the result generated by the multimodal large model is matched with a preset regular expression to extract the question Q and the corresponding answer A. If the number of extracted questions or answers is 0, or the number of questions is not equal to the number of answers, it means that the match is wrong, and the regular expression is replaced to continue matching.

[0086] In step 403, all failures mean that the output format of the multimodal large model in this self-question and answering does not meet the requirements, and another instruction prompt is changed. If the output formats of all three instruction prompts do not meet the requirements, the generation of the question and answer pair for the image fails and the image is skipped.

[0087] In a preferred embodiment, the information of the picture data obtained in step 201 and the image content description obtained in step 202 are used to make the multimodal large model give a second answer to the question raised in step 204 and at the same time infer the basis or reason for generating the answer, specifically including:

[0088] The questions generated in step 204 are answered in the form of a chain of thoughts, and a reasoning basis and a new answer are generated for each question; that is, the secondary question-answering and reasoning generation stage will answer the questions generated in step four in the form of a chain of thoughts, and a reasoning basis and a new answer are generated for each question.

[0089] The instruction prompt of the secondary question-answering and reasoning generation stage (i.e., step 205) is obtained by combining the predefined prompt trunk p3, the detailed description d of the image, the information Info of the image data sorted out from the annotation, and the question Q generated in step 204. The predefined prompt trunk part is randomly selected from three instructions with the same semantics but different expressions.

[0090] If a single self-question-answering output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped. The purpose of this step is to first generate reasoning information data for the answer to the question and answer pair, which not only helps to expand the training data, but also can use the thought chain feature to generate higher quality answers. In addition, the reasoning results and secondary answers generated in this stage will be used for consistency detection in subsequent stages to evaluate the quality of the question and answer pair.

[0091] Among them, in order to control the output format of the model, the instruction prompts in the secondary question-answering and reasoning generation stage require the output of the generated thinking chain in the format of "question: [first question] answer: [answer corresponding to the first question] basis: [reasoning basis corresponding to the first question]". Then, regular expressions are used to extract the information in the output one by one and save them as two lists, reasoning reason Reason and secondary answer A'.

[0092] In one embodiment, based on the information of the image data collated in step 201 and the questions and answers generated in step 204, the multimodal large model is enabled to determine whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid, specifically including:

[0093] To determine whether the question-answer pair is related to the text information, the multimodal large model will give a binary judgment of "yes" or "no" for each question-answer pair. The instruction prompt used by the multimodal large model to determine whether the question-answer pair is related to the text information is obtained by combining the predefined prompt trunk p4 and the information Info of the image data sorted out from the annotation with the questions Q and A generated in step 204.

[0094] For questions that are judged as "no", discard the question and answer pair and keep the questions that are judged as "yes".

[0095] Among them, in order to control the output format of the model, the instruction prompt in the judgment stage (i.e., step 206) requires the output of the generated thinking chain one by one in the format of "Judgment: [the judgment corresponding to the i-th question and answer]", and then use regular expressions to extract the information in the output one by one and save it as a list Judge.

[0096] In an optional implementation, the information of the picture data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer and reason to the question generated in step 205 are used to perform consistency check on the question and answer pair, specifically including:

[0097] The multimodal large model determines whether the initial answer generated in step 203 and the secondary answer generated in step 204 express the same meaning based on the two different context prefixes and the reasoning basis generated in step 204. The multimodal large model will give a binary judgment of "yes" or "no" for each question-answer pair.

[0098] The instruction prompt used for consistency check of the question-answer pair consists of a predefined prompt trunk p5, the question Q generated in step 203 and step 204, the answer A, the secondary answer A' generated in step 205 and the reasoning basis Reason.

[0099] Delete any questions that were answered as “No”.

[0100] The purpose of consistency check is to delete the question-answer pairs that are confusing, incorrect, or have a low grasp of the multimodal model. Most of the answers to these question-answer pairs cannot accurately answer the questions. These question-answer pairs reduce the overall quality of the generated question-answer pair dataset, introduce noise to the model training, and affect the performance of the model. In order to control the output format of the model, the instruction prompt in the consistency check phase (i.e., step 207) requires the output to be “Judge:<True or False> " format to output the generated thought chain one by one, and then use regular expressions to extract the information in the output one by one and save it as a consistency judgment list Judge; where Judge i ={j i,k ∈{True,False}|k=1,2,...,Q num}.

[0101] Step 202 and step 204 to step 207 in this embodiment can be understood as: obtaining the results of the corresponding stages by inputting corresponding instruction prompts into the multimodal large model.

[0102] Embodiment 2:

[0103] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0104] The present invention provides a method for generating image text understanding question and answer data for receipt images based on a multimodal large model, the purpose of which is to use a multimodal large model to achieve large-scale batch automatic generation of image text information question and answer data in the receipt scenario. This solves the shortage of large-scale multimodal instruction fine-tuning data centered on text for receipts in terms of quantity and quality, and overcomes the high time and labor costs of obtaining large-scale image and text understanding question and answer data annotation.

[0105] like Figure 4As shown, this embodiment proposes a method for generating question-answering data for receipt image text understanding based on a multimodal large model, including the following implementation steps:

[0106] Step 1: Data preprocessing: Use the public receipt dataset as image data to generate the corresponding text comprehension question and answer data. Sort out the annotations that come with the dataset to get the information of the image data.

[0107] Specifically: In step 1, the annotations of the dataset are sorted to obtain information about the image data, and the key information in the annotations is extracted and simplified to carry as much information as possible with as few words as possible, providing reliable text information for the multimodal large model's self-question and answer of images, effectively improving the multimodal large model's ability to understand image content and the accuracy of the generated question and answer pairs about text information. The specific extracted information may include text content, text content location, the type of information represented by the text content, and brief information on the receipt. The text content location is represented by the lower left corner coordinates and the lower right corner coordinates of the text content. The types of information represented by the text content include store name, store address, product name, unit price of product, total price of product, etc.; the brief information on the receipt includes brief information such as store name, store address, and amount spent. The specific saving format is that the text content Text, the text content location Coordinate and the information category represented by the text content Category are saved together as a tuple (Text, Category, Corrdinate), and the brief information of the receipt is saved as a separate dictionary Info_dict: {"Store name": "Store address": "Total price of items":}.

[0108] Step 2: Use the multimodal large model and the information of the image data obtained from the annotation in step 1 to describe the content of the target image in detail, paying special attention to the text information in the target image.

[0109] Specifically: In step 2, a multimodal large model is used to generate a detailed description of the target image, and the text information in the image is emphasized in the instruction prompt. In order to maintain the detailed description of the image content and control the length of the total prompt instruction, the generated detailed description is about 70 to 100 words. A reference instruction prompt is as follows:

[0110] "You are an expert in image content analysis for receipt images. The text information of the input receipt image is: (Text, Category, Corrdinate); brief information: Info_dict. Please combine the provided image information to generate a short but accurate description of the input receipt image. The length of the description is about 70 to 100 words. Please pay special attention to the text information in the image."

[0111] Using instruction prompts similar to the above-mentioned reference instructions can enable the multimodal large model to make full use of the annotation information of the receipt dataset, improve the multimodal large model's ability to understand the receipt images, and generate more detailed and accurate image descriptions. The image description of this step will be used as auxiliary information, just like the image information obtained by organizing the image annotations in step 1, and integrated into the instruction prompts of subsequent steps, to help the multimodal large model further understand the image, improve the performance of the multimodal large model in subsequent steps, and help the multimodal large model generate high-quality and accurate question-answer pairs.

[0112] Step 3: Using the information of the image data obtained in step 1, determine the number of questions related to the text information generated for a picture and the corresponding answers.

[0113] Specifically: In step 3, we determine the number of question-answer pairs we generate later based on the image information obtained in step 1. Determining the appropriate number of question-answer pairs is conducive to improving the accuracy of the question-answer pairs generated by the multimodal large model and the correlation between the question-answer pairs and the text information in the image. The number of question-answer pairs we control to be generated is related to the number of texts in the image. For a receipt image I, the text content Text in the image information obtained from the corresponding dataset label determines the number of questions about the text information generated for this image and the corresponding answers Num = F (len (Text)), where len (Text) refers to the number of texts, and F is a function with the number of texts as a variable. The specific form of the function varies depending on the annotation of the dataset.

[0114] Step 4: Using the information of the image data obtained in step 1 and the image content description obtained in step 2, the multimodal large model is allowed to perform self-questioning and answering of the text information, generating questions about the text information and corresponding answers, the number of which is equal to the number of questions about the text information and corresponding answers generated in step 2.

[0115] Specifically: Step 4 is mainly responsible for generating self-questioning and answering data based on image information. The self-questioning and answering stage will generate a number of question-answer pairs based on the amount of text information in the image. Each question should be clear and meaningful and related to the text information in the image. Each question corresponds to an answer, and the answer to each question should be a few short words or phrases related to the text in the image; the instruction prompts at this stage are composed of a predefined prompt trunk p2, a detailed description of the image d, and the information Info of the image data obtained from the annotation; the predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions.

[0116] A reference instruction prompt is as follows:

[0117] "This is a receipt image. Here is the text information in the image in the format of (text content, text type, (text upper left corner position coordinates, text lower right corner position coordinates)): (Text, Category, Corrdinate); as well as basic information about the image<Info_dict> and detailed description <description>,generate num > different and detailed questions and their corresponding answers. The questions should be as detailed as possible, and the corresponding answers should be as concise and comprehensive as possible, and each question and answer pair must be related to the text in the picture.

[0118] Each question and answer pair should follow the following format:

[0119] Question: [Your first question] Answer: [Answer based on the text in the picture]

[0120] Question: [Your second question] Answer: [Answer based on the text in the picture] ......

[0122] num >.Question: [Your num >question]Answer: [Answer based on the text in the picture]

[0123] For each question use '1.2.3.4... num >'Such a number. "

[0124] Using command prompts similar to the above reference commands, the multimodal large model can generate a series of question-answer pairs that focus on the text content of the image. While ensuring the number of generated question-answer pairs, it can also ensure high-quality output. At the same time, strict format requirements strictly control the number of question-answer pairs generated by the multimodal large model, and make the output format of the multimodal large model more standardized and easier to match using regular expressions.

[0125] Step 5: Using the information of the image data obtained in step 1 and the image content description obtained in step 2, the multimodal large model is used to give a second answer to the question raised in step 4 and generate the basis or reason for the answer at the same time.

[0126] Specifically: The main function of step five is to generate the reasoning basis for the question-answer pair and the secondary answer required in the subsequent consistency detection stage. The secondary question-answer and reasoning generation stage will answer the questions generated in step four in the form of a thought chain, generating a reasoning basis and a new answer for each question. The instruction prompts in this stage are composed of a predefined prompt trunk p2, a detailed description of the image d, information Info of the image data sorted out from the annotation, and the question Q generated in the previous stage; the predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions.

[0127] ​​​​"This is a receipt image. Here is the text information in the image in the format of (text content, text type, (text upper left corner position coordinates, text lower right corner position coordinates)): (Text, Category, Corrdinate); and basic information about the image<Info_dict> and detailed description <description>, answer the following questions. The answers to the questions must be related to the text information. Please think step by step, first give a short reasoning, explain the basis for answering the question, and then give a specific answer. The question to be answered is <q>

[0128] The output should match the following format:

[0129] Question: [First question] Answer: [Answer to the first question] Basis: [Reasoning to the first question]

[0130] Question: [Second question] Answer: [Answer to the second question] Basis: [Reasoning to the second question] ......

[0132] num >. Question: [No. num >Question] Answer: [Question num >The answer to the question] Based on: [ num >The reasoning basis for each question]

[0133] For each question use '1.2.3.4... num >'Such a number. "

[0134] Using instruction prompts similar to the above reference instructions can enable the multimodal large model to answer questions twice in the form of thought chains. The thought chain answering process can not only improve the accuracy of the model's answer to questions, but also generate relevant reasoning evidence. The reasoning evidence can not only enrich the information of the data set, but also provide relevant information for the subsequent consistency verification steps.

[0135] Step 6: Based on the information of the image data organized in step 1 and the questions and answers generated in step 4, the multimodal large model determines whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid.

[0136] Specifically: The main function of step six is ​​to determine whether the generated question-answer pair is related to the text in the image. The question-answer pair data generated in step four can be screened based on the generated judgment. In actual operation, it is observed that the question-answer pair data in step four may contain some irrelevant information that is not related to the text in the image. The content of these question-answer pairs is not the text information in the image, but some other visual information. Therefore, this type of problem needs to be screened to a certain extent to avoid adversely affecting the quality of the question-answer dataset generated by the multimodal large model. In the stage of judging whether the question-answer pair is related to the text information in the image, the multimodal large model will give a binary judgment of "yes" or "no" for each question-answer pair. The instruction prompt at this stage is obtained by combining the predefined prompt trunk p4, the information Info of the image data sorted from the annotation, and the question-answer pair Q&A generated in step four.

[0137] A reference instruction prompt is as follows:​​​​​

[0138] "This is a receipt image. Here is the text information in the image in the format of (text content, text type, (text upper left corner position coordinates, text lower right corner position coordinates)): (Text, Category, Corrdinate); and basic information about the image<Info_dict> , now here are some questions about this image and their answers. The question is <q>, the corresponding answer is As a data verification expert, please judge whether these question-answer pairs are related to the text in the image. Output "yes" or "no" judgments for these questions in turn. The output should conform to the following format:

[0139] Judgment: [Judgment corresponding to the first question and answer]

[0140] Judgment: [Judgment corresponding to the second question and answer] ......

[0142] num >.Judgement: [No. num >The judgment corresponding to each question and answer]

[0143] For each judgment use '1.2.3.4... num >'Such a number. "

[0144] Using command prompts similar to the reference commands above, the multimodal large model can output judgments on a series of question-answer pairs. The judgments generated in this stage will work together with the judgments generated in the consistency detection stage to screen the question-answer pairs.

[0145] Step 7: Use the information of the image data obtained in step 1, the image content description obtained in step 2, the question and answer generated in step 4, and the secondary answer and reason to the question generated in step 5 to perform consistency check on the question and answer pair. If the answers to the same question are inconsistent, the question and answer pair will be marked as invalid.

[0146] Specifically: The main function of step seven is to perform consistency check on the question-answer pair data. The results generated by the multimodal large model may be diverse, that is, for the same question, the model may generate multiple different answers. Through consistency checking, incorrect question-answer pairs can be screened out, thereby improving the accuracy and consistency of the answers and improving the quality of the generated question-answer pair data. In the consistency detection stage, the multimodal large model determines whether the initial answer generated in step four and the secondary answer generated in step five express the same meaning based on the reasoning basis generated in step five. The multimodal large model will give a binary judgment of "yes" or "no" for each question-answer pair. The instruction prompts at this stage consist of a predefined prompt trunk, the questions generated in steps four and five, two sets of answers and reasoning basis.

[0147] A reference instruction prompt is as follows:

[0148] "You are an expert in image content analysis. Below are some questions about a receipt image and two sets of answers. You need to judge whether the two sets of answers are consistent and correct based on the content of the image and the reasoning given. If the content of the first set of answers is different from the second set of answers, the output judgment is "no". The question to be judged is​​​ <q>, the reasoning used to answer the question is <reason> , the first set of answers is< / reason> < / q> , the second set of answers is Output "yes" or "no" for each of these questions. The output should be in the following format:

[0149] Judgment: [Judgment corresponding to the first question and answer]

[0150] Judgment: [Judgment corresponding to the second question and answer] ......

[0152] num >.Judgement: [No. num >The judgment corresponding to each question and answer]

[0153] For each judgment use '1.2.3.4... num >'Such a number. "

[0154] Using command prompts similar to the above reference commands, the multimodal large model can output judgments on a series of question-answer pairs. The judgments at this stage are used together with the judgments at the previous stage to perform the final screening operation.

[0155] Step 8: Delete invalid question-answer pairs based on the question-answer pair tags obtained in steps 6 and 7.

[0156] Specifically: Step 8 uses the judgments obtained in steps 6 and 7 to screen the question-answer pairs. The text-related judgment and consistency check judgment of each question-answer pair are summed to obtain the judgment result of the question-answer pair. According to the final judgment result, the question-answer pairs judged as "no" are deleted, and the question-answer pairs judged as "yes" are retained.

[0157] The present invention uses a multimodal large model to automatically generate large-scale, high-quality text-centric image-text question-and-answer data for receipts, thereby making up for the current situation in the academic community where the text-centric receipt image question-and-answer instruction fine-tuning data is insufficient and of low quality. The present invention designs an automated data generation pipeline, and the entire process of data generation does not require manual annotation, saving a lot of manpower and time costs. The present invention implements a method for generating image-text comprehension question-and-answer data that mainly processes receipt images, and its workflow and instruction prompts can process various types of receipt images. At the same time, it has the potential to be extended to more image types, such as natural scene images, mobile phone screenshots, table images, chart images, and document images.

[0158] Embodiment 3:

[0159] like​​​ Figure 5 , is a schematic diagram of the architecture of a device for generating sequential text bill image question and answer data based on a multimodal large model according to an embodiment of the present invention. The device for generating sequential text bill image question and answer data based on a multimodal large model according to this embodiment includes one or more processors 21 and a memory 22. Figure 5 A processor 21 is taken as an example.

[0160] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0161] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for generating sequential text bill image question and answer data based on a multimodal large model in Example 1. The processor 21 executes the method for generating sequential text bill image question and answer data based on a multimodal large model by running the non-volatile software programs and instructions stored in the memory 22.

[0162] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0163] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the method for generating sequential text bill image question and answer data based on a multimodal large model in the above-mentioned embodiment 1 is executed.

[0164] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0165] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.< / q> < / q> < / description> < / description>

Claims

1. A method for generating sequential text bill image question-answering data based on a multimodal large model, characterized in that: include: In step 201, the public receipt dataset is used as image data to generate corresponding text comprehension question and answer data, and the annotations of the dataset are sorted to obtain information of the image data; In step 202, the target image content is described in detail using the multimodal large model and the information of the image data obtained in step 201; In step 203, using the information of the picture data obtained in step 201, determine the number of questions related to the text information generated for a picture and the corresponding answers; In step 204, using the information of the picture data obtained in step 201 and the image content description obtained in step 202, the multimodal large model is made to perform self-questioning and answering on the text information, and generate questions related to the text information and corresponding answers; wherein the number of generated questions and answers is equal to the number of questions related to the text information and corresponding answers determined in step 202; In step 205, using the information of the image data obtained in step 201 and the image content description obtained in step 202, the multimodal large model is made to give a second answer to the question raised in step 204 and at the same time infer the basis or reason for generating the answer; In step 206, based on the information of the image data sorted in step 201 and the questions and answers generated in step 204, the multimodal large model is used to determine whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid. In step 207, the consistency check is performed on the question-answer pair using the information of the image data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer and reason to the question generated in step 205; if the two answers corresponding to the same question are inconsistent, the question-answer pair is marked as invalid; In step 208, invalid question-answer pairs are deleted based on the question-answer pair tags obtained in steps 206 and 207.

2. The method for generating sequential text bill image question-answering data based on a multimodal large model according to claim 1 is characterized in that: The use of the public receipt dataset as image data to generate corresponding text comprehension question and answer data, and the information of the image data obtained by sorting out the annotations of the dataset, specifically includes: Arrange the annotations of the dataset to obtain information of the image data, extract key information from the annotations and simplify them; the specific extracted information includes one or more of the text content, the location of the text content, the type of information represented by the text content, and the brief information of the receipt; The position of the text content is represented by the coordinates of the lower left corner and the lower right corner of the text content. The types of information represented by the text content include store name, store address, product name, unit price of product and total price of product. The brief information of the receipt includes store name, store address and amount spent. The specific saving format is that for the text content Text, the text content position Coordinate and the type of information Category represented by the text content are saved together as a tuple (Text, Category, Corrdinate). The brief information of the receipt is saved as a separate dictionary Info_dict: {"store name":,"store address":,"total price":,"}.

3. The method for generating sequential text bill image question-answering data based on a multimodal large model according to claim 2 is characterized in that: The use of the multimodal large model and the information of the image data obtained in step 201 to describe the target image content in detail specifically includes: Use a multimodal large model to generate a detailed description of the target image, and emphasize the text information in the image in the instruction prompts during the detailed description generation stage. At the same time, control the length of the total prompt instructions so that the generated detailed description is between 70 and 100 words. The instruction prompts in the detailed description generation stage are randomly selected from three instructions with the same semantics but different expressions. The generated question and answer pairs are extracted from the answers of the multimodal large model using a preset regular expression matching method. If the match fails, the regular expression is replaced to continue matching. If all failures occur, it means that the self-question and answer output does not meet the format requirements, then another instruction prompt is used. If the outputs of all three instruction prompts do not meet the format requirements, the image is skipped.

4. The method for generating sequential text bill image question-answering data based on a multimodal large model according to claim 1 is characterized in that: The method of using the information of the picture data obtained in step 201 to determine the number of questions related to the text information generated for a picture and the number of corresponding answers specifically includes: For the input image I, the corresponding image annotation information in the dataset is L, and the sorted image information is Info. Then the number of question-answer pairs generated for the input image I is Q num =Num(Info); wherein Num(Info) is a preset function.

5. The method for generating sequential text bill image question-answering data based on a multimodal large model according to claim 1 is characterized in that: The information of the picture data obtained in step 201 and the image content description obtained in step 202 are used to enable the multimodal large model to perform self-questioning and answering of the text information, and generate questions about the text information and corresponding answers, specifically including: The instruction prompt used by the multimodal large model to perform self-questioning and answering of text information is obtained by combining a predefined prompt trunk p2, a detailed description d of the image, and information Info of the image data obtained from the annotation; the predefined prompt trunk part is randomly selected from three instructions with the same semantics but different expressions; In the self-questioning and answering stage, the instruction prompt requires that the question and answer pairs be output one by one in the format of "Question: [Your first question] Answer: [Answer according to the text in the picture]"; The results generated by the multimodal large model are matched with a preset regular expression to extract the question Q and the corresponding answer A. If the number of questions or answers extracted is 0, or the number of questions is not equal to the number of answers, it means that the match is wrong, and the regular expression is changed to continue matching; If all three fail, it means that the output format of the multimodal large model in this self-question and answering does not meet the requirements. Change to another instruction prompt. If the output formats of all three instruction prompts do not meet the requirements, the generation of question and answer pairs for the image fails and the image is skipped.

6. The method for generating sequential text bill image question-answering data based on a multimodal large model according to claim 1 is characterized in that: The information of the picture data obtained in step 201 and the image content description obtained in step 202 are used to make the multimodal large model give a second answer to the question raised in step 204 and at the same time infer the basis or reason for generating the answer, specifically including: Answer the questions generated in step 204 in the form of a thought chain, generating a reasoning basis and a new answer for each question; The instruction prompts in the secondary question answering and reasoning generation stage are obtained by combining the predefined prompt trunk p3, the detailed description d of the image, the information Info of the image data sorted out from the annotation, and the question Q generated in step 204. The predefined prompt trunk is randomly selected from three instructions with the same semantics but different expressions; If a single self-question and answer output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped.

7. The method for generating sequential text bill image question-answering data based on a multimodal large model according to claim 6 is characterized in that: The instruction prompts in the secondary question-answering and reasoning generation stage require the output of the generated thought chain in the format of "question: [first question] answer: [answer corresponding to the first question] basis: [reasoning basis corresponding to the first question]". Then, regular expressions are used to extract the information in the output one by one and save them as two lists, reasoning reason Reason and secondary answer A'.

8. The method for generating sequential text bill image question-answering data based on a multimodal large model according to claim 1 is characterized in that: Based on the information of the image data collated in step 201 and the questions and answers generated in step 204, the multimodal large model is made to determine whether the question-answer pair is related to the text information. If not, the question-answer pair is marked as invalid, specifically including: The instruction prompt used by the multimodal large model to determine whether the question-answer pair is related to the text information is obtained by combining the predefined prompt trunk p4 and the information Info of the image data obtained from the annotation with the questions Q and A generated in step 204; For questions that are judged as "no", the correct answer is discarded and the questions that are judged as "yes" are retained; Among them, the instruction prompt in the judgment stage requires outputting the generated thinking chain one by one in the format of "judgment: [judgment corresponding to the i-th question and answer]", and then using regular expressions to extract the information in the output one by one and save it as a list Judge.

9. The method for generating sequential text bill image question-answering data based on a multimodal large model according to claim 1, characterized in that: The consistency check of the question-answer pair is performed by utilizing the information of the picture data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer and reason to the question generated in step 205, specifically including: The instruction prompt used for consistency check of the question-answer pair is composed of the predefined prompt trunk p5, the question Q generated in step 203 and step 204, the answer A, the secondary answer A' generated in step 205 and the reasoning basis Reason; Delete the questions that are judged as "no"; Among them, the instruction prompt in the consistency check phase requires the output to be "Judge:<True or False> " format to output the generated thought chain one by one, and then use regular expressions to extract the information in the output one by one and save it as a consistency judgment list Judge; where Judge i ={j i,k ∈{True,False}|k=1,2,...,Q num }.

10. The method for generating sequential text bill image question-answering data based on a multimodal large model according to any one of claims 1 to 9, characterized in that: The multimodal big model is a closed-source multimodal big model and a corresponding API interface, or the multimodal big model is an open-source multimodal big model.

Images