A Method for Generating Ticket Image Question-Answer Data Based on a Multimodal Large Model

Abstract

This invention relates to the field of natural language processing technology, and provides a method for generating sequential text-based image question-and-answer data for receipts based on a multimodal large-scale model. The method includes: providing a detailed description of the target image content; instructing the multimodal large-scale model to perform self-questioning and answering on the text information, generating questions and answers; instructing the multimodal large-scale model to provide a secondary answer to the questions and simultaneously reasoning for the generated answers; instructing the multimodal large-scale model to determine whether the question-and-answer pairs are relevant to the text information and performing consistency checks on the question-and-answer pairs; and deleting irrelevant and inconsistent question-and-answer pairs. This invention uses a multimodal large-scale model to achieve large-scale, automated batch generation of image-text information question-and-answer data in receipt scenarios. This solves the shortcomings in quantity and quality of large-scale multimodal instruction fine-tuning data centered on text for receipts, and overcomes the high time and manpower costs of acquiring and annotating large-scale image-text understanding question-and-answer data.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for generating sequential text ticket image question-and-answer data based on a multimodal large model. Background Technology

[0002] In recent years, multimodal large models have made significant progress in the field of artificial intelligence, demonstrating broad application prospects. A multimodal large model is an AI model capable of processing various types of data, including text, images, and audio, exhibiting advanced intelligence in a variety of tasks. For example, multimodal large models can be used for scene understanding in autonomous driving systems, voice and image command responses in intelligent assistants, generating natural language descriptions for images, translating text and images, and answering questions about given images. These applications greatly benefit from the multimodal model's deep understanding of various data types, especially image data. One factor contributing to the strong image data understanding capability of multimodal large models is their ability to recognize and understand textual information within images, such as road signs, trademarks, and document content.

[0003] However, despite breakthroughs in image content understanding and recognition, multimodal large models still face challenges in processing textual information within images. Current multimodal models are generally less accurate than dedicated optical character recognition systems in extracting textual information from images, and they also lack in understanding the semantics and context of this textual information. To overcome these limitations, researchers are actively developing new training strategies and techniques, one of which is instruction fine-tuning. Instruction fine-tuning involves additional training on a pre-trained model to enable it to better understand and execute given task instructions. This approach typically involves using a carefully designed set of instruction-output pairs as training data, designed to teach the model how to correctly generate the expected output, which can significantly improve the performance of the pre-trained model when performing specific tasks. By training with an appropriate instruction dataset, the model can be better adapted to specific application scenarios, thereby producing more accurate and useful results. Generating high-quality instruction datasets is a key issue in this process.

[0004] However, the quantity and quality of publicly available instruction fine-tuning question-and-answer data for receipt images are currently limited, making this a deficiency in large-scale model instruction fine-tuning question-and-answer datasets and impacting the performance of large models on related tasks. Manually acquiring large-scale, high-quality text-centric instruction fine-tuning question-and-answer data is a challenging and time-consuming task. Manual data collection and annotation are not only costly but also difficult to meet the needs of rapidly evolving multimodal large-scale models.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method for generating sequential text ticket image question and answer data based on a multimodal large model.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for generating sequential text document image question-and-answer data based on a multimodal large model, comprising:

[0009] In step 201, the publicly available receipt dataset is used as image data to generate corresponding text comprehension question-and-answer data, and the information of the image data is obtained by organizing the annotations that come with the dataset.

[0010] In step 202, the content of the target image is described in detail using the information from the multimodal large model and the image data obtained in step 201;

[0011] In step 203, using the information from the image data obtained in step 201, the number of questions about the text information generated for an image and the corresponding number of answers are determined.

[0012] In step 204, using the information from the image data obtained in step 201 and the image content description obtained in step 202, the multimodal large model performs self-questioning and answering on the text information, generating questions and corresponding answers related to the text information. In subsequent embodiments, the questions and corresponding answers related to the text information are also referred to as question-answer pairs. The number of questions and answers generated is equal to the number of questions and corresponding answers related to the text information determined in step 202.

[0013] In step 205, using the information from the image data obtained in step 201 and the image content description obtained in step 202, the multimodal large model is made to provide a second answer to the question raised in step 204 and simultaneously reason about the basis or reason for generating the answer.

[0014] In step 206, based on the information from the image data organized in step 201 and the questions and answers generated in step 204, the multimodal large model determines whether the question-answer pair is related to the text information. If it is not related, the question-answer pair is marked as invalid.

[0015] In step 207, the consistency of the question-answer pair is checked using the information from the image data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer and reasoning generated in step 205. If the two answers to the same question are inconsistent (i.e., the answer generated in step 204 is inconsistent with the answer generated in the secondary answer in step 205), the question-answer pair is marked as invalid.

[0016] In step 208, invalid question-answer pairs are deleted based on the question-answer pair tags obtained in steps 206 and 207.

[0017] Preferably, the step of using publicly available receipt datasets as image data to generate corresponding text comprehension question-and-answer data, and organizing the information from the image data obtained by processing the annotations provided in the dataset, specifically includes:

[0018] The annotations provided in the dataset are organized to obtain information from the image data. Key information in the annotations is extracted and simplified. Specifically, the extracted information includes one or more of the following: text content, text content location, the type of information represented by the text content, and brief information from the receipt.

[0019] The text content is positioned using the coordinates of its bottom left and bottom right corners. The text content represents various information types, including store name, store address, product name, unit price, and total price. The receipt contains brief information such as store name, store address, and amount spent. The specific saving format is as follows: [Format details for text content]. Text content location and the types of information represented by the text content Save them together as a tuple The receipt's brief information is saved as a separate dictionary. :{“Store Name”:,“Store Address”:,“Total Price”:,}.

[0020] Preferably, the step of using the information from the multimodal large model and the image data processed in step 201 to describe the content of the target image in detail specifically includes:

[0021] A multimodal large model is used to generate a detailed description of the target image. In the instruction prompts during the detailed description generation stage, attention is paid to the text information in the image, while the length of the total prompts is controlled so that the generated detailed description is between 70 and 100 words.

[0022] The instruction prompts in the detailed description generation stage are randomly selected from three instructions with the same semantics but different expressions. The generated question-answer pairs are extracted from the answers of the multimodal large model using a pre-set regular expression matching method. If the matching fails, the regular expression is changed and the matching continues. If all fail, it means that the output of this self-question and answer does not meet the format requirements, and another instruction prompt is used. If the output of all three instruction prompts does not meet the format requirements, the image is skipped.

[0023] The instructions for the detailed description generation stage are generated based on the following template: "You are an image content analysis expert for receipt images; the text information of the input receipt image is:" Brief information: Please generate a brief but accurate description of the input receipt image based on the provided image information. The description should be approximately 70 to 100 words long. Please pay special attention to the text information in the image.

[0024] Preferably, the step of using the image data obtained in step 201 to determine the number of questions and corresponding answers related to the text information generated for an image specifically includes:

[0025] For the input image The corresponding image annotation information in the dataset is The processed image information is For the input image Number of generated question-answer pairs ;in, This is a preset function.

[0026] Preferably, the step of using the image data information obtained in step 201 and the image content description obtained in step 202 to enable the multimodal large model to perform self-questioning and answering of text information, generating questions and corresponding answers related to the text information, specifically includes:

[0027] The instruction prompts used by the multimodal large model to perform self-questioning and answering of text information are based on a predefined prompt backbone. Detailed description of the image Information from image data obtained from annotations The predefined main body of the prompt is randomly selected from three instructions that have the same semantics but different expressions.

[0028] During the self-questioning and answering phase, the instructions require the output to be in the format of "Question: [Your first question] Answer: [Answer based on the text in the image]" to output the question and answer pairs one by one.

[0029] Problems are extracted from the results generated by the multimodal large model using pre-defined regular expressions. and the corresponding answer If the number of questions or answers extracted is 0, or the number of questions and answers are not equal, it means that the matching has failed, and the regular expression should be changed to continue matching;

[0030] If all three instructions fail, it means that the output format of the multimodal large model in this self-questioning session does not meet the requirements. Try another instruction. If the output format of all three instructions does not meet the requirements, the generation of question-and-answer pairs for the image will fail, and the image will be skipped.

[0031] Preferably, the step of using the information from the image data obtained in step 201 and the image content description obtained in step 202 to enable the multimodal large model to provide a secondary answer to the question raised in step 204 and simultaneously infer the basis or reasoning for generating the answer specifically includes:

[0032] The questions generated in step 204 are answered in the form of a thought chain, generating a reasoning basis and a new answer for each question;

[0033] The instruction prompts in the secondary question-and-answer and reasoning generation stages are based on a predefined prompt framework. Detailed description of the image Information from image data obtained from annotations And the problem generated in step 204 The predefined prompt core is obtained by combining three semantically identical but differently expressed instructions;

[0034] If a single self-answer output does not meet the format requirements, another instruction prompt will be used. If all three instruction prompts fail to generate, the image will be skipped.

[0035] Preferably, in the instruction prompts of the secondary question-and-answer and reasoning generation stage, the generated thought chain is output one by one in the format of "Question: [First Question] Answer: [Answer to the first question] Basis: [Reasoning basis for the first question]". Then, regular expressions are used to extract the information from the output one by one and save it as two lists of reasoning reasons. and secondary answers .

[0036] Preferably, based on the information from the image data organized in step 201 and the questions and answers generated in step 204, the multimodal large model determines whether the question-answer pair is related to the text information. If it is not related, the question-answer pair is marked as invalid. Specifically, this includes:

[0037] The instructions used by the multimodal large model to determine whether a question-answer pair is relevant to textual information are based on a predefined prompt framework. Information from image data obtained from annotations The problem generated in step 204 and Combined to obtain;

[0038] For questions that are answered "no", discard the correct answer and keep the questions that are answered "yes".

[0039] In the judgment phase, the instructions require the output to be in the format of "judgment: [judgment corresponding to the i-th question and answer]", outputting the generated thought chain one by one. Then, regular expressions are used to extract the information from the output one by one and save it as a list. .

[0040] Preferably, the consistency check of the question-answer pair using the information from the image data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer and reasoning generated in step 205 specifically includes:

[0041] The prompts used to perform consistency checks on question-answer pairs consist of a predefined prompt backbone. The problems generated in steps 203 and 204 ,Answer The secondary answer generated in step 205 and reasoning basis composition;

[0042] Remove the questions that are judged as "no";

[0043] Specifically, the instruction prompts during the consistency check phase require the output to include "Judgment:<True or False> The generated thought chain is output line by line in the format "", and then regular expressions are used to extract the information from the output line by line and save it as a consistency judgment list. ;in, .

[0044] Preferably, the multimodal large model is a closed-source multimodal large model and its corresponding API interface, or the multimodal large model is an open-source multimodal large model.

[0045] Secondly, the present invention also provides a device for generating sequential text document image question-and-answer data based on a multimodal large model, used to implement the method for generating sequential text document image question-and-answer data based on a multimodal large model described in the first aspect, the device comprising:

[0046] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the method for generating sequential text ticket image question-and-answer data based on a multimodal large model as described in the first aspect.

[0047] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the method for generating sequential text ticket image question-and-answer data based on a multimodal large model as described in the first aspect.

[0048] Fourthly, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in memory, performing the method as described in the first aspect.

[0049] Fifthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the method as described in the first aspect.

[0050] This invention utilizes a multimodal large-scale model to achieve large-scale, automated batch generation of image-text information question-and-answer data in receipt scenarios. This addresses the shortcomings in quantity and quality of large-scale, text-centric multimodal instruction fine-tuning data for receipts, and overcomes the high time and manpower costs associated with acquiring and annotating large-scale image-text understanding question-and-answer data. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0052] Figure 1 This is a flowchart illustrating a method for generating sequential text document image question-and-answer data based on a multimodal large model, as provided in an embodiment of the present invention.

[0053] Figure 2 This is a flowchart illustrating a method for generating sequential text document image question-and-answer data based on a multimodal large model, as provided in an embodiment of the present invention.

[0054] Figure 3 This is a flowchart illustrating a method for generating sequential text document image question-and-answer data based on a multimodal large model, as provided in an embodiment of the present invention.

[0055] Figure 4This is a schematic diagram of a method for generating sequential text ticket image question-and-answer data based on a multimodal large model provided in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the architecture of a sequence text ticket image question-and-answer data generation device based on a multimodal large model provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0059] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0060] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0061] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0062] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] Example 1:

[0064] To alleviate the current situation of insufficient quantity and low quality of fine-tuning data for text-based receipt image question-and-answer commands, this invention proposes an innovative solution. Specifically, it utilizes existing multimodal large-scale models and publicly available receipt (i.e., sequential text receipt) datasets to design an automated data generation pipeline. Employing a series of meticulously designed instructions, it completes the processes of information organization, self-questioning, self-reasoning, and self-filtering of image annotations in the receipt dataset. This achieves the automated generation of high-quality and large-scale receipt image text information question-and-answer data. Embodiment 1 of this invention provides a method for generating sequential text receipt image question-and-answer data based on a multimodal large-scale model, such as... Figure 1 As shown, it includes:

[0065] In step 201, the publicly available receipt dataset is used as image data to generate corresponding text comprehension question-and-answer data, and the information of the image data is obtained by organizing the annotations that come with the dataset.

[0066] In step 202, the content of the target image is described in detail using the information from the multimodal large model and the image data obtained in step 201; wherein, the multimodal large model is a closed-source multimodal large model and its corresponding API interface, or the multimodal large model is an open-source multimodal large model.

[0067] In step 203, using the information from the image data obtained in step 201, the number of questions and corresponding answers related to the text information generated for an image are determined.

[0068] In step 204, using the information from the image data obtained in step 201 and the image content description obtained in step 202, the multimodal large model performs self-questioning and answering on the text information, generating questions and corresponding answers related to the text information; wherein, the number of questions and answers generated is equal to the number of questions and corresponding answers related to the text information determined in step 202.

[0069] In step 205, using the information from the image data obtained in step 201 and the image content description obtained in step 202, the multimodal large model is made to provide a second answer to the question raised in step 204 and simultaneously infer the basis or reason for generating the answer.

[0070] In step 206, based on the information from the image data organized in step 201 and the questions and answers generated in step 204, the multimodal large model determines whether the question-answer pair is related to the text information. If it is not related, the question-answer pair is marked as invalid.

[0071] In step 207, the consistency of the question-answer pair is checked using the information from the image data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer and reasoning generated in step 205. If the two answers to the same question are inconsistent, the question-answer pair is marked as invalid.

[0072] In step 208, invalid question-answer pairs are deleted based on the question-answer pair tags obtained in steps 206 and 207.

[0073] This embodiment uses a multimodal large model to achieve large-scale, automated batch generation of image-text information question-and-answer data in the context of receipts. This addresses the shortcomings in quantity and quality of large-scale, text-centric multimodal instruction fine-tuning data for receipts, and overcomes the high time and manpower costs associated with acquiring and annotating large-scale image-text understanding question-and-answer data.

[0074] The process of using publicly available receipt datasets as image data to generate corresponding text comprehension question-and-answer data, and organizing the information from the image data obtained by processing the annotations provided in the dataset, specifically includes:

[0075] The annotations provided in the dataset are organized to obtain information from the image data. Key information in the annotations is extracted and simplified. To convey as much information as possible with as few words as possible, reliable text information is provided for the multimodal large-scale model's self-questioning based on images. This effectively improves the multimodal large-scale model's ability to understand image content and the accuracy of the generated question-and-answer pairs based on relevant text information. For the input image... The corresponding image annotation information in the dataset is The processed image information is The specific information extracted includes one or more of the following: text content, text content location, the type of information represented by the text content, and brief information on the receipt.

[0076] The text content is positioned using the coordinates of its bottom left and bottom right corners. The text content represents various information, including store name, store address, product name, unit price, and total price. The receipt contains brief information such as store name, store address, and amount spent. The specific saving format is as follows: [Format details for text content]. Text content location and the types of information represented by the text content Save them together as a tuple The receipt's brief information is saved as a separate dictionary. :{“Store Name”:,“Store Address”:,“Total Price”:,}.

[0077] In one embodiment, the information from the multimodal large model and the image data processed in step 201 is used to describe the content of the target image in detail, such as... Figure 2 As shown, it specifically includes:

[0078] In step 301, a multimodal large model is used to generate a detailed description of the target image. The instructions during the detailed description generation stage emphasize textual information within the image, while controlling the total length of the instructions to between 70 and 100 words. This aims to strike a balance between controlling the total length of the instructions and ensuring the detail of the image content description. Considering the randomness of the large model and the requirement for diversity in the training data, the instructions during the detailed description generation stage are randomly selected from three semantically identical but differently worded instructions.

[0079] In step 302, the generated question-and-answer pairs are extracted from the answers of the multimodal large model using a pre-set regular expression matching method. If a matching error occurs, the regular expression is changed and matching continues. If all matches fail, it means that the output of this self-answering session does not meet the format requirements, and another instruction prompt is used. If the output of all three instruction prompts does not meet the format requirements, the image is skipped. Let the input image be... The predefined instruction prompt is The multimodal large model used is Then describe in detail This design effectively improves the success rate of model generation and the diversity of the obtained data. This idea and method are implemented in all steps of the entire process involving the generation of large models. The number of prompts, design methods, and usage methods in subsequent steps are the same as in this step. Therefore, the design ideas and usage methods of prompts will not be explained in detail in each step.

[0080] The instruction prompt in the detailed description generation stage (i.e., step 202) is generated based on the following template: "You are an image content analysis expert for receipt images; the text information of the input receipt image is:" Brief information: Please generate a brief but accurate description of the input receipt image based on the provided image information. The description should be approximately 70 to 100 words long. Please pay special attention to the text information in the image.

[0081] In one optional implementation, the step of using the image data obtained in step 201 to determine the number of questions and corresponding answers related to the text information generated for an image specifically includes:

[0082] To balance the number of questions and corresponding answers generated for a given image, the relevance of the questions and answers to the text in the image, and the accuracy of the generated question-answer pairs, we use the information from the image data obtained by organizing the annotations provided in the dataset in step 201 to limit the number of question-answer pairs generated by the multimodal large model for a specific image. For the input image... The corresponding image annotation information in the dataset is The processed image information is For the input image Number of generated question-answer pairs ;in, This is a pre-defined function, derived by those skilled in the art based on requirements. The annotations and resulting image information may differ depending on the specific receipt dataset. This design effectively improves the relevance and accuracy of the generated question-answer pairs to the text, while generating as many question-answer pairs as possible.

[0083] In practical applications, the information from the image data obtained in step 201 and the image content description obtained in step 202 are used to enable the multimodal large model to perform self-questioning and answering of textual information, generating questions and corresponding answers related to the textual information, such as... Figure 3 As shown, it specifically includes:

[0084] In step 401, the self-questioning phase generates several question-answer pairs. Each question should be clear and meaningful, and related to the text information in the image. Each question corresponds to an answer, which should be a few short words or phrases related to the text in the image. The instruction prompts used by the multimodal large model to perform self-questioning on the text information are based on a predefined prompt backbone. Detailed description of the image Information from image data obtained from annotations The predefined prompt core is randomly selected from three instructions with the same semantics but different expressions. In the self-questioning stage (i.e., step 204), the prompt requires the output to be in the format of "Question: [Your first question] Answer: [Answer according to the text in the picture]" to output the question and answer pairs one by one.

[0085] In step 402, the problem is extracted from the results generated by the multimodal large model using a pre-set regular expression matching. and the corresponding answer If the number of questions or answers extracted is 0, or the number of questions and answers are not equal, it means that the matching has failed, and the regular expression should be changed to continue matching.

[0086] In step 403, if all fail, it means that the output format of the multimodal large model in this self-questioning does not meet the requirements. Another instruction prompt is then used. If the output format of all three instruction prompts does not meet the requirements, the generation of question-and-answer pairs for the image fails, and the image is skipped.

[0087] In a preferred embodiment, the step of using the information from the image data obtained in step 201 and the image content description obtained in step 202 to enable the multimodal large model to provide a secondary answer to the question raised in step 204 and simultaneously infer the basis or reasoning for generating the answer specifically includes:

[0088] The questions generated in step 204 are answered in the form of a thought chain, generating a reasoning basis and a new answer for each question; that is, the second question-and-answer and reasoning generation stage will answer the questions generated in step four in the form of a thought chain, generating a reasoning basis and a new answer for each question.

[0089] The instruction prompts in the second question-and-answer and reasoning generation stage (i.e., step 205) are based on a predefined prompt backbone. Detailed description of the image Information from image data obtained from annotations And the problem generated in step 204 The predefined prompt core is obtained by combining three instructions that have the same semantics but different expressions.

[0090] If a single self-answering output does not meet the format requirements, another instruction prompt is used. If all three instruction prompts fail to generate, the image is skipped. The purpose of this step is firstly to generate reasoning information data for the answers to the question-answer pair, which not only helps to expand the training data but also utilizes the characteristics of thought chains to generate higher-quality answers. In addition, the reasoning results and secondary responses generated in this stage will be used for consistency detection in subsequent stages to evaluate the quality of the question-answer pair.

[0091] To control the model's output format, the instructions during the secondary question-and-answer and reasoning generation stages require the output to follow the format "Question: [First Question] Answer: [Answer to the First Question] Basis: [Reasoning Basis for the First Question]". The generated thought chains are then output line by line using regular expressions, and the information is extracted from the output line by line and saved as two separate lists containing reasoning arguments. and secondary answers .

[0092] In one embodiment, based on the information from the image data organized in step 201 and the questions and answers generated in step 204, the multimodal large model determines whether the question-answer pair is related to the text information. If it is not related, the question-answer pair is marked as invalid. Specifically, this includes:

[0093] To determine whether a question-and-answer pair is relevant to the text information, the multimodal large model provides a binary "yes" or "no" answer for each pair. The instructions used by the multimodal large model to determine the relevance of a question-and-answer pair to the text information are based on a predefined prompt framework. Information from image data obtained from annotations The problem generated in step 204 and The combination yields the desired result.

[0094] For questions that are answered "no", discard the correct answer and keep the questions that are answered "yes".

[0095] In order to control the output format of the model, the instruction prompt in the judgment stage (i.e., step 206) requires the output to be in the format of "judgment: [judgment corresponding to the i-th question and answer]" to output the generated thought chain one by one. Then, regular expressions are used to extract the information from the output one by one and save it as a list. .

[0096] In an optional implementation, the consistency check of the question-answer pair using the information from the image data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer and reasoning generated in step 205 specifically includes:

[0097] The multimodal large model, based on two different contextual prefixes and the reasoning basis generated in step 204, determines whether the initial answer generated in step 203 and the secondary answer generated in step 204 express the same meaning. The multimodal large model provides a "yes" or "no" binary judgment for each question-answer pair.

[0098] The prompts used to perform consistency checks on question-answer pairs consist of a predefined prompt backbone. The problems generated in steps 203 and 204 ,Answer The secondary answer generated in step 205 and reasoning basis composition.

[0099] Remove the questions that are judged as "no".

[0100] The purpose of consistency checking is primarily to remove question-answer pairs that are misleading, incorrect, or have low multimodal model confidence. These pairs often fail to accurately answer the questions, lowering the overall quality of the generated question-answer pair dataset and introducing noise into model training, thus impacting model performance. To control the model's output format, the instructions in the consistency check phase (step 207) require the output to begin with "Judgment:<True or False> The generated thought chain is output line by line in the format "", and then regular expressions are used to extract the information from the output line by line and save it as a consistency judgment list. ;in, .

[0101] In this embodiment, steps 202 and 204-207 can be understood as: by inputting corresponding instruction prompts into the multimodal large model, the results of the corresponding stage are obtained.

[0102] Example 2:

[0103] Based on the method described in Embodiment 1, this invention combines specific application scenarios and uses technical descriptions in relevant scenarios to illustrate the implementation process of the features of this invention in those scenarios.

[0104] This invention provides a method for generating image-text understanding question-and-answer data based on a multimodal large-scale model for frontal receipt images. Its purpose is to achieve large-scale, automated batch generation of image-text information question-and-answer data in receipt scenarios using a multimodal large-scale model. This addresses the shortcomings in quantity and quality of large-scale multimodal instruction fine-tuning data centered on text for receipts, and overcomes the high time and manpower costs associated with acquiring and annotating large-scale image-text understanding question-and-answer data.

[0105] like Figure 4 As shown, this embodiment proposes a method for generating question-and-answer data for receipt images based on a multimodal large model, including the following implementation steps:

[0106] Step 1: Data Preprocessing: Use publicly available receipt datasets as image data to generate corresponding text comprehension question-and-answer data. Organize the information from the image data obtained using the built-in annotations in the dataset.

[0107] Specifically: In step one, the annotations provided with the dataset are organized to obtain information from the image data. Key information in the annotations is extracted and simplified to carry as much information as possible with as few words as possible. This provides reliable text information for the multimodal large model's self-questioning based on images, effectively improving the multimodal large model's ability to understand image content and the accuracy of the generated question-and-answer pairs related to text information. The extracted information may include text content, text content location, the type of information represented by the text content, and brief information from the receipt. The text content location is represented by the coordinates of the lower left and lower right corners of the text content. The types of information represented by the text content include store name, store address, product name, unit price, and total price. The brief information from the receipt includes store name, store address, and amount spent. The specific saving format is as follows: for text content... Text content location and the types of information represented by the text content Save them together as a tuple The receipt's brief information is saved as a separate dictionary. :{“Store Name”:,“Store Address”:,“Total Price”:,}.

[0108] Step 2: Use the information from the multimodal large model and the image data obtained from the annotations in Step 1 to provide a detailed description of the target image content, paying particular attention to the text information in the target image.

[0109] Specifically: In step two, a multimodal large model is used to generate a detailed description of the target image, emphasizing textual information within the image in the instruction prompts. To maintain the detail of the image content description while controlling the overall length of the prompts, the generated detailed description is approximately 70 to 100 words. A sample instruction prompt is as follows:

[0110] "You are an image content analysis expert working with receipt images. The text information of the input receipt image is:" Brief information: Please generate a brief but accurate description of the input receipt image, based on the provided image information. The description should be approximately 70 to 100 characters long. Please pay special attention to the text information within the image.

[0111] Using instruction prompts similar to those mentioned above allows the multimodal large model to fully utilize the annotation information of the receipt dataset, improving its understanding of receipt images and generating more detailed and accurate image descriptions. These image descriptions, along with the image information obtained from image annotation in step one, will be used as supplementary information and integrated into the instruction prompts for subsequent steps. This will help the multimodal large model further understand the images, improve its performance in subsequent steps, and help it generate high-quality and accurate question-answer pairs.

[0112] Step 3: Using the information from the image data obtained in Step 1, determine the number of questions and corresponding answers to be generated for a given image.

[0113] Specifically, in step three, we determine the number of question-answer pairs to be generated based on the image information obtained in step one. Determining the appropriate number of question-answer pairs is beneficial for improving the accuracy of the generated pairs from the multimodal large model and their relevance to the text information in the image. We control the desired number of question-answer pairs to be related to the amount of text in the image. For a receipt image... The text content in the image information obtained from the corresponding dataset labels To determine the number of questions and corresponding answers generated related to the text information generated for this image. ,in This refers to the number of texts. It is a function that takes the number of texts as a variable, and the specific form of the function varies depending on the annotation of the dataset.

[0114] Step 4: Using the information from the image data obtained in Step 1 and the image content description obtained in Step 2, the multimodal large model performs self-questioning and answering on the text information, generating questions and corresponding answers related to the text information, the number of which is equal to the number of questions and corresponding answers related to the text information generated in Step 2.

[0115] Specifically: Step four is mainly responsible for generating self-answering data based on image information. During this phase, several question-answer pairs are generated based on the amount of text information in the image. Each question should be clear and meaningful, and related to the text information in the image. Each question corresponds to one answer, which should be a few short words or phrases related to the text in the image. The instructions in this stage are based on a predefined prompt framework. Detailed description of the image Information from image data obtained from annotations The combination yields a predefined main body of instructions, which is randomly selected from three semantically identical but expressively different instructions.

[0116] The following is a sample instruction prompt:

[0117] "This is an image of a receipt. Here is some text information about the image, formatted as (text content, text type, (coordinates of the top left corner of the text, coordinates of the bottom right corner of the text)):" ; and basic information about the image. and detailed description ,generate Each image contains a different and detailed question and its corresponding answer. The questions should be as detailed as possible, and the answers should be as concise and comprehensive as possible. Each question-and-answer pair must be related to the text in the image.

[0118] Each question and answer should follow the format below:

[0119] Question: [Your first question] Answer: [Answer based on the text in the picture]

[0120] Question: [Your second question] Answer: [Answer based on the text in the image] ......

[0121] Question: [Your first...] [Question] Answer: [Answer based on the text in the image]

[0122] For each question, use '1.2.3.4...' 'Such a number.'

[0123] Using instruction prompts similar to those mentioned above, the multimodal large model can generate a series of question-and-answer pairs focusing on image-text content. This ensures both a high quantity of generated question-and-answer pairs and high-quality output. Furthermore, strict formatting requirements tightly control the number of question-and-answer pairs generated by the multimodal large model, while also making the output format more standardized and easier to match using regularization expressions.

[0124] Step 5: Using the information from the image data obtained in Step 1 and the image content description obtained in Step 2, the multimodal large model provides a second answer to the question raised in Step 4 and generates the basis or reason for the answer.

[0125] Specifically, step five primarily serves to generate the reasoning basis for question-and-answer pairs, as well as the secondary answers needed for the subsequent consistency check stage. The secondary question-and-answer and reasoning generation stage addresses the questions generated in step four using a thought chain format, generating a reasoning basis and a new answer for each question. The instructions in this stage are based on a predefined prompt framework. Detailed description of the image Information from image data obtained from annotations And the problems generated in the previous stage The combination yields a predefined main body of instructions, which is randomly selected from three semantically identical but expressively different instructions.

[0126] "This is an image of a receipt. Here is some text information about the image, formatted as (text content, text type, (coordinates of the top left corner of the text, coordinates of the bottom right corner of the text)):" ; and basic information about the image. and detailed description Please answer the following questions. The answers must be related to the textual information. Please think step by step, first providing a brief reasoning to explain the basis for your answer, and then giving a specific answer. The questions to be answered are:

[0127] The output should conform to the following format:

[0128] Question: [The first question] Answer: [The answer to the first question] Basis: [The reasoning behind the first question]

[0129] Question: [The second question] Answer: [The answer to the second question] Basis: [The reasoning behind the second question] ......

[0130] Question: [The first question] [Question #] Answer: [Question #] [Answer to the question] Based on: [the question] The reasoning basis for each question]

[0131] For each question, use '1.2.3.4...' 'Such a number.'

[0132] Using prompts similar to the reference instructions mentioned above allows multimodal large models to provide secondary answers to questions in the form of thought chains. This thought chain-based answering process not only improves the model's accuracy in answering questions but also generates relevant reasoning. This reasoning not only enriches the information in the dataset but also provides relevant information for subsequent consistency verification steps.

[0133] Step Six: Based on the information from the image data organized in Step One, and the questions and answers generated in Step Four, the multimodal large model determines whether the question-answer pair is related to the text information. If it is not related, the question-answer pair is marked as invalid.

[0134] Specifically, the main function of step six is ​​to determine whether the generated question-and-answer pairs are related to the text in the image. This determination will then be used to filter the question-and-answer pair data generated in step four. In practice, it was observed that some of the question-and-answer pairs in step four may contain irrelevant information that is not related to the text in the image. These pairs focus on other visual information rather than the text itself. Therefore, these types of pairs need to be filtered out to avoid adversely affecting the quality of the question-and-answer dataset generated by the multimodal large model. In the stage of determining whether a question-and-answer pair is related to the text information in the image, the multimodal large model will provide a binary "yes" or "no" judgment for each pair. The instruction prompts in this stage are based on a predefined prompt backbone. Information from image data obtained from annotations and the question-answer pairs generated in step four The combination yields the desired result.

[0135] The following is a sample instruction prompt:

[0136] "This is an image of a receipt. Here is some text information about the image, formatted as (text content, text type, (coordinates of the top left corner of the text, coordinates of the bottom right corner of the text)):" ; and basic information about the image. Below are some questions and answers about this image. The questions are... The corresponding answer is As a data validation expert, determine whether these question-and-answer pairs are relevant to the text in the images. Output "Yes" or "No" for each question. The output should conform to the following format:

[0137] Judgment: [Judgment corresponding to the first question and answer]

[0138] Judgment: [The judgment corresponding to the second question and answer] ......

[0139] Judgment: [the first] [Judgment corresponding to each question and answer]

[0140] For each judgment, use '1.2.3.4...' 'Such a number.'

[0141] Using instruction prompts similar to those in the reference instructions described above, the multimodal large model can output judgments on a series of question-answer pairs. The judgments generated in this stage will work together with the judgments generated in the consistency detection stage to filter the question-answer pairs.

[0142] Step 7: Using the information from the image data obtained in Step 1, the image content description obtained in Step 2, the questions and answers generated in Step 4, and the secondary answers and reasons generated in Step 5, perform a consistency check on the question-answer pairs. If the answers to the same question are inconsistent, the question-answer pair is marked as invalid.

[0143] Specifically, step seven primarily serves to perform consistency checks on the question-and-answer pairs. The results generated by the multimodal large model may exhibit diversity; that is, for the same question, the model may generate multiple different answers. Consistency checks can filter out erroneous question-and-answer pairs, thereby improving the accuracy and consistency of the answers and enhancing the quality of the generated question-and-answer pair data. In the consistency detection phase, the multimodal large model, based on the reasoning generated in step five, determines whether the initial answer generated in step four and the secondary answer generated in step five express the same meaning. The multimodal large model provides a "yes" or "no" binary judgment for each question-and-answer pair. The instruction prompts in this stage consist of a predefined prompt framework, the questions generated in steps four and five, two sets of answers, and the reasoning basis.

[0144] The following is a sample instruction prompt:

[0145] "You are an image content analysis expert. Below are some questions and two sets of answers about an image of a receipt. You need to determine whether the two sets of answers are consistent and correct based on the content of the image and the given reasoning. If the content of the first set of answers differs from that of the second set of answers, output 'No'. The questions to be judged are..." The reasoning used to answer the question is The first set of answers is The second set of answers is Output "Yes" or "No" for each of these questions. The output should conform to the following format:

[0146] Judgment: [Judgment corresponding to the first question and answer]

[0147] Judgment: [The judgment corresponding to the second question and answer] ......

[0148] Judgment: [the first] [Judgment corresponding to each question and answer]

[0149] For each judgment, use '1.2.3.4...' 'Such a number.'

[0150] Using instruction prompts similar to those in the reference instructions above, the multimodal large model can output judgments on a series of question-answer pairs. These judgments, along with those from the previous stage, are then used for the final filtering operation.

[0151] Step 8: Based on the question-answer pair tags obtained in Step 6 and Step 7, delete invalid question-answer pairs.

[0152] Specifically: Step eight uses the judgments obtained in steps six and seven to filter question-and-answer pairs. The text-related judgment and consistency check judgment for each question-and-answer pair are ANDed together to obtain the judgment result for that pair. Based on the final judgment result, question-and-answer pairs judged as "no" are deleted, and those judged as "yes" are retained.

[0153] This invention utilizes a multimodal large-scale model to automatically generate large-scale, high-quality text-centric image-text question-and-answer data for receipts, thereby addressing the current academic situation where the amount and quality of text-centric receipt image question-and-answer instruction fine-tuning data are insufficient. This invention designs an automated data generation pipeline; the entire data generation process requires no manual annotation, saving significant manpower and time costs. This invention implements a method for generating image-text comprehension question-and-answer data primarily for receipt images. Its workflow and instruction prompts can handle various types of receipt images. It also has the potential to be extended to more image types, such as natural scene images, mobile phone screenshots, table images, chart images, and document images.

[0154] Example 3:

[0155] like Figure 5 The diagram shown is an architectural schematic of a sequence text document image question-and-answer data generation device based on a multimodal large model according to an embodiment of the present invention. This embodiment of the sequence text document image question-and-answer data generation device includes one or more processors 21 and a memory 22. Wherein, Figure 5 Take a processor 21 as an example.

[0156] Processor 21 and memory 22 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0157] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the method for generating sequential text document image question-and-answer data based on a multimodal large model in Embodiment 1. The processor 21 executes the method for generating sequential text document image question-and-answer data based on a multimodal large model by running the non-volatile software programs and instructions stored in the memory 22.

[0158] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0159] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they execute the sequence text ticket image question-and-answer data generation method based on multimodal large model in Embodiment 1.

[0160] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0161] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating ticket image question-and-answer data based on a multimodal large model, characterized in that, include: In step 201, a publicly available receipt image dataset is obtained, the annotations provided in the dataset are organized, the key information in the annotations is extracted and simplified, and the information of the image data is obtained. In step 202, the content of the target image is described in detail using the information from the multimodal large model and the image data obtained in step 201, thus obtaining an image content description; In step 203, using the information from the image data obtained in step 201, the number of questions about the text information in an image and the corresponding number of answers are determined. In step 204, using the information from the image data obtained in step 201 and the image content description obtained in step 202, the multimodal large model performs self-questioning and answering on the text information in the image, generating questions and corresponding answers about the text information in the image; wherein, the number of questions and answers generated is equal to the number of questions and corresponding answers about the text information in the image determined in step 202. In step 205, using the information from the image data obtained in step 201 and the image content description obtained in step 202, the multimodal large model is made to provide a second answer to the question raised in step 204 and simultaneously reason for the reasoning behind the answer. In step 206, based on the information from the image data organized in step 201 and the questions and answers generated in step 204, the multimodal large model determines whether the question-answer pair is related to the text information in the image. If it is not related, the question-answer pair is marked as invalid. In step 207, the consistency of the question-answer pair is checked using the information from the image data obtained in step 201, the image content description obtained in step 202, the question and answer generated in step 204, and the secondary answer and reasoning generated in step 205. If the two answers to the same question are inconsistent, the question-answer pair is marked as invalid. In step 208, invalid question-answer pairs are deleted based on the question-answer pair tags obtained in steps 206 and 207.

2. The method for generating ticket image question-and-answer data based on a multimodal large model according to claim 1, characterized in that, The process of obtaining the publicly available receipt image dataset involves organizing the annotations provided in the dataset, extracting and simplifying the key information from the annotations to obtain the image data information, specifically including: The annotations provided in the dataset are organized to obtain information from the image data. Key information in the annotations is extracted and simplified. Specifically, the extracted information includes one or more of the following: text content, text content location, the type of information represented by the text content, and brief information from the receipt. The text content is positioned using the coordinates of its bottom left and bottom right corners. The text content represents various information types, including store name, store address, product name, unit price, and total price. The receipt contains brief information such as store name, store address, and amount spent. The specific saving format is as follows: [Format details for text content]. Text content location and the types of information represented by the text content Save them together as a tuple The receipt's brief information is saved as a separate dictionary. :{"Store Name":,"Store Address":,"Total Price of Items":,}.

3. The method for generating ticket image question-and-answer data based on a multimodal large model according to claim 2, characterized in that, The description of the target image content using the information from the multimodal large model and the image data processed in step 201 specifically includes: A multimodal large model is used to generate a detailed description of the target image. In the instruction prompts during the detailed description generation stage, attention is paid to the text information in the image, while the length of the total prompts is controlled so that the generated detailed description is between 70 and 100 words. The instruction prompts in the detailed description generation stage are randomly selected from three semantically identical but differently expressed instructions. The generated question-answer pairs are extracted from the answers of the multimodal large model using a pre-set regular expression matching method. If a matching error occurs, the regular expression is changed and the matching continues. If all fail, it means that the output of this self-question and answer does not meet the format requirements, and another instruction prompt is used. If the output of all three instruction prompts does not meet the format requirements, the image is skipped.

4. The method for generating ticket image question-and-answer data based on a multimodal large model according to claim 1, characterized in that, The step of using the image data obtained in step 201 to determine the number of questions and corresponding answers related to the text information in an image, specifically includes: For the input image The corresponding image annotation information in the dataset is The processed image information is For the input image Number of generated question-answer pairs ;in, This is a preset function.

5. The method for generating ticket image question-and-answer data based on a multimodal large model according to claim 1, characterized in that, The step of using the image data obtained in step 201 and the image content description obtained in step 202 to enable the multimodal large model to perform self-questioning and answering on the text information in the image, generating questions and corresponding answers about the text information in the image, specifically including: The instruction prompts used by the multimodal large model to perform self-questioning and answering of text information in images are based on a predefined prompt backbone. Detailed description of the image Information from image data obtained from annotations The predefined main body of the prompt is randomly selected from three instructions that have the same semantics but different expressions. During the self-answering phase, the instructions require outputting the question-and-answer pairs one by one in the format of "Question: [Your first question] Answer: [Answer based on the text in the image]". Problems are extracted from the results generated by the multimodal large model using pre-defined regular expressions. and the corresponding answer If the number of questions or answers extracted is 0, or the number of questions and answers are not equal, it means that the matching has failed, and the regular expression should be changed to continue matching; If all three instructions fail, it means that the output format of the multimodal large model in this self-questioning session does not meet the requirements. Try another instruction. If the output format of all three instructions does not meet the requirements, the generation of question-and-answer pairs for the image will fail, and the image will be skipped.

6. The method for generating ticket image question-and-answer data based on a multimodal large model according to claim 1, characterized in that, The step of using the information from the image data obtained in step 201 and the image content description obtained in step 202 to enable the multimodal large model to provide a secondary answer to the question raised in step 204 and simultaneously reason for generating the answer includes: The questions generated in step 204 are answered in the form of a thought chain, generating a reasoning basis and a new answer for each question; The instruction prompts in the secondary question-and-answer and reasoning generation stages are based on a predefined prompt framework. Detailed description of the image Information from image data obtained from annotations And the problem generated in step 204 The predefined prompt core is obtained by combining three semantically identical but differently expressed instructions; If a single self-answer output does not meet the format requirements, another instruction prompt will be used. If all three instruction prompts fail to generate, the image will be skipped.

7. The method for generating ticket image question-and-answer data based on a multimodal large model according to claim 6, characterized in that, The instructions in the secondary question-and-answer and reasoning generation stage require the output to be in the format of "Question: [First Question] Answer: [Answer to the First Question] Basis: [Reasoning Basis for the First Question]" to output the generated thought chain one by one. Then, regular expressions are used to extract the information from the output one by one and save it as two lists of reasoning reasons. and secondary answers .

8. The method for generating ticket image question-and-answer data based on a multimodal large model according to claim 1, characterized in that, Based on the information from the image data organized in step 201 and the questions and answers generated in step 204, the multimodal large model determines whether the question-answer pair is related to the text information in the image. If it is not related, the question-answer pair is marked as invalid. Specifically, this includes: The instructions used by the multimodal large model to determine whether a question-answer pair is related to text information in an image are based on a predefined prompt framework. Information from image data obtained from annotations The problem generated in step 204 and Combined to obtain; For questions that are answered "no", discard the correct answer and keep the questions that are answered "yes". In the judgment phase, the instructions require the output to be in the format of "judgment: [judgment corresponding to the i-th question and answer]" to output the generated thought chain one by one. Then, regular expressions are used to extract the information from the output one by one and save it as a list. .

9. The method for generating ticket image question-and-answer data based on a multimodal large model according to claim 1, characterized in that, The process of using the image data obtained in step 201, the image content description obtained in step 202, the questions and answers generated in step 204, and the secondary answers and reasons generated in step 205 to the questions to perform a consistency check on the question-answer pairs specifically includes: The prompts used to perform consistency checks on question-answer pairs consist of a predefined prompt backbone. The problems generated in steps 203 and 204 ,Answer The secondary answer generated in step 205 and reasoning basis composition; Remove the questions that are judged as "no"; Specifically, the instruction prompts during the consistency check phase require the output to include "Judgment:<True or False > The generated thought chain is output line by line in the format "", and then regular expressions are used to extract the information from the output line by line and save it as a consistency judgment list. ;in, .

10. The method for generating ticket image question-and-answer data based on a multimodal large model according to any one of claims 1-9, characterized in that, The multimodal large model is a closed-source multimodal large model and its corresponding API interface, or the multimodal large model is an open-source multimodal large model.

Images