A layout- and task-aware zero-shot document image question answering method with textual hints
Through OCR processing and layout information retention of text prompt method, combined with task description template, the problem that the instruction fine-tuning model in the existing technology cannot directly perform zero-sample document image question answering is solved, which realizes the effective solution of zero-sample document image question answering and improves the accuracy and adaptability of question answering.
Patent Information
- Application Number
- CN202310607370.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2025-09-05
- Estimated Expiration
- 2043-05-26
AI Technical Summary
Existing layout-aware multimodal pre-trained models cannot directly utilize instruction-fine-tuned large language-based models for zero-shot document image question answering, hindering their powerful capabilities in zero-shot learning.
By performing OCR on document images, extracting text fragments and using spaces and line breaks to retain the original layout information, combined with task description templates, and designing a language model with text prompts to input instruction fine-tuning, zero-sample question answering is achieved.
The instruction fine-tuning language model is used to effectively solve the document image question answering task under zero-shot conditions, with performance comparable to the pre-training fine-tuning paradigm, improving the model's question answering accuracy and adaptability.
Smart Images
Figure CN116775822B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the intersection of image understanding and natural language processing, and specifically relates to the application of prompt learning based on instruction fine-tuning of a large language training model in zero-sample document image question answering. Background Art
[0002] Intelligent document image question answering, a key application of document intelligence, aims to develop artificial intelligence systems that can understand documents and automatically answer questions in natural language. Compared to text documents, document images contain textual, visual, and layout information, which poses unique challenges for machine understanding.
[0003] Recently, layout-aware multimodal pre-training models under the pre-training-fine-tuning paradigm have made significant progress in document image question answering. Their core strategy is to introduce additional visual perception modules and layout perception modules on top of the language model, and then learn these modules through pre-training tasks that understand text, visual content, and layout. However, pre-training and fine-tuning on the visual, layout, and task modules prevent these methods from directly using existing instructions to adjust large language-based models, thereby failing to take advantage of these models' strong capabilities in zero-shot learning.
[0004] Therefore, how to enable the instruction-fine-tuned language-based model to directly solve the document image question answering task is a technical problem that needs to be solved urgently. Summary of the Invention
[0005] The purpose of the present invention is to solve the problems existing in the prior art and provide a layout and task-aware text-prompted zero-sample document image question answering method.
[0006] Inspired by the powerful zero-shot learning capabilities of instruction-based fine-tuned large-scale language-based models, this paper proposes layout- and task-aware text prompts, enabling the instruction-based fine-tuned language-based models to be directly applied to document image question answering tasks. By using appropriate spaces and line breaks, the paper ensures that the document content prompt text retains the original document layout information as much as possible. Furthermore, different task description templates are designed for different question answering tasks, enabling the model to generate answers that meet the requirements of document image question answering tasks.
[0007] The technical solutions specifically adopted in the present invention are as follows:
[0008] A layout- and task-aware text-cued zero-shot document image question answering method, whose steps are as follows:
[0009] S1: Perform OCR processing on the document image to extract all text segments contained in the document image and the text box coordinates corresponding to each text segment;
[0010] S2: All text fragments extracted in S1 are judged according to the corresponding text box coordinates to determine their original rows in the document image, and all text fragments are combined into document content prompt text by adding spaces and line breaks. During the combination process, all text fragments are sorted according to their corresponding original rows, and text fragments in the same original row continue to be placed in the same row of the document content prompt text, and line breaks are added between the upper and lower rows. Spaces are added between adjacent text fragments in the same row to ensure that their horizontal spacing is consistent with their original layout in the document image, so that the combined document content prompt text retains the layout information in the original document.
[0011] S3: Read the task description prompt text template designed for the target question-answering task. The template contains a specific task description, a document content prompt placeholder containing layout information, and a specific question placeholder. Replace the corresponding placeholders in the template with the document content prompt text combined in S2 and the specific question entered by the user to obtain the final task description prompt text and input it into the language model after instruction fine-tuning. The final question-answering result is output through model decoding.
[0012] Preferably, in S2, the specific method of combining all text fragments into document content prompt text by adding spaces and line breaks is as follows:
[0013] S21. First, all text segments are arranged in order from top to bottom and from left to right based on the coordinates of the extracted text boxes of each text segment. The text segments in the same original line in the document image and their corresponding text box coordinates are recorded. Then, the sum of the widths of all text boxes in a single line is divided by the total number of characters in the line to obtain the average character width of a single character.
[0014] S22. Then, text segments in the same original line in the document image are grouped according to their respective horizontal spatial order in the document image, and spaces are added between two adjacent text segments to maintain their horizontal layout information, where the number of spaces added is calculated by dividing the minimum horizontal spacing between the text boxes corresponding to the two text segments by the average character width;
[0015] S23. Arrange each line of combined text from top to bottom according to its vertical spatial order in the document image, and separate two adjacent lines of combined text by adding a line break.
[0016] Preferably, in said S21, when calculating the average character width of a single character, first find the row with the largest number of characters, then use this row as a basis to calculate the average character width, and then represent the average character width of all rows in the entire document image.
[0017] Preferably, in S3, different task description prompt text templates are set for different question-answering tasks.
[0018] Preferably, in S3, for complex question-answering tasks, the language model needs to output corresponding confidence levels while outputting question-answering results, and select the question-answering result with the highest confidence level as the final output result.
[0019] Preferably, the language model after fine-tuning of the instructions includes GPT-3.5, Claude or GPT-4.
[0020] Compared with the prior art, the present invention has the following beneficial effects:
[0021] Compared with the existing technology, the present invention combines the OCR results of document images by adding spaces and line breaks into document content prompt text that retains the layout information of the original document as much as possible. This can utilize the powerful zero-shot learning ability of the instruction fine-tuning large language model, realize zero-shot question answering of document images, and achieve performance comparable to that of the same type of model under the pre-training fine-tuning paradigm. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 Schematic diagram of the steps for zero-shot document image question answering with layout- and task-aware text prompts.
[0023] Figure 2 is a flow chart of the method of the present invention.
[0024] Figure 3 This is the task description prompt text template for DocVQA in the present invention (the input language is English).
[0025] Figure 4 This is the task description prompt text template for InfographicVQA in the present invention (the input language is English).
[0026] Figure 5 This is the task description prompt text template for MP-DocVQA in the present invention (the input language is English). DETAILED DESCRIPTION
[0027] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.
[0028] like Figure 1 As shown, in a preferred embodiment of the present invention, a layout and task-aware text prompt zero-sample document image question answering method is provided, and its steps are shown as S1 to S3:
[0029] S1: Perform OCR processing on the document image to extract all text fragments S contained in the document image = {s1, s2, ..., s n} and the corresponding text box coordinates B={b1,b2,…,b n}, where n represents the number of extracted text segments.
[0030] S2: Using the OCR results obtained in S1, all text fragments are determined according to the corresponding text box coordinates to determine the original lines in the document image. All text fragments are combined into document content prompt text by adding spaces and line breaks, preserving the layout information in the original document as much as possible.
[0031] The process of forming the document content prompt text by the above combination can be expressed as follows:
[0032] S′=Recover(S,B)
[0033] The string S′ is the document content prompt text, and Recover() represents a function that restores document layout information. The combination process of the Recover() function can essentially be expressed as follows: all text fragments are sorted according to their corresponding original lines. Text fragments in the same original line continue to be placed in the same line of the document content prompt text, and line breaks are added between the upper and lower lines. Spaces are added between adjacent text fragments in the same line to ensure that their horizontal spacing is consistent with their original layout in the document image. This ensures that the combined document content prompt text retains the layout information of the original document.
[0034] As a preferred implementation of the embodiment of the present invention, the implementation steps of Recover() are:
[0035] S21. First, all text segments are arranged in order from top to bottom and from left to right according to the coordinates of the extracted text boxes. The text segments in the same original row in the document image and their corresponding text box coordinates are recorded. The text in the i-th row and the coordinates of its corresponding text box can be inserted into the list S21. i and B i Then, the character width is counted and the sum of the widths of all text boxes in a single line is divided by the total number of characters in this line to obtain the average character width of a single character. In this embodiment, in order to make the calculation result representative, when calculating the average character width of a single character, the line with the largest number of characters is first found, and then the average character width is calculated based on this line, thereby representing the average character width of all lines in the entire document image. i and w i Represents the number of characters in the i-th row and the width of the union of all text boxes in the i-th row, respectively. The average character width It can be calculated by the following formula:
[0036]
[0037] That is, the i * The line is the one with the largest number of characters among all the lines.
[0038] S22, then it is necessary to add spaces between adjacent text segments in the same row from left to right. The text segments in the same original row in the document image are combined according to their respective horizontal spatial order in the document image, and the horizontal layout information of the two adjacent text segments is maintained by adding spaces between them, and the number of spaces added is obtained by dividing the minimum horizontal spacing of the text boxes corresponding to the two text segments by the average character width. Specifically, the two adjacent text segments S in the i-th row are i,j and S i,k The number of blank lines added between where h i,jk For two adjacent text boxes B i,j and B i,k The minimum horizontal distance between
[0039] S23. Arrange each line of combined text from top to bottom according to its vertical spatial order in the document image, and separate two adjacent lines of combined text by adding a line break.
[0040] S3: Design a specific task description prompt text template P based on the specific target question-answering task. This template contains a specific task description, a document content prompt placeholder containing layout information, and a specific question placeholder.
[0041] As a preferred implementation of the present invention, the task description prompt text template P is different for different question-answering tasks. For more complex question-answering problems, the model can be instructed to output the corresponding confidence level when outputting the question-answering result, and the prediction result with the highest confidence level is selected as the final output result. Examples of task description prompt text templates for different question-answering tasks are as follows: Figure 3 、 Figure 4 and Figure 5 shown.
[0042] When actually executing the task, the document content prompt text S′ formed by combining the document content in S2 and the corresponding question q input by the user are used to replace the corresponding placeholders in the template P in S3 to obtain the final prompt text P(S′,q). The language model after fine-tuning the input instructions is used to obtain the final question-answering result, which can be expressed as follows:
[0043] a′=f M (P(S′,q))
[0044] Where a′ represents the prediction result of the model, f M () represents the decoding process of the model.
[0045] For the specific implementation process of the above-mentioned S1 to S3 method flow in an example, see Figure 2 shown.
[0046] It should be noted that the specific form of the language model after instruction fine-tuning used in the present invention is not limited. As a preferred implementation method of an embodiment of the present invention, the above-mentioned instruction fine-tuning large language model can use GPT-3.5, Claude or GPT-4.
[0047] In order to further demonstrate the technical effect of the present invention, the layout and task-aware text prompt zero-sample document image question answering method described in S1 to S3 above is applied to a specific dataset example.
[0048] Example
[0049] The specific steps of the layout and task-aware text prompt zero-shot document image question answering method in this embodiment are as described in S1 to S3 above and will not be repeated here. For ease of description, the above layout and task-aware text prompt zero-shot document image question answering method is abbreviated as LATIN-Prompt.
[0050] This example uses three document image question answering datasets: DocVQA, InfographicVQA, and MP-DocVQA, to evaluate the performance of the present invention on this task.
[0051] DocVQA is an extractive question answering task consisting of 50,000 questions defined on 12,767 document images. The training set has 39,463 questions, the validation set has 5,349 questions, and the test set has 5,188 questions. DocVQA contains a large number of questions related to tables and layouts in images, placing high demands on the model's ability to understand document image layout. InfographicVQA consists of 5,485 infographics containing a variety of text, graphics, and visual elements. Compared to DocVQA, the InfographicVQA task also requires basic reasoning and arithmetic skills, and the answer sources are more complex. MP-DocVQA extends the document visual question answering task to the more realistic multi-page scenario, where a document often consists of multiple pages that should be processed together. It contains 46,000 questions covering 48,000 scanned pages from 6,000 industry documents, with page images containing different layouts. The variability between documents in MP-DocVQA is very high. The number of pages in each document ranges from 1 to 20, and the number of recognized OCR words ranges from 1 to 42,313. Figure 3 、 Figure 4and Figure 5 The task description prompt text templates P designed by the present invention for DocVQA, InfographicVQA and MP-DocVQA are shown respectively. The two placeholder positions marked with {} are the two placeholders that need to be replaced.
[0052] This embodiment uses three instruction fine-tuning language models to verify the effectiveness of the method, namely Claude, GPT-3.5 and Alpaca models. This implementation verifies the performance of the method on three data sets according to the processes described in S1 to S4 above, and the results are shown in Table 1, Table 2 and Table 3. Among them, Plain Prompt is used as a comparison with LATIN-Prompt in the present invention. Unlike LATIN-Prompt in the present invention, the prompt text used by Plain Prompt is "Document:{document}Questiion:{question}Directly extract the answer of the questionfrom the document.Answer:", {document} and {question} respectively represent the original text fragment obtained by OCR and the placeholder for the question.
[0053] Table 1 Model performance on the DocVQA dataset
[0054]
[0055] * This refers to the results of GPT-4 reported on the OpenAI blog. Although the details are lacking, GPT-4 utilizes visual information compared to Cluade and GPT-3.5. LATIN-Prompt and GPT-4 are orthogonal.
[0056] Table 2 Model performance on the InfographicVQA dataset
[0057]
[0058] Table 3 Model performance on the MP-DocVQA dataset
[0059]
[0060] In addition to inputting specific questions into the model, the LATIN-Prompt of the present invention mainly includes two components: document content prompt text containing layout information and task description prompt text. To verify the effectiveness of each part, this embodiment conducted ablation experiments on the validation datasets of DocVQA and InfographicVQA. The results are shown in Tables 4 and 5. Among them, Plain represents Plain Prompt for comparison, Layout represents only inputting document content prompt text containing layout information, and Task represents only inputting task description prompt text. LATIN-Prompt corresponds to the situation in the present invention where document content prompt text containing layout information and task description prompt text are input simultaneously.
[0061] Table 4 Ablation experiment results of Claude model on DocVQA and InfographicVQA datasets
[0062]
[0063] Table 5 Ablation test results of GPT-3.5 model on DocVQA and InfographicVQA datasets
[0064]
[0065] The results in the table above show that both document content hints containing layout information and task description hints significantly improve the model's zero-shot prediction performance. The improvement provided by the task description hint is even more significant because it ensures that the format of the generated answer meets the task requirements. Furthermore, document content hints containing layout information further improve model performance by enabling the model to leverage layout information between text segments, building on the correct format.
[0066] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.
Claims
1. A layout- and task-aware text-prompted zero-shot document image question answering method, characterized in that: Here are the steps: S1: Perform OCR processing on the document image to extract all text segments contained in the document image and the text box coordinates corresponding to each text segment; S2: All text fragments extracted in S1 are judged according to the corresponding text box coordinates to determine their original rows in the document image, and all text fragments are combined into document content prompt text by adding spaces and line breaks. During the combination process, all text fragments are sorted according to their corresponding original rows, and text fragments in the same original row continue to be placed in the same row of the document content prompt text, and line breaks are added between the upper and lower rows. Spaces are added between adjacent text fragments in the same row to ensure that their horizontal spacing is consistent with their original layout in the document image, so that the combined document content prompt text retains the layout information in the original document. S3: Read the task description prompt text template designed for the target question-answering task. The template contains a specific task description, a document content prompt placeholder containing layout information, and a specific question placeholder. The document content prompt text combined in S2 and the specific question entered by the user replace the corresponding placeholders in the template to obtain the final task description prompt text. This text is input into the language model after instruction fine-tuning, and the final question-answering result is output through model decoding. In S2, the specific method of combining all text fragments into document content prompt text by adding spaces and line breaks is as follows: S21. First, all text segments are arranged in order from top to bottom and from left to right based on the coordinates of the extracted text boxes of each text segment. The text segments in the same original line in the document image and their corresponding text box coordinates are recorded. Then, the sum of the widths of all text boxes in a single line is divided by the total number of characters in the line to obtain the average character width of a single character. When calculating the average character width of a single character, first find the row with the largest number of characters, then use this row as the basis for calculating the average character width, and then represent the average character width of all rows in the entire document image; S22. Then, text segments in the same original line in the document image are grouped according to their respective horizontal spatial order in the document image, and spaces are added between two adjacent text segments to maintain their horizontal layout information, where the number of spaces added is calculated by dividing the minimum horizontal spacing between the text boxes corresponding to the two text segments by the average character width; S23. Arrange each line of combined text from top to bottom according to its vertical spatial order in the document image, and separate two adjacent lines of combined text by adding a line break.
2. The layout and task-aware text-prompted zero-shot document image question answering method according to claim 1, characterized in that: In S3, different task description prompt text templates are set for different question-answering tasks.
3. The layout and task-aware text-prompted zero-shot document image question answering method according to claim 1, characterized in that: In S3, for complex question-answering tasks, the language model needs to output the corresponding confidence level while outputting the question-answering results, and select the question-answering result with the highest confidence level as the final output result.
4. The layout and task-aware text-prompted zero-shot document image question answering method according to claim 1, characterized in that: The language model fine-tuned by the instructions includes GPT-3.5, Claude or GPT-4.