Certificate information extraction method and device, computer equipment and storage medium

Abstract

The invention discloses a certificate information extraction method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a to-be-processed document image; extracting text elements and bounding box coordinates of the to-be-processed document image; analyzing the text elements and the bounding box coordinates of the text elements to determine relative spatial positions among the text elements, deducing a logic structure, and generating a formatted text simulating an original document layout by inserting a space and a line feed; inputting the formatted text into a large language model, and extracting required information in combination with a prompt engineering strategy to obtain formatted data; and outputting the formatted data. By implementing the method provided by the invention, the accuracy and efficiency of the large-scale language model in the structured data extraction task can be remarkably enhanced, so that the problems of high calculation cost, complex model modification or data privacy risk in the aspect of effectively utilizing a plain text LLM to understand a document space structure in the prior art are solved.

Description

Technical Field

[0001] This invention relates to artificial intelligence, and more specifically to methods, apparatus, computer equipment, and storage media for extracting document information. Background Technology

[0002] Traditional document information extraction primarily relies on optical character recognition (OCR) combined with template-based rule-based processing. This approach performs well with documents in fixed formats and single languages, but its limitation lies in its inability to adapt to complex scenarios involving multiple templates and languages. Traditional OCR systems treat documents as simple "planar character sets," often losing crucial two-dimensional spatial structure and contextual information when converting image text into machine-readable formats. This includes the relative positions of text blocks, table structures, and the flow of multi-column text. This lack of layout information makes subsequent information interpretation and processing difficult, typically requiring the use of downstream tools or manual review.

[0003] The development of large language models (LLMs) has revolutionized the field of document information extraction. These models, trained on massive amounts of text data, possess exceptional language understanding and adaptability, capable of handling various types of structured and unstructured documents, and demonstrating strong contextual understanding and few-shot or even zero-shot learning capabilities. However, when LLMs only receive linear text from OCR output, crucial spatial contextual information within the document is ignored, limiting the model's overall performance. For example, with flattened text input, LLMs may fail to correctly distinguish between headings and body paragraphs, or accurately associate key-value pairs in forms, relationships that are visually obvious. To overcome this challenge, existing technologies have explored various solutions. Firstly, multimodal LLMs such as Qwen2.5-VL, GPT-4o, and Gemini can simultaneously process document images and their text content, holistically parsing the document's context and structure by performing computer vision and natural language processing tasks. While this approach holds immense potential, it also introduces higher computational costs, speed considerations, and data privacy and security issues. Furthermore, subtle variations in the input can lead to inconsistent outputs, and the model is at risk of generating seemingly plausible but actually incorrect information. Secondly, another approach utilizes layout tokens / positional encoding, representing layout information as independent text tokens and interleaving them with their corresponding text content as input to the LLM. While this method helps preserve layout information, it may also affect the model's ability to learn longer texts and introduce ambiguity. Finally, converting the OCR output to a structured markup language format such as HTML or Markdown is also an effective strategy that can significantly improve performance. However, this approach may lead to increased token usage and latency, and the conversion from raw OCR output to semantically meaningful HTML or Markdown structures requires the support of intelligent algorithms.

[0004] Therefore, it is necessary to design a new method to significantly enhance the accuracy and efficiency of large language models in structured data extraction tasks, in order to address the problems of high computational cost, complex model modification, or data privacy risks in the effective use of plain text LLMs to understand document spatial structure. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus, computer equipment and storage medium for extracting document information.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for extracting document information, comprising: Obtain the image of the document to be processed; Extract the text elements and bounding box coordinates of the document image to be processed; The text elements and their bounding box coordinates are analyzed to determine the relative spatial positions between the text elements, infer the logical structure, and generate formatted text that simulates the layout of the original document by inserting spaces and line breaks. The formatted text is input into a large language model, and the required information is extracted by combining the prompting engineering strategy to obtain formatted data. Output the formatted data.

[0007] The further technical solution is as follows: Analyzing the text elements and their bounding box coordinates to determine the relative spatial positions between text elements, inferring the logical structure, and generating formatted text that simulates the original document layout by inserting spaces and line breaks includes: The text elements and their bounding box coordinates are analyzed to determine the relative spatial positions between the text elements and to infer the logical structure. Based on the logical structure, the text elements are converted into formatted plain text, using spaces and line breaks to reflect the visual layout and semantic structure of the original document, thus obtaining formatted text.

[0008] The further technical solution is as follows: the logical structure includes table rows and columns, key-value pairs, and header paragraph hierarchy.

[0009] The further technical solution is as follows: Based on the logical structure, the text elements are converted into formatted plain text, using spaces and line breaks to reflect the visual layout and semantic structure of the original document, to obtain formatted text, including: According to the logical structure, the text elements are converted into formatted plain text. During the conversion process, spaces are inserted to maintain the horizontal alignment, indentation, and column structure of the text blocks in the document image to be processed. Line breaks are used to vertically divide the text to distinguish different logical lines or semantic units.

[0010] The further technical solution is as follows: Based on the logical structure, the text elements are converted into formatted plain text, and during the conversion process, spaces are inserted to maintain the horizontal alignment, indentation, and column structure of the text blocks in the document image to be processed; line breaks are used to vertically divide the text to distinguish different logical lines or semantic units, including: Sort all the text elements according to their ordinate and abscissa; Calculate the horizontal and vertical distances between each pair of consecutive text elements; When the vertical distance exceeds the first threshold, insert a newline character; when the horizontal distance exceeds the second threshold, insert the corresponding number of spaces between the text to obtain the inserted spaces and newline characters. Concatenate all the text elements with the inserted spaces and line breaks to form formatted text that mimics the layout of the original document.

[0011] The further technical solution is as follows: the prompting engineering strategy includes explicitly instructing the large language model in the prompt how to interpret the spatial format, and includes examples to demonstrate the expected input format and the required structured output.

[0012] A further technical solution is as follows: after outputting the formatted data, it further includes: Fine-tuning large language models on formatted datasets, enhancing performance through instruction and contextual learning, and combining hybrid processes with post-correction mechanisms for fine-tuning large language models.

[0013] The present invention also provides a document information extraction device, comprising: The acquisition unit is used to acquire the image of the document to be processed; An extraction unit is used to extract the text elements and bounding box coordinates of the document image to be processed; The analysis and processing unit is used to analyze the text elements and their bounding box coordinates to determine the relative spatial positions between text elements, infer the logical structure, and generate formatted text that simulates the layout of the original document by inserting spaces and line breaks. An extraction unit is used to input the formatted text into a large language model, and extract the required information in conjunction with a prompting engineering strategy to obtain formatted data. An output unit is used to output the formatted data.

[0014] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0015] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0016] The advantages of this invention compared to existing technologies are as follows: By converting the document image to be processed into formatted text containing text elements and their bounding box coordinates, this invention cleverly preserves the spatial layout information of the original document. It simulates the original document's layout by inserting spaces and line breaks, enabling large-scale plain text language models to understand the document's logical structure without directly processing image data or making complex visual embedding layer modifications. This significantly reduces computational costs and avoids high-cost multimodal models while improving the accuracy and efficiency of structured data extraction tasks. Furthermore, this method reduces reliance on original image data, enhances data privacy protection, and solves the privacy leakage risks and technical complexity issues arising from image data processing in existing technologies. Combined with prompting engineering strategies, the information extraction process is further optimized, enabling plain text LLMs to efficiently and accurately complete document understanding tasks.

[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of the document information extraction method provided in this embodiment of the invention; Figure 2 A flowchart illustrating the document information extraction method provided in this embodiment of the invention; Figure 3 A schematic diagram of the sub-process of the document information extraction method provided in the embodiments of the present invention. Figure 1 ; Figure 4 A schematic diagram of the sub-process of the document information extraction method provided in the embodiments of the present invention. Figure 2 ; Figure 5 A schematic block diagram of a document information extraction device provided in an embodiment of the present invention; Figure 6 A schematic block diagram of the analysis and processing unit of the document information extraction device provided in the embodiments of the present invention; Figure 7 A schematic block diagram of the layout format subunit of the document information extraction device provided in the embodiments of the present invention; Figure 8 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the document information extraction method provided in an embodiment of the present invention. Figure 2This is a schematic flowchart illustrating the document information extraction method provided in this embodiment of the invention. The method is applied in a server. The server interacts with the terminal, acquiring document images and extracting text elements and their bounding box coordinates. It analyzes the relative spatial positions of these elements to infer the logical structure and generates formatted text simulating the original document layout using spaces and line breaks. This text is then input into a large language model, combined with prompting engineering strategies to extract the required information. This method further refines the understanding of the logical structure, including table rows and columns, key-value pairs, and heading paragraph levels, ensuring that the formatted text accurately reflects the visual and semantic layout of the original document. Furthermore, it optimizes the horizontal alignment and column structure of text blocks using sorting and distance calculation based on text element coordinates, and uses specific thresholds to control the insertion of spaces and line breaks, thereby improving formatting accuracy. Finally, by fine-tuning the large language model on the formatted dataset, enhancing performance using instruction and context learning, and continuously improving it by combining a hybrid process and post-correction mechanisms, the accuracy and efficiency of the large language model in structured data extraction tasks are significantly enhanced, effectively solving the problems of high computational cost, complex model modification, or data privacy risks in existing technologies.

[0025] Figure 2 This is a flowchart illustrating the document information extraction method provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S150.

[0026] S110. Obtain the image of the document to be processed.

[0027] In this embodiment, the document image to be processed refers to any form of digital document image from which information needs to be extracted. These images may include, but are not limited to, personal identification documents such as ID cards, passports, and driver's licenses; business documents such as bank statements, invoices, and contracts; and any image data containing text information and having a specific layout structure. The process of acquiring these document images typically involves taking high-quality photographs using a scanner or camera on a mobile device, ensuring that the text is clearly readable and that distortion and shadows are minimized as much as possible.

[0028] Specifically, the main purpose of step S110 is to provide a basic data source for subsequent information extraction. This means that when acquiring document images, the following points should be noted to ensure the accuracy and efficiency of OCR processing: Resolution: Ensure the document image has sufficient resolution so that the OCR engine can accurately recognize the characters. A resolution of 300 DPI (dots per inch) or higher is generally recommended.

[0029] Sharpness and contrast: Maximize image sharpness and ensure sufficient contrast between text and background, which helps improve the accuracy of OCR recognition.

[0030] Distortion-free: Avoid tilting or bending the image during shooting or scanning, and keep the document flat to avoid affecting the accuracy of the bounding box coordinates.

[0031] Lighting conditions: Appropriate lighting is key. Avoid overly bright or dark conditions, reduce shadows and reflections, and ensure clear character edges.

[0032] After completing the above preparations, the document image is converted into machine-readable text format using Optical Character Recognition (OCR) technology. Simultaneously, the precise location information of each text element, i.e., the bounding box coordinates, is extracted, laying the foundation for the next step of spatial structure analysis. This process is the first and crucial step in implementing the method described in this invention, as it directly relates to the accuracy and reliability of the final extracted information.

[0033] S120. Extract the text elements and bounding box coordinates of the document image to be processed.

[0034] In this embodiment, a text element refers to each individual character, number, symbol, or word identified from the document image to be processed using Optical Character Recognition (OCR) technology. These text elements are the basic units that constitute the document content, and each text element carries specific information. For example, on a bank statement, text elements may include information such as account number, transaction date, and amount.

[0035] Bounding box coordinates refer to the precise location of each identified text element in the original document image, typically represented by a set of four values: (xi, yi, wi, hi). Here: xi and yi are the horizontal and vertical coordinates of the top-left corner of the text element relative to the top-left corner of the document image. They determine the starting point of the text element in two-dimensional space.

[0036] wi and hi represent the width and height of the text element, respectively, and are used to describe the size of the space occupied by the text element.

[0037] Specifically, the main objective of step S120 is to extract text elements and their corresponding bounding box coordinates from the document image to be processed. This process mainly includes the following two aspects: Text Recognition: Utilizing a high-precision OCR engine to scan and analyze document images to accurately identify all text elements such as characters, numbers, and symbols. This process needs to consider factors such as document quality, font style, and font size to ensure correct parsing of various text types.

[0038] Location positioning: For each identified text element, the OCR engine also needs to calculate its precise location in the original document, i.e., the bounding box coordinates. This step is crucial because it not only helps us understand the relative positional relationships between various text elements, but also provides the basic data support for subsequent spatial structure inference and formatting.

[0039] Through the above process, a list containing all text elements and their corresponding bounding box coordinates can be obtained. This list will become the key input for the next step of analyzing the text's logical structure, thereby achieving effective encoding and utilization of the document's two-dimensional spatial layout information. This precise location information helps improve the accuracy and efficiency of large language models (LLMs) when performing structured data extraction tasks.

[0040] S130. Analyze the text elements and their bounding box coordinates to determine the relative spatial positions between text elements, infer the logical structure, and generate formatted text that simulates the original document layout by inserting spaces and line breaks.

[0041] In this embodiment, relative spatial position refers to the specific position of each text element in a two-dimensional coordinate system within the document image. This includes the horizontal (left-right) and vertical (up-down) distances of the text element relative to other text elements. By analyzing these coordinates, it is possible to understand how the text elements are arranged, such as whether they belong to the same row or column, or whether there are hierarchical relationships such as headings and paragraphs, key-value pairs, etc.

[0042] Horizontal position: refers to the position of text elements in the horizontal direction, that is, how they are distributed within a line.

[0043] Vertical position: refers to the position of text elements in the vertical direction, used to determine whether they are on the same line or belong to different paragraphs, table rows, etc.

[0044] Inferring logical structure involves identifying and interpreting the roles and relationships of text elements within a document based on their positional information. This process includes, but is not limited to: Table structure identification: By analyzing the bounding box coordinates of text elements, determine which text elements constitute a table, as well as the rows and columns within the table.

[0045] Distinguishing key-value pairs: Identify key-value pairs appearing in a document, such as "Name" as the key and "Zhang San" as the value in a form.

[0046] Hierarchical relationship between headings and paragraphs: Determine whether a text element is a heading or body text, and the hierarchical relationship between them, such as multiple paragraphs under a first-level heading.

[0047] Lists and other structured content: Identify ordered or unordered lists, bullet points, etc.

[0048] Formatting text involves converting the raw OCR output into a plain text representation that accurately reflects the original visual layout of the document. This process mainly includes the following steps: Sorting text fragments: First, sort the text fragments in order from top to bottom and from left to right.

[0049] Calculate the distance between adjacent segments: Calculate the horizontal and vertical distances between adjacent text segments.

[0050] Insert spaces and newlines: Horizontal alignment: If the horizontal distance between two text segments exceeds a certain threshold, insert an appropriate number of spaces between them to maintain the relative position and indentation of the text blocks.

[0051] Vertical separation: If the vertical distance between two text segments exceeds another threshold, a newline character is inserted after the previous text segment to separate different logical lines or semantic blocks.

[0052] Result of concatenation: All processed text fragments, along with inserted spaces and line breaks, are concatenated into a final string that accurately reproduces the visual layout of the original document.

[0053] This approach not only makes text data easier for large language models to understand and parse, but also preserves the spatial structure information of the original document, thereby improving the accuracy and efficiency of information extraction tasks. In this way, even plain text LLMs can effectively handle complex document layouts without relying on the complexity of multimodal models.

[0054] In one embodiment, please refer to Figure 3 The above-mentioned step S130 may include steps S131 to S132.

[0055] S131. Analyze the text elements and their bounding box coordinates to determine the relative spatial positions between the text elements and infer the logical structure.

[0056] In this embodiment, the logical structure includes table rows and columns, key-value pairs, and heading paragraph hierarchy.

[0057] In this embodiment, the relative spatial positions of text elements and their bounding box coordinates are determined by analyzing these elements, and the logical structure of the document is inferred based on this positional information. Specifically, this includes the following aspects: Table row and column recognition: By analyzing the bounding box coordinates of text elements, the system identifies rows and columns in a table and understands the relationships between the contents of each cell.

[0058] Key-value pair differentiation: Based on the spatial position of text elements, identify which parts are "keys" (such as field names in a form) and which parts are "values" (such as the specific content entered in a form).

[0059] Heading paragraph hierarchy: By analyzing the position and size of text elements, determine which text elements belong to headings and which belong to body paragraphs, and identify the hierarchical relationship between them.

[0060] S132. Based on the logical structure, the text elements are converted into formatted plain text, using spaces and line breaks to reflect the visual layout and semantic structure of the original document, so as to obtain formatted text.

[0061] In this embodiment, the text elements are converted into formatted plain text according to the logical structure. During the conversion process, spaces are inserted to maintain the horizontal alignment, indentation, and column structure of the text blocks in the document image to be processed. Line breaks are used to vertically divide the text to distinguish different logical lines or semantic units.

[0062] In this embodiment, based on the logical structure deduced in the previous step, the text elements are converted into formatted plain text that can simulate the original document layout. The specific operations are as follows: Horizontal alignment and indentation: By inserting an appropriate number of spaces, the horizontal alignment, indentation, and column structure of text blocks are maintained, ensuring that the relative positional relationships between different text elements are preserved.

[0063] Vertical splitting: Using line breaks to vertically split text, dividing it into different logical lines or semantic units, so that each independent piece of content can be clearly displayed.

[0064] In one embodiment, please refer to Figure 4 The above step S132 may include steps S1321 to S1324.

[0065] S1321. Sort all the text elements according to their vertical and horizontal coordinates.

[0066] First, all text elements are sorted according to their vertical coordinates (from top to bottom) and horizontal coordinates (from left to right). This step ensures that the text elements are arranged in the order of their actual positions in the document during subsequent processing.

[0067] S1322. Calculate the horizontal and vertical distances between each pair of consecutive text elements.

[0068] Next, for each pair of consecutive text elements after sorting, the horizontal distance (Δx) and vertical distance (Δy) between them are calculated. This distance information is used to determine whether spaces or newlines need to be inserted.

[0069] S1323. When the vertical distance exceeds the first threshold, insert a newline character; when the horizontal distance exceeds the second threshold, insert the corresponding number of spaces between the text to obtain the inserted spaces and newline characters.

[0070] If the vertical distance between two text elements exceeds a preset first threshold, a newline character is inserted between the two elements to distinguish different logical lines or semantic units. Similarly, if the horizontal distance exceeds a preset second threshold, a certain number of spaces are inserted between them to maintain the relative position and column structure of the text blocks.

[0071] S1324. Concatenate all the text elements and the inserted spaces and line breaks to form formatted text that simulates the layout of the original document.

[0072] Finally, all text elements, along with the spaces and newlines inserted according to the above rules, are concatenated into a complete string. The resulting formatted text not only contains all the text content of the original document but also accurately reflects its two-dimensional visual layout and semantic structure.

[0073] Through this series of steps, the text elements output by OCR are efficiently and accurately transformed into a formatted text representation that is easy for large language models to understand and parse, thereby improving the accuracy and efficiency of information extraction tasks.

[0074] In this embodiment, horizontal alignment is achieved by inserting an appropriate number of spaces to simulate the relative positions, indentation, and column structure of text blocks observed in the original document image. Vertical separation is achieved using line breaks to divide logical lines or independent semantic blocks. This ensures that the generated formatted text accurately reproduces the visual layout of the original document, thus explicitly encoding two-dimensional spatial information into the text stream.

[0075] Specifically, a high-precision OCR engine is employed that can provide accurate bounding box coordinates for each recognized word or character. The quality of the OCR output (especially the bounding box accuracy) directly determines the fidelity and accuracy of the spatial encoding.

[0076] Spaces and line breaks are systematically inserted into the original OCR text to accurately simulate the relative positions, indentation, column structure, and table alignment of text blocks observed in the original document image, and to restore the layout information between text fragments extracted by OCR through spaces and line breaks.

[0077] Algorithm input: A set of text elements T={t1, t2, ..., tn}, each fragment ti is accompanied by its bounding box coordinates (xi, yi, wi, hi) in the document image, where xi and yi are the top left corner coordinates, and wi and hi are the width and height.

[0078] Horizontal distance threshold θh (used to determine whether a space needs to be inserted). Vertical distance threshold θv (used to determine whether a line break is needed).

[0079] Output a formatted string S, which simulates the original document layout by inserting spaces and newlines.

[0080] Specifically, based on the coordinates of the text fragments, they are sorted in a top-to-bottom, left-to-right order. First, they are sorted in ascending order by yi (vertical direction). If yi is the same, they are then sorted in ascending order by xi (horizontal direction).

[0081]

[0082] For two adjacent segments ti and ti+1: Calculate horizontal distance:

[0083] Calculate vertical distance:

[0084] If Δyi, i+1 > θv, then insert a newline character \\n after ti. If Δxi, i+1 > θh, then insert n spaces between ti and ti+1, satisfying: , where k is the character width per unit pixel (e.g., k = 10 pixels / character).

[0085] Concatenate all text fragments and inserted spaces / newlines into the final string S: Gap i These are spaces or newlines inserted according to the rules mentioned above.

[0086] S140. Input the formatted text into the large language model, and extract the required information by combining the prompting engineering strategy to obtain formatted data.

[0087] In this embodiment, formatted data refers to plain text strings with explicit spatial layout information converted from the original OCR text. This formatting not only preserves the order and hierarchy of the text content in the document, but also simulates the two-dimensional visual layout of the original document by inserting appropriate spaces and line breaks. The purpose of this is to enable LLM to understand and utilize these spatial cues, thereby performing structured data extraction tasks such as key-value pair extraction and table cell recognition more accurately.

[0088] The prompting engineering strategy includes explicitly instructing the large language model in the prompt how it interprets the spatial format, and includes examples to demonstrate the expected input format and the required structured output.

[0089] To maximize the performance of LLM when performing these tasks, a series of hint engineering strategies were adopted, mainly including the following aspects: Explicit Instructions: The prompts provided to the LLM should clearly state that the input text has been spatially formatted using spaces and line breaks to simulate its original visual layout. For example, “The following document text has been horizontally aligned with spaces and vertically separated by line breaks to simulate its original visual layout. Please extract key-value pairs from the following text, assuming keys are left-aligned and values ​​are right-aligned, and output it as a JSON object.” This explicit instruction helps the LLM correctly understand and interpret the spatial structure of the input text.

[0090] Few-shot Examples: Include a few concrete examples in the prompts to demonstrate the expected input format and the desired structured output format (such as JSON, tables). This not only helps LLM learn the correct output format faster but also improves its adaptability and generalization ability to different input formats.

[0091] Multi-step workflow: For more complex documents or tasks requiring higher accuracy, a multi-step workflow approach can be used. This involves breaking down the entire information extraction process into multiple consecutive subtasks, with the context or output of each subtask serving as input for the next. This method can progressively refine and optimize the final extraction results, improving overall accuracy.

[0092] By applying the aforementioned prompting engineering strategies, LLMs can more efficiently extract the required structured information from formatted text. For example, when processing a form containing multiple pieces of personal information, an LLM can accurately identify the location of fields such as name, address, and phone number based on the instructions and examples in the prompts, and organize them into an easy-to-use structured data format (such as a JSON object).

[0093] Furthermore, to further improve the performance of the LLM on specific tasks, fine-tuning can be considered. Specifically, the LLM can be trained on a specially constructed dataset containing documents formatted according to the method described in this embodiment and their corresponding structured outputs. Such fine-tuning not only enhances the model's ability to understand specific types of documents but also improves its robustness and versatility in practical applications.

[0094] This embodiment employs an LLM tokenizer that effectively preserves the structural information of spaces and newlines, such as the BPE algorithm used in GPT-4 (e.g., cl100k_base). These tokenizers are more efficient at handling spaces and are optimized to preserve and understand the structural role of spaces. Tokenizers that over-normalize spaces are avoided to prevent the loss of spatial cues. The method leverages the inherent sensitivity to textual spatial patterns learned by LLMs during pre-training from a large corpus of text containing programming code, poetry, and other structured documents. LLMs can handle text layouts represented by spatial tags, and their performance degrades sharply when these spatial tags are excluded.

[0095] In summary, by cleverly combining formatting techniques and prompting engineering strategies, the method in this embodiment can significantly improve the performance and accuracy of large text-based language models in document information extraction tasks without relying on complex multimodal models.

[0096] S150, Output the formatted data.

[0097] In step S150, the Large Language Model (LLM) has completed parsing the formatted text and extracted the required structured information. This information is typically output in an easy-to-process and use format (such as a JSON object) for further use by subsequent applications or systems.

[0098] In one embodiment, the step S150 described above is followed by: Fine-tuning large language models on formatted datasets, enhancing performance through instruction and contextual learning, and combining hybrid processes with post-correction mechanisms for fine-tuning large language models.

[0099] In one embodiment, after step S150, the method further includes fine-tuning the LLM using a formatted dataset and employing techniques such as Instruction Tuning (IT) and In-Context Learning (ICL) to enhance model performance. Fine-tuning refers to the process of additionally training a pre-trained LLM on a task-specific dataset. For the OCR and LLM combination method proposed in this invention, fine-tuning can be performed through the following steps: Build a dedicated formatted dataset: Create a dataset containing documents formatted according to the space / newline character method and their corresponding structured outputs. These documents should cover as many layout variations as possible to ensure that the model can generalize to a wide range of situations.

[0100] Fine-tuning process: This dataset is used to fine-tune the selected LLM. Since the LLM has already learned how to understand spatial symbols during the pre-training phase, fine-tuning can make it more accurately interpret specific document layout patterns and improve its accuracy in practical applications.

[0101] To further improve the model's performance, the following strategies can be adopted: Instruction Tuning (IT): This is a special type of fine-tuning that involves not only training on task-specific data but also learning the specific instructions on how to perform the task. For example, explicitly informing the LLM of the type of task to be performed in the prompts (such as "extract all cells in a table and organize them by column"), and then teaching the model to follow these instructions through numerous examples.

[0102] In-Context Learning (ICL): This approach does not rely on explicit fine-tuning. Instead, it leverages the powerful contextual understanding capabilities of LLMs to guide model behavior directly at inference time by providing a few examples (few-shot learning). This allows the model to adjust its response based on the specific context of the current input without retraining.

[0103] Although the method in this embodiment mainly relies on plain text LLM to extract document information, in a production environment, other technologies can be combined to improve the robustness and accuracy of the overall system: The method described in this embodiment can be integrated into a larger intelligent document processing workflow, such as using multimodal LLM to perform post-correction on the OCR output. This way, even if there are errors in the initial OCR and text formatting, these inaccuracies can be corrected by analyzing the original image and formatted text.

[0104] Introducing a manual review process, where professionals examine and correct the model's output, not only allows for the timely detection and correction of errors but also enables continuous improvement of the model's performance through a feedback loop.

[0105] Through the above series of measures, not only can the performance and accuracy of text-based LLM in document information extraction tasks be significantly improved, but the dependence on complex multimodal models is also reduced, resulting in a more flexible and efficient solution.

[0106] The method in this embodiment systematically formats the optical character recognition (OCR) output text using spaces and newlines, thereby explicitly encoding the two-dimensional spatial structure information of the document into a layout-aware text representation. This method can significantly enhance the accuracy and efficiency of large language models (LLMs) in structured data extraction tasks, while avoiding the high cost and complexity of multimodal models, as well as the token efficiency problems that may be introduced by other text encoding methods.

[0107] For example, obtain a scanned image of a passport. Then, process this image using a high-precision OCR engine (such as Tesseract). The OCR engine will recognize all text content and its precise location coordinates. For instance, it might recognize "Name: John Doe" and provide the bounding box coordinates of that text block in the image.

[0108] Based on the bounding box coordinates provided by the OCR engine, the relative positions between these text blocks can be analyzed. For example, there is a clear spatial relationship between "Name" and "John Doe," indicating that they are a key-value pair.

[0109] Generate formatted text: Based on these spatial relationships, the original OCR text is converted into a more structured plain text string. For example, "Name: John Doe" might be formatted as: Name: John Doe; Birth Date: January 1, 1980; Nationality: American; ID Number: 123456789; The layout of the original document is simulated by inserting appropriate spaces and line breaks.

[0110] To guide the LLM to correctly understand the formatted text, the following prompt was designed: "The following is information from the passport. Please extract the content of each field according to the following format and return it as a JSON object." The formatted text is input into an LLM (e.g., GPT-4). Since the LLM has already learned how to handle the spatial patterns of the text during pre-training, it can accurately parse this information and convert it into structured data.

[0111] The final output of LLM is a JSON object as shown below: { "name": "John Doe", "birth_date": "01 / 01 / 1980", "nationality": "American", "id_number": "123456789" } To further improve accuracy, we can fine-tune the LLM using a specially formatted passport dataset. This allows the LLM to better understand and handle subtle differences between different passport templates.

[0112] Tesseract was chosen as the OCR engine because it provides precise bounding box coordinates for each character. This is crucial for subsequent spatial encoding. The text fragments output by the OCR are reorganized into a form closer to the original passport layout. For example, if the horizontal distance between "Name" and "John Doe" is greater than a set threshold, appropriate spaces are inserted between the two text fragments; if the vertical distance exceeds the threshold, line breaks are inserted.

[0113] The BPE algorithm (such as cl100k_base) used by GPT-4 is chosen because it effectively preserves information about spaces and newlines. LLM has learned how to understand spatial patterns in text from a large amount of programming code, poetry, and other structured documents. Therefore, it can effectively parse spatial relationships when encountering formatted text.

[0114] The prompts explicitly inform the LLM that the input text has been spatially formatted with spaces and newlines, and provide several examples to demonstrate the expected input format and the required structured output.

[0115] The method in this embodiment cleverly transforms the two-dimensional spatial layout information of a document into plain text format, enabling a small and efficient Language Model (LLM) to extract document information. This approach not only significantly reduces the reliance on large multimodal models but also fully utilizes the inherent text understanding capabilities of LLMs, achieving a "small but powerful" model deployment.

[0116] While traditional or multimodal language models are powerful, they typically require large model sizes and expensive computational resources to handle the complex interactions between images and text. This leads to higher deployment costs, longer inference latency, and potential data privacy and security issues. In contrast, the method in this embodiment transforms visual layout information into a text pattern that an LLM can understand during the OCR stage. This allows plain text LLMs to avoid directly processing image pixels, thus avoiding the high overhead and complexity associated with multimodal models.

[0117] During pre-training, LLMs have already learned sensitivity and understanding of spatial markers such as spaces and line breaks from massive amounts of text data. This data includes programming code, structured logs, and formatted documents. The method in this embodiment leverages this inherent advantage, activating the pre-existing "visual grammar" capabilities of the model through carefully designed text formatting. This means that we can enable the LLM to understand document layout without making large-scale modifications to its underlying architecture or introducing complex visual embedding layers.

[0118] Through this efficient data preparation method, even relatively small plain-text LLMs can achieve performance comparable to large multimodal models on specific document understanding tasks. This "small yet powerful" LLM not only reduces the computational costs of training and inference and improves processing speed, but also makes model deployment more flexible, especially suitable for resource-constrained scenarios or those with strict requirements for response speed. For example, lightweight frameworks such as LayTokenLLM, by optimizing the encoding of layout information, outperform comparable multimodal LLMs on multi-page document understanding tasks. The method of this invention aligns with this concept, aiming to improve the performance of LLMs in the field of document intelligence by optimizing the input data format, rather than blindly increasing the model size.

[0119] Since plain text LLMs do not directly process raw image data, this reduces the risk of sensitive information leakage to some extent, providing a more controllable solution for industries with strict requirements for data privacy and compliance.

[0120] This embodiment's method, by explicitly encoding spatial context, enables LLM to better interpret relationships between text elements (such as key-value pairs, table cells, and hierarchical headings), achieving more accurate structured data extraction that goes beyond simple keyword matching and achieves a deeper level of contextual understanding. For tasks primarily involving text and layout (excluding complex visual elements such as charts or images), this method allows plain-text LLM to achieve competitive performance, potentially avoiding the higher computational costs, speed considerations, and data privacy issues associated with multimodal models. This embodiment's method is compatible with existing plain-text LLM architectures and primarily requires a preprocessing step, rather than fundamentally modifying or specifically training the model. This makes it more accessible for deployment and fine-tuning. This embodiment's method leverages the inherent ability of LLM to interpret and reason about structured text formats, as demonstrated by their proficiency in handling code, poetry, and other whitespace-sensitive content. Maintaining human-readable text as input facilitates debugging and understanding the reasons behind specific extraction decisions made by the LLM (compared to opaque visual embeddings). By cleverly transforming visual layout information into a text pattern that LLM can understand, the method in this embodiment can be better generalized to previously unseen document formats without the need for predefined templates.

[0121] Therefore, the method of this embodiment provides an effective, economical and flexible way to improve the accuracy and efficiency of document information extraction, while ensuring data security and privacy.

[0122] The aforementioned document information extraction method cleverly preserves the spatial layout information of the original document by converting the document image into formatted text containing text elements and their bounding box coordinates. It simulates the original document's layout by inserting spaces and line breaks, allowing a plain text LLM to understand the document's logical structure without directly processing image data or making complex visual embedding layer modifications. This significantly reduces computational costs and avoids high-cost multimodal models while improving the accuracy and efficiency of structured data extraction tasks. Furthermore, this method reduces reliance on original image data, enhances data privacy protection, and solves the privacy leakage risks and technical complexity issues associated with image data processing in existing technologies. Combined with prompting engineering strategies, the information extraction process is further optimized, enabling plain text LLMs to efficiently and accurately complete document understanding tasks.

[0123] Figure 5 This is a schematic block diagram of a document information extraction device 300 provided in an embodiment of the present invention. Figure 5 As shown, corresponding to the above-described document information extraction method, the present invention also provides a document information extraction device 300. This document information extraction device 300 includes a unit for performing the above-described document information extraction method, and the device can be configured in a server. Specifically, please refer to... Figure 5 The document information extraction device 300 includes an acquisition unit 301, an extraction unit 302, an analysis and processing unit 303, an extraction unit 304, and an output unit 305.

[0124] The document acquisition unit 301 is used to acquire an image of the document to be processed; the extraction unit 302 is used to extract the text elements and bounding box coordinates of the document image to be processed; the analysis and processing unit 303 is used to analyze the text elements and their bounding box coordinates to determine the relative spatial positions between the text elements, infer the logical structure, and generate formatted text that simulates the layout of the original document by inserting spaces and line breaks; the extraction unit 304 is used to input the formatted text into a large language model, and extract the required information in combination with the prompting engineering strategy to obtain formatted data; the output unit 305 is used to output the formatted data.

[0125] In one embodiment, such as Figure 6 As shown, the analysis and processing unit 303 includes an analysis subunit 3031 and a layout format subunit 3032.

[0126] Analysis subunit 3031 is used to analyze the text elements and their bounding box coordinates to determine the relative spatial positions between text elements and infer the logical structure; layout formatting subunit 3032 is used to convert the text elements into formatted plain text according to the logical structure, using spaces and line breaks to reflect the visual layout and semantic structure of the original document to obtain formatted text.

[0127] In one embodiment, the layout format subunit 3032 is used to convert the text element into formatted plain text according to the logical structure, and during the conversion process, the horizontal alignment, indentation and column structure of the text block in the document image to be processed are maintained by inserting spaces; and the text is vertically split using line breaks to distinguish different logical lines or semantic units.

[0128] In one embodiment, such as Figure 7 As shown, the layout format subunit 3032 includes a sorting module 30321, a distance calculation module 30322, an insertion module 30323, and a splicing module 30324.

[0129] The sorting module 30321 is used to sort all the text elements according to their ordinate and abscissa; the distance calculation module 30322 is used to calculate the horizontal and vertical distances between each pair of consecutive text elements; the insertion module 30323 is used to insert a newline character when the vertical distance exceeds a first threshold and to insert a corresponding number of spaces between the text elements when the horizontal distance exceeds a second threshold, so as to obtain the inserted spaces and newline characters; the concatenation module 30324 is used to concatenate all the text elements with the inserted spaces and newline characters to form formatted text that simulates the layout of the original document.

[0130] In one embodiment, the document information extraction device 300 further includes: The fine-tuning unit is used to fine-tune large language models on formatted datasets, enhance performance by leveraging instruction and context learning, and combine a hybrid process with a post-correction mechanism for fine-tuning large language models.

[0131] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned document information extraction device 300 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0132] The aforementioned document information extraction device 300 can be implemented as a computer program, which can, for example... Figure 8 It runs on the computer device shown.

[0133] Please see Figure 8 , Figure 8This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0134] See Figure 8 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0135] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a document information extraction method.

[0136] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0137] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a document information extraction method.

[0138] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps: Acquire the image of the document to be processed; extract the text elements and bounding box coordinates of the image of the document to be processed; analyze the text elements and their bounding box coordinates to determine the relative spatial positions between the text elements, infer the logical structure, and generate formatted text that simulates the layout of the original document by inserting spaces and line breaks; input the formatted text into a large language model, and extract the required information in combination with prompting engineering strategies to obtain formatted data; output the formatted data.

[0140] The prompting engineering strategy includes explicitly instructing the large language model in the prompt how it interprets the spatial format, and includes examples to demonstrate the expected input format and the required structured output.

[0141] In one embodiment, when the processor 502 performs the steps of analyzing the text elements and their bounding box coordinates to determine the relative spatial positions between text elements, inferring the logical structure, and generating formatted text that simulates the original document layout by inserting spaces and line breaks, the processor specifically implements the following steps: The text elements and their bounding box coordinates are analyzed to determine the relative spatial positions between the text elements and infer the logical structure. Based on the logical structure, the text elements are converted into formatted plain text, and spaces and line breaks are used to reflect the visual layout and semantic structure of the original document to obtain formatted text.

[0142] The logical structure includes table rows and columns, key-value pairs, and heading paragraph hierarchy.

[0143] In one embodiment, when the processor 502 implements the step of converting the text elements into formatted plain text according to the logical structure, and using spaces and line breaks to reflect the visual layout and semantic structure of the original document to obtain formatted text, the specific implementation steps are as follows: According to the logical structure, the text elements are converted into formatted plain text. During the conversion process, spaces are inserted to maintain the horizontal alignment, indentation, and column structure of the text blocks in the document image to be processed. Line breaks are used to vertically divide the text to distinguish different logical lines or semantic units.

[0144] In one embodiment, when the processor 502 implements the steps of converting the text elements into formatted plain text according to the logical structure, and maintaining the horizontal alignment, indentation, and column structure of the text blocks in the document image to be processed by inserting spaces during the conversion process; and using line breaks to vertically divide the text to distinguish different logical lines or semantic units, the processor 502 specifically implements the following steps: Sort all text elements according to their ordinates and abscissas; calculate the horizontal and vertical distances between each pair of consecutive text elements; insert a newline character when the vertical distance exceeds a first threshold, and insert a corresponding number of spaces between text elements when the horizontal distance exceeds a second threshold, to obtain the inserted spaces and newline characters; concatenate all text elements with the inserted spaces and newline characters to form formatted text that simulates the layout of the original document.

[0145] In one embodiment, after implementing the step of outputting the formatted data, the processor 502 further implements the following steps: Fine-tuning large language models on formatted datasets, enhancing performance through instruction and contextual learning, and combining hybrid processes with post-correction mechanisms for fine-tuning large language models.

[0146] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0147] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0148] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the following steps: Acquire the image of the document to be processed; extract the text elements and bounding box coordinates of the image of the document to be processed; analyze the text elements and their bounding box coordinates to determine the relative spatial positions between the text elements, infer the logical structure, and generate formatted text that simulates the layout of the original document by inserting spaces and line breaks; input the formatted text into a large language model, and extract the required information in combination with prompting engineering strategies to obtain formatted data; output the formatted data.

[0149] The prompting engineering strategy includes explicitly instructing the large language model in the prompt how it interprets the spatial format, and includes examples to demonstrate the expected input format and the required structured output.

[0150] The prompting engineering strategy includes explicitly instructing the large language model in the prompt how it interprets the spatial format, and includes examples to demonstrate the expected input format and the required structured output.

[0151] In one embodiment, when the processor executes the computer program to perform the steps of analyzing the text elements and their bounding box coordinates to determine the relative spatial positions between text elements, inferring the logical structure, and generating formatted text that simulates the original document layout by inserting spaces and line breaks, the processor specifically implements the following steps: The text elements and their bounding box coordinates are analyzed to determine the relative spatial positions between the text elements and infer the logical structure. Based on the logical structure, the text elements are converted into formatted plain text, and spaces and line breaks are used to reflect the visual layout and semantic structure of the original document to obtain formatted text.

[0152] The logical structure includes table rows and columns, key-value pairs, and heading paragraph hierarchy.

[0153] In one embodiment, when the processor executes the computer program to implement the step of converting the text elements into formatted plain text according to the logical structure, using spaces and line breaks to reflect the visual layout and semantic structure of the original document to obtain formatted text, the specific implementation is as follows: According to the logical structure, the text elements are converted into formatted plain text. During the conversion process, spaces are inserted to maintain the horizontal alignment, indentation, and column structure of the text blocks in the document image to be processed. Line breaks are used to vertically divide the text to distinguish different logical lines or semantic units.

[0154] In one embodiment, when the processor executes the computer program to implement the steps of converting the text elements into formatted plain text according to the logical structure, and maintaining the horizontal alignment, indentation, and column structure of the text blocks in the document image to be processed by inserting spaces during the conversion process; and using line breaks to vertically segment the text to distinguish different logical lines or semantic units, the specific implementation is as follows: Sort all text elements according to their ordinates and abscissas; calculate the horizontal and vertical distances between each pair of consecutive text elements; insert a newline character when the vertical distance exceeds a first threshold, and insert a corresponding number of spaces between text elements when the horizontal distance exceeds a second threshold, to obtain the inserted spaces and newline characters; concatenate all text elements with the inserted spaces and newline characters to form formatted text that simulates the layout of the original document.

[0155] In one embodiment, after executing the computer program to perform the step of outputting the formatted data, the processor further performs the following steps: Fine-tuning large language models on formatted datasets, enhancing performance through instruction and contextual learning, and combining hybrid processes with post-correction mechanisms for fine-tuning large language models.

[0156] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0157] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0158] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0159] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0161] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for extracting document information, characterized in that, include: Obtain the image of the document to be processed; Extract the text elements and bounding box coordinates of the document image to be processed; The text elements and their bounding box coordinates are analyzed to determine the relative spatial positions between the text elements, infer the logical structure, and generate formatted text that simulates the layout of the original document by inserting spaces and line breaks. The formatted text is input into a large language model, and the required information is extracted by combining the prompting engineering strategy to obtain formatted data. Output the formatted data.

2. The document information extraction method according to claim 1, characterized in that, The process of analyzing the text elements and their bounding box coordinates to determine the relative spatial positions between text elements, inferring the logical structure, and generating formatted text that simulates the original document layout by inserting spaces and line breaks includes: The text elements and their bounding box coordinates are analyzed to determine the relative spatial positions between the text elements and to infer the logical structure. Based on the logical structure, the text elements are converted into formatted plain text, using spaces and line breaks to reflect the visual layout and semantic structure of the original document, thus obtaining formatted text.

3. The method for extracting document information according to claim 2, characterized in that, The logical structure includes table rows and columns, key-value pairs, and heading paragraph hierarchy.

4. The method for extracting document information according to claim 3, characterized in that, The process of converting the text elements into formatted plain text according to the logical structure, using spaces and line breaks to reflect the visual layout and semantic structure of the original document, to obtain formatted text includes: According to the logical structure, the text elements are converted into formatted plain text. During the conversion process, spaces are inserted to maintain the horizontal alignment, indentation, and column structure of the text blocks in the document image to be processed. Line breaks are used to vertically divide the text to distinguish different logical lines or semantic units.

5. The method for extracting document information according to claim 4, characterized in that, The process involves converting the text elements into formatted plain text according to the logical structure, and during the conversion, maintaining the horizontal alignment, indentation, and column structure of the text blocks in the document image by inserting spaces; and using line breaks to vertically segment the text to distinguish different logical lines or semantic units, including: Sort all the text elements according to their ordinate and abscissa; Calculate the horizontal and vertical distances between each pair of consecutive text elements; When the vertical distance exceeds the first threshold, insert a newline character; when the horizontal distance exceeds the second threshold, insert the corresponding number of spaces between the text to obtain the inserted spaces and newline characters. Concatenate all the text elements with the inserted spaces and line breaks to form formatted text that mimics the layout of the original document.

6. The method for extracting document information according to claim 1, characterized in that, The prompting engineering strategy includes explicitly instructing the large language model in the prompt how it interprets the spatial format, and includes examples to demonstrate the expected input format and the required structured output.

7. The method for extracting document information according to claim 1, characterized in that, After outputting the formatted data, the method further includes: Fine-tuning large language models on formatted datasets, enhancing performance through instruction and contextual learning, and combining hybrid processes with post-correction mechanisms for fine-tuning large language models.

8. A document information retrieval device, characterized in that, include: The acquisition unit is used to acquire the image of the document to be processed; An extraction unit is used to extract the text elements and bounding box coordinates of the document image to be processed; The analysis and processing unit is used to analyze the text elements and their bounding box coordinates to determine the relative spatial positions between text elements, infer the logical structure, and generate formatted text that simulates the layout of the original document by inserting spaces and line breaks. An extraction unit is used to input the formatted text into a large language model, and extract the required information in conjunction with a prompting engineering strategy to obtain formatted data. An output unit is used to output the formatted data.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images