Document analysis method and system based on multiple modes

Abstract

The invention belongs to the technical field of artificial intelligence and multi-modal data processing, and provides a multi-modal-based document analysis method and system. The method comprises the following steps: S1, inputting a to-be-analyzed document, and converting all pages in the to-be-analyzed document into pictures with the same size as an original page page by page; s2, performing layout analysis on the converted picture page by page by utilizing a visual model, detecting layout elements existing in the picture, and obtaining a position coordinate of each layout element; s3, inputting the current picture and the detected layout elements into a large language model for analysis, and adopting different analysis methods for different layout elements to obtain analyzed contents; and S4, displaying the analyzed content according to needs, or performing layout recovery by using the obtained position coordinates of the layout elements and the analyzed content. According to the method, the input document supports all document formats which are almost visible at present, and the analysis speed can be increased.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal data processing technology, specifically relating to a document parsing method and system based on multimodality. Background Technology

[0002] Document parsing has long been a popular research area in computer vision (CV) and natural language processing (NLP). However, current methods all have limitations to varying degrees.

[0003] Chinese patent application CN 111581948A discloses a document parsing method, apparatus, device, and storage medium. The method involves receiving a document parsing request and downloading the corresponding document based on the request; converting the document into an Extensible Markup Language (XML) file; parsing the converted XML file to extract document content; and storing the extracted document content in a database in JS object-based JSON format. However, the parsing method is limited to certain document formats, primarily targeting Word documents, and lacks generalization capabilities. Furthermore, the parsing results are relatively simple, and it cannot recover the original document layout from the parsing results.

[0004] Chinese patent application CN110795919A discloses a method, apparatus, device, and medium for identifying and extracting tables from PDF documents. The method involves extracting structured data corresponding to the content of at least one table from a PDF document containing at least one table; matching the structured data of the at least one table with a labeled dataset corresponding to the content of the table to be extracted to determine a target table matching the labeled dataset; identifying a set of target field cells matching the labeled dataset in the target table; and writing the content of the target field cell set in the target table into an editable document. This solution is mainly for extracting tables from PDFs and has limitations. Summary of the Invention

[0005] This invention aims to address the technical problems existing in the prior art by providing a multimodal document parsing method that can improve the accuracy of detecting blurred table boundaries and recognizing cross-page tables.

[0006] To achieve the above technical objectives, the present invention adopts the following technical solution: This invention provides a multimodal document parsing method, comprising the following steps: S1. Input the document to be parsed, and convert each page of the document to an image with the same size as the original page. S2. Using a visual model, perform layout analysis on the converted images page by page to detect the layout elements present in the images and obtain the position coordinates of each layout element. S3. Input the current image and the detected layout elements into the large language model for parsing. Use different parsing methods for different layout elements to obtain the parsed content. S4. Display the parsed content as needed, or restore the layout using the obtained position coordinates of the layout elements and the parsed content.

[0007] Preferably, in step S2, the layout elements include text, footer, header, table, formula, and illustration.

[0008] Preferably, in step S3, for layout elements that are text, the layout analysis also needs to detect the font size, font, whether it is bold, whether it is italic, and whether it is a superscript or subscript style information of the corresponding text.

[0009] Preferably, in step S3, for layout elements that are illustrations, the content of the illustrations remains unchanged during parsing, and the text information in the illustrations is not parsed.

[0010] Preferably, in step S3, for page elements that are tables, the parsing process requires analyzing the table format based on the parsed text content using a large language model to ensure that the table format is consistent with the table format in the original document.

[0011] Preferably, the document formats to be parsed include pdf, .doc, .docx, .xls, .xlsx, .ppt, .pptx, .jpg, and .png.

[0012] Preferably, in step S4, for cases where only text content needs to be extracted, the parsed content is directly output and displayed in Markdown format.

[0013] Preferably, in step S4, when both text content and layout information are needed, all parsed information is output in JOSN format, and the layout of the original document is restored using this parsed information.

[0014] Preferably, if translation is required, in step S3, during parsing, the text content is translated into the target language by using prompt words to drive the large language model.

[0015] Preferably, for a document to be parsed that has more than a preset threshold of pages, the document to be parsed is split into multiple documents according to the specified number of pages, and then parsed in parallel by the program. After the parsing is completed, the parsing results are re-integrated and summarized according to the original order before splitting to obtain the final parsing result.

[0016] Preferably, for real-time browsing and presentation requirements, the current page browsed by the user is captured. If the current page has been parsed, it is presented immediately; if the current page has not been parsed, the current page and the adjacent preset pages before and after it are parsed immediately. The capture strategy for the user's browsing page is as follows: if the user's page scrolling speed is greater than the preset speed, the page that the user finally stops on is taken as the valid current page to be captured.

[0017] In another aspect, the present invention provides a multimodal document parsing system, including a processor capable of executing a computer program, wherein the computer program, when executed, can implement the above-described multimodal document parsing method.

[0018] Compared with the prior art, the beneficial effects of the present invention are: (1) After obtaining the document to be parsed, this invention first converts all pages in the document into images of the same size, and then sequentially identifies and parses the image content of all converted pages. The input document can support a variety of document formats, such as pdf, .doc, .docx, .xls, .xlsx, .ppt, .pptx, etc., all currently available document formats.

[0019] (2) This invention outputs text content in different formats (markdown and josn) for two needs: extracting only text content and needing the original document layout at the same time, with higher parsing efficiency.

[0020] (3) The present invention employs two different strategies to improve parsing speed for documents with many pages and for instant browsing. Attached Figure Description

[0021] Figure 1 This is a flowchart of a multimodal document parsing method according to an embodiment of the present invention.

[0022] Figure 2 Illustrations, tables, and text diagrams detected by layout analysis in one embodiment of the present invention; Figure 3 This is a schematic diagram of illustrations, tables, text, and footers / headers detected by layout analysis according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figures 1-3 As shown, this invention provides a multimodal document parsing method, comprising the following steps: S1. Input the document to be parsed, and convert each page of the document to an image with the same size as the original page. S2. Using a visual model, perform layout analysis on the converted images page by page to detect the layout elements present in the images and obtain the position coordinates of each layout element. S3. Input the current image and the detected layout elements into the large language model for parsing. Use different parsing methods for different layout elements to obtain the parsed content. S4. Display the parsed content as needed, or restore the layout using the obtained position coordinates of the layout elements and the parsed content.

[0025] In one specific embodiment of the present invention, in step S2, the layout elements include text, footer, header, table, formula, and illustration.

[0026] In a specific embodiment of the present invention, in step S3, for layout elements that are text, the layout analysis also needs to detect the font size, font, whether it is bold, whether it is italic, and whether it is a superscript or subscript style information of the corresponding text.

[0027] In a specific embodiment of the present invention, in step S3, for layout elements that are illustrations, the content of the illustrations remains unchanged during parsing, and the text information in the illustrations is not parsed.

[0028] In a specific embodiment of the present invention, in step S3, for page elements that are tables, the table format needs to be analyzed based on the parsed text content using a large language model during parsing to ensure that the table format is consistent with the table format in the original document.

[0029] In one specific embodiment of the present invention, the document formats to be parsed include pdf, .doc, .docx, .xls, .xlsx, .ppt, .pptx, .jpg, and .png.

[0030] In a specific embodiment of the present invention, in step S4, for cases where only text content needs to be extracted, the parsed content is directly output and displayed in Markdown format.

[0031] In a specific embodiment of the present invention, in step S4, when both text content and layout information are needed, all parsed information is output in JOSN format, and the layout of the original document is restored through this parsed information.

[0032] In one specific embodiment of the present invention, if there is a need for translation, in step S3, during parsing, the text content is translated into the target language by using prompt words to drive the large language model.

[0033] In one specific embodiment of the present invention, for a document to be parsed with more than a preset threshold number of pages, the document to be parsed is divided into multiple documents according to a specified number of pages, and then parsed in parallel by a program. After the parsing is completed, the parsing results are re-integrated and summarized according to the original order before the division to obtain the final parsing result.

[0034] In a specific embodiment of the present invention, for real-time browsing and presentation requirements, the current page browsed by the user is captured. If the current page has been parsed, it is presented immediately; if the current page has not been parsed, the current page and the adjacent preset pages before and after it are parsed immediately. The capture strategy for the user's browsing page is as follows: if the user's page scrolling speed is greater than the preset speed, the page that the user finally stops on is taken as the valid current page.

[0035] In another aspect, the present invention provides a multimodal document parsing system, including a processor capable of executing a computer program, wherein the computer program, when executed, can implement the above-described multimodal document parsing method.

[0036] Example 1 The multimodal document parsing method of the present invention will be described in detail below according to a specific embodiment of the present invention.

[0037] like Figures 1-3 As shown, this invention provides a multimodal document parsing method, comprising the following steps: First, input a file of any format, and convert each page into an image of the same size. Image formats can be common, such as .jpg and .png. Input documents support various formats, including pdf, .doc, .docx, .xls, .xlsx, .ppt, and .pptx—virtually all currently available document formats. Once the document is received, all pages are converted into images of the same size, and then the content of each image is sequentially recognized and parsed.

[0038] Then, layout analysis is performed to examine the elements on the page.

[0039] The system combines precise visual models to perform layout analysis on the converted images. This primarily involves detecting layout elements such as text, footers, headers, tables, formulas, and inserted images. After detection, the coordinates of each page element are obtained for layout restoration during subsequent editing or online reading. If only the document content needs to be extracted without editing or online reading requirements, the parsed text can be directly output. A diagram illustrating the layout analysis is shown below. Figures 2-3 As shown.

[0040] For text-based layout elements, the detection and analysis should not only detect the position coordinates, but also the font size, font, whether it is bold, italic, and whether it is a superscript or subscript.

[0041] Based on the detected layout elements, the large model uses prompts to drive content parsing of the corresponding images on that page. The content of illustrations remains original; no text information within the illustrations needs to be parsed. For tables, the large model analyzes the table format after parsing the text content to ensure consistency with the original document's table format. For text content, if only the text content needs to be extracted (i.e., Markdown output [Markdown syntax can be referenced here]), only the text content needs to be output. If both text content and corresponding layout information are needed (JOIN output), then the layout analysis information is used to format the output, extracting all information, which can be used to reconstruct the original document layout in subsequent applications.

[0042] Based on layout analysis information (information on page elements such as illustrations, tables, formulas, footers, headers, and text blocks detected by computer vision, including the coordinates of these elements within the page) as input, a large language model and corresponding prompts are used to recognize the text within these elements. After recognition, if the element is text-based, the output will include the text content and the coordinates of the text block. Output can be in Markdown format or as a JSON file with position coordinates.

[0043] (1) Markdown file: This file contains only the recognized illustration files, the extracted text content, and the corresponding text formatting. For example, if it's normal text, the extracted content will be pure text; if the current text is a newline character (\n), it will be marked with a newline character (\n). If it's a table, the extracted content will include both the table text and the table style displayed using Markdown syntax. If it's an illustration, the text within it will not be recognized; the illustration will remain as is, and Markdown syntax will be used to indicate that it's an illustration. If it's a formula, it will be displayed as a LaTeX formula, maintaining the correct formatting for bold, italics, etc.

[0044] (2) JSON file JSON files contain more information than Markdown files. They include the content of the Markdown file, the coordinates of each page element within the page (to help to recover the original document's layout as much as possible after recognition, similar to the PDF to Word conversion function we often use), and the page numbers of the source document corresponding to the page parsing results.

[0045] If translation is required, prompt words can be used to drive the large model to translate the text content into the target language during parsing. If no translation is needed, the original language of the text can be directly output. The large language model's translation function generates a pre-translated result in the specified target language while parsing the document content.

[0046] To improve the speed of parsing and rendering: For documents with many pages, hundreds or thousands of pages, there are two acceleration strategies: (1) The original document with a large number of pages is split into many smaller files according to the specified number of pages, and then parsed in parallel by the program to speed up the parsing of the entire document. After the parallel parsing is completed, the parsing results are re-integrated and summarized into the final parsing result according to the original order before splitting.

[0047] (2) For real-time browsing requirements, the current page range viewed by the user is captured. If the current page has been parsed, it is displayed immediately. If the current page has not been parsed, the current page and the adjacent pages before and after it are parsed immediately. The page capture strategy for the user's browsing page is as follows: If the user is scrolling quickly, the page where the user finally stops is considered a valid capture. Then, the current page is used as a reference to execute the current pagination parsing strategy (parsing the current page and the adjacent pages before and after it).

[0048] The above description is merely an embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for document parsing based on multi-modal, characterized in that, The method comprises the following steps: S1, inputting a document to be parsed, and converting all pages in the document to be parsed into pictures with the same size as the original pages; S2, performing layout analysis on the converted pictures page by page using a visual model, detecting the layout elements existing in the pictures, and obtaining the position coordinates of each layout element; S3, inputting the current picture and the detected layout elements into a large language model for parsing, using different parsing methods for different layout elements, and obtaining the parsed content; S4, displaying the parsed content as needed, or performing layout restoration using the position coordinates of the layout elements and the parsed content.

2. The multi-modal based document parsing method of claim 1, wherein, In step S2, the layout elements include text, footnotes, headings, tables, formulas, and illustrations.

3. The multi-modal based document parsing method of claim 2, wherein, In step S3, for the layout element being text, the font size, font, boldness, italic, and superscript / subscript style information of the corresponding text are also detected during layout analysis.

4. The multi-modal based document parsing method of claim 2, wherein, In step S3, for the layout element being an illustration, the content of the illustration remains unchanged and the text information in the illustration is not parsed.

5. The multi-modal based document parsing method of claim 2, wherein, In step S3, for the layout element being a table, the table format is analyzed based on the parsed text content using the large language model during parsing to ensure that the table format is consistent with that in the original document.

6. The multi-modal based document parsing method of claim 1, wherein, The document formats to be parsed include pdf,.doc,.docx,.xls,.xlsx,.ppt,.pptx, and.jpg and.png.

7. The multi-modal based document parsing method of claim 1, wherein, In step S4, for only extracting text content, the parsed content is directly output in markdown form for display.

8. The multi-modal based document parsing method of claim 1, wherein, In step S4, when both text content and layout information are needed, all parsed information is output in josn form, and the layout of the original document is restored through the parsed information.

9. The multi-modal based document parsing method of claim 1, wherein, If translation is required, in step S3, the text content is translated into the target language by prompting the large language model during parsing.

10. The multi-modal based document parsing method of claim 1, wherein, For a document to be parsed with a number of pages exceeding a preset threshold, the document to be parsed is divided into multiple documents according to a specified number of pages, and then parallel parsing is performed through a program. After parsing is completed, the parsed results are integrated and summarized in the original order before division to obtain the final parsed results.

11. The multi-modal based document parsing method of claim 1, wherein, For instant browsing and presentation requirements, the current page browsed by the user is captured. If the current page has been parsed, it is immediately presented. If the current page has not been parsed, the current page and the adjacent front and rear preset pages are immediately parsed. The page capture strategy for user browsing is as follows: if the user slides the page at a speed greater than a preset speed, the page where the user finally stops is taken as the current page for effective capture. 12.A multi-modal based document parsing system, characterized in that, The system comprises a processor capable of executing a computer program, which, when executed, can implement the multi-modal based document parsing method of any one of claims 1-11.

Images