Structured parsing method and device for a document

Abstract

The application provides a structured parsing method and device of a document, and the technical field of document processing. The structured parsing method of the document comprises: in the case that a to-be-parsed document is in a portable document format, parsing a current document page of the to-be-parsed document based on a rule engine, obtaining page meta statistical information and an initial parsing result of the current document page; determining a correction strategy of the current document page according to the page meta statistical information, the correction strategy being used to indicate whether the initial parsing result needs to be supplemented based on optical character recognition, whether the initial parsing result needs to be corrected based on a visual language model, and whether the initial parsing result needs to be re-extracted based on the visual language model; and obtaining a structured parsing result of the current document page according to the initial parsing result and the correction strategy. The application is used to improve processing efficiency while ensuring parsing accuracy.

Description

Technical Field

[0001] This application relates to the field of document processing technology, specifically to a method and apparatus for structured document parsing. Background Technology

[0002] Document structure parsing is a technology that converts unstructured or semi-structured documents into machine-readable structured data, and it is a core foundational capability for enterprise digital transformation and knowledge management.

[0003] In related technologies, document structured parsing mainly includes two categories: The first category is document parsing methods based on rule engines. These methods have high accuracy, fast processing speed, and low computational cost for extracting plain text pages; however, they often fail to extract complete or even completely fail for scanned documents, image documents, and tables containing complex merged cells, limiting their applicability. The second category is document parsing methods based on artificial intelligence models. These methods have a strong ability to understand scanned documents and complex layouts, covering scenarios that rule engines struggle to handle; however, calling visual language models on plain text pages causes unnecessary computational overhead and response delays, and the model output lacks strict structural constraints, easily leading to problems such as illusions or inconsistent formatting. Both types of methods operate independently and cannot automatically select the optimal parsing strategy based on page characteristics. In actual business scenarios, a document often contains multiple page types, including plain text pages, pages containing tables, and scanned pages, making it difficult for a single parsing engine to simultaneously achieve both accuracy and processing efficiency. Summary of the Invention

[0004] One of the purposes of this application is to provide a document parsing method and apparatus to improve processing efficiency while ensuring parsing accuracy.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows: Firstly, a structured document parsing method is provided, including: When the document to be parsed is in portable document format, the current document page of the document to be parsed is parsed based on the rule engine to obtain the page meta statistics information and initial parsing results of the current document page; The correction strategy for the current document page is determined based on the page meta statistics. The correction strategy is used to indicate whether the initial parsing result needs to supplement the text content based on optical character recognition, whether it needs to correct the image content based on visual language model, and whether it needs to re-extract the table content based on visual language model. Based on the initial parsing results and the correction strategy, obtain the structured parsing results of the current document page.

[0006] In some embodiments, obtaining page meta statistics of the current document page includes: obtaining the number of characters in the current document page, determining whether the current document page includes a table, and determining whether the current document page includes an image.

[0007] In some embodiments, determining the correction strategy for the current document page based on the page meta statistics includes: If the number of characters is less than a preset threshold and the current document page includes images, it is determined that the initial parsing result needs to supplement the text content based on optical character recognition, needs to correct the image content based on a visual language model, and does not need to re-extract the table content based on a visual language model. If the number of characters is greater than or equal to the preset threshold, a correction strategy for the current document page is determined based on whether the current document page includes a table and whether the current document page includes an image.

[0008] In some embodiments, a correction strategy for the current document page is determined based on whether the current document page includes a table and whether the current document page includes an image, including: In the case that the current document page includes tables and images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, requires correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model. In the case where the current document page includes tables but does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model. In the case where the current document page does not include tables but includes images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, requires correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model. In the case where the current document page does not include tables and does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model.

[0009] In some embodiments, where the correction strategy indicates that the initial parsing result needs to supplement the text content based on optical character recognition, needs to correct the image content based on a visual language model, and does not need to re-extract the table content based on a visual language model, the structured parsing result of the current document page is obtained according to the initial parsing result and the correction strategy, including: Perform optical character recognition on the current document page to obtain the optical character recognition results for the current document page; Based on the optical character recognition result and the initial parsing result, the intermediate parsing result is obtained; The intermediate parsing results and the page image of the current document page are input into the visual language model, and the structured parsing results of the current document page output by the visual language model are received.

[0010] In some embodiments, where the correction strategy indicates that the initial parsing result does not require supplementing the text content based on optical character recognition, requires correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model, the structured parsing result of the current document page is obtained based on the initial parsing result and the correction strategy, including: Based on the page image of the current document page and the visual language model, obtain the image description text corresponding to each image in the current document page; The image description text corresponding to each image is added to the initial parsing result to obtain the structured parsing result of the current document page.

[0011] In some embodiments, obtaining the image description text corresponding to each image in the current document page based on the page image of the current document page and the visual language model includes: Retrieve the image type corresponding to each image in the current document page; Based on the image type corresponding to each image and the preset mapping relationship, obtain the prompt words corresponding to each image. The preset mapping relationship includes the correspondence between each image type and the prompt words. Input the prompt words corresponding to each image and the page image of the current document page into the visual language model, and obtain the image description text corresponding to each image output by the visual language model.

[0012] In some embodiments, where the correction strategy indicates that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model, the structured parsing result of the current document page is obtained based on the initial parsing result and the correction strategy, including: Based on the initial parsing results, obtain the header information of each table in the current document page; The header information of each table is concatenated to form the corresponding header sequence for each table. Input the header sequence of each table and the page image of the current document page into the visual language model, and obtain the table content of each table output by the visual language model; Based on the initial parsing results and the table contents of each table output by the visual language model, the structured parsing results of the current document page are obtained.

[0013] In some embodiments, obtaining the structured parsing result of the current document page based on the initial parsing result and the table content of each table output by the visual language model includes: The content of each table output by the visual language model is validated. If the content of the first table passes the validation, the content of the first table in the initial parsing result is replaced with the content of each table output by the visual language model. If the content of the second table fails the validation, the content of the second table in the initial parsing result remains unchanged.

[0014] In some embodiments, where the correction strategy indicates that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model, the structured parsing result of the current document page is obtained based on the initial parsing result and the correction strategy, including: The initial parsing result is determined as the structured parsing result of the current document page.

[0015] In some embodiments, the method further includes: After obtaining the structured parsing result of the current document page, the page image of the current document page and the structured parsing result of the current document page are input into the visual language model, and the confidence evaluation decision output by the visual language model is obtained. The confidence evaluation decision is a first decision result used to indicate that the confidence is greater than or equal to a preset value or a second decision result used to indicate that the confidence is less than the preset value. The structured parsing results of the current document page are marked based on the confidence assessment decision.

[0016] In some embodiments, the method further includes: After obtaining the structured parsing results of all document pages of the document to be parsed, the structured parsing results of each document page of the document to be parsed are concatenated according to the page number to obtain the structured parsing result of the document to be parsed.

[0017] In some embodiments, the method further includes: If the document to be parsed is in word processing document format, spreadsheet workbook format, or presentation format, the document to be parsed is parsed according to the corresponding parsing engine to obtain the structured parsing result corresponding to the document to be parsed.

[0018] Secondly, embodiments of this application also provide a document structure parsing apparatus, comprising: The parsing unit is used to parse the current document page of the document to be parsed based on the rule engine when the document to be parsed is in portable document format, and to obtain the page meta statistics information and initial parsing results of the current document page; The determining unit is used to determine the correction strategy of the current document page based on the page meta statistics information. The correction strategy is used to indicate whether the initial parsing result needs to supplement the text content based on optical character recognition, whether it needs to correct the image content based on visual language model, and whether it needs to re-extract the table content based on visual language model. The acquisition unit is used to acquire the structured parsing result of the current document page based on the initial parsing result and the correction strategy.

[0019] In some embodiments, the parsing unit is specifically used to obtain the number of characters in the current document page, determine whether the current document page includes a table, and determine whether the current document page includes an image.

[0020] In some embodiments, the determining unit is specifically configured to, when the number of characters is less than a preset threshold and the current document page includes an image, determine that the initial parsing result requires supplementation of the text content based on optical character recognition, correction of the image content based on a visual language model, and no need to re-extract the table content based on a visual language model; and when the number of characters is greater than or equal to the preset threshold, determine a correction strategy for the current document page based on whether the current document page includes a table and whether the current document page includes an image.

[0021] In some embodiments, the determining unit is specifically used to determine, when the current document page includes tables and images, that the initial parsing result does not need to supplement the text content based on optical character recognition, needs to correct the image content based on a visual language model, and needs to re-extract the table content based on a visual language model; In the case where the current document page includes tables but does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model. In the case where the current document page does not include tables but includes images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, requires correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model. In the case where the current document page does not include tables and does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model.

[0022] In some embodiments, the acquisition unit is specifically configured to perform optical character recognition on the current document page to obtain the optical character recognition result of the current document page when the correction strategy indicates that the initial parsing result needs to supplement the text content based on optical character recognition, needs to correct the image content based on visual language model, and does not need to re-extract the table content based on visual language model; obtain intermediate parsing results based on the optical character recognition result and the initial parsing result; input the intermediate parsing result and the page image of the current document page into the visual language model, and receive the structured parsing result of the current document page output by the visual language model.

[0023] In some embodiments, the acquisition unit is specifically configured to, when the correction strategy indicates that the initial parsing result does not need to supplement the text content based on optical character recognition, needs to correct the image content based on a visual language model, and does not need to re-extract the table content based on a visual language model, acquire image description text corresponding to each image in the current document page based on the page image of the current document page and the visual language model; add the image description text corresponding to each image to the initial parsing result, and acquire the structured parsing result of the current document page.

[0024] In some embodiments, the acquisition unit is specifically used to acquire the image type corresponding to each image in the current document page; acquire the prompt word corresponding to each image according to the image type corresponding to each image and a preset mapping relationship, wherein the preset mapping relationship includes the correspondence between each image type and the prompt word; input the prompt word corresponding to each image and the page image of the current document page into the visual language model, and acquire the image description text corresponding to each image output by the visual language model.

[0025] In some embodiments, the acquisition unit is specifically configured to: acquire the header information of each table in the current document page based on the initial parsing result when the correction strategy indicates that the initial parsing result does not require supplementation of text content based on optical character recognition, does not require correction of image content based on visual language model, and requires re-extraction of table content based on visual language model; concatenate the header information of each table into a header sequence corresponding to each table; input the header sequence corresponding to each table and the page image of the current document page into the visual language model, and acquire the table content of each table output by the visual language model; and acquire the structured parsing result of the current document page based on the initial parsing result and the table content of each table output by the visual language model.

[0026] In some embodiments, the acquisition unit is further configured to verify the table content of each table output by the visual language model; if the table content of the first table passes the verification, the table content of the first table in the initial parsing result is replaced with the table content of each table output by the visual language model; if the table content of the second table fails the verification, the table content of the second table in the initial parsing result remains unchanged.

[0027] In some embodiments, the acquisition unit is specifically used to determine the initial parsing result as the structured parsing result of the current document page when the correction strategy indicates that the initial parsing result does not need to supplement the text content based on optical character recognition, does not need to correct the image content based on visual language model, and does not need to re-extract the table content based on visual language model.

[0028] In some embodiments, the acquisition unit is further configured to, after acquiring the structured parsing result of the current document page, input the page image of the current document page and the structured parsing result of the current document page into the visual language model, and acquire the confidence evaluation decision output by the visual language model, wherein the confidence evaluation decision is a first decision result indicating that the confidence is greater than or equal to a preset value or a second decision result indicating that the confidence is less than the preset value; and mark the structured parsing result of the current document page according to the confidence evaluation decision.

[0029] In some embodiments, the acquisition unit is further configured to, after acquiring the structured parsing results of all document pages of the document to be parsed, concatenate the structured parsing results of each document page of the document to be parsed according to the page number, so as to obtain the structured parsing results of the document to be parsed.

[0030] In some embodiments, the parsing unit is further configured to parse the document to be parsed according to the corresponding parsing engine when the document to be parsed is in word processing document format, spreadsheet workbook format, or presentation format, and obtain the structured parsing result corresponding to the document to be parsed.

[0031] Thirdly, embodiments of this application also provide an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program and the processor is used to cause the electronic device to implement the document structured parsing method described in any of the above claims when executing the computer program.

[0032] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the document structure parsing method described in any of the preceding claims.

[0033] Fifthly, embodiments of this application also provide a computer program product that, when run on a computer, enables the computer to implement the document structure parsing method described in any of the preceding claims.

[0034] The document structure parsing method provided in this application, when the document to be parsed is in portable document format, first parses the current document page of the document to be parsed based on a rule engine to obtain the page meta-statistics and initial parsing results of the current document page. Then, it determines the correction strategy for the current document page based on the page meta-statistics and obtains the structured parsing result of the current document page based on the initial parsing results and the correction strategy. Since the correction strategy is adaptively determined based on the page meta-statistics, it can accurately match the corresponding correction method for different page features. For pages that can be accurately parsed by the rule engine, there is no need to call optical character recognition, visual language model correction, or visual table correction, effectively avoiding unnecessary computational overhead. For pages where the rule engine extraction is insufficient, optical character recognition supplementation, visual language model correction, or visual table correction are triggered as needed to make up for the possible defects in the rule engine extraction. Therefore, it can improve the processing efficiency of document structure parsing while ensuring the accuracy of document structure parsing. Attached Figure Description

[0035] Figure 1 One of the flowcharts of the structured parsing method for the document provided in this application; Figure 2 The second flowchart of the structured parsing method for the documents provided in this application; Figure 3 The third step in the flowchart of the structured parsing method for the documents provided in this application; Figure 4 The fourth step of the flowchart for the structured parsing method of the document provided in this application; Figure 5 The fifth step in the flowchart of the structured parsing method for the documents provided in this application; Figure 6 A schematic diagram of the structured parsing device for the documents provided in this application; Figure 7 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0036] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0037] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0038] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0039] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0040] The term "at least one of..." in the specification and claims of this application means that it includes any one or more of the listed items. For example, "at least one of A, B, and C" can be: 1. including only A; 2. including only B; 3. including only C; 4. including A and B; 5. including A and C; 6. including B and C; 7. including A, B, and C.

[0041] The following is a description of the terminology used in the embodiments of this application: Document Structured Parsing: Converts unstructured or semi-structured documents (PDF, scanned documents, Office documents, etc.) into machine-readable structured data (Markdown / JSON).

[0042] Autonomous Planning Agent: An intelligent agent that autonomously decomposes, makes decisions, and schedules tasks based on the reasoning and planning capabilities of an artificial intelligence model. In the embodiments of this application, the agent autonomously decides whether to use a rule engine or a VLM / OCR correction path for each page based on features such as text length, number of tables, number of images, and text density, thereby achieving a balance between processing efficiency and effectiveness.

[0043] Rule-based parsing is a programmatic method that uses predefined rules (regular expressions, coordinate and line detection, layout analysis, etc.) rather than machine learning models for parsing. Its advantages include near 100% accuracy for plain text pages, high speed (approximately 0.1 seconds per page), and zero additional cost. Its disadvantages include incomplete or even ineffective extraction of scanned documents, image-based PDFs, and tables containing complex merged cells.

[0044] Visual Language Model (VLM): A multimodal large model (such as GPT-4V, Qwen-VL) capable of simultaneously understanding images and text, and can recognize image content to output structured results. In this application, it is used for full-text correction of scanned documents, table re-extraction, image description generation, and confidence assessment. Its advantages are: strong understanding of scanned documents and complex layouts; its disadvantages are: long processing time per page, high API cost, and susceptibility to visual illusions.

[0045] Dual-Engine Architecture: An architecture where two parsing engines, a rule engine and a visual language model, work collaboratively. The rule engine and the visual language model do not operate independently; instead, they are scheduled by an agent to achieve complementary capabilities, prioritizing the rule engine and allowing VLM / OCR to make corrections as needed.

[0046] Multimodal Large Model (VLM): A large model capable of simultaneously processing multiple modalities of input, such as text, images, audio, and video. VLM is the specific form of multimodal large model in the vision-language direction.

[0047] Hallucination: The phenomenon where a large model generates seemingly reasonable but actually erroneous / fabricated content (such as fabricated data, disordered rows and columns).

[0048] Page Classifier: A module that classifies document pages based on the features extracted by the rule engine.

[0049] This application provides a structured document parsing method, referring to... Figure 1 As shown, the structured parsing method for this document includes the following steps: S11. If the document to be parsed is in Portable Document Format (PDF) format, the current document page of the document to be parsed is parsed based on the rule engine to obtain the page meta statistics information and initial parsing results of the current document page.

[0050] The rule engine in this application embodiment refers to a parsing engine that uses regular expressions and layout analysis libraries (such as PyMuPDF, pdfplumber, PDFMiner, python-docx, etc.) to extract text, tables, and images from a document.

[0051] In some embodiments, the method of parsing the current document page of the document to be parsed based on the rule engine and obtaining the initial parsing result of the current document page may include the following steps a to d: Step a: Extract text blocks from the current document page to obtain a list of text blocks for the current document page.

[0052] Each text block in the text block list includes: coordinates (bbox), text content, and block type information.

[0053] In some embodiments, the page text extraction interface of the PDF document parsing library (PyMuPDF) can be called to obtain a list of text blocks in block mode, sort them by coordinates, and filter out entries that are not text blocks to obtain an ordered list of text blocks.

[0054] Step b: Extract tables from the current document page to obtain all tables on the current document page.

[0055] In some embodiments, the corresponding page of the PDF table extraction library (pdfplumber) can be opened, and the table detection interface can be called to detect the table area of ​​the current document page; for each detected table, its two-dimensional cell structure is parsed, and the table header and data rows are extracted; by detecting the pattern of consecutive cells sharing the same bounding box, the row number, column number, row span and column span information of the merged cells are inferred, and the merged cell structure is constructed; at the same time, the bounding box coordinates of each table are obtained.

[0056] Step c: Extract images from the current document page to obtain each image of the current document page.

[0057] In some embodiments, PyMuPDF can be used to extract embedded image resources from the page and filter out invalid images smaller than a preset number of bytes (e.g., 100 bytes), save the valid images as temporary files, and record the image reference structure (ImageRef), where ImageRef includes: position coordinates and file path.

[0058] Step d: Arrange the text block list, tables, and images of the current document page to obtain the initial parsing results of the current document page.

[0059] In some embodiments, the text block list, tables, and images of the current document page are arranged to obtain the initial parsing result of the current document page, including: arranging the text blocks and tables in an alternating manner according to the y-coordinate to ensure that the tables are embedded in the correct position on the page.

[0060] Specifically, the y0 coordinates of the text block and the y1 coordinates of the table bbox can be sorted uniformly, and the text paragraphs and text-formatted tables (Markdown) can be output alternately according to the reading order.

[0061] In some embodiments, arranging the list of text blocks, tables, and images of the current document page to obtain the initial parsing result of the current document page further includes: calculating the ratio of the intersection area of ​​the text block and the table bounding box to the area of ​​the text block, and determining text blocks whose ratio exceeds a preset threshold as text within the table area and removing them. For example, the preset threshold can be 0.7.

[0062] When the ratio exceeds a preset threshold, it indicates that the text block falls within the table area and its content is already included in the table extraction results. Therefore, removing the text block can effectively avoid the duplication of text and table content, thereby improving the accuracy of the initial parsing results.

[0063] In some embodiments, the text block list, tables, and images of the current document page are arranged to obtain the initial parsing result of the current document page, including: when the current document page is a multi-column layout, arranging the reading order of the text block list, tables, and images of the current document page based on the arrangement model to ensure that the multi-column content is output in the correct order.

[0064] In some embodiments, obtaining page meta statistics of the current document page includes: obtaining the number of characters in the current document page, determining whether the current document page includes a table, and determining whether the current document page includes an image.

[0065] It should be noted that whether the current document page includes tables can be identified by the number of tables in the current document page. If the number of tables is 0, the current document page does not include tables; if the number of tables is not 0, the current document page includes tables. Similarly, whether the current document page includes images can be identified by the number of images in the current document page. If the number of images is 0, the current document page does not include images; if the number of images is not 0, the current document page includes images.

[0066] In some embodiments, before parsing the current document page of the document to be parsed based on the rule engine, the method further includes: Determine the document format of the document to be parsed.

[0067] The document to be parsed can be in the following formats: Portable Document Format (PDF), Word document, Excel spreadsheet, PowerPoint presentation, WPS document, Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG), Tagged Image File Format (TIFF), Bitmap (BMP), HyperText Markup Language (HTML) web page document, Text File (TXT), Rich Text Format (RTF), etc.; it can also include Open Fixed-layout Document (OFD), Electronic Publication (EPUB), etc.

[0068] S12. Determine the correction strategy for the current document page based on the page metadata.

[0069] The correction strategy is used to indicate whether the initial parsing result needs to be supplemented based on Optical Character Recognition (OCR), whether it needs to be corrected based on Vision-Language Model (LVM), and whether the table content needs to be re-extracted based on the Vision-Language Model.

[0070] In some embodiments, the correction strategy may include a first syntax element (needs_ocr) indicating whether the initial parsing result needs to supplement the text content based on optical character recognition (OCR), a second syntax element (needs_vision) indicating whether the initial parsing result needs to correct the image content based on a visual language model (VLM), and a third syntax element (needs_vl_table_correction) indicating whether the initial parsing result needs to re-extract the table content based on a VLM. The first, second, and third syntax elements are represented by binarization flags. For example, the binarization flags may have values ​​of 0 or 1, or True or False.

[0071] In some embodiments, determining the correction strategy for the current document page based on the page meta statistics includes: when the number of characters is less than a preset threshold and the current document page includes images, determining that the initial parsing result needs to supplement the text content based on optical character recognition, needs to correct the image content based on a visual language model, and does not need to re-extract the table content based on a visual language model.

[0072] For example, the preset threshold can be 50, 100, etc.

[0073] If the number of characters on the current document page is less than a preset threshold and includes images, then the current document page can be determined to be a scanned page. The content of a scanned page exists in image form, and the rule engine cannot directly extract the text from it. This is manifested by a very small number of extracted characters and the page containing an entire image. For such pages, the text must first be extracted using optical character recognition (OCR), and then the recognition result must be corrected using a visual language model. Since the entire page content requires visual processing, there is no need to trigger visual table correction separately. Accordingly, the initial parsing result needs to supplement the text content based on OCR, correct the image content based on the visual language model, and does not need to re-extract the table content based on the visual language model.

[0074] In some embodiments, determining the correction strategy for the current document page based on the page meta statistics includes: when the number of characters is greater than or equal to the preset threshold, determining the correction strategy for the current document page based on whether the current document page includes a table and whether the current document page includes an image.

[0075] That is, if text_len < preset threshold and image_count > 0, then the current document page is determined to be a scanned page (SCANNED), and needs_vision=True, needs_ocr=True, and needs_vl_table_correction=False are set.

[0076] If the number of characters on the current document page is greater than or equal to the preset threshold, it indicates that the rule engine has been able to extract valid text content from the page, and the page does not belong to the scanned document page. At this time, different page types can be further divided according to whether the page contains tables or images, and corresponding correction strategies can be matched: In pages containing tables, the rule engine may miss the extraction of complex merged cell tables, and visual table correction needs to be triggered; In pages containing images, embedded images lack descriptive information, and image descriptions need to be generated with the help of visual language models or content correction needs to be performed; For plain text pages that contain neither tables nor images, the extraction results of the rule engine are accurate enough, and no visual correction is required.

[0077] In some embodiments, a correction strategy for the current document page is determined based on whether the current document page includes a table and whether the current document page includes an image, including: When the current document page includes tables and images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, requires correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model.

[0078] That is, if text_len > preset threshold, and table_count > 0 and image_count > 0, then it is determined to be a mixed image and table page (MIXED), and needs_vision = True, needs_ocr = False, and needs_vl_table_correction = True are set.

[0079] In some embodiments, a correction strategy for the current document page is determined based on whether the current document page includes a table and whether the current document page includes an image, including: In the case where the current document page includes tables but does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model.

[0080] That is, if text_len > preset threshold, and table_count > 0 and image_count = 0, then it is determined to be a table mixed page (MIXED), and needs_vision = False, needs_ocr = False, and needs_vl_table_correction = True.

[0081] In some embodiments, a correction strategy for the current document page is determined based on whether the current document page includes a table and whether the current document page includes an image, including: In the case where the current document page does not include tables but includes images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, requires correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model.

[0082] That is, if text_len > preset threshold, and table_count = 0 and image_count > 0, then it is determined to be an image mixed page (MIXED), and needs_vision = True, needs_ocr = False, and needs_vl_table_correction = False are set.

[0083] In some embodiments, a correction strategy for the current document page is determined based on whether the current document page includes a table and whether the current document page includes an image, including: In the case where the current document page does not include tables and does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model.

[0084] That is, if text_len > preset threshold, and table_count = 0 and image_count = 0, then it is determined to be plain text (TEXT), and needs_vision = False, needs_ocr = False, and needs_vl_table_correction = False are set.

[0085] In some embodiments, the above-described structured document parsing method further includes: calculating the text density of the current document page. The text density of the current document page is the ratio of the number of text characters on the current document page to the page pixel area.

[0086] S13. Based on the initial parsing result and the correction strategy, obtain the structured parsing result of the current document page.

[0087] That is, depending on whether the initial parsing result needs to supplement the text content based on optical character recognition, whether it needs to correct the image content based on visual language model, and whether it needs to re-extract the table content based on visual language model, the corresponding correction operation is selectively performed to supplement and improve the initial parsing result, thereby obtaining the structured parsing result of the current document page.

[0088] In some embodiments, obtaining the structured parsing result of the current document page based on the initial parsing result and the correction strategy includes: if the correction strategy indicates that the initial parsing result needs to supplement the text content based on optical character recognition, then supplement the text content of the initial parsing result based on optical character recognition; if the correction strategy indicates that the initial parsing result needs to correct the image content based on a visual language model, then correct the image content of the initial parsing result based on a visual language model; if the correction strategy indicates that the initial parsing result needs to correct the image content based on a visual language model, then re-extract the table content of the current document page based on a visual language model.

[0089] In some embodiments, the document structure parsing method provided in the above embodiments further includes: after obtaining the structure parsing results of all document pages of the document to be parsed, concatenating the structure parsing results of each document page of the document to be parsed according to the page number to obtain the structure parsing result of the document to be parsed.

[0090] In some embodiments, the document to be parsed in this application can be application materials submitted by users in financial transactions. By using the document structured parsing method provided in this application to perform structured parsing on the application materials submitted by users, structured fields such as the applicant's identity information, income information, asset information, liability information, and credit information can be obtained, which can then be used for credit assessment and credit approval of the applicant.

[0091] In some embodiments, the document to be parsed in this application can be a contract text to be reviewed in legal business. By using the document structure parsing method provided in this application to perform structured parsing on the contract text, structured fields such as contract subject, contract amount, performance period, liability for breach of contract, and dispute resolution clauses can be obtained, which can then be used for contract risk review and batch comparison of clauses.

[0092] In some embodiments, the document to be parsed in this application can be a medical record or examination report in medical practice. By using the document structure parsing method provided in this application to perform structured parsing on the medical record or examination report, structured fields such as chief complaint, present medical history, diagnostic conclusion, examination indicators, and medication records can be obtained, which can then be used to assist in diagnosis and treatment and clinical data statistics.

[0093] In some embodiments, the document to be parsed in this application can be a resume submitted by a job seeker in a recruitment process. By using the document structure parsing method provided in this application to parse the resume in a structured manner, structured fields such as education experience, work experience, skill tags, and expected salary can be obtained, which can then be used for automatic initial screening of resumes and the construction of a talent pool.

[0094] The document structure parsing method provided in this application, when the document to be parsed is in portable document format, first parses the current document page of the document to be parsed based on a rule engine to obtain the page meta-statistics and initial parsing results of the current document page. Then, it determines the correction strategy for the current document page based on the page meta-statistics and obtains the structured parsing result of the current document page based on the initial parsing results and the correction strategy. Since the correction strategy is adaptively determined based on the page meta-statistics, it can accurately match the corresponding correction method for different page features. For pages that can be accurately parsed by the rule engine, there is no need to call optical character recognition, visual language model correction, or visual table correction, effectively avoiding unnecessary computational overhead. For pages where the rule engine extraction is insufficient, optical character recognition supplementation, visual language model correction, or visual table correction are triggered as needed to make up for the possible defects in the rule engine extraction. Therefore, it can improve the processing efficiency of document structure parsing while ensuring the accuracy of document structure parsing.

[0095] As an extension and refinement of the above embodiments, this application provides another method for structured document parsing, referring to... Figure 2 As shown, the structured parsing method for this document includes the following steps: S201. If the document to be parsed is in portable document format, the current document page of the document to be parsed is parsed based on the rule engine to obtain the initial parsing result of the current document page, the number of characters in the current document page, whether the current document page includes a table, and whether the current document page includes an image.

[0096] The implementation of step S201 can refer to step S11 above. To avoid redundancy, it will not be repeated here.

[0097] If the number of characters is less than a preset threshold, and the current document page includes images, then the following steps are performed: S202. It is determined that the initial parsing result requires supplementing the text content based on optical character recognition, correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model.

[0098] S203. Perform optical character recognition on the current document page to obtain the optical character recognition result of the current document page.

[0099] In some embodiments, an optical character recognition (OCR) tool library can be invoked via an Application Programming Interface (API) to perform OCR on the page image of the current document page in order to obtain the OCR results for the current document page.

[0100] For example, the page image of the current document page can be an image obtained by taking a screenshot of the current document page, or it can be an image obtained by rendering the current document page.

[0101] S204. Based on the optical character recognition result and the initial parsing result, obtain the intermediate parsing result.

[0102] In some embodiments, obtaining intermediate parsing results based on the optical character recognition result and the initial parsing result includes: if the initial parsing result extracted by the rule engine is empty, directly using the optical character recognition result as the intermediate parsing result; if the initial parsing result extracted by the rule engine is not empty, then supplementing the initial parsing result with the optical character recognition result to obtain the intermediate parsing result.

[0103] In some embodiments, intermediate parsing results include: line-by-line text content and confidence level.

[0104] S205. Input the intermediate parsing results and the page image of the current document page into the visual language model, and receive the structured parsing results of the current document page output by the visual language model.

[0105] In some embodiments, the page correction function of the visual language model can be invoked, and the page image of the current document page and the intermediate parsing results can be input into the visual language model. The visual language model verifies and corrects the preliminary extraction results based on the page image, and then outputs the corrected complete structured parsing results.

[0106] As an extension and refinement of the above embodiments, this application provides another method for structured document parsing, referring to... Figure 3 As shown, the structured parsing method for this document includes the following steps: S301. If the document to be parsed is in portable document format, the current document page of the document to be parsed is parsed based on the rule engine to obtain the initial parsing result of the current document page, the number of characters in the current document page, whether the current document page includes a table, and whether the current document page includes an image.

[0107] The implementation of step S301 can refer to step S11 above. To avoid redundancy, it will not be repeated here.

[0108] If the number of characters is greater than or equal to a preset threshold, and the current document page does not include tables but includes images, then the following steps are performed: S302. Based on the page image of the current document page and the visual language model, obtain the image description text corresponding to each image in the current document page.

[0109] In some embodiments, based on the page image of the current document page and the visual language model, image description text corresponding to each image in the current document page is obtained, including the following steps a to d: Step a: Obtain the image type corresponding to each image in the current document page.

[0110] For example, the image type can be: chart, flowchart, photo, etc.

[0111] Step b: Based on the image type and preset mapping relationship of each image, obtain the prompt words corresponding to each image.

[0112] The preset mapping relationship includes the correspondence between various image types and prompt words.

[0113] For example, the prompts corresponding to the charts could be as follows: You are a professional document table extraction expert. Below is a screenshot of a document page that contains a table.

[0114] Please carefully identify all the tables in the image and output the complete table content.

[0115] Require: 1. Strictly follow the table structure in the image to extract the content of each row and each column. 2. Merged cells must be rendered using the HTML table tag, with the rowspan / colspan attributes used to indicate the merging relationship. - Example: "Economy" is displayed vertically across 2 lines. - Example: Total means "Total" spans 3 columns horizontally. 3. Simple tables without merged cells can still use Markdown pipe table formatting. 4. The header row must be complete, and each column must contain content. 5. Maintain the original content of the text in the table; do not delete or rewrite it. 6. If the table has category columns such as "Area" or "Category," you must fill in the category value for each cell (e.g., "Economy," "Policy," "Market," "Competitors," etc.). These columns usually require vertical merging. 7. If a page contains both text paragraphs and tables, the text paragraphs should be retained, and the tables should be embedded in their correct positions on the page. 8. If there are multiple tables on a page, output each table separately. 9. Output format: Maintain the original reading order of the page; use plain text for text paragraphs. 10. Do not output any explanatory text; only output the document content itself.

[0116] Step c: Input the prompt words corresponding to each image and the page image of the current document page into the visual language model, and obtain the image description text corresponding to each image output by the visual language model.

[0117] S303. Add the image description text corresponding to each image to the initial parsing result to obtain the structured parsing result of the current document page.

[0118] In some embodiments, the image description text corresponding to each image is added to the initial parsing result to obtain the structured parsing result of the current document page. This includes: determining the insertion position of each image description text in the text stream of the initial parsing result based on the position coordinates of each image in the current document page, and adding the corresponding image description text to the initial parsing result according to the insertion position, thereby ensuring that the image description is consistent with the page reading order.

[0119] In some embodiments, adding the image description text corresponding to each image to the initial parsing result to obtain the structured parsing result of the current document page includes: uniformly adding the image description text corresponding to each image to the end of the initial parsing result of the current document page to form a complete structured parsing result containing image descriptions.

[0120] As an extension and refinement of the above embodiments, this application provides another method for structured document parsing, referring to... Figure 4 As shown, the structured parsing method for this document includes the following steps: S401. If the document to be parsed is in portable document format, the current document page of the document to be parsed is parsed based on the rule engine to obtain the initial parsing result of the current document page, the number of characters in the current document page, whether the current document page includes a table, and whether the current document page includes an image.

[0121] The implementation of step S401 can refer to step S11 above. To avoid redundancy, it will not be repeated here.

[0122] If the number of characters is greater than or equal to a preset threshold, and the current document page includes tables but does not include images, then the following steps are performed: S402. Obtain the header information of each table in the current document page based on the initial parsing result.

[0123] For example, the header information of a table may include: "Economy", "Growth Rate", and "Investment Amount".

[0124] S403. Concatenate the header information of each table to form the corresponding header sequence for each table.

[0125] In some embodiments, the header information of each table is concatenated into a header sequence corresponding to each table, including: concatenating the header information of the first table sequentially to obtain the header sequence corresponding to the first table. The first table is any table in the current document.

[0126] For example, the header sequence of a table can be obtained by concatenating the header information: "Economy|Growth Rate|Investment Amount".

[0127] S404. Input the header sequence of each table and the page image of the current document page into the visual language model, and obtain the table content of each table output by the visual language model.

[0128] In some embodiments, the image table extraction function of the visual language model can be invoked to input the page image of the current document page and the header sequence of each table into the visual language model, and the visual language model outputs Markdown or HTML format text containing the complete table structure.

[0129] S405. Based on the initial parsing results and the table contents of each table output by the visual language model, obtain the structured parsing results of the current document page.

[0130] In some embodiments, obtaining the structured parsing result of the current document page based on the initial parsing result and the table content of each table output by the visual language model includes: replacing the content of each table in the initial parsing result with the table content of each table output by the visual language model, thereby obtaining the structured parsing result of the current document page.

[0131] In other embodiments, the structured parsing result of the current document page is obtained based on the initial parsing result and the table content of each table output by the visual language model, including the following steps 1 to 3: Step 1: Verify the content of each table output by the visual language model.

[0132] In some embodiments, validating the table content of each table output by the visual language model includes: a process for validating the table content of each table output by the visual language model, comprising: parsing the table text output by the visual language model to obtain a list of structured table data, each table data containing a header and data rows; performing the following validation items on each structured table data, wherein if all are satisfied, the validation is deemed successful, and if any item is not satisfied, the validation is deemed unsuccessful: Column consistency: The number of columns in the table is consistent with the number of columns in the reference header, which is derived from the header information of the corresponding table extracted by the rule engine; Reasonable number of columns per row: The actual number of columns in each row is within a reasonable range, taking into account the number of columns covered by merged cells when calculating the number of columns; The number of rows is reasonable: there is no obvious missing rows in the table compared to the corresponding number of rows extracted by the rule engine.

[0133] Step 2: If the content of the first table passes the verification, then replace the content of the first table in the initial parsing result with the content of each table output by the visual language model.

[0134] Step 3: If the content of the second table fails the validation, keep the content of the second table unchanged in the initial parsing result.

[0135] Since the above embodiments can verify the table content of each table output by the visual language model, and replace the corresponding table in the initial parsing result with the table that passes the verification, and keep the corresponding table in the initial parsing result unchanged for the table that fails the verification, the above embodiments can give full play to the advantage of the visual language model's strong ability to understand complex table structures and ensure the quality of table extraction, while avoiding the impact of the uncertain output of the visual language model on the quality of downstream data, and ensuring the stability and reliability of the parsing results.

[0136] As an extension and refinement of the above embodiments, this application provides another method for structured document parsing, referring to... Figure 5 As shown, the structured parsing method for this document includes the following steps: S501. If the document to be parsed is in portable document format, the current document page of the document to be parsed is parsed based on the rule engine to obtain the initial parsing result of the current document page, the number of characters in the current document page, whether the current document page includes a table, and whether the current document page includes an image.

[0137] The implementation of step S501 can refer to step S11 above. To avoid redundancy, it will not be repeated here.

[0138] If the number of characters is greater than or equal to a preset threshold, and the current document page does not contain tables or images, then the following steps are performed: S502. The initial parsing result is determined as the structured parsing result of the current document page.

[0139] Since the current document page is a plain text page when the number of characters is greater than or equal to the preset threshold and does not include tables or images, the initial parsing result extracted by the rule engine has sufficient accuracy and does not require additional visual correction processing. Therefore, the initial parsing result is directly determined as the structured parsing result of the current document page, which can effectively avoid unnecessary calls to visual language models or optical character recognition, thereby saving computing resources, improving processing efficiency and reducing parsing costs.

[0140] In some embodiments, if the number of characters is greater than or equal to a preset threshold, and the current document page includes tables and images, then the above steps S302, S303 and steps S402 to S405 are executed to obtain the structured parsing result of the current document page.

[0141] It should be noted that there is a specific execution order between steps S302 and S303, and between steps S402 and S405; however, the execution order between steps S302 and S303 and between steps S402 and S405 is not limited. Steps S302 and S303 can be executed first, followed by steps S402 and S405; or steps S402 and S405 can be executed first, followed by steps S302 and S303; or steps S302 and S303 and steps S402 and S405 can be executed in parallel.

[0142] In some embodiments, if the structured parsing result of the current document page has undergone at least one of optical character recognition text supplementation, visual language model image correction, and visual language model table re-extraction, the above-mentioned document structured parsing method further includes: after obtaining the structured parsing result of the current document page, inputting the page image of the current document page and the structured parsing result of the current document page into the visual language model, and obtaining the confidence evaluation decision output by the visual language model, wherein the confidence evaluation decision is a first decision result indicating that the confidence is greater than or equal to a preset value or a second decision result indicating that the confidence is less than the preset value; and marking the structured parsing result of the current document page according to the confidence evaluation decision.

[0143] For example, the first decision result is CONFIDENT, and the second decision result is UNCERTAIN.

[0144] In the above embodiments, when the structured parsing result of the current document page has undergone at least one of the following processing steps—optical character recognition text supplementation, visual language model image correction, and visual language model table re-extraction—after obtaining the structured parsing result of the current document page, the page image of the current document page and the structured parsing result of the current document page are input into the visual language model, and the confidence evaluation decision output by the visual language model is obtained. The structured parsing result of the current document page is then marked according to the confidence evaluation decision. Therefore, the above embodiments can provide a quality reference for the extraction results for downstream applications, making it easier to determine whether manual review is required.

[0145] In some embodiments, the above-mentioned document structured parsing method further includes: when the document to be parsed is a word processing document format (DOCX), a spreadsheet workbook format (Excel), or a presentation format (PPT), parsing the document to be parsed using the corresponding parsing engine to obtain the structured parsing result corresponding to the document to be parsed.

[0146] In other words, for DOCX files, using python-docx to extract paragraphs and tables allows the rule engine to complete the full parsing; for Excel files, the openpyxl rule engine reads them directly with nearly 100% accuracy, requiring no VLM / OCR correction, and the page classifier automatically identifies them as TEXT type and skips the correction step; for PPT files, using python-pptx to extract text and images also requires no VLM correction.

[0147] It should be noted that even when the document to be parsed is in another format, it can still be parsed using the corresponding parsing method to obtain the structured parsing result. For example, the document to be parsed can also be in image format, HyperText Markup Language (HTML) format, plain text (Text, TXT) format, rich text format (RTF) format, open fixed-layout document (OFD) format, electronic publication (EPUB) format, etc. The embodiments of this application can use the parsing method corresponding to each format to parse the document to be parsed, and the embodiments of this application do not limit the specific parsing method.

[0148] In some embodiments, the above formats can be accessed through a unified BaseConverter abstract interface and CONVERTER_MAP extended mapping table to achieve a unified parsing entry point for multi-format documents.

[0149] The above embodiments define a unified parsing specification through the BaseConverter abstract interface and complete the mapping relationship between format types and parsers by combining the CONVERTER_MAP extended mapping table. This enables multiple formats to be connected to the dual-engine architecture in a consistent manner, providing a unified parsing entry point to the outside world. This not only shields the differences in the underlying parsing logic of different formats and reduces the usage cost for callers, but also decouples the format parsing logic from the main scheduling process. When adding new format support, it is only necessary to implement the abstract interface and register the mapping relationship, which significantly improves the scalability and maintainability of the system.

[0150] Based on the same inventive concept, as an implementation of the above method, this application embodiment also provides a document structure parsing device. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the document structure parsing device in this embodiment can correspondingly implement all the contents of the aforementioned method embodiment.

[0151] This application provides a structured document parsing apparatus. Figure 6 This is a schematic diagram of the structure of the document's structured parsing device, such as... Figure 6 As shown, the document structure parsing device 600 includes: The parsing unit 61 is used to parse the current document page of the document to be parsed based on the rule engine when the document to be parsed is in portable document format, and to obtain the page meta statistics information and initial parsing results of the current document page; The determining unit 62 is used to determine the correction strategy of the current document page based on the page meta statistics information. The correction strategy is used to indicate whether the initial parsing result needs to supplement the text content based on optical character recognition, whether it needs to correct the image content based on visual language model, and whether it needs to re-extract the table content based on visual language model. The acquisition unit 63 is used to acquire the structured parsing result of the current document page based on the initial parsing result and the correction strategy.

[0152] In some embodiments, the parsing unit 61 is specifically used to obtain the number of characters in the current document page, determine whether the current document page includes a table, and determine whether the current document page includes an image.

[0153] In some embodiments, the determining unit 62 is specifically used to determine, when the number of characters is less than a preset threshold and the current document page includes an image, that the initial parsing result needs to supplement the text content based on optical character recognition, needs to correct the image content based on a visual language model, and does not need to re-extract the table content based on a visual language model; and when the number of characters is greater than or equal to the preset threshold, to determine the correction strategy for the current document page based on whether the current document page includes a table and whether the current document page includes an image.

[0154] In some embodiments, the determining unit 62 is specifically used to determine, when the current document page includes tables and images, that the initial parsing result does not need to supplement the text content based on optical character recognition, needs to correct the image content based on a visual language model, and needs to re-extract the table content based on a visual language model; In the case where the current document page includes tables but does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model. In the case where the current document page does not include tables but includes images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, requires correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model. In the case where the current document page does not include tables and does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model.

[0155] In some embodiments, the acquisition unit 63 is specifically configured to perform optical character recognition on the current document page to obtain the optical character recognition result of the current document page when the correction strategy indicates that the initial parsing result needs to supplement the text content based on optical character recognition, needs to correct the image content based on visual language model, and does not need to re-extract the table content based on visual language model; obtain intermediate parsing results based on the optical character recognition result and the initial parsing result; input the intermediate parsing result and the page image of the current document page into the visual language model, and receive the structured parsing result of the current document page output by the visual language model.

[0156] In some embodiments, the acquisition unit 63 is specifically configured to, when the correction strategy indicates that the initial parsing result does not need to supplement the text content based on optical character recognition, needs to correct the image content based on visual language model, and does not need to re-extract the table content based on visual language model, acquire the image description text corresponding to each image in the current document page based on the page image of the current document page and the visual language model; add the image description text corresponding to each image to the initial parsing result, and acquire the structured parsing result of the current document page.

[0157] In some embodiments, the acquisition unit 63 is specifically used to acquire the image type corresponding to each image in the current document page; acquire the prompt word corresponding to each image according to the image type corresponding to each image and a preset mapping relationship, wherein the preset mapping relationship includes the correspondence between each image type and the prompt word; input the prompt word corresponding to each image and the page image of the current document page into the visual language model, and acquire the image description text corresponding to each image output by the visual language model.

[0158] In some embodiments, the acquisition unit 63 is specifically configured to: acquire the header information of each table in the current document page based on the initial parsing result when the correction strategy indicates that the initial parsing result does not require supplementation of text content based on optical character recognition, does not require correction of image content based on visual language model, and requires re-extraction of table content based on visual language model; concatenate the header information of each table into a header sequence corresponding to each table; input the header sequence corresponding to each table and the page image of the current document page into the visual language model, and acquire the table content of each table output by the visual language model; and acquire the structured parsing result of the current document page based on the initial parsing result and the table content of each table output by the visual language model.

[0159] In some embodiments, the acquisition unit 63 is further configured to verify the table content of each table output by the visual language model; if the table content of the first table passes the verification, the table content of the first table in the initial parsing result is replaced with the table content of each table output by the visual language model; if the table content of the second table fails the verification, the table content of the second table in the initial parsing result remains unchanged.

[0160] In some embodiments, the acquisition unit 63 is specifically used to determine the initial parsing result as the structured parsing result of the current document page when the correction strategy indicates that the initial parsing result does not need to supplement the text content based on optical character recognition, does not need to correct the image content based on visual language model, and does not need to re-extract the table content based on visual language model.

[0161] In some embodiments, the acquisition unit 63 is further configured to, after acquiring the structured parsing result of the current document page, input the page image of the current document page and the structured parsing result of the current document page into the visual language model, and acquire the confidence evaluation decision output by the visual language model, wherein the confidence evaluation decision is a first decision result indicating that the confidence is greater than or equal to a preset value or a second decision result indicating that the confidence is less than the preset value; and mark the structured parsing result of the current document page according to the confidence evaluation decision.

[0162] In some embodiments, the acquisition unit 63 is further configured to, after acquiring the structured parsing results of all document pages of the document to be parsed, concatenate the structured parsing results of each document page of the document to be parsed according to the page number, so as to obtain the structured parsing results of the document to be parsed.

[0163] In some embodiments, the parsing unit 61 is further configured to parse the document to be parsed according to the corresponding parsing engine when the document to be parsed is in word processing document format, spreadsheet workbook format, or presentation format, and obtain the structured parsing result corresponding to the document to be parsed.

[0164] Based on the same inventive concept, embodiments of this application also provide an electronic device. Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 7 As shown, the electronic device provided in this embodiment includes a memory 701 and a processor 702. The memory 701 is used to store a computer program, and the processor 702 is used to execute the structured document parsing method provided in the above embodiment when executing the computer program.

[0165] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a computing device, enables the computing device to implement the document structure parsing method described in any of the above embodiments.

[0166] Based on the same inventive concept, this application provides a computer program product that, when run on a computer, enables the computer to implement the document structure parsing method described in any of the above embodiments.

[0167] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0168] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0169] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0170] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A structured document parsing method, characterized in that, include: When the document to be parsed is in portable document format, the current document page of the document to be parsed is parsed based on the rule engine to obtain the page meta statistics information and initial parsing results of the current document page; The correction strategy for the current document page is determined based on the page meta statistics. The correction strategy is used to indicate whether the initial parsing result needs to supplement the text content based on optical character recognition, whether it needs to correct the image content based on visual language model, and whether it needs to re-extract the table content based on visual language model. Based on the initial parsing results and the correction strategy, obtain the structured parsing results of the current document page.

2. The method according to claim 1, characterized in that, Obtain page metadata for the current document page, including: the number of characters on the current document page, whether the current document page includes tables, and whether the current document page includes images.

3. The method according to claim 2, characterized in that, The step of determining the correction strategy for the current document page based on the page metadata includes: If the number of characters is less than a preset threshold and the current document page includes images, it is determined that the initial parsing result needs to supplement the text content based on optical character recognition, needs to correct the image content based on a visual language model, and does not need to re-extract the table content based on a visual language model. If the number of characters is greater than or equal to the preset threshold, a correction strategy for the current document page is determined based on whether the current document page includes a table and whether the current document page includes an image.

4. The method according to claim 3, characterized in that, The correction strategy for the current document page is determined based on whether it contains tables and images, including: In the case that the current document page includes tables and images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, requires correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model. In the case where the current document page includes tables but does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model. In the case where the current document page does not include tables but includes images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, requires correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model. In the case where the current document page does not include tables and does not include images, it is determined that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model.

5. The method according to claim 1, characterized in that, When the correction strategy indicates that the initial parsing result needs to supplement the text content based on optical character recognition, needs to correct the image content based on a visual language model, and does not need to re-extract the table content based on a visual language model, the structured parsing result of the current document page is obtained according to the initial parsing result and the correction strategy, including: Perform optical character recognition on the current document page to obtain the optical character recognition results for the current document page; Based on the optical character recognition result and the initial parsing result, the intermediate parsing result is obtained; The intermediate parsing results and the page image of the current document page are input into the visual language model, and the structured parsing results of the current document page output by the visual language model are received.

6. The method according to claim 1, characterized in that, When the correction strategy indicates that the initial parsing result does not require supplementing the text content based on optical character recognition, but requires correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model, the structured parsing result of the current document page is obtained based on the initial parsing result and the correction strategy, including: Based on the page image of the current document page and the visual language model, obtain the image description text corresponding to each image in the current document page; The image description text corresponding to each image is added to the initial parsing result to obtain the structured parsing result of the current document page.

7. The method according to claim 6, characterized in that, The step of obtaining image description text corresponding to each image in the current document page based on the page image of the current document page and the visual language model includes: Retrieve the image type corresponding to each image in the current document page; Based on the image type corresponding to each image and the preset mapping relationship, obtain the prompt words corresponding to each image. The preset mapping relationship includes the correspondence between each image type and the prompt words. Input the prompt words corresponding to each image and the page image of the current document page into the visual language model, and obtain the image description text corresponding to each image output by the visual language model.

8. The method according to claim 1, characterized in that, When the correction strategy indicates that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and requires re-extracting the table content based on a visual language model, the structured parsing result of the current document page is obtained based on the initial parsing result and the correction strategy, including: Based on the initial parsing results, obtain the header information of each table in the current document page; The header information of each table is concatenated to form the corresponding header sequence for each table. Input the header sequence of each table and the page image of the current document page into the visual language model, and obtain the table content of each table output by the visual language model; Based on the initial parsing results and the table contents of each table output by the visual language model, the structured parsing results of the current document page are obtained.

9. The method according to claim 8, characterized in that, The step of obtaining the structured parsing result of the current document page based on the initial parsing result and the table content of each table output by the visual language model includes: The content of each table output by the visual language model is validated. If the content of the first table passes the validation, the content of the first table in the initial parsing result is replaced with the content of each table output by the visual language model. If the content of the second table fails the validation, the content of the second table in the initial parsing result remains unchanged.

10. The method according to claim 1, characterized in that, When the correction strategy indicates that the initial parsing result does not require supplementing the text content based on optical character recognition, does not require correcting the image content based on a visual language model, and does not require re-extracting the table content based on a visual language model, the structured parsing result of the current document page is obtained based on the initial parsing result and the correction strategy, including: The initial parsing result is determined as the structured parsing result of the current document page.

11. The method according to any one of claims 5-9, characterized in that, The method further includes: After obtaining the structured parsing result of the current document page, the page image of the current document page and the structured parsing result of the current document page are input into the visual language model, and the confidence evaluation decision output by the visual language model is obtained. The confidence evaluation decision is a first decision result used to indicate that the confidence is greater than or equal to a preset value or a second decision result used to indicate that the confidence is less than the preset value. The structured parsing results of the current document page are marked based on the confidence assessment decision.

12. The method according to claim 8, characterized in that, The method further includes: After obtaining the structured parsing results of all document pages of the document to be parsed, the structured parsing results of each document page of the document to be parsed are concatenated according to the page number to obtain the structured parsing result of the document to be parsed.

13. The method according to claim 1, characterized in that, The method further includes: If the document to be parsed is in word processing document format, spreadsheet workbook format, or presentation format, the document to be parsed is parsed according to the corresponding parsing engine to obtain the structured parsing result corresponding to the document to be parsed.

14. A structured document parsing apparatus, characterized in that, include: The parsing unit is used to parse the current document page of the document to be parsed based on the rule engine when the document to be parsed is in portable document format, and to obtain the page meta statistics information and initial parsing results of the current document page; The determining unit is used to determine the correction strategy of the current document page based on the page meta statistics information. The correction strategy is used to indicate whether the initial parsing result needs to supplement the text content based on optical character recognition, whether it needs to correct the image content based on visual language model, and whether it needs to re-extract the table content based on visual language model. The acquisition unit is used to acquire the structured parsing result of the current document page based on the initial parsing result and the correction strategy.

15. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program and the processor being used to cause the electronic device to implement the document structure parsing method according to any one of claims 1-13 when executing the computer program.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the document structure parsing method according to any one of claims 1-13.

17. A computer program product, characterized in that, When the computer program product is run on a computer, the computer implements the document structure parsing method according to any one of claims 1-13.

Images