Multi-format document analysis method and device, computer equipment and storage medium
By using global layout analysis and multi-resolution image processing, the problem of garbled characters in the parsing of diverse documents was solved, enabling precise positioning and efficient structured reorganization of document elements, thus improving the accuracy and efficiency of document parsing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING SOKON IND GRP CO LTD
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-24
AI Technical Summary
Existing document parsing technologies are prone to garbled characters when dealing with diverse documents, especially when handling complex text and image layouts, complex tables, and independently stored images, making it difficult to achieve efficient and structured multimodal deep understanding.
By converting documents into images, global layout analysis is performed to obtain location information and hierarchical relationships. Combining low-resolution and high-resolution image processing, target regions of elements to be processed are identified and extracted, descriptive information is generated, and document content is reorganized according to location and hierarchical relationships.
It enables precise location and accurate content extraction of documents in multiple formats, improving the accuracy and efficiency of document parsing and ensuring the integrity and logical order of the document structure.
Smart Images
Figure CN121921799A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of document processing technology, and in particular to a method, apparatus, computer device, and storage medium for parsing multi-format documents. Background Technology
[0002] In today's information processing field, automated parsing and deep understanding of multi-format documents are key to improving data processing efficiency. Currently, the mainstream document parsing technologies can be broadly categorized into two types: modular pipeline systems and end-to-end multimodal large-scale models.
[0003] Modular pipeline systems use multiple dedicated modules, such as text detection, character recognition, layout analysis, and table detection, to process documents in stages. However, this approach is complex to integrate, requires frequent data format conversions, and is prone to information fragmentation and semantic loss. Furthermore, each module typically targets a specific document type, making it susceptible to crashes and garbled text when dealing with diverse and unconventional documents. Summary of the Invention
[0004] Based on this, a method, apparatus, computer device, and storage medium for parsing multi-format documents are provided to solve the problem that existing document parsing methods are prone to garbled characters when dealing with diverse documents.
[0005] In a first aspect, the present invention provides a method for parsing multi-format documents, the method comprising: For the first type of document, the modality of the first type of document is converted into an image to obtain the image to be processed, wherein the first type of document includes images and text; Using a preset layout analysis model, a global layout analysis is performed on the image to be processed to obtain global layout information; wherein, the global layout information includes the position information and hierarchical relationship of the elements to be processed; Based on the location information, a target region containing the element to be processed is determined in the image to be processed, and descriptive information of the element to be processed is generated; Based on the location information and the hierarchical relationship, the description information is reorganized to obtain the parsed document.
[0006] Optionally, before converting the modality of the first type of document into an image for the first type of document, the method further includes: determining the type of the received document; the type includes a first type and a second type; for the second type of document, extracting the structured information and content text of the second type of document; wherein the second type of document does not include images; converting the structured information into plain text format data, and merging the plain text format data with the content text to obtain the parsed document.
[0007] Optionally, the image to be processed includes a first resolution image; the resolution of the first resolution image is lower than a preset resolution threshold; the step of performing global layout analysis on the image to be processed using a preset layout analysis model to obtain the global layout information includes: performing layout analysis on the first resolution image using a first layout analysis model to obtain preliminary layout information, the preliminary layout information including the border of the area corresponding to the element to be processed, and the type corresponding to each element to be processed; extracting text from the first resolution image using a text recognition model to obtain text fragments, and a set of text fragments composed of multiple text fragments; and analyzing the global layout of the first resolution image based on the preliminary layout information and the set of text fragments using a second layout analysis model to obtain the global layout information.
[0008] Optionally, the step of analyzing the global layout of the first resolution image based on the preliminary layout information and the set of text fragments using the second layout analysis model to obtain the global layout information includes: extracting visual features from the first resolution image; converting text fragments in the set of text fragments into geometric channels and sparse semantic features; fusing the preliminary layout information, the visual features, the geometric channels, and the sparse semantic features to obtain multimodal fusion features; and analyzing the multimodal fusion features using the second layout analysis model to obtain the global layout information.
[0009] Optionally, the image to be processed further includes a second resolution image, the second resolution image having a higher resolution than the first resolution image; the elements to be processed include text elements, image elements, table elements, and formula elements; the step of determining the target region of the element to be processed in the image to be processed based on the position information and generating descriptive information of the element to be processed further includes: extracting the target table region of the table element from the second resolution image according to the position information of the table element; performing grid structure detection on the table elements in the target table region to determine the grid topology information of the table elements; extracting the text content of the table elements; and reorganizing the text content based on the grid topology information to obtain the descriptive information of the table elements.
[0010] Optionally, the step of determining the target region of the element to be processed in the image to be processed based on the location information and generating description information of the element to be processed further includes: extracting the target image region of the image element from the second resolution image according to the location information of the image element; obtaining an image from the target image region, uploading the image to a remote storage endpoint, and receiving an access link corresponding to the image sent by the remote storage endpoint; inputting the image into a preset image processing model to obtain text description content of the image; and associating the access link with the text description content to obtain description information of the image element.
[0011] Optionally, the parameters of the preset layout analysis model include a layout quality evaluation index, which reflects the overlap ratio between the output global layout and the reference layout. The layout quality evaluation index is expressed as:
[0012] in, This represents the set of regions corresponding to the elements to be processed in the global layout information. This represents the set of regions corresponding to each element in the reference layout. This indicates that the image to be processed is divided into a set of fine-grained grid cells. This indicates the global layout of the output in the grid cells. Coverage on Indicates the reference layout in the grid cells Coverage on the surface.
[0013] Secondly, the present invention provides a multi-format document parsing apparatus, the apparatus comprising: The conversion module is used to convert the modality of the first type of document into an image to be processed, wherein the first type of document includes images and text; The analysis module is used to perform global layout analysis on the image to be processed using a preset layout analysis model to obtain global layout information; wherein, the global layout information includes the position information and hierarchical relationship of the elements to be processed; A generation module is used to determine the target region of the element to be processed in the image to be processed based on the location information, and to generate descriptive information of the element to be processed. The reorganization module is used to reorganize the description information according to the location information and the hierarchical relationship to obtain the parsed document.
[0014] Optionally, the apparatus is further configured to determine the type of the received document; the type includes a first type and a second type; for the second type of document, extract the structured information and content text of the second type of document; wherein the second type of document does not include images; convert the structured information into plain text format data, and merge the plain text format data with the content text to obtain the parsed document.
[0015] Optionally, the image to be processed includes a first resolution image; the resolution of the first resolution image is lower than a preset resolution threshold; the analysis module is further configured to perform layout analysis on the first resolution image using a first layout analysis model to obtain preliminary layout information, the preliminary layout information including the border of the area corresponding to the element to be processed, and the type corresponding to each element to be processed; extract text from the first resolution image using a text recognition model to obtain text fragments, and a set of text fragments composed of multiple text fragments; and analyze the global layout of the first resolution image using a second layout analysis model based on the preliminary layout information and the set of text fragments to obtain the global layout information.
[0016] Optionally, the analysis module is further configured to extract visual features from the first resolution image; convert the text fragments in the text fragment set into geometric channels and sparse semantic features; fuse the preliminary layout information, the visual features, the geometric channels, and the sparse semantic features to obtain multimodal fusion features; and analyze the multimodal fusion features through the second layout analysis model to obtain the global layout information.
[0017] Optionally, the image to be processed further includes a second resolution image, the second resolution image having a higher resolution than the first resolution image; the elements to be processed include text elements, image elements, table elements, and formula elements; the generation module is further configured to extract the target table region of the table element from the second resolution image based on the position information of the table element; perform grid structure detection on the table element in the target table region to determine the grid topology information of the table element; extract the text content of the table element; and reorganize the text content based on the grid topology information to obtain the description information of the table element.
[0018] Optionally, the generation module is further configured to: extract the target image region of the image element from the second resolution image based on the position information of the image element; obtain an image from the target image region; upload the image to a remote storage endpoint; receive an access link corresponding to the image sent by the remote storage endpoint; input the image into a preset image processing model to obtain the text description content of the image; and associate the access link with the text description content to obtain the description information of the image element.
[0019] Optionally, the parameters of the preset layout analysis model include a layout quality evaluation index, which reflects the overlap ratio between the output global layout and the reference layout. The layout quality evaluation index is expressed as:
[0020] in, This represents the set of regions corresponding to the elements to be processed in the global layout information. This represents the set of regions corresponding to each element in the reference layout. This indicates that the image to be processed is divided into a set of fine-grained grid cells. This indicates the global layout of the output in the grid cells. Coverage on Indicates the reference layout in the grid cells Coverage on the surface.
[0021] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-format document parsing method of the first aspect described above.
[0022] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multi-format document parsing method of the first aspect described above.
[0023] The aforementioned multi-format document parsing method, apparatus, computer equipment, and storage medium, through global layout analysis of the document, obtain global layout information containing the location information and hierarchical relationships of the elements to be processed. Based on the global layout information, the various elements to be processed in the document can be accurately located, and then the content of each element to be processed can be accurately extracted. Finally, based on the location information and hierarchical relationships, the extracted accurate content is reorganized to obtain a highly accurate parsed document. Attached Figure Description
[0024] Figure 1 This is a flowchart illustrating a multi-format document parsing method in one embodiment; Figure 2 This is a flowchart illustrating the global layout analysis process in one embodiment; Figure 3 This is a structural block diagram of a multi-format document parsing device in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this invention, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing together, or B existing alone. A connected to B can represent: A and B directly connected, or A and B connected through C. Furthermore, in the description of this invention, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.
[0026] To facilitate understanding by those skilled in the art, the technical terms involved in this invention will first be explained.
[0027] Page layout: In the field of document processing, page layout specifically refers to a formal representation obtained after performing geometric structure analysis and logical structure inference on document images or digital documents through computational models. It describes the type, spatial position, geometric shape, and hierarchical and sequential relationships between all constituent elements on a document page.
[0028] In this invention, the acquisition, transmission, storage, and use of data all comply with the requirements of relevant national laws and regulations.
[0029] Before introducing the multi-format document parsing method provided by this invention, the technical background of this invention will be described in detail below for ease of understanding.
[0030] In today's information processing field, automated parsing and deep understanding of multi-format documents are key to improving data processing efficiency. Traditional document digitization methods mainly rely on Optical Character Recognition (OCR) technology to extract text content from scanned documents or images. However, this method often struggles to effectively identify and preserve the complex structure of documents (such as page layout and chapter hierarchy) and non-text elements (such as images and tables), leading to information fragmentation and semantic loss. For plain text (such as TXT and Markdown) or partially marked documents (such as HTML (Hypertext Markup Language) and Word), although computers can directly read the text and limited structural information, when processing unmarked or fixed-format documents such as PDFs, PPTs, or scanned images, it is still necessary to first convert the text using OCR and then perform additional complex post-processing to reconstruct the structure.
[0031] Currently, the mainstream technical approaches mainly include two categories: Modular pipeline system: This system processes documents step-by-step by connecting independent dedicated modules such as text detection, character recognition, layout analysis, and table recognition. While each module in this solution is highly independent and accurate, the system integration is complex, the data processing flow is lengthy, and format conversions are frequent, making it difficult to adapt to diverse documents with varying layouts and mixed elements.
[0032] End-to-end multimodal large-scale model: This approach utilizes a vision-language integrated pre-trained model to directly parse the entire document image. While the process is simple, it relies on a very large-scale model, resulting in high inference costs, low efficiency, and limited capabilities in fine-grained structured output (such as precise coordinates and nested tables).
[0033] In recent years, solutions for document parsing and structured processing have covered multiple stages, including document parsing, table recognition, and image processing. However, these solutions generally have the following limitations: document type classification is not refined enough, making it difficult to distinguish between fine-grained categories such as "plain text / scanned version / image-text fusion"; OCR extraction often ignores page layout hierarchy and precise image coordinates, lacking structured information; image processing only performs local optimization, failing to achieve structured output and cannot be independently invoked or located; tables only recognize content, with limited expression of their hierarchical structure.
[0034] Therefore, how to achieve efficient, structured, and multimodal deep understanding of document parsing, especially in areas such as mixed text and graphics, complex tables, and independent storage and reconstruction of images, remains a technical challenge that urgently needs to be overcome.
[0035] In view of this, the present invention provides a method, apparatus, computer device, and storage medium for parsing multi-format documents to solve the above problems.
[0036] The technical solution provided by the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0037] Figure 1 This is a flowchart illustrating a multi-format document parsing method in one embodiment. This process can be executed by a multi-format document parsing device, which can be implemented in software, hardware, or a combination of both. Figure 1 As shown, the process includes the following steps: S101, for the first type of document, convert the modality of the first type of document into an image to obtain the image to be processed.
[0038] In one embodiment, before executing step S101, the type of the received document is first determined so that the document is routed to the corresponding type of parsing path for processing. The document types are preset to a first type and a second type. The first type of document may include images and text. For example, it may be a complex document containing images or a document from which text cannot be directly extracted, such as PDF (scanned or mixed version), PPT, various scanned images or photos, etc. It should be noted that the images included in the first type of document are image elements within the document; an image is a modality of the document, a form of document, and the converted image may include multiple images. The second type of document does not contain images. For example, it may be a plain text document or a markup document, such as Markdown, TXT, HTML, JSON, PDF / Office documents with copyable text, etc.
[0039] In some implementations, the received document is decoded to obtain a decoded document, and it is determined whether the decoded document contains images. If the decoded document contains images, the received document is determined to be a first type of document; if the decoded document does not contain images, the received document is determined to be a second type of document. For first type documents, they are routed to a first parsing path for processing, i.e., step S101 described above is executed. For second type documents, they are routed to a second parsing path for processing.
[0040] In one embodiment, the processing flow for the second parsing path of the second type of document includes the following steps: S201, perform plain text parsing on the second type of document to extract structured information and content text.
[0041] Structured information is used to describe a document's attributes and interpretive context. Interpretive context refers to how structured information helps a system or user understand the background, meaning, and relationships of specific data, terms, or content. Structured information includes, but is not limited to, metadata, which includes, but is not limited to: document-level data (e.g., encoding and language, title / author / creation and modification time, page number / resolution, color mode, paper / page size, paragraph / heading level, etc.), page-level data (e.g., page number, page rotation angle, text extractability, white space ratio, estimated reading direction, page density index, etc.), and quality and risk marker data (e.g., garbled text risk score, quality ratings for blur / noise / shadow / perspective, etc.).
[0042] In some implementations, the structured information and content text of the document are extracted by directly calling the corresponding parser. In other implementations, the document is also subjected to garbled character detection to filter out garbled fields before the structured information and content text are extracted from the filtered document.
[0043] S202 converts the structured information into plain text data, merges the plain text data with the content text, and obtains the parsed document.
[0044] In some implementations, the extracted structured information is converted into Markdown format and then merged with the extracted content text to obtain a well-structured Markdown text as the parsed text output of the document.
[0045] In one embodiment of step S101, if the first type of document received is a pure image document, then the process proceeds directly to step S1012.
[0046] In one embodiment, if the received first type of document is a PDF or PPT, the document modality is converted into a graphic to obtain an image to be processed. The image to be processed can be the original image after the document modality conversion, or it can include a first resolution image and a second resolution image after processing the original image, wherein the resolution of the first resolution image is lower than the resolution of the second resolution image.
[0047] In some implementations, each page of the document is rendered as an image with a DPI (Dots Per Inch) value ranging from 220 to 450. Preprocessing is then applied to the image, including grayscale enhancement, binarization, denoising, tilt / rotation correction, perspective correction, and contrast enhancement, to obtain a high-resolution image, i.e., a second-resolution image, to improve the accuracy of subsequent detection and recognition. In some implementations, after the above preprocessing, the image is downsampled to obtain a downsampled full-page image, i.e., a first-resolution image. For example, the downsampled full-page image may have a pixel count of 1036 × 1036.
[0048] S102, using a preset layout analysis model, performs global layout analysis on the image to be processed to obtain global layout information.
[0049] Figure 2 This is a flowchart illustrating the global layout analysis of the image to be processed in one embodiment, as shown below. Figure 2 As shown, the process includes the following steps: S1021, Using the first layout analysis model, perform layout analysis on the first resolution image to obtain preliminary layout information.
[0050] In some implementations, the first layout analysis model is a lightweight layout analysis model. Preliminary layout information includes the borders of the areas corresponding to the elements to be processed, and the type of each element. Elements to be processed are elements in the document that need to be identified or extracted; the types of elements include, but are not limited to, text, images, formulas, and tables. Text elements include, but are not limited to, titles and body paragraphs. The areas corresponding to the elements to be processed include text areas, table areas, image areas, and formula areas.
[0051] S1022, using a text recognition model, extract text from the first-resolution image to obtain a set of text fragments.
[0052] In some implementations, the text recognition model is a low-precision OCR model that can quickly extract multiple text fragments from a first-resolution image, and these multiple text fragments form a text fragment set. Each text fragment includes, but is not limited to: a unique identifier (ocr_id), a bounding box / quad (bbox / quad), coarse text (text_coarse), baseline angle (orientation), quality flags, etc.
[0053] In one embodiment, after obtaining the preliminary layout information and the set of text fragments, the fragments in the text fragment set can be compared and associated with the regions corresponding to the elements to be processed in the preliminary layout information to assign each fragment to the correct region. The comparison content is a composite metric of the geometry and semantics of the fragments and regions. This composite metric includes, but is not limited to, the geometric overlap, directional consistency, adjacency distance, and semantic consistency between the fragments and each region. The geometric overlap can be compared by calculating the area intersection-union ratio (IoU) / area intersection ratio (IoA) between the region and the fragment, or by determining the center point, or by calculating the polygon intersection ratio, etc. The directional consistency can be compared by determining the angle constraint between the fragment baseline angle and the region flow direction. The adjacency distance can be compared by calculating the normalized distance from the fragment to the region boundary and center, or by the spacing between lines / segments within the same region. The semantic consistency can be compared by performing script / language consistency checks on the coarsely recognized text (text_coarse) (e.g., the main text area is biased towards Chinese / English, the formula area contains a high proportion of LaTeX symbols, etc.).
[0054] After obtaining a composite metric of the geometry and semantics of fragments and regions, an association algorithm is used to assign each fragment to the correct region, thus obtaining association information. Association information includes a page-level layout sketch of the first-resolution image, a preliminary reading order, and key attributes other than location information. The page-level layout sketch includes, but is not limited to, region nodes, fragment nodes, and association edges. Key attributes other than location information include, but are not limited to: text-side attributes (such as character / word confidence statistics, language / script, baseline and line height, orientation angle, etc.), region-side attributes (such as region type, type confidence, density / line spacing statistics within the region, hierarchy / adjacency relationship with adjacent regions, etc.), page-side attributes (such as page number, DPI, global rotation angle, page-level reading flow, blank rate, noise score, etc.), and association-side attributes (such as matching score, overlap metric, conflict flag, etc.).
[0055] S1023, using the second layout analysis model, based on the preliminary layout information and text fragment set, analyzes the global layout of the first resolution image to obtain global layout information.
[0056] In one embodiment, visual features are first extracted from a first-resolution image. In some implementations, a backbone network (e.g., a lightweight convolutional neural network CNN or a vision-based neural network ViT) is used for visual feature extraction.
[0057] The fragments in the text fragment set are converted into geometric channels and sparse semantic features. For example, characters or word boxes are converted into binary maps or intensity maps, such as character density maps, orientation field maps, and line baseline heatmaps.
[0058] Multimodal fusion features are obtained by integrating preliminary layout information, visual features, geometric channels, and sparse semantic features. A second layout analysis model is then used to analyze these multimodal fusion features to obtain global layout information. In some implementations, the second layout analysis model is a high-precision model. The second layout analysis model integrates visual and preliminary OCR-extracted information to identify the regions corresponding to each element to be processed in the first-resolution image, as well as the sequential relationships between these regions. For example, the layout analysis model can identify one block of text as a title and another block as a body paragraph, determining that the title precedes the body paragraph. The final global layout information output by the second layout analysis model includes, but is not limited to, the positional information and hierarchical relationships of the elements to be processed. The positional information includes the positional information of the regions corresponding to the elements to be processed, and the sequence of the elements, i.e., the reading order of the elements.
[0059] Using a lower-resolution first-resolution image for global layout analysis facilitates rapid global coverage, reduces redundant calculations in blank areas, and improves computational efficiency. Global layout analysis yields the complete layout structure of the image to be processed, ensuring the correct logical order and hierarchical structure of the parsed document's content during subsequent reconstruction based on this global layout information.
[0060] In one embodiment, the parameters of the aforementioned preset layout analysis model include a layout quality evaluation index, which reflects the overlap ratio between the output global layout and the reference layout. This layout quality evaluation index is expressed as:
[0061] in, The set of page layout elements obtained from the model parsing is the set of regions corresponding to the elements to be processed in the global layout information, including the text region, title region, table region, image region, etc. This represents the set of regions corresponding to each element of the reference layout of the corresponding page, i.e., the set of actual element regions of the corresponding page. This set of element regions of the reference layout comes from an offline labeled / constructed dataset, which is used to train and validate the model and does not depend on the input during online inference. This indicates that the image to be processed (i.e., the page of a document) is divided into a set of fine-grained grid cells (e.g., dividing the page into sufficiently fine grid cells). This indicates the global layout of the output in the grid cells. Coverage on Indicates the reference layout in the grid cells Coverage is the percentage of the predicted area. Coverage can be 1 or 0, where 1 indicates that the area is covered and 0 indicates that it is not covered. Coverage can also be expressed as a probability value.
[0062] This layout quality assessment metric is used in the offline training and validation phases of the layout analysis model. Training is complete when the metric meets preset conditions, such as a value greater than 70%, resulting in a well-trained layout analysis model. Traditional methods often judge correctness based on whether the IoU between the detected page area and the manually labeled area exceeds a certain threshold. This hard threshold judgment has limitations in layout evaluation: if the IoU is slightly below the threshold, it is considered a failure regardless of whether the area location is almost correct, failing to reflect the overall closeness between the parsed result and the original page layout. The layout quality assessment metric does not make a binary judgment of pass / fail for each area, but rather calculates the coverage overlap ratio between the system's predicted layout and the actual layout on a whole-page scale, thus quantifying the overall fit. Even if the boundaries of some areas do not completely overlap, as long as the overall page structure (headings on top, body paragraphs below, charts on the right, etc.) remains consistent with the actual layout, PageIoU will still give a high score. Based on PageIoU, the layout analysis model is trained, and the final trained layout analysis model has strong analytical capabilities and outputs highly accurate global page layout information.
[0063] S103, Based on the location information, determine the target region containing the element to be processed in the image to be processed, and generate descriptive information of the element to be processed.
[0064] In one embodiment, the image to be processed for element identification can be a second-resolution image, i.e., a high-resolution image, to ensure the extraction of fine details for each element. Location information includes, but is not limited to: location information for text elements, table elements, image elements, and formula elements.
[0065] In some implementations, the target text region of the text element is located from the second-resolution image based on the positional information of the text element, and the target text region is cropped out. A second character recognition model is then invoked to perform character-by-character recognition of the target text region, obtaining high-precision text content as descriptive information for the text element. The second character recognition model used here is a more accurate OCR model than the aforementioned character recognition model used to extract text from the first-resolution image, ensuring accurate reading of small or complex fonts in the target text region. In some implementations, the descriptive information of the text element is associated with the corresponding target text region, that is, the parsed text content is precisely associated with its coordinate region in the original page. Simultaneously, based on its positional information in the global layout (such as title, body text, header, footer, etc.), its semantic role is automatically determined and labeled. This mechanism ensures that the extracted text not only retains "what it is," but also clearly defines "where it is" and "what role it plays."
[0066] In some implementations, the target table region of the table element is extracted from the second-resolution image based on the position information of the table element. A preset table parsing algorithm is then used to perform table recognition and parsing on the document, yielding the parsed table results. In other implementations, the target table region of the table element is located from the second-resolution image based on the marked table region, and this target table region is cropped. Then, grid structure detection is performed on the table elements in the target table region to identify row and column lines or infer grid division using OCR text coordinates, determining the grid topology information of the table elements. Next, the text content of each cell in the table element is extracted, and the table structure (including row and column indices and cell merging relationships) is reconstructed based on the extracted text content and grid topology information, resulting in a structured table data structure, such as TableJSON. This reconstructed table data structure serves as the descriptive information for the table elements. This table data structure can fully express both the table content and the table logic, preserving the semantic relationships of the tables in the original document. This table parsing and reconstruction method ensures the accuracy of the table information in the parsed document subsequently reconstructed based on the descriptive information of the table elements. In one embodiment, if the target table area contains nested tables, the internal table structure can be recursively detected by a table parsing algorithm, and finally a structured table data structure, such as a two-dimensional array or a table JSON with span information, can be output to fully represent the original table content.
[0067] In some implementations, the target image region of the image element is located from the second-resolution image based on the positional information of the image element, and the target image region is cropped. The image is then obtained from the target image region and uploaded to a remote storage endpoint (e.g., a server, cloud storage), obtaining the corresponding access link. Generating an access link for the image facilitates direct referencing of the image during subsequent reconstruction of the parsed document, without needing to convert the image into binary data before embedding it, thus maintaining the plain text nature of the parsed document. An image understanding / description model (e.g., an image captioning generation model) is used to analyze the obtained image and generate a textual description of the image. This textual description accurately summarizes the information expressed by the image; for example, the textual description might be "The bar chart shows the sales growth trend over the past 5 years" or "A schematic diagram of the internal structure of the machine equipment." Finally, the generated access link and the textual description are used together as the descriptive information of the image element. In one embodiment, if the image has a corresponding text description (caption), the caption text content is also analyzed and associated with the image.
[0068] In one embodiment, after obtaining the descriptive information (text content) of text elements and the descriptive information (access links and text descriptions corresponding to the images) of image elements, a text-image fusion process is performed, integrating the information contained in the images back into the text content. Specifically, based on the image's position in the document, the access links and text descriptions corresponding to the images are inserted into the parsed text content. Plain text format (e.g., Markdown format) can be used for insertion. For example, if the original document contains a chart, the corresponding location in the parsed text will display the description "Chart: The bar chart shows the sales growth trend over the past 5 years." Through text-image fusion processing, the information carried by the images can be converted into plain text and integrated into the document parsing content, ensuring that no image content in the document is lost during the parsing process, thus compensating for the lack of visual content in traditional document parsing solutions.
[0069] In some implementations, the target formula region of the formula element is located from the second-resolution image based on the position information of the formula element, and the target formula region is cropped out. The formula in the target formula region is parsed using a formula OCR model or parsing engine, and the formula is converted into an editable text form, such as LaTeX format, which is the formula's description information.
[0070] Based on the global layout information, it is possible to accurately locate various elements to be processed in the document, thereby enabling the precise extraction of the content of these elements and ensuring the accuracy of the information in the parsed document reconstructed based on the extracted content.
[0071] S104. Based on the location information and hierarchical relationship, reorganize all descriptive information to obtain the parsed document.
[0072] In some implementations, the recombination process includes the following steps: S1041, according to the original reading order (sequence) and hierarchical relationship between the elements to be processed in the image to be processed, concatenate all the text content in the description information of the text elements, and use appropriate tags to represent the hierarchical structure. For example, use Markdown syntax such as # / ## to indicate the level of the title and chapter name, treat the paragraph text directly as text, and use - or number prefixes to indicate the list structure for list items.
[0073] S1042, based on the position of the image element in the image to be processed, embed the image element's description information into the corresponding position. That is, embed the image's access link and corresponding text description into the corresponding position in the original document.
[0074] S1043, according to the position of the table element in the image to be processed, embed the description information of the table element in a two-dimensional table format into the corresponding position.
[0075] S1044, according to the position of the formula element in the image to be processed, the description information of the formula element is embedded into the corresponding position to obtain the parsed document. In some implementations, the embedded formula text can also be wrapped with mathematical formula tags (e.g., using $...$ or $$...$$ to represent inline or standalone mathematical formulas) to ensure that the formula can be recognized as a whole formula rather than a plain string when read by both humans and machines.
[0076] In one embodiment, when outputting the final parsed document, a consistency check is performed on the overall format of the parsed document. For example, this verifies the correctness of Markdown syntax (complete list and table closures), checks for valid image links, and ensures correct escape sequences for formula symbols. Necessary citations or source information can also be added to ensure the parsed document is traceable and complete during use.
[0077] The multi-format document parsing method provided in this application offers a solution that can automatically identify document formats and route them to corresponding parsing strategies. Different processing paths are selected for different document types (plain text and image scans), thus ensuring parsing efficiency and accuracy. This routing mechanism overcomes the limitations of traditional single-route processing and achieves wide applicability. For image scan documents, a global layout analysis is also proposed, combined with a parsing process that finely identifies local areas corresponding to each document element. By performing global layout analysis at low resolution and local detail identification at high resolution, the efficiency and accuracy of complex layout processing are significantly improved.
[0078] It should be understood that, although Figure 1-2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1-2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0079] In one embodiment, such as Figure 3 As shown, a multi-format document parsing device is provided, including: The conversion module 301 is used to convert the modality of the first type of document into an image to be processed, wherein the first type of document includes images and text; The analysis module 302 is used to perform global layout analysis on the image to be processed using a preset layout analysis model to obtain global layout information; wherein, the global layout information includes the position information and hierarchical relationship of the elements to be processed; The generation module 303 is used to determine the target region of the element to be processed in the image to be processed based on the location information, and to generate description information of the element to be processed. The reorganization module 304 is used to reorganize the description information according to the location information and the hierarchical relationship to obtain the parsed document.
[0080] Specific limitations regarding the multi-format document parsing device can be found in the limitations of the multi-format document parsing method above, and will not be repeated here. Each module in the aforementioned multi-format document parsing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0081] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores multi-format document parsing data. The network interface communicates with external terminals via a network. When executed by the processor, the computer program implements a multi-format document parsing method. The display screen can be an LCD screen or an e-ink display screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device casing, or an external keyboard, touchpad, or mouse.
[0082] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0083] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the steps in the multi-format document parsing method described above.
[0084] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the steps in the multi-format document parsing method described above.
[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0087] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.
Claims
1. A method for parsing multi-format documents, characterized in that, The method includes: For the first type of document, the modality of the first type of document is converted into an image to obtain the image to be processed, wherein the first type of document includes images and text; Using a preset layout analysis model, a global layout analysis is performed on the image to be processed to obtain global layout information; wherein, the global layout information includes the position information and hierarchical relationship of the elements to be processed; Based on the location information, a target region containing the element to be processed is determined in the image to be processed, and descriptive information of the element to be processed is generated; Based on the location information and the hierarchical relationship, the description information is reorganized to obtain the parsed document.
2. The method according to claim 1, characterized in that, Before converting the modality of the first type of document into an image, the method further includes: Determine the type of the received document; the type includes a first type and a second type; For the second type of document, extract the structured information and content text of the second type of document; wherein, the second type of document does not include images; The structured information is converted into plain text data, and the plain text data is merged with the content text to obtain the parsed document.
3. The method according to claim 1, characterized in that, The image to be processed includes a first resolution image; the resolution of the first resolution image is lower than a preset resolution threshold. The step involves performing a global layout analysis on the image to be processed using a preset layout analysis model to obtain the global layout information, including: Using the first layout analysis model, the first resolution image is analyzed to obtain preliminary layout information, which includes the border of the area corresponding to the element to be processed, and the type corresponding to each element to be processed. By using a text recognition model, text is extracted from the first resolution image to obtain text fragments and a set of text fragments composed of multiple text fragments. Using the second layout analysis model, based on the preliminary layout information and the set of text fragments, the global layout of the first resolution image is analyzed to obtain the global layout information.
4. The method according to claim 3, characterized in that, The second layout analysis model, based on the preliminary layout information and the set of text fragments, analyzes the global layout of the first resolution image to obtain the global layout information, including: Extract visual features from the first resolution image; The text fragments in the text fragment set are converted into geometric channels and sparse semantic features; By fusing the preliminary layout information, the visual features, the geometric channels, and the sparse semantic features, a multimodal fusion feature is obtained; The global layout information is obtained by analyzing the multimodal fusion features using the second layout analysis model.
5. The method according to any one of claims 1-4, characterized in that, The image to be processed also includes a second resolution image, the second resolution image having a higher resolution than the first resolution image; the elements to be processed include text elements, image elements, table elements, and formula elements; The step of determining the target region of the element to be processed in the image to be processed based on the location information and generating descriptive information of the element to be processed further includes: Based on the position information of the table elements, the target table region of the table elements is extracted from the second resolution image; Perform grid structure detection on the table elements in the target table area to determine the grid topology information of the table elements; Extract the text content of the table elements; Based on the grid topology information, the text content is reorganized to obtain the description information of the table elements.
6. The method according to claim 5, characterized in that, The step of determining the target region of the element to be processed in the image to be processed based on the location information and generating descriptive information of the element to be processed further includes: Based on the position information of the image element, the target image region of the image element is extracted from the second resolution image; Obtain an image from the target image area, upload the image to a remote storage endpoint, and receive an access link corresponding to the image sent by the remote storage endpoint; The image is input into a preset image processing model to obtain the text description of the image; By associating the access link with the text description, the description information of the image element can be obtained.
7. The method according to claim 1, characterized in that, The parameters of the preset layout analysis model include layout quality evaluation indicators, which reflect the overlap ratio between the output global layout and the reference layout. The layout quality evaluation indicators are expressed as follows: ， in, This represents the set of regions corresponding to the elements to be processed in the global layout information. This represents the set of regions corresponding to each element in the reference layout. This indicates that the image to be processed is divided into a set of fine-grained grid cells. This indicates the global layout of the output in the grid cells. Coverage on Indicates the reference layout in the grid cells Coverage on the surface.
8. A multi-format document parsing device, characterized in that, The device includes: A conversion module is used to convert the modality of a first type of document into an image to be processed, wherein the first type of document includes images and text; The analysis module is used to perform global layout analysis on the image to be processed using a preset layout analysis model to obtain global layout information; wherein, the global layout information includes the position information and hierarchical relationship of the elements to be processed; A generation module is used to determine the target region of the element to be processed in the image to be processed based on the location information, and to generate descriptive information of the element to be processed. The reorganization module is used to reorganize the description information according to the location information and the hierarchical relationship to obtain the parsed document.
9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.