Document intelligent analysis system and method based on multi-modal fusion

Abstract

The application discloses a document intelligent analysis system and method based on multi-modal fusion, relates to the field of artificial intelligence and document processing technology, and a multi-modal large model core layer is used for deep analysis of document data to obtain an analysis result; a layout analysis module in the multi-modal large model core layer is used for analyzing the layout of the document data, identifying the type, position and reading order of different regions; a formula recognition module is used for recognizing a recognized formula region, and converting a formula image into code; a table structure analysis module is used for structurally analyzing a recognized table region to obtain structured data; a knowledge level block module is used for extracting text content of a recognized text region, and dividing the document data into a plurality of hierarchical knowledge blocks based on semantic analysis; and an output and application layer is used for outputting and applying the analysis result in a structured format. The application can deeply analyze a complex document, extract and structure multi-modal information in the complex document.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and document processing technology, and in particular to a document intelligent parsing system and method based on multimodal fusion. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] With the deepening of digital transformation, various documents (such as PDFs, Word documents, and scanned copies) have become important carriers of information. However, these documents often contain complex layouts and rich multimedia content, posing a significant challenge to automated parsing. Traditional document parsing technologies mainly rely on Optical Character Recognition (OCR) to extract text, combined with rule engines or simple machine learning models to process layout and structure. This method performs reasonably well when processing documents with simple structures and regular layouts, but it has significant shortcomings when facing the following complex scenarios: Modern documents often employ complex layouts such as multi-column layouts, mixed text and images, and content spanning multiple pages. Traditional methods struggle to accurately interpret this layout logic, leading to disordered content order or structural gaps in the extracted content. For example, in a two-column paper, traditional OCR might output the content from both columns in an alternating manner, disrupting the original reading order.

[0004] Documents contain not only text but also non-text elements such as images, tables, and mathematical formulas. Traditional OCR focuses only on text recognition, with limited ability to understand the structure of text and tables in images, as well as the semantics of formulas, resulting in insufficient multimodal information fusion. For example, traditional methods often fail to correctly identify the structure of mathematical formulas embedded in documents, treating them as ordinary image processing and leading to information loss.

[0005] Traditional technologies lack a deep semantic understanding of document content. They can only "see" the text and layout, but cannot "understand" the meaning and hierarchical structure of the content. This limits the value of data output by traditional methods in scenarios requiring intelligent content-based analysis (such as question answering and summarizing).

[0006] Many companies have a large number of unstructured documents (such as contracts, reports, invoices, etc.). Traditional technologies struggle to convert the information in these documents into structured data, thus limiting subsequent data analysis and knowledge mining. Summary of the Invention

[0007] To overcome the shortcomings of the existing technologies, this invention provides a document intelligent parsing system and method based on multimodal fusion. By integrating key technologies such as layout analysis, formula recognition, table structure parsing, knowledge hierarchy segmentation, title extraction, and metadata addition, it can perform in-depth parsing of complex documents, extract and structure the multimodal information within them, thereby meeting the needs of large model training and applications for high-quality and diverse data.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention provides a document intelligent parsing system based on multimodal fusion, comprising: The input layer is used to acquire multimodal document data; The multimodal large model core layer is used for deep analysis of document data to obtain analysis results. This core layer includes a layout analysis module, a formula recognition module, a table structure parsing module, and a knowledge hierarchy segmentation module. The layout analysis module analyzes the layout of the document data, identifying the type, position, and reading order of different areas. The formula recognition module identifies the formula areas and converts the formula images into code. The table structure parsing module performs structured parsing on the identified table areas to obtain structured data. The knowledge hierarchy segmentation module extracts text content from the identified text areas and divides the document data into several hierarchical knowledge blocks based on semantic analysis. The output and application layer is used to output and apply the parsed results in a structured format.

[0009] A further technical solution is that the layout analysis module adopts a multimodal layout analysis architecture that integrates visual, semantic and relational information, including a two-stream convolutional network, a multi-scale adaptive aggregation module and a relation learning module.

[0010] In a further technical solution, the dual-stream convolutional network processes visual information and semantic information respectively to obtain visual features and text features; the multi-scale adaptive aggregation module receives visual features and text features, and obtains fused features through an adaptive fusion mechanism; the relationship learning module generates a candidate set of layout elements based on the fused features, and learns the relationship between each element to obtain the layout analysis results.

[0011] In a further technical solution, the formula recognition module includes a formula detection submodule and a formula recognition submodule. The formula detection submodule uses a target detection model to locate the formula region, and the formula recognition submodule uses a sequence-to-sequence model to convert the formula image into a LaTeX sequence.

[0012] In a further technical solution, the table structure parsing module first identifies the row and column structure of the table and the position of each cell, then processes merged cells and cross-page tables, and finally converts the table content into structured data.

[0013] In a further technical solution, the core layer of the multimodal large model also includes a title extraction module, which is used to automatically identify and extract the top headings in the document data and construct the title hierarchy structure of the document.

[0014] In a further technical solution, the core layer of the multimodal large model also includes a metadata addition module, which is used to add metadata information to the parsed document content, including the document's basic attributes and content attributes.

[0015] Secondly, this invention provides a document intelligent parsing method based on multimodal fusion, including: Acquire multimodal document data; The document data is subjected to in-depth analysis to obtain the analysis results. Specifically, the layout of the document data is analyzed to identify the type, position and reading order of different areas; the identified formula areas are identified and the formula images are converted into code; the identified table areas are subjected to structured analysis to obtain structured data; and the identified text areas are subjected to text content extraction, and the document data is divided into several hierarchical knowledge blocks based on semantic analysis. The parsing results are output in a structured format and applied.

[0016] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the document intelligent parsing method based on multimodal fusion as described in the second aspect.

[0017] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the document intelligent parsing method based on multimodal fusion as described in the second aspect.

[0018] The above one or more technical solutions have the following beneficial effects: This invention provides a system capable of efficiently parsing complex documents containing multimodal information such as text, images, tables, and formulas, extracting and structuring key information to provide high-quality data support for downstream large-scale model training and applications. By integrating key technologies such as layout analysis, formula recognition, table structure parsing, knowledge hierarchy segmentation, title extraction, and metadata addition, this invention constructs an end-to-end intelligent document parsing solution, aiming to address the bottleneck issues of traditional document processing technologies when dealing with complex layouts and cross-modal information.

[0019] This invention improves the accuracy and completeness of parsing. Through cross-modal understanding of a multimodal large model, this system can simultaneously recognize multiple elements such as text, images, tables, and formulas, ensuring more complete and accurate parsing results. For example, for academic papers containing complex formulas, traditional methods may treat the formulas as images and ignore their content, while this system can accurately identify the formulas and convert them into editable LaTeX, avoiding information loss.

[0020] This invention enhances layout comprehension capabilities. The layout analysis module of this invention can handle various complex layouts, including multi-column, multi-page, and mixed text and image layouts, correctly extracting the reading order and logical structure of the content. This solves the problem of disordered content order in complex document layouts encountered by traditional methods, making the analysis results more consistent with human reading habits.

[0021] This invention overcomes the challenges of processing tables and formulas. Addressing the traditional technical difficulties associated with tables and formulas, this invention provides a specialized solution. The table structure parsing module can process wired tables, non-wired tables, and tables spanning multiple pages, outputting structured table data. The formula recognition module can identify complex mathematical formulas, including inline formulas, and convert them into a standard format. These capabilities significantly expand the applicability of document parsing.

[0022] This invention constructs an intelligent content structure. Through hierarchical knowledge segmentation and title extraction, this system automatically builds a hierarchical content structure for documents, organizing scattered text into logically clear knowledge units. This structured output is highly beneficial for large models to understand and apply document content. For example, in a question-and-answer system, relevant paragraphs can be quickly located based on titles, and in knowledge base construction, each knowledge block can be stored as an independent knowledge entry.

[0023] This invention maximizes the value of data. The metadata addition module adds rich semantic information to the parsing results, making the originally static document data searchable, manageable, and associative. This is of great significance for applications such as building enterprise knowledge bases, realizing intelligent retrieval, and conducting data mining, and can fully realize the value of data.

[0024] This invention offers a highly efficient and scalable system. It employs a modular architecture, allowing modules to operate in parallel or pipelined fashion, thus improving parsing efficiency. Furthermore, the multimodal large model exhibits excellent generalization capabilities; with appropriate fine-tuning, it can adapt to documents from different domains and formats, demonstrating high scalability. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0026] Figure 1 This is a schematic diagram of the document intelligent parsing system architecture based on multimodal fusion according to an embodiment of the present invention; Figure 2 This is a flowchart of the document intelligent parsing method based on multimodal fusion according to an embodiment of the present invention. Detailed Implementation

[0027] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0029] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0030] In recent years, next-generation artificial intelligence technologies, represented by multimodal large models, have brought revolutionary breakthroughs to the field of document parsing. Multimodal large models refer to large-scale deep learning models capable of processing multiple types of data simultaneously (such as text, images, and audio). Unlike traditional models that only process single-modality data, multimodal large models, by fusing information from different modalities, can more comprehensively and accurately understand and process complex documents.

[0031] Given the enormous potential of multimodal large models in the field of document parsing, this invention aims to design a document parsing system architecture and process based on multimodal large models, so as to give full play to its technical advantages in layout analysis, formula recognition, table structure parsing, knowledge hierarchy segmentation, title extraction and metadata addition, and provide a comprehensive, efficient and innovative solution for intelligent document processing.

[0032] Example 1 like Figure 1 As shown, this embodiment discloses a document intelligent parsing system based on multimodal fusion, including: The input layer is used to acquire multimodal document data; The core layer of the multimodal large model is used for deep parsing of document data to obtain the parsing results; The output and application layer is used to output and apply the parsed results in a structured format.

[0033] In this embodiment, the document intelligent parsing system adopts a modular design, with each module working collaboratively to achieve comprehensive document parsing. The system architecture is divided into an input layer, a multimodal large model core layer, and an output and application layer from bottom to top.

[0034] In this embodiment, the input layer is used to receive document data (multimodal document data) in various formats input by the user, including but not limited to PDF, Word, and scanned images (such as JPEG and PNG). The input layer passes document data from different sources to the multimodal large model core layer for processing through a unified interface.

[0035] Necessary preprocessing is performed on document data. For scanned documents or images, preprocessing includes image denoising, binarization, and skew correction to improve the accuracy of subsequent recognition. For electronic documents, preprocessing may include format conversion and text extraction. Preprocessing aims to improve the accuracy of subsequent multimodal large-scale model processing.

[0036] Furthermore, the input layer uses a PDF parser or Office document parser to extract text and image resources, converting the document into a unified intermediate representation (such as XML or JSON format containing text and image coordinates).

[0037] In this embodiment, the multimodal large model core layer is the core of the system, consisting of multiple functional modules, which utilize the powerful capabilities of the multimodal large model to perform in-depth analysis of documents.

[0038] The core layer of the multimodal large model includes a layout analysis module, a formula recognition module, a table structure parsing module, a knowledge hierarchy segmentation module, a title extraction module, and a metadata addition module.

[0039] (1) Page Layout Analysis Module This module analyzes the document's layout, identifying the location and type of different areas such as text, images, tables, and formulas, and determining their reading order. This provides contextual information for subsequent detailed analysis.

[0040] The preprocessed document enters the layout analysis module, which uses a multimodal large model to analyze the document's layout, identifying different types of areas such as text blocks, image blocks, table blocks, and formula blocks, and determining their position coordinates and reading order on the page. The results of the layout analysis provide context for subsequent modules, such as informing the formula recognition module which areas contain formulas, and informing the table parsing module which areas are tables.

[0041] Furthermore, the layout analysis module can employ a deep learning-based layout analysis model, such as using a multimodal large model to perform object detection or semantic segmentation on document images, dividing the page into different layout elements.

[0042] In some implementations, the layout analysis module employs a multimodal layout analysis architecture (VSR) that integrates visual, semantic, and relational information. This architecture comprises three main components: a two-stream convolutional network, a multi-scale adaptive aggregation module, and a relation learning module.

[0043] Two-stream convolutional neural networks (DNNs): This network comprises two parallel convolutional neural network branches: a visual branch and a semantic branch. One branch processes the visual information (images) of the document, while the other processes its semantic information (text). The visual branch takes the document's image as input and extracts image features through multiple convolutional layers, resulting in visual feature maps at different scales. The semantic branch converts the document's text into a two-dimensional representation, for example, mapping each character or word to its position on the page, forming a "text grid," and then extracts text features through the convolutional network. Through this two-stream network, the model can simultaneously capture both the document's visual layout and its semantic information.

[0044] The multi-scale adaptive aggregation module, after obtaining visual and textual features, fuses the features from both modalities through an adaptive aggregation mechanism to obtain a fused feature. It doesn't simply concatenate the two features; instead, it uses an attention mechanism to learn where and to what extent to rely on visual or textual features, thereby generating a multi-modal feature representation that integrates visual and semantic elements. This fused feature can more accurately represent the layout elements of a document.

[0045] The Relationship Learning Module generates a candidate set of layout elements based on fused features and learns the relationships between these elements. It treats each layout element (such as text blocks, image blocks, table blocks, etc.) as a node in a graph and uses graph neural networks or self-attention mechanisms to learn the relationships between nodes. For example, it can learn relationships such as "tables are usually located below their titles" and "images usually have captions." Through relationship learning, the model can correct initial detection results, such as adjusting the position of text boxes based on co-occurrence relationships and removing overlapping boxes based on spatial relationships. Finally, the Relationship Learning Module outputs accurate layout analysis results, including the category (text, image, table, formula, etc.) and position coordinates of each element, as well as their reading order.

[0046] Through the above architecture, the layout analysis module can handle various complex layouts, including multi-column, cross-page, and mixed text and image layouts, and output accurate layout information, laying the foundation for detailed analysis by subsequent modules.

[0047] (2) Formula recognition module The formula recognition module is specifically designed to identify mathematical formulas in documents and convert them into structured representations (such as LaTeX or MathML). This module uses a deep learning model to understand the two-dimensional structure of formulas, solving the problem that traditional OCR struggles to handle complex formulas.

[0048] Guided by layout analysis, the formula recognition module performs precise identification of detected formula regions. It employs a specialized mathematical formula recognition model to convert formula images into editable LaTeX or MathML code. For formulas inlined within text lines, the module can also accurately locate and recognize them, ensuring the consistency between the formula and the text.

[0049] Furthermore, the mathematical formula recognition model is a deep learning model specifically designed for mathematical formulas. It can understand the two-dimensional structure of formulas and convert formula images into editable LaTeX or MathML code. The mathematical formula recognition model includes a formula detection submodule and a formula recognition submodule. The formula detection submodule uses object detection or segmentation models to locate the formula region, while the formula recognition submodule uses sequence-to-sequence models (such as Transformer-based models) to convert the formula image sequence into a LaTeX sequence. Through the collaboration of these two submodules, it can handle formulas of various complexities, including multi-line formulas, matrices, integrals, and differentials.

[0050] In some implementations, the formula recognition module includes two sub-modules: a formula detection sub-module and a formula recognition sub-module.

[0051] Formula Detection Submodule: This submodule utilizes object detection or semantic segmentation models to locate formula regions. Detection is relatively easy for explicit formulas (formulas on a separate line); however, it is more challenging for inline formulas (formulas embedded within lines of text). The formula detection submodule can employ a deep learning-based model, for example, inputting a document image into a trained convolutional neural network to output bounding boxes of formula regions. This model can be trained on a dataset of documents containing formulas, learning to recognize the visual features of formulas (such as mathematical symbols, fractions, square roots, etc.).

[0052] Formula Recognition Submodule: This submodule utilizes a sequence-to-sequence model to convert a sequence of formula images into a LaTeX sequence. It takes detected formula images as input and performs recognition through an encoder-decoder architecture. The encoder, which can be a convolutional neural network or a visual Transformer, extracts features from the formula images; the decoder, which can be a recurrent neural network (RNN) or a Transformer, progressively generates the LaTeX sequence. During training, the model uses pairs of formula images and their corresponding LaTeX sequences as supervision signals to learn the mapping relationship between images and text sequences. In this way, the formula recognition submodule can recognize formulas of various complexities, including multi-line formulas, matrices, integrals, and differentials, and output accurate LaTeX representations.

[0053] Through the collaboration of these two sub-modules, the formula recognition module can process all formulas in a document, whether explicit or inline, and convert them into structured LaTeX code. This solves the problem of traditional OCR struggling to handle complex formulas and avoids the situation of treating formulas as ordinary images and losing information.

[0054] (3) Table structure parsing module The table structure parsing module performs structured parsing of tables in a document, including detecting table areas, identifying row and column structures, and handling merged cells, ultimately converting the table content into structured data (such as HTML or JSON format). This module can handle complex cases such as wired tables, non-wired tables, and tables spanning multiple pages.

[0055] The table structure parsing module performs in-depth analysis of the table areas detected by the layout analysis. It first identifies the table's row and column structure (including determining the number of rows and columns) and the position of each cell; then it handles complex cases such as merged cells and tables spanning multiple pages. Finally, the module converts the table content into a structured data format, such as an HTML table or a JSON object, preserving the table's original structure and content.

[0056] In some implementations, the table structure parsing module employs an end-to-end table recognition method based on a multimodal large model. This method includes the following steps: Table area detection: This step is completed by the layout analysis module, which has already determined the bounding box of the table area.

[0057] Cell detection: Within a table area, detect the position of each cell. This can be achieved using an object detection model that treats each cell as an object. The model needs to learn to recognize cell boundaries, including the boundaries of merged cells.

[0058] Row and column structure identification: Based on the detected cell positions, infer the row and column structure of the table. This includes determining how many rows and columns the table has, and the number of rows and columns each cell spans (i.e., merging information). This step can be implemented using graph algorithms, such as treating cells as nodes in a graph and constructing row and column relationships based on their relative positions.

[0059] Cell content recognition: Perform OCR recognition on each cell range to extract the text content. For cells containing formulas, the formula recognition module can be used for processing.

[0060] Structured Output: The identified row and column structure and cell content are output in a structured format. For example, HTML table code can be generated, where backticks (`) represent rows, backticks (`) represent cells, and `rowspan` and `colspan` attributes represent merge information; or a JSON object can be generated containing an array of "cells", where each element contains the cell's row and column index, merge information, and text content.

[0061] Using the methods described above, the table structure parsing module can handle various types of tables, including wired tables, wireless tables, and multi-page tables. For multi-page tables, this module can combine multiple pages of tables into a complete logical table, ensuring data integrity. The final output structured table data can be directly used for data analysis or large model training.

[0062] Furthermore, when handling merged cells, the module identifies which cells are merged across rows or columns; for multi-page tables, the module can combine multiple pages of tables into a complete logical table.

[0063] Furthermore, the table structure parsing module can employ a deep learning-based table recognition model, such as using a multimodal large model to simultaneously understand the image and text of the table and predict its structure. In addition, this module can incorporate rule-based post-processing to correct the structure output by the model, ensuring the accuracy of the results.

[0064] (4) Knowledge hierarchy block module For text areas that are neither formulas nor tables, the system uses a high-precision OCR engine to extract the text content. This means that, based on the semantic logic of the document content, the document is divided into several hierarchical knowledge blocks. After acquiring the document's text content (obtained through OCR or text extraction) and layout information, the knowledge hierarchy segmentation module performs semantic analysis on the document to identify the logical structure of the content. More importantly, a multimodal large model performs semantic understanding on the extracted text, including word segmentation, named entity recognition, and syntactic analysis, laying the foundation for subsequent knowledge segmentation and title extraction.

[0065] The knowledge hierarchy segmentation module divides a document into hierarchical knowledge blocks, such as chapters, paragraphs, and lists, based on the semantic logic of the document content. By understanding the hierarchical structure of the document, this module achieves logical segmentation of the content, providing structured input for building knowledge graphs or conducting question answering in large models.

[0066] The knowledge hierarchy segmentation module divides documents into logically related knowledge blocks based on semantic understanding of the document content. For example, it can divide a document into chapters such as introduction, methods, experiments, and conclusions, or divide long paragraphs into several thematic paragraphs. This process is achieved by analyzing the semantic similarity and structural clues of the text, ensuring that the content within each knowledge block is highly related and the logic between blocks is clear.

[0067] After acquiring the document's text content (obtained through OCR or text extraction) and layout information, the knowledge hierarchy segmentation module performs semantic analysis on the document to identify the logical structure of the content. For example, it can divide an academic paper into chapters such as introduction, related work, methods, experiments, and conclusions, or divide a long report into several thematic paragraphs.

[0068] The knowledge hierarchy segmentation module can employ natural language processing techniques, such as topic modeling, text clustering, and sequence labeling, combined with layout information (such as title positions) to achieve segmentation. For example, it can treat a document as a sequence and use sequence labeling models to label each paragraph or sentence with its corresponding chapter or topic. Through knowledge hierarchy segmentation, the document's content is organized into logically related knowledge units, facilitating subsequent applications (such as building knowledge graphs and question-answering systems).

[0069] In some implementations, the knowledge hierarchy segmentation module employs a document hierarchy segmentation method based on a multimodal large model. This method includes the following steps: Text segmentation: First, the text content of the document is divided into basic paragraphs or sentence units. This can be achieved through simple rules (such as segmenting by line breaks) or more complex semantic segmentation models.

[0070] Semantic Representation: A multimodal large model is used to perform semantic representation on each paragraph or sentence. The multimodal large model can simultaneously consider text content and layout information (such as whether the paragraph is below a heading, whether it is a list item, etc.) to generate a representation vector that integrates semantics and layout.

[0071] Clustering or classification: Based on semantic representation, paragraphs or sentences are clustered or classified to identify the topic or chapter of the content. For example, clustering algorithms (such as K-means) can be used to group semantically similar paragraphs into one class, with each class constituting a knowledge block; or classification models (such as training a document chapter classifier) ​​can be used to predict the chapter category to which each paragraph belongs.

[0072] Hierarchical Structure Construction: Based on the clustering or classification results, construct the hierarchical structure of the document. For example, if clustering is used, each cluster can be considered a knowledge block and arranged according to the order in which the clusters appear in the document; if classification is used, consecutive paragraphs of the same category can be merged into a knowledge block and arranged according to the order in which the category appears in the document. For multi-level hierarchies (such as chapters, sections, subsections), the above method can be applied iteratively: first divide into chapters, then divide into sections within each chapter, and so on.

[0073] Using the methods described above, the knowledge hierarchy segmentation module can automatically divide a long document into logically related knowledge units. For example, an academic paper can be divided into chapters such as introduction, related work, methods, experiments, and conclusions; a report can be divided into several thematic paragraphs. The content within each knowledge block is highly related, and the logic between blocks is clear. This structured segmentation result is very suitable for building knowledge graphs or conducting intelligent question answering, because each knowledge block can be regarded as an independent knowledge unit, which is easy to retrieve and understand.

[0074] (5) Title extraction module The title extraction module automatically identifies and extracts headings at all levels from a document, constructing the document's heading hierarchy. This module helps in understanding the document's outline, facilitating subsequent tasks such as summary generation and content retrieval.

[0075] The title extraction module uses text formatting features (such as font size, bold, italics, and whether it is centered) and semantic features (such as whether it is at the beginning of a paragraph, whether it contains a chapter number, and whether it is a question or interrogative sentence) to determine whether a text block is a title and to determine the hierarchical relationship of the titles. The extracted title structure helps to understand the outline of the document and provides a basis for subsequent generation of summaries or navigation.

[0076] Furthermore, the title extraction module uses text formatting features (such as font size, bold, italics, and whether it is centered) and semantic features (such as whether it is at the beginning of a paragraph, whether it contains a chapter number, and whether it is a question or interrogative sentence) to determine whether a text block is a title. If it is a title, the module further determines its level, such as first-level heading, second-level heading, etc.

[0077] Furthermore, the title extraction module can employ a combination of rules and models: first, rules (such as font size exceeding a certain threshold and centered alignment) are used to initially filter candidate titles; then, a multimodal large-scale model is used to semantically classify the candidate titles, determining whether they are titles and their hierarchical level. Through title extraction, the system can obtain the document's outline structure, which is very helpful for understanding the document's context and generating summaries.

[0078] In some implementations, the title extraction module employs a title extraction method that combines rules and models. This method includes the following steps: Candidate title detection: First, preliminary screening of candidate titles is conducted using rules. Rules may include: the font size of the text block is greater than a certain threshold, the text block is centered or left-aligned and located at the beginning of a paragraph, and the text block contains chapter numbers (such as "1.", "I.", etc.). These rules can quickly filter out most non-title text, resulting in a set of candidate titles.

[0079] Heading Classification: A multimodal large-scale model is then used to classify candidate headings, determining whether they are headings and their heading level. This model considers both the text content and layout information (such as font size and position) of the candidate headings, outputting a classification result such as "Level 1 Heading," "Level 2 Heading," or "No Heading." This model can be trained on a dataset labeled with heading levels, learning features to distinguish between headings and body text, as well as heading levels.

[0080] Hierarchical Structure Construction: Based on the classification results, construct the document's heading hierarchy. For example, arrange all text blocks identified as first-level headings in the order of their appearance in the document, forming chapters; arrange all text blocks identified as second-level headings in the order of their appearance, forming sections, and so on. For headings without explicit hierarchical identifiers, the hierarchy can be inferred from their relative position to the preceding and following headings.

[0081] Using the methods described above, the title extraction module can accurately extract the document's title structure, including chapters, sections, and subsections. The extracted title structure helps in understanding the document's outline, facilitating subsequent tasks such as summary generation and content retrieval. For example, in an intelligent question-answering system, the title structure can be used to quickly locate relevant paragraphs based on the chapter keywords in the user's question, improving the accuracy and efficiency of question-answering.

[0082] (6) Metadata Addition Module The metadata addition module adds metadata information to the parsed document content, including basic document attributes (such as author, creation date, and document type) and content attributes (such as keywords and abstract). Metadata enriches the semantic information of the document, facilitating subsequent retrieval, management, and knowledge mining.

[0083] The metadata addition module adds rich metadata to the document based on the parsing results. This includes basic metadata extracted from document attributes (such as author, creation time, and document type) as well as semantic metadata obtained through content analysis (such as keywords, topic categories, and summaries).

[0084] Metadata is appended to the parsing results in a structured form, improving the data's searchability and manageability. For example, adding a "metadata" field to the output JSON object, containing all metadata items, will greatly enhance the searchability and manageability of the parsing results, facilitating subsequent knowledge base construction and intelligent retrieval.

[0085] Furthermore, the metadata includes: Basic metadata includes information such as the document's title, author, creation date, modification date, and document type (e.g., paper, report, contract). This information can usually be extracted from the document's attributes or inferred by analyzing the document's content (e.g., determining its type based on its format and content).

[0086] Content metadata includes keywords, topic categories, and summaries. The metadata addition module can utilize natural language processing techniques to extract keywords (such as TF-IDF and TextRank algorithms) or categorize topics from documents (e.g., using a pre-trained text classification model to determine the document's domain). Summaries can be automatically generated by this module (e.g., extractive or generative summaries) or provided by the user. For technical documents, reference lists, figure lists, etc., can also be extracted as metadata.

[0087] Structural metadata includes statistical information such as the number of pages, chapters, tables, and formulas in the document, as well as the document's language (Chinese, English, etc.). This information helps in document management and retrieval.

[0088] In some implementations, the metadata addition module performs the following operations: Basic Metadata Extraction: Extracts basic metadata from document attributes. For example, for PDF documents, it can read their metadata fields (such as title, author, creation date, modification date, etc.); for scanned documents, users may need to manually enter some metadata. For documents without explicit metadata, this module can attempt to infer it from the content, such as determining its type (paper, report, contract, etc.) based on the document's format and content.

[0089] Content metadata extraction: Extracting content metadata from documents using natural language processing techniques.

[0090] Keyword extraction: Keywords are extracted from documents using keyword extraction algorithms (such as TF-IDF and TextRank). These keywords can reflect the main content of the document.

[0091] Topic classification: This uses a pre-trained text classification model to determine the topic domain (e.g., computer science, medicine, law, etc.) of a document. This can be achieved by classifying the document's summary or full text.

[0092] Summary generation: Use automatic summarization techniques (such as extractive summarization or generative summarization) to generate concise summaries of documents. Summaries serve as a summary of the document, helping users quickly understand its content.

[0093] Reference extraction: For documents such as academic papers, a list of references can be extracted and included as part of the metadata.

[0094] Structural metadata extraction: The structural information of statistical documents is used as metadata. For example, the number of pages, chapters, tables, and formulas in a statistical document. This information helps in document management and retrieval.

[0095] Metadata integration: All the extracted metadata is integrated into a single structured object and appended to the parsed results. For example, a "metadata" field is added to the output JSON object, containing all metadata items.

[0096] Through the metadata addition module, the parsed document data not only includes the content itself but also rich metadata information. This metadata will greatly improve the searchability and manageability of the parsing results, facilitating subsequent knowledge base construction and intelligent retrieval. For example, in the knowledge base, documents can be quickly filtered based on metadata (such as by author or by topic); in intelligent retrieval, the relevance of search results can be improved based on metadata (such as by keyword matching or by topic filtering).

[0097] In this embodiment, the output and application layer is responsible for outputting the results parsed by the core layer in a structured format and supporting integration with downstream applications. Output formats may include structured text (such as Markdown and JSON), knowledge graph entries, database records, etc., for use in applications such as large model training, intelligent question answering, and data analysis.

[0098] The system will output all the parsed information (including structured text, tabular data, formula code, knowledge blocks, title structures, metadata, etc.) in a predetermined format. The output can be a uniform JSON object, or separate Markdown files, knowledge graph triples, etc., to meet the needs of different downstream applications.

[0099] Furthermore, the output and application layers can provide multiple output formats to meet the needs of different application scenarios: Structured text output: The parsed document content is output in a structured text format, such as Markdown or JSON. For example, the document's text content can be output as Markdown format according to heading levels, tables can be output as Markdown tables, and formulas can be output as LaTeX code. This format is easy for humans to read and also easy for large models to use directly.

[0100] Knowledge Graph Output: The parsed results are converted into a triplet form of a knowledge graph. For example, each knowledge block is treated as an entity, with attributes such as title and author, and relationships between knowledge blocks (such as "contains" and "cites") as edges, thus constructing a knowledge graph. This output is well-suited for knowledge base construction and intelligent question answering.

[0101] Database record output: The parsed results are inserted into the database. Each record corresponds to a document or a knowledge block, containing its text, metadata, structural information, etc. This output facilitates the storage and retrieval of large-scale data.

[0102] Application Interface: The output and application layer can also provide API interfaces for downstream applications to directly call the parsed results. For example, a RESTful API can be provided, allowing applications to retrieve parsed JSON data via document ID. This facilitates the integration of this system into existing enterprise workflows, such as content management systems, intelligent search systems, and large-scale model training platforms.

[0103] Through the above process, this invention can transform a complex multimodal document into data with clear structure and rich semantics, providing high-quality input for the training and application of large models.

[0104] In summary, this invention achieves: Improve parsing accuracy: Through the cross-modal understanding capabilities of the multimodal large model, significantly improve the accuracy of recognizing elements such as text, images, tables, and formulas in documents, and reduce false recognition and missed recognition.

[0105] Enhanced layout understanding: Enables intelligent analysis of complex layouts, including multi-column, cross-page, and mixed text and image layouts, ensuring that the extracted content maintains the correct reading order and logical structure.

[0106] Supports multimodal content: It not only extracts text, but also identifies and parses non-text content such as table structures and mathematical formulas in documents, transforming them into structured data to provide a foundation for subsequent in-depth analysis.

[0107] Building a knowledge hierarchy: By segmenting knowledge into blocks and extracting headings, the hierarchical structure of the document is automatically built, and the content is organized into logical units such as chapters and paragraphs, which facilitates understanding and application of large models.

[0108] Enrich metadata: Add metadata such as document type, author, keywords, and summary to the parsed content to improve the searchability and manageability of the data, laying the foundation for knowledge base construction and intelligent retrieval.

[0109] This invention provides a comprehensive and efficient document parsing solution by integrating the powerful capabilities of multimodal large models, significantly improving the intelligence level of document parsing and laying a solid foundation for document processing and application in the era of large models.

[0110] Example 2 like Figure 2 As shown, this embodiment discloses a document intelligent parsing method based on multimodal fusion, including: Acquire multimodal document data; The document data is subjected to in-depth analysis to obtain the analysis results. Specifically, the layout of the document data is analyzed to identify the type, position and reading order of different areas; the identified formula areas are identified and the formula images are converted into code; the identified table areas are subjected to structured analysis to obtain structured data; and the identified text areas are subjected to text content extraction, and the document data is divided into several hierarchical knowledge blocks based on semantic analysis. The parsing results are output in a structured format and applied.

[0111] In this embodiment, the document intelligent parsing method includes the following steps: (1) Document Input and Preprocessing. The system first receives the document input by the user and performs necessary preprocessing. For scanned documents or images, preprocessing includes image denoising, binarization, and tilt correction; for electronic documents, it may include format conversion and text extraction. Preprocessing aims to improve the accuracy of subsequent modules.

[0112] (2) Layout Analysis. The preprocessed document enters the layout analysis module. This module uses a multimodal large model to analyze the document's layout, identify different areas such as text blocks, image blocks, table blocks, and formula blocks, and determine their coordinates and reading order. The results of the layout analysis provide context for subsequent modules, such as informing the formula recognition module which areas contain formulas, and informing the table structure parsing module which areas are tables, etc.

[0113] (3) Formula Recognition. Guided by the layout analysis, the formula recognition module performs precise identification of detected formula areas. It uses a specialized mathematical formula recognition model to convert formula images into editable LaTeX or MathML code. For formulas inlined in text lines, the module can also accurately locate and recognize them, ensuring the consistency between formulas and text.

[0114] (4) Table Structure Parsing. The table structure parsing module performs in-depth analysis of the table areas detected by the layout analysis. It first identifies the row and column structure of the table, and then handles complex cases such as merged cells and tables spanning multiple pages. Finally, the module converts the table content into a structured data format, such as an HTML table or a JSON object.

[0115] (5) Text recognition and semantic understanding. For text regions that are not formulas or tables, the system uses a high-precision OCR engine to extract the text content. More importantly, the multimodal large model performs semantic understanding on the extracted text, including word segmentation, named entity recognition, and syntactic analysis, laying the foundation for subsequent knowledge segmentation and title extraction.

[0116] (6) Knowledge Hierarchical Blocking. The knowledge hierarchical blocking module divides the document into logically related knowledge blocks based on the semantic understanding of the document content. For example, it can divide the document into chapters such as introduction, methods, experiments, and conclusions, or divide long paragraphs into several thematic paragraphs. Through knowledge hierarchical blocking, the content of the document is organized into logically related knowledge units, which facilitates subsequent application.

[0117] (7) Title Extraction. The title extraction module automatically identifies headings at all levels in the document and constructs a title hierarchy structure. It uses the format and semantic features of the text to determine whether a text block is a title and to determine its level. The extracted title structure helps to understand the outline of the document and provides a basis for generating a summary or navigation.

[0118] (8) Metadata Addition. The metadata addition module adds rich metadata to the document based on the parsing results. This includes basic metadata (such as author, creation date, document type), content metadata (such as keywords, topic categories, and abstracts), and structural metadata (such as page count, chapter count, table count, and formula count). The metadata is appended to the parsing results in a structured form, improving the data's searchability and manageability.

[0119] (9) Output Results. Finally, the system outputs all the parsed information (including structured text, tabular data, formula code, knowledge blocks, title structure, metadata, etc.) in a predetermined format. The output can be a uniform JSON object, or separate Markdown files, knowledge graph triples, etc., to meet the needs of different downstream applications.

[0120] Through the above process, the method of this invention can transform a complex multimodal document into data with clear structure and rich semantics, providing high-quality input for the training and application of large models.

[0121] To better understand this invention, a specific example is provided below: Scenario: Suppose a user inputs a PDF document containing a complex layout of an academic paper, including multi-column text, double-page tables, mathematical formulas, and chapter headings. The user wants to parse this document into structured data for use in building a knowledge base or training a large model.

[0122] step: (1) Document Input and Preprocessing: The system receives the PDF document and uses a PDF parser to extract the text and image resources. Since the document is an electronic PDF, no preprocessing such as image denoising is required. The system converts the document into an intermediate representation containing text and image coordinates for subsequent processing.

[0123] (2) Page Layout Analysis: The page layout analysis module performs layout analysis on each page of the document. Since the document uses a two-column layout, the module identifies the text areas in the left and right columns and determines the reading order as left to right and top to bottom. Simultaneously, it identifies the table areas (located on pages 3 and 4, spanning multiple pages) and formula areas (including explicit and inline formulas) in the document. The page layout analysis module outputs a page description containing the coordinates and reading order of each text block, table block, and formula block.

[0124] (3) Formula Recognition: The formula recognition module identifies the detected formula regions. For explicit formulas, it converts them into LaTeX code; for inline formulas, it accurately locates and recognizes them. For example, an integral formula in the document is recognized as `\int_{a}^{b}fxdx`, and an inline radical formula is recognized as `\sqrt{x^2+y^2}`. These LaTeX codes retain the original structure and meaning of the formulas and can be directly used for typesetting or further analysis.

[0125] (4) Table Structure Parsing: The table structure parsing module parses tables that span multiple pages. It first identifies the table's boundaries on pages 3 and 4 and merges the two pages into a single logical table. Then, it identifies the table's row and column structure, including merged cells. Finally, the module outputs an HTML representation of the table, which correctly represents the structure of spanning pages and merged cells. For example, a merged cell spanning two rows and two columns is represented in HTML using `rowspan="2"colspan="2"`.

[0126] (5) Text Recognition and Semantic Understanding: For text regions that are not formulas or tables, the system uses an OCR engine to extract the text content (since it is an electronic PDF, the text can be extracted directly). The multimodal large model performs semantic understanding on the extracted text, including word segmentation, named entity recognition, and syntactic analysis. For example, it identifies entities such as people's names, place names, and organization names in the document, as well as the grammatical relationships between sentences.

[0127] (6) Knowledge Hierarchical Chunking: The knowledge hierarchical chunking module divides the document into several knowledge blocks based on its semantic structure. It identifies the document's chapter structure, including chapters such as "Introduction," "Related Work," "Methods," "Experiments," and "Conclusions." Each chapter is considered a knowledge block. Within each chapter, it further divides the content into several thematic paragraphs. For example, in the "Methods" chapter, it divides the content into paragraphs such as "Dataset," "Model Architecture," and "Training Strategies." Through knowledge hierarchical chunking, the document's content is organized into logically related knowledge units.

[0128] (7) Title Extraction: The title extraction module extracts the title structure of the document. It identifies the first-level headings (such as "1. Introduction", "2. Related Work", etc.) and second-level headings (such as "2.1 Traditional Methods", "2.2 Deep Learning Methods", etc.). It constructs a title hierarchy tree, where the first-level headings are the root nodes, the second-level headings are the child nodes of the first-level headings, and so on. The extracted title structure clearly reflects the outline of the document.

[0129] (8) Metadata Addition: The metadata addition module adds metadata to the parsing results. It extracts basic metadata such as author, conference name, and publication date from PDF attributes; it extracts keywords (such as "multimodal large model", "document parsing", "layout analysis") and topic categories (such as "computer science" and "artificial intelligence") from the document content; it also generates a short summary of the document. In addition, it counts structural metadata such as the number of pages (10 pages), number of chapters (5 chapters), number of tables (2), and number of formulas (15). All of this metadata is appended to the parsing results.

[0130] (9) Output: The system will output the parsed result as a JSON object. This object contains the following fields: metadata: A sub-object containing all metadata.

[0131] content: Contains the document's structured text content, organized by heading hierarchy, with each chapter and paragraph clearly marked.

[0132] tables: An array containing parsed table data, with each table represented in HTML format.

[0133] formulas: An array containing the recognized formulas, each represented in LaTeX.

[0134] knowledge_blocks: An array containing the results of knowledge hierarchy blocks, each containing its title and content.

[0135] outline: An object representing the document's heading hierarchy.

[0136] Through the above processing, the originally complex academic paper is transformed into clearly structured and semantically rich data. Users can utilize this data for various downstream applications, such as inserting knowledge blocks into knowledge bases, generating navigation using title structures, and retrieving data based on metadata. This fully demonstrates the powerful capabilities and application value of this invention in the field of document parsing.

[0137] Example 3 The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 2.

[0138] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 2.

[0139] The steps and methods involved in the apparatuses of Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0140] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0141] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0142] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A document intelligent parsing system based on multimodal fusion, characterized in that, include: The input layer is used to acquire multimodal document data; The core layer of the multimodal large model is used for deep parsing of document data to obtain the parsing results; The core layer of the multimodal large model includes a layout analysis module, a formula recognition module, a table structure parsing module, and a knowledge hierarchy segmentation module. The layout analysis module analyzes the layout of the document data and identifies the type, position, and reading order of different areas; the formula recognition module identifies the formula areas and converts the formula images into code; the table structure parsing module performs structured parsing on the identified table areas to obtain structured data. The knowledge hierarchy segmentation module extracts text content from the identified text regions and divides the document data into several hierarchical knowledge blocks based on semantic analysis. The output and application layer is used to output and apply the parsed results in a structured format.

2. The document intelligent parsing system based on multimodal fusion as described in claim 1, characterized in that, The layout analysis module adopts a multimodal layout analysis architecture that integrates visual, semantic, and relational information, including a two-stream convolutional network, a multi-scale adaptive aggregation module, and a relation learning module.

3. The document intelligent parsing system based on multimodal fusion as described in claim 2, characterized in that, The dual-stream convolutional network processes visual information and semantic information respectively to obtain visual features and text features; the multi-scale adaptive aggregation module receives visual features and text features and obtains fused features through an adaptive fusion mechanism. The relationship learning module generates a candidate set of layout elements based on fusion features and learns the relationships between the elements to obtain layout analysis results.

4. The document intelligent parsing system based on multimodal fusion as described in claim 1, characterized in that, The formula recognition module includes a formula detection submodule and a formula recognition submodule. The formula detection submodule uses a target detection model to locate the formula region, and the formula recognition submodule uses a sequence-to-sequence model to convert the formula image into a LaTeX sequence.

5. The document intelligent parsing system based on multimodal fusion as described in claim 1, characterized in that, The table structure parsing module first identifies the row and column structure of the table and the position of each cell, then processes merged cells and cross-page tables, and finally converts the table content into structured data.

6. The document intelligent parsing system based on multimodal fusion as described in claim 1, characterized in that, The core layer of the multimodal large model also includes a title extraction module, which is used to automatically identify and extract the top headings in the document data and construct the title hierarchy structure of the document.

7. The document intelligent parsing system based on multimodal fusion as described in claim 1, characterized in that, The core layer of the multimodal large model also includes a metadata addition module, which is used to add metadata information to the parsed document content, including the document's basic attributes and content attributes.

8. A document intelligent parsing method based on multimodal fusion, characterized in that, include: Acquire multimodal document data; The document data is subjected to in-depth analysis to obtain the analysis results; specifically, the layout of the document data is analyzed to identify the type, location, and reading order of different areas. The identified formula regions are identified and converted into code; the identified table regions are parsed in a structured manner to obtain structured data. Text content is extracted from the identified text regions, and the document data is divided into several hierarchical knowledge blocks based on semantic analysis; The parsing results are output in a structured format and applied.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the document intelligent parsing method based on multimodal fusion as described in claim 8.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the document intelligent parsing method based on multimodal fusion as described in claim 8.

Images