Multi-modal document processing method, system and equipment

Abstract

The invention provides a multi-mode document processing method, system and equipment, and the method comprises the steps: obtaining a multi-source heterogeneous file, and converting the multi-source heterogeneous file into a standard document; performing layout element analysis on the standard document, and determining a section set; according to each document semantic block in the section set, analyzing the spatial relationship, visual flow and semantic association between the document semantic blocks so as to generate a global content sequence conforming to a reading sequence; constructing an optimal analysis pipeline according to the type of each document semantic block in the global content sequence, analyzing each document semantic block based on the optimal analysis pipeline, and determining an analysis result; and generating the analysis result into structured data, and outputting the structured data. According to the method, the reading sequence is corrected, sequence logic coherence is ensured, and the semantic disjunction problem caused by only relying on the spatial position is solved; according to the method, scattered recognition results are associated and integrated and converted into structured data, so that the problem that information is scattered and difficult to reuse is solved.

Description

Technical Field

[0001] This application relates to the field of intelligent document processing, and in particular to a multimodal document processing method, system and device. Background Technology

[0002] With the rapid development of information and intelligent technologies, document parsing and generation technologies have been widely applied in fields such as office automation, data processing, and content generation. Currently, document parsing technology has evolved to the multimodal fusion stage, enhancing understanding capabilities by combining visual, textual, and layout information, and employing model integration or pipeline approaches to handle complex tasks.

[0003] However, document parsing technology still suffers from the following shortcomings, limiting its application in real-world, ever-changing scenarios: First, the processing flow and strategies are fixed, unable to dynamically adjust according to significant differences in document type, quality, and complexity, leading to over-processing and resource waste for simple documents, while under-processing complex or low-quality documents; Second, it relies on individual heavy models or stacks a large number of single models, resulting in low computational efficiency; adding new document types requires re-labeling data and retraining, leading to poor scalability and maintainability; Third, multiple models exist in isolation, lacking effective collaboration mechanisms and knowledge sharing capabilities, making it impossible to achieve rapid adaptation to small samples and feedback-based autonomous evolution. Therefore, there is an urgent need for a technology capable of autonomously evaluating documents, intelligently scheduling multi-model resources, and possessing adaptive parsing capabilities to overcome the current technological bottlenecks in flexibility, efficiency, and generalization. Summary of the Invention

[0004] This application provides a multimodal document processing method, system, and device to solve the problem in the prior art that multimodal document processing cannot accurately and efficiently process documents based on document type and complexity when faced with multi-source heterogeneous documents.

[0005] In a first aspect, this application provides a multimodal document processing method, comprising: acquiring multi-source heterogeneous files and converting them into standard documents; performing layout element analysis on the standard documents to determine a set of blocks, the set of blocks including the geometric coordinates and type labels corresponding to each document semantic block; analyzing the spatial relationships, visual flow, and semantic associations between the document semantic blocks based on the geometric coordinates and type labels of each document semantic block in the set of blocks, so as to generate a global content sequence that conforms to the reading order; constructing an optimal parsing pipeline based on the type of each document semantic block in the global content sequence, parsing each document semantic block based on the optimal parsing pipeline, and determining the parsing result; generating structured data from the parsing result and outputting it.

[0006] In some possible embodiments of the first aspect, before performing layout analysis on the layout elements in the standard document, the method further includes: extracting features from at least a portion of the standard document to determine a feature profile, the feature profile including document type, layout style, and element distribution; if the document type in the feature profile is a preset type, then preprocessing the standard document to determine a standard document with document enhancement, wherein the preprocessing includes resolution enhancement, page rotation correction, and image quality optimization; if the document type in the feature profile is not a preset type, then directly outputting the standard document.

[0007] In some possible embodiments of the first aspect, the standard document is analyzed for layout elements to determine a set of blocks, including: calling a preset document layout analysis model to analyze the layout elements of the standard document, determining the geometric coordinates and type labels corresponding to each document semantic block; associating the geometric coordinates and type labels corresponding to each document semantic block into independent data, and packaging the independent data of each document semantic block to form a set of blocks, wherein the type labels include at least one of title, body paragraph, table, mathematical formula, chart, list, code block and comment.

[0008] In some possible embodiments of the first aspect, based on the geometric coordinates and type labels of each document semantic block in the block set, the spatial relationships, visual flow, and semantic associations between the document semantic blocks are analyzed to generate a global content sequence that conforms to the reading order. This includes: analyzing the spatial relationships, visual flow, and semantic associations between each document semantic block in the block set using a preset reading order sorting model to generate spatial relationship scores, visual flow scores, and semantic association scores; calculating the weighted scores of the spatial relationship scores, visual flow scores, and semantic association scores of each document semantic block according to preset weighting coefficients to determine the weighted scores; and sorting the document semantic blocks according to their weighted scores and reorganizing them from high to low to generate a global content sequence that conforms to the reading order.

[0009] In some possible embodiments of the first aspect, based on the geometric coordinates and type labels of each document semantic block in the block set, the spatial relationships, visual flow, and semantic associations between the document semantic blocks are analyzed to generate a global content sequence that conforms to the reading order. This includes: determining a spatial relationship matrix between the document semantic blocks based on the geometric coordinates of each document semantic block; establishing a semantic association matrix between the document semantic blocks based on the type labels and the spatial relationship matrix; determining a visual flow path based on the version structure formed by the spatial relationship matrix of the document semantic blocks and a preset global reading direction; and weightedly fusing the spatial relationship matrix, the visual flow path, and the semantic association matrix to determine a global content sequence that conforms to the reading order.

[0010] In some possible embodiments of the first aspect, an optimal parsing pipeline is constructed based on the type of each document semantic block in the global content sequence. The document semantic blocks are then parsed based on the optimal parsing pipeline to determine the parsing results. This includes: using a preset optical character recognition model to perform character recognition on document semantic blocks with at least one of the type labels: title, body paragraph, code block, and comment, to determine the document recognition result; using a preset formula positioning model to locate document semantic blocks with the type label: mathematical formula, and using a preset formula parsing model to parse the located mathematical formula, to determine the formula recognition result; using a preset table classification model to classify document semantic blocks with the type labels: chart, list, and table, to determine wired tables and wireless tables; determining the table recognition result by recognizing the content of each wired and wireless table; recognizing the type of each document semantic block in the global content sequence to construct the optimal parsing pipeline; and calling the preset parsing model according to the optimal parsing pipeline to parse each document semantic block, taking at least one of the document recognition result, table recognition result, and formula recognition result as the parsing result.

[0011] In some possible embodiments of the first aspect, the table recognition result is determined by identifying the contents of the wired table and the wireless table, including: performing wireframe detection on the wired table using a preset wired table parsing model to determine the reconstructed cells; performing text alignment and semantic spacing inference on the wireless table using a preset wireless table parsing model to determine the table structure; and fusing the recognition results of the reconstructed cells and the table structure to generate a table recognition result containing complete row and column logic and two-dimensional structured data.

[0012] In some possible embodiments of the first aspect, after generating structured data from the parsing results, the method further includes: optimizing the structured data based on reading order and semantic analysis, wherein the optimization method includes automatically connecting broken paragraphs, list items or tables across pages and filling in missing logical connectors, wherein the reading order and semantic analysis are determined by the global content sequence; verifying whether the document structure in the optimized structured data has discontinuous title numbering, missing chart references or missing key chapters, and generating a verification report.

[0013] In a second aspect, this application provides a multimodal document processing system, comprising: an acquisition module for acquiring multi-source heterogeneous files and converting them into standard documents; a layout analysis module for analyzing the layout elements of the standard documents to determine a set of blocks, the set of blocks including the geometric coordinates and type labels corresponding to each document semantic block; a sequence reorganization module for analyzing the spatial relationships, visual flow, and semantic associations between document semantic blocks based on the geometric coordinates and type labels of each document semantic block in the set of blocks, so as to generate a global content sequence that conforms to the reading order; an intelligent parsing module for constructing an optimal parsing pipeline based on the type of each document semantic block in the global content sequence, parsing each document semantic block based on the optimal parsing pipeline, and determining the parsing result; and a structured data module for generating structured data from the parsing result and outputting it.

[0014] In a third aspect, this application also provides an electronic device, including a processor, a memory, and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the multimodal document processing system described above.

[0015] The beneficial effects of this application are as follows: It acquires multi-source heterogeneous files and converts them into standard documents; it eliminates format barriers, forming a unified standard document; it analyzes the layout elements of the standard document to determine the block set, and by obtaining the geometric coordinates and type labels of the document semantic blocks, it solves the problems of inaccurate positioning and ambiguous types in traditional layout analysis, improving the accuracy of document layout element recognition; based on the geometric coordinates and type labels of each document semantic block in the block set, it analyzes the spatial relationships, visual flow, and semantic associations between the document semantic blocks to generate a global content sequence that conforms to the reading order; it corrects the reading order, ensuring logical coherence of the sequence and solving the semantic disconnect problem caused by relying solely on spatial location; it constructs an optimal parsing pipeline based on the type of each document semantic block in the global content sequence, and parses each document semantic block based on the optimal parsing pipeline to determine the parsing results; it generates structured data from the parsing results and outputs it, correcting recognition errors through deep recognition combined with global semantic cross-validation, solving the drawbacks of independent recognition of various types of content, and ensuring the accuracy of the recognition results; it converts the parsing results into structured data, solving the problems of information dispersion and difficulty in reuse. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0017] In the attached diagram:

[0018] Figure 1 A flowchart of a multimodal document processing method provided in an embodiment of this application; Figure 2 A flowchart illustrating the document standardization process in a multimodal document processing method provided in an embodiment of this application; Figure 3 A flowchart of document parsing in a multimodal document processing method provided in an embodiment of this application; Figure 4 A flowchart illustrating the document structure generation process in a multimodal document processing method provided in an embodiment of this application; Figure 5 This is a structural block diagram of a multimodal document processing system provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 7 An example diagram of a multimodal document processing system provided in an embodiment of this application; Figure 8 This is another example diagram of a multimodal document processing system provided in an embodiment of this application. Detailed Implementation

[0019] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0022] In related technologies, multi-source heterogeneous files include documents, tables, and image documents of different formats. Due to the lack of uniformity in format, they are difficult to process and parse in a unified manner. The lack of systematic analysis of document layout elements makes it impossible to accurately obtain the position and type information of each document's semantic blocks, resulting in inaccurate recognition of spatial, visual, and semantic relationships between document semantic blocks. Consequently, it is impossible to generate content sequences that conform to human reading habits. The recognition of documents, tables, and formulas is independent of each other, lacking an effective information fusion mechanism. It is difficult to output structured data, making it impossible to efficiently extract and reuse effective information in documents. This seriously affects the efficiency and accuracy of document processing and fails to meet the needs of enterprise-level data integration, knowledge management, and other scenarios for the structured processing of multi-source heterogeneous documents.

[0023] To address the aforementioned problems, this application proposes a multimodal document processing method, system, and device. Please refer to... Figure 1 The above is a flowchart illustrating a multimodal document processing method provided in an embodiment of this application, including: Step S110: Obtain multi-source heterogeneous files and convert them into standard documents; For example, multi-source heterogeneous files are acquired, including but not limited to PDF, Word, Excel, image, TXT, and document data returned by Web APIs. Format parsing is performed on each type of heterogeneous file, employing corresponding parsing strategies for different formats: for editable documents, the original content and format information are extracted using toolchains such as PyPDF2 and Python-docx; for image files and scanned documents, OCR technology is used to extract text content and layout outline information; for PDF files, editable PDFs and scanned PDFs are distinguished, with content directly extracted from editable PDFs, while scanned PDFs are first converted to editable text using OCR; all parsed files are converted into standard documents of a preset format, ensuring that the standard documents have a unified encoding format, layout specifications, and content structure.

[0024] Step S120: Analyze the layout elements of the standard document to determine the block set. The block set includes the geometric coordinates and type labels corresponding to the semantic blocks of each document. For example, in one implementation, a Transformer-based instance segmentation architecture performs layout element analysis on a standard document, preprocesses the document to remove interfering information such as headers, footers, watermarks, and hyperlinks, enhancing document layout clarity. A CNN backbone network extracts multi-scale features from the document page, including shallow features for capturing details such as text edges and deep features for capturing semantic category information. A Transformer decoder is used to determine a potential document semantic block corresponding to each query, generating a pixel-level mask and polygonal bounding box for each semantic block. Precise geometric coordinates are extracted from the mask; for example, using a page coordinate system with the top-left corner of the page as the origin, the x-axis as the horizontal axis, and the y-axis as the vertical axis, representing the position and size of the semantic block. A multi-task classification model is used to type-label each semantic block, generating type labels, including text blocks, table blocks, formula blocks, image blocks, title blocks, etc. The geometric coordinates and type labels of all semantic blocks are then aggregated to form a block set.

[0025] Step S130: Based on the geometric coordinates and type labels of each document semantic block in the block set, analyze the spatial relationship, visual flow and semantic association between the document semantic blocks to generate a global content sequence that conforms to the reading order; For example, based on the geometric coordinates of each document semantic block in the section set, the spatial relationships between document semantic blocks are analyzed, including adjacency, inclusion, overlap, and relative positional relationships, such as top, bottom, left, right, and center. The degree of spatial association is quantified by calculating the distance and intersection of the bounding boxes of document semantic blocks. Combining human reading habits, such as from left to right, from top to bottom, title before body text, and table before description, a visual flow model is constructed based on the geometric coordinates and type labels of document semantic blocks to simulate the eye movement trajectory during human reading and determine the reading order of document semantic blocks. Through pre-trained language models, such as Huawei's Pangu Big Data Model or Baidu's Big Data Model, the textual semantic features of each document semantic block are extracted, semantic similarity is calculated, and the semantic associations between document semantic blocks, such as causal, subordinate, and parallel relationships, are analyzed to correct the reading order. By integrating the results of spatial relationship, visual flow, and semantic association analysis, a global content sequence that conforms to human reading habits and is semantically coherent is generated, so that the arrangement logic of each document semantic block is consistent with the document semantics.

[0026] Step S140: Construct an optimal parsing pipeline based on the type of each document semantic block in the global content sequence, parse each document semantic block based on the optimal parsing pipeline, and determine the parsing result; For example, a global content sequence corresponding to the semantic blocks of the document to be parsed is obtained. This global content sequence consists of several document semantic blocks arranged in spatial order or reading logical order. Each document semantic block is type-identified to determine its category, including at least one of text semantic blocks, table semantic blocks, formula semantic blocks, and other structured semantic blocks. Based on the correspondence between semantic block types and preset parsing strategies, corresponding parsing models, parsing orders, and parsing parameters are assigned to different types of semantic blocks, forming an optimal parsing pipeline adapted to the current document's global content sequence. The optimal parsing pipeline indicates the parsing path, model calling method, and execution priority for each semantic block. Following the parsing order and model calling rules defined by the optimal parsing pipeline, the corresponding text recognition model, table recognition model, and formula recognition model are called sequentially or in parallel to perform parsing processing on the corresponding types of document semantic blocks. The recognition results corresponding to each semantic block are obtained, and the results are spliced, verified, and structurally integrated to remove redundant information and ensure logical coherence, ultimately forming the overall parsing result.

[0027] Step S150: Generate structured data from the parsed results and output it.

[0028] For example, each parsing result is assigned a corresponding data identifier and associated index, text content is mapped to key-value pair data, table content is mapped to a data array structure corresponding to rows and columns, and formula content is mapped to standardized expression data, forming structured data containing field identifiers, content information, and relationships.

[0029] In some embodiments, before performing layout analysis on layout elements in a standard document, the method further includes: Feature extraction is performed on at least a portion of the standard document to determine a feature profile, which includes document type, layout style, and element distribution. If the document type in the feature profile is a preset type, then the standard document is preprocessed to determine the standard document for document enhancement. The preprocessing includes resolution enhancement, page rotation correction and image quality optimization. If the document type in the feature profile is not a preset type, then a standard document will be output directly.

[0030] For example, a standard document is acquired, and a feature extraction algorithm is used to extract features from all or part of the document's content. During the extraction process, features such as document type, layout style, and element distribution are collected. Document type features include document format identifiers, content keywords, and file extensions. Layout style features include typesetting, font specifications, line spacing, and margins. Element distribution features include the position, proportion, and arrangement patterns of elements such as text, images, tables, and formulas. Integrating and normalizing the collected feature information allows for the construction of a feature profile.

[0031] For example, the document type in the feature profile is matched with a preset type in a preset type library. After confirming a successful match, the preprocessing process is initiated. The preprocessing performs the following operations in sequence: by detecting the current resolution of the standard document, if it is lower than a preset resolution threshold, an interpolation algorithm (such as bilinear interpolation) is used to supplement the document pixels to improve the document resolution; the tilt angle of the document page is detected by Hough transform, and correction parameters are calculated based on the tilt angle to rotate and adjust the tilted page so that the document page remains horizontal or vertical, thus completing the page rotation correction; noise, blur, and insufficient contrast in the document are detected, and a Gaussian filtering algorithm is used to remove noise, and a contrast enhancement algorithm is used to optimize image brightness and contrast, repair blurry areas, and improve the overall image quality of the document; after the preprocessing is completed, the enhanced standard document is output.

[0032] The document type in the feature profile is matched with the preset type in the preset type library. If the match fails, the standard document is determined to be unnecessary for preprocessing. The original standard document can be directly output as the processing result without performing any additional enhancement or correction operations.

[0033] By employing the above methods and constructing feature profiles through feature extraction, precise differentiation of document types is achieved. This allows for targeted preprocessing of documents of preset types, while documents of non-preset types are directly output. This addresses the lack of specificity in traditional uniform processing models. Differentiated processing based on document type accurately matches processing needs, avoiding indiscriminate processing. Simultaneously, preprocessing operations such as resolution enhancement, page rotation correction, and image quality optimization specifically address issues related to document resolution, tilt, and quality, eliminating interfering factors that affect document usability and resolving quality defects present in documents of preset types. This improves document clarity, regularity, and readability. Furthermore, simple documents using non-preset types do not require preprocessing operations, reducing unnecessary calculations, shortening the processing cycle, and avoiding unnecessary preprocessing steps that consume computational resources. This simplifies the processing workflow as needed, improving document processing efficiency and resource utilization.

[0034] In some embodiments, page layout element analysis is performed on a standard document to determine a set of sections, including: The preset document layout analysis model is invoked to perform layout element analysis on the standard document and determine the geometric coordinates and type labels corresponding to each semantic block of the document. The geometric coordinates and type labels corresponding to each document semantic block are associated as independent data. The independent data of each document semantic block are packaged to form a block set. The type labels include at least one of the following: title, body paragraph, table, mathematical formula, chart, list, code block and comment.

[0035] For example, a preset document layout analysis model is invoked to process a standard document. The standard document is scanned and its features analyzed region by region to identify each semantic block within the document and delineate the boundary range of each semantic block. Based on the boundary range, the geometric coordinates of each semantic block are calculated. These geometric coordinates represent the two-dimensional coordinate system of the document page, determining the specific position of the semantic block on the page, such as the coordinates of the top left corner, bottom right corner, or center point, as well as its width and height. Simultaneously, based on the content and format characteristics of each semantic block, its type is determined, and a corresponding type label is assigned to each semantic block. The type label is selected from at least one of the following: title, body paragraph, table, mathematical formula, chart, list, code block, and comment. The geometric coordinates and type label corresponding to each semantic block are output, forming a block set.

[0036] Based on the geometric coordinates and type labels of each document semantic block, a one-to-one correspondence is established. The geometric coordinate data and type label data of a single document semantic block are integrated into a set of independent data, and a unique document semantic block identifier is assigned to each set of data to distinguish different document semantic blocks. All document semantic blocks are sorted and classified according to their spatial position on the document page, such as from top to bottom, from left to right, or by type label. All sorted independent data are packaged to form a block set, which contains the unique identifier, geometric coordinates, and type label association information of each document semantic block.

[0037] By using the above method and a pre-set document layout analysis model, the boundaries, geometric coordinates, and type labels of document semantic blocks can be accurately identified, solving the problem of chaotic document semantic block identification. By associating geometric coordinates and type labels as independent data, the positional information and type information of document semantic blocks correspond to each other, solving the problems of scattered and insufficiently associated document semantic block information. At the same time, the management difficulty of document semantic block data is reduced by using block sets, improving the efficiency of document processing.

[0038] In some embodiments, based on the geometric coordinates and type labels of each document semantic block in the block set, the spatial relationships, visual flow, and semantic associations between the document semantic blocks are analyzed to generate a global content sequence that conforms to the reading order, including: By using a pre-defined reading order sorting model, the spatial relationships, visual flow, and semantic associations between semantic blocks of documents in the section set are analyzed, and spatial relationship scores, visual flow scores, and semantic association scores are generated. The spatial relationship score, visual flow score, and semantic association score of each document semantic block are weighted according to preset weighting coefficients to determine the weighted score value; The document is sorted according to the weighted scores corresponding to its semantic blocks, and then reorganized in descending order of scores to generate a global content sequence that conforms to the reading order.

[0039] For example, a preset reading order sorting model is invoked. Based on the geometric coordinates of each document semantic block, the positional relationship between adjacent document semantic blocks is analyzed, such as top-bottom, left-right, and containment relationships, quantifying the degree of spatial association and generating a spatial relationship score. Combining the visual layout rules of the document page, the visual guidance order of each document semantic block is analyzed, for example, through text reading direction and visual focus priority, quantifying the visual flow fit and generating a visual flow score. The content features of each document semantic block are extracted, the logical relationship between document semantic blocks is analyzed, the degree of semantic fit is quantified, and a semantic association score is generated. The spatial relationship score, visual flow score, and semantic association score corresponding to each document semantic block are output, generating the spatial relationship score, visual flow score, and semantic association score for each document semantic block. Preset weight coefficients are invoked, which include preset weight coefficients corresponding to the three-dimensional scores. Among them, the weight of the semantic association score is not lower than the weight of the spatial relationship and visual flow scores to ensure semantic logic priority. For each document semantic block, the scores of the three dimensions are multiplied by the corresponding weight coefficients and then summed to calculate the weighted score value of each document semantic block, and the weighted score value of all document semantic blocks is output. Furthermore, for each document semantic block, a weighted score is assigned, and the unique identifier of the document semantic block is associated with its corresponding weighted score. All document semantic blocks are then sorted in descending order of weighted score, with higher scores indicating higher reading priority. During the sorting process, if document semantic blocks have the same weighted score, they are re-sorted based on their spatial relationship scores to ensure the sorting result is unique. After sorting, all document semantic blocks are reorganized in this order, and the content and related information of each document semantic block are integrated to generate a global content sequence.

[0040] By comprehensively analyzing spatial relationships, visual flow, and semantic connections across three dimensions, and combining weighted calculations and sorting, the sorting results are aligned with user reading habits, resolving the issue of illogical sorting logic and ensuring the sequence conforms to normal reading logic. Quantitative analysis generates scores, and combined with a preset weighting mechanism, avoiding biases from single-dimensional sorting. Multi-dimensional score quantification eliminates the ambiguity of qualitative analysis, generating a global content sequence that conforms to the reading order. This simplifies the user reading process, eliminates reading obstacles caused by disordered arrangement, and improves reading efficiency and document usage experience.

[0041] In some embodiments, based on the geometric coordinates and type labels of each document semantic block in the block set, the spatial relationships, visual flow, and semantic associations between the document semantic blocks are analyzed to generate a global content sequence that conforms to the reading order, including: Based on the geometric coordinates of each document semantic block, determine the spatial relationship matrix between the document semantic blocks; Based on type tags and spatial relationship matrices, a semantic association matrix is ​​established between semantic blocks of each document. The visual flow path is determined based on the version structure formed by the spatial relationship matrix of document semantic blocks and the preset global reading direction; By weighted and fused spatial relationship matrix, visual flow path, and semantic association matrix, a global content sequence that conforms to the reading order is determined.

[0042] For example, the geometric coordinates of each document semantic block are extracted, the total number of document semantic blocks N is counted, and an N×N spatial relationship matrix is ​​constructed. The rows and columns in the spatial relationship matrix correspond to the unique identifiers of each document semantic block, and the matrix element values ​​represent the degree of spatial association between the document semantic blocks in the corresponding rows and columns. Based on the geometric coordinates, the spatial parameters of each pair of document semantic blocks, such as distance, orientation, and containment relationship, are calculated to ensure that the spatial relationship matrix can accurately represent the spatial association features between all document semantic blocks, so as to output the spatial relationship matrix.

[0043] For example, an N×N dimensional semantic association matrix with the same dimensions as the spatial relationship matrix is ​​constructed. Based on type labels, the type association between semantic blocks of two documents is analyzed. If the semantic blocks of two documents have the same type or have a logical relationship, such as title and body text, table and annotation, a basic association score is assigned. The basic association score is corrected by combining the corresponding element values ​​in the spatial relationship matrix. The stronger the spatial association, the greater the correction of the semantic association score, thus realizing the synergistic empowerment of spatial relationship and type labels. The corrected association score is quantized to the 0-1 range and assigned to the corresponding element of the semantic association matrix. The semantic association matrix is ​​then output.

[0044] For example, based on the spatial correlation of each element in the spatial relationship matrix, the arrangement rules of the document semantic blocks are sorted out to construct the document version structure, such as hierarchical structure, parallel structure, and nested structure, and the level and position of each document semantic block in the version structure are determined; the preset global reading direction configuration is invoked, such as the conventional reading direction from left to right or from top to bottom or the custom reading direction for special documents; combining the version structure and the global reading direction, the visual flow path is planned: the global reading direction is given priority, and the layers are sorted according to the spatial correlation within the same level, and the layers are advanced sequentially according to the hierarchical order of the version structure, ensuring that the visual flow path conforms to the document layout logic and user reading habits, and the visual flow path is output.

[0045] The process involves obtaining the spatial relationship matrix, visual flow path, and semantic association matrix, and assigning them corresponding preset weight coefficients. The sum of these preset weight coefficients is 1, with the semantic association matrix having the highest weight. The weights of the spatial relationship matrix and visual flow path are dynamically adjusted based on the document type. The spatial relationship matrix and semantic association matrix are standardized, unifying the matrix element values ​​to the same quantization range, consistent with the quantization scores of the visual flow path (assigned scores from 1 to N according to the path order). A weighted summation algorithm is used to multiply the quantization values ​​of these three types of data by their corresponding weight coefficients and then sum them to obtain the comprehensive ranking score for each document semantic block. All document semantic blocks are sorted in descending order of comprehensive ranking scores. If scores are the same, a secondary sort is performed, referring to the order of the visual flow path. After sorting, all document semantic blocks are reorganized to generate a global content sequence that conforms to the reading order.

[0046] By employing the above methods, the spatial relationships and semantic associations between semantic blocks of each document are represented in a matrix manner, avoiding omissions and quantification biases in association information and solving the problem of inaccurate association representation. Combining version structure and preset global reading direction addresses the issue of unreasonable path planning; the version structure reflects the physical layout of the document, and the global reading direction aligns with user habits. A standardized weighted fusion mechanism avoids sorting biases, solves the problem of unscientific multi-dimensional fusion, and improves sorting accuracy and coherence. A global content sequence that conforms to the reading order is generated, eliminating reading obstacles caused by disordered arrangement, aligning with reading habits, simplifying the reading process, and improving reading efficiency and document user experience.

[0047] In some embodiments, an optimal parsing pipeline is constructed based on the type of each document semantic block in the global content sequence. The optimal parsing pipeline is used to parse each document semantic block, and the parsing result is determined, including: By using a pre-defined optical character recognition model, character recognition is performed on document semantic blocks whose type labels are at least one of title, body paragraph, code block and comment, and the document recognition result is determined. By using a preset formula localization model, the semantic blocks of documents with the type tag of mathematical formula are located. By using a preset formula parsing model, the located mathematical formulas are parsed to determine the formula recognition result. By using a pre-defined table classification model, semantic blocks of documents labeled as charts, lists, and tables are classified to determine wired and wireless tables; the table recognition result is determined by identifying the content of each wired and wireless table. Identify the types of semantic blocks in each document within the global content sequence to construct the optimal parsing pipeline; The optimal parsing pipeline calls the preset parsing model to parse each document semantic block, and takes at least one of the document recognition result, table recognition result and formula recognition result as the parsing result.

[0048] For example, the document semantic blocks with type labels are filtered out to identify at least one of the following types: title, body paragraph, code block, and comment. The geometric coordinates and content images of such document semantic blocks are extracted. A preset optical character recognition model is called, and the content image is input. The characters in the content image are scanned line by line, and features are extracted to identify the font, size, format, and content of the characters. The grammatical format of the document semantic blocks is distinguished from the annotation format of the comments, and the layout logic of the titles and body paragraphs is preserved. The identified character content is verified, and recognition errors are corrected, such as correcting similar-looking characters and blurry characters. By integrating the identified content of each document semantic block and combining its geometric coordinates and type labels, a document recognition result is formed.

[0049] The process involves: acquiring document semantic blocks labeled "mathematical formula" from a collection of blocks, extracting their geometric coordinates and content images; calling a preset formula positioning model, inputting the content image of the mathematical formula document semantic block, and accurately locating the formula's specific range within the document semantic block by recognizing the formula's outline and symbol features, eliminating interference from non-formula areas, and outputting the formula's precise positioning coordinates to obtain the positioned formula image; inputting the positioned formula image into a preset formula parsing model to identify and parse the symbols, operators, variables, and structural relationships in the formula, restoring the formula's logical structure and mathematical meaning, and converting the image-based formula into editable and calculable formula text; verifying the parsing results, correcting symbol recognition errors and structural parsing deviations, and combining the formula positioning coordinates with the document semantic block information to form the formula recognition result.

[0050] The system retrieves document semantic blocks tagged with charts, lists, and tables from a collection of sections, extracting their geometric coordinates, content images, and type labels. A pre-defined table classification model is then invoked, inputting the content images and feature information of the document semantic blocks. By recognizing the structural features of the document semantic blocks, the model classifies them, distinguishing between charts, lists, wired tables, and wireless tables. Chart and list-type document semantic blocks are eliminated, while wired and wireless table document semantic blocks are retained. For wired tables, the distribution of table lines is identified to determine row and column boundaries, extracting the content within table cells (e.g., text and numbers) to reconstruct the table structure. For wireless tables, the element arrangement patterns and spacing features within cells are identified to determine row and column divisions, extracting cell content. By integrating the structural and content information of wired and wireless tables, combined with their geometric coordinates, a table recognition result is generated.

[0051] By employing the above methods, document semantic block types are accurately identified, and an optimal parsing pipeline is constructed, improving parsing accuracy and efficiency. Dedicated recognition models and differentiated processing methods are used for different types of document semantic blocks to avoid errors caused by uniform recognition, solving the problem of low recognition accuracy and effectively reducing recognition errors. Through a two-step positioning-parsing method, the location and semantic information of formulas are acquired simultaneously, solving the problem of disconnect between formula recognition and parsing. The two work together to achieve complete formula recognition, improving the practicality of formula recognition. A differentiated recognition method is used to distinguish between wired and wireless tables, solving the problem of chaotic table recognition. Dedicated recognition methods are used for the structural differences between wired and wireless tables to ensure accurate restoration of table structure and content, achieving accurate classification and recognition of semantic blocks in table-type documents. Dedicated models and differentiated processing methods balance recognition accuracy and efficiency, improving the overall efficiency of document processing. Pre-set parsing models are used to perform targeted parsing of each document's semantic blocks, and the results are verified and integrated to ensure the accuracy and completeness of the parsing results.

[0052] In some embodiments, the table identification result is determined by identifying the content of the wired table and the wireless table, including: By using a pre-defined wired table parsing model, wireframe detection is performed on the wired table to determine the cells to be reconstructed; By using a pre-defined wireless table parsing model, text alignment and semantic spacing inference are performed on the wireless table to determine its structure. By integrating the recognition results of reconstructed cells and table structures, a table recognition result containing complete row and column logic and two-dimensional structured data is generated.

[0053] For example, the semantic blocks of a wired table document are obtained, and its content image and geometric coordinates are extracted. A preset wired table parsing model is called, and the wired table content image is input. The image is preprocessed (denoising and enhancement) to highlight the outline features of the table lines. Then, wireframe detection is performed to identify the horizontal borders, vertical borders, and intersections of the table, and the coordinate information of each border and the position of the intersection are recorded. Based on the border coordinates and the distribution of intersections, the row and column boundaries of the table are divided, merged cells are identified, and the boundary range and belonging relationship of each cell are determined by the missing status of adjacent borders and the distribution pattern of intersections. The cell boundaries are calibrated to correct the boundary errors caused by border offset and distortion, and standardized reconstructed cells are generated. The reconstructed cells contain cell boundary coordinates, row numbers, column numbers, and belonging relationship information.

[0054] For example, the semantic blocks of the wireless table document are obtained, and their content images, geometric coordinates, and text content are extracted. A preset wireless table parsing model is called, and the content images and text content of the wireless table are input into the preset wireless table parsing model. Character recognition and word segmentation are performed on the text content, and the alignment features of the text, such as horizontal alignment, vertical alignment, and indentation alignment, are extracted. Then, the semantic spacing between the texts is analyzed. Combined with the layout rules of the wireless table, the row and column belonging relationships of the text blocks are inferred. Text blocks with close semantic association and spacing that meet the preset threshold are divided into the same row or the same column. Based on the text alignment features and the semantic spacing inference results, the row and column boundaries of the wireless table are divided, the cell position corresponding to each text block is determined, the number of rows and columns of the table and the distribution of cells are determined, and the table structure is generated to ensure that the table structure is consistent with the semantic logic and layout rules of the text. The process involves acquiring reconstructed cell data from wired tables and structural data from wireless tables. Through standardization, a unified coordinate system and data format are established, and fusion rules are created. The boundary coordinates and attribution relationships of the reconstructed cells are matched with the row and column divisions and text block information of the wireless tables to align row and column logic and correct conflicting information in the two types of data, such as inconsistent row and column numbers or cell position deviations. The text and numerical content from the reconstructed cells and wireless tables are extracted and filled into the corresponding cells to supplement the cell content information. Finally, the row and column logic, cell boundaries, and cell content are integrated to generate a table recognition result containing complete row and column logic and two-dimensional structured data.

[0055] The above methods achieve accurate cell reconstruction through wireframe detection, identify merged cells and correct boundary deviations, thus solving the problem of inaccurate wired table structure restoration and improving the accuracy of wired table cell reconstruction. By inferring text alignment and semantic spacing, combined with text layout and semantic logic, the table structure is determined, resolving the problem of chaotic wireless table structure recognition and improving the rationality of wireless table structure recognition. Standardized processing and fusion rules eliminate data conflicts, integrate row and column logic with content information, and solve the problem of inconsistent fusion, achieving efficient fusion of the two types of table recognition results. The generated table recognition results contain complete row and column logic and structured data, reducing manual correction costs, improving table processing efficiency, and enhancing the practicality of the table recognition results.

[0056] In some embodiments, after generating structured data from the parsed results, the method further includes: The structured data is optimized based on reading order and semantic analysis. The optimization methods include automatically connecting broken paragraphs, list items or tables across pages and filling in missing logical connectors. The reading order and semantic analysis are determined by the global content sequence. The system verifies whether the document structure in the optimized structured data has discontinuous title numbering, missing chart references, or missing key chapters, and generates a verification report.

[0057] Optionally, the document recognition results, table recognition results, and formula recognition results are input into a preset parsing model to extract content features, semantic features, and layout features of each type of result. Based on semantic importance, the recognition results are hierarchically divided, prioritizing the retention of titles, core text, and other content to determine the affiliation of each level of the directory. Combining the logical relationships between the recognition results, such as similar themes, causal relationships, and progressive relationships, the content arrangement order is sorted out, and the correspondence between tables, formulas, and text content is integrated. Referring to document layout features, such as title levels, paragraph spacing, and content arrangement, a multi-level document directory and chapter structure are generated, and tables and formulas are embedded into the corresponding chapters in a logical order. All information is integrated to generate structured data containing multi-level directories, chapter structures, text, tables, and formulas.

[0058] For example, the reading order information and semantic features of each content item are extracted from the structured data. Based on the reading order, the structured data is checked segment by segment to identify broken paragraphs, list items, or tables across pages. By analyzing the semantic similarity and layout of the breaks, the broken content is automatically pieced together to ensure the integrity and coherence of the content. Semantic analysis is performed on the structured data to identify content fragments with logical breaks, such as missing causal, progressive, or adversative relationships. Based on the contextual semantics, corresponding logical connectors, such as "therefore" and "however," are automatically filled in to improve the content logic. After optimization, the data is validated to check the accuracy of the piecing and the suitability of the connectors, ensuring that the optimized structured data is logically fluent, complete, and conforms to user reading habits.

[0059] The system invokes a preset validation rule library, which includes heading numbering standards, chart citation standards, and key chapter judgment criteria. It then validates the document structure of the structured data one by one: checking the heading numbers at all levels to determine if there are any discontinuous issues such as skipped numbers, duplicates, or inconsistent formats; checking all charts for missing references; comparing the key chapter judgment criteria to check for core chapters and determining if there are any omissions in the introduction, conclusion, or implementation methods; recording all issues found during validation, noting the location, type, and rectification suggestions, and generating a validation report in a preset format.

[0060] By employing the above methods and a pre-defined parsing model, multi-type recognition results are fused to generate structured data containing multi-level directories and chapter structures. This addresses the issues of scattered multi-source data and low structuring levels, improving the fusion efficiency and structuring level of multi-source recognition results. Furthermore, by fixing page breaks and filling in logical connectors, issues of incoherent content and logical gaps are resolved, enhancing the coherence and readability of the structured data. Specialized verification promptly identifies document structure vulnerabilities, generating verification reports to provide a basis for rectification, resolving structural non-standardization issues, and ensuring the standardization and reliability of the structured data. Finally, it eliminates the need for repeated corrections of existing data; standardized structured data reduces manual rectification costs and improves document processing efficiency.

[0061] Phase 1: Document Standardization and Multidimensional Feature Awareness (Please refer to [link]). Figure 3 The following is a detailed flowchart of the document parsing process in a multimodal document processing method provided in an embodiment of this application: It receives heterogeneous documents (i.e., heterogeneous files) from users, such as Word, Excel, PPT, PDF, and images (JPG, PNG). Through document standardization, it converts all non-PDF original documents into PDF format (i.e., standard documents) without loss or with high fidelity, eliminating the diversity of source formats and providing stable and consistent input for all vision- or text-based analysis modules.

[0062] Through lightweight and fast classification and perception, it calls a fast classification model based on a simplified YOLO architecture to perform millisecond-level scanning of PDF documents; it can not only identify common document types such as contracts, invoices, academic papers, and reports with high accuracy, but also perform preliminary visual feature analysis at the same time, and output feature profiles including document type, layout style prediction, and distribution of main elements such as table density and formula proportion.

[0063] The second phase, intelligent parsing orchestration and collaborative execution, constructs an optimal parsing pipeline. Please refer to [link / reference needed]. Figure 4 The following is a flowchart of document structure generation in a multimodal document processing method provided in an embodiment of this application, which is described in detail below: Enhancements through document preprocessing include at least the following: High-resolution rendering renders PDF pages at a high resolution of 300-600, ensuring the clarity of fine fonts and typographic lines.

[0064] Intelligent orientation correction uses the paddle_orientation_classification model (document orientation classification model) to detect and correct page rotation, ensuring consistent visual orientation of the content.

[0065] Image quality optimization employs adaptive algorithms to perform denoising, sharpening, contrast stretching, and intelligent binarization on images, effectively improving the recognition rate of low-quality original documents.

[0066] By detecting fine-grained layout elements, such as driving doclayout_YOLO, the document layout analysis model, pixel-level layout analysis is performed to accurately detect and classify more than ten types of document semantic blocks, including headings, body paragraphs, tables, mathematical formulas, charts, headers and footers, lists, and code blocks, and outputting a set of blocks with geometric coordinates and type labels.

[0067] By reorganizing the reading order through deep reading analysis, and using a layout_reader (reading order sorting model) based on a set of page layouts, the system reconstructs a global content order that conforms to human reading logic by analyzing the spatial relationships, visual flow, and semantic connections between pages. This allows for intelligent handling of complex scenarios such as multi-column layouts, text-image wrapping, irregular content areas, and cross-page content connections, generating a coherent and orderly text flow blueprint.

[0068] Perform high-precision text recognition services, for example, using the Paddle OCR engine (high-precision OCR recognition model), and combine it with page layout information. For document semantic blocks such as body paragraphs and titles, perform high-priority OCR; for document semantic blocks such as formulas, mask or process them with low priority to avoid invalid recognition and achieve resource optimization.

[0069] The table parsing service, classification, and routing utilizes the paddle_table_cls table classification model to categorize detected table regions as "wired," "wireless," or "complex." A dedicated engine then parses the data: wired tables are handled by the unet_structure finite table parsing model for wireframe detection and cell reconstruction; wireless tables are parsed using the slanet_plus wireless table parsing model, which infers structure through text alignment and semantic spacing; and complex tables are parsed using a model fusion strategy. The reconstructed table structure is combined with the OCR results of each cell to generate two-dimensional structured data containing complete row and column logic and content.

[0070] Formulas are located using mathematical formula recognition services. For example, YOLO_v8_mfd (an improved object detection model based on YOLOv8) is used to accurately detect all formula regions. unimernet_small (a mathematical expression recognition model) is used to convert formulas in the formula region into standard LaTeX or MathML formats, achieving symbolization, editability, and computability of the formulas, and enabling structured encoding.

[0071] Through semantic reconstruction and knowledge enhancement, the extracted titles, their visual hierarchy (inferred from font size and position), and content are input into a large language model (qwen3). By integrating semantic importance, logical relationships, and layout features, a logically rigorous multi-level document directory and chapter structure are generated. Content coherence repair includes automatically connecting broken paragraphs, list items, or tables across pages based on reading order and semantic analysis, and filling in missing logical connectors to ensure the overall fluency of the output text. Completeness quality verification includes automatically checking the generated document structure for issues such as discontinuous title numbering, missing chart references, and omissions of key chapters, and generating a verification report.

[0072] Phase 3: Structured JSON generation and standardized output. Please refer to [link / reference]. Figure 5 The following is a structural block diagram of a multimodal document processing system provided in an embodiment of this application, which is described in detail below: All processing results are encapsulated into a product, a structured JSON data packet containing a complete information hierarchy and supporting both machine and human-friendly interfaces. The output structure is detailed below: Document metadata includes, but is not limited to, unique identifier, original file name, processing timestamp, total number of pages, language, and confidence score.

[0073] Fine-grained layout information includes, but is not limited to, the geometric coordinates, type, identified text, and reading order of the semantic blocks of a document organized by page.

[0074] The structured content entities are detailed below: Text streams include, but are not limited to, reconstructed, complete, coherent plain text.

[0075] Tabular data, stored in a row-column structure, with original region coordinates and classification information, and optionally provides simplified JSON export in key-value pair format.

[0076] Mathematical formulas, with LaTeX / MathML code including coordinates and context.

[0077] Intelligent catalog, a hierarchical chapter outline generated by a large language model.

[0078] The key entity database extracts key business entities and their attributes, such as dates, amounts, company names, and terms.

[0079] Knowledge graphs are graph-structured data that express the relationships between concepts.

[0080] For details of the report generated using the above method, please refer to [link / reference]. Figure 7 and Figure 8 These are two example diagrams of a multimodal document processing system provided in an embodiment of this application.

[0081] The list of models used is as follows:

[0082] 2. List of Recognition Result Types

[0083] In some embodiments, by employing a multimodal document processing method, an intelligent, collaborative, and adaptive document parsing and structuring framework is constructed. Through a unified knowledge representation and scheduling mechanism, the transformation from a static pipeline to a dynamic pipeline is achieved. This application has the following technical effects: First, an adaptive intelligent orchestration engine is constructed, abandoning the fixed pipeline and introducing an intelligent scheduling center based on a lightweight multi-dimensional perception and decision-making engine. The input document is inspected, its image quality and layout complexity are evaluated, and its document type is predicted, thereby dynamically generating and executing the optimal processing path.

[0084] For simple documents, unnecessary enhancement or complex analysis modules can be skipped, and a lightweight approach can be adopted to reduce costs and increase efficiency. For complex or low-quality documents, more powerful preprocessing, layout analysis and recognition processing can be adaptively called or combined to address specific problems, thereby achieving the optimal balance between efficiency and accuracy overall.

[0085] Second, for complex structures, a dynamic parsing strategy pool is pre-built. For the most challenging complex structures in documents, such as irregular tables, dense formulas, and multi-column mixed layouts, a strategy pool composed of multiple dedicated parsers (rules, deep learning, image processing, etc.) is constructed. The intelligent orchestration engine not only calls the strategies, but also dynamically combines multiple strategies for parallel or serialized processing based on the characteristics of local areas, and selects or merges the results based on confidence.

[0086] When faced with complex substructures, instead of relying on a single model for blind guessing, the success rate and accuracy of processing are improved through the collaboration and competition of multiple methods, and the ability to generalize to new formats and irregular documents is enhanced.

[0087] Third, by integrating multi-source information and structured output, a structured information fusion system is established to receive intermediate results from multiple heterogeneous sources such as text streams, page layouts, visual cues, and entity types; through disambiguation and conflict decision-making mechanisms, a final globally consistent and logically clear structured output is generated.

[0088] When the global arbitrator and assembler integrate the fragmented information generated into a high-quality organic whole based on semantic and spatial relationships, it not only improves the credibility and usability of the output results, but the structured data itself can also serve as feedback to optimize the upstream orchestration and collaboration process, forming a feedback adjustment process.

[0089] Please refer to Figure 5 This embodiment of the application also provides a structural block diagram of a multimodal document processing system, including: The acquisition module 510 is used to acquire multi-source heterogeneous files and convert them into standard documents; The feature extraction module 520 is used to extract features from at least a portion of a standard document and determine a feature profile, which includes document type, layout style and element distribution. The layout analysis module 530 is used to analyze the layout elements of a standard document and determine the block set. The block set includes the geometric coordinates and type labels corresponding to each semantic block of the document. The sequence reorganization module 540 is used to analyze the spatial relationships, visual flow and semantic associations between the semantic blocks of each document based on the geometric coordinates and type labels of each semantic block in the block set, so as to generate a global content sequence that conforms to the reading order. The intelligent parsing module 550 is used to construct an optimal parsing pipeline based on the type of each document semantic block in the global content sequence, parse each document semantic block based on the optimal parsing pipeline, and determine the parsing result; The Structured Data Module 560 is used to generate structured data from the parsed results and output it.

[0090] It should be noted that the multimodal document processing system and multimodal document processing method provided in the above embodiments belong to the same concept. Specifically, the acquisition module 510 executes step S110, the layout analysis module 530 executes step S120, the sequence reorganization module 540 executes step S130, the intelligent parsing module 550 executes step S140, the structured data module 560 executes step S150, and the feature extraction module 520 performs feature extraction. The specific methods of execution of each module have been described in detail in the method embodiments and will not be repeated here. In practical applications, the multimodal document processing method provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0091] The above method acquires multi-source heterogeneous files and converts them into standard documents, eliminating format barriers and forming a unified standard document. Page element analysis is performed on the standard document to determine the page block set. By obtaining the geometric coordinates and type labels of document semantic blocks, the problems of inaccurate positioning and type ambiguity in traditional page analysis are solved, improving the accuracy of document page element recognition. Based on the geometric coordinates and type labels of each document semantic block in the page block set, the spatial relationships, visual flow, and semantic associations between document semantic blocks are analyzed to generate a global content sequence that conforms to the reading order. The reading order is corrected to ensure the sequence... Logically coherent, it solves the semantic disconnect problem caused by relying solely on spatial location; it performs deep recognition of the block set based on the global content sequence, and determines the document recognition result, table recognition result, and formula recognition result separately. By combining deep recognition with global semantic cross-validation, recognition errors are corrected, and the drawbacks of independent recognition of various types of content are solved, ensuring the accuracy of recognition results; information is fused based on document recognition result, table recognition result, and formula recognition result to generate structured data and output it. By information fusion, the scattered recognition results are associated and integrated and converted into structured data, solving the problem of information dispersion and difficulty in reuse.

[0092] In some embodiments, an electronic device is also provided, which may be a server, and its internal structure diagram is shown below. Figure 5 As shown, the electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. The computer program is executed by the processor to implement the functions or steps of the server-side method described above.

[0093] In some embodiments, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the methods described above.

[0094] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0095] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or electronic device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0097] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A multimodal document processing method, characterized in that, include: Obtain multi-source heterogeneous files and convert them into standard documents; The standard document is analyzed for layout elements to determine a set of sections, which includes the geometric coordinates and type labels corresponding to each semantic block of the document. Based on the geometric coordinates and type labels of each document semantic block in the block set, the spatial relationships, visual flow and semantic associations between the document semantic blocks are analyzed to generate a global content sequence that conforms to the reading order; An optimal parsing pipeline is constructed based on the type of each document semantic block in the global content sequence. Each document semantic block is parsed based on the optimal parsing pipeline, and the parsing result is determined. The parsing results are used to generate structured data, which is then output.

2. The multimodal document processing method according to claim 1, characterized in that, Before performing layout analysis on the layout elements in the standard document, the following steps are also included: At least a portion of the standard document is subjected to feature extraction to determine a feature profile, which includes document type, layout style and element distribution; If the document type in the feature profile is a preset type, then the standard document is preprocessed to determine the standard document for document enhancement, wherein the preprocessing includes resolution enhancement, page rotation correction and image quality optimization; If the document type in the feature profile is not a preset type, the standard document is output directly.

3. The multimodal document processing method according to claim 1, characterized in that, The standard document is analyzed for layout elements to determine the set of sections, including: The standard document is analyzed by calling a preset document layout analysis model to determine the geometric coordinates and type labels corresponding to each semantic block of the document. The geometric coordinates corresponding to each document semantic block are associated with the type label to form independent data. The independent data of each document semantic block are packaged to form a block set. The type label includes at least one of the following: title, body paragraph, table, mathematical formula, chart, list, code block and comment.

4. The multimodal document processing method according to claim 1, characterized in that, Based on the geometric coordinates and type labels of each document semantic block in the block set, the spatial relationships, visual flow, and semantic associations between the document semantic blocks are analyzed to generate a global content sequence that conforms to the reading order, including: By using a preset reading order sorting model, the spatial relationships, visual flow and semantic associations between the semantic blocks of each document in the section set are analyzed, and spatial relationship scores, visual flow scores and semantic association scores are generated. The spatial relationship score, visual flow score, and semantic association score of each document semantic block are weighted and calculated according to preset weighting coefficients to determine the weighted score value; The document semantic blocks are sorted according to their weighted scores, and then reorganized from highest to lowest score to generate a global content sequence that conforms to the reading order.

5. The multimodal document processing method according to claim 1, characterized in that, Based on the geometric coordinates and type labels of each document semantic block in the block set, the spatial relationships, visual flow, and semantic associations between the document semantic blocks are analyzed to generate a global content sequence that conforms to the reading order, including: Based on the geometric coordinates of each document semantic block, a spatial relationship matrix between the document semantic blocks is determined; Based on the type tags and the spatial relationship matrix, a semantic association matrix is ​​established between each document semantic block; The visual flow path is determined based on the version structure formed by the spatial relationship matrix of the document semantic blocks and the preset global reading direction. The spatial relationship matrix, the visual flow path, and the semantic association matrix are weighted and fused to determine a global content sequence that conforms to the reading order.

6. The multimodal document processing method according to claim 1, characterized in that, An optimal parsing pipeline is constructed based on the type of each document semantic block in the global content sequence. The optimal parsing pipeline is used to parse each document semantic block, and the parsing result is determined, including: By using a preset optical character recognition model, character recognition is performed on the semantic blocks of the document whose type labels are at least one of title, body paragraph, code block and comment, and the document recognition result is determined. The document semantic block with the type tag of mathematical formula is located by using a preset formula positioning model, and the positioned mathematical formula is parsed by using a preset formula parsing model to determine the formula recognition result. By using a preset table classification model, the semantic blocks of the document labeled as chart, list, and table are classified to determine wired tables and wireless tables; by recognizing the content of the wired tables and the wireless tables, the table recognition result is determined. Identify the type of each document semantic block in the global content sequence to construct an optimal parsing pipeline; The document semantic blocks are parsed according to the optimal parsing pipeline by calling the preset parsing model, and at least one of the document recognition result, the table recognition result and the formula recognition result is used as the parsing result.

7. The multimodal document processing method according to claim 6, characterized in that, The table recognition result is determined by identifying the contents of the wired table and the wireless table, including: By using a preset wired table parsing model, wireframe detection is performed on the wired table to determine the reconstructed cells; By using a preset wireless table parsing model, text alignment and semantic spacing inference are performed on the wireless table to determine the table structure; By integrating the recognition results of the reconstructed cells and the table structure, a table recognition result containing complete row and column logic and two-dimensional structured data is generated.

8. The multimodal document processing method according to any one of claims 1 to 7, characterized in that, After generating structured data from the parsed results, the process also includes: The structured data is optimized based on reading order and semantic analysis. The optimization methods include automatically connecting broken paragraphs, list items or tables across pages and filling in missing logical connectors. The reading order and the semantic analysis are determined by the global content sequence. The optimized structured data is checked for discrepancies in document structure, such as discontinuous title numbering, missing chart references, or omissions of key chapters, and a verification report is generated.

9. A multimodal document processing system, characterized in that, include: The acquisition module is used to acquire multi-source heterogeneous files and convert the multi-source heterogeneous files into standard documents; The layout analysis module is used to perform layout element analysis on the standard document and determine the block set, which includes the geometric coordinates and type labels corresponding to each semantic block of the document. The sequence reorganization module is used to analyze the spatial relationships, visual flow and semantic associations between the document semantic blocks based on the geometric coordinates and type labels of each document semantic block in the block set, so as to generate a global content sequence that conforms to the reading order; The intelligent parsing module is used to construct an optimal parsing pipeline based on the type of each document semantic block in the global content sequence, parse each document semantic block based on the optimal parsing pipeline, and determine the parsing result; The structured data module is used to generate structured data from the parsing results and output it.

10. An electronic device, characterized in that, It includes a processor, a memory, and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the multimodal document processing method as described in any one of claims 1 to 8.

Images