A parsing and retrieval enhancement method for complex modal documents
By using a multimodal document parsing system and a retrieval enhancement engine, the problems of parsing accuracy and retrieval accuracy of complex modal documents in the field of mechanical experiments have been solved, realizing efficient intelligent question answering and cross-modal retrieval, and improving the quality and application efficiency of the knowledge base.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF STRUCTURE & ENVIRONMENT ENG
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-26
AI Technical Summary
Existing document parsing technologies have low accuracy in recognizing complex modal documents in the field of mechanical experiments, making it difficult to understand deep semantics, resulting in low parsing efficiency and poor retrieval results, and failing to achieve efficient cross-modal retrieval.
A multimodal document parsing system and retrieval enhancement engine are adopted. PaddleOCR, MinerU2.5 and LoRA modules are used for text detection and recognition. Combined with context-aware error correction and multimodal information completion, a RAG knowledge base is built for efficient retrieval.
It significantly improves the parsing accuracy and retrieval accuracy of complex modal documents, realizes efficient intelligent question answering and cross-modal retrieval, and improves the construction quality and application efficiency of the knowledge base.
Smart Images
Figure CN122285703A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and information retrieval technology, specifically relating to an enhanced method for parsing and retrieving complex modal documents in the field of mechanical testing. Background Technology
[0002] The field of mechanical testing has accumulated a large number of unstructured documents, including numerous images, formulas, complex tables, and domain terminology. Efficiently and accurately parsing information from these historical documents and performing intelligent retrieval has become a pressing issue. Existing document parsing technologies primarily rely on traditional OCR engines such as Tesseract. While these products are technologically mature, they suffer from high text recognition error rates in blurry, tilted, or complex background scanned documents, struggling to accurately segment and recognize complex tables, images, and formulas, easily leading to content errors. Furthermore, they lack effective error correction and completion mechanisms for technical terms and domain-specific abbreviations. In terms of retrieval, traditional keyword matching technologies cannot understand the deep semantics of user queries, nor can they achieve cross-modal retrieval, resulting in low efficiency in intelligent question answering.
[0003] Currently, there is a lack of a comprehensive technical solution that can integrate high-precision multimodal parsing with deep semantic understanding retrieval, which directly restricts the construction quality and application effectiveness of domain knowledge bases. Therefore, developing a parsing and retrieval enhancement technology for complex modal documents is of vital practical significance for knowledge management and intelligent applications in the field of impulsive dynamics experiments. Summary of the Invention
[0004] To address the aforementioned issues, this invention provides a parsing and retrieval enhancement technology for complex modal documents. This technology addresses the problems of low parsing efficiency and poor accuracy caused by historical documents in the field of mechanics experiments containing a large number of images, formulas, complex tables, and domain terminology, as well as the inability of traditional keyword matching technology to understand the deep semantics of user queries and to achieve cross-modal retrieval. The technology mainly includes an innovative multimodal document parsing system and a retrieval enhancement engine based on the parsing results, enabling efficient extraction and identification of multimodal data and providing efficient intelligent question answering capabilities.
[0005] Working principle: A parsing and retrieval enhancement method for complex modal documents, comprising the following steps: S1: Receive PDF and image documents uploaded by users and convert them into high-resolution image sequences for subsequent processing; S2: Use the PaddleOCR engine to perform preliminary text detection and recognition on the document image, and mark the detected formula areas. S3: Input the text block and its coordinate information output by S2 into the MinerU2.5 model to perform in-depth physical layout and logical structure analysis of the document, especially to identify formula blocks, formula numbers and their logical relationship with the context text; S4: Intelligent enhanced OCRFlux technology fuses and intelligently corrects the outputs of S2 and S3 to improve the accuracy of table data recognition; S5: Based on the high-quality structured data produced by S4, it builds and queries the RAG knowledge base, performs efficient similarity search in the Milvus vector database, finds the text fragments and formula content most relevant to the query semantics, and supports a hybrid search mode, that is, combining vector semantic search, traditional keyword-based search, and formula structure similarity search to ensure recall and precision.
[0006] Furthermore, S2 also includes the following specific content: outputting the original content of all text lines in the document and their coordinate positions in the image; performing special marking and preprocessing on the mechanical formula area; and providing scattered text cell content for complex table areas.
[0007] Furthermore, S3 also includes the following specific content: When parsing the table, MinerU2.5 accurately identifies the table boundaries, the division of rows and columns, and the topological structure of merged cells, and assigns row and column index semantic labels to each cell; at the same time, the model performs physical layout analysis on the mechanical formula area marked by S2, and identifies the spatial structural features of the mechanical formulas; extracts the semantic theme of the table from the table title and header information, and converts the unstructured image table into preliminary structured data; For the mechanics formula region, the system further activates the LoRA enhancement module; based on the pre-trained vision-language model, by injecting and training a low-rank adaptation matrix, the model can quickly learn the special symbol system and complex structural features of mechanics formulas in the modal testing field while maintaining its general capabilities, and achieve accurate recognition of formula symbols and structured conversion of LaTeX format.
[0008] Furthermore, S4 also includes the following specific features: context-aware error correction, multimodal information completion, formula LoRA refined parsing, and knowledge document output to improve the accuracy of tabular data recognition.
[0009] Furthermore, the context-aware error correction is that OCRFlux uses the table structure information obtained in S3 to verify the content of low-confidence cells identified by OCR in S2.
[0010] Furthermore, the multimodal information completion means that for blank or extremely low confidence cells that OCR cannot recognize due to stains or blurriness, OCRFlux will combine the contents of other cells in the same row and column to perform reasonable numerical or text inference and completion. For ambiguous symbols in mechanical formulas, it will combine the domain prior knowledge learned by LoRA to perform probabilistic inference and completion.
[0011] Furthermore, for the formula regions marked in the mechanics formula region processed by the LoRA module in S3, the LoRA enhancement module is activated. It utilizes a domain-adapted low-rank network to perform refined identification and LaTeX structured transformation of complex mechanics formulas, significantly improving the parsing accuracy of multi-line formulas, matrix equations, and special symbols.
[0012] Furthermore, the knowledge document output is to output a high-quality, structurally complete, and content-accurate JSON-formatted structured data from the data enhanced by OCRFlux, wherein the table part is completely restored into a machine-readable data structure.
[0013] Furthermore, step S5 also includes the following specific steps: Knowledge base construction: The structured text output by S4 is segmented into text fragments of appropriate size, and the LaTeX formulas in the text are specially processed; the abstract syntax tree features of the formulas are extracted, the structural fingerprint of the formulas is calculated, and they are converted into high-dimensional vectors with the help of a pre-trained vectorized language model. At the same time, a dedicated LoRA encoder for mechanics formulas is introduced to map the LaTeX code of the formulas to the semantic space and capture the physical semantic relationship between the formulas; at the same time, the multimodal data metadata is stored in the Milvus vector database to form the core RAG knowledge base. Semantic retrieval and hybrid query: When a user submits a query, the system first converts the query statement into a vector; then, it performs an efficient similarity search in the Milvus vector database to find the text fragments most relevant to the query semantics; since the table data has been parsed and quantized, even numerical conditions hidden deep within the table can be accurately retrieved; the system also supports hybrid retrieval modes, that is, combining vector semantic retrieval and traditional keyword-based retrieval to ensure recall and precision. Results generation and presentation: The system takes the top K most relevant text fragments retrieved as context, where K is a positive integer, and feeds them into a large model for information integration and refinement to generate a direct and accurate answer. Finally, it provides users with source citations for the search results, including highlighted original paragraphs and complete original tables, ensuring the verifiability of the results and the accuracy of the generated content.
[0014] The beneficial effects of this invention are as follows: To address the key challenges in building professional knowledge bases and intelligent applications in the field of mechanical testing, this paper presents a parsing and retrieval enhancement technology for complex modal documents. By constructing an innovative multimodal document parsing system and a retrieval enhancement engine based on the parsing results, it achieves accurate and efficient extraction and identification of multi-source heterogeneous information, thereby significantly improving the accuracy and responsiveness of intelligent applications. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the method steps of the present invention.
[0016] Figure 2 An enhanced generation system for multimodal document parsing and retrieval. Detailed Implementation
[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection claimed by the present invention.
[0018] This invention provides a method for parsing and enhancing retrieval of complex modal documents. Its core technology lies in first performing preliminary text detection and recognition on the document image, then deeply analyzing its physical layout and logical structure, and based on this, performing multi-source information fusion and intelligent correction. Finally, by constructing a retrieval enhancement knowledge base, namely a vector knowledge base, it achieves accurate retrieval of semantically related fragments, and leverages a large language model to fuse contextual knowledge to generate enhanced responses. Specific implementation details of this invention are as follows:
[0019] S1: Receive PDF and image documents uploaded by users and convert them into high-resolution image sequences for subsequent processing; S2: Use the PaddleOCR engine to perform preliminary text detection and recognition on the document image, and mark the detected formula areas. Output the original content of all text lines in the document and their coordinate positions in the image, and perform special marking and preprocessing for the mechanical formula area; for complex table areas, this step can only provide scattered text cell content, cannot understand the overall structure of the table, and is prone to recognition errors due to image quality and font reasons (such as misidentifying the number "0" as the letter "O").
[0020] In particular, for mechanical formulas containing complex subscripts and superscripts, Greek letters, and special operators (such as characteristic equations, modal expressions, etc.), conventional OCR can only treat the formula as ordinary text lines and cannot recognize its inherent mathematical structure and physical semantics. Furthermore, it is prone to recognition deviations due to issues such as image quality, font style, or dense symbols, resulting in a significantly higher error rate in formula parsing than in ordinary text.
[0021] S3: During the table parsing process, the text blocks and their coordinate information output by S2 are input into the MinerU2.5 model to perform in-depth physical layout and logical structure analysis of the document, especially identifying formula blocks, formula numbers and their logical relationship with the context text; When parsing tables, MinerU2.5 can accurately identify table boundaries, row and column divisions, and the topological structure of merged cells, and assign row and column index semantic labels to each cell. At the same time, the model performs physical layout analysis on the mechanical formula area marked by S2, identifies the spatial structural features of mechanical formulas (such as fractional upper and lower structure, integral upper and lower limits, matrix row and column layout, etc.), extracts the semantic theme of the table from the table title and header information, and converts the unstructured image table and mechanical formulas into structured data with preliminary semantic information.
[0022] For the mechanics formula region, the system further activates the LoRA (Low-Rank Adaptation) enhancement module: based on a pre-trained vision-language model, by injecting and training a low-rank adaptation matrix, the model can quickly learn the special symbol system and complex structural features of mechanics formulas in the modal testing field while maintaining its general capabilities, and achieve accurate recognition of formula symbols and structured conversion of LaTeX format.
[0023] S4: This step covers four main functions: context-aware error correction, multimodal information completion, refined LoRA parsing of formulas, and knowledge document output. The intelligent enhancement OCRFlux technology fuses and intelligently corrects the outputs of S2 and S3, improving the accuracy of table data recognition. For blank or extremely low-confidence cells that OCR cannot recognize due to stains or blurriness, OCRFlux will combine the content of other cells in its row and column to perform reasonable numerical or text inference and completion.
[0024] The core of improving the accuracy of tabular data recognition includes: context-aware error correction, multimodal information completion, and knowledge document output; among them, context-aware error correction is OCRFLux using the table structure information obtained in S3 to verify the content of low-confidence cells identified by OCR in S2.
[0025] For the formula region processed by the LoRA module in S3, OCRFlux further optimizes post-processing, including formula structure integrity verification (such as bracket matching and superscript / subscript hierarchy verification), physical quantity unit consistency check (such as dimension matching of stiffness matrix K and mass matrix M), and symbol correction based on domain knowledge base. Multimodal information completion refers to the process of making reasonable inferences and completions for recognition gaps or low-confidence areas caused by contamination, blurring, etc., by combining other content in the same row and column; for ambiguous symbols in mechanical formulas, probabilistic inferences and completions are made by combining the domain prior knowledge learned by LoRA (such as common symbol sets in modal analysis).
[0026] For the formula regions marked by the formula region processed by the LoRA module in S3, the LoRA enhancement module is activated. It uses a low-rank network adapted to the domain to perform fine recognition and LaTeX structured transformation of complex mechanics formulas, which significantly improves the parsing accuracy of multi-line formulas, matrix equations and special symbols. For example, when a cell content is recognized by PaddleOCR as "15O" (low confidence), but the column header is "weight (kg)", OCRFlux will correct "15O" to "150" with a high probability based on the semantics of the numerical context. The knowledge document output is to output high-quality, structurally complete, and accurate JSON-formatted structured data from the data enhanced by OCRFlux, in which the table part is completely restored to a machine-readable data structure.
[0027] S5: Based on the high-quality structured data produced by S4, it builds and queries a vector knowledge base, performs efficient similarity search in the Milvus vector database, finds the text fragments most relevant to the query semantics, and supports a hybrid search mode, that is, combining vector semantic search, traditional keyword-based search, and formula structure similarity search to ensure recall and precision. The specific steps include: 1) Knowledge base construction: The structured text output by S4 is segmented into text fragments of appropriate size, and the LaTeX formulas in the text are specially processed; the abstract syntax tree (AST) features of the formula are extracted, the formula structure fingerprint is calculated, and it is converted into a high-dimensional vector with the help of a pre-trained vectorized language model. At the same time, a dedicated LoRA encoder for mechanics formulas is introduced to map the LaTeX code of the formulas to the semantic space and capture the physical semantic relationship between formulas (such as the mathematical equivalence of different forms of eigenvalue problems); at the same time, the multimodal data element information (including the formula AST structure, physical quantity type, and the label of the mechanics branch to which it belongs) is stored in the Milvus vector database to form the core RAG knowledge base; Semantic retrieval and hybrid query: When a user submits a query, the system first converts the query statement into a vector; then, it performs an efficient similarity search in the Milvus vector database to find the text fragments most relevant to the query semantics; since the table data has been parsed and quantized, even numerical conditions hidden deep within the table can be accurately retrieved; the system also supports hybrid retrieval modes, that is, combining vector semantic retrieval and traditional keyword-based retrieval to ensure recall and precision. Results generation and presentation: The system takes the top K most relevant text fragments retrieved as context, where K is a positive integer, and feeds them into a large model for information integration and refinement to generate a direct and accurate answer; finally, it provides users with source citations for the search results, including highlighted original paragraphs and complete original tables, ensuring the verifiability of the results and the accuracy of the generated content. The proposed enhanced parsing and retrieval method for complex modal documents, based on mechanical experiment design management, categorizes the data sources of each chapter and heading level in mechanical experiment documents into five types: input documents, knowledge base, fixed prompts, no specific input, and common library. Different interfaces are invoked based on these five data sources in the template. For input document data sources, after the user uploads the document using RAG (Reference Artificial Intelligence), the retrieved content and user-input prompts are fed into a large model for generation (e.g., experiment overview, experiment purpose, test piece information, etc.). For knowledge base data sources, documents are first retrieved from the database based on tags, potentially locating multiple documents. A list and prompts are passed to the algorithm, which performs RAG retrieval across multiple documents and then directly returns the retrieved chapters to the front end (e.g., experiment methods, measurement schemes, etc.). For fixed input data sources, RAG retrieval is performed first, and then the prompt template is filled in (e.g., "This task comes from..."). For no specific input data sources, the large model is directly invoked, and the large model generates information based on the prompts. For common library data sources, database information is first queried, and then all child elements of the template parent are returned (e.g., experiment site, experiment equipment, etc.).
[0028] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for enhancing the parsing and retrieval of complex modal documents, characterized in that, Includes the following steps: S1: Receive PDF and image documents uploaded by users and convert them into high-resolution image sequences for subsequent processing; S2: Use the PaddleOCR engine to perform preliminary text detection and recognition on the document image, and mark the detected formula areas. S3: Input the text block and its coordinate information output by S2 into the MinerU2.5 model to perform in-depth physical layout and logical structure analysis of the document, especially to identify formula blocks, formula numbers and their logical relationship with the context text; S4: Intelligent enhanced OCRFlux technology fuses and intelligently corrects the outputs of S2 and S3 to improve the accuracy of table data recognition; S5: Based on the high-quality structured data produced by S4, it builds and queries the RAG knowledge base, performs efficient similarity search in the Milvus vector database, finds the text fragments and formula content most relevant to the query semantics, and supports a hybrid search mode, that is, combining vector semantic search, traditional keyword-based search, and formula structure similarity search to ensure recall and precision.
2. The parsing and retrieval enhancement method according to claim 1, characterized in that, The S2 also includes the following specific content: outputting the original content of all text lines in the document and their coordinate positions in the image, performing special marking and preprocessing on the mechanical formula area; and providing scattered text cell content for complex table areas.
3. The parsing and retrieval enhancement method according to claim 1, characterized in that, S3 also includes the following specific content: When parsing the table, MinerU2.5 accurately identifies the table boundaries, the division of rows and columns, and the topological structure of merged cells, and assigns row and column index semantic labels to each cell; at the same time, the model performs physical layout analysis on the mechanical formula area marked by S2, and identifies the spatial structural features of the mechanical formulas; extracts the semantic theme of the table from the table title and header information, and converts the unstructured image table into preliminary structured data; For the mechanics formula region, the system further activates the LoRA enhancement module; based on the pre-trained vision-language model, by injecting and training a low-rank adaptation matrix, the model can quickly learn the special symbol system and complex structural features of mechanics formulas in the modal testing field while maintaining its general capabilities, and achieve accurate recognition of formula symbols and structured conversion of LaTeX format.
4. The parsing and retrieval enhancement method according to claim 1, characterized in that, The S4 also includes the following specific features: text-based error correction, multimodal information completion, formula LoRA fine-grained parsing, and knowledge document output to improve the accuracy of tabular data recognition.
5. The parsing and retrieval enhancement method according to claim 4, characterized in that, The context-aware error correction is that OCRFLlux uses the table structure information obtained in S3 to verify the content of low-confidence cells identified by OCR in S2.
6. The parsing and retrieval enhancement method according to claim 4, characterized in that, The multimodal information completion refers to the process where, for blank or extremely low-confidence cells that OCR cannot recognize due to stains or blurriness, OCRFlux combines the contents of other cells in the same row and column to perform reasonable numerical or text inference and completion. For ambiguous symbols in mechanics formulas, it combines the domain prior knowledge learned by LoRA to perform probabilistic inference and completion.
7. The parsing and retrieval enhancement method according to claim 6, characterized in that, For the formula regions marked in the mechanics formula region processed by the LoRA module in S3, the LoRA enhancement module is activated. It uses a domain-adapted low-rank network to perform refined identification and LaTeX structured transformation of complex mechanics formulas, which significantly improves the parsing accuracy of multi-line formulas, matrix equations and special symbols.
8. The parsing and retrieval enhancement method according to claim 4, characterized in that, The knowledge document output is to output high-quality, structurally complete, and accurate JSON-formatted structured data from the data enhanced by OCRFlux, in which the table part is completely restored into a machine-readable data structure.
9. The parsing and retrieval enhancement method according to claim 1, characterized in that, S5 further includes the following specific steps: The first step is knowledge base construction: The structured text output by S4 is segmented into text fragments of appropriate size, and the LaTeX formulas in the text are specially processed; the abstract syntax tree features of the formulas are extracted, the structural fingerprint of the formulas is calculated, and they are converted into high-dimensional vectors with the help of a pre-trained vectorized language model. At the same time, a dedicated LoRA encoder for mechanics formulas is introduced to map the LaTeX code of the formulas to the semantic space and capture the physical semantic relationships between formulas; at the same time, the multimodal data metadata is stored in the Milvus vector database to form the core RAG knowledge base. The second step is semantic retrieval and hybrid query: When a user submits a query, the system first converts the query statement into a vector; then, it performs an efficient similarity search in the Milvus vector database to find the text fragments most relevant to the query semantics; since the table data has been parsed and quantized, even numerical conditions hidden deep within the table can be accurately retrieved; the system also supports hybrid retrieval modes, that is, combining vector semantic retrieval and traditional keyword-based retrieval to ensure recall and precision. The third step is result generation and presentation: The system takes the top K most relevant text fragments retrieved as context, where K is a positive integer, and feeds them into a large model for information integration and refinement to generate a direct and accurate answer; finally, it provides the user with source citations for the search results, including highlighted original paragraphs and complete original tables, ensuring the verifiability of the results and the accuracy of the generated content.