Long document intelligent retrieval method and system based on hierarchical analysis and multi-modal fusion

Abstract

The invention discloses a long document intelligent retrieval method and system based on hierarchical analysis and multi-modal fusion, and the method comprises the following steps: carrying out hierarchical analysis on an input long document, and obtaining a physical layout structure, a logic structure and a division and association relationship of multi-modal elements of the long document; respectively extracting text features of the text elements and visual features of the multi-modal elements; performing deep fusion on the text features and the visual features through a cross-modal fusion model to generate a unified multi-modal semantic representation vector; constructing a hybrid index; receiving user query, and performing preliminary retrieval by using the hybrid index to obtain a candidate result set; reordering the candidate result set; and outputting the retrieval result after reordering. Through visual-logic collaborative analysis, gating enhanced bilinear attention fusion and graph attention-based context reordering, a collaborative gain effect is generated in a long document intelligent retrieval task, and the retrieval precision and the user experience are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, information retrieval, and multimodal artificial intelligence, specifically to a method and system for intelligent retrieval of long documents based on hierarchical parsing and multimodal fusion. It is applicable to scenarios such as enterprise knowledge management and intelligent question answering, academic literature and patent retrieval, legal document analysis and case retrieval, and medical and health record processing. Background Technology

[0002] In the fields of natural language processing and information retrieval, intelligent retrieval of long documents (such as academic papers, technical reports, and legal documents) has always been a key and challenging issue. These documents typically contain lengthy text, complex layouts, and information in multiple modalities, including images, tables, and formulas. Existing retrieval technologies can be broadly categorized into two types: one is dense retrieval models based on deep learning. These models, such as Dense Paragraph Retrieval (DPR) and its variants, map queries and document fragments into high-dimensional vectors using pre-trained language models (such as BERT) and match them based on vector similarity. The construction of such methods typically includes a text editor, vector indexing, and similarity calculation modules. Their working principle leverages the semantic understanding capabilities of neural networks to achieve semantic retrieval beyond keywords. However, these models are essentially designed for pure text and lack modules for processing multimodal information, thus failing to understand and retrieve images, tables, and other content within the document. The other type is visual-language cross-modal models. To handle multimodal content, visual-language models such as ViLBERT, CLIP, LayoutLM, and ViLT have emerged. These models typically consist of a text encoder, an image encoder, and a fusion module for cross-attention or feature concatenation. Their working principle involves the model learning the alignment representations of text and images in a shared space, thereby supporting tasks such as text-image retrieval.

[0003] However, when the aforementioned, especially the second type of existing technology, is applied to the specific scenario of long documents, inherent defects in its construction and working principle lead to the following problems: The structural understanding is superficial. Existing cross-membrane models (such as ViLT) are constructed to typically treat an entire document page or a large sequence of text as a flat input. While they can perceive coarse layout information (such as 2D information), they lack dedicated modules for collaboratively resolving and explicitly modeling the document's inherent, hierarchical physical layout (such as columns and heading positions) and logical structure (such as chapter, paragraph, and chart reference relationships). The direct consequence is that the model cannot understand… Figure 1 The semantic structure of "located in Section 3, Subsection 2" makes it difficult to achieve precise intra-section retrieval or understand the attribution relationship between charts and specific text paragraphs, resulting in coarse-grained search results and limited accuracy. The multi-modal fusion mechanism is coarse-grained. The existing models use fusion modules such as simple feature splicing or basic cross-attention mechanisms, which are not capable of dealing with complex and variable text-image relationships in long documents. Since the fusion module lacks an adaptive mechanism that can dynamically adjust the contribution of text and visual features according to specific context content, it cannot optimally reflect the semantic association between modalities when faced with scenes where multiple charts and long text contexts are intertwined, resulting in insufficiently fine-grained representations after fusion, which can lead to misassociation or information flooding. The retrieval results are disconnected from the context. The ranking module of the existing retrieval system mostly relies on the direct semantic similarity between the query and the candidate segments (or images) for ranking. Since the local context information determined by the document structure is not introduced and utilized in the construction, there is an inherent limitation in the working principle of the system. The disadvantage is that the retrieval results may be semantically related but contextually inconsistent, for example, returning a chart discussing the "experimental setup" to a query asking about "experimental results", which seriously affects the practicality and user experience of the retrieval results.

[0004] These specific technical bottlenecks severely restrict the accuracy and user experience of long document intelligent retrieval. Therefore, there is an urgent need in the field for a new method that can deeply integrate document structure information, achieve fine-grained adaptive multi-modal fusion, and utilize context information to optimize retrieval results. SUMMARY

[0005] The purpose of the present application is to provide a long document intelligent retrieval method and system based on hierarchical parsing and multi-modal fusion to address the above technical problems.

[0006] The technical solution of the present application is as follows: A long document intelligent retrieval method based on hierarchical parsing and multi-modal fusion, comprising the following steps: Performing hierarchical parsing on the input long document to obtain its physical layout structure, logical structure, and division and association relationship of multi-modal elements; Based on the hierarchical parsing results, extracting text features of text elements and visual features of multi-modal elements, respectively; Through a cross-modal fusion model, the text features and visual features are deeply fused to generate a unified multi-modal semantic representation vector; Based on the results of hierarchical parsing and the multi-modal semantic representation vector, a hybrid index is constructed; the hybrid index at least includes a structured metadata index, a vector index, and a keyword inverted index; Receiving a user query, performing preliminary retrieval using the hybrid index to obtain a candidate result set; Based on the hierarchical context information of the long document, re-ranking the candidate result set; Outputting the re-ranked retrieval results.

[0007] Furthermore, the cross-modal fusion model is a gated enhanced bilinear attention network; the fusion process is as follows: Calculate text feature vectors With visual feature vectors Bilinear interaction matrix between : , in, For learnable parameter tensors; Generate dynamic gating vectors : ,in, This indicates a splicing operation. For the sigmoid function, and These are learnable parameters; Calculate the final fusion features : ,in This is element-wise multiplication.

[0008] Using the above method, the gated enhanced bilinear attention fusion network (GEBAN) captures fine-grained cross-modal associations through bilinear interactions and introduces dynamic gating vectors to adaptively balance modal contributions, thus solving the robust fusion problem of complex text-image relationships in long documents.

[0009] Furthermore, the reordering of the candidate result set specifically includes: For each candidate result, construct its local context graph based on the logical structure tree. The nodes in the graph include the candidate result and its parent node, child node, sibling node, and multimodal element nodes associated through anchor points. A graph attention network is used to calculate the correlation between the user query vector and each node in the local context graph; The relevance scores of the candidate result itself and its neighboring nodes are aggregated to obtain the final context relevance score of the candidate result, and the candidate result set is reordered based on this score.

[0010] Using the above method, the context-aware re-ranking algorithm based on graph attention network (GAT) constructs a local context graph for the retrieval results, and uses GAT to quantify the correlation between the query and the context of the results, which significantly improves the relevance and accuracy of the results.

[0011] Furthermore, the hierarchical parsing of the input long document specifically includes: Using an instance segmentation model pre-trained for document layout analysis tasks, the physical structure of document pages is analyzed to locate and classify text blocks, images, tables, and formula areas. A graph neural network for document logical structure is constructed, with the regions output by physical structure parsing as nodes. The features of the nodes are integrated with visual features, textual semantic features, and layout features. Logical relationships between nodes are predicted through graph convolution operations to generate the document's logical structure tree.

[0012] Furthermore, the instance segmentation model is the Mask R-CNN model, and in its region proposal network and post-processing non-maximum suppression step, constraint rules based on prior knowledge of document layout are introduced to preferentially merge similar regions that are adjacent in the horizontal or vertical direction.

[0013] Using the above methods, the hierarchical document parsing mechanism with "visual-logical" dual-stage collaboration employs an improved MaskR-CNN for high-precision layout analysis and combines it with logical relationship inference based on graph neural networks (GNN) to achieve automated and accurate collaborative parsing of the physical layout and logical structure of long documents.

[0014] Furthermore, the constructed hybrid index, whose structured metadata index, vector index, and keyword inverted index are linked by a unique identifier, supports the execution of joint queries that simultaneously involve structural scope, semantic similarity, and keyword matching.

[0015] Using the methods described above, the multi-dimensional associated hybrid index architecture organically integrates structured metadata, dense vectors, and keyword indexes through unique identifier mapping, supporting efficient and complex multi-condition joint queries.

[0016] This application also includes a long document intelligent retrieval system based on hierarchical parsing and multimodal fusion, implementing a long document intelligent retrieval method based on hierarchical parsing and multimodal fusion, including: The hierarchical document parsing module performs hierarchical parsing on the input long document to obtain its physical layout structure, logical structure, and the division and association of multimodal elements; The multimodal feature extraction and fusion module extracts text features of text elements and visual features of multimodal elements based on hierarchical parsing results; through a cross-modal fusion model, it deeply fuses text features and visual features to generate a unified multimodal semantic representation vector. The hybrid index building module constructs a hybrid index based on the results of hierarchical parsing and multimodal semantic representation vectors; the hybrid index includes at least a structured metadata index, a vector index, and a keyword inverted index; The retrieval and reordering engine receives user queries, performs preliminary retrieval using a hybrid index, and obtains a candidate result set; based on the hierarchical context information of long documents, it reorders the candidate result set.

[0017] Furthermore, the hierarchical document parsing module includes: A physical structure parser is configured to perform physical layout segmentation of a document using a modified Mask R-CNN model; A logical structure parser configured to use a graph neural network to infer the logical hierarchy of document elements.

[0018] This application also includes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a long document intelligent retrieval method based on hierarchical parsing and multimodal fusion.

[0019] This application also includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a long document intelligent retrieval method based on hierarchical parsing and multimodal fusion.

[0020] Compared with existing technologies, the advantages of this invention are: By employing "visual-logical" collaborative parsing, gated enhanced bilinear attention fusion, and graph attention-based context reordering, a synergistic gain effect was achieved in long document intelligent retrieval tasks, significantly improving retrieval accuracy and user experience. Attached Figure Description

[0021] Figure 1 This is the architecture diagram of the system in this application. Detailed Implementation

[0022] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0024] Please see Figure 1 A method for intelligent retrieval of long documents based on hierarchical parsing and multimodal fusion includes: A hierarchical document parsing method based on a two-stage collaboration of "visual-logical" approach: In the physical structure analysis stage, a Mask R-CNN model pre-trained for document layout analysis is employed. This model, tailored to the characteristics of text blocks, charts, tables, formulas, and other regions in document images, incorporates constraint rules based on prior knowledge of document layout (such as preferentially merging similar regions that are horizontally or vertically adjacent) in the Region Proposal Network (RPN) and post-processing Non-Maximum Suppression (NMS) steps. This achieves high-precision and robust segmentation of complex layouts. This differs from the application of general object detection models and represents a specialized improvement for document structure understanding.

[0025] Logical structure parsing stage: Constructing a document logical structure graph neural network. This network uses regions from the physically parsed output as nodes, and node features integrate visual features (position, size), textual semantic features (encoded via Sentence-BERT), and typographic features (font, indentation). Through graph convolution operations, the network explicitly predicts parent-child relationships (e.g., chapter-subsection) and sibling relationships (e.g., consecutive paragraphs) between nodes, ultimately generating an accurate document logical structure tree. This method overcomes the limitation of traditional rule-based or heuristic algorithms, which have poor generalization ability on complex documents.

[0026] Gated and enhanced bilinear attention fusion networks for multimodal feature interaction: First, calculate the text feature vector. With visual feature vectors Bilinear interaction matrix between To capture fine-grained cross-modal correlations: ,in It is a learnable parameter tensor.

[0027] The key innovation lies in introducing a dynamic gating vector g. This gating vector is generated jointly from textual and visual features: ,in Indicates splicing, This is the sigmoid function. This gating vector adaptively modulates the contribution of fused information from bilinear interactions to the original textual information.

[0028] Final fusion features Represented as: ( (For element-wise multiplication). This gated bilinear design allows the model to dynamically select a fusion strategy based on the specific content, exhibiting stronger robustness and accuracy than a fixed fusion method, especially when dealing with scenarios where the relationship between descriptive text and charts in a document is uncertain.

[0029] Context-aware reordering algorithm based on graph attention network: After obtaining a set of candidate fragments through initial retrieval, the algorithm reorders them to improve the relevance of the results.

[0030] The algorithm constructs a local context graph for each candidate fragment. The nodes in the graph include the candidate fragment and its parent, child, and sibling nodes in the document's logical structure tree, as well as the anchored graph nodes.

[0031] A graph attention network is used to calculate the relevance of the query vector to all nodes in the local context graph. Finally, the context relevance score of the candidate fragment is a weighted aggregation of its relevance to all its neighboring nodes.

[0032] This algorithm quantifies the structured contextual information of documents and incorporates it into the sorting process, effectively solving the classic problem of "semantically related but irrelevant content," which is an essential improvement over sorting based solely on content similarity.

[0033] A hybrid index architecture that supports multi-dimensional join queries: This architecture is not a simple coexistence of multiple indexes, but rather an organic connection between the structured metadata index (stored in Elasticsearch), the dense vector index (stored in FAISS), and the keyword inverted index through a unique identifier mapping mechanism.

[0034] This enables the system to efficiently execute complex joint queries, such as: "Within the scope of Chapter 3, find paragraphs or charts that are semantically similar to 'neural network optimization' and contain the keyword 'gradient descent'."

[0035] like Figure 1 As shown, this invention proposes an intelligent long-document retrieval system based on hierarchical parsing and multimodal fusion. Its architecture mainly consists of three layers: an application layer, a core processing layer, and a storage layer. Each layer works collaboratively to complete the entire process from document parsing to intelligent retrieval.

[0036] The application layer is the data interaction interface between the system and the outside world, and is responsible for receiving two types of input: Long document upload: Supports users to upload long documents in various formats such as PDF and DOCX, which contain multimodal content such as text, images, and tables to be processed.

[0037] User query input: Receives search queries submitted by users through the interface. The query can be a natural language question or a combination of keywords.

[0038] The core processing layer, the core of the system's technological innovation, includes the following modules: The hierarchical document parsing module is responsible for deep parsing of the uploaded original documents. First, it uses a physical structure parser employing a Mask R-CNN model pre-trained for document layout analysis tasks to analyze page layout, columns, and region divisions, accurately locating and classifying elements such as text blocks, images, tables, and formulas. Then, a logical structure parser uses a graph neural network (GNN) to identify logical units such as chapters, paragraphs, and lists by analyzing visual, semantic, and typographical features.

[0039] The multimodal feature extraction and fusion module receives the parsed elements and uses a text feature extractor (based on pre-trained models such as BERT) and a visual feature extractor (based on pre-trained models such as CNN / ViT) in parallel to obtain high-dimensional semantic features. Subsequently, the cross-modal attention fusion unit adopts a gated enhanced bilinear attention network (GEBAN), which achieves fine-grained and adaptive fusion of text and visual features through bilinear interaction and dynamic gating mechanisms to generate a unified multimodal representation vector.

[0040] Query processing interface: Receives and performs initial parsing of user queries.

[0041] The retrieval and re-ranking engine is the core of online retrieval. Its internal hybrid retrieval unit concurrently sends requests to various indexes in the storage layer; the context-aware re-ranking unit employs a graph attention network (GAT)-based algorithm to construct a local context graph for candidate results, performing fine-grained ranking based on their structural relevance to the query, semantic similarity, and cross-modal association. Storage layer: Hybrid Index Library: This is the system's "memory" center, receiving and storing all output from the offline processing pipeline. It is a composite database containing: Structured metadatabase: Stores the physical and logical structure information of documents (such as chapter structure, chart index and location information) in a non-relational format.

[0042] Vector Index Library: Efficiently stores and manages massive high-dimensional vectors generated by text and multimodal fusion, and supports fast approximate nearest neighbor search.

[0043] Keyword inverted index: Establishes a mapping from keywords to document fragments, supporting efficient traditional keyword Boolean retrieval and highlighting.

[0044] This application, through the implementation of the above technical solution, has achieved significant improvements in the core performance indicators of intelligent long document retrieval, and rigorous ablation experiments have verified the necessity and effectiveness of each innovative module. Specific technical effects are reflected in the following aspects: 1. Overall retrieval accuracy reaches industry-leading levels. Experimental results on the LongDocMM (LongDoc Multimodal) benchmark set demonstrate that the complete solution of this invention significantly outperforms existing mainstream technologies in key evaluation metrics. Specifically, in the core metric of mean average precision (mAP@5), the solution achieves 0.641, representing an absolute improvement of 0.114 and a relative improvement of 21.6% compared to the best baseline model (ViLT, 0.527). In the NDCG@10 metric, which measures ranking quality, the solution achieves 0.708, also showing a significant advantage over the baseline model's 0.591.

[0045] 2. Each core innovation module has been proven to have made substantial contributions. Through systematic ablation experiments, the specific contribution of each innovative module to the final performance was quantitatively verified: The hierarchical document parsing module is the foundation of the system. Removing this module resulted in a 9.0% decrease in the mAP@5 metric, demonstrating the importance of a deep understanding of the physical and logical structure of documents for improving retrieval accuracy.

[0046] The gated-enhanced bilinear attention fusion module is key to improving multimodal understanding capabilities. Removing this module resulted in a 5.6% decrease in mAP@5, confirming its effectiveness in fine-grained cross-modal semantic fusion.

[0047] The context-aware reordering module is crucial for optimizing the final results. Removing this module resulted in a 3.6% decrease in the mAP@5 metric, demonstrating the value of leveraging document context information to improve the relevance of the results.

[0048] 3. The system demonstrates superior performance in complex query scenarios. When dealing with complex queries that require combining semantic understanding, structural filtering, and keyword matching, the hybrid indexing architecture and multi-module collaborative mechanism of this invention demonstrate unique advantages. For example, in multi-condition queries such as "retrieving charts in Chapter 3 related to neural network optimization and containing the keyword gradient descent," the accuracy of the solution proposed in this invention is about 30% higher than that of traditional single-retrieval methods.

[0049] 4. The system response efficiency meets the requirements of practical applications. Through optimized hybrid index design and parallel processing, the system maintains high response speed while ensuring retrieval quality. For a test database with tens of millions of documents, the average query response time is controlled within 200 milliseconds, fully meeting the real-time requirements of practical applications.

[0050] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. A method for intelligent retrieval of long documents based on hierarchical parsing and multimodal fusion, characterized in that, Includes the following steps: The input long document is parsed hierarchically to obtain its physical layout structure, logical structure, and the division and relationship of multimodal elements; Based on the hierarchical parsing results, text features of text elements and visual features of multimodal elements are extracted respectively; By using a cross-modal fusion model, textual and visual features are deeply fused to generate a unified multimodal semantic representation vector. Based on the results of hierarchical parsing and multimodal semantic representation vectors, a hybrid index is constructed; the hybrid index includes at least a structured metadata index, a vector index, and a keyword inverted index. Receive user queries, perform preliminary retrieval using a hybrid index, and obtain a candidate result set; The candidate result set is reordered based on the hierarchical context information of long documents; Output the reordered search results.

2. The intelligent long document retrieval method based on hierarchical parsing and multimodal fusion according to claim 1, characterized in that, The cross-modal fusion model is a gated enhanced bilinear attention network; the fusion process is as follows: Calculate text feature vectors With visual feature vectors Bilinear interaction matrix between : ， in, For learnable parameter tensors; Generate dynamic gating vectors : ,in, This indicates a splicing operation. For the sigmoid function, and These are learnable parameters; Calculate the final fusion features : ,in This is element-wise multiplication.

3. The intelligent long document retrieval method based on hierarchical parsing and multimodal fusion according to claim 1, characterized in that, The reordering of the candidate result set specifically includes: For each candidate result, construct its local context graph based on the logical structure tree. The nodes in the graph include the candidate result and its parent node, child node, sibling node, and multimodal element nodes associated through anchor points. A graph attention network is used to calculate the correlation between the user query vector and each node in the local context graph; The relevance scores of the candidate result itself and its neighboring nodes are aggregated to obtain the final context relevance score of the candidate result, and the candidate result set is reordered based on this score.

4. The intelligent long document retrieval method based on hierarchical parsing and multimodal fusion according to claim 1, characterized in that, The hierarchical parsing of the input long document specifically includes: Using an instance segmentation model pre-trained for document layout analysis tasks, the physical structure of document pages is analyzed to locate and classify text blocks, images, tables, and formula areas. A graph neural network for document logical structure is constructed, with the regions output by physical structure parsing as nodes. The features of the nodes are integrated with visual features, textual semantic features, and layout features. Logical relationships between nodes are predicted through graph convolution operations to generate the document's logical structure tree.

5. The intelligent long document retrieval method based on hierarchical parsing and multimodal fusion according to claim 4, characterized in that, The instance segmentation model is the Mask R-CNN model, and in its region proposal network and post-processing non-maximum suppression step, constraint rules based on prior knowledge of document layout are introduced to preferentially merge similar regions that are adjacent in the horizontal or vertical direction.

6. The intelligent long document retrieval method based on hierarchical parsing and multimodal fusion according to claim 1, characterized in that, The constructed hybrid index, whose structured metadata index, vector index, and keyword inverted index are linked by a unique identifier, supports the execution of joint queries that simultaneously involve structural scope, semantic similarity, and keyword matching.

7. A long document intelligent retrieval system based on hierarchical parsing and multimodal fusion, characterized in that, To implement the intelligent long document retrieval method based on hierarchical parsing and multimodal fusion as described in any one of claims 1-6, the method includes: The hierarchical document parsing module performs hierarchical parsing on the input long document to obtain its physical layout structure, logical structure, and the division and association of multimodal elements; The multimodal feature extraction and fusion module extracts text features of text elements and visual features of multimodal elements based on hierarchical parsing results; through a cross-modal fusion model, it deeply fuses text features and visual features to generate a unified multimodal semantic representation vector. The hybrid index building module constructs a hybrid index based on the results of hierarchical parsing and multimodal semantic representation vectors; the hybrid index includes at least a structured metadata index, a vector index, and a keyword inverted index; The retrieval and reordering engine receives user queries, performs preliminary retrieval using a hybrid index, and obtains a candidate result set; based on the hierarchical context information of long documents, it reorders the candidate result set.

8. The intelligent long document retrieval system based on hierarchical parsing and multimodal fusion according to claim 7, characterized in that, The hierarchical document parsing module includes: A physical structure parser is configured to perform physical layout segmentation of a document using a modified Mask R-CNN model; A logical structure parser configured to use a graph neural network to infer the logical hierarchy of document elements.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the long document intelligent retrieval method based on hierarchical parsing and multimodal fusion as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the long document intelligent retrieval method based on hierarchical parsing and multimodal fusion as described in any one of claims 1 to 6.

Images