Multi-modal long document retrieval method based on hierarchical representation and double-branch retrieval

Abstract

The invention belongs to the technical field of document retrieval, and particularly relates to a multi-modal long document retrieval method based on hierarchical representation and double-branch retrieval. According to the method, three-level hierarchical document representation composed of an element level, a page level and a chapter and section level is constructed by designing hierarchical features oriented to a document structure; a double-branch retrieval mechanism is used for conducting semantic-level matching and feature-level matching on queries, and a final evidence set is generated through a fusion strategy. According to the method, the cross-page semantic association and the fine-grained visual clues can be utilized at the same time, effective retrieval and information positioning of the multi-modal long document are achieved, the retrieval requirements of documents with different layout forms and different lengths in a real scene are met, and the method has important practical application value.

Description

A Multimodal Long Document Retrieval Method Based on Hierarchical Representation and Bi-branch Retrieval Technical Field

[0001] This invention belongs to the field of document retrieval technology, specifically relating to a multimodal long document retrieval method based on hierarchical representation and bi-branch retrieval. Background Technology

[0002] While Large Language Models (LLMs) have seen rapid development in natural language processing tasks, they still face challenges in long document understanding and multimodal document question answering. Real-world long documents are typically lengthy, potentially containing hundreds of pages of content, and include multimodal elements such as text, tables, images, formulas, and their headings. In this scenario, it's often necessary to use retrieval to locate key pages within long documents. Existing retrieval methods tend to lose cross-page semantic information when processing long documents, or have shortcomings in understanding multimodal elements, leading to incomplete or inaccurate search results.

[0003] Currently, document retrieval methods mainly focus on two research directions: one is cross-page relationship modeling, which involves constructing page-level graph structures and chapter trees. [1-2] Alternatively, documents can be clustered into page / block-level units. [3] The first aspect is to achieve high-dimensional semantic representation and retrieval; the second is to model multimodal elements by encoding visual components such as images, tables, and formulas through a visual encoder and a layout-aware language model. [4-6] However, current technologies lack a retrieval method that can uniformly model multimodal information at three levels (element level, page level, and chapter / block level) while taking into account cross-page semantic association and fine-grained feature matching. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal long document retrieval method based on hierarchical representation and bi-branch retrieval that has accurate information location and high retrieval efficiency, so as to meet the retrieval application needs of documents of different lengths and layouts in real-world scenarios.

[0005] The multimodal long document retrieval method based on hierarchical representation and bi-branch retrieval provided by this invention constructs a three-level document representation consisting of element-level, page-level, and chapter / block-level features by designing a hierarchical feature modeling framework oriented towards document structure. It then matches queries based on semantic retrieval branches and feature retrieval branches to obtain the final evidence set. The specific steps are as follows:

[0006] (1) Constructing element-level representations: For each page of the input document, aggregate multimodal element information such as text blocks, images, tables, and charts, and utilize a layout-aware visual encoder. [7] Embedded encoding is performed on the page to obtain the element feature representation of the page;

[0007] (2) Constructing page-level and block-level representations: Inputting the overall image of the page into the visual language model [8] This generates a corresponding page-level text description, which serves as the page-level representation; after obtaining the representation of a single page, a page chunker is used. [9] Based on the page-level representation, several semantically consecutive document pages are divided into a semantic block, and finally, a summary text of the corresponding page block is generated as the block-level semantic representation.

[0008] (3) Dual-branch retrieval and result fusion: The query text is encoded using a text encoder and then matched with the semantic representation of each page block and the element-level feature representation of each page for similarity matching. The results of semantic-level matching and feature-level matching are obtained. Finally, the first m1 pages of the semantic branch and the first m2 pages of the feature branch are selected respectively, and the union of the two is used as the final set of evidence pages.

[0009] Furthermore:

[0010] The encoder described in step (1) needs to use a multimodal layout-aware visual encoder specifically designed for document images, and to process the document pages. pass Encode to obtain element-level feature representation .

[0011] In step (2), the page-level representation needs to be achieved using the general visual language model LVLM. Page It is transformed into a page-level text description as a representation, while simultaneously using a specific large model prompt. [9] This allows the text description to include text descriptions of elements within the page (such as paragraphs, images, tables, etc.) and a text summary of the entire page.

[0012] Next, in step (2), the page divider is driven by an LLM. [9] Semantic continuity analysis is performed on the page-level representation, including: firstly, dividing the document into several initial page batches according to a preset page number threshold. Then in each batch Internally, based on page presentation Semantic consistency is determined for consecutive pages to obtain a set of initial semantic blocks within the batch. , where each semantic block It consists of semantically continuous and thematically consistent pages; and further defines the boundary semantic blocks of adjacent batches. and A second joint review is conducted, and the semantic discontinuity at the batch boundary is refined using the following formula:

[0013] ;

[0014] The final refined semantic block set is obtained. ; where each semantic block Represented as , This indicates the range of consecutive pages corresponding to the semantic block. This represents a block-level summary generated by LLM and is used as a block-level representation.

[0015] The dual-branch retrieval in step (3) includes: (1) Block-level semantic matching branch: matching query q with block-level summary Semantic matching is performed to obtain block-level relevance scores. And the rating is inherited by all pages within the block, that is The specific process is as follows:

[0016] 1) Input preparation: Obtain the query and a pre-generated set of refined semantic blocks. Each semantic block contains its corresponding contiguous page range and a block-level summary generated by LLM.

[0017] 2) Semantic matching: Matching queries with block-level summaries in the semantic space by calculating the cosine similarity between text embedding vectors.

[0018] 3) Score inheritance: The calculated block-level relevance score is directly assigned to all pages within the scope of the semantic block. (2) Element-level feature matching branch: The query q is matched with the element-level features of each page. Perform visual-semantic matching and calculate the maximum similarity of the most relevant elements on each page. The specific process is as follows:

[0019] 1) Feature acquisition: Each document page is encoded using a multimodal layout-aware visual encoder (LAE) to obtain element-level feature representations, which include fine-grained features of elements such as paragraphs, images, and tables.

[0020] 2) Visual-semantic matching: Matching and comparing the query with the features of each element on the page.

[0021] 3) Maximum Similarity Aggregation: A max-pooling strategy is employed, selecting the score of the element on the page most relevant to the query as the page's final score. This ensures that as long as there is one key piece of evidence (such as a specific chart or formula) matching on the page, the page will receive a high score.

[0022] (3) Hierarchical fusion: Select the first m1 pages of the semantic branch and the first m2 pages of the element branch respectively, and combine the two to form the final retrieval set.

[0023] Among them, m1 and m2 can be optimized on the validation set to balance global semantic coverage and local visual accuracy under a fixed candidate page budget m.

[0024] This invention proposes a multimodal long document retrieval method based on hierarchical representation and bi-branch retrieval. It constructs a three-level hierarchical document representation consisting of element-level, page-level, and chapter / block-level features oriented towards document structure. A bi-branch retrieval mechanism is then used to perform semantic-level and feature-level matching on the query, and a fusion strategy is employed to generate the final evidence set. This invention can simultaneously utilize cross-page semantic associations and fine-grained visual cues to achieve effective retrieval and information location for multimodal long documents, meeting the retrieval needs of documents with different layouts and lengths in real-world scenarios, and possesses significant practical application value. Attached Figure Description

[0025] Figure 1 is a framework diagram of the present invention.

[0026] Figure 2. VQA performance for different question types on the MMLongBench-Doc dataset.

[0027] Figure 3 shows an analysis of retrieval examples for unanswerable questions. Detailed Implementation

[0028] The present invention will be further described below with reference to the embodiments and accompanying drawings. However, the scope of protection of the present invention is not limited to the embodiments described.

[0029] The specific steps are as follows:

[0030] (1) Element-level representation construction

[0031] For each page of the input document First, the multimodal element information of the page is aggregated, including text blocks, images, tables, charts, etc. Then, a layout-aware encoder (LAE, such as ColPali[4]) is used to perform multimodal embedding encoding on the page to obtain element-level feature representations:

[0032] in, This is a fine-grained feature vector containing multimodal elements within the page, capable of simultaneously reflecting textual semantics, visual appearance, and spatial layout structure. This feature is used for subsequent element-level visual semantic retrieval.

[0033] (2) Page-level representation construction

[0034] Overall image of the page Input a general visual language model (LVLM, such as the Qwen2.5-VL series), and then: Generate page-level text descriptions This includes: the semantic content of each text paragraph within the page (structured textual descriptions of elements such as images, tables, and diagrams); and the semantic summary of the entire page.

[0035] (3) Block-level representation construction (LLM-driven semantic continuity chunking)

[0036] To obtain the page-level representation of each page Then, a page chunking mechanism driven by a large language model is used. Semantic continuity analysis is performed on the page sequence to construct a chapter-level representation. The steps include: first, dividing the document into several initial page batches according to a preset page number threshold (e.g., 20 pages). Then in each batch Internally, based on page presentation Semantic consistency is determined for consecutive pages to obtain a set of initial semantic blocks within the batch. , where each semantic block It consists of semantically continuous and thematically consistent pages; and further defines the boundary semantic blocks of adjacent batches. and A second joint review was conducted, and it was approved.

[0037] ;

[0038] The semantic discontinuities at batch boundaries are refined to obtain a refined set of semantic blocks. ; where each semantic block Represented as , This indicates the range of consecutive pages corresponding to the semantic block. This represents a block-level summary generated by LLM and is used as a block-level representation.

[0039] (4) Dual-branch retrieval and hierarchical fusion

[0040] This invention uses a dual-branch retrieval framework that combines semantic branches and feature branches to simultaneously capture the global semantic relevance of a document and the matching of local visual elements.

[0041] Perform block-level semantic matching branch: match query q with block-level summary Semantic matching is performed to obtain block-level relevance scores. And the rating is inherited by all pages within the block, that is Simultaneously, perform element-level feature matching branch: match query q with the element-level features of each page. Perform visual-semantic matching and calculate the maximum similarity of the most relevant elements on each page. Finally, the pages are merged: the first m1 pages of the semantic branch and the first m2 pages of the element branch are selected respectively, and the union of the two forms the final search set.

[0042] m1 and m2 can be tuned on the validation set to balance global semantic coverage and local visual accuracy under a fixed candidate page budget m.

[0043] The model of this invention is evaluated on four representative multimodal long document benchmarks: MmLongBench-Doc

[10] , LongDoCurl

[11] , Dude

[12] , and SliDevQA

[13] . These public datasets cover different document types and formats, including text, tables, charts, layouts, and images, with answers distributed across single-page, multi-page, and unanswerable cases. Among them, Dude and SliDevQA contain relatively short documents and primarily test the model's ability to understand fine-grained multimodal elements. In contrast, MmLongBench-Doc and LongDoCurl have longer documents, averaging about 40 and 80 pages respectively, and focus on evaluating cross-page retrieval and reasoning. In summary, these benchmarks provide a comprehensive evaluation of element-level multimodal understanding and chunk-level structure modeling. Table 1 shows the retrieval performance of different retrieval methods on four datasets. MMLongBench-Doc and LongDocURL are long document datasets (averaging 40 and 85 pages respectively), while the other two datasets have relatively shorter document pages. This demonstrates that our method achieves the highest recall while maintaining high precision across both long and short documents. Table 2 shows the VQA performance of pages retrieved by various retrieval methods paired with different understanding models, reflecting how the pages retrieved by our method enable understanding models of various sizes and types to answer questions better.

[0044] To demonstrate the performance of the method of this invention under different evidence distribution scenarios, question-answer pairs are divided into three categories based on basic truth evidence: (1) single-page evidence, where all supporting information is contained within one page; (2) cross-page evidence, where the answer requires reasoning across multiple pages; and (3) unanswerable questions, where there is no relevant evidence in the document. Figure 2 shows a performance comparison among ten existing representative methods, with the results arranged in descending order of accuracy for each case. It can be seen that on the MMLongBench-Doc dataset, the method of this invention can improve the accuracy of answering single-page and cross-page questions, but may be accompanied by a decrease in the accuracy of unanswerable questions.

[0045] Figure 3 illustrates an analysis of unanswerable question-answering examples. This case, an unanswerable question in MmLongBench-Doc, is due to fine-grained semantic confusion, illustrating that the high retrieval performance but low question-answering performance of the method in this invention is caused by the illusion inherent in the subsequent understanding model. The method in this invention, Hi3Doc, can retrieve more relevant pages, while the Colpali method retrieves irrelevant pages.

[0046] Table 1, Retrieval Performance Table

[0047] .

[0048] Table 2, VQA Performance Table

[0049] .

[0050] References

[0051] [1] Wu,

[0052] [2] Sourati, Z., Wang, Z., Liu, MM, Hu, Y., Guo, M., Bharadwaj,S., ... & Roth, D. (2025). LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding. arXiv preprint arXiv:2510.07233.

[0053] [3] Gong, Z., Huang, Y., & Mai, C. (2025). Mmrag-docqa: A multi-modalretrieval-augmented generation method for document question-answering withhierarchical index and multi-granularity retrieval. arXiv e-prints, arXiv-2508.

[0054] [4] Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot,C., & Colombo, P. (2025, January). ColPali: Efficient document retrieval withvision-language models. In Proceedings of the International Conference onLearning Representations (ICLR).

[0055] [5] Yu, S., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y., … & Sun, M.(2025). VisRAG: Vision-based retrieval-augmented generation on multi-modalitydocuments. In Proceedings of the Thirteenth International Conference onLearning Representations (ICLR).

[0056] [6] Tanaka, R., Iki, T., Hasegawa, T., Nishida, K., Saito, K., &Suzuki, J. (2025). Vdocrag: Retrieval-augmented generation over visually-richdocuments. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition (CVPR) (pp. 24827–24837).

[0057] [7] Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and CongYao. Layoutllm: Layout instruction tuning with large language models fordocument understanding. In Proceedings of the IEEE / CVF conference on computervision and pattern recognition, pages 15630–15640, 2024. 1, 2.

[0058] [8] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, MicheleBevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How languagemodels use long contexts. Transactions of the Association for ComputationalLinguistics, 12: 526 157–173, 2024. 1, 2.

[0059] [9] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, SiboSong, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vltechnical report. arXiv preprint arXiv:2502.13923, 2025. 1, 5.

[0060]

[10] Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. 2024. MMLONGBENCH-DOC: benchmarking long-context document understanding with visualizations. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NIPS '24), Vol. 37. Article 3041, 95963–96010.

[0061]

[11] Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, JianXu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, et al. Longdocurl: acomprehensive multimodal long document benchmark integrating understanding,reasoning, and locating. In Proceedings of the 63rd Annual Meeting of theAssociation for Computational Linguistics (Volume 1: Long Papers), pages1135–1159, 2025. 5.

[0062]

[12] Jordy Van Landeghem; Rafa Powalski; Rubèn Tito; DawidJurkiewicz; Matthew Blaschko; Lukasz Borchmann, "Document UnderstandingDataset and Evaluation (DUDE)," 2023 IEEE / CVF International Conference onComputer Vision (ICCV), Paris, France, 2023, pp. 19471-19483, doi: 10.1109 / ICCV51070.2023.01789.

[0063]

[13] Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa,Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visualquestion answering on multiple images. In Proceedings of the AAAI Conferenceon Artificial Intelligence, pages 13636–13645, 2023. 5。

Claims

1. A multimodal long document retrieval method based on hierarchical representation and bi-branch retrieval, characterized in that, By designing a hierarchical feature modeling framework oriented towards document structure, a three-level document representation consisting of element level, page level and chapter block level is constructed, and the query is matched based on semantic retrieval branch and feature retrieval branch to obtain the final evidence set; the specific steps are as follows: (1) Construct element level representation: For each page of the input document, aggregate multimodal element information of text blocks, images, tables and charts, and use layout-aware visual encoder to embed and encode the page to obtain the element feature representation of the page; (2) Constructing page-level and block-level representations: Input the overall image of the page into the visual language model to generate the corresponding page-level text description as the page-level representation; After obtaining the representation of a single page, the page blocker is used to divide several semantically continuous document pages into a semantic block according to the page-level representation. Finally, the summary text of the corresponding page block is generated as the block-level semantic representation. (3) Dual-branch retrieval and result fusion: The query text is encoded using a text encoder and then matched with the semantic representation of each page block and the element-level feature representation of each page for similarity matching to obtain the semantic-level matching and feature-level matching results. Finally, the first m1 pages of the semantic branch and the first m2 pages of the feature branch are selected respectively, and the union of the two is used as the final evidence page set.

2. The multimodal long document retrieval method according to claim 1, characterized in that, The page element features mentioned in step (1) are represented as follows: Specifically ， For document pages, Operations for layout-aware visual encoders.

3. The multimodal long document retrieval method according to claim 2, characterized in that, In step (2), the visual language model used for the page-level representation is LVLM, which represents the page. Converted to a page-level text description, it is represented as: ； This represents the operation of the Visual Language Model (LVLM); then, through a specific large model prompt, the text description includes text descriptions of elements within the page and a text summary of the entire page.

4. The multimodal long document retrieval method according to claim 3, characterized in that, In step (2), the page partitioning is further performed using an LLM-driven page partitioner. Semantic continuity analysis is performed on the page-level representation, including: firstly, dividing the document into several initial page batches according to a preset page number threshold. Then in each batch Internally, based on page presentation Semantic consistency is determined for consecutive pages to obtain a set of initial semantic blocks within the batch. , where each semantic block It consists of semantically continuous and thematically consistent pages; and further defines the boundary semantic blocks of adjacent batches. and A second joint review is conducted, and the semantic discontinuity at the batch boundary is refined using the following formula: The final result is a refined set of semantic blocks. ; where each semantic block Represented as ， This indicates the range of consecutive pages corresponding to the semantic block. This represents a block-level summary generated by LLM and is used as a block-level representation.

5. The multimodal long document retrieval method according to claim 4, characterized in that, The dual-branch retrieval described in step (3) includes: (1) Block-level semantic matching branch: matching query q with block-level summary Semantic matching is performed to obtain block-level relevance scores. And the rating is inherited by all pages within the block, that is (2) Element-level feature matching branch: Match query q with the element-level features of each page Perform visual-semantic matching and calculate the maximum similarity of the most relevant elements on each page. (3) Hierarchical fusion: Select the first m1 pages of the semantic branch and the first m2 pages of the element branch respectively, and combine the two to form the final retrieval set.

6. The multimodal long document retrieval method according to claim 5, characterized in that, The operation process of the block-level semantic matching branch in the dual-branch retrieval in step (3) is as follows: 1) Input preparation: obtain the query and the pre-generated refined semantic block set; each semantic block contains its corresponding continuous page range and the block-level summary generated by LLM; 2) Semantic matching: match the query and the block-level summary in the semantic space by calculating the cosine similarity between the text embedding vectors; 3) Score inheritance: directly assign the calculated block-level relevance score to all pages within the range to which the semantic block belongs.

7. The multimodal long document retrieval method according to claim 5, characterized in that, The operation process of the element-level feature matching branch in the dual-branch retrieval in step (3) is as follows: 1) Feature acquisition: Encode each document page using a multimodal layout-aware visual encoder (LAE) to obtain element-level feature representations, which include fine-grained features of elements such as paragraphs, images, and tables; 2) Visual-semantic matching: Match and compare the query with the features of each element in the page; 3) Maximum similarity aggregation: Use the maximum pooling strategy to select the score of the element in the page that is most relevant to the query as the final score of the page.

8. The multimodal long document retrieval method according to claim 5, characterized in that, In the hierarchical fusion step (3), m1 and m2 are tuned on the validation set to balance global semantic coverage and local visual accuracy under a fixed candidate page budget m.

Images