A lightweight and efficient multi-modal document retrieval enhancement method

Abstract

The application discloses a kind of light weight high-efficiency multimodal document retrieval enhancement method, by utilizing multimodal large language model, training sparse encoder, multi-vector encoder and multimodal document rearranger, combine sparse retrieval with multi-vector fine-grained delay interaction strategy;Through the design of hybrid retrieval architecture combining sparse index and multi-vector representation, the memory bottleneck and response delay problem faced by traditional multimodal retrieval method in mass data scene are effectively alleviated;The time cost perception loading strategy introduced and the clustering storage mechanism cooperatively optimize the delay interaction score calculation efficiency;Finally, the sparse and delay interaction score are fused, and the multimodal language model reorderer is combined, the document matching precision is significantly improved, and the response efficiency and user experience of multimodal retrieval enhancement system are comprehensively improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of multi-modal information retrieval, and particularly relates to a lightweight and efficient multi-modal document retrieval enhancement method. BACKGROUND

[0002] With the rapid development of big data and artificial intelligence technology, multi-modal documents (such as documents in formats such as PDF, Word, PPT, etc., which contain various modal information such as text, images, charts, etc.) are widely used in industries such as law, medicine, finance, and insurance, and have become an indispensable information resource in data lakes and data warehouses. Using natural language queries to accurately locate relevant documents from a large number of multi-modal documents is an important basis for implementing the retrieval-augmented generation (RAG) task. Multi-modal document retrieval enhancement methods (such as the Chinese patent application technology with publication number CN119988588A) usually first accurately retrieve multi-modal documents related to user queries, and then input the contents of these documents and user queries into a large language model to generate answers, thereby effectively improving the accuracy and reliability of the generated results.

[0003] Currently, multi-modal document retrieval enhancement technology usually relies on optical character recognition (OCR) technology and traditional text retrieval technology; specifically, these methods first convert visual content such as images and charts in the document into text through OCR technology (such as the literature [Wang B, Xu C, Zhao X, et al. Mineru: An open-source solution for precise document content extraction [J]. arXiv preprint: 2409.18839, 2024]), and then use traditional text retrieval methods (such as BM25) or dense vector retrieval methods for subsequent retrieval processing.

[0004] In the process of implementing the present application, we found that at least the following problems exist in the prior art:

[0005] On the one hand, OCR technology itself has certain recognition errors, especially when dealing with complex charts, images, and other content; on the other hand, visual and spatial structure information inevitably suffers a lot of loss during the OCR conversion process, making it difficult to fully capture the complete semantics of the document during retrieval, which significantly reduces the accuracy of the retrieval results and makes it difficult to effectively support the subsequent enhancement generation task. In addition, the OCR process is tedious and time-consuming to process, and usually requires the use of specialized recognition models for different modal information, such as specialized table recognition models or formula recognition models, thereby further increasing the implementation difficulty and cost.

[0006] Therefore, it is urgent to propose a lightweight multi-modal document retrieval enhancement method and system with simple structure, efficient processing and retrieval accuracy, to effectively meet the urgent need of efficient and accurate retrieval enhancement for large-scale multi-modal documents in practical applications. SUMMARY

[0007] In view of the above, the present application provides a lightweight and efficient multi-modal document retrieval enhancement method, which trains a sparse encoder, a multi-vector encoder and a multi-modal document rearranger by using a multi-modal large language model, combines sparse retrieval with multi-vector fine-grained post-interaction strategy, and introduces a time cost-aware loading mechanism, thereby significantly reducing memory and computing overhead while improving the accuracy and response efficiency of multi-modal document retrieval.

[0008] A lightweight and efficient multi-modal document retrieval enhancement method, comprising the following steps:

[0009] (1) Obtain a multi-modal document library, wherein the documents in the library contain one or more modal information such as text, image, chart and table;

[0010] (2) Based on an open source dataset, train a sparse encoder, a multi-vector encoder and a multi-modal document rearranger based on a multi-modal large language model;

[0011] (3) Use the trained sparse encoder and multi-vector encoder to encode the documents in the multi-modal document library, and extract the sparse vector representation and dense multi-vector representation of the documents, respectively;

[0012] (4) Based on the sparse vector representation, construct an inverted index structure and store it in the disk, and use a balanced clustering-based disk storage algorithm to store the dense multi-vector representation;

[0013] (5) Load the inverted index structure and use sparse retrieval to quickly filter out candidate documents and calculate sparse scores; for the candidate documents, use a time cost-aware loading strategy to dynamically load the dense multi-vector representation of the candidate documents from the disk and calculate the delayed interaction scores; based on the scores after fusion of the sparse scores and the delayed interaction scores, further screen the candidate documents;

[0014] (6) After screening, splice the remaining candidate documents and the user's natural language query through a template, input them into the trained multi-modal document rearranger for reordering, and obtain the most relevant documents;

[0015] (7) Input the natural language query and the most relevant document into the multi-modal large language model to automatically generate the question and answer and return the final result to the user.

[0016] Further, the format of the documents in the multi-modal document library is any one of PDF, Word, PPT, and the like, and each document is composed of structured metadata (such as title, chapter hierarchy, author information), natural language paragraphs, illustrations, charts, tables, and formulas.

[0017] Further, the sparse encoder in step (2) is used to learn a sparse representation with a high compression rate and suitable for large-scale inverted index construction; the multi-vector encoder is used to capture the fine-grained semantic structure within the document to support subsequent token-level delayed interactive retrieval; and the multi-modal document rearranger is used for high-precision semantic matching and sorting of candidate documents; and LoRA (Low Rank Adaptation) lightweight parameter fine-tuning update strategy is used in the training process of the above three.

[0018] Further, the specific implementation of step (3) is as follows: first, the documents in the multi-modal document library are uniformly converted into PDF format, and the documents are image-processed at page-level granularity to generate image sequences; then the image sequences are sequentially input into the trained sparse encoder and multi-vector encoder for encoding, wherein the sparse encoder performs projection, ReLU (Rectified Linear Unit) function, and maximum pooling processing on the token-level hidden state obtained by image encoding to generate a bag-of-words sparse representation, which is used to construct an inverted index structure and support preliminary document recall; the multi-vector encoder retains all token hidden states output by image encoding and uniformly compresses them into a set of 128-dimensional vectors through linear mapping to form a fine-grained dense multi-vector representation, which is used for subsequent token-level fine-grained delayed interactive retrieval.

[0019] Further, in step (4), for the sparse vector representation of each document, only the weights of the non-zero dimensions are retained, and the document ID and weight corresponding to the non-zero dimensions are mapped into a sparse inverted list, thereby constructing an inverted index structure from the keyword dimension to the document set; at the same time, all documents are divided into several clusters that are similar in semantics and balanced in size by an equal clustering algorithm using the sparse vector representation as a feature, and then the dense multi-vector representation of the documents is written to the disk according to the clustering results in continuous physical positions, so that the dense multi-vector representations of the documents in the same cluster are stored in the same block of disk to reduce the randomness of disk access when reading.

[0020] Further, the step (5) is implemented as follows: first, the natural language query of the user is encoded by a sparse encoder and a multi-vector encoder to obtain a sparse query vector and a dense query multi-vector; then, the inverted index structure is loaded, the sparse score between the sparse query vector and the sparse vector representation of each document is calculated based on the vector overlap dimension, and the top M documents with the highest sparse scores are taken as candidate documents; further, the dense multi-vector representation of the candidate documents is loaded by using a time cost-aware loading strategy, when a cluster contains more candidate documents, the dense multi-vector representation of the entire cluster of documents is loaded; if a cluster contains only a small number of candidate documents, only the dense multi-vector representation of the candidate documents is loaded; the delayed interaction score between the dense query multi-vector and the dense multi-vector representation of each candidate document is calculated based on the token-level delayed interaction matching similarity algorithm, and finally the fusion score of each candidate document is calculated according to the following formula, and the candidate documents are further screened according to the fusion score.

[0021]

[0022] Wherein: score is the fusion score of the candidate document, score spa is the sparse score of the candidate document, score mul is the delayed interaction score of the candidate document, Z( ) is a standardization function, a is a weight coefficient, and M is a natural number greater than 1.

[0023] Further, in the step (6), each candidate document retained after screening is concatenated with the natural language query of the user as an input sequence, and these input sequences are input one by one into the multi-modal document rearranger, the rearranger takes the logarithmic probability of "Yes" obtained by projecting the hidden layer state corresponding to the last token of the input sequence into the vocabulary space of the multi-modal large language model as the relevance score, and then reorders the candidate documents according to the relevance score, and takes the candidate document with the highest relevance score as the most relevant document.

[0024] A computer device comprising a memory and a processor, the memory having a computer program stored therein, and the processor being configured to execute the computer program to implement the above-mentioned multi-modal document retrieval enhancement method.

[0025] A computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the above-mentioned multi-modal document retrieval enhancement method.

[0026] The application effectively alleviates the memory bottleneck and response delay problems faced by traditional multi-modal retrieval methods in a massive data scenario by designing a hybrid retrieval architecture combining sparse indexing and multi-vector representation; the time cost perception loading strategy introduced cooperates with the clustering storage mechanism to optimize the delay interaction calculation efficiency; finally, the sparse and delay interaction scores are fused, and combined with a multi-modal language model reorderer, the document matching precision is significantly improved, and the response efficiency and user experience of the multi-modal retrieval enhancement system are comprehensively improved. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A flowchart of the lightweight and efficient multi-modal document retrieval enhancement method of the application. DETAILED DESCRIPTION

[0028] In order to describe the application more specifically, the technical solutions of the application will be described in detail below in combination with the drawings and specific embodiments.

[0029] The multi-modal document retrieval enhancement scheme proposed by the application is driven by a multi-modal large language model, based on a sparse vector encoder and a multi-vector encoder trained to convert documents into sparse representations with clear structure and rich semantics and fine-grained multi-vector representations, and to perform efficient representation matching and retrieval score integration through a fusion algorithm. At the same time, the application introduces a disk continuous storage mechanism for document clustering and a time cost perception loading strategy, which reduces memory usage and access delay while ensuring the calculation efficiency of the delay interaction score calculation stage. Finally, a reorderer driven by a multi-modal large language model is used to reorder the candidate documents with high precision, significantly improving the final retrieval relevance, and achieving the design goals of "storage saving, delay optimization, and precision improvement", which is suitable for efficient retrieval enhancement tasks in various scenarios such as large-scale RAG and multi-modal intelligent systems.

[0030] EMBODIMENT

[0031] As shown in Figure 1 The embodiment provides a lightweight and efficient multi-modal document retrieval enhancement method, and the specific implementation process is as follows:

[0032] Step S11: Obtain a multi-modal document library, wherein the document format can be PDF, Word, PPT, etc., and the document contains one or more modal information such as text, image, chart, and table.

[0033] Specifically, the present application can be widely applied to government affairs disclosure, scientific research literature management, enterprise archives archiving, medical archives search, legal regulations review, patent examination assistance, and other practical scenarios. Without loss of generality, the multi-modal document library in the present embodiment is represented as a data set containing heterogeneous documents, which are derived from Internet public resources, enterprise knowledge bases, or scientific research databases. Each document is composed of structured metadata (such as title, chapter hierarchy, author information), natural language paragraphs, illustrations, charts, tables, and formulas, etc. For example, in the scientific research literature scenario, the document may be embedded with experimental visualization graphs, LaTeX formulas, and structured result tables.

[0034] Step S12: training a sparse encoder, a multi-vector encoder, and a multi-modal rearranger based on an open-source data set and a multi-modal large language model to achieve efficient representation and fine sorting judgment capability of the document.

[0035] Specifically, each sample in the open-source data set used in the present embodiment contains a query and a corresponding document page for training the model. A unified encoding and sorting framework is constructed using a multi-modal large language model, and independent optimization objectives are designed for the three types of sub-modules: the sparse encoder is dedicated to learning a sparse representation with high compression rate suitable for large-scale inverted index construction; the multi-vector encoder is used to capture the fine-grained semantic structure within the document to support subsequent token-level delayed interactive retrieval; and the multi-modal rearranger is used for high-precision semantic matching and sorting of candidate documents. In order to improve training efficiency and enhance deployment flexibility, each sub-module adopts a low-rank adaptive fine-tuning lightweight parameter updating strategy, which significantly reduces the training resource overhead while ensuring the generalization ability of the model in multiple scenarios.

[0036] The specific training process is as follows: the sparse encoder is optimized by a contrastive learning loss, the hidden state extracted after inputting the document image into the multi-modal large language model is mapped to the vocabulary space, and the sparse features are extracted through ReLU activation and max-pooling operation, and finally the bag-of-words representation for sparse retrieval is generated. The multi-vector encoder retains all token-level hidden states generated by image encoding, and further compresses them into a set of vectors with a uniform dimension (such as 128 dimensions) through linear mapping to form a dense vector representation that supports delayed loading and interactive computation. The training objective is also to minimize the contrastive learning loss. The training of the rearranger is based on the instruction fine-tuning method, a multi-modal input sequence containing the query and the document image is constructed, and the model sorting ability is optimized by maximizing the logarithmic probability of the "Yes" token. The above three sub-modules share the underlying multi-modal large language base model, and through the modular switching mechanism, they efficiently complete their respective functions in the video memory, thereby constructing a multi-modal retrieval system with collaborative optimization.

[0037] Step S13: First, the multi-modal document is uniformly converted to PDF format, and each page PDF is converted to an image. The sparse encoder and multi-vector encoder are used to encode the document to obtain sparse vector representation and multi-vector representation respectively, which can be used for sparse retrieval and delay interaction score calculation.

[0038] Specifically, the embodiment first converts the original multi-modal document (including PDF, Word, PPT, etc.) into PDF format to ensure input format consistency and visual structure fidelity; then image processing is performed on each PDF document at page level granularity to generate a high-resolution image page sequence as the subsequent visual input source.

[0039] Then, encoding is performed. The system parallelly inputs each page image into the sparse encoder and multi-vector encoder trained in step S12 to extract two types of complementary representation information respectively. The sparse encoder generates a bag-of-words sparse vector representation by projecting the token-level hidden state obtained by image encoding, ReLU activation and max-pooling processing. The sparse vector is used to construct an inverted index structure and support preliminary document recall. At the same time, the multi-vector encoder retains all token hidden states output by the image encoder and compresses them into a set of 128-dimensional vectors through a linear mapping module to form a fine-grained dense semantic multi-vector representation for subsequent token-level fine-grained delay interaction retrieval. The above process supports independent vector encoding for each page of the document, and the vectors of all pages can be aggregated to form the global representation structure of the entire document.

[0040] It should be noted that the entire preprocessing and encoding process can be efficiently parallelized in a GPU (graphics processing unit) or a dedicated acceleration device, supporting batch document vectorization processing, and decoupled from the subsequent index construction and retrieval module, which can be independently deployed and updated, significantly improving the maintainability and engineering usability of the system.

[0041] Step S14: Based on the sparse vector representation, an efficient and scalable inverted index structure is constructed, and sparse vectors are used for document clustering. The clustering results are used to optimize the balanced disk storage of multi-vector representation to improve the loading efficiency.

[0042] Specifically, the extracted sparse vector representation is represented as: where i represents the i-th document page, |V| is the vocabulary size of the multi-modal large language model, w ji The importance weight of the document on the j-th token is represented as w The inverted structure supports the online query stage, which quickly retrieves a candidate document set through the non-zero dimensions of the sparse query vector, and calculates a sparse score based on the overlapping dimensions of the vectors ).

[0043] In addition, to reduce the disk I / O cost of the subsequent multi-vector delay interaction score calculation stage, the embodiment further adopts a document clustering strategy based on the semantic distribution of sparse vectors, realizing the optimization of disk sequential storage of multi-vector representation. Specifically, taking the document sparse vector as the input feature, the document is divided into several clusters with similar semantics and balanced size through an improved balanced clustering algorithm, which ensures that the number of documents in each cluster meets the balance constraint, avoiding the instability problem of loading performance caused by extreme large or small clusters. Subsequently, the system sequentially writes the multi-vector representation of the document to the disk according to the clustering results, ensuring that the multi-vector representation of the documents in the same cluster is physically stored in the same disk block, so that after the candidate documents are screened out by subsequent sparse retrieval, efficient batch sequential loading can be realized with less random I / O cost.

[0044] Step S15: In the retrieval stage, first use sparse retrieval to quickly filter out candidate documents, and then according to the time cost-aware reading algorithm, dynamically select full clustering loading or specified document multi-vector loading for the filtered candidate documents to perform refined delay interaction calculation, fuse sparse score and multi-vector delay interaction score, and select the top-K relevant documents according to the ranking.

[0045] Specifically, first, based on the constructed inverted index structure, and using the non-zero dimensions in the sparse vector to match the keywords in the inverted index, efficient document preliminary screening operation is performed on the user query. Specifically, the query input is encoded into a sparse vector using a sparse encoder, and a candidate document set is quickly retrieved through the non-zero dimensions of the sparse query vector, and a sparse score is calculated based on the overlapping dimensions of the vectors ).

[0046] After screening out the Top-M candidate documents with the highest scores in the sparse retrieval stage, the user query is first input through a multi-vector encoder to encode it into a set of multi-vector representations to capture various fine-grained semantic information in the query, and then a dynamic decision is made for the loading strategy of the multi-vector representation of each of the M candidate documents, which comprehensively considers the number of candidate documents, the disk read-write throughput capacity, and the cluster size. Specifically, the system considers the distribution of the vector storage clusters to which the candidate documents belong in the disk, the number of documents in the cluster, the proportion of the target document in the cluster, and the efficiency of sequential reading and random reading of the storage device, and adopts one of the two loading strategies for the storage cluster where each candidate document is located: one is "cluster loading", that is, when the cluster contains a large number of target documents, the vector data in the entire cluster is loaded into the memory at one time to reduce the number of disk random accesses and improve the loading efficiency; the other is "specified loading", that is, when the cluster contains only a small number of target documents, the multi-vector representation corresponding to the target document is directly located and read from the disk to avoid unnecessary loading of redundant data. The time-cost-aware dynamic loading strategy ensures that the overall efficiency of document multi-vector loading is maximized while ensuring retrieval accuracy.

[0047] After completing the vector loading, the system performs token-level delayed interactive matching similarity score calculation for each query q and candidate document, calculates the multi-vector delayed interactive score , and fuses the obtained sparse score ) and the normalized multi-vector delayed interactive score to obtain the final document comprehensive score to fully utilize the complementary advantages of sparse and dense, and according to the fused score, the final document ranking is obtained, and the fusion formula of the process is:

[0048]

[0049] wherein: a is a weight coefficient, Z( ) is a standardization function, is the sparse retrieval score, is the multi-vector delayed interactive score, and the fusion formula is used to balance the influence of the two types of scores on the final result.

[0050] Step S16: splice the Top-K relevant documents and the user query according to the ranking template, input into the multi-modal large language model reorderer, and output the logarithmic probability of the "Yes" word as the final relevance score according to the last token of the sequence, and further reorder the documents.

[0051] Specifically, the embodiment adopts a multimodal large language model based on instruction fine-tuning as the rearranger, and supports multimodal input, including query text and document image content. In actual operation, the system first splices each pair of “query-candidate document” into a complete input sequence according to a predefined prompt template, which explicitly guides the model to judge the relevance between the document and the query. For example, the input template can be “Given the user query: <query>, is the following document relevant? <document>Answer Yes or No.”, wherein the document part is the image embedding or encoded representation of the document, and the query part is the user natural language input.

[0052] The spliced sequence is then input into the multi-modal rearranger, which predicts the logarithmic probability of the "Yes" token according to the input sequence based on the judgment ability learned during the training phase, as the final relevance score between the candidate document and the current query; the system performs the same process on all candidate documents and finally rearranges them according to the score to filter out the document results that best match the user's intent.

[0053] Through the above operation, step S16 realizes high-precision reordering based on prior sparse and delayed interaction, effectively improving the adaptability of the document retrieval system to modal heterogeneity and semantic complex scenarios, and lays a precise document input foundation for the final question and answer generation task.

[0054] Step S17: input the user query and the most relevant document into the multi-modal large language model to realize various downstream tasks such as question and answer, summary, etc., and return the final result to the user to improve the user's interactive experience.

[0055] The above description of the embodiments is to facilitate those skilled in the art to understand and apply the present application. Those skilled in the art can easily make various modifications to the above embodiments, and apply the general principles described herein to other embodiments without creative labor. Therefore, the present application is not limited to the above embodiments, and the improvements and modifications made by those skilled in the art to the present application should be within the scope of protection of the present application.< / document> < / query>

Claims

1. A lightweight and efficient multimodal document retrieval enhancement method, characterized in that, Includes the following steps: (1) Obtain a multimodal document library, in which the documents contain one or more modal information including text, images, charts, and tables; (2) Based on the open-source dataset and the multimodal large language model, train a sparse encoder, a multi-vector encoder and a multimodal document reordering machine. (3) Use the trained sparse encoder and multi-vector encoder to encode the documents in the multimodal document library, and extract the sparse vector representation and dense multi-vector representation of the documents respectively; (4) Construct an inverted index structure based on sparse vector representation and store it on disk. Use a disk storage algorithm based on balanced clustering to store dense multi-vector representation. (5) Load the inverted index structure and use sparse retrieval to quickly filter out candidate documents and calculate the sparse score; for candidate documents, use a time cost-aware loading strategy to dynamically load the dense multi-vector representation of the candidate document from the disk and calculate the delayed interaction score; further filter the candidate documents based on the score obtained by fusing the sparse score and the delayed interaction score; specifically: First, a sparse encoder and a multi-vector encoder are used to encode the user's natural language query to obtain sparse query vectors and dense query multi-vectors. Then, an inverted index structure is loaded, and the sparse score between the sparse query vector and the sparse vector representation of each document is calculated based on the vector overlap dimension. The top M documents with the highest sparse scores are selected as candidate documents. Next, a time-cost-aware loading strategy is used to load the dense multi-vector representations of the candidate documents. If a cluster contains many candidate documents, the dense multi-vector representations of the entire cluster are loaded; if a cluster contains only a few candidate documents, only the dense multi-vector representations of the candidate documents are loaded. A token-level delayed interaction matching similarity algorithm is used to calculate the delayed interaction score between the dense query multi-vector and the dense multi-vector representation of each candidate document. Finally, the fusion score of each candidate document is calculated according to the following formula, and the candidate documents are further filtered based on the fusion score. score=α·Z(score spa )+Z(score mul ) Where: score is the fusion score of the candidate documents, score spa The sparse score for candidate documents, score mul Z() is the delayed interaction score of the candidate document, α is the weight coefficient, and M is a natural number greater than 1. (6) The candidate documents retained after filtering are combined with the user's natural language query through a template and then input into the trained multimodal document reorderer for reordering to obtain the most relevant documents; (7) Input the natural language query and the most relevant document into the multimodal large language model to automatically generate questions and answers and return the final results to the user.

2. The multimodal document retrieval enhancement method according to claim 1, characterized in that: The documents in the multimodal document library can be in any format, including PDF, Word, and PPT. Each document consists of structured metadata, natural language paragraphs, illustrations, charts, tables, and formulas.

3. The multimodal document retrieval enhancement method according to claim 1, characterized in that: The sparse encoder in step (2) is used to learn a sparse representation with high compression ratio that is suitable for large-scale inverted index construction; The multi-vector encoder is used to capture the fine-grained semantic structure within the document to support subsequent token-level delayed interactive retrieval; the multimodal document reorderer is used to perform high-precision semantic matching and ranking of candidate documents; the LoRA lightweight parameter fine-tuning update strategy is adopted during the training of the above three components.

4. The multimodal document retrieval enhancement method according to claim 1, characterized in that: The specific implementation of step (3) is as follows: First, the documents in the multimodal document library are uniformly converted into PDF format, and the documents are image-processed at the page level to generate image sequences. Then, the image sequences are sequentially input into the trained sparse encoder and multi-vector encoder for encoding. The sparse encoder generates a bag-of-words sparse representation by projecting, using the ReLU function and max pooling to the token-level hidden states obtained from the image encoding. This is used to construct an inverted index structure and support preliminary document retrieval. The multi-vector encoder retains all the token hidden states output by the image encoding and compresses them uniformly into a 128-dimensional vector set through linear mapping to form a fine-grained dense multi-vector representation for subsequent token-level fine-grained delayed interactive retrieval.

5. The multimodal document retrieval enhancement method according to claim 1, characterized in that: In step (4), for the sparse vector representation of each document, only the weights of the non-zero dimensions are retained, and the document IDs corresponding to the non-zero dimensions are mapped to the weights to form a sparse inverted index, thereby constructing an inverted index structure from the keyword dimension to the document set; at the same time, using the sparse vector representation as a feature, all documents are divided into several semantically similar and equally sized clusters through a balanced clustering algorithm, and then the dense multi-vector representation of the document is written to the disk according to the clustering results in a continuous physical location, ensuring that the dense multi-vector representation of the document in the same cluster is stored on the same disk.

6. The multimodal document retrieval enhancement method according to claim 1, characterized in that: In step (6), each candidate document retained after filtering is concatenated with the user's natural language query in the order of "query-candidate document" to form an input sequence. These input sequences are then input into the multimodal document reorderer one by one. The reorderer projects the hidden layer state corresponding to the last token of the input sequence onto the vocabulary space of the multimodal large language model and obtains the log probability of "Yes" as the relevance score. Then, the candidate documents are reordered according to the relevance score, and the candidate document with the highest relevance score is selected as the most relevant document.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: The processor is used to execute the computer program to implement the multimodal document retrieval enhancement method as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the multimodal document retrieval enhancement method as described in any one of claims 1 to 6.

Images