A multi-modal retrieval method and system based on fine-grained interaction

Abstract

This application provides a multimodal retrieval method and system based on fine-grained interaction. The method includes: converting each page of a document into an image and attaching a text summary, preserving the contextual relationship between the image and text, integrating the text and image content within the same framework, and ensuring the integrity and consistency of information during the conversion process; parsing the page image to generate multi-dimensional embeddings for each region, ensuring that information in each region can be accurately extracted and processed; generating fine-grained embeddings for each region, accurately understanding the semantic information contained in different parts of the image, and combining it with text information to ensure efficient information matching; and mapping the features of the image and text into the same latent space through multi-dimensional embedding and fine-grained interaction alignment, thereby capturing finer-grained relevance. This method can deeply explore the deep connections between images and text, improving the accuracy and efficiency of information retrieval and generation.

Description

Technical Field

[0001] This application belongs to the field of multimodal document retrieval, specifically involving a multimodal retrieval method and system based on fine-grained interaction. Background Technology

[0002] With the rapid development of information technology, the generation and dissemination of multimedia data have become increasingly widespread. Modern documents are no longer limited to plain text, but increasingly include multiple media formats such as tables and images. This emergence of multimodal documents presents new challenges to information retrieval and processing technologies. Traditional information retrieval systems, primarily based on text data, struggle to fully understand and process complex documents containing multiple media formats, resulting in insufficient accuracy and relevance of search results. With the diversification of document content, especially the introduction of non-textual information such as images, charts, and audio, existing technologies are unable to effectively integrate and process this heterogeneous information, urgently requiring new methods to improve information comprehension and retrieval effectiveness. Summary of the Invention

[0003] The purpose of this application is to explore and implement efficient multimodal information retrieval technology to improve the accuracy and comprehensiveness of information retrieval. Multimodal information retrieval can more comprehensively understand and represent information in complex documents, and by integrating text, table and image data, it can provide more relevant and comprehensive retrieval results, and reduce information omissions and misjudgments.

[0004] To achieve the above objectives, this application proposes a multimodal retrieval method based on fine-grained interaction, including:

[0005] Each page of the document to be queried is converted into an image, and a page embedding model is used to generate the page embedding and corresponding text summary for each image; the image is parsed into multiple logical regions, and a multi-modal model is used to generate multi-dimensional text and image embeddings for each logical region;

[0006] This process involves extracting high-dimensional features from images using a multimodal model to generate multi-dimensional image embeddings; converting user query text into multi-dimensional text embeddings using word or sentence embedding techniques; decomposing the multi-dimensional image and text embeddings into multiple image tokens and text tokens; calculating the association score between each pair of image and text token embeddings to form an interaction matrix, where each element corresponds to the similarity of a token pair; aggregating the scores in the matrix to generate the final global relevance score; mapping image tokens and text tokens to the same latent space using an alignment network; and optimizing the mapping function by training the alignment network to make the features of images and text as similar as possible in the latent space.

[0007] Calculate the similarity between the text embedding of the user query text and the page embedding, and combine the semantic filtering of the page text summary to select the top N page images that are most likely to contain the search results as candidate pages, thereby narrowing the scope of subsequent searches.

[0008] Based on the logical regions of the candidate pages and the multi-dimensional embeddings in the alignment network, the similarity between the user query text and the multi-dimensional embeddings of each logical region is calculated. The key regions within the candidate pages are further filtered out by combining the type features of the logical regions. The set of local content most relevant to the user query text is returned through cross-modal fusion analysis.

[0009] As an improvement to the above method, the page embedding model is ColQwen2, Qwen2.5-VL-32B or VisRAG-Ret.

[0010] As an improvement to the above method, the MinerU tool is used to parse the image into multiple logical regions.

[0011] As an improvement to the above method, the multimodal model is CLIP or ViT-BERT.

[0012] As an improvement to the above method, the word embedding or sentence embedding technology is BERT or RoBERTa.

[0013] As an improvement to the above method, cosine similarity or dot product is used to calculate the association score between each pair of image token and text token embeddings.

[0014] As an improvement to the above method, the alignment network is SBANet or MagNet.

[0015] This application also provides a multimodal retrieval system based on fine-grained interaction, implemented based on the above method, the system comprising:

[0016] The document semantic fine-grained representation module is used to convert each page of the document to be queried into an image, and use the page embedding model to generate the page embedding and corresponding text summary of each image; the image is parsed into multiple logical regions, and a multi-modal model is used to generate multi-dimensional text and image embeddings for each logical region.

[0017] The cross-modal interaction module based on image-text alignment is used to extract high-dimensional features from images using a multimodal model to generate multi-dimensional image embeddings; convert user query text into multi-dimensional text embeddings using word embedding or sentence embedding techniques; decompose the multi-dimensional embeddings of images and text into multiple image tokens and text tokens; calculate the association score between each pair of image token and text token embeddings to form an interaction matrix; map image tokens and text tokens to the same latent space using an alignment network; and optimize the mapping function by training the alignment network to make the features of images and text as similar as possible in the latent space.

[0018] The coarse-grained retrieval module is used to calculate the similarity between the text embedding of the user query text and the page embedding. Combined with the semantic filtering of the page text summary, it selects the top N page images that are most likely to contain the search results as candidate pages.

[0019] The fine-grained retrieval module is used to calculate the similarity between the user query text and the multi-dimensional embeddings of each logical region based on the logical regions and the alignment network of the candidate pages. It further filters out key regions within the candidate pages by combining the type features of the logical regions, and returns the set of local content most relevant to the user query text through cross-modal fusion analysis.

[0020] Compared with existing technologies, the advantages of this application are:

[0021] 1. This invention designs a fine-grained representation method for document semantics, which involves a two-stage retrieval process. The first stage is coarse-grained retrieval, which retrieves relevant page images based on the user's input query. The second stage is fine-grained retrieval, which further retrieves the text and image content in the page images based on the page images retrieved in the first stage.

[0022] 2. This invention designs a cross-modal interaction method for image-text alignment. It uses multi-dimensional embedding technology to extract high-dimensional features of images and text respectively, generating rich multi-modal embedding vectors. Then, the image and text features are aligned through an alignment network to ensure the consistency and integrity of information. Attached Figure Description

[0023] Figure 1 The diagram shows a flowchart of a multimodal retrieval method based on fine-grained interaction.

[0024] Figure 2 The diagram shows a structural schematic of a document semantic fine-grained representation technique.

[0025] Figure 3The diagram shows a schematic of cross-modal interaction based on image-text alignment; where PositionalEmbedding is the location embedding; Textual token is the text token; Visual token is the image token; and WeightedSum is the weighted sum. Detailed Implementation

[0026] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0027] Multimodal information retrieval technology, by comprehensively utilizing data from various media formats such as text, images, video, and audio, can more comprehensively understand and represent information content, thereby significantly improving the accuracy and relevance of information retrieval. Compared to traditional text retrieval methods, multimodal retrieval can better capture the relationships and semantics between different media formats, reducing comprehension biases caused by limited or missing information. Furthermore, multimodal information retrieval can better meet users' needs for diverse information, providing richer and more intuitive search results, significantly enhancing the user experience. For documents containing complex elements (such as images and charts), traditional retrieval methods often fail to effectively extract and match relevant information, while multimodal information retrieval can comprehensively handle these elements, ensuring higher retrieval accuracy.

[0028] Simultaneously, the development of this technology will also drive technological advancements in related fields such as natural language processing, computer vision, and audio processing, promoting their widespread application. Through deeper cross-domain collaboration, multimodal technology can strengthen interaction between various fields, further advancing multimodal learning and understanding. For example, in fields such as healthcare, education, entertainment, and security, multimodal information retrieval can help users obtain the information they need more accurately, promoting the popularization and implementation of intelligent applications.

[0029] Multimodal information retrieval technology, by comprehensively processing text, tables, and images, significantly improves the accuracy and comprehensiveness of information retrieval, enhances the quality and diversity of information generation, improves automation and intelligence, and enhances user experience. It not only provides new ideas for traditional information retrieval systems but also offers strong technical support for various complex multimodal data processing. These advantages make multimodal retrieval technology highly valuable in practical applications, especially demonstrating great potential in processing large document databases, news recommendations, and cross-platform information retrieval.

[0030] To address the limitations of existing technologies, this application proposes a multimodal retrieval method based on fine-grained cross-modal interaction. This method preserves the contextual relationship between images and text by converting each page of a document into a single image and attaching a detailed text summary. In this way, text and image content can be integrated within the same framework, ensuring the integrity and consistency of information during conversion and processing. Furthermore, the page images are parsed to generate multi-dimensional embeddings for each region, ensuring that information in each sub-region can be accurately extracted and processed. Through fine-grained embedding generation for each region, the system can accurately understand the semantic information contained in different parts of the image and combine it with text information, ensuring efficient information matching.

[0031] By employing alignment techniques involving multi-dimensional embedding and fine-grained interaction, the features of images and text are mapped into the same latent space, further enabling fine-grained cross-modal interactions and capturing more nuanced relevance. This method can deeply explore the profound connections between images and text, improving the accuracy and efficiency of information retrieval and generation. Furthermore, the introduction of fine-grained interaction allows the system to capture more subtle differences and connections when associating image and text information, significantly enhancing the accuracy and relevance of retrieval results.

[0032] By combining these technologies, this invention not only effectively processes multimodal documents but also provides a more efficient and accurate solution in the field of multimodal information retrieval. This method, by comprehensively integrating image and text information, enhances the system's cross-modal understanding capabilities and has broad application prospects and market value.

[0033] This invention proposes a multimodal retrieval method based on fine-grained interaction. First, the document to be retrieved undergoes fine-grained semantic processing; then, cross-modal processing based on image-text alignment is performed; finally, semantic retrieval can be performed on the document. This method includes:

[0034] Step 1) Fine-grained representation techniques based on document semantics;

[0035] Step 2) Cross-modal interaction technology based on image-text alignment;

[0036] Step 3) A two-stage retrieval process from coarse to fine.

[0037] In the above technical solution, step 1) specifically includes:

[0038] Step 101) Convert each page of the document into a complete page image. This process preserves the overall layout and visual information of the page. Through this conversion, the structured information such as typography, fonts, colors, and text-image relationships on the page is fully captured, allowing subsequent processing to perform fine-grained analysis at the image level, ensuring that every part of the information in the document is reasonably expressed and understood.

[0039] Step 102) For each page image, generate page embeddings and corresponding text summaries using mainstream page embedding models (such as ColQwen2 and VisRAG-Ret) so that the language model can better understand the content of the image. The generation of text summaries can rely on the visual model Qwen2.5-VL-32B, which can effectively extract key information from the image and provide more concise and explicit text data for subsequent retrieval tasks.

[0040] Step 103) The page image is parsed into multiple logical regions using tools such as MinerU, such as title areas, body text areas, chart areas, and list areas. The content features and structural information of each region are extracted and analyzed separately, including text, images, and tables, thus providing a foundation for subsequent cross-modal alignment and embedding generation. This process not only helps to extract the diversity of document content but also allows each part of the document's information to be processed and analyzed accurately on its own.

[0041] Step 104) Apply a multimodal model (such as CLIP, ViT-BERT, etc.) to each parsed fine-grained element (such as text, image, table) to generate multi-dimensional text and image embeddings for each region. Through these models, image regions and text information can be mapped into a common embedding space. These embedding representations can capture the complex semantic relationships between regions and provide effective support for subsequent cross-modal retrieval.

[0042] Step 105) Employing efficient vector storage technologies (such as FAISS) allows for rapid storage and retrieval of embeddings for each page and fine-grained elements, reducing computational and storage overhead. Based on the needs of actual application scenarios, the region partitioning strategy and embedding generation method are dynamically adjusted to flexibly address different document types and complexities. Efficient storage and retrieval technologies like FAISS significantly improve the system's operating efficiency on large-scale datasets and further optimize the retrieval process by dynamically adjusting strategies to adapt to various complex situations.

[0043] In the above technical solution, step 2) specifically includes:

[0044] Step 201) Encode image and text information using multi-dimensional embeddings, rather than simplifying them to a single one-dimensional embedding. Multi-dimensional embeddings allow for more precise representation of image and text information, ensuring fine-grained understanding and effective cross-modal integration. This approach better captures the diversity and complexity of information, avoiding information loss common in traditional methods. Specific steps include: extracting high-dimensional features from the image using multi-modal models (such as CLIP, ViT-BERT, etc.) to generate multi-dimensional embeddings of the image. Each feature dimension in the image corresponds to a different semantic level, allowing the embedding representation to encompass diverse information within the image.

[0045] Step 202) Use word embedding or sentence embedding techniques (such as pre-trained models like BERT and RoBERTa) to convert the query text into a multi-dimensional vector representation. These pre-trained models can capture the complex semantic information and contextual relationships of the text through deep learning methods, thus laying the foundation for the effective fusion of images and text. Text embedding can effectively map the lexical, grammatical, and semantic information in the text, generating richer text representations.

[0046] Step 203) decomposes the multi-dimensional embeddings of images and text into multiple tokens, each token representing a local feature or word. This step further subdivides the image and text embedding representations into independent parts through a fine-grained tokenization method, facilitating the processing of local features one by one. In this way, the system can better capture the subtle connections between images and text, and improve the accuracy of cross-modal retrieval through precise tokenization processing.

[0047] Step 204) Achieve fine-grained semantic alignment between images and text by constructing an interaction matrix: First, calculate the association score between each pair of image-text token embeddings using cosine similarity or dot product to form a complete interaction matrix; then, aggregate the scores in the matrix using methods such as maximum value, average value, or attention weighting to generate the final global relevance score. Each element in the matrix corresponds to the similarity of a specific token pair, and the aggregation operation integrates these local matching signals into a quantitative indicator of overall semantic fit, preserving micro-level semantic associations while achieving macro-level matching evaluation.

[0048] Step 205) By introducing an alignment network (such as SBANet, MagNet, etc.), visual features (image tokens) and linguistic features (text tokens) are mapped to the same latent space, allowing them to be compared and fused in the same space. The alignment network learns to optimize the mapping function, bringing the features of the image and text as close as possible in the latent space, enabling a more seamless fusion. This mapping process improves the interaction and understanding depth between images and text.

[0049] Step 206) By training the alignment network, the mapping function is optimized to make the features of the image and text as similar as possible in the latent space. The alignment network maps visual and linguistic features to the same latent space, enabling them to be compared and fused more effectively. This optimization process improves the accuracy of cross-modal retrieval, allowing data from different modalities to work together better.

[0050] In the above technical solution, step 3) specifically includes:

[0051] A two-stage retrieval strategy is adopted to achieve accurate information positioning from the overall to the local: The first stage, coarse-grained page retrieval, takes the user query as input and calculates the similarity (e.g., cosine similarity) between the query text embedding and the page embedding generated in step 102). Combined with semantic filtering of page text summaries (e.g., Qwen3-8B-Reranker model), the top-N page images most likely to contain the answer are quickly selected, narrowing the retrieval scope. The second stage, fine-grained region retrieval, directly targets these candidate pages. Based on the logical regions (title, body text, charts, etc.) divided in step 103) and the cross-modal interaction technology based on image-text alignment in step 2), multi-dimensional region embeddings are generated. The similarity between the query text and each region embedding is calculated, and combined with region type features (e.g., prioritizing matching "chart area" to queries containing data visualization), key regions within the page are further filtered out. Finally, the most relevant set of local content is returned through cross-modal fusion analysis (e.g., linking visual features of charts with text descriptions in the body text). This progressive retrieval strategy, moving from page to region, ensures efficiency through coarse-grained retrieval while improving accuracy through fine-grained analysis, effectively balancing the scope of retrieval with the accuracy of information location.

[0052] Through this series of steps, the method of this invention can effectively improve the efficiency and accuracy of multimodal retrieval, especially demonstrating powerful processing capabilities when dealing with complex semantic relationships between images and text. This retrieval method based on fine-grained cross-modal interaction has broad application prospects in the field of information retrieval.

[0053] Example 1

[0054] like Figure 1As shown, this embodiment utilizes a fusion retrieval technique combining multi-dimensional embedding and fine-grained interaction. It combines high-dimensional image-based features with text-based word or sentence embeddings. By weighing and integrating the contributions of different modal features, it more comprehensively models the relationship between queries and documents, enhancing the retrieval model's capabilities in multi-modal information understanding and relevance determination. Specifically, each page of a document is converted into a single image and appended with a detailed text summary, preserving the contextual relationship between images and text. Furthermore, the page images are parsed to generate multi-dimensional embeddings for each region. Simultaneously, an alignment network (such as a Multilayer Perceptron (MLP) or Convolutional Neural Network (CNN)) is introduced to map visual and linguistic features into the same latent space, achieving fine-grained cross-modal interaction and capturing more granular relevance. Through the combination of these techniques, the multi-modal document semantic retrieval system can more effectively handle complex multi-modal documents, providing more accurate and comprehensive retrieval and generation results.

[0055] like Figure 2 As shown, the document semantic fine-grained representation method provided in this embodiment includes:

[0056] Step 1) Preliminary processing of page images

[0057] In this stage, each page of the document is first converted into a complete page image, a process that preserves the overall layout and visual information of the page. Image processing combines traditional textual and visual information, providing a unified format and content for subsequent processing steps. Then, for each page image, a page embedding and corresponding text summary are generated so that the language model can better understand the image's content. The text summary, as a supplement to the image information, enhances the system's ability to capture important information from the image, making the retrieval task more accurate and comprehensive.

[0058] Step 2) Dividing the page image into regions

[0059] The page image is parsed into multiple logical regions, such as title areas, body text areas, chart areas, and list areas. Tools like MinerU are used to identify the content type of each region, such as plain text, images, and tables. Region segmentation not only facilitates the structured processing of document content but also allows for the refined extraction of information features from each region, providing more detailed data support for subsequent retrieval tasks. The accuracy of region identification directly affects the quality of the retrieval results; therefore, this step is a crucial component of the entire system.

[0060] Step 3) Efficient Vector Storage Based on FAISS

[0061] By employing efficient vector storage technologies such as FAISS, the embeddings of each page and fine-grained element in a document can be quickly stored and retrieved, effectively reducing computational and storage overhead. Simultaneously, by dynamically adjusting the region partitioning strategy and embedding generation method according to the needs of real-world application scenarios, it can flexibly adapt to different document types and complexities. These methods not only significantly improve the system's operating efficiency on large-scale datasets but also optimize the retrieval process dynamically to better handle complex situations.

[0062] like Figure 3 As shown, the cross-modal interaction method for image-text alignment provided in this embodiment includes:

[0063] Step 1) Multi-dimensional embedding

[0064] Instead of simplifying them to a single one-dimensional embedding, multi-dimensional embeddings are used to encode image and text information. Specific steps include: extracting high-dimensional features from images using multimodal models (such as CLIP, ViT-BERT, etc.) to generate multi-dimensional image embeddings; and converting text into multi-dimensional vector representations using word embeddings or sentence embedding techniques (such as pre-trained models like BERT, RoBERTa, etc.). These high-dimensional representations not only capture information from images and text more accurately but also preserve more semantics and details, avoiding information loss common in traditional methods. Combining the multi-dimensional embeddings of images and text forms a comprehensive multimodal embedding representation, which can more fully represent document content and improve retrieval accuracy and efficiency.

[0065] Step 2) Fine-grained interaction

[0066] When calculating the relevance score, the interactions between each pair of token embeddings are considered, including cross-modal interactions. The specific steps include: decomposing the multi-dimensional embeddings of images and text into multiple tokens, each token representing a local feature or word; constructing an interaction matrix and calculating the similarity or relevance score between each pair of token embeddings; and obtaining the final relevance score by aggregating the scores from the interaction matrix. Fine-grained interactions capture more nuanced relevance, improving retrieval accuracy and relevance, thus providing more precise matching results when handling complex multimodal documents. This step effectively enhances the system's detailed understanding of document content, ensuring that every detail is fully considered.

[0067] Step 3) Align the network

[0068] By introducing an alignment network, visual and linguistic features are mapped to the same latent space, enabling comparison and fusion within that space. The specific steps include: using the alignment network to map the multi-dimensional embeddings of images and text to the same latent space; in this latent space, image and text features can be directly compared and fused; and training the alignment network to optimize the mapping function, making the image and text features as similar as possible in the latent space. By mapping visual and linguistic features to the same latent space, the alignment network allows for more effective comparison and fusion, further improving system performance. This process significantly enhances the ability to integrate cross-modal information, ensuring seamless information exchange and further improving the intelligence and accuracy of the retrieval system.

[0069] like Figure 2 As shown, the coarse-to-fine retrieval process provided in this embodiment includes:

[0070] The first stage is coarse-grained retrieval, which retrieves relevant page images based on the user's query. The second stage is fine-grained retrieval, which further retrieves text and image content from these page images based on the images retrieved in the first stage. Ultimately, it returns precise local information. This progressive retrieval from the overall to the specific ensures both efficiency and accurate location of detailed information.

[0071] By combining these technologies, this invention provides an efficient and accurate multimodal retrieval method that can deliver more precise matching results in complex document environments. This method can not only process standard documents but also handle comprehensive retrieval of various data types such as images, charts, and text, and has broad application prospects.

[0072] Example 2

[0073] This application also provides a multimodal retrieval system based on fine-grained interaction, implemented based on the above method, the system comprising:

[0074] The document semantic fine-grained representation module is used to convert each page of the document to be queried into an image, and use the page embedding model to generate the page embedding and corresponding text summary of each image; the image is parsed into multiple logical regions, and a multi-modal model is used to generate multi-dimensional text and image embeddings for each logical region.

[0075] The cross-modal interaction module based on image-text alignment is used to extract high-dimensional features from images using a multimodal model to generate multi-dimensional image embeddings; convert user query text into multi-dimensional text embeddings using word embedding or sentence embedding techniques; decompose the multi-dimensional embeddings of images and text into multiple image tokens and text tokens; calculate the association score between each pair of image token and text token embeddings to form an interaction matrix; map image tokens and text tokens to the same latent space using an alignment network; and optimize the mapping function by training the alignment network to make the features of images and text as similar as possible in the latent space.

[0076] The coarse-grained retrieval module is used to calculate the similarity between the text embedding of the user's query text and the page embedding. Combined with the semantic filtering of the page text summary, it selects the top N page images that are most likely to contain the search results as candidate pages, thereby narrowing the scope of subsequent searches.

[0077] The fine-grained retrieval module is used to calculate the similarity between the user query text and the multi-dimensional embeddings of each logical region based on the logical regions and the alignment network of the candidate pages. It further filters out key regions within the candidate pages by combining the type features of the logical regions, and returns the set of local content most relevant to the user query text through cross-modal fusion analysis.

[0078] This application may also provide a computer device, including: at least one processor, memory, at least one network interface, and a user interface. The various components in this device are coupled together via a bus system. It is understood that the bus system is used to implement communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0079] The user interface can include a display, keyboard, or clicking device. Examples include a mouse, trackball, touchpad, or touchscreen.

[0080] It is understood that the memory in the embodiments disclosed in this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0081] In some implementations, the memory stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0082] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application programs include various applications, such as media players and browsers, used to implement various application functions. Programs implementing the methods of the embodiments of this disclosure can be included in the application programs.

[0083] In the above embodiments, the processor can also invoke programs or instructions stored in memory, specifically programs or instructions stored in an application program, for the following purposes:

[0084] Follow the steps described above.

[0085] The above methods can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the disclosed methods, steps, and logic block diagrams. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the disclosed methods can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0086] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof.

[0087] For software implementation, the technology of this application can be implemented by executing the functional modules (e.g., procedures, functions, etc.) of this application. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0088] This application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, it can implement the steps in the above method embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A multimodal retrieval method based on fine-grained interaction, comprising: Each page of the document to be queried is converted into an image, and the page embedding model is used to generate the page embedding and corresponding text summary for each image; The image is parsed into multiple logical regions, and a multimodal model is used to generate multi-dimensional text and image embeddings for each logical region. High-dimensional features are extracted from images using a multimodal model to generate multidimensional embeddings of the images; Use word embedding or sentence embedding techniques to convert user query text into multi-dimensional text embeddings; The multi-dimensional embedding of images and text is decomposed into multiple image tokens and text tokens; Calculate the association score between each pair of image token and text token embeddings to form an interaction matrix, where each element in the interaction matrix corresponds to the similarity of a token pair; An alignment network is used to map image tokens and text tokens into the same latent space; By training an alignment network and optimizing the mapping function, the features of images and text are made as close as possible in the latent space. Calculate the similarity between the text embedding of the user query text and the page embedding, and combine the semantic filtering of the page text summary to select the top N page images that are most likely to contain the search results as candidate pages, thereby narrowing the scope of subsequent searches. Based on the logical regions of the candidate pages and the multi-dimensional embeddings in the alignment network, the similarity between the user query text and the multi-dimensional embeddings of each logical region is calculated. The key regions within the candidate pages are further filtered out by combining the type features of the logical regions. The set of local content most relevant to the user query text is returned through cross-modal fusion analysis.

2. The multimodal retrieval method based on fine-grained interaction according to claim 1, characterized in that, The page embedding model is ColQwen2, Qwen2.5-VL-32B, or VisRAG-Ret.

3. The multimodal retrieval method based on fine-grained interaction according to claim 1, characterized in that, Use the MinerU tool to parse the image into multiple logical regions.

4. The multimodal retrieval method based on fine-grained interaction according to claim 1, characterized in that, The multimodal model is CLIP or ViT-BERT.

5. The multimodal retrieval method based on fine-grained interaction according to claim 1, characterized in that, The word embedding or sentence embedding technology is BERT or RoBERTa.

6. The multimodal retrieval method based on fine-grained interaction according to claim 1, characterized in that, The association score between each pair of image token and text token embeddings is calculated using cosine similarity or dot product.

7. The multimodal retrieval method based on fine-grained interaction according to claim 1, characterized in that, The alignment network is either SBANet or MagNet.

8. A multimodal retrieval system based on fine-grained interaction, implemented according to the method of any one of claims 1-7, characterized in that, The system includes: The document semantic fine-grained representation module is used to convert each page of the document to be queried into an image, and use the page embedding model to generate the page embedding and corresponding text summary of each image; the image is parsed into multiple logical regions, and a multi-modal model is used to generate multi-dimensional text and image embeddings for each logical region. The cross-modal interaction module based on image-text alignment is used to extract high-dimensional features from images using a multimodal model to generate multi-dimensional image embeddings; convert user query text into multi-dimensional text embeddings using word embedding or sentence embedding techniques; decompose the multi-dimensional embeddings of images and text into multiple image tokens and text tokens; calculate the association score between each pair of image token and text token embeddings to form an interaction matrix; map image tokens and text tokens to the same latent space using an alignment network; and optimize the mapping function by training the alignment network to make the features of images and text as similar as possible in the latent space. The coarse-grained retrieval module calculates the similarity between the text embedding of the user's query text and the page embedding. Combined with semantic filtering of the page text summary, it selects the top N page images most likely to contain the search results as candidate pages, thereby narrowing the scope of subsequent searches. The fine-grained retrieval module is used to calculate the similarity between the user query text and the multi-dimensional embeddings of each logical region based on the logical regions and the alignment network of the candidate pages. It further filters out key regions within the candidate pages by combining the type features of the logical regions, and returns the set of local content most relevant to the user query text through cross-modal fusion analysis.

Images