Table-based multi-modal retrieval enhancement generation method
By constructing a multimodal knowledge graph and amplifying the central weight of the table, the problems of knowledge timeliness lag and insufficient long-tail knowledge coverage in knowledge-intensive scenarios of multimodal large models are solved, realizing accurate cross-modal knowledge linking and improving the accuracy of retrieval results.
Patent Information
- Application Number
- CN202610292967.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-11
- Publication Date
- 2026-05-15
AI Technical Summary
Existing multimodal large models suffer from illusions, delayed knowledge timeliness, and insufficient coverage of long-tail knowledge in knowledge-intensive scenarios. Traditional retrieval enhancement generation techniques ignore tabular modalities, resulting in poor collaboration among multi-source heterogeneous data and making it difficult to meet the needs of practical applications.
A table-based multimodal retrieval enhancement generation method is constructed. By acquiring anchor tables, images, and text related to subqueries, a multimodal knowledge graph is built, and the table center weight is amplified. Anchor text is filtered using a hybrid relevance function and semantic similarity calculation. A table center bias factor is introduced to adjust the edge weights, and the Leiden algorithm is used to optimize the graph structure.
It achieves precise linking of cross-modal knowledge, improves the accuracy and reliability of retrieval results, meets the deep retrieval needs of knowledge-intensive scenarios, alleviates the drawbacks of multimodal large models, and improves the reliability of generation.
Smart Images
Figure CN122045451A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database and data processing technology, specifically a table-based multimodal retrieval enhancement generation method. Background Technology
[0002] Currently, artificial intelligence technology is rapidly evolving from single-text modality to multimodal general intelligence. Multimodal large models (MLLMs, such as GPT4V and Gemini) inherit the powerful semantic understanding and generation capabilities of large language models (LLMs) and further integrate visual perception capabilities, potentially becoming the core technological foundation supporting complex multimodal interactions in the real world. However, while MLLMs have shown good performance on general task leaderboards, their practical applications still suffer from drawbacks such as frequent "illusion" phenomena, delayed knowledge timeliness, and insufficient coverage of long-tail knowledge, resulting in poor reliability in knowledge-intensive scenarios.
[0003] To alleviate these problems, the industry has proposed the Retrieval Augmentation Generative Paradigm (RAG), which involves introducing external knowledge to assist model generation. However, traditional retrieval augmentation generative techniques are mostly limited to the single modality of text retrieval. Although some recent studies have incorporated visual-text joint embedding into the retrieval augmentation generative process, their retrieval granularity is mostly limited to "image-text" pairs, often neglecting the crucial table modality, making it difficult to meet the decision-making needs of multi-source heterogeneous data such as text, tables, and images in the real world.
[0004] Tables, as an important information carrier, play an irreplaceable role in real-world scenarios. For example, medical diagnosis requires combining medical record tables with medical images, while financial analysis requires linking financial statements with market news. In these scenarios, images often carry background information unrelated to the query, and text may contain stop words irrelevant to the main point. Tables, being high-density structured data compiled manually, naturally possess advantages such as low noise, strong logic, and clear entity relationships. Furthermore, they can serve as a core hub connecting textual and image information, enabling precise cross-modal knowledge linking.
[0005] Therefore, constructing a multimodal retrieval enhancement system centered on tables, capable of bridging the modal divide and efficiently integrating structured and unstructured knowledge, can not only improve the efficiency of multimodal retrieval fusion but also enhance the accuracy of the generated data. Summary of the Invention
[0006] The purpose of this invention is to address the shortcomings of existing technologies and provide a table-based multimodal retrieval enhancement generation method.
[0007] This invention is achieved through the following technical solution: A table-based multimodal retrieval enhancement generation method includes the following steps: S1. Obtain the subquery or the user's subquery, filter the table with the highest relevance to the subquery from the table database based on the subquery, and serialize the table to obtain the anchor table set; S2. Filter the anchor point image set from the image library based on the anchor point table set, and filter the anchor point text set from the text library based on the anchor point table set; S3. Using the anchor table set as the center, construct the modal association boundary between the anchor table set, the anchor image set, and the anchor text set to obtain a multimodal knowledge graph; S4. Increase the weight of the central position of the multimodal knowledge graph and output the search results. Then, build a query generator based on the adjusted multimodal knowledge graph. The query generator can generate a new round of sub-queries. Determine whether the user terminates the search. If they do, end the process; otherwise, re-execute step S1 with a new round of sub-queries as input.
[0008] Furthermore, step S1 specifically includes: S11. Obtain the subquery or the user's subquery, and filter the table with the highest relevance to the subquery from the table database based on the subquery. S12. Split the table to obtain several anchor texts; S13. Filter anchor texts based on their lexical and semantic relevance, and form an anchor table set based on the filtered anchor texts.
[0009] Furthermore, in step S13, the lexical relevance and semantic relevance of the anchor text are determined by a hybrid relevance function to filter the anchor text. The hybrid relevance function is as follows: ; In the formula, For tables; For subqueries; () represents operations on text sequences; (·,·) is used to estimate the lexical relevance between two terms; (·,·) is used to calculate the cosine similarity between two vectors; For pre-trained dense embedding functions; It is a balancing factor. .
[0010] Furthermore, the process between step S1 and step S2 also includes: The importance of each row in the anchor table set relative to the subquery is calculated, and a number of rows and columns that are higher than the first threshold are selected. An enhanced query vector is generated from the selected rows and columns. The anchor table set improves the selection accuracy of the image library and text library based on the enhanced query vector.
[0011] Furthermore, the importance of each row in the anchor table relative to the subquery is calculated using the semantic similarity formula: ; In the formula, The text in the i-th row of the anchor table set.
[0012] Furthermore, after selecting several rows and columns that exceed the first threshold, these are grouped into a core row set. An enhanced query vector is then generated from this core row set. The calculation formula is: ; In the formula, For the core row set, The number of core rows; As a weighting factor, .
[0013] Furthermore, step S3 specifically includes: S31. Extract table entity sets from anchor table sets, extract text entity sets from anchor text sets, and extract image entity sets from anchor image sets; S32. Establish modal association edges based on table entities and non-table entities to obtain a multimodal knowledge graph.
[0014] Furthermore, the modal correlation edge formula for: ; In the formula, It is a table entity; It is a non-table entity; For table entity sets; A set of text entities; For image entity set; The threshold is set manually. ; Furthermore, in step S4, By introducing a table center bias factor to adjust the weights of the center positions in a multimodal knowledge graph, the adjusted edge weights are obtained. for: ; In the formula, These are the original edge weights; This is the table center bias factor; It is an indicator function, if the node or Belongs to the table entity set If the condition is met, return 1; otherwise, return 0.
[0015] Furthermore, In step S4, after adjusting the weights of the center positions of the multimodal knowledge graph, the central modularity of the multimodal knowledge graph is calculated, and the Leiden algorithm is used to maximize the central modularity of the multimodal knowledge graph and perform clustering.
[0016] The formula for calculating the central modularity is: ; In the formula, It is a node Effective weighting; It is the sum of the weights of all valid edges; It is the Kronecker function.
[0017] The advantages and beneficial effects of this invention are as follows: This method fully utilizes the advantages of table-based structure and low noise, linking text and image information to achieve accurate cross-modal knowledge linkage, solving the problems of traditional multimodal retrieval neglecting table modalities and poor synergy of multi-source heterogeneous data. It constructs a multimodal knowledge graph and amplifies the central weight, improving the priority and accuracy of table-related retrieval results. Simultaneously, it uses a query generator to achieve iterative retrieval, continuously optimizing retrieval results to meet the deep retrieval needs of knowledge-intensive scenarios. It effectively alleviates the drawbacks of the "illusion" of large multimodal models, the lag in knowledge timeliness, and insufficient coverage of long-tail knowledge, improving the reliability of multimodal retrieval enhancement generation. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart of the table-based multimodal retrieval enhancement generation method of the present invention.
[0020] Figure 2This is a model diagram of the table-based multimodal retrieval enhancement generation method of the present invention. Detailed Implementation
[0021] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described below with reference to specific embodiments.
[0022] This embodiment provides a table-based multimodal retrieval enhancement generation method, such as... Figure 1 and Figure 2 As shown, it includes the following steps: S1. Obtain the subquery or the user's subquery, filter the table with the highest relevance to the subquery from the table database based on the subquery, and serialize the table to obtain the anchor table set; S2. Filter the anchor point image set from the image library based on the anchor point table set, and filter the anchor point text set from the text library based on the anchor point table set; S3. Using the anchor table set as the center, construct the modal association boundary between the anchor table set, the anchor image set, and the anchor text set to obtain a multimodal knowledge graph; S4. Increase the weight of the central position of the multimodal knowledge graph and output the search results. Then, build a query generator based on the adjusted multimodal knowledge graph. The query generator can generate a new round of sub-queries. Determine whether the user terminates the search. If they do, end the process; otherwise, re-execute step S1 with a new round of sub-queries as input.
[0023] Understandably, this method fully leverages the advantages of table-based structure and low noise, linking text and image information to achieve precise cross-modal knowledge linkage, addressing the problems of traditional multimodal retrieval neglecting table modalities and poor synergy among multi-source heterogeneous data. It constructs a multimodal knowledge graph and amplifies the central weights to improve the priority and accuracy of table-related retrieval results. Simultaneously, it uses a query generator to achieve iterative retrieval, continuously optimizing search results to meet the deep retrieval needs of knowledge-intensive scenarios. This effectively alleviates the drawbacks of the "illusion" of large multimodal models, the lag in knowledge timeliness, and insufficient coverage of long-tail knowledge, improving the reliability of multimodal retrieval enhancement generation.
[0024] Furthermore, such as Figure 1 and Figure 2 As shown, step S1 specifically includes: S11. Obtain the subquery or the user's subquery, and filter the table with the highest relevance to the subquery from the table database based on the subquery. S12. Split the table to obtain several anchor texts; S13. Filter anchor texts based on their lexical and semantic relevance, and form an anchor table set based on the filtered anchor texts.
[0025] Understandably, splitting the table and then filtering anchor texts based on lexical and semantic relevance to form an anchor table set enables refined filtering of the table data, eliminating table content irrelevant to the subquery, avoiding redundant data interference, improving the matching degree between the anchor table set and the subquery, and laying a precise foundation for subsequent cross-modal filtering.
[0026] Furthermore, such as Figure 1 and Figure 2 As shown, in step S13, the lexical relevance and semantic relevance of the anchor text are determined by a hybrid relevance function to filter the anchor text. The hybrid relevance function is as follows: ; In the formula, For tables; For subqueries; () represents operations on text sequences; (·,·) is used to estimate the lexical relevance between two terms; (·,·) is used to calculate the cosine similarity between two vectors; For pre-trained dense embedding functions; It is a balancing factor. .
[0027] Understandably, by using a hybrid relevance function to integrate BM25 lexical relevance and cosine similarity semantic relevance, the limitations of single relevance judgment can be addressed by taking into account both literal matching and deep semantic understanding. The introduction of a balance factor α allows for flexible adjustment of the weights of both factors according to different retrieval scenarios, adapting to different types of subqueries such as textual and semantic queries, and further improving the accuracy and scenario adaptability of anchor text filtering.
[0028] Furthermore, such as Figure 1 and Figure 2 As shown, the process between step S1 and step S2 also includes: The importance of each row in the anchor table set relative to the subquery is calculated, and a number of rows and columns that are higher than the first threshold are selected. An enhanced query vector is generated from the selected rows and columns. The anchor table set improves the selection accuracy of the image library and text library based on the enhanced query vector.
[0029] Understandably, calculating the importance of table rows and generating enhanced query vectors before cross-modal filtering allows the filtering of image and text libraries to be based on the core information of the table, rather than simply the original subqueries. This effectively improves the accuracy of cross-modal filtering, avoids image and text filtering biases caused by insufficient original query information, and reduces the amount of irrelevant cross-modal data retrieved.
[0030] Furthermore, the importance of each row in the anchor table relative to the subquery is calculated using the semantic similarity formula: ; In the formula, The text in the i-th row of the anchor table set.
[0031] It is understandable that by calculating the semantic similarity between table rows and subqueries using cosine similarity, importance can be determined, accurately quantifying the relevance of each row's content to the retrieval requirements. This provides an objective and quantifiable criterion for selecting core table rows, ensuring that the selected rows and columns are all core information highly relevant to the subqueries.
[0032] Furthermore, after selecting several rows and columns that exceed the first threshold, these are grouped into a core row set. An enhanced query vector is then generated from this core row set. The calculation formula is: ; In the formula, For the core row set, The number of core rows; As a weighting factor, .
[0033] Understandably, by generating an enhanced query vector that integrates the original subquery and the core information of the table through the core row set, the introduction of the weight factor β can flexibly balance the proportion of the original query intent and the table-derived information, thus preserving the user's core search needs while enriching the query dimensions with the structured information of the table, making the basis for cross-modal filtering more comprehensive and accurate.
[0034] Furthermore, such as Figure 1 and Figure 2 As shown, step S3 specifically includes: S31. Extract table entity sets from anchor table sets, extract text entity sets from anchor text sets, and extract image entity sets from anchor image sets; S32. Establish modal association edges based on table entities and non-table entities to obtain a multimodal knowledge graph.
[0035] Understandably, by extracting entity sets from each modality and dividing them into tabular and non-tabular entities to construct modal association boundaries, the nodes and relationships of the multimodal knowledge graph become clearer, the modal association logic between tables, text, and images is clarified, and the orderly integration of structured tabular knowledge with unstructured text and image knowledge is achieved, thus avoiding the chaos of multimodal knowledge fusion.
[0036] Furthermore, such as Figure 1 and Figure 2 As shown, the modal correlation boundary formula for: ; In the formula, It is a table entity; It is a non-table entity; For table entity sets; A set of text entities; For image entity set; The threshold is set manually. ; Understandably, by using the modal association boundary formula to quantify the association between table entities and text and image entities, only association pairs with similarity higher than the threshold τ are retained, while weak or irrelevant modal connections are eliminated. This makes the association relationships in the multimodal knowledge graph more accurate, reduces noisy connections in the graph, and improves the effectiveness of cross-modal knowledge association.
[0037] Furthermore, in step S4, By introducing a table center bias factor to adjust the weights of the center positions in a multimodal knowledge graph, the adjusted edge weights are obtained. for: ; In the formula, These are the original edge weights; This is the table center bias factor; It is an indicator function, if the node or Belongs to the table entity set If the condition is met, return 1; otherwise, return 0.
[0038] Understandably, introducing a table center bias factor to adjust the edge weights of the knowledge graph can specifically increase the weight of table entity-related connections, strengthen the core hub status of the table in the multimodal knowledge graph, make the search results more focused on multimodal information highly related to the table, avoid non-core modal information from obscuring key search results, and improve the targeting of search results.
[0039] Furthermore, in step S4, after adjusting the weights of the center positions of the multimodal knowledge graph, the central modularity of the multimodal knowledge graph is calculated, and the Leiden algorithm is used to maximize the central modularity of the multimodal knowledge graph and cluster it.
[0040] The formula for calculating the central modularity is: ; In the formula, It is a node Effective weighting; It is the sum of the weights of all valid edges; It is the Kronecker function.
[0041] Understandably, by calculating the central modularity and combining it with the Leiden algorithm to cluster the adjusted knowledge graph, accurate clustering of multimodal knowledge can be achieved, allowing highly correlated multimodal knowledge to form independent modules, which facilitates the rapid location of core retrieval modules. Maximizing the central modularity can improve the compactness of modules within the graph and the distinguishability between modules, further optimizing the organization of retrieval results and improving the readability and utilization efficiency of retrieval results.
[0042] Specifically, for step S4, a large language model can be used in conjunction with the Leiden algorithm to generate a hierarchical community structure. The system aggregates community summaries at different levels and generates the next round of subqueries based on the aggregated summaries, forming a closed-loop iterative process until all information for solving the problem is collected.
[0043] Specifically, for each community Using an LLM-driven summary generator Generate summary : ; in These are parameters of the LLM; Next, all current community summaries are compiled to form the current iteration's dynamic multimodal knowledge graph summary. ; Then, based on the initial query Current subquery and aggregated summary Input query generator Generate the next round of subqueries. : ; Next, determine whether the user terminates the search. If they do, the process ends; otherwise, iterate with a new round of subqueries as input.
[0044] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A table-based multimodal retrieval enhancement generation method, characterized in that, Includes the following steps: S1. Obtain the subquery or the user's subquery, filter the table with the highest relevance to the subquery from the table database based on the subquery, and serialize the table to obtain the anchor table set; S2. Filter the anchor point image set from the image library based on the anchor point table set, and filter the anchor point text set from the text library based on the anchor point table set; S3. Using the anchor table set as the center, construct the modal association boundary between the anchor table set, the anchor image set, and the anchor text set to obtain a multimodal knowledge graph; S4. Increase the weight of the central position of the multimodal knowledge graph and output the search results. Then, build a query generator based on the adjusted multimodal knowledge graph. The query generator can generate a new round of sub-queries. Determine whether the user terminates the search. If they do, end the process; otherwise, re-execute step S1 with a new round of sub-queries as input.
2. The table-based multimodal retrieval enhancement generation method according to claim 1, characterized in that, Step S1 specifically includes: S11. Obtain the subquery or the user's subquery, and filter the table with the highest relevance to the subquery from the table database based on the subquery. S12. Split the table to obtain several anchor texts; S13. Filter anchor texts based on their lexical and semantic relevance, and form an anchor table set based on the filtered anchor texts.
3. The table-based multimodal retrieval enhancement generation method according to claim 2, characterized in that, In step S13, the lexical relevance and semantic relevance of the anchor text are determined by a hybrid relevance function to filter the anchor text. The hybrid relevance function is as follows: In the formula, For tables; For subqueries; () represents operations on text sequences; (·,·) is used to estimate the lexical relevance between two terms; (·,·) is used to calculate the cosine similarity between two vectors; For pre-trained dense embedding functions; It is a balancing factor. .
4. The table-based multimodal retrieval enhancement generation method according to claim 1, characterized in that, Between step S1 and step S2, the following is also included: The importance of each row in the anchor table set relative to the subquery is calculated, and a number of rows and columns that are higher than the first threshold are selected. An enhanced query vector is generated from the selected rows and columns. The anchor table set improves the selection accuracy of the image library and text library based on the enhanced query vector.
5. The table-based multimodal retrieval enhancement generation method according to claim 4, characterized in that, The importance of each row in the anchor table relative to the subquery is calculated using the semantic similarity formula: In the formula, The text in the i-th row of the anchor table set.
6. The table-based multimodal retrieval enhancement generation method according to claim 5, characterized in that, After selecting a number of rows and columns that exceed the first threshold, these are grouped into a core row set. An enhanced query vector is then generated from this core row set. The calculation formula is: In the formula, For the core row set, The number of core rows; As a weighting factor, .
7. The table-based multimodal retrieval enhancement generation method according to claim 1, characterized in that, Step S3 specifically includes: S31. Extract table entity sets from anchor table sets, extract text entity sets from anchor text sets, and extract image entity sets from anchor image sets; S32. Establish modal association edges based on table entities and non-table entities to obtain a multimodal knowledge graph.
8. The table-based multimodal retrieval enhancement generation method according to claim 7, characterized in that, Modal correlation boundary formula for: In the formula, It is a table entity; It is a non-table entity; For table entity sets; A set of text entities; For image entity set; The threshold is set manually. .
9. The table-based multimodal retrieval enhancement generation method according to claim 1, characterized in that, In step S4, By introducing a table center bias factor to adjust the weights of the center positions in a multimodal knowledge graph, the adjusted edge weights are obtained. for: In the formula, These are the original edge weights; This is the table center bias factor; It is an indicator function, if the node or Belongs to the table entity set If the condition is met, return 1; otherwise, return 0.
10. The table-based multimodal retrieval enhancement generation method according to any one of claims 1 or 9, characterized in that, In step S4, after adjusting the weights of the center positions of the multimodal knowledge graph, the central modularity of the multimodal knowledge graph is calculated, and the Leiden algorithm is used to maximize the central modularity of the multimodal knowledge graph and perform clustering. The formula for calculating the central modularity is: In the formula, It is a node Effective weighting; It is the sum of the weights of all valid edges; It is the Kronecker function.