A dynamic enhanced multi-modal hypergraph retrieval enhancement generation method and system
By performing cross-modal coding and dynamic incremental hypergraph construction on multi-source heterogeneous data, the problems of semantic fragmentation and insufficient expression of high-order relations in cross-document scenarios in existing technologies are solved, and efficient and stable retrieval and generation of multimodal information are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA TOWER CO LTD
- Filing Date
- 2026-04-29
- Publication Date
- 2026-07-10
AI Technical Summary
Existing technologies suffer from problems such as semantic fragmentation, insufficient ability to express higher-order relations, difficulty in real-time updating of static indexes, amplification of errors caused by the judgment bias of a single large language model, and limited ability to process multimodal information in cross-document, multi-knowledge source and complex reasoning scenarios.
By performing cross-modal encoding on multi-source heterogeneous data, a cross-modal vector index and a candidate topic/entity set are generated. Knowledge drift detection is performed, triggering incremental updates of local subgraphs. Multi-expert voting alignment verification is adopted to achieve dynamic incremental hypergraph construction. Coarse and fine searches are performed in the two-stage retrieval process to generate multimodal context.
It improves the timeliness and stability of retrieval, enhances the ability to model high-order relationships of multimodal knowledge, reduces the risk of error propagation, improves the completeness and interpretability of retrieval results, and supports the unified representation and associated retrieval of non-textual information such as images and tables.
Smart Images
Figure CN122364527A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of information retrieval and knowledge engineering technology, and specifically relates to a dynamically enhanced multimodal hypergraph retrieval enhancement generation method and system. Background Technology
[0002] With the rapid development of large language model technology, Retrieval-Augmented Generation (RAG) has gradually become an important technical approach in the fields of complex knowledge question answering, intelligent retrieval, and decision support. However, when facing cross-document, multi-knowledge-source, and complex reasoning scenarios, traditional vector similarity-based RAG methods still suffer from problems such as semantic fragmentation and insufficient ability to express higher-order relations.
[0003] To address these issues, improved solutions such as GraphRAG and Cog-RAG have emerged in recent years. These solutions introduce knowledge graphs, dual hypergraph structures, and two-stage retrieval mechanisms to structurally model text fragments, topic concepts, and entity relationships, attempting to simulate the coarse-to-fine cognitive reasoning process of humans, thereby enhancing knowledge organization and reasoning abilities in complex problems. These methods have, to some extent, enhanced the ability to model semantic associations across text fragments and improved reasoning performance in complex query scenarios.
[0004] However, existing technologies still have significant shortcomings. First, existing solutions mostly rely on pre-built hypergraphs or knowledge graph structures, and their indexing systems are usually statically generated, making it difficult to adjust in real time as the knowledge base continues to evolve or content is frequently updated, resulting in untimely search results. Second, key steps such as topic-entity alignment and node summarization heavily depend on the initial judgment of a single large language model. If there are deviations in the initial extraction or alignment results, the errors can easily be amplified in subsequent retrieval and generation processes, affecting the overall system stability. Third, existing methods are mainly geared towards textual knowledge construction, and their ability to process non-textual information such as images and tables is limited. They lack a unified multimodal semantic representation and high-order association modeling mechanism, making it difficult to fully explore the potential relationships between multimodal knowledge. Summary of the Invention
[0005] To address the aforementioned issues, this application provides a dynamically enhanced multimodal hypergraph retrieval enhancement generation method, comprising: In the offline phase, cross-modal encoding is performed on multi-source heterogeneous data to generate cross-modal vector indexes and candidate topic / entity sets; Knowledge drift detection is performed on the cross-modal vector index and candidate topic / entity set. If drift is detected, the incremental hypergraph update of the local subgraph is triggered. After alignment and verification by multi-expert voting, the data is written into the multimodal dual hypergraph index library. During the online phase, semantic parsing is performed on user requests to extract topic words and entity words; Based on the subject terms and entity terms, coarse and fine searches are performed on the multimodal dual hypergraph index library to extract evidence subgraphs and obtain multimodal context. The user request and multimodal context are input into the multimodal large language model to generate an answer and provide feedback to the user.
[0006] Optionally, the step of performing cross-modal encoding on multi-source heterogeneous data to generate a cross-modal vector index and a candidate topic / entity set includes: The parsing algorithm is used to standardize multi-source heterogeneous data, performing noise reduction, deduplication, and metadata completion operations. The cleaned multi-source heterogeneous data is semantically segmented to generate standardized intermediate data units, which include semantic text blocks, image regions of interest, and table cell rows. Intermediate data units are input into the cross-modal encoder and mapped to a unified high-dimensional vector space to generate cross-modal vector indices and candidate topic / entity sets.
[0007] Optionally, the knowledge drift detection of the cross-modal vector index and the candidate topic / entity set includes: The cross-modal vector index and candidate topic / entity set are compared with the knowledge features in the multimodal dual hypergraph index library. When the proportion of newly added topic / entity candidates to the total data exceeds the threshold θ1, or the distribution difference index between the newly added data vector and the historical index vector exceeds the threshold θ2, it is determined to be a drift. The distribution difference indicators include mean drift, cluster center changes, or similarity distribution shifts.
[0008] Optionally, the incremental hypergraph update that triggers the local subgraph, after being aligned and verified by multi-expert voting, is written into the multimodal dual hypergraph index library, including: Using entities / keywords in drifting data as anchor points, the most semantically similar associated subgraph regions are retrieved and located in the multimodal hypergraph index library; Within the locked associated subgraph region, calculate the distance between the new entity node and the existing topic node in the vector space to determine the optimal attribution relationship; A hyperedge is constructed based on the attribution relationship, which connects the new entity node, the corresponding topic node, and the original multimodal evidence supporting the entity-topic attribution relationship; After alignment verification through multi-expert voting, the hyperedges are merged into the corresponding local subgraphs of the multimodal dual hypergraph index library, completing the incremental update of the hypergraph topology and obtaining the updated multimodal dual hypergraph index library.
[0009] Optionally, the multi-expert voting alignment verification includes: Input the entity-topic-hyperedge relationship into three expert models simultaneously; Semantic experts use large language models to determine the semantic coherence between entities and topics; The stability of hyperedges is determined by structural experts using graph topology algorithms. The confidence level of the attribution relationship is determined by statistical experts based on the co-occurrence frequency; A weighted voting system is used to merge the voting results of three experts to generate a comprehensive confidence score. When the comprehensive confidence score is lower than the threshold θ3, a feedback retry mechanism is triggered to perform knowledge drift detection within the constraint of not exceeding the preset number of retries. When the comprehensive confidence score is higher than the threshold θ3, the entity-topic-hyperedge relationship is determined as valid knowledge and written into the multimodal dual hypergraph index library.
[0010] Optionally, the step of performing coarse and fine searches based on subject terms and entity terms using a multimodal dual-hypergraph index to extract evidence subgraphs and obtain multimodal context includes: Using keywords as input, a coarse search is performed based on the topic hypergraph in the multimodal dual hypergraph index library. After multi-expert voting alignment verification, a verified topic-entity alignment set is output. Using the validated topic-entity alignment set and entity words as input, a fine search is performed based on the entity hypergraph in the multimodal dual hypergraph index library to extract the evidence subgraph and obtain the multimodal context.
[0011] Optionally, the step of performing a coarse search based on the topic hypergraph in the multimodal dual hypergraph index library, using topic terms as input, and outputting the verified topic range after multi-expert voting alignment verification, includes: Using topic words as input, the topic hypergraph portion of the multimodal dual hypergraph index library is accessed, the semantic similarity between the topic words and topic nodes in the library is calculated, and matching candidate topic nodes are selected based on the semantic similarity. Then, relevant topic subgraphs are recalled based on the candidate topic nodes. The candidate topics and associated entity sets of the relevant topic subgraphs are jointly verified. The voting results of three experts are merged through weighted voting to generate a comprehensive confidence score. Candidate relationships with a comprehensive confidence score lower than θ3 are eliminated, and the verified topic-entity alignment set is output.
[0012] Optionally, the step involves taking the verified topic-entity alignment set and entity words as input, performing a fine search based on the entity hypergraph in the multimodal dual hypergraph index, extracting the evidence subgraph, and obtaining the multimodal context, including: Using the verified topic-entity alignment set and entity words as input, the entity hypergraph in the multimodal dual hypergraph index library is accessed. The specific cross-modal hyperedge is located through entity linking and cross-modal matching. The heterogeneous data of text, images and tables contained in the cross-modal hyperedge are extracted. The located cross-modal hyperedge, the nodes connected to the hyperedge, and the extracted heterogeneous data are assembled into a structured evidence subgraph. The heterogeneous data in the structured evidence subgraph are integrated, and the text paragraphs, image evidence, and data tables contained in the heterogeneous data are sorted and deduplicated according to logical relevance to form a structured multimodal context.
[0013] Based on the same inventive concept, this application also provides a dynamically enhanced multimodal hypergraph retrieval enhancement generation system, comprising: The data processing unit is used in the offline phase to perform cross-modal encoding on multi-source heterogeneous data, and generate cross-modal vector indexes and candidate topic / entity sets; The incremental update unit is used to perform knowledge drift detection on the cross-modal vector index and candidate topic / entity set. If drift is detected, the incremental hypergraph update of the local subgraph is triggered. After alignment and verification by multi-expert voting, it is written into the multimodal dual hypergraph index library. The query parsing unit is used in the online phase to perform semantic parsing on user requests and extract subject terms and entity terms; The two-stage retrieval unit is used to perform coarse and fine retrieval based on the multimodal dual hypergraph index library according to the subject terms and entity terms, extract the evidence subgraph, and obtain the multimodal context; The answer generation unit is used to take user requests and multimodal context as input to the multimodal large language model, generate answers, and provide feedback to the user.
[0014] Furthermore, this application also provides a computing device, comprising: at least one processor and a memory; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, a dynamically enhanced multimodal hypergraph retrieval enhancement generation method as described above is implemented.
[0015] In another aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the dynamically enhanced multimodal hypergraph retrieval enhancement generation method as described above.
[0016] Compared with the prior art, this application has the following advantages: This application provides a dynamically enhanced multimodal hypergraph retrieval enhancement method and system, including: an offline stage, performing cross-modal encoding on multi-source heterogeneous data to generate a cross-modal vector index and a candidate topic / entity set; performing knowledge drift detection on the cross-modal vector index and the candidate topic / entity set, and if drift is detected, triggering incremental hypergraph updates of the local subgraph, and writing it into a multimodal dual hypergraph index library after alignment verification through multi-expert voting; an online stage, performing semantic parsing on user requests to extract topic words and entity words; performing coarse and fine searches based on topic words and entity words using the multimodal dual hypergraph index library to extract evidence subgraphs and obtain multimodal context; and inputting the user request and multimodal context into a multimodal large language model to generate an answer and provide feedback to the user.
[0017] By introducing knowledge drift detection and dynamic incremental hypergraph construction mechanisms, the timeliness of retrieval and the availability of new knowledge are significantly improved, solving the problem that pre-built dual hypergraphs are difficult to adapt to updates in real time; The alignment verification framework using multi-expert voting significantly reduces error propagation and inference chain pollution caused by single LLM extraction bias, thereby improving the stability and reliability of retrieval enhancement generation. By explicitly modeling higher-order relationships of "one-to-many / many-to-many" using a hypergraph structure, and by progressively narrowing the search space through "topic-level coarse search → hyperedge / entity fine search" in the two-stage retrieval process, the problem of traditional RAG's cross-fragment semantic fragmentation and difficulty in expressing higher-order relationships is alleviated, thereby improving the completeness and interpretability of complex knowledge reasoning. Cross-modal hyperedge encoding and cross-modal hyperedge implementation enable semantic alignment and association retrieval of non-textual information in hypergraphs, overcoming the problem of insufficient support for modalities such as images and tables in existing schemes; The incremental update mechanism reduces the cost of offline construction and maintenance, and provides more stable and usable engineering capabilities for application scenarios that require dynamic knowledge and multimodal evidence, such as intelligent customer service and business decision support.
[0018] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This paper illustrates a flowchart of the dynamically enhanced multimodal hypergraph retrieval enhancement generation method provided in this application. Figure 2 The flowchart of cross-modal parsing segmentation and unified coding provided in this application is shown; Figure 3 This application illustrates the dynamic incremental hypergraph construction process. Figure 4 This application illustrates the multi-expert voting alignment framework provided. Figure 5 This application provides a flowchart of the two-stage retrieval and evidence subgraph acquisition process. Figure 6 The diagram shows the structure of the dynamically enhanced multimodal hypergraph retrieval enhancement generation system provided in this application. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0022] Example 1 This application provides a dynamically enhanced multimodal hypergraph retrieval enhancement generation method, such as... Figure 1 ,include: In the offline phase, cross-modal encoding is performed on multi-source heterogeneous data to generate cross-modal vector indexes and candidate topic / entity sets; Knowledge drift detection is performed on the cross-modal vector index and candidate topic / entity set. If drift is detected, the incremental hypergraph update of the local subgraph is triggered. After alignment and verification by multi-expert voting, the data is written into the multimodal dual hypergraph index library. During the online phase, semantic parsing is performed on user requests to extract topic words and entity words; Based on the subject terms and entity terms, coarse and fine searches are performed on the multimodal dual hypergraph index library to extract evidence subgraphs and obtain multimodal context. The user request and multimodal context are input into the multimodal large language model to generate an answer and provide feedback to the user.
[0023] Optionally, the multi-source heterogeneous data includes unstructured text (such as documents and papers), image data (such as illustrations and scanned documents), tabular data, and some structured data.
[0024] Optionally, the step of performing cross-modal encoding on multi-source heterogeneous data to generate a cross-modal vector index and a candidate topic / entity set, such as... Figure 2 ,include: Parsing and cleaning: Standardize multi-source heterogeneous data using parsing algorithms, and perform noise reduction, deduplication, and metadata (such as permissions and timestamps) completion operations; Multi-granularity segmentation: The cleaned multi-source heterogeneous data is segmented at the semantic level to generate standardized intermediate data units, which include semantic text blocks, image regions of interest, and table cell rows; Cross-modal unified encoding: The intermediate data units are input into the cross-modal encoder and mapped to a unified high-dimensional vector space to generate cross-modal vector indices and candidate topic / entity sets.
[0025] Optionally, the knowledge drift detection of the cross-modal vector index and the candidate topic / entity set includes: The cross-modal vector index and candidate topic / entity set are compared with the knowledge features in the multimodal dual hypergraph index library. When the proportion of newly added topic / entity candidates to the total data exceeds the threshold θ1, or the distribution difference index between the newly added data vector and the historical index vector exceeds the threshold θ2, it is determined to be a drift. If any of the above conditions are not met, it is determined that there is no change, and the index preservation operation is performed; The distribution difference indicators include mean drift, cluster center changes, or similarity distribution shifts.
[0026] Optionally, the incremental hypergraph update that triggers the local subgraph is written into the multimodal dual hypergraph index library after being aligned and verified by multi-expert voting, such as... Figure 3 ,include: Using entities / keywords in the drift data as anchor points, calculate their vector similarity with existing nodes in the multimodal hypergraph index, and retrieve and lock the Top-K associated subgraph regions with the highest vector similarity scores; Within the locked associated subgraph region, calculate the distance between the new entity node and the existing topic node in the vector space to determine the optimal attribution relationship; A hyperedge is constructed based on the attribution relationship, which connects the new entity node, the corresponding topic node, and the original multimodal evidence (such as text paragraphs or image features) that supports the entity-topic attribution relationship. After alignment verification through multi-expert voting, the hyperedges are merged into the corresponding local subgraphs of the multimodal dual hypergraph index, completing the incremental update of the hypergraph topology and updating the multimodal dual hypergraph index.
[0027] Local hypergraph updates do not involve full database reconstruction.
[0028] Optionally, the multi-expert voting alignment verification, such as Figure 4 ,include: Input the entity-topic-hyperedge relationship into three expert models simultaneously; Semantic experts use large language models to determine the semantic coherence between entities and topics; The stability of hyperedges is determined by structural experts using graph topology algorithms. The confidence level of the attribution relationship is determined by statistical experts based on the co-occurrence frequency; A weighted voting system is used to merge the voting results of three experts to generate a comprehensive confidence score. When the comprehensive confidence score is lower than the threshold θ3, a feedback retry mechanism is triggered to perform knowledge drift detection within the constraint of not exceeding the preset number of retries. When the comprehensive confidence score is higher than the threshold θ3, the entity-topic-hyperedge relationship is determined as valid knowledge and written into the multimodal dual hypergraph index library.
[0029] Optionally, the internal structure of the multimodal dual hypergraph index library includes three parts: Topic Hypergraph: Used to store high-level topic semantic relationships.
[0030] Entity Hypergraph: Used to store fine-grained entity and multimodal details.
[0031] Cross-modal feature storage: Used to store aligned cross-modal vector indices.
[0032] Optionally, semantic parsing of user requests can be performed to extract topic terms and entity terms, including: Semantic parsing of user requests is performed, and natural language processing technology is used to identify the user's core query intent. Key topic words and entity words are extracted from the user requests to generate a structured representation for retrieval.
[0033] Optionally, the step involves performing coarse and fine searches based on subject terms and entity terms using a multimodal dual-hypergraph index, extracting evidence subgraphs, and obtaining the multimodal context, such as... Figure 5 ,include: Using keywords as input, a coarse search is performed on the topic hypergraph in the multimodal dual hypergraph index library. After multi-expert voting alignment verification, the verified topic-entity alignment set is output. Using the validated topic-entity alignment set and entity words as input, a fine search is performed based on the entity hypergraph in the multimodal dual hypergraph index to extract the evidence subgraph and obtain the multimodal context.
[0034] Optionally, the step of taking topic terms as input, performing a coarse search based on the topic hypergraph in the multimodal dual hypergraph index, and outputting a verified topic-entity alignment set after multi-expert voting alignment verification includes: Using topic words as input, the topic hypergraph portion of the multimodal dual hypergraph index library is accessed, the semantic similarity between the topic words and topic nodes in the library is calculated, and matching candidate topic nodes are selected based on the semantic similarity. Then, relevant topic subgraphs are recalled based on the candidate topic nodes. The candidate topics and associated entity sets of the relevant topic subgraphs are jointly verified. The voting results of three experts are merged through weighted voting to generate a comprehensive confidence score. Candidate relationships with a comprehensive confidence score lower than θ3 are eliminated, and the verified topic-entity alignment set is output.
[0035] Optionally, the step involves taking the verified topic-entity alignment set and entity words as input, performing a fine search based on the entity hypergraph in the multimodal dual hypergraph index, extracting the evidence subgraph, and obtaining the multimodal context, including: Using the verified topic-entity alignment set and entity words as input, the entity hypergraph / hyperedge index in the multimodal hypergraph index library is accessed. The specific cross-modal hyperedge is located through entity linking and cross-modal matching. The heterogeneous data of text, images and tables contained in the cross-modal hyperedge are extracted. The located cross-modal hyperedge, the nodes connected to the hyperedge, and the extracted heterogeneous data are assembled into a structured evidence subgraph. The heterogeneous data in the structured evidence subgraph are integrated, and the text paragraphs, image evidence, and data tables contained in the heterogeneous data are sorted and deduplicated according to logical relevance to form a structured multimodal context.
[0036] The beneficial effects of this application are as follows: Significantly improved knowledge update timeliness: By introducing knowledge drift detection and dynamic incremental hypergraph construction mechanism, incremental updates of nodes and hyperedges are performed only on the affected local subgraphs, avoiding the lag and high cost of full reconstruction. This enables the index structure to quickly and adaptively update with knowledge changes, significantly improving the timeliness of retrieval and the availability of new knowledge, and solving the problem that pre-built dual hypergraphs are difficult to adapt to updates in real time.
[0037] The alignment error rate is reduced, and the cascading propagation of errors from a single model is avoided: The alignment verification framework adopts multi-expert voting, which integrates multi-dimensional information such as semantic consistency, structural consistency and statistical confidence to make decisions. It can cross-validate topic-entity alignment, entity-evidence association and hyperedge connection relationship, which significantly reduces the propagation of errors and inference chain pollution caused by single LLM extraction bias, and improves the stability and reliability of retrieval enhancement generation.
[0038] Semantic fragmentation across text fragments is alleviated, and the ability to express higher-order relationships is enhanced: Higher-order relationships of "one-to-many / many-to-many" are explicitly modeled through a hypergraph structure, and the retrieval space is narrowed layer by layer in the two-stage retrieval process by "topic-level coarse retrieval → hyperedge / entity fine retrieval". This allows information scattered in multiple text fragments, table units or image evidence to be aggregated into a structured evidence chain through hyperedges, thereby alleviating the problem of semantic fragmentation across fragments and difficulty in expressing higher-order relationships in traditional RAG, and improving the integrity and interpretability of complex knowledge reasoning.
[0039] Enhanced Multimodal Knowledge Unified Representation and Semantic Association Capabilities: This invention creates a cross-modal hyperedge encoder that maps heterogeneous knowledge such as text, images, and tables to a unified representation space. It binds entities, topics, and multimodal evidence through cross-modal hyperedges, enabling semantic alignment and association retrieval of non-textual information in the hypergraph. This overcomes the limitations of existing solutions in terms of insufficient support for modalities such as images and tables, and the difficulty in unifying representation and joint reasoning, thereby improving recall quality and evidence sufficiency in multimodal scenarios.
[0040] Improved retrieval efficiency and engineering usability: Incremental update mechanism reduces offline construction and maintenance costs, multi-expert verification mechanism reduces the pressure of manual rule tuning and error correction; online two-stage retrieval, coarse first and then fine, can effectively reduce the search space and the number of invalid candidates while ensuring retrieval accuracy, improve end-to-end question and answer response efficiency, and provide more stable and usable engineering capabilities for application scenarios that require dynamic knowledge and multimodal evidence, such as intelligent customer service and business decision support.
[0041] Example 2 Based on the same inventive concept, this application also provides a dynamically enhanced multimodal hypergraph retrieval enhancement generation system, such as... Figure 6 ,include: The data processing unit is used in the offline phase to perform cross-modal encoding on multi-source heterogeneous data, and generate cross-modal vector indexes and candidate topic / entity sets; The incremental update unit is used to perform knowledge drift detection on the cross-modal vector index and candidate topic / entity set. If drift is detected, the incremental hypergraph update of the local subgraph is triggered. After alignment and verification by multi-expert voting, it is written into the multimodal dual hypergraph index library. The query parsing unit is used in the online phase to perform semantic parsing on user requests and extract subject terms and entity terms; The two-stage retrieval unit is used to perform coarse and fine retrieval based on the multimodal dual hypergraph index library according to the subject terms and entity terms, extract the evidence subgraph, and obtain the multimodal context; The answer generation unit is used to take user requests and multimodal context as input to the multimodal large language model, generate answers, and provide feedback to the user.
[0042] Optionally, the step of performing cross-modal encoding on multi-source heterogeneous data to generate a cross-modal vector index and a candidate topic / entity set includes: The parsing algorithm is used to standardize multi-source heterogeneous data, performing noise reduction, deduplication, and metadata completion operations. The cleaned multi-source heterogeneous data is semantically segmented to generate standardized intermediate data units, which include semantic text blocks, image regions of interest, and table cell rows. Intermediate data units are input into the cross-modal encoder and mapped to a unified high-dimensional vector space to generate cross-modal vector indices and candidate topic / entity sets.
[0043] Optionally, the knowledge drift detection of the cross-modal vector index and the candidate topic / entity set includes: The cross-modal vector index and candidate topic / entity set are compared with the knowledge features in the multimodal dual hypergraph index library. When the proportion of newly added topic / entity candidates to the total data exceeds the threshold θ1, or the distribution difference index between the newly added data vector and the historical index vector exceeds the threshold θ2, it is determined to be a drift. The distribution difference indicators include mean drift, cluster center changes, or similarity distribution shifts.
[0044] Optionally, the incremental hypergraph update that triggers the local subgraph, after being aligned and verified by multi-expert voting, is written into the multimodal dual hypergraph index library, including: Using entities / keywords in drifting data as anchor points, the most semantically similar associated subgraph regions are retrieved and located in the multimodal hypergraph index library; Within the locked associated subgraph region, calculate the distance between the new entity node and the existing topic node in the vector space to determine the optimal attribution relationship; A hyperedge is constructed based on the attribution relationship, which connects the new entity node, the corresponding topic node, and the original multimodal evidence supporting the entity-topic attribution relationship; After alignment verification through multi-expert voting, the hyperedges are merged into the corresponding local subgraphs of the multimodal dual hypergraph index library, completing the incremental update of the hypergraph topology and obtaining the updated multimodal dual hypergraph index library.
[0045] Optionally, the multi-expert voting alignment verification includes: Input the entity-topic-hyperedge relationship into three expert models simultaneously; Semantic experts use large language models to determine the semantic coherence between entities and topics; The stability of hyperedges is determined by structural experts using graph topology algorithms. The confidence level of the attribution relationship is determined by statistical experts based on the co-occurrence frequency; A weighted voting system is used to merge the voting results of three experts to generate a comprehensive confidence score. When the comprehensive confidence score is lower than the threshold θ3, a feedback retry mechanism is triggered to perform knowledge drift detection within the constraint of not exceeding the preset number of retries. When the comprehensive confidence score is higher than the threshold θ3, the entity-topic-hyperedge relationship is determined as valid knowledge and written into the multimodal dual hypergraph index library.
[0046] Optionally, the step of performing coarse and fine searches based on subject terms and entity terms using a multimodal dual-hypergraph index to extract evidence subgraphs and obtain multimodal context includes: Using keywords as input, a coarse search is performed based on the topic hypergraph in the multimodal dual hypergraph index library. After multi-expert voting alignment verification, a verified topic-entity alignment set is output. Using the validated topic-entity alignment set and entity words as input, a fine search is performed based on the entity hypergraph in the multimodal dual hypergraph index library to extract the evidence subgraph and obtain the multimodal context.
[0047] Optionally, the step of performing a coarse search based on the topic hypergraph in the multimodal dual hypergraph index library, using topic terms as input, and outputting the verified topic range after multi-expert voting alignment verification, includes: Using topic words as input, the topic hypergraph portion of the multimodal dual hypergraph index library is accessed, the semantic similarity between the topic words and topic nodes in the library is calculated, and matching candidate topic nodes are selected based on the semantic similarity. Then, relevant topic subgraphs are recalled based on the candidate topic nodes. The candidate topics and associated entity sets of the relevant topic subgraphs are jointly verified. The voting results of three experts are merged through weighted voting to generate a comprehensive confidence score. Candidate relationships with a comprehensive confidence score lower than θ3 are eliminated, and the verified topic-entity alignment set is output.
[0048] Optionally, the step involves taking the verified topic-entity alignment set and entity words as input, performing a fine search based on the entity hypergraph in the multimodal dual hypergraph index, extracting the evidence subgraph, and obtaining the multimodal context, including: Using the verified topic-entity alignment set and entity words as input, the entity hypergraph in the multimodal dual hypergraph index library is accessed. The specific cross-modal hyperedge is located through entity linking and cross-modal matching. The heterogeneous data of text, images and tables contained in the cross-modal hyperedge are extracted. The located cross-modal hyperedge, the nodes connected to the hyperedge, and the extracted heterogeneous data are assembled into a structured evidence subgraph. The heterogeneous data in the structured evidence subgraph are integrated, and the text paragraphs, image evidence, and data tables contained in the heterogeneous data are sorted and deduplicated according to logical relevance to form a structured multimodal context.
[0049] Example 3 Based on the same inventive concept, this application also provides an electronic device. The electronic device of this application includes at least one processor and at least one storage medium electrically connected to the processor. The storage medium is electrically connected to the processor, wherein the storage medium stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.
[0050] Example 4 Based on the same inventive concept, this application also provides a storage medium storing instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to perform the method described above.
[0051] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A dynamically enhanced multimodal hypergraph retrieval enhancement generation method, characterized in that, include: In the offline phase, cross-modal encoding is performed on multi-source heterogeneous data to generate cross-modal vector indexes and candidate topic / entity sets; Knowledge drift detection is performed on the cross-modal vector index and candidate topic / entity set. If drift is detected, the incremental hypergraph update of the local subgraph is triggered. After alignment and verification by multi-expert voting, the data is written into the multimodal dual hypergraph index library. During the online phase, semantic parsing is performed on user requests to extract topic words and entity words; Based on the subject terms and entity terms, coarse and fine searches are performed on the multimodal dual hypergraph index library to extract evidence subgraphs and obtain multimodal context. The user request and multimodal context are input into the multimodal large language model to generate an answer and provide feedback to the user.
2. The method according to claim 1, characterized in that, The process of performing cross-modal encoding on multi-source heterogeneous data to generate cross-modal vector indexes and candidate topic / entity sets includes: The parsing algorithm is used to standardize multi-source heterogeneous data, performing noise reduction, deduplication, and metadata completion operations. The cleaned multi-source heterogeneous data is semantically segmented to generate standardized intermediate data units, which include semantic text blocks, image regions of interest, and table cell rows. Intermediate data units are input into the cross-modal encoder and mapped to a unified high-dimensional vector space to generate cross-modal vector indices and candidate topic / entity sets.
3. The method according to claim 1, characterized in that, The knowledge drift detection of cross-modal vector indexes and candidate topic / entity sets includes: The cross-modal vector index and candidate topic / entity set are compared with the knowledge features in the multimodal dual hypergraph index library. When the proportion of newly added topic / entity candidates to the total data exceeds the threshold θ1, or the distribution difference index between the newly added data vector and the historical index vector exceeds the threshold θ2, it is determined to be a drift. The distribution difference indicators include mean drift, cluster center changes, or similarity distribution shifts.
4. The method according to claim 1, characterized in that, The incremental hypergraph update that triggers the local subgraph is written into the multimodal dual hypergraph index library after being aligned and verified by multi-expert voting, including: Using entities / keywords in drifting data as anchor points, the most semantically similar associated subgraph regions are retrieved and located in the multimodal hypergraph index library; Within the locked associated subgraph region, calculate the distance between the new entity node and the existing topic node in the vector space to determine the optimal attribution relationship; A hyperedge is constructed based on the attribution relationship, which connects the new entity node, the corresponding topic node, and the original multimodal evidence supporting the entity-topic attribution relationship; After alignment verification through multi-expert voting, the hyperedges are merged into the corresponding local subgraphs of the multimodal dual hypergraph index library, completing the incremental update of the hypergraph topology and obtaining the updated multimodal dual hypergraph index library.
5. The method according to claim 4, characterized in that, The multi-expert voting alignment verification includes: Input the entity-topic-hyperedge relationship into three expert models simultaneously; Semantic experts use large language models to determine the semantic coherence between entities and topics; The stability of hyperedges is determined by structural experts using graph topology algorithms. The confidence level of the attribution relationship is determined by statistical experts based on the co-occurrence frequency; A weighted voting system is used to merge the voting results of three experts to generate a comprehensive confidence score. When the comprehensive confidence score is lower than the threshold θ3, a feedback retry mechanism is triggered to perform knowledge drift detection within the constraint of not exceeding the preset number of retries. When the comprehensive confidence score is higher than the threshold θ3, the entity-topic-hyperedge relationship is determined as valid knowledge and written into the multimodal dual hypergraph index library.
6. The method according to claim 1, characterized in that, The process involves performing coarse and fine searches based on the subject terms and entity terms using the multimodal dual hypergraph index, extracting evidence subgraphs, and obtaining the multimodal context, including: Using keywords as input, a coarse search is performed based on the topic hypergraph in the multimodal dual hypergraph index library. After multi-expert voting alignment verification, a verified topic-entity alignment set is output. Using the validated topic-entity alignment set and entity words as input, a fine search is performed based on the entity hypergraph in the multimodal dual hypergraph index library to extract the evidence subgraph and obtain the multimodal context.
7. The method according to claim 6, characterized in that, The process involves taking keywords as input, performing a coarse search based on the topic hypergraph in a multimodal dual hypergraph index, and then outputting the verified topic range after multi-expert voting alignment and verification. This range includes: Using topic words as input, the topic hypergraph portion of the multimodal dual hypergraph index library is accessed, the semantic similarity between the topic words and topic nodes in the library is calculated, and matching candidate topic nodes are selected based on the semantic similarity. Then, relevant topic subgraphs are recalled based on the candidate topic nodes. The candidate topics and associated entity sets of the relevant topic subgraphs are jointly verified. The voting results of three experts are merged through weighted voting to generate a comprehensive confidence score. Candidate relationships with a comprehensive confidence score lower than θ3 are eliminated, and the verified topic-entity alignment set is output.
8. The method according to claim 6, characterized in that, The process involves taking the verified topic-entity alignment set and entity words as input, performing a refined search based on the entity hypergraph in the multimodal dual hypergraph index, extracting the evidence subgraph, and obtaining the multimodal context, including: Using the verified topic-entity alignment set and entity words as input, the entity hypergraph in the multimodal dual hypergraph index library is accessed. The specific cross-modal hyperedge is located through entity linking and cross-modal matching. The heterogeneous data of text, images and tables contained in the cross-modal hyperedge are extracted. The located cross-modal hyperedge, the nodes connected to the hyperedge, and the extracted heterogeneous data are assembled into a structured evidence subgraph. The heterogeneous data in the structured evidence subgraph are integrated, and the text paragraphs, image evidence, and data tables contained in the heterogeneous data are sorted and deduplicated according to logical relevance to form a structured multimodal context.
9. A dynamically enhanced multimodal hypergraph retrieval enhancement generation system, characterized in that, include: The data processing unit is used in the offline phase to perform cross-modal encoding on multi-source heterogeneous data, and generate cross-modal vector indexes and candidate topic / entity sets; The incremental update unit is used to perform knowledge drift detection on the cross-modal vector index and candidate topic / entity set. If drift is detected, the incremental hypergraph update of the local subgraph is triggered. After alignment and verification by multi-expert voting, it is written into the multimodal dual hypergraph index library. The query parsing unit is used in the online phase to perform semantic parsing on user requests and extract subject terms and entity terms; The two-stage retrieval unit is used to perform coarse and fine retrieval based on the multimodal dual hypergraph index library according to the subject terms and entity terms, extract the evidence subgraph, and obtain the multimodal context; The answer generation unit is used to take user requests and multimodal context as input to the multimodal large language model, generate answers, and provide feedback to the user.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the dynamically enhanced multimodal hypergraph retrieval enhancement generation method according to any one of claims 1-8.
11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the dynamically enhanced multimodal hypergraph retrieval enhancement generation method according to any one of claims 1-8.