A multi-modal knowledge fusion retrieval method and system for a heterogeneous knowledge graph
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2026-03-27
- Publication Date
- 2026-08-07
AI Technical Summary
[0005]本发明的目的是针对将多模态异构文档的非结构化数据进行融合形成知识图谱，进而提升LLM模型推理过程中的感知的效率的技术问题，提出一种异构知识图谱的多模态知识融合检索方法及系统
[0041]1. The method described in this invention achieves a standardized and unified representation of multimodal data through deep fusion of visual language models and context. This method effectively solves the semantic loss problem caused by multimodal information conversion in traditional methods, preserving the original structural features and deep semantics of images, tables, and formulas, and significantly improving the understandability and usability of heterogeneous data.
Smart Images
Figure CN122527331A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a multimodal knowledge fusion retrieval method and system for heterogeneous knowledge graphs, belonging to the field of multimodal knowledge fusion retrieval technology, and applied to knowledge question answering reasoning scenarios of heterogeneous documents. Background Technology
[0002] With the acceleration of digitalization, knowledge carriers such as corporate documents, research papers, and technical manuals are exhibiting high heterogeneity and multimodal characteristics. Traditional knowledge retrieval systems are mainly based on plain text processing, returning results through keyword matching or simple semantic similarity calculations. However, in real-world applications, a large amount of key information is hidden in images, tables, mathematical formulas, and complex layouts, which cannot be effectively captured by traditional text retrieval technologies.
[0003] Current research on multimodal knowledge retrieval mainly suffers from the following problems: First, most existing methods adopt a sequential model of extraction followed by retrieval, that is, multimodal content is forcibly converted into text descriptions before unified retrieval. This approach leads to a serious loss of modal information; the spatial structure of images, the row and column logic of tables, and the symbolic semantics of formulas are flattened during the conversion process, making it difficult to restore their original semantics. Second, existing knowledge graph construction methods mainly target structured text data and lack the ability to model multimodal entities as independent nodes, resulting in a semantic disconnect between multimodal content and structured knowledge in the graph, failing to form organic knowledge associations. Finally, in the retrieval and ranking stage, existing algorithms often only consider semantic similarity, ignoring the importance of the topological structure of the knowledge graph and the differences in adaptability of different modalities to user query intent, and lack the ability to generate deep reasoning based on retrieval results.
[0004] Therefore, how to fuse unstructured data from multimodal heterogeneous documents to form a knowledge graph, thereby improving the efficiency of perception in the inference process of LLM models, has become an urgent technical problem to be solved. Summary of the Invention
[0005] The purpose of this invention is to address the technical problem of fusing unstructured data from multimodal heterogeneous documents to form a knowledge graph, thereby improving the perceptual efficiency of LLM model reasoning. This invention proposes a multimodal knowledge fusion retrieval method and system based on heterogeneous knowledge graphs. The invention constructs a heterogeneous knowledge graph containing associations between multimodal carriers and structured knowledge, combines a multidimensional perceptual retrieval mechanism based on structural importance, semantic matching degree, and modality preference weights, and generates accurate reasoning answers based on the retrieval results using a large language model.
[0006] This invention discloses a multimodal knowledge fusion retrieval method for heterogeneous knowledge graphs, comprising the following steps:
[0007] Step 1: Extract multimodal elements and contextual information using the minerU of the visual language model, and then fuse them using the prompt word template of the large language model;
[0008] Step 1.1: Use a visual language model to identify multimodal elements, including text blocks, images, tables, and formulas, in the original document converted to PDF format;
[0009] Step 1.2: Use minerU to differentially extract the contextual information of multimodal elements consisting of titles, surrounding text, page numbers, and footnotes;
[0010] Step 1.3: Use the prompt word template of the large language model to fuse multimodal elements with contextual information for semantic coherence and scene adaptability, and output it in JSON format;
[0011] Step 2: Using an end-to-end joint extraction method, the structured descriptions of multimodal elements and contextual information are extracted into entities and relations, and nodes are combined with multimodal elements to form triples of heterogeneous knowledge graphs.
[0012] Step 2.1: Perform end-to-end joint extraction of entities and relations from the structured description of multimodal elements and contextual information to prevent semantic fragmentation;
[0013] Step 2.1.1: Abandon the serial approach of first extracting entities and then matching relationships, and adopt a large language model to simultaneously identify entity boundaries and relationship types within text blocks;
[0014] Step 2.1.2: Optimize entity boundary definition and relationship classification through contextual semantic mutual verification to ensure semantic consistency between entities and relationships.
[0015] Step 2.2: Use the entities of the multimodal elements and the structured descriptions extracted from the end-to-end joint extraction as entity nodes, and the relationships extracted from the structured descriptions from the end-to-end joint extraction as edges to construct triples of the heterogeneous knowledge graph;
[0016] Step 3: The structural importance weights obtained using the personalized PageRank iterative method, the semantic matching degree obtained using cosine similarity, and the modality preference coefficient are fused together, and the top-K nodes sorted according to decreasing weights are associated with subgraphs.
[0017] Step 3.1: Use the personalized PageRank iterative method shown in Equation (1) to obtain the structural importance weights of nodes in the heterogeneous knowledge graph, which are used to simulate the random walk process of users in the graph;
[0018] (1)
[0019] in, Representative node Personalized PageRank score, S is the damping coefficient. ) represents the personalized seed vector, N( ) is a node The set of directly adjacent nodes, |N( )| for node The degree value.
[0020] Step 3.2: Use cosine similarity as shown in equation (2) to obtain the semantic matching degree between the query and the node;
[0021] (2)
[0022] in, For the feature vector queried by the user, For the first Feature vectors of each node;
[0023] Step 3.3: Determine the modality preference coefficient, analyze the user's query intent using the large model, determine the appropriate response modality, and if the node modality matches the appropriate modality, increase its weight; otherwise, decrease its weight.
[0024] Step 3.3.1: Preconstruct a modal weight mapping table to set differentiated basic weight coefficients for different types of modal nodes in the graph. The modal types include text, image, table, and formula.
[0025] Step 3.3.2: Traverse the graph nodes to be searched and identify the modality type of each node;
[0026] Step 3.3.3: Based on the identified node modality type, query the corresponding weight coefficient from the modality weight mapping table and directly assign it to the node as the modality preference coefficient. This enables differentiated weighting of nodes across different modalities, prioritizing node relevance based on problem correlation. Distinguishing between weighting and deweighting in assignment;
[0027] Step 3.4: Combine structural importance weights, semantic matching degree, and modality preference coefficients. Weight fusion is performed in the manner shown in equation (3);
[0028] (3)
[0029] in, For structural importance weights, For semantic importance weights, For modal preference coefficients, and The fusion coefficient;
[0030] Step 3.5: Perform retrieval and sorting of heterogeneous knowledge graph nodes using weight fusion, and select the Top-K nodes sorted by decreasing weight for subgraph association;
[0031] Step 4: Generate question-and-answer reasoning answers based on the search results;
[0032] Step 4.1: Construct the retrieved multimodal knowledge fragments into a prompt context;
[0033] Step 4.2: Input the large language model to generate the final answer, achieving accurate reasoning based on multimodal knowledge;
[0034] On the other hand, in order to achieve the purpose of the present invention, based on the above method, the present invention further proposes a multimodal knowledge fusion retrieval system for heterogeneous knowledge graphs, including a heterogeneous data preprocessing module, a knowledge graph construction module, a multidimensional perception retrieval module, and a question-answering reasoning generation module;
[0035] The heterogeneous data preprocessing module is used to receive raw documents in various formats, identify and separate multimodal elements using a visual language model, generate standardized semantic descriptions in combination with context, and construct a unified representation of heterogeneous data; this will serve as input to the knowledge graph construction module.
[0036] The knowledge graph construction module is used to perform joint extraction of entities and relations based on the unified representation and using a large language model, taking multimodal elements as independent entity nodes, and constructing a heterogeneous knowledge graph containing multimodal carriers and structured knowledge associations; this will serve as the input to the multidimensional perception retrieval module.
[0037] The multidimensional perception retrieval module is used to respond to user queries, calculate the structural importance weight, semantic importance weight, and modal preference coefficient of each node in the graph, fuse them to obtain the final weight of the nodes, perform retrieval ranking, and output Top-K retrieval results and their associated subgraphs;
[0038] The question-and-answer reasoning generation module is used to receive the search results, construct them into a large language model with prompting context input, and generate the final reasoning answer that matches the user's query.
[0039] A computer-readable storage device storing a computer program, characterized in that, when the computer program is executed, it implements the aforementioned multimodal knowledge fusion retrieval method and system for heterogeneous knowledge graphs.
[0040] Compared with existing technologies, it has the following beneficial effects:
[0041] 1. The method described in this invention achieves a standardized and unified representation of multimodal data through deep fusion of visual language models and context. This method effectively solves the semantic loss problem caused by multimodal information conversion in traditional methods, preserving the original structural features and deep semantics of images, tables, and formulas, and significantly improving the understandability and usability of heterogeneous data.
[0042] 2. The method described in this invention constructs a novel heterogeneous knowledge graph, innovatively incorporating multimodal carriers themselves as independent nodes into the graph and establishing semantic associations between them and structured knowledge. This graph construction method breaks down the barriers between modalities, enabling the retrieval system to directly locate specific charts or formulas, rather than merely remaining at the textual description level, greatly enhancing the richness and accuracy of knowledge associations.
[0043] 3. The method described in this invention proposes a multi-dimensional perceptual retrieval mechanism based on structure, semantics, and modality preferences. By integrating PageRank structural weights, semantic similarity weights, and preset modality differentiation weights, this method can more accurately capture users' potential needs, prioritize returning knowledge fragments with high topological coreness, semantic matching, and modality adaptation, and significantly improve retrieval recall and ranking quality in complex scenarios.
[0044] 4. The method described in this invention adds a question-answering reasoning step based on the retrieval results. By using high-precision multimodal retrieval results as context input to the large language model, it not only provides relevant information, but also integrates multi-source information to generate a logically rigorous and well-supported final answer, thus achieving a leap from retrieval to reasoning. Attached Figure Description
[0045] Figure 1 is a schematic diagram of the method flow of the present invention;
[0046] Figure 2 It is a heterogeneous knowledge graph constructed based on the method proposed in this invention; Detailed Implementation
[0047] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.
[0048] This embodiment uses the classic academic paper in the field of computer vision, "Deep Residual Learning for Image Recognition" (He et al., CVPR 2016), as an example. The user inputs the query: "How does ResNet solve the network degradation problem through residual blocks? Please provide the derivation of the core formula and show the specific performance improvement data on ImageNet." The system first performs multimodal parsing on the paper PDF; then it constructs a heterogeneous knowledge graph containing formula nodes, structure graph nodes, and experimental data nodes; next, it uses a multidimensional perception retrieval mechanism to increase the weight of the corresponding modal nodes based on the needs of formula derivation and performance data; finally, based on the retrieved core formula, structure graph, and experimental tables, a comprehensive reasoning answer is generated by the large model.
[0049] Example
[0050] like Figure 1 As shown in this embodiment, a multimodal knowledge fusion retrieval method for heterogeneous knowledge graphs includes the following steps:
[0051] Step 1: Extract multimodal elements and contextual information using the minerU of the visual language model, and then fuse them using the prompt word template of the large language model;
[0052] Step 1.1: Use a visual language model to identify multimodal elements, including text blocks, images, tables, and formulas, in the original document converted to PDF format;
[0053] Step 1.2: Use minerU to differentially extract the contextual information of multimodal elements consisting of titles, surrounding text, page numbers, and footnotes;
[0054] Step 1.3: Use the prompt word template of the large language model to fuse multimodal elements with contextual information for semantic coherence and scene adaptability, and output it in JSON format;
[0055] In this embodiment, the system reads the PDF file of "Deep Residual Learning for Image Recognition" and uses a visual language model to identify Figure 2 (residual block structure diagram) on page 3, Formula (1), and Table 2 (ImageNet validation set Top-1 error rate comparison) on page 4. For Formula (1), the model generates a standardized description based on the content of Section 3.1: "This formula defines the basic form of residual learning." ,in For input, For the residual mapping to be learned (e.g., two convolutional layers) The model utilizes shortcut connections via identity mappings to achieve element-wise addition, aiming to allow the network to fit the residuals rather than the original mappings, thereby addressing the degradation problem in deep networks. Regarding Table 2, the model concludes: "The table data shows that the Top-1 error rate of the 34-layer Plain Net is 28.54%, higher than the 27.94% of the 18-layer Plain Net, indicating degradation; while the error rate of the 34-layer ResNet drops to 25.03%, significantly better than the 27.55% of the 18-layer ResNet, demonstrating that residual learning can effectively benefit from increased depth."
[0056] Step 2: Using an end-to-end joint extraction method, the structured descriptions of multimodal elements and contextual information are extracted into entities and relations, and nodes are combined with multimodal elements to form triples of heterogeneous knowledge graphs.
[0057] Step 2.1: Perform end-to-end joint extraction of entities and relations from the structured description of multimodal elements and contextual information to prevent semantic fragmentation;
[0058] Step 2.1.1: Abandon the serial approach of first extracting entities and then matching relationships, and adopt a large language model to simultaneously identify entity boundaries and relationship types within text blocks;
[0059] Step 2.1.2: Optimize entity boundary definition and relationship classification through contextual semantic mutual verification to ensure semantic consistency between entities and relationships.
[0060] In this embodiment, the system simultaneously extracts the entities "Degradation Problem," "Identity Mapping," "ResNet-34," and "ResNet-152," as well as the relations "solve," "contain," and "superior" from text blocks in the paper's introduction and Methodology section. The model determines that there is a "solve" relationship between "ResNet-34" and "Degradation Problem," and that "ResNet-34" performs "superior" to Plain-34 on ImageNet, ensuring semantic accuracy.
[0061] Step 2.2: Use the entities of the multimodal elements and the structured descriptions extracted from the end-to-end joint extraction as entity nodes, and the relationships extracted from the structured descriptions from the end-to-end joint extraction as edges to construct triples of the heterogeneous knowledge graph;
[0062] In this embodiment, the system creates Figure 2 as an independent node Node_Fig_2 (type: Image), described as "a normal stacked layer on the left, a residual block with shortcut connections on the right, executing..." The operation involves creating a node Node_Formula_1 (type: Formula) for Formula (1) and a node Node_Table_2 (type: Table) for Table 2. Then, the following edges are established: Node_Fig_2 --[visualizes_structure_of]--> Node_Residual_Block, Node_Formula_1 --[defines_mechanism_for]--> Node_Residual_Block, Node_Table_2 --[provides_evidence_for]--> Node_ResNet_Superiority. This way, when the concept of a "residual block" is retrieved, it can be directly associated with the specific structure diagram, mathematical formula, and experimental data table.
[0063] Step 3: The structural importance weights obtained using the personalized PageRank iterative method, the semantic matching degree obtained using cosine similarity, and the modality preference coefficient are fused together, and the top-K nodes sorted according to decreasing weights are associated with subgraphs.
[0064] Step 3.1: Use the personalized PageRank iterative method shown in Equation (1) to obtain the structural importance weights of nodes in the heterogeneous knowledge graph, which are used to simulate the random walk process of users in the graph;
[0065] (1)
[0066] in, Representative node Personalized PageRank score, S is the damping coefficient. ) represents the personalized seed vector, N( ) is a node The set of directly adjacent nodes, |N( )| for node The degree value.
[0067] In this embodiment, for the keywords "residual block" and "degradation problem" in the user query, their corresponding nodes are set as the seed vector of PageRank. After iterative calculations, Node_Formula_1 (definition mechanism) and Node_Table_1 (providing evidence), located at the core of the graph, received higher structural importance weights due to their close connection with the seed nodes. .
[0068] Step 3.2: Use cosine similarity as shown in equation (2) to obtain the semantic matching degree between the query and the node;
[0069] (2)
[0070] in, For the feature vector queried by the user, For the first Feature vectors of each node;
[0071] In this embodiment, the user query "provide the derivation of the core formula" is converted into a vector. The cosine similarity is calculated between the description vector of Node_Formula_1 and the description vector of Node_Formula_1. Since the description contains "formula", The node receives a higher semantic weight for highly matched words such as "residual mapping" and "remainder mapping". .
[0072] Step 3.3: Determine the modality preference coefficient, analyze the user's query intent using the large model, determine the appropriate response modality, and if the node modality matches the appropriate modality, increase its weight; otherwise, decrease its weight.
[0073] Step 3.3.1: Preconstruct a modal weight mapping table to set differentiated basic weight coefficients for different types of modal nodes in the graph. The modal types include text, image, table, and formula.
[0074] Step 3.3.2: Traverse the graph nodes to be searched and identify the modality type of each node;
[0075] Step 3.3.3: Based on the identified node modality type, query the corresponding weight coefficient from the modality weight mapping table and directly assign it to the node as the modality preference coefficient. This enables differentiated weighting of nodes across different modalities, prioritizing node relevance based on problem correlation. Distinguishing between weighting and deweighting in assignment;
[0076] In the embodiment, the following is set For Node_Formula_1, assume its ,and Then the final weight However, for a node that only contains background text, even... Up to 0.8, because Its final score is also much lower than that of the formula node. This ensures that the search results prioritize returning the formulas and data structures that users need most.
[0077] Step 3.4: Combine structural importance weights, semantic matching degree, and modality preference coefficients. Weight fusion is performed in the manner shown in equation (3);
[0078] (3)
[0079] in, For structural importance weights, For semantic importance weights, For modal preference coefficients, and The fusion coefficient;
[0080] Step 3.5: Perform retrieval and sorting of heterogeneous knowledge graph nodes using weight fusion, and select the Top-K nodes sorted by decreasing weight for subgraph association;
[0081] Step 4: Generate question-and-answer reasoning answers based on the search results;
[0082] Step 4.1: Construct the retrieved multimodal knowledge fragments into a prompt context;
[0083] Step 4.2: Input the large language model to generate the final answer, achieving accurate reasoning based on multimodal knowledge;
[0084] In this embodiment, the system will sort the top-ranked Node_Formula_1 (containing LaTeX formulas) and its semantic interpretation: Fitting residuals Comparison Fit The nodes Node_Fig_2 (containing the structural description of Figure 2: shortcut connections perform element-wise addition without adding parameters) and Node_Table_2 (containing the specific data of Table 2: 34-layer Plain Net error rate 28.54% vs 34-layer ResNet error rate 25.03%; 152-layer ResNet error rate is further reduced) are assembled into a Prompt context.
[0085] On the other hand, in order to achieve the purpose of the present invention, based on the above method, the present invention further proposes a multimodal knowledge fusion retrieval system for heterogeneous knowledge graphs, including a heterogeneous data preprocessing module, a knowledge graph construction module, a multidimensional perception retrieval module, and a question-answering reasoning generation module;
[0086] The heterogeneous data preprocessing module is used to receive raw documents in various formats, identify and separate multimodal elements using a visual language model, generate standardized semantic descriptions in combination with context, and construct a unified representation of heterogeneous data; this will serve as input to the knowledge graph construction module.
[0087] The knowledge graph construction module is used to perform joint extraction of entities and relations based on the unified representation and using a large language model, taking multimodal elements as independent entity nodes, and constructing a heterogeneous knowledge graph containing multimodal carriers and structured knowledge associations; this will serve as the input to the multidimensional perception retrieval module.
[0088] The multidimensional perception retrieval module is used to respond to user queries, calculate the structural importance weight, semantic importance weight, and modal preference coefficient of each node in the graph, fuse them to obtain the final weight of the nodes, perform retrieval ranking, and output Top-K retrieval results and their associated subgraphs;
[0089] The question-and-answer reasoning generation module is used to receive the search results, construct them into a large language model with prompting context input, and generate the final reasoning answer that matches the user's query.
[0090] A computer-readable storage device storing a computer program, characterized in that, when the computer program is executed, it implements the aforementioned multimodal knowledge fusion retrieval method and system for heterogeneous knowledge graphs.
[0091] To further illustrate the advantages of the present invention, the following experimental data are used for explanation;
[0092] As shown in Table 1, for the heterogeneous document benchmark dataset DocBench, the multimodal knowledge fusion retrieval method of heterogeneous knowledge graph proposed in this patent is compared with various baseline methods. From the perspective of comprehensive indicators, the proposed method achieves the best performance with an overall score of 55.1. In each sub-domain, the proposed method achieves the highest or near-highest performance, especially in the fields of law (59.7) and news (56.4), which demonstrates its strong adaptability in heterogeneous document scenarios in different fields.
[0093] Table 1. Performance of the method in various domains and overall metrics of DocBench.
[0094] method Aca. Fin. Gov. Law. News. Overall VLM 39.4 38.8 53.1 52.6 52.2 45.3 LightRAG 47.6 40.1 52.8 52.5 55.2 48.4 VisRAG 30.9 35.2 29.6 37.9 20.1 31.3 ViDoRAG 30.1 60.1 44.3 40.2 42.9 43.6 RAGAnything 52.4 53.3 54.2 52.5 58.5 53.8 This article's method 53.1 52.8 56.1 59.7 56.4 55.1 (+0.7) (-7.3) (+1.9) (+7.2) (-2.1) (+1.3)
[0095] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A multimodal knowledge fusion retrieval method for heterogeneous knowledge graphs, characterized in that: Includes the following steps, Step 1: Extract multimodal elements and contextual information using the minerU of the visual language model, and then fuse them using the prompt word template of the large language model; Step 2: Using an end-to-end joint extraction method, the structured descriptions of multimodal elements and contextual information are extracted into entities and relations, and nodes are combined with multimodal elements to form triples of heterogeneous knowledge graphs. Step 3: The structural importance weights obtained using the personalized PageRank iterative method, the semantic matching degree obtained using cosine similarity, and the modality preference coefficient are fused together, and the top-K nodes sorted according to decreasing weights are associated with subgraphs. Step 3.1: Use the personalized PageRank iterative method shown in Equation (1) to obtain the structural importance weights of nodes in the heterogeneous knowledge graph, which are used to simulate the random walk process of users in the graph; （1） in, Representative node Personalized PageRank score, S is the damping coefficient. ) represents the personalized seed vector, N( ) is a node The set of directly adjacent nodes, |N( )| for node The degree value; Step 3.2: Use cosine similarity as shown in equation (2) to obtain the semantic matching degree between the query and the node; （2） in, For the feature vector queried by the user, For the first Feature vectors of each node; Step 3.3: Determine the modality preference coefficient, analyze the user's query intent using the large model, determine the appropriate response modality, and if the node modality matches the appropriate modality, increase its weight; otherwise, decrease its weight. Step 3.3.1: Preconstruct a modal weight mapping table to set differentiated basic weight coefficients for different types of modal nodes in the graph. The modal types include text, image, table, and formula. Step 3.3.2: Traverse the graph nodes to be searched and identify the modality type of each node; Step 3.3.3: Based on the identified node modality type, query the corresponding weight coefficient from the modality weight mapping table and directly assign it to the node as the modality preference coefficient. This enables differentiated weighting of nodes across different modalities, prioritizing node relevance based on problem correlation. Distinguishing between weighting and deweighting in assignment; Step 3.4: Combine structural importance weights, semantic matching degree, and modality preference coefficients. Weight fusion is performed in the manner shown in equation (3); （3） in, For structural importance weights, For semantic importance weights, For modal preference coefficients, and The fusion coefficient; Step 3.5: Perform retrieval and sorting of heterogeneous knowledge graph nodes using weight fusion, and select the Top-K nodes sorted by decreasing weight for subgraph association; Step 4: Generate question-and-answer reasoning answers based on the search results.
2. The multimodal knowledge fusion retrieval method for heterogeneous knowledge graphs as described in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Use a visual language model to identify multimodal elements, including text blocks, images, tables, and formulas, in the original document converted to PDF format; Step 1.2: Use minerU to differentially extract the contextual information of multimodal elements consisting of titles, surrounding text, page numbers, and footnotes; Step 1.3: Use the prompt word template of the large language model to fuse multimodal elements with contextual information for semantic coherence and scene adaptability, and output it in JSON format.
3. The multimodal knowledge fusion retrieval method for heterogeneous knowledge graphs as described in claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Perform end-to-end joint extraction of entities and relations from the structured description of multimodal elements and contextual information to prevent semantic fragmentation; Step 2.2: Use the entities of the multimodal elements and the structured descriptions extracted from the end-to-end joint extraction as entity nodes, and the relationships extracted from the structured descriptions from the end-to-end joint extraction as edges to construct triples of the heterogeneous knowledge graph.
4. The multimodal knowledge fusion retrieval method for heterogeneous knowledge graphs as described in claim 3, characterized in that: Step 2.1 is implemented as follows: Step 2.1.1: Abandon the serial approach of first extracting entities and then matching relationships, and adopt a large language model to simultaneously identify entity boundaries and relationship types within text blocks; Step 2.1.2: Optimize entity boundary definition and relationship classification through contextual semantic mutual verification to ensure semantic consistency between entities and relationships.
5. The multimodal knowledge fusion retrieval method for heterogeneous knowledge graphs as described in claim 1, characterized in that: Step 4 is implemented as follows: Step 4.1: Construct the retrieved multimodal knowledge fragments into a prompt context; Step 4.2: Input the large language model to generate the final answer, and realize accurate reasoning of multimodal knowledge.
6. A multimodal knowledge fusion retrieval system for heterogeneous knowledge graphs, as described in claim 1, characterized in that: It includes a heterogeneous data preprocessing module, a knowledge graph construction module, a multi-dimensional perception retrieval module, and a question-answering reasoning generation module; The heterogeneous data preprocessing module is used to receive raw documents in multiple formats, identify and separate multimodal elements using a visual language model, generate standardized semantic descriptions in combination with context, and construct a unified representation of heterogeneous data. It will be used as input to the knowledge graph construction module; The knowledge graph construction module is used to perform joint extraction of entities and relations based on the unified representation and using a large language model, and to construct a heterogeneous knowledge graph containing multimodal carriers and structured knowledge associations by treating multimodal elements as independent entity nodes. It will be used as input for the multidimensional perception retrieval module; The multidimensional perception retrieval module is used to respond to user queries, calculate the structural importance weight, semantic importance weight, and modal preference coefficient of each node in the graph, fuse them to obtain the final weight of the nodes, perform retrieval ranking, and output Top-K retrieval results and their associated subgraphs; The question-and-answer reasoning generation module is used to receive the search results, construct them into a large language model with prompting context input, and generate the final reasoning answer that matches the user's query.
7. A computer-readable storage device storing a computer program, characterized in that, When the computer program is executed, it implements the multimodal knowledge fusion retrieval method and system for heterogeneous knowledge graphs.