Universal reasoning method and system for knowledge graph based on multi-modal fusion and dynamic self-adaption
By transforming the graph structure and extracting dense feature of multi-angle angles on the knowledge graph, combined with text adaptive fusion, the shortcomings of the knowledge graph model in multi-modal fusion and cross-data set generalization are solved, and unified processing of internal and external tasks and cross-data set migration is realized, and the performance of knowledge graph inference is improved.
Patent Information
- Application Number
- CN202510567165.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-30
- Publication Date
- 2025-08-08
AI Technical Summary
The existing basic knowledge graph model has shortcomings in multimodal fusion, task universality and cross-dataset generalization, and cannot effectively handle intra- and off-map inference tasks, and lacks unified processing capabilities and robust generalization performance across datasets.
By transforming the graph structure of the knowledge graph, the task is uniformly modeled into a triple-completion form, the relationship diagram and entity diagram are constructed, multi-angle dense features are extracted using graph neural network, and the model adaptability is enhanced through text adaptive dynamic fusion, combining multi-modal information with multi-layer perceptrons to achieve dynamic adjustment of the dual-modal contribution degree and construct cross-data set migration capabilities.
The model's knowledge graph reasoning capabilities in the fields of medical care, finance, science and education have been improved, and the deep integration of multimodal information has been achieved, which has enhanced task universality and robust performance across data sets, and adapted to diversified downstream tasks.
Smart Images
Figure CN120450046A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of knowledge graph reasoning technology, and specifically relates to a general knowledge graph reasoning method and system based on multimodal fusion and dynamic adaptation. Background Art
[0002] Amid the rapid development of artificial intelligence and information processing technologies, knowledge graphs, as a structured representation of entities and their relationships, provide fundamental support for reasoning and information retrieval tasks in fields such as healthcare, finance, science, and education. With their rich entity representations and rigorous logical connections, knowledge graphs have become a core component in numerous practical applications, including classification, recommendation, knowledge retrieval, and question-answering systems.
[0003] In recent years, foundational models in natural language processing (NLP) and computer vision (CV) have made significant progress through transfer learning, demonstrating excellent performance across datasets and tasks. Inspired by this, researchers have begun working on building foundational models for knowledge graphs, aiming to achieve cross-dataset generalization and universal adaptability for diverse reasoning tasks. Knowledge graphs naturally integrate bimodal information, both structural and textual, but existing research has primarily focused on developing structural modalities, with relatively limited attention paid to textual modalities. In reality, however, fully integrating bimodal information is crucial, as the contextual knowledge provided by textual information can effectively supplement structural representations, giving the model a more comprehensive and in-depth understanding of knowledge. For example, in the medical field, textual descriptions of medical literature and clinical notes in the knowledge graph can supplement simple structural relationships, helping the model more accurately assess medication risks. In finance, the textual information from corporate annual reports contained in the knowledge graph can enhance the mining of implicit risk associations in equity networks. In science or education, the detailed content contained in the knowledge graph can enrich the logical connections between knowledge points, thereby optimizing the generation of personalized reasoning. This multimodal fusion is particularly critical for downstream applications such as commonsense reasoning and knowledge graph question answering (KGQA). The organic combination of relational knowledge and contextual knowledge has been demonstrated by numerous works, such as QAGNN and GreaseLM, to significantly improve task performance.
[0004] Furthermore, previous research has been limited to in-graph (in-KG) reasoning tasks such as knowledge graph completion (KGC), and has failed to fully address the challenges of out-of-graph (out-of-KG) reasoning tasks such as KGQA. In-graph tasks primarily reason around existing structures and information within the knowledge graph, while out-of-graph tasks require models to break through the limitations of the knowledge graph's explicit structure and handle more complex reasoning scenarios through bimodal fusion. For a foundational model in the knowledge graph field, simultaneously completing both in-graph and out-of-graph tasks is a fundamental requirement and the key to achieving wider application and higher value.
[0005] In the past, researchers have conducted extensive research to address both of these tasks. Regarding in-graph tasks, knowledge graph completion is often used to effectively evaluate the performance of research methods in this area. Recently, drawing on pre-training paradigms from the fields of natural language processing (NLP) and computer vision (CV), ULTRA (likely a machine learning framework) has achieved zero-shot transfer by identifying meta-topological types within knowledge graph structures and constructing dataset-independent entity and relationship representations. However, ULTRA only utilizes structural modality information and ignores textual modality, which to some extent limits its performance. Furthermore, recent work has also begun to explore incorporating textual information to enhance reasoning capabilities. BLP and StAR utilize language models to initialize embedding tables to improve representation learning; StATik combines node text encoding with structural information captured by message passing by jointly training a language model and a graph neural network. While these methods effectively integrate multimodal information, their reliance on fine-tuning limits their generalization to unseen datasets and other tasks. Furthermore, these approaches are limited to in-graph reasoning tasks and are unable to address tasks outside of graphs.
[0006] For off-graph tasks, the most common one is the knowledge graph question answering scenario. The core goal of this task is to link the subject entities in the query question to the knowledge graph and improve the accuracy of the answer through the relationship and context reasoning in the knowledge graph. In recent research progress, QAGNN jointly updates the language model and graph neural network layer through message passing; GreaseLM enhances multimodal fusion by aligning the graph neural network and the Transformer layer. However, the above methods only focus on off-graph reasoning tasks, while most knowledge graph completion methods are limited to on-graph reasoning. This task specificity highlights their key flaw: the lack of a general framework that can uniformly handle on-graph and off-graph reasoning.
[0007] Breaking through these limitations requires addressing three core challenges in model design: (1) alleviating the semantic differences between text and structural information to promote effective fusion; (2) balancing the contributions of both modalities to adapt to diverse task requirements, especially in off-graph reasoning tasks; and (3) maintaining an unbiased training process to achieve robust generalization across datasets and avoid biased learning of specific entities or relationships. Only in this way can the model maintain stable performance across different datasets and tasks. Therefore, it is urgent to build a knowledge graph basic model with multimodal deep fusion capabilities, the ability to uniformly handle on-graph and off-graph reasoning tasks, and strong generalization and versatility, in order to overcome the many difficulties in current knowledge graph applications and promote the efficient implementation and innovative development of knowledge graphs in a wider range of fields such as medicine, finance, science, and education. Summary of the Invention
[0008] In view of the above, the purpose of the present invention is to provide a general reasoning method and system for knowledge graphs based on multimodal fusion and dynamic adaptation. In order to effectively cope with reasoning tasks in fields such as medicine, finance, science or education, the text and structural information are deeply integrated to effectively reconcile the semantic differences between the two, and the key text information is selectively retained through dynamic text adaptive fusion to enhance the multi-task adaptation capability. In addition, through a flexible edge scoring mechanism, corresponding adjustments can be made according to different downstream tasks, which can significantly improve the model's migration ability in diversified reasoning scenarios of multi-type reasoning tasks such as knowledge graph completion and knowledge question and answer, and achieve a universal solution for efficient migration by significantly reducing domain adaptation requirements.
[0009] To achieve the above-mentioned purpose, the present invention provides the following technical solutions:
[0010] In a first aspect, an embodiment of the present invention provides a general reasoning method for a knowledge graph based on multimodal fusion and dynamic adaptation, comprising the following steps:
[0011] Perform graph structure transformation on knowledge graphs in the medical, financial, scientific, or educational fields to transform knowledge graph reasoning tasks into the same triple completion form to achieve unified modeling of task objectives;
[0012] Each relationship in the knowledge graph after graph structure conversion is used as a node, and the meta-relationship features of entity connections are used as edges to construct a relationship graph;
[0013] Each edge in the knowledge graph after graph structure conversion is scored based on its contribution to the knowledge graph reasoning task, and the knowledge graph after edge scoring is obtained as the entity graph;
[0014] Input the entity graph, relationship graph, and query question into the general reasoning model of the knowledge graph and perform reasoning according to the unified modeling task goal to obtain the predicted answer, including: encoding all nodes in the entity graph and relationship graph into text modal features, extracting dense features from the query condition perspective and the global semantic perspective based on the entity graph and relationship graph and their text modal features through a graph neural network, fusing the dense features from the two perspectives through a multi-layer perceptron to obtain the final dense feature containing text information and structural information, and performing text-adaptive dynamic fusion of the final dense feature with the text modal feature of the entity graph and inputting it into the decoder to calculate the predicted answer corresponding to the query question;
[0015] After training, the general reasoning model of knowledge graph can be applied to knowledge graph reasoning tasks in the fields of medicine, finance, science or education.
[0016] Preferably, the graph structure conversion of the knowledge graph in the medical field, the financial field, the scientific field or the educational field to convert the knowledge graph reasoning task into the same triple completion form to complete the unified modeling of the task goal includes:
[0017] For knowledge graph completion tasks in the medical, financial, scientific, or educational fields, the entity nodes and relationships of the original knowledge graph are maintained. For knowledge graph question-answering tasks in the medical, financial, scientific, or educational fields, the query question is used as a query node and the candidate answer is used as a candidate answer node to insert into the original knowledge graph, where the query node is connected to all the corresponding topic nodes in the original knowledge graph, and the candidate answer node is connected to all the entity nodes it contains.
[0018] By transforming the knowledge graph, various knowledge graph reasoning tasks including knowledge graph completion tasks and knowledge graph question-answering tasks are unified into a given knowledge graph and a query triple prefix containing only the query head entity node and relationship. The goal is to find a correct node as the tail entity node to complete the query triple prefix into a logically correct triple.
[0019] Preferably, the meta-relationship features of entity connections include:
[0020] For any knowledge graph after graph structure conversion, for relations r1 and r2, if the head entity of r1 is the tail entity of r2, r1 and r2 are connected through a head-to-tail h2t edge; if the head entity of r1 is the head entity of r2, r1 and r2 are connected through a head-to-head h2h edge; if the tail entity of r1 is the head entity of r2, r1 and r2 are connected through a tail-to-head t2h edge; if the tail entity of r1 is the tail entity of r2, r1 and r2 are connected through a tail-to-tail t2t edge. h2t, h2h, t2h and t2t are the meta-relation features of entity connections.
[0021] Preferably, the rules for edge scoring include:
[0022] For knowledge graph completion tasks, the contribution of each edge is consistent by default, that is, the scores of all edges are the same; for knowledge graph question answering tasks, the text features of the knowledge graph after graph structure conversion and the text features of the query question are passed through a bilinear layer, and then Softmax regularized to obtain the relevance score with the query question as the edge score and participate in feature calculation when the knowledge graph general reasoning model is reasoned.
[0023] Preferably, the entity graph, relationship graph and text modality features thereof are extracted from the query condition perspective and the global semantic perspective by using a graph neural network to obtain dense features from two perspectives, including:
[0024] The relationship graph and its text modal features are extracted from the query condition perspective and the global semantic perspective through a graph neural network to obtain comprehensive features from two perspectives. The entity graph and its text modal features are extracted from the query condition perspective and the global semantic perspective to obtain dense features from two perspectives. When extracting dense features at each step at each perspective, the features of each edge in the entity graph are updated by the comprehensive features extracted from the relationship graph in the previous step. Finally, dense features from the query condition perspective and dense features from the global semantic perspective are extracted.
[0025] Preferably, when feature extraction is performed on the entity graph from the perspective of query conditions, the head entity in the query triple prefix corresponding to the entity graph is initialized to an assignment vector in the node initialization stage, and other entities are initialized to all-zero vectors to participate in the message propagation calculation in the graph neural network; when feature extraction is performed on the entity graph from the perspective of global semantics, all entity nodes corresponding to the entity graph are initialized to assignment vectors in the node initialization stage.
[0026] Preferably, the step of performing text-adaptive dynamic fusion of the final dense features and the text modality features of the entity graph and inputting the fused features into a decoder to calculate a predicted answer corresponding to the query question includes:
[0027] The final dense features are dynamically fused with the text modal features of the entity graph through text adaptiveness to obtain the fused dense features. According to the query question, the fused dense features corresponding to the query head entity are used as the query input, and the fused dense features corresponding to the remaining entities except the query head entity are used as the key input. The correct probability of each candidate node is calculated through the cross-attention mechanism, and the candidate node with the highest probability is used as the predicted answer.
[0028] Preferably, the text adaptive dynamic fusion adopts a dynamic linear weighting method, which is expressed as follows:
[0029]
[0030] in, represents the dense features obtained after text adaptive dynamic fusion, x r Text modality features representing entity graphs, represents the final dense feature, and α represents the weight.
[0031] In a second aspect, an embodiment of the present invention further provides a general reasoning system for a knowledge graph based on multimodal fusion and dynamic self-adaptation, which is implemented using the general reasoning method for a knowledge graph based on multimodal fusion and dynamic self-adaptation, and includes: a task target modeling module, a relationship graph construction module, an entity graph construction module, a general model reasoning module, and a model training module;
[0032] The task goal modeling module is used to perform graph structure conversion on the knowledge graph in the medical field, the financial field, the scientific field or the educational field to convert the knowledge graph reasoning task into the same triple completion form to complete the unified modeling of the task goal;
[0033] The relationship graph construction module is used to use each relationship in the knowledge graph after graph structure conversion as a node and the meta-relationship features of entity connections as edges to construct a relationship graph;
[0034] The entity graph construction module is used to score each edge in the knowledge graph after graph structure conversion based on its contribution to the knowledge graph reasoning task, and obtain the knowledge graph after edge scoring as the entity graph;
[0035] The general model reasoning module is used to input the entity graph, relationship graph and query question into the knowledge graph general reasoning model and perform reasoning according to the unified modeling task goal to obtain a predicted answer, including: encoding all nodes in the entity graph and relationship graph into text modal features, extracting dense features from the query condition perspective and the global semantic perspective based on the entity graph and relationship graph and their text modal features through a graph neural network, fusing the dense features from the two perspectives through a multi-layer perceptron to obtain a final dense feature containing text information and structural information, and performing text-adaptive dynamic fusion of the final dense feature with the text modal feature of the entity graph and inputting it into a decoder to calculate the predicted answer corresponding to the query question;
[0036] The model training module is used to train the general reasoning model of the knowledge graph and then apply it to knowledge graph reasoning tasks in the medical field, financial field, scientific field or educational field.
[0037] In the third aspect, an embodiment of the present invention also provides an electronic device, comprising a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation when executing the computer program.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] (1) The present invention proposes a technical solution that effectively reconciles the semantic differences between structural modality and textual modality. By extracting textual modality features from entity graphs and relationship graphs and performing dense feature extraction from two perspectives through graph neural networks, and combining them with multi-layer perceptron fusion, the deep fusion and complementary enhancement of bimodal information are achieved to improve the model's contextual understanding ability in complex reasoning scenarios such as medical, financial, scientific or educational fields.
[0040] (2) The present invention constructs a unified reasoning framework that can adapt to diverse downstream tasks through unified modeling of task objectives. The framework design breaks through the limitations of task adaptation and overcomes the defects of the separation of the processing mechanism of in-graph tasks (knowledge graph completion) and out-of-graph tasks (knowledge graph question and answer). By dynamically fusing the final dense features with the text modal features of the entity graph in a text-adaptive manner, the dynamic adjustment of the dual-modal contribution is achieved, which helps to enhance the task versatility and applicability of the model.
[0041] (3) The present invention constructs a relationship graph so that a basic topological unit can be constructed in any data set. When extracting dense features in each step of model training, the features of each edge in the entity graph are updated by the comprehensive features extracted from the relationship graph in the previous step. This can adapt to the unbiased training mechanism of cross-dataset migration, solve the problem of the model's preference for learning specific entities or relationships, ensure robust performance under different knowledge graph data distributions, and further improve the model's generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 This is a flowchart of a general reasoning method for a knowledge graph based on multimodal fusion and dynamic adaptation provided by an embodiment of the present invention;
[0044] Figure 2 This is a schematic diagram of a framework of a general reasoning method for a knowledge graph based on multimodal fusion and dynamic adaptation provided by an embodiment of the present invention;
[0045] Figure 3 It is a structural diagram of a general reasoning system of knowledge graph based on multimodal fusion and dynamic adaptation provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.
[0047] The inventive concept of the present invention is: in response to the shortcomings of the knowledge graph basic model in the prior art in multimodal fusion, task versatility and cross-dataset generalization, the embodiment of the present invention provides a knowledge graph general reasoning method and system based on multimodal fusion and dynamic adaptation. By adopting a unified representation method for knowledge graph completion (KGC) and knowledge graph question answering (KGQA) tasks, the input data are input into the graph neural network after the two processing steps of relationship graph construction and edge scoring. A dual-tower encoder layer is used to obtain a dense feature representation containing text and structural bimodal information, and the downstream decoder layer outputs the option with the highest correct probability among all candidate answers as the predicted answer based on the query features of the query question, thereby improving the performance of the model in multimodal fusion, task versatility and cross-dataset generalization, so that it can be better applied to knowledge graph reasoning tasks in the fields of medicine, finance, science or education.
[0048] Figure 1 This is a flow chart of a general reasoning method for knowledge graphs based on multimodal fusion and dynamic adaptation provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of the framework of the general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation provided by the embodiment of the present invention. Figure 1 and Figure 2 As shown, the embodiment provides a general reasoning method for knowledge graph based on multimodal fusion and dynamic adaptation, including the following steps:
[0049] S1, performs graph structure transformation on knowledge graphs in the medical, financial, scientific or educational fields to convert various knowledge graph reasoning tasks into the same triple completion form to complete the unified modeling of task objectives.
[0050] In the embodiment, taking the field of science as an example, Figure 2 As shown in (A) in the figure, for the knowledge graph completion task in the scientific field, the input is a graph consisting of a list of triples, and the query question is a triple prefix (containing only the head entity and the relationship. The head entity specifically includes scientific concepts, experimental methods, research tools or subject entities, etc., and the relationship specifically includes composition relationship, application relationship, causal relationship, evolutionary relationship, factual logical relationship, subject logical relationship, etc.). The task goal is to find an entity in an entity set (specifically including matter or structure, application scenario or category, evolutionary node, related scientific entity, etc.) so that the completed triple (head entity, relationship, tail entity) is logical and correct in the knowledge graph, for example Figure 2The prefix of the triple shown in (A) is (person A, genre), and the corresponding answer target is the tail entity (popular). For the knowledge graph question answering task in the scientific field, its input is a knowledge graph, a triple prefix of a query question (including a head entity and a relationship. The head entity specifically includes the question covering the corresponding topic node in the knowledge graph. The relationship specifically includes composition relationship, application relationship, causal relationship, evolution relationship, factual logic relationship, subject logic relationship, etc., and fixed relationships such as "the correct answer is" can also be directly defined). The task goal is to require that a correct answer be found in a list of candidate answers with the help of the knowledge in the knowledge graph to answer the question as the tail entity node (specifically including scientific concepts, scientific facts, etc., and the options corresponding to the question can also be directly defined). For example Figure 2 The prefix of the triple shown in (A) is (Question: What attitude will a lion take if it perceives its competitor to be timid? "The correct answer is"), and the corresponding answer targets are the five candidate tail entities A, B, C, D, and E. In the knowledge graph question answering task, the knowledge graph needs to be further transformed. The query question is treated as a special entity node (query node), and each candidate answer is also treated as a special entity node (candidate answer node). They are all inserted into the knowledge graph, where the query node is connected to all topic nodes, and the candidate answer node is connected to all the entity nodes it contains.
[0051] Through the above transformation of the knowledge graph, various knowledge graph reasoning tasks including knowledge graph completion tasks and knowledge graph question-answering tasks are unified into a given knowledge graph and a query triple prefix (containing only the query head entity node and relationship). The goal is to find a correct node as the tail entity node to complete the query triple prefix into a logically correct triple.
[0052] S2, each relationship in the knowledge graph after graph structure conversion is used as a node, and the meta-relationship features of entity connections are used as edges to construct a relationship graph.
[0053] In the embodiment, in order to achieve the generalization migration of any knowledge graph data, the dense vector expressing any relationship and any entity is calculated through four meta-relation features. Here, it is necessary to build a relationship graph to prepare for feature calculation. Figure 2As shown in (B) in the figure, for any knowledge graph after graph structure conversion, each edge (i.e., relationship) is regarded as a node. For relations r1 and r2, if the head entity of r1 is the tail entity of r2, r1 and r2 are connected through a head-to-tail h2t (head-tail) edge; if the head entity of r1 is the head entity of r2, r1 and r2 are connected through a head-to-head h2h edge; if the tail entity of r1 is the head entity of r2, r1 and r2 are connected through a tail-to-head t2h edge; if the tail entity of r1 is the tail entity of r2, r1 and r2 are connected through a tail-to-tail t2t edge. Finally, the relationship graph is constructed through the meta-relationship features of the four entity connections h2t, h2h, t2h, and t2t.
[0054] S3, each edge in the knowledge graph after graph structure conversion is scored based on its contribution to the knowledge graph reasoning task, and the knowledge graph after edge scoring is obtained as the entity graph.
[0055] In the embodiment, for different downstream tasks, the contribution of edge information is different. A flexible edge scoring mechanism is used to assign weights to each edge so that its contribution will be different in different tasks. For the knowledge graph completion task, the contribution of each edge is the same by default, and the score of all edges is 1. Figure 2 As shown in (B) in the figure, for the knowledge graph question answering task, the text features of each edge (i.e., triple) in the knowledge graph after graph structure conversion and the text features of the query question are passed through a bilinear layer and then Softmax regularized to obtain its relevance score with the query question. This relevance score is used as the score of the edge and is involved in feature calculation when the knowledge graph general reasoning model performs reasoning.
[0056] S4, input the entity graph, relationship graph and query question into the general reasoning model of the knowledge graph and perform reasoning based on the unified modeling task objectives to obtain the predicted answer.
[0057] In the embodiment, for the core model structure part, its input is the entity graph, relationship graph, and query question obtained after executing the above three steps. Figure 2 The model framework shown in (C) is used for reasoning.
[0058] At the encoding layer, all nodes in the entity graph and relationship graph are first encoded into textual modal features using a language model (LM). The relationship graph and entity graph are then encoded sequentially using the QCMP (Query Conditional Message Passing) module and the GCPM (Global Conditional Message Passing) module, both based on graph neural networks (GNNs). For the relationship graph, the relationship graph and its textual modal features are extracted from both the query condition perspective (R-CMP (Relationship Graph Conditional Message Passing) module in QCMP) and the global semantic perspective (R-CMP module in GCMP) using the GNN. This yields comprehensive features from both perspectives (i.e., features for all relationships in the knowledge graph). For entity graphs, the entity graph and its text modal features are extracted from the query condition perspective (E-CMP (entity graph conditional message passing) in QCMP) and the global semantic perspective (E-CMP module in GCMP) through graph neural networks to obtain dense features from two perspectives. From the perspective of query conditions, the head entity in the query triple prefix corresponding to the entity graph is initialized to a full 1 vector in the node initialization stage, and other entities are initialized to a full 0 vector. When extracting dense features at each step at each perspective, the features of each edge in the entity graph are updated by the comprehensive features extracted from the previous step relationship graph, participating in the graph neural network. The message propagation calculation in the network is carried out, and from the global semantic perspective, all entity nodes corresponding to the entity graph are initialized to all-1 vectors in the node initialization stage, and the subsequent calculation process is consistent with the query condition perspective. Through this multi-angle design, dense features from the query condition perspective and dense features from the global semantic perspective are finally extracted. Finally, the dense features from the two perspectives are fused through a multi-layer perceptron (MLP) to obtain the final dense features containing text information and structural information. In order to emphasize the contribution of text information, the final dense features are dynamically fused with the text modal features of the entity graph using text adaptive dynamic fusion (DTAF) to obtain the fused dense features. Specifically, the dynamic linear weighting method is used to retain and emphasize the text semantic contribution of each node to a certain extent. The formula is expressed as:
[0059]
[0060] in, represents the dense features obtained after text adaptive dynamic fusion, x r Text modality features representing entity graphs, represents the final dense feature, and α represents the weight.
[0061] In the decoding layer, based on the fused dense features and the query question, the fused dense features corresponding to the query head entity are used as the query input (QueryEmbedding), and the fused dense features corresponding to the remaining entities except the query head entity are used as the key input (KeyEmbedding). The correct probability of each candidate node is calculated through the cross-attention mechanism (CrossAttention), and the candidate node with the highest probability is used as the predicted answer.
[0062] S5, after training the general reasoning model of knowledge graph, it is applied to knowledge graph reasoning tasks in the medical field, financial field, scientific field or education field.
[0063] In this embodiment, due to the high versatility of the model design, the knowledge graph universal reasoning model can be pre-trained for self-supervised knowledge graph completion on a large number of knowledge graphs. For the same knowledge graph completion task, zero-shot transfer can be achieved when dealing with any downstream data without training. For knowledge graph question-answering tasks with large differences, the model only needs to perform a small amount of training on the downstream data to achieve task transfer.
[0064] In summary, the embodiment of the present invention provides a general reasoning method for knowledge graphs based on multimodal fusion and dynamic adaptation, and its reasoning effect is verified through experiments in the following in-graph task fields and out-of-graph task fields.
[0065] In the field of on-graph tasks, as shown in Table 1, we compare the two most advanced graph-based models, ULTRA (source document titled "Towards foundation models for knowledge graph reasoning") and ProLINK (source document titled "LLM as prompter: Low-resource inductive reasoning on arbitrary knowledge graphs"), with the optimal model (Supervised SOTA) obtained by supervised training with existing models adapted to various datasets. This is combined with the general knowledge graph reasoning model (hereinafter referred to as MERRY) provided in an embodiment of the present invention, and a variant of MERRY obtained by using the PNA method in MERRY to aggregate the features of all tokens output by the LM as the text features of the text segment. PNA, a knowledge graph completion task was tested on the dataset in a zero-shot setting, where MRR represents Mean Reciprocal Rank and Hits@10 represents Hits at 10. On average, MERRY provided by the embodiments of the present invention achieved the best results.
[0066] Table 1 Ability evaluation results of domain tasks within the graph (knowledge graph completion task)
[0067]
[0068] Dataset Description:
[0069] IndE: Inductive Entity Datasets, which refers to the test set that only contains entities that have not been seen during training;
[0070] IndER: Inductive Entity and Relation Datasets, which refers to the test set containing entities and relations that were not seen during training;
[0071] IndE(WN), IndE(FB), and IndE(NL): refer to the three datasets WN18RR, FB15k-237, and NELL-995, respectively. Following the work of GraIL, each dataset has four sub-datasets, v1 to v4. The results in the table represent the average results of the four sub-datasets.
[0072] IndER(FB), IndER(WK), IndER(NL): refer to the three datasets FB15k-237, Wikidata68K and NELL-995 respectively. Following the work of InGram (the source document is named Ingram: Inductive knowledge graph embedding via relation graphs, Ingram: Inductive knowledge graph embedding method based on relational graphs), the first two datasets have four sub-datasets, v25, v50, v75, and v100, and IndER(NL) has five sub-datasets, v0, v25, v50, v75, and v100. The results in the table represent the average results of these sub-datasets.
[0073] In the field of off-graph tasks, as shown in Table 2, eight methods with excellent performance are compared, including: RoBERTa-Large, LLaMA-3-8b-instruct, RGCN (source document name: Modeling relational data with graph convolutional networks), GconAttn (source document name: Improving natural language inference using external knowledge in the science questions domain), KagNet (source document name: KagNet: Knowledge-aware graph networks for commonsense reasoning), RN (source document name: A simple neural network module for relational reasoning), MHGRN (source document name: Scalable multi-hop relational reasoning for knowledge-aware question answering), QA-GNN (source document name: QA-GNN: Reasoning with language models and knowledge We used the commonly used CommonsenseQA dataset to evaluate knowledge graph question answering capabilities, using two other approaches: MERRY (QA-GNN: Question Answering System Based on Collaborative Reasoning of Language Models and Knowledge Graphs), GreaseLM (GreaseLM: Graph Reasoning Enhanced Language Models), and GreaseLM (GreaseLM: Language Models Enhanced by Graph Reasoning). Accuracy represents accuracy. MERRY also achieved the best results, achieving an accuracy improvement of up to 5.7% compared to the language model LLaMA-3-8b-instruct.
[0074] Table 2 Ability evaluation results of out-of-graph domain tasks (knowledge graph question answering task)
[0075]
[0076] Dataset Description:
[0077] IHdev: In-House split development dataset, following the KagNet validation dataset;
[0078] IHtest: In-House split test dataset, following the test dataset of KagNet.
[0079] Based on the same inventive concept, Figure 3 As shown, an embodiment of the present invention also provides a general reasoning system 300 of knowledge graph based on multimodal fusion and dynamic adaptation, including: a task target modeling module 310, a relationship graph construction module 320, an entity graph construction module 330, a general model reasoning module 340, and a model training module 350.
[0080] The task goal modeling module 310 is used to perform graph structure conversion on the knowledge graph in the medical field, financial field, scientific field or scientific field to convert various knowledge graph reasoning tasks into the same triple completion form to complete the unified modeling of the task goals.
[0081] The relationship graph construction module 320 is used to use each relationship in the knowledge graph after graph structure conversion as a node and the meta-relationship features of entity connections as edges to construct a relationship graph.
[0082] The entity graph construction module 330 is used to score each edge in the knowledge graph after graph structure conversion based on its contribution to the knowledge graph reasoning task, and obtain the knowledge graph after edge scoring as the entity graph.
[0083] The general model reasoning module 340 is used to input the entity graph, relationship graph and query question into the general reasoning model of the knowledge graph and perform reasoning according to the unified modeling task objectives to obtain the predicted answer, including: encoding all nodes in the entity graph and relationship graph into text modal features, extracting dense features from the query condition perspective and the global semantic perspective based on the entity graph and relationship graph and their text modal features through the graph neural network, fusing the dense features from the two perspectives through a multi-layer perceptron to obtain the final dense features containing text information and structural information, and performing text-adaptive dynamic fusion of the final dense features with the text modal features of the entity graph and inputting them into the decoder to calculate the predicted answer corresponding to the query question.
[0084] The model training module 350 is used to train the general reasoning model of the knowledge graph and apply it to knowledge graph reasoning tasks in the medical field, financial field, scientific field or educational field.
[0085] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation when executing the computer program.
[0086] It should be noted that the general reasoning system of knowledge graph based on multimodal fusion and dynamic adaptation, and the electronic device provided in the above embodiments belong to the same inventive concept as the general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation. The specific implementation process is detailed in the embodiment of the general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation, which will not be repeated here.
[0087] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A general reasoning method for knowledge graphs based on multimodal fusion and dynamic adaptation, characterized by: The following steps are involved: Perform graph structure transformation on knowledge graphs in the medical, financial, scientific, or educational fields to transform knowledge graph reasoning tasks into the same triple completion form to achieve unified modeling of task objectives; Each relationship in the knowledge graph after graph structure conversion is used as a node, and the meta-relationship features of entity connections are used as edges to construct a relationship graph; Each edge in the knowledge graph after graph structure conversion is scored based on its contribution to the knowledge graph reasoning task, and the knowledge graph after edge scoring is obtained as the entity graph; Input the entity graph, relationship graph, and query question into the general reasoning model of the knowledge graph and perform reasoning according to the unified modeling task goal to obtain the predicted answer, including: encoding all nodes in the entity graph and relationship graph into text modal features, extracting dense features from the query condition perspective and the global semantic perspective based on the entity graph and relationship graph and their text modal features through a graph neural network, fusing the dense features from the two perspectives through a multi-layer perceptron to obtain the final dense feature containing text information and structural information, and performing text-adaptive dynamic fusion of the final dense feature with the text modal feature of the entity graph and inputting it into the decoder to calculate the predicted answer corresponding to the query question; After training, the general reasoning model of knowledge graph can be applied to knowledge graph reasoning tasks in the fields of medicine, finance, science or education.
2. The general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation according to claim 1 is characterized in that: The graph structure conversion of the knowledge graph in the medical field, the financial field, the scientific field, or the educational field to convert the knowledge graph reasoning task into the same triple completion form to complete the unified modeling of the task objectives includes: For knowledge graph completion tasks in the medical, financial, scientific, or educational fields, the entity nodes and relationships of the original knowledge graph are maintained. For knowledge graph question-answering tasks in the medical, financial, scientific, or educational fields, the query question is used as a query node and the candidate answer is used as a candidate answer node to insert into the original knowledge graph, where the query node is connected to all the corresponding topic nodes in the original knowledge graph, and the candidate answer node is connected to all the entity nodes it contains. By transforming the knowledge graph, various knowledge graph reasoning tasks including knowledge graph completion tasks and knowledge graph question-answering tasks are unified into a given knowledge graph and a query triple prefix containing only the query head entity node and relationship. The goal is to find a correct node as the tail entity node to complete the query triple prefix into a logically correct triple.
3. The general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation according to claim 1 is characterized in that: The meta-relational characteristics of entity connections include: For any knowledge graph after graph structure conversion, for relations r1 and r2, if the head entity of r1 is the tail entity of r2, r1 and r2 are connected through a head-to-tail h2t edge; if the head entity of r1 is the head entity of r2, r1 and r2 are connected through a head-to-head h2h edge; if the tail entity of r1 is the head entity of r2, r1 and r2 are connected through a tail-to-head t2h edge; if the tail entity of r1 is the tail entity of r2, r1 and r2 are connected through a tail-to-tail t2t edge. h2t, h2h, t2h and t2t are the meta-relation features of entity connections.
4. The general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation according to claim 1 is characterized in that: The rules for side scoring include: For knowledge graph completion tasks, the contribution of each edge is consistent by default, that is, the scores of all edges are the same; for knowledge graph question answering tasks, the text features of the knowledge graph after graph structure conversion and the text features of the query question are passed through a bilinear layer, and then Softmax regularized to obtain the relevance score with the query question as the edge score and participate in feature calculation when the knowledge graph general reasoning model is reasoned.
5. The general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation according to claim 1 is characterized in that: The entity graph, relationship graph and their text modal features are extracted through graph neural networks from the query condition perspective and the global semantic perspective to obtain dense features from two perspectives, including: The relationship graph and its text modal features are extracted from the query condition perspective and the global semantic perspective through a graph neural network to obtain comprehensive features from two perspectives. The entity graph and its text modal features are extracted from the query condition perspective and the global semantic perspective to obtain dense features from two perspectives. When extracting dense features at each step at each perspective, the features of each edge in the entity graph are updated by the comprehensive features extracted from the relationship graph in the previous step. Finally, dense features from the query condition perspective and dense features from the global semantic perspective are extracted.
6. The general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation according to claim 2 is characterized in that: When extracting features from the entity graph from the perspective of query conditions, the head entity in the query triple prefix corresponding to the entity graph is initialized to an assignment vector in the node initialization stage, and other entities are initialized to all-zero vectors to participate in the message propagation calculation in the graph neural network; when extracting features from the entity graph from the perspective of global semantics, all entity nodes corresponding to the entity graph are initialized to assignment vectors in the node initialization stage.
7. The general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation according to claim 1 is characterized in that: The final dense features are dynamically fused with the text modality features of the entity graph in a text-adaptive manner and then input into the decoder to calculate the predicted answer corresponding to the query question, including: The final dense features are dynamically fused with the text modal features of the entity graph through text adaptiveness to obtain the fused dense features. According to the query question, the fused dense features corresponding to the query head entity are used as the query input, and the fused dense features corresponding to the remaining entities except the query head entity are used as the key input. The correct probability of each candidate node is calculated through the cross-attention mechanism, and the candidate node with the highest probability is used as the predicted answer.
8. The general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation according to claim 1 is characterized in that: Text adaptive dynamic fusion adopts dynamic linear weighting method, and the formula is expressed as: in, Represents the dense features obtained after text adaptive dynamic fusion, Text modality features representing entity graphs, represents the final dense feature, and α represents the weight.
9. A general reasoning system for knowledge graphs based on multimodal fusion and dynamic self-adaptation, implemented using the general reasoning method for knowledge graphs based on multimodal fusion and dynamic self-adaptation according to any one of claims 1 to 8, characterized in that: include: Task target modeling module, relationship graph construction module, entity graph construction module, general model reasoning module, and model training module; The task goal modeling module is used to perform graph structure conversion on the knowledge graph in the medical field, the financial field, the scientific field or the educational field to convert the knowledge graph reasoning task into the same triple completion form to complete the unified modeling of the task goal; The relationship graph construction module is used to use each relationship in the knowledge graph after graph structure conversion as a node and the meta-relationship features of entity connections as edges to construct a relationship graph; The entity graph construction module is used to score each edge in the knowledge graph after graph structure conversion based on its contribution to the knowledge graph reasoning task, and obtain the knowledge graph after edge scoring as the entity graph; The general model reasoning module is used to input the entity graph, relationship graph and query question into the knowledge graph general reasoning model and perform reasoning according to the unified modeling task goal to obtain a predicted answer, including: encoding all nodes in the entity graph and relationship graph into text modal features, extracting dense features from the query condition perspective and the global semantic perspective based on the entity graph and relationship graph and their text modal features through a graph neural network, fusing the dense features from the two perspectives through a multi-layer perceptron to obtain a final dense feature containing text information and structural information, and performing text-adaptive dynamic fusion of the final dense feature with the text modal feature of the entity graph and inputting it into a decoder to calculate the predicted answer corresponding to the query question; The model training module is used to train the general reasoning model of the knowledge graph and then apply it to knowledge graph reasoning tasks in the medical field, financial field, scientific field or educational field.
10. An electronic device comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the general reasoning method of knowledge graph based on multimodal fusion and dynamic adaptation as described in any one of claims 1 to 8 when executing the computer program.