Knowledge graph enhanced reasoning framework based on graph neural network and large language model

Abstract

The invention discloses a knowledge graph enhanced reasoning framework based on a graph neural network and a large language model. The framework comprises a vectorization representation module, a graph neural network coding module, a large language model interaction module and a joint reasoning module. The vectorization representation module is used for aligning the structured knowledge graph and the unstructured text and converting the structured knowledge graph and the unstructured text into unified vectorization representation so as to provide a basis for subsequent processing; the graph neural network coding module obtains graph structure characterization by utilizing low-dimensional embedding of graph neural network learning data; the big language model interaction module performs semantic completion on the knowledge graph entity relationship by means of a big language model to generate an enhanced context vector; and the joint reasoning module fuses the results of the two modules and outputs an enhanced knowledge graph tetrad. Through multi-module cooperation, the accuracy and integrity of knowledge graph reasoning are remarkably improved, and the method has wide application prospects in the fields of intelligent questions and answers, intelligent decisions and the like.

Description

Technical Field

[0001] The present invention relates to the field of knowledge representation and reasoning technology, and specifically to a knowledge graph enhanced reasoning framework based on graph neural network and large language model. Background Art

[0002] As a structured knowledge representation method, knowledge graph describes the relationship between entities through graph structure and is widely used in intelligent question answering, information retrieval and recommendation systems. Traditional knowledge graph construction mainly relies on structured data and stores knowledge in the form of triples (head entity, relationship, tail entity). However, most knowledge in the real world exists in the form of unstructured text, such as news reports and academic papers, which makes knowledge graphs have problems such as incomplete information and untimely updates.

[0003] In order to acquire knowledge from unstructured text, researchers began to use natural language processing technology to extract entities and relations in text and integrate them into knowledge graphs. However, due to the ambiguity and diversity of natural language, as well as the semantic gap between structured knowledge graphs and unstructured text, entity alignment and relationship extraction tasks face many challenges.

[0004] In terms of knowledge reasoning, early reasoning methods were mainly based on rules and logic. Although they had high accuracy, the formulation of rules relied on domain experts and it was difficult to handle large-scale complex knowledge graphs. With the development of deep learning, graph neural networks (GNNs) have gradually become the mainstream technology for knowledge graph reasoning. They can automatically learn the features and relationships in the graph structure and effectively improve the reasoning performance. However, graph neural networks have deficiencies in handling long-distance dependencies and semantic understanding. Especially when there is missing information in the knowledge graph, the accuracy of the reasoning results will be greatly affected.

[0005] At the same time, the Large Language Model (LLM) has achieved remarkable results in natural language processing tasks with its powerful language understanding and generation capabilities. Introducing the Large Language Model into knowledge graph reasoning can make use of its semantic completion capabilities to make up for the information missing in the knowledge graph.

[0006] Therefore, how to effectively combine large language models with graph neural networks to give full play to the advantages of both remains a problem to be solved. Summary of the invention

[0007] The purpose of the present invention is to provide a knowledge graph enhanced reasoning framework based on graph neural network and large language model. Through the entity alignment algorithm, accurate alignment of structured knowledge graph entities and unstructured text entities is achieved, a joint vector representation is generated, and multi-source knowledge is effectively integrated. With the help of a multi-layer graph attention network, the relationship between entities is deeply learned, and the graph structure representation is output to improve the understanding of graph structure information. The missing information of the graph structure representation is semantically completed using a large language model to generate a semantic completion vector to fill the information gap of the knowledge graph. By combining the weighted graph structure representation and the semantic completion vector, an enhanced knowledge graph quadruple containing a confidence score is output to provide users with more reliable reasoning results. The framework of the present invention can effectively improve the quality and reasoning performance of the knowledge graph, and has important theoretical significance and practical application value.

[0008] To achieve the above objectives, the present invention provides a knowledge graph enhanced reasoning framework based on graph neural network and large language model, including the following modules: vectorization representation module, graph neural network encoding module, large language model interaction module and joint reasoning module; The vectorized representation module aligns the structured knowledge graph entities and the unstructured text entities through an entity alignment algorithm to generate a joint vector representation; The graph neural network encoding module uses a multi-layer graph attention network to learn the relationship between entities based on the joint vector representation and outputs a graph structure representation; The large language model interaction module uses the large language model to semantically complete the entity relationship of the knowledge graph and generate a semantic completion vector; The joint reasoning module outputs an enhanced knowledge graph quadruple (head entity, relationship, tail entity, confidence) containing a confidence score by combining and weighting the graph structure representation and the semantic completion vector.

[0009] Furthermore, the entity alignment algorithm adopts a matching strategy based on cosine similarity, which is expressed as follows: Suppose the set of structured knowledge graph entity vectors is , is the total number of structured knowledge graph entity vectors, each knowledge graph entity vector corresponds to a knowledge graph entity; the set of unstructured text entity vectors is , is the total number of unstructured text entity vectors; the cosine similarity formula is expressed as follows: in, is the first The vector representation of an entity, is the first The vector representation of the entity; when the similarity is calculated If it exceeds the preset threshold, it is considered a match, otherwise it is not a match; Further, the joint vector representation is composed of a fusion vector and unmatched structured knowledge graph entity vectors; The fusion vector The weighted fusion generation of the successfully matched structured knowledge graph entity vector and the unstructured text entity vector is expressed as follows: in, is the first The vector representation of an entity, is the first The vector representation of an entity, for and The similarity.

[0010] Furthermore, the multi-layer graph attention network uses the vector in the joint vector representation as the initial feature vector of the node, which is expressed as follows: Suppose the vector set in the joint vector representation is , is the total number of vectors in the joint vector representation; the multi-layer graph attention network node feature vector update formula is as follows: in, is the updated feature vector of the node, For the node With neighbor nodes The attention weights between are the parameters of the multi-layer graph attention network, For Node The set of neighbor nodes of is the activation function; The multi-layer graph attention network updates all nodes through the multi-layer graph attention network node feature vector update formula, and updates the updated node feature vector As the input of the next layer of graph attention network, repeat the node update formula to extract and update features; The graph structure represents nodes, which are composed of feature vectors of the final nodes in the multi-layer graph attention network.

[0011] Furthermore, the attention weight The calculation formula is: in, is the softmax activation function, is the LeakyReLU activation function; are the parameters of the multi-layer graph attention network; is the attention parameter vector; is the feature vector of node i, is the feature vector of node j, the neighbor of node i; and Vector concatenation ; and a are updated via back-propagation.

[0012] Furthermore, the large language model interaction module receives the graph structure representation , is the total number of vectors in the joint vector representation, and the semantic completion vector is generated according to the following steps: First, the entity pairs in the knowledge graph and , the corresponding graph structure representation and , and the embedding vector corresponding to the unstructured text Concatenate to generate enhanced input vector ; Then the enhanced input vector is input into the large language model to generate an enhanced relationship The probability distribution of ; Finally, select the one with the highest probability Enhanced Relationship , generate the semantic completion vector set corresponding to the enhanced input vector , Z is the preset threshold.

[0013] Furthermore, the combination weighting is expressed as follows: in, Characterize the structure of the graph The corresponding inference score, is the semantic completion vector The corresponding inference score, is the preset threshold, Score the final reasoning; This framework selects the corresponding relationship with the highest score y , generate knowledge graph quadruple after enhanced reasoning , and is the corresponding entity pair in the knowledge graph.

[0014] Compared with the prior art, the advantages of the present invention are: (1) Multi-source knowledge fusion and precise alignment: This paper uses an entity alignment algorithm based on cosine similarity matching to break through the limitations of traditional reliance on a single knowledge source or simple text matching, aligns structured knowledge graphs and unstructured text entities, generates a joint vector representation, and improves the accuracy and comprehensiveness of knowledge fusion.

[0015] (2) Graph feature learning and missing information completion: This paper uses a multi-layer graph attention network to learn entity relationships, output graph structure representations, and use a large language model to complete missing information. Compared with traditional technologies, it can better capture complex relationships and improve the accuracy and reliability of reasoning.

[0016] (3) Cross-model fusion and reasoning result optimization: This paper realizes the deep fusion of graph neural network and large language model by combining weighted fusion graph structure representation and semantic completion vector, and outputs enhanced knowledge graph quadruple with confidence score, providing valuable reference for user decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a framework module diagram of the present invention; Figure 2 It is a structural diagram of the vectorized representation module of the present invention; Figure 3 This is a structural diagram of the joint reasoning module of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0020] Example 1: Please refer to Figure 1 As shown, the present embodiment describes a knowledge graph enhanced reasoning framework based on a graph neural network and a large language model, the framework comprising a vectorized representation module, a graph neural network encoding module, a large language model interaction module and a joint reasoning module; The vectorized representation module, reference Figure 1 As shown, the structured knowledge graph entities and unstructured text entities are aligned through the entity alignment algorithm to generate a joint vector representation; The entity alignment algorithm adopts a matching strategy based on cosine similarity, which is expressed as follows: Suppose the set of structured knowledge graph entity vectors is , is the total number of structured knowledge graph entity vectors, each knowledge graph entity vector corresponds to a knowledge graph entity; the set of unstructured text entity vectors is , is the total number of unstructured text entity vectors; the cosine similarity formula is expressed as follows: in, is the first The vector representation of an entity, is the first The vector representation of the entity; when the similarity is calculated If it exceeds the preset threshold, it is considered a match, otherwise it is not a match; The knowledge graph entity vector Adopting the knowledge graph embedding model ComplEx: Modeling based on complex space can handle symmetric and antisymmetric relationships in the knowledge graph. In ComplEx, entities and relationships are represented as complex vectors. Through the scoring function, various relationship patterns in the knowledge graph are captured, and then entity vectors are learned. The text entity vector It is obtained through the pre-trained BERT model. The BERT model uses a bidirectional Transformer architecture to encode the input text. When processing sentences containing text entities, the model can generate a vector representation corresponding to the entity based on the context information. The joint vector representation is composed of a fusion vector and unmatched structured knowledge graph entity vectors; The fusion vector The weighted fusion generation of the successfully matched structured knowledge graph entity vector and the unstructured text entity vector is expressed as follows: in, is the first The vector representation of an entity, is the first The vector representation of an entity, for and The similarity.

[0021] The graph neural network encoding module uses a multi-layer graph attention network to learn the relationship between entities based on the joint vector representation and outputs a graph structure representation; The multi-layer graph attention network uses the vector in the joint vector representation as the initial feature vector of the node, which is expressed as follows: Suppose the vector set in the joint vector representation is , is the total number of vectors in the joint vector representation; the multi-layer graph attention network node feature vector update formula is as follows: in, is the updated feature vector of the node, For the node With neighbor nodes The attention weights between are the parameters of the multi-layer graph attention network, For Node The set of neighbor nodes of is the activation function; The multi-layer graph attention network updates all nodes through the multi-layer graph attention network node feature vector update formula, and updates the updated node feature vector As the input of the next layer of graph attention network, repeat the node update formula to extract and update features; The graph structure represents nodes, which are composed of feature vectors of the final nodes in the multi-layer graph attention network.

[0022] The attention weight The calculation formula is: in, is the softmax activation function, is the LeakyReLU activation function; are the parameters of the multi-layer graph attention network; is the attention parameter vector; is the feature vector of node i, is the feature vector of node j, the neighbor of node i; and Vector concatenation ; parameter and The update is performed through the back-propagation algorithm and the optimizer. The update process is as follows: Initialization parameters: Parameters of the multi-layer graph attention network before training begins and the attention parameter vector Initialize; this framework uses random initialization method to set parameters by randomly sampling from Gaussian distribution and The initial value of Forward propagation: The vector set in the joint vector representation Input the multi-layer graph attention network as the initial feature vector of the node; For each node , calculate its relationship with neighboring nodes The attention weight between : First calculate and , and then concatenate them to get ; Then calculate and through The activation function performs nonlinear transformation; finally, the results of all neighbor nodes are transformed by The activation function is normalized to obtain the attention weight ; Calculate the loss function: Define the loss function based on specific tasks including but not limited to node classification and link prediction. , the loss function is used to measure the difference between the model’s prediction results and the true label; Backpropagation: Calculating the loss function Parameters and Gradient and This step uses the chain rule, starting from the loss function, and gradually calculating the gradient of each layer backward; for each layer, the node feature vector update formula is , the derivative of the activation function needs to be considered when calculating the gradient , attention weight right and The derivative of, and the neighbor node feature vector right The derivative of Update parameters: Use the optimizer (Adam optimizer in this example) to calculate the gradient and Update parameters separately and ; The optimizer sets the learning rate , update the parameters as follows: Repeat the training process: repeat the steps of forward propagation, calculating the loss function, backpropagation, and updating parameters until the model converges.

[0023] The large language model interaction module uses the large language model to semantically complete the entity relationship of the knowledge graph and generate a semantic completion vector; The large language model interaction module receives the graph structure representation , is the total number of vectors in the joint vector representation, and the semantic completion vector is generated according to the following steps: First, the entity pairs in the knowledge graph and , the corresponding graph structure representation and , and the embedding vector corresponding to the unstructured text Concatenate to generate enhanced input vector : Among them, emb represents the text embedding operation; Then the enhanced input vector Input large language model to generate enhanced relations The probability distribution of : in, are the pre-trained classification layer model parameters, Represents the processing of a large language model; Finally, select the one with the highest probability Enhanced Relationship , through the feature fusion function Generate the semantic completion vector set corresponding to the enhanced input vector , Z is the preset threshold, and the formula is as follows: in, For the relationship The relationship vector obtained after the embedding operation; The feature fusion function The gating mechanism is used to effectively fuse the target entity pairs. Spectral structural characterization , and the predicted relationship The embedding representation of , thereby generating a more representative and semantically rich semantic completion vector ; The gating mechanism includes a gating coefficient Calculation and semantic completion vector Calculation of The gating coefficient Calculation of: Gating coefficient For control of graph structure characterization and The weight distribution when generating the semantic completion vector is calculated as follows: , is the learnable parameter in the gating mechanism; The semantic completion vector The calculation formula is: in, is the gating coefficient, is the semantic completion vector, , The target entity The spectral structure representation of For the predicted relationship The embedding representation of the weighted fusion result and the relationship between the prediction The embedding representation of Add together to get the final semantic completion vector , relation embedding The semantic information of the predicted relationship is carried and combined with the weighted fusion result of the graph structure representation, so that the semantic completion vector can comprehensively consider the structural information of the entity pair and the semantic information of the predicted relationship; The gating mechanism can be used to learn parameters It is learned through the back propagation algorithm during the training process of the model; specifically, during the training process of the entire framework, the loss value is calculated according to the loss function (cross entropy loss in this embodiment), and then the loss function is calculated through back propagation. Finally, the optimization algorithm (Adam in this example) is used to update the gradient The value of makes the loss function gradually decrease, so that the model can learn the optimal gating mechanism parameters.

[0024] The joint reasoning module, reference Figure 3 As shown, by combining and weighting the graph structure representation and the semantic completion vector, an enhanced knowledge graph quadruple (head entity, relationship, tail entity, confidence) containing a confidence score is output; The combination weighting is expressed as follows: in, Characterize the structure of the graph The corresponding inference score, is the semantic completion vector The corresponding inference score, is the preset threshold, Score the final reasoning; and The calculation formula is as follows: in, is the first multi-layer perceptron, is the second multi-layer perceptron, both of which are two-layer fully connected networks; the first multi-layer perceptron model parameters and the multi-layer graph attention network parameters and Joint training; the second multi-layer perceptron model parameters are obtained through the classification layer parameters Come to train; This framework selects the corresponding relationship with the highest score y , generate knowledge graph quadruple after enhanced reasoning , and is the corresponding entity pair in the knowledge graph.

[0025] The above formulas are all dimensionless and only use numerical values ​​for calculation. These formulas are based on a large amount of data and are derived through software simulation, aiming to be as close to the actual situation as possible. The preset parameters in the formulas can be adjusted by those skilled in the art according to specific needs.

[0026] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0027] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A knowledge graph enhanced reasoning framework based on graph neural network and large language model, characterized by: include: The vectorized representation module aligns structured knowledge graph entities with unstructured text entities through an entity alignment algorithm to generate a joint vector representation; A graph neural network encoding module, based on the joint vector representation, uses a multi-layer graph attention network to learn the relationship between entities and output a graph structure representation; A large language model interaction module uses a large language model to semantically complete the entity relationship of the knowledge graph and generate a semantic completion vector; The joint reasoning module outputs an enhanced knowledge graph quadruple (head entity, relationship, tail entity, confidence) containing a confidence score by combining and weighting the graph structure representation and the semantic completion vector.

2. The frame according to claim 1, characterized in that The entity alignment algorithm adopts a matching strategy based on cosine similarity, which is expressed as follows: Suppose the set of structured knowledge graph entity vectors is , is the total number of structured knowledge graph entity vectors, each knowledge graph entity vector corresponds to a knowledge graph entity; the set of unstructured text entity vectors is , is the total number of unstructured text entity vectors; the cosine similarity formula is expressed as follows: in, is the first The vector representation of an entity, is the first The vector representation of the entity; when the similarity is calculated If it exceeds the preset threshold, it is considered a match, otherwise it is not a match.

3. The frame according to claim 2, characterized in that The joint vector representation is composed of a fusion vector and unmatched structured knowledge graph entity vectors; The fusion vector The weighted fusion generation of the successfully matched structured knowledge graph entity vector and the unstructured text entity vector is expressed as follows: in, is the first The vector representation of an entity, is the first The vector representation of an entity, for and The similarity.

4. The frame according to claim 1, characterized in that The multi-layer graph attention network uses the vector in the joint vector representation as the initial feature vector of the node, which is expressed as follows: Suppose the vector set in the joint vector representation is , is the total number of vectors in the joint vector representation; the multi-layer graph attention network node feature vector update formula is as follows: in, is the updated feature vector of the node, For the node With neighbor nodes The attention weights between are the parameters of the multi-layer graph attention network, For Node The set of neighbor nodes of is the activation function; The multi-layer graph attention network updates all nodes through the multi-layer graph attention network node feature vector update formula, and updates the updated node feature vector As the input of the next layer of graph attention network, repeat the node update formula to extract and update features; The graph structure represents nodes, which are composed of updated feature vectors of all nodes in the multi-layer graph attention network.

5. The frame according to claim 4, characterized in that The attention weight The calculation formula is: in, is the softmax activation function, is the LeakyReLU activation function; are the parameters of the multi-layer graph attention network; is the attention parameter vector; is the feature vector of node i, is the feature vector of node j, the neighbor of node i; and Vector concatenation ; and a are updated via back-propagation.

6. The frame according to claim 1, characterized in that The large language model interaction module receives the graph structure representation , is the total number of vectors in the joint vector representation, and the semantic completion vector is generated according to the following steps: First, the entity pairs in the knowledge graph and , the corresponding graph structure representation and , and the embedding vector corresponding to the unstructured text Concatenate to generate enhanced input vector ; Then the enhanced input vector is input into the large language model to generate an enhanced relationship The probability distribution of ; Finally, select the one with the highest probability Enhanced Relationship , generate the semantic completion vector set corresponding to the enhanced input vector , Z is the preset threshold.

7. The frame according to claim 1, characterized in that The combination weighting is expressed as follows: in, For the structural characterization of the graph The corresponding inference score, is the semantic completion vector The corresponding inference score, is the preset threshold, Score the final reasoning; This framework selects the corresponding relationship with the highest score y , generate knowledge graph quadruple after enhanced reasoning , and is the corresponding entity pair in the knowledge graph.

Images