Knowledge graph completion method based on graph context and relation perception attention

The CRGAT model addresses inefficiencies in knowledge graph completion by using relationship-aware attention and biased random walks to enhance entity embeddings, improving accuracy and capturing non-local contexts.

CN119067211AActive Publication Date: 2024-12-03SOUTHWEST PETROLEUM UNIV

Patent Information

Application Number
CN202411545385.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-01
Publication Date
2024-12-03
Estimated Expiration
2044-11-01

AI Technical Summary

Technical Problem

Existing knowledge graph completion methods face challenges in capturing key semantic information under different relationships and non-local contexts, leading to inefficiencies in large-scale knowledge graphs due to over-smoothing and noise.

Method used

A knowledge graph completion method using a graph neural network model (CRGAT) with a local relation-aware graph context aggregation module and a non-local graph context aggregation module, incorporating relationship-aware attention and biased random walks to enhance entity embeddings.

Benefits of technology

The method effectively captures relationship information and non-local contexts, improving the accuracy of entity representation and knowledge graph completion performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067211A_ABST
    Figure CN119067211A_ABST
Patent Text Reader

Abstract

The invention discloses a knowledge graph completion method based on graph context and relation perception attention. A graph neural network knowledge graph completion model (CRGAT) based on graph context and relation perception attention is adopted. The CRGAT learns local entity characterization by introducing specific network parameters for each type of relations, aggregating local relation perception graph contexts of entities. Then designing a random walk sampling strategy to obtain non-local context of the entity, and modeling non-local entity representation through a long short-term memory (LSTM) network; and then fusing local and non-local entity representations by using a gating mechanism to obtain a final entity embedding representation, and generating a relation embedding representation through dimension transformation. And finally, calculating a score for each candidate triple, and completing knowledge graph completion. According to the method, a local relation sensing mechanism is provided, relation information between entities is fully utilized, in addition, non-local graph context is effectively used for enhancing entity embedding, and the complementation effect is remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph completion under a knowledge graph, and specifically to a knowledge graph completion method based on graph context and relationship-aware attention. Background Art

[0002] A knowledge graph is a graph structure that stores knowledge in the form of triples (entity, relationship, entity). It plays a key role in knowledge-driven practical applications such as question-answering systems, recommendation systems, and dialogue generation, and has revolutionized the solution paradigm for many natural language processing problems.

[0003] Knowledge graph completion aims to use existing graph information to infer missing fact triples and verify the validity of new triples; the latest methods of knowledge graph completion mainly rely on knowledge graph embedding, by converting entities and relations into low-dimensional distributed representations; these methods can be roughly divided into three categories: knowledge graph completion methods based on translation, knowledge graph completion methods based on bilinear transformation, and knowledge graph completion methods based on graph neural networks.

[0004] Existing knowledge graph completion methods mainly include methods based on translation operations, methods based on bilinear transformations, and methods based on graph neural networks. Methods based on translation operations regard relationships as a transformation process from head entities to tail entities, and methods based on bilinear transformations associate the latent semantics of entities and relationships through scoring functions. Methods based on graph neural networks effectively integrate the contextual information of entities through graph aggregation mechanisms. In addition, some models combine encoder-decoder architectures to further improve the accuracy of knowledge graph completion. Although these methods have achieved certain results in improving prediction performance, there is still a problem of high resource consumption on large-scale knowledge graphs.

[0005] Although knowledge graph completion methods based on graph neural networks have made progress, they still face two major challenges. First, existing methods use fixed weight matrices to aggregate neighbor entities, which makes it difficult to effectively capture key semantic information under different relationships, resulting in insufficient recognition of important relationships. Second, these methods have limitations in capturing non-local contexts in knowledge graphs. Although a multi-hop neighbor information aggregation mechanism is introduced, layer-by-layer propagation may lead to over-smoothing or noise, ignoring important information in non-local contexts. Summary of the invention

[0006] The present invention proposes a knowledge graph completion method based on graph context and relationship-aware attention, which can effectively identify key relationships and make full use of the relationship information between entities. In addition, it can deeply mine important contextual information, enhance entity embedding, and significantly improve the knowledge graph completion performance.

[0007] The technical solution adopted by the present invention is as follows: a knowledge graph completion method based on graph context and relationship-aware attention, adopting a graph neural network knowledge graph completion model (CRGAT) based on graph context and relationship-aware attention, including the following specific steps: S1, introducing a local relationship-aware graph context aggregation module, and using a graph attention network containing relationship-specific parameters to obtain a relationship-aware local representation of each entity in the knowledge graph; S2, introducing a non-local graph context aggregation module, using random walks to sample the non-local context of the entity, and obtaining the non-local representation of the entity through a long short-term memory network (LSTM); S3, using a gating mechanism to aggregate the local representation and non-local representation of the entity to obtain the final embedded representation of the entity, and at the same time, generating the final relationship embedded representation through dimensional transformation; S4, calculating the score for each candidate triple based on the obtained final embedded representation of the entity and the relationship, and completing the knowledge graph based on the result.

[0008] The local relation-aware graph context aggregation module in step S1 introduces the network parameters of specific relations into the graph attention network. , adaptively learn information from neighbors under different relations; For through relationship Connect to a specific neighbor entity The central entity , use the following formula to combine their representations and calculate the neighbor entity information under a specific relationship: ; ; in, It is a representation obtained by combining relations and entities. It is a combination operation that integrates the relational representation into the entity representation. , It is a physical representation. is a relational representation, is a learnable relation-specific projection matrix, It is Hadamard. Neighbor entity In a specific relationship Pass it to the central entity Information, is a single-layer feedforward neural network. is a learnable weight matrix; In order to assign different importance to information, a multi-head relation-aware attention mechanism is used to measure the importance of each neighbor in the aggregation process, as shown in the following formula: ; ; ; in, It is the attention head The final message below is embedded in It is the attention head The representation obtained by combining the following relations and entities, It is the attention head The following learnable weight matrix, is the normalization function, is the attention score, It is a single-layer feedforward neural network, composed of the weight matrix Parameterization, followed by the LeakyReLU nonlinear function, is the normalized attention weight, Is the central entity The first-order neighbor set of Is the connection center entity and neighbors relationship; All first-order neighbors and corresponding relationship information are aggregated according to the learned attention weights, as shown in the following formula: ; in, Is the central entity The final local relation-aware representation of is the number of attention heads, Is the central entity The set of first-order neighbor entities and relations.

[0009] The non-local graph context aggregation module in step S2 uses Node2Vec to perform biased random walks from the central entity Departure, proceed The secondary length is fixed to The biased random walk of Entities are sorted in descending order according to the frequency of their appearance in the walk, and the top 4 entities with the highest frequency are selected as the central entities in the knowledge graph. The non-local graph context of ; Modeling central entities using long short-term memory networks Its non-local graph context The dependencies between Input the LSTM network in reverse order and define the output of the last step as the central entity Non-local representation of , the formula is as follows: ; in, It is a recurrent neural network used to capture long-term dependencies in sequence data. yes The reverse order of .

[0010] In step S3, the local representation and non-local representation of the entity are aggregated using a gating mechanism to obtain the final embedding representation of the entity. At the same time, the final relational embedding representation is generated through dimensionality transformation, as shown in the following formula: ; ; in, Is the central entity The final embedding representation of is a learnable vector, is the sigmoid function, It is Hadamard. It's a relationship The initial embedding representation of It's a relationship The relational embedding representation of is a transformation matrix that projects the relations to the same embedding dimension as the entities.

[0011] The triple scoring module described in step S4 uses ConvE and InteractE networks as decoders to calculate the score of the triple (s, r, o). The ConvE related formula is as follows: ; in, represents the triple score, and Respectively and The two-dimensional reconstruction of represents the standard convolution operation, is the weight matrix, Representing Entities Embedded representation of .

[0012] The InteractE related formula is as follows: ; ; ; in, Represents t-random permutation, a random permutation operation performed on the input feature sequence within a given time step t, which is used to disrupt the order of the input features to enhance the robustness of the model. represents a chessboard reshaping operation, It indicates that the arranged feature matrix is ​​reshaped in a chessboard format, that is, the feature matrix is ​​rearranged in a chessboard format to optimize the convolution calculation process between features, thereby improving the calculation efficiency of the model. Score the triples, represents the sigmoid function, represents the process of flattening a tensor into a vector, represents a deep circular convolution, is the convolution filter, is the weight matrix; Based on the triplet scores, the network is optimized using the standard cross entropy loss with label smoothing, as follows: ; in, is a binary cross entropy loss function, which is used to measure the gap between the predicted probability and the true label. Represents the total number of samples, used to calculate the average loss, It is a triple Tags, is the predicted probability.

[0013] Add the triples with the highest scores to the original knowledge graph to complete the knowledge graph.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The knowledge graph completion method based on graph context and relationship-aware attention proposed in the present invention adopts a relationship-aware aggregation strategy and introduces relationship-specific network parameters in the process of neighbor information aggregation, which can more accurately capture the relationship information between neighbor nodes and target nodes, and help improve the accuracy of entity representation; (2) The knowledge graph completion method based on graph context and relationship-aware attention proposed in the present invention uses a biased random walk sampling module to obtain the most relevant high-order neighbors of an entity, and uses a long short-term memory (LSTM) network to model the dependency between an entity and its non-local context, effectively alleviating the over-smoothing problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall framework diagram of the model of the present invention.

[0016] Figure 2 It is a graph of the experimental results of the present invention on the data sets FB15k-237 and WN18RR. DETAILED DESCRIPTION

[0017] The present invention is further described in detail below with reference to the accompanying drawings: A knowledge graph completion method based on graph context and relationship-aware attention, using a graph neural network knowledge graph completion model CRGAT based on graph context and relationship-aware attention, the overall structure of the model is as follows Figure 1 As shown, , , Represent the initial embedding representations of entity 1, entity 2, and entity 3 respectively, , , Represents three different types of relationships. , , Respectively express the relationship , , A learnable relation-specific projection matrix, represents the attention mechanism, , , Represents the central entity Neighbor entities with higher wandering frequency, , , Represent entities ,entity and entities The initial embedding representation of , and other symbol descriptions are detailed in the content of the invention.

[0018] The construction and training of the model include the following specific steps: S1. Introducing a local relation-aware graph context aggregation module, and using a graph attention network with relation-specific parameters to obtain a relation-aware local representation of each entity in the knowledge graph; S2. Introducing a non-local graph context aggregation module, and using random walks to sample the non-local context of entities, and using a long short-term memory network (LSTM) to obtain the non-local representation of entities; S3. Using a gating mechanism to aggregate the local and non-local representations of entities to obtain the final embedded representation of entities, and at the same time, generating the final relational embedded representation through dimensionality transformation; S4. Based on the final embedded representations of entities and relations obtained, the score of each candidate triple is calculated, and the knowledge graph is completed based on the results.

[0019] The local relation-aware graph context aggregation module in step S1 introduces the network parameters of specific relations into the graph attention network. , adaptively learn information from neighbors under different relations; For through relationship Connect to a specific neighbor entity The central entity , use the following formula to combine their representations and calculate the neighbor entity information under a specific relationship: ; ; in, It is a representation obtained by combining relations and entities. It is a combination operation that integrates the relational representation into the entity representation. , It is a physical representation. is a relational representation, is a learnable relation-specific projection matrix, It is Hadamard. Neighbor entity In a specific relationship Pass it to the central entity Information, is a single-layer feedforward neural network. is a learnable weight matrix; In order to assign different importance to information, a multi-head relation-aware attention mechanism is used to measure the importance of each neighbor in the aggregation process, as shown in the following formula: ; ; ; in, It is the attention head The final message below is embedded in It is the attention head The representation obtained by combining the following relations and entities, It is the attention head The following learnable weight matrix, is the normalization function, is the attention score, It is a single-layer feedforward neural network, composed of the weight matrix Parameterization, followed by the LeakyReLU nonlinear function, is the normalized attention weight, Is the central entity The first-order neighbor set of Is the connection center entity and neighbors relationship; All first-order neighbors and corresponding relationship information are aggregated according to the learned attention weights, as shown in the following formula: ; in, Is the central entity The final local relation-aware representation of is the number of attention heads, Is the central entity The set of first-order neighbor entities and relations.

[0020] The non-local graph context aggregation module in step S2 uses Node2Vec to perform biased random walks from the central entity Departure, proceed The secondary length is fixed to The biased random walk of Entities are sorted in descending order according to the frequency of wandering, and the top 4 entities with the highest frequency are selected as the central entities in the knowledge graph. The non-local graph context of ; Modeling central entities using long short-term memory networks Its non-local graph context The dependencies between Input the LSTM network in reverse order and define the output of the last step as the central entity Non-local representation of , the formula is as follows: ; in, It is a recurrent neural network used to capture long-term dependencies in sequence data. yes The reverse order of .

[0021] In step S3, the local representation and non-local representation of the entity are aggregated using a gating mechanism to obtain the final embedding representation of the entity. At the same time, the final relational embedding representation is generated through dimensionality transformation, as shown in the following formula: ; ; in, Is Entity The final embedding representation of is a learnable vector, is the sigmoid function, It is Hadamard. It's a relationship The initial embedding representation of It's a relationship The relational embedding representation of is a transformation matrix that projects the relations to the same embedding dimension as the entities.

[0022] The triple scoring module described in step S4 uses ConvE and InteractE networks as decoders to calculate the score of the triple (s, r, o). The ConvE related formula is as follows: ; in, represents the triple score, and Respectively and The two-dimensional reconstruction of represents the standard convolution operation, is the weight matrix, Representing Entities Embedded representation of .

[0023] The InteractE related formula is as follows: ; ; ; in, Represents t-random permutation, a random permutation operation performed on the input feature sequence within a given time step t, which is used to disrupt the order of the input features to enhance the robustness of the model. represents a chessboard reshaping operation, It indicates that the arranged feature matrix is ​​reshaped in a chessboard format, that is, the feature matrix is ​​rearranged in a chessboard format to optimize the convolution calculation process between features, thereby improving the calculation efficiency of the model. Score the triples, represents the sigmoid function, represents the process of flattening a tensor into a vector, represents a deep circular convolution, is the convolution filter, is the weight matrix; Based on the triplet scores, the network is optimized using the standard cross entropy loss with label smoothing, as follows: ; in, is a binary cross entropy loss function, which is used to measure the gap between the predicted probability and the true label. Represents the total number of samples, used to calculate the average loss, It is a triple Tags, is the predicted probability.

[0024] Add the triples with the highest scores to the original knowledge graph to complete the knowledge graph.

[0025] For the graph neural network knowledge graph completion model CRGAT based on graph context and relationship-aware attention designed by the present invention, link prediction is used to benchmark its performance.

[0026] The present invention adopts the mean reciprocal rank (MRR), the average rank mean (MR) and the top N percentage of hits@N as evaluation indicators of algorithm performance.

[0027] When training CRGAT, the optimizer Adam is used to optimize the model parameters, and the optimal parameters are found through the random grid search method.

[0028] In terms of the selection of data sets, this paper uses two standard data sets commonly used in knowledge graph completion tasks, FB15k-237 and WN18RR, which include training sets, validation sets, and test sets. The experimental results are shown in the attached Figure 2 As shown in Figure 5, CRGAT achieves leading results on the WN18RR dataset, outperforming other baseline models in all five metrics, and performs well on the FB15k-237 dataset, outperforming other baseline models in 4 out of 5 metrics.

Claims

1. A knowledge graph completion method based on graph context and relationship-aware attention, characterized in that: This method adopts the graph neural network knowledge graph completion model CRGAT based on graph context and relationship-aware attention, including the following steps: S1. Introduce a local relation-aware graph context aggregation module, and use a graph attention network with specific relational network parameters to obtain a relation-aware local representation of each entity in the knowledge graph. S2, introduce the non-local graph context aggregation module, use random walks to sample the non-local context of entities, and obtain the non-local representation of entities through long short-term memory networks; S3. Aggregate the local representation and non-local representation of the entity using the gating mechanism to obtain the final embedding representation of the entity. At the same time, generate the final relation embedding representation through dimensionality transformation. S4. Based on the final embedded representation of entities and relations, use the triple scoring module to calculate the score for each candidate triple, and complete the knowledge graph based on the results.

2. The knowledge graph completion method based on graph context and relationship-aware attention according to claim 1 is characterized in that: The local relation-aware graph context aggregation module in step S1 introduces the network parameters of specific relations into the graph attention network. , adaptively learn information from neighbors under different relations; For through relationship Connect to a specific neighbor entity The central entity , use the following formula to combine their representations and calculate the neighbor entity information under a specific relationship: ; ; in, It is a representation obtained by combining relations and entities. It is a combination operation that integrates the relational representation into the entity representation. , It is a physical representation. is a relational representation, is a learnable relation-specific projection matrix, It is Hadamard. Neighbor entity In a specific relationship Pass it to the central entity Information, is a single-layer feedforward neural network. is a learnable weight matrix; In order to assign different importance to information, a multi-head relation-aware attention mechanism is used to measure the importance of each neighbor in the aggregation process, as shown in the following formula: ; ; ; in, It is the attention head The final message below is embedded in It is the attention head The representation obtained by combining the following relations and entities, It is the attention head The following learnable weight matrix, is the normalization function, is the attention score, It is a single-layer feedforward neural network, composed of the weight matrix Parameterization, followed by the LeakyReLU nonlinear function, is the normalized attention weight, Is the central entity The first-order neighbor set of Is the connection center entity and neighbors relationship; All first-order neighbors and corresponding relationship information are aggregated according to the learned attention weights, as shown in the following formula: ; in, Is the central entity The final local relation-aware representation of is the number of attention heads, Is the central entity The set of first-order neighbor entities and relations.

3. The knowledge graph completion method based on graph context and relationship-aware attention according to claim 1 is characterized in that: The non-local graph context aggregation module in step S2 uses Node2Vec to perform biased random walks from the central entity Departure, proceed The secondary length is fixed to The biased random walk of Entities are sorted in descending order according to the frequency of their appearance in the walk, and the top 4 entities with the highest frequency are selected as the central entities in the knowledge graph. The non-local graph context of ; Modeling central entities using long short-term memory networks Its non-local graph context The dependencies between Input the LSTM network in reverse order and define the output of the last step as the central entity Non-local representation of , the formula is as follows: ; in, It is a recurrent neural network used to capture long-term dependencies in sequence data. yes The reverse order of .

4. The knowledge graph completion method based on graph context and relationship-aware attention according to claim 1 is characterized in that: In step S3, the local representation and non-local representation of the entity are aggregated using a gating mechanism to obtain the final embedding representation of the entity. At the same time, the final relational embedding representation is generated through dimensionality transformation, as shown in the following formula: ; ; in, Is the central entity The final embedding representation of is a learnable vector, is the sigmoid function, It is Hadamard. It's a relationship The initial embedding representation of It's a relationship The relational embedding representation of is a transformation matrix that projects the relations to the same embedding dimension as the entities.

5. The knowledge graph completion method based on graph context and relationship-aware attention according to claim 1, characterized in that: The triple scoring module described in step S4 uses ConvE and InteractE networks as decoders to calculate the score of the triple (s, r, o). The ConvE related formula is as follows: ; in, represents the triple score, and Respectively and The two-dimensional reconstruction of represents the standard convolution operation, is the weight matrix, Representing Entities Embedded representation of The InteractE related formula is as follows: ; ; ; in, Represents t-random permutation, a random permutation operation performed on the input feature sequence within a given time step t, which is used to disrupt the order of the input features to enhance the robustness of the model. represents a chessboard reshaping operation, It indicates that the arranged feature matrix is ​​reshaped in a chessboard format, that is, the feature matrix is ​​rearranged in a chessboard format to optimize the convolution calculation process between features, thereby improving the calculation efficiency of the model. Score the triples, represents the sigmoid function, represents the process of flattening a tensor into a vector, represents a deep circular convolution, is the convolution filter, is the weight matrix; Based on the triplet scores, the network is optimized using the standard cross entropy loss with label smoothing, as follows: ; in, is a binary cross entropy loss function, which is used to measure the gap between the predicted probability and the true label. Represents the total number of samples, used to calculate the average loss, It is a triple Tags, is the predicted probability; Add the triples with the highest scores to the original knowledge graph to complete the knowledge graph.

6. The knowledge graph completion method based on graph context and relationship-aware attention according to claim 1, characterized in that: The graph neural network knowledge graph completion model CRGAT based on graph context and relationship-aware attention adopts link prediction to benchmark its performance, and uses the reciprocal mean MRR, average mean MR, and the top N percentage Hits@N as evaluation indicators of algorithm performance.

7. The knowledge graph completion method based on graph context and relationship-aware attention according to claim 1, characterized in that: When training the graph neural network knowledge graph completion model CRGAT based on graph context and relationship-aware attention, the optimizer Adam is used to optimize the model parameters, and the optimal parameters are found through the random grid search method.

8. The knowledge graph completion method based on graph context and relationship-aware attention according to claim 1, characterized in that: When training the graph neural network knowledge graph completion model CRGAT based on graph context and relationship-aware attention, two standard datasets FB15k-237 and WN18RR, which are commonly used in knowledge graph completion tasks, are used, including training sets, validation sets, and test sets.

Citation Information

Patent Citations

  • Knowledge graph completion method and device, electronic equipment and storage medium

    CN113569056A

  • Emotion recognition method based on bimodal graph network information bottleneck

    CN115631504A

  • Knowledge graph completion method based on global and local comparative learning

    CN117668256A

  • System and method of managing knowledge for knowledge graphs

    US20230081891A1

Cited By

  • Knowledge graph completion method based on topology perception hybrid convolutional network

    CN120317346A

  • Knowledge graph link reasoning method

    CN120409652A

  • A knowledge graph link reasoning method

    CN120409652B

  • Construction method of intelligent manufacturing scene knowledge base based on artificial intelligence

    CN121413745A