A knowledge graph completion method fusing GNN and global attention mechanism

Abstract

The application designs a knowledge graph completion method fusing GNN and global attention mechanism, inputs the knowledge graph into a GNN structure extraction module of a Transformer global perception embedding module, introduces reverse relations and self-loop relation types and introduces centrality coding to obtain initial embedding of entities and relations; extracts structure information of the knowledge graph through a graph attention mechanism to obtain entity and relation embedding containing local structure information of the entity; inputs into the Transformer global perception embedding module, calculates entity embedding after fusing multi-relation fusion degree coding, calculates spatial position coding and edge feature coding, adds the bias term into attention matrix calculation, processes according to the standard Transformer global perception embedding module process to obtain entity embedding representation as the output of the Transformer global perception embedding module; inputs the entity and relation embedding processed through the Transformer global perception embedding module into a scoring function to evaluate and complete.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graph technology and is a knowledge graph completion method that integrates GNN and global attention mechanism. Background Technology

[0002] In recent years, knowledge graphs have attracted significant attention from both academia and industry. A knowledge graph, composed of entities, relations, and semantic descriptions, is a heterogeneous directed graph data structure. Existing knowledge graphs include Freebase, YAGO, WordNet, NELL, and Wikidata, and have been widely applied in artificial intelligence fields such as search engines, question-answering systems, dialogue systems, recommendation systems, knowledge reasoning, and event prediction. However, since most knowledge graphs are constructed manually or semi-automatically, problems such as incomplete and sparse data frequently arise. To address these issues, the Knowledge Graph Completion (KGC) task has emerged.

[0003] Knowledge graph completion aims to derive potential triples from existing triples in a knowledge graph, further refining the knowledge graph and reducing the manual cost of its construction. Specifically, the knowledge graph completion task is currently abstracted as a prediction problem: predicting the missing parts of triples, such as predicting the relationship between known head and tail entities; predicting the tail entity given the head entity and relation; or predicting the head entity given the tail entity and relation. Currently, mainstream knowledge graph completion methods obtain entity representations through methods such as Knowledge Graph Embedding (KGE) and use these representations to predict and infer missing entities and relations. Knowledge Graph Embedding aims to map elements (entities and relations) in a knowledge graph to a low-dimensional continuous vector space, learning the embedded representations of entities and relations while preserving the inherent structure and semantic information of the knowledge graph. This involves converting the symbolic representation of the knowledge graph into a numerical representation, thereby achieving efficient knowledge graph completion.

[0004] Currently, the most commonly used knowledge graph completion methods are the translation distance-based model embedding method and the graph neural network-based model embedding method.

[0005] Firstly, there's the translation distance model embedding method, which is based on a simple assumption: entities and relations can be matched through translation vectors. The earliest translation distance model was the knowledge representation learning model, or TransE model, proposed by Bordes et al. in their 2013 paper "Translating Embeddings for Modeling Multi-relational Data." This model maps entities and relations in a knowledge graph to a low-dimensional vector space, obtaining embedded vector representations of entities and relations, and uses the vector difference between the head and tail entities to represent the relationships between entities.

[0006] Next, we will discuss embedding methods based on graph neural network (GNN) models. A GNN is a type of neural network capable of processing graph-structured data, extracting features by aggregating and updating node information in the graph. The most basic GNN is the Graph Convolutional Network (GCN) proposed by Kipf and Welling et al. in their 2017 paper, "Semi-Supervised Classification with Graph Convolutional Networks." GCN updates the graph data by aggregating its own and neighborhood features using a mapping function. Since the basic GCN uses a uniform function to aggregate neighborhoods, it cannot effectively distinguish between different entities and relationships within the neighborhood. To handle knowledge graph data with multiple relationship types, researchers have successively proposed graph neural network models that comprehensively consider entity and relationship information in the knowledge graph, such as the R-GCN model proposed by Schlichtkrull et al. in their 2018 paper, "Modeling Relational Data with Graph Convolutional Networks," and the CompGCN model proposed by Vashishth et al. in their 2020 paper, "Composition-based Multi-Relational Graph Convolutional Networks." R-GCN distinguishes different relations by assigning different weight matrices to each type of relation, thereby separating the features of different relations and preserving relation-specific information to better capture the associations between entities; CompGCN further considers the complex interactions between entities and relations, embedding entities and relations simultaneously in the knowledge graph.

[0007] However, both the translation distance model and the graph neural network model have obvious drawbacks: the translation distance model has limitations in handling complex one-to-many, many-to-one, and many-to-many relationships, is sensitive to hyperparameters, has limited scalability, cannot meet the needs of automatically adding new entities and new relationships, and is completely unable to perceive information about non-adjacent entities; while the graph neural network model has a certain ability to capture information about the first-order and higher-order neighbors of an entity, it is limited by the number of GNN layers (maximum of 3-4 layers, too many layers will cause oversmoothing and over-compression problems), and often only cares about the nearby neighbor entities during aggregation, unable to perceive information about the more distant neighbor entities, which seriously affects the effect of knowledge graph completion.

[0008] In recent years, Transformer has demonstrated powerful performance in fields such as Natural Language Processing (NLP), Computer Vision (CV), and Generative Intelligence Generated Content (AIGC). Transformer has strong expressive power and the ability to capture global information, but few scholars have explored its application in the graph domain. There are two main reasons for this: (1) Graph data has an irregular structure and lacks a natural order, making it difficult to apply Transformer directly to graph data. Knowledge graphs are a very complex type of heterogeneous graph data, and applying Transformer to knowledge graph embedding will face huge challenges; (2) In knowledge graphs, the local structure of entities is very important information. However, when Transformer updates entity embeddings, it focuses too much on global information, which leads to its inability to effectively capture information of directly adjacent entities, that is, its ability to capture local structural information of entities is poor. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a knowledge graph completion method called Comphormer, which integrates GNN and global attention mechanisms. By fusing learning between the GNN structure extraction module and the Transformer global perception embedding module, GNN and Transformer can fully leverage their respective advantages. The global perception capability of Transformer solves the defect of GNN in capturing information about distant neighbors, while the powerful local structure extraction capability of GNN solves the defect of Transformer in easily losing local structural information of entities. This further optimizes the model's ability to express knowledge graphs and improves the effect of knowledge graph completion.

[0010] The technical problem solved by this invention is achieved through the following technical solution:

[0011] A knowledge graph completion method integrating GNN and global attention mechanism includes the following steps:

[0012] Step 1: Input the knowledge graph into the Comphormer model. The Comphormer model consists of a GNN structure extraction module and a Transformer global awareness embedding module. After the knowledge graph is input into the Comphormer model, it first enters the GNN structure extraction module. The GNN structure extraction module introduces reverse relation and self-loop relation types and performs initial embedding to obtain the initial representation of entities and relations.

[0013] Step 2: Integrate the degree structure encoding into the entity initialization embedding; the relation initial embedding is represented by the initialized basis vectors.

[0014] Step 3: Based on the selected attention operator, use the CompGAT layer in the GNN structure extraction module to aggregate and update the initial embeddings of entities and relations, and obtain entity and relation embeddings containing local structural information of entities, which are used as the output of the GNN structure extraction module.

[0015] Step 4: Input the output of the GNN structure extraction module into the Transformer global perception embedding module. In this module, the multi-relation fusion degree encoding is integrated into the entity embedding, and the edge features of each edge in the knowledge graph are calculated to obtain the entity embedding with multi-relation fusion degree encoding and the edge features of each edge.

[0016] Step 5: In the Transformer global awareness embedding module, calculate the spatial location encoding and edge feature encoding between each pair of entities, and add the spatial location encoding and edge feature encoding as bias terms to the attention matrix. Calculate the attention matrix for each attention head in each layer.

[0017] Step 6: Based on the obtained attention matrix, update the entity embedding after multi-relation fusion degree encoding according to the data processing flow of the standard Transformer global awareness embedding module, and perform LayerNorm and residual linking; the entity embedding updated by the Transformer global awareness embedding module is used as the output of the Transformer global awareness embedding module; the entity embedding output by the Transformer global awareness embedding module and the relation embedding output by the GNN structure extraction module are used as the final entity and relation embeddings output by the Transformer global awareness embedding module.

[0018] Step 7: Input the entity embeddings and relation embeddings output by the Transformer global awareness embedding module into the scoring function module, calculate the scores of triples in the knowledge graph, and evaluate the quality of the entity embeddings and relation embeddings output by the Transformer global awareness embedding module.

[0019] Moreover, the specific implementation method for obtaining the initial representation of entities and relations in step 1 is as follows: In the GNN structure extraction module of the Transformer global awareness embedding module, reverse relation type and self-loop relation type are introduced to enrich the edge type and initialize the embedding to obtain the initial representation of entities and relations.

[0020] Moreover, the specific implementation method of step 2 is as follows: after initializing the knowledge graph, degree structure encoding is introduced to enrich the representation of entities, and the initial embedding of entities containing graph centrality information is obtained; the initial embedding of relations is represented by the initialized basis vectors, so that the number of parameters is reduced.

[0021] Moreover, the specific implementation method of step 3 is as follows: This module designs three attention operators, namely Add, Multiply, and Concatenate. Different operators can be selected according to different knowledge graphs to achieve the best effect. Based on the selected attention operator, the CompGAT layer in the GNN structure extraction module is used to aggregate and update the initial embeddings of entities and relations, so as to obtain entity embeddings and relation embeddings containing local structural information of the knowledge graph, which are used as the output of the GNN structure extraction module.

[0022] Moreover, the specific implementation method of step 4 is as follows: input the output of the GNN structure extraction module into the Transformer global perception embedding module, calculate the multi-relation fusion degree encoding of the entity and incorporate it into the entity embedding, so as to obtain the entity embedding containing graph centrality and multi-relation centrality information; calculate the edge features of each edge for use in the edge feature encoding calculation in step 5.

[0023] Moreover, the specific implementation method of step 5 is as follows: calculate the spatial position code between entities based on the shortest path distance and the shortest resistance distance between each pair of entities, calculate the edge feature code between entities based on the shortest path between each pair of entities and the edge features of each edge on the shortest path, and use these two codes as bias terms for the attention matrix calculation. The spatial position code enables the model to perceive the relative position information between entities, and the edge feature code can embed the relationship into the attention mechanism.

[0024] Moreover, the specific implementation method of step 6 is as follows: calculate the attention matrix of each attention head in each layer of Transformer, update the entity embedding according to the standard Transformer global awareness embedding module process, and perform LayerNorm and residual linking; after being updated by the Transformer global awareness embedding module, an entity embedding that simultaneously contains local structural information, global relative position information and relational information can be obtained.

[0025] Furthermore, the specific implementation method of step 7 is as follows: A ConvE model is used as the scoring function. The embedded representations of entities and relations are reshaped into a two-dimensional image, and convolutional layers are used to capture local features between entities and relations. Then, a fully connected layer is used to output the score of the triples. For each positive triple, a set of negative triples is generated by randomly replacing the head and tail entities. The positive triple includes (e... i , r k , e j Negative sample triples include (e i ',r k ',e j The model is trained to distinguish between real and incorrect triples by comparing the scores of positive and negative samples, and the results are used to calculate the loss function and perform backpropagation, thus completing one round of model training.

[0026] The advantages and positive effects of this invention are:

[0027] This invention designs a knowledge graph completion method that integrates GNN and global attention mechanisms, namely the Transformer Global Awareness Embedding Module. The Transformer Global Awareness Embedding Module consists of a GNN structure extraction module and a Transformer Global Awareness Embedding Module. By inputting the knowledge graph into the GNN structure extraction module, inverse and self-loop relation types are introduced, and degree structure encoding is applied to obtain initial embeddings of entities and relations. Then, the structural information of the knowledge graph is extracted using the graph attention mechanism to obtain entity and relation embeddings containing local structural information of the entities. These are then input into the Transformer Global Awareness Embedding Module to calculate the entity embedding after fusing multi-relation degree encoding, and to calculate the spatial location encoding. Code and edge feature encodings are added as bias terms to the attention matrix calculation, and the entity embeddings are updated according to the standard Transformer global awareness embedding module processing flow. This yields an entity embedding representation that simultaneously contains local structural information, global relative position information, and relational information, which serves as the output of the Transformer global awareness embedding module. The entity embeddings output by the Transformer global awareness embedding module and the relation embeddings output by the GNN structure extraction module are used as the final entity and relation embeddings output by the Transformer global awareness embedding module. The entity and relation embeddings output by the Transformer global awareness embedding module are then input into a scoring function for embedding quality evaluation and knowledge graph completion. This invention further optimizes the deep learning model's ability to represent knowledge graphs. The Transformer global awareness embedding module can simultaneously capture local structural information and global information in the knowledge graph, improving the effectiveness of knowledge graph completion. Attached Figure Description

[0028] Figure 1 is a diagram of the overall architecture of the Transformer global awareness embedding module of the present invention;

[0029] Figure 2 is an architecture diagram of the GNN structure extraction module of the present invention;

[0030] Figure 3 is an architecture diagram of the Transformer global awareness embedding module of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the accompanying drawings.

[0032] The knowledge graph completion method disclosed in this invention consists of a Transformer global awareness embedding module and a scoring function. The Transformer global awareness embedding module comprises a GNN structure extraction module and a Transformer global awareness embedding module. The overall architecture is as follows: Figure 1 As shown.

[0033] For the GNN structure extraction module of the Transformer global awareness embedding module, entities and relations are the basic units constituting the graph structure in a knowledge graph. This invention requires a model that can simultaneously handle entities and relations. Furthermore, to effectively extract structural information from the knowledge graph, this invention designs a novel attention-based GNN, namely CompGAT, as the basic framework for the GNN structure extraction module. To adapt the module to the specific needs of local structure extraction and further improve the model's performance on knowledge graph completion tasks, CompGAT introduces degree structure encoding and integrates an attention mechanism, designing three different attention combination operators. CompGAT can not only encode entity information but also perform joint embedding of relations, generating entity and relation embeddings containing local structural properties, providing feature representations for subsequent tasks. First, this invention introduces reverse relations and self-looping relation types into the GNN structure extraction module and performs initial embedding. Next, degree structure encoding is introduced, using the out-degree and in-degree of nodes as the measure of centrality, and integrating centrality information into the entity embedding using direct addition. Next, the entity embeddings encoded by the fusion degree structure and the relation embeddings represented by basis vectors are input into a multi-layer CompGAT for aggregation and updating. Before aggregating the entity and relation embeddings, attention operators can be selected: Add, Multiply, and Concatenate. After aggregation and updating by the multi-layer CompGAT, entity embeddings and relation embeddings containing local structural information are obtained and used as the output of the GNN structure extraction module. The architecture of the GNN structure extraction module is as follows: Figure 2 .

[0034] For the Transformer Global Aware Embedding Module, this invention designs and introduces three encoding methods specifically for knowledge graph data, enabling the model to adapt well to such data. These three encoding methods are: multi-relation fusion degree encoding, spatial location encoding, and edge feature encoding. After inputting the output of the GNN structure extraction module into the Transformer entity embedding module, multi-relation fusion degree encoding is first introduced into the entity embedding to obtain entity embeddings containing graph centrality and multi-relation centrality information. The entity embedding matrix after incorporating multi-relation fusion degree encoding is then processed as the Query, Key, and Value matrices in the Transformer input. When calculating the attention matrix, spatial distance encoding and edge feature encoding are introduced as bias terms. Based on the shortest path distance and shortest resistance distance between each pair of entities, spatial location encoding is generated using an exponentially decaying mask and added as a bias term in the attention matrix calculation. Edge feature encoding comprehensively considers information about the entities linked by each edge and their relation types. First, the edge features of each edge are calculated, derived by fusing the embedding of the relation corresponding to the edge with the embeddings of the head and tail entities it connects to. After obtaining the edge features, the edge encoding between each entity pair is calculated. For each entity pair, a shortest path is found, and the average of the sum of the dot products of the edge features of each edge on that path and the corresponding learnable embeddings is calculated to obtain the edge encoding between each entity pair. Finally, the obtained edge encoding is added to the attention matrix calculation using a bias term. By introducing spatial distance encoding and edge feature encoding, a new attention matrix calculation method is obtained. Finally, the entity embeddings after fusing multi-relation fusion degree encoding are input into the Transformer global awareness embedding module for processing. The resulting entity embedding representation is used as the output of the Transformer global awareness embedding module. The architecture of the Transformer global awareness embedding module is as follows: Figure 3 .

[0035] In this invention, the scoring function is responsible for calculating the score of triples in the knowledge graph to evaluate the quality of entity and relation embeddings output by the Transformer global awareness embedding module. This invention uses ConvE as the scoring function, mapping the embeddings of the scoring function triples to a score that represents the probability of the triple. For each positive sample triple, a set of negative sample triples is generated by randomly replacing the head and tail entities. The embeddings of the positive and negative samples are input into the ConvE model to obtain the score for each triple. By comparing the scores of the positive and negative samples, the model is trained to distinguish between true and false triples, and this is used to calculate the loss function and perform backpropagation, thus completing the model training.

[0036] In this application, the structured triples of a knowledge graph are represented as follows: , where e i Called the head entity, e j Called the tail entity, r k Represents entity e i and e j The relationship between them, E = {e1, e2, ..., e M Let} be the entity set, R = {r1, r2, ..., r N} represents the set of relations between entities, where entity e is an entity. i Relationship r k The corresponding embedding vector has a dimension of d.

[0037] A knowledge graph completion method integrating GNN and global attention mechanism includes the following steps:

[0038] Step 1: Input the knowledge graph into the GNN structure extraction module of the Transformer global perception embedding module and preprocess it.

[0039] The knowledge graph to be completed is divided into three parts: a training set, a validation set, and a test set. Reverse relation types and self-loop relation types are introduced into the knowledge graph to enrich the edge types, and an initial representation of the knowledge graph is generated. Edges of reverse relation and self-loop relation types are only used in the GNN structure extraction module; these edges do not actually exist and are only used for message passing between entities. In this module, entity e... i and relation r k The embedding representations are respectively , Their initial representations after initialization are respectively , After initialization, the initial representation of the knowledge graph from the training set is input into the GNN structure extraction module of the Transformer global perception embedding module to extract the local structural information of entities.

[0040] Step 2: Introduce degree structure encoding in entity embedding and represent relation embedding using basis vectors.

[0041] In the GNN structure extraction module, degree structure encoding is first introduced to enrich the structural representation of entities. The out-degree and in-degree of an entity are used as measures of centrality, and the degree structure encoding containing centrality information is integrated into the entity embedding. Specifically, for each entity, the in-degree encoding and out-degree encoding are calculated based on its in-degree and out-degree, respectively. Then, these encodings are added to the initial entity embedding vector to obtain the entity embedding containing centrality information. The initial entity embedding after incorporating the degree structure encoding containing centrality information can be represented as:

[0042] (1)

[0043] in, For initial embedding of entities, , It is based on entity e i The learnable embedding vectors obtained from the in-degree and out-degree of entities do not include the edges of inverse and self-loop relationships introduced in step 1 when calculating the in-degree and out-degree of entities; they only include the edges originally present in the knowledge graph. To allow the model to scale with the number of relationships, a basis vector method is used to represent the relationship embeddings, with the initial relationship embeddings as follows:

[0044] (2)

[0045] in, For the initial embedding of the relation, It is a learnable coefficient vector, v b is the basis vector of relation embedding, and B is the number of basis vectors.

[0046] Step 3: Aggregate and update to obtain the output of the GNN structure extraction module.

[0047] By selecting an appropriate attention operator, entity and relation embeddings are input into a multi-layered CompGAT for aggregation and update. The embedding and update formulas for entities and relations in the CompGAT layer are as follows:

[0048] (3)

[0049] (4)

[0050] (5)

[0051] in, It is entity e i The next layer of embedding representation, f(...) is an aggregation function, e j For entity e i In relation r k The neighboring entity, e m For entity e i In relation r n The neighboring entities below, For neighboring entity e j The embedded representation of this layer, For neighboring entity e j The corresponding relation r k The embedded representation of this layer, That is, there are 3 types of relations (existing relation R, reverse relation R) inv Self-circulating relation R self-loopIn each layer's respective projection matrix, Ψ is a fusion operator based on an attention mechanism. This invention designs three different attention combination mechanisms, allowing for selectable combinations of entity and relation representations: Add, Multiply, and Concatenate. The specific formulas are as follows:

[0052] (6)

[0053] After multiple layers of CompGAT aggregation and updating, entity embeddings and relation embeddings containing local structural information of the knowledge graph are obtained, which serve as the output of the GNN structure extraction module.

[0054] Step 4: Input the output of the GNN structure extraction module into the Transformer global perception embedding module, introduce multi-relation fusion degree encoding and calculate the edge features of each edge.

[0055] The output of the GNN structure extraction module is input into the Transformer global perception embedding module, and a multi-relation fusion degree encoding is first introduced. Considering the diversity of relation types connected to each entity in the knowledge graph, this invention proposes a multi-relation fusion degree encoding that can capture the connection patterns of entities under different relation types. Specifically, the in-degree and out-degree of the entity under different relations are first calculated, and a learnable in-degree embedding vector and a learnable out-degree embedding vector are assigned to each relation. The in-degree and out-degree encodings of the entity under each relation are then calculated. The weighted average of the in-degree and out-degree encodings under each relation is then obtained to obtain the final multi-relation fusion degree encoding of the entity. Finally, the encoding is added to the entity embedding to obtain the entity embedding containing graph centrality and multi-relation centrality information. The formula is as follows:

[0056] (7)

[0057] in, The entity embedding vector is the output of the GNN structure extraction module, where N is the number of relations. and It is based on entity e i In relation r k The learnable embedding vectors obtained from the in-degree and out-degree are given below. This is for embedding entities after integrating multi-relational fusion degree encoding.

[0058] Considering that knowledge graphs not only contain a large number of entities but also edges connecting these entities—that is, relationships between entities—this invention proposes an edge encoding method, namely edge feature encoding, to more effectively encode edge information in the attention layer. Edge feature encoding comprehensively considers the embeddings of head and tail entities and relationships. It aims to capture the correlation between connected entities and relationships and integrate it into the attention mechanism to improve the model's expressive ability of the knowledge graph. This method comprehensively considers the embeddings of head and tail entities and relationships. First, it needs to calculate the edge features of each edge in the knowledge graph. Specifically, for each edge in the knowledge graph, the embedding of the relationship corresponding to that edge is combined with the embeddings of its connected head and tail entities to calculate the edge feature d of each edge in the knowledge graph. ijk Edge feature d ijk The calculation formula is as follows:

[0059] (8)

[0060] (9)

[0061] Where, r k h i and h j The relationship r between the outputs of the GNN structure extraction module and the outputs of the module are respectively k Relational embedding and relation r k The entity embeddings of the link's head and tail are obtained by multi-relation fusion degree encoding, g ijk To integrate the head and tail entities and relationships on the edges, W1 and W2 are weight matrices. The obtained edge features will be used for subsequent calculation of edge feature encoding between entities.

[0062] Step 5: Calculate the attention matrix.

[0063] When calculating the attention matrix, this invention adds two additional encoding methods: spatial location encoding and edge feature encoding.

[0064] First, spatial location encoding. To encode the spatial information of the knowledge graph in the model, this invention uses an encoding method based on the shortest resistance distance and the shortest path distance. Specifically, for a knowledge graph G, consider a function... Used to measure entity e in knowledge graph G i and e j Spatial relationships between them. Functions It can be defined by the connectivity between entities in the graph. If two nodes are connected, select... For e i and e j Choose the shortest resistance distance between them. For e iand e j The shortest path distance between them; if no distance exists, then... and The output is set to a special value, -1. Next, for and Each output value is accompanied by an exponentially decaying mask, which decreases exponentially as the distance between entities in the graph structure decreases. The calculation... and After encoding the two spatial locations, they are used as bias terms in the attention calculation. Let A be the attention matrix, then the calculation formula is as follows:

[0065] (10)

[0066] Among them, h i and h j To integrate entity embeddings after multi-relation fusion degree encoding, W Q and W K This is the weight matrix. and For different distance encoding functions, respectively by entity e i and e j The shortest resistance distance and shortest path distance are obtained by adding an exponential decay mask, and this is shared across all layers. This indicates element-wise multiplication.

[0067] Next is edge feature encoding, for each entity pair (e i , e j ), found a connection e i and e j Shortest path SP ij = (p1, p2, ..., p N Find the edge features d of all edges on the path. ijk Next, we calculate the average of the sum of the dot products of these edge features and their corresponding learnable embedding vectors to obtain the edge encoding c. ij Finally, the edge encoding c obtained in the attention calculation is... ij As a bias term, the updated attention matrix is ​​calculated as follows:

[0068] (11)

[0069] in, It is SP ij The nth edge e n Edge features, It is the nth weighted embedding, where N is the entity e. i and e j The shortest path distance between them.

[0070] Step 6: Calculate the output of the Transformer Global Aware Embedding Module.

[0071] After calculating the attention matrix for each attention head in each layer, the entity embedding matrix after fusing multi-relation fusion degree encoding is processed as the Query, Key, and Value matrices in the Transformer input, following the standard Transformer global awareness embedding module processing flow. The entity embedding is updated, and LayerNorm and residual connections are performed. The entity embedding updated by the Transformer global awareness embedding module is used as the output of the Transformer entity embedding module. The entity embedding output by the Transformer entity embedding module and the relation embedding output by the GNN structure extraction module are used as the final entity and relation embeddings output by the Transformer global awareness embedding module.

[0072] Step 7: Input the output of the Transformer global awareness embedding module into the scoring function module to evaluate and complete the knowledge graph.

[0073] The output of the Transformer global awareness embedding module is input into the scoring function, which is responsible for calculating the score of triples in the knowledge graph to evaluate the quality of entity and relation embeddings output by the Transformer global awareness embedding module. This invention uses a ConvE model as the scoring function, reshaping the embedded representations of entities and relations into two-dimensional images, using convolutional layers to capture local features between entities and relations, and then outputting the triple score through a fully connected layer. The specific scoring process is as follows: The output of the Transformer global awareness embedding module is input into the scoring function; for each positive sample triple (e... i , r k , e j ), randomly replace the first and last entities to generate a set of negative sample triples (e i ',r k ',e j The embeddings of positive and negative samples are input into the ConvE model to obtain a score for each triple. By comparing the scores of positive and negative samples, the model is trained to distinguish between true and false triples, and the scores are used to calculate the loss function and perform backpropagation, thus completing one round of model training. The loss function uses binary cross-entropy loss, and the optimization algorithm uses the Adam optimizer.

[0074] Next, the validation set of the knowledge graph is fed into the Transformer global awareness embedding module to calculate entity and relation embeddings. The embedding results are then fed into the evaluation function to obtain the evaluation results for each round. The evaluation function will calculate the following metrics:

[0075] Mean Reciprocal Rank (MRR): MRR is a commonly used metric to measure model performance, used to evaluate link prediction effectiveness in knowledge graph completion tasks. It considers the ranking of candidate answers. MRR calculates the average of the reciprocal ranks of the correct answers. Specifically, for each question, all candidate answers are ranked according to the model's predicted score, then the reciprocal rank of the correct answer (i.e., 1 / rank) is calculated, and finally, the average of the reciprocal ranks of all questions is used as the MRR value. A higher MRR value indicates that the model performs better in ranking the correct answers. The MRR calculation formula is as follows:

[0076] (12)

[0077] Where test is the test set, and rank is the rank set. h and rank t These represent the ranking of the test triplet in the set of candidate tuples that replace the head and tail entities, respectively.

[0078] Hits@N: The Hits@N metric measures whether the model produces the correct answer among the top N candidate answers. It considers the proportion of the head / tail entities of the effective test triples that rank among the top N in the overall entity set score. N represents a positive integer, typically 1, 3, or 10. The significance of Hits@N lies in measuring whether the model can rank the correct answer within the top N positions. A higher Hits@N value indicates higher model accuracy. The formula for calculating Hits@N is as follows:

[0079] (13)

[0080] Where C (...) is a conditional function, which has a value of 1 when the condition is true and 0 otherwise.

[0081] After obtaining the MRR metric from this round of evaluation, the model with the best performance is retained compared to the MRR metric from the previous round. After multiple rounds of training, the parameters of the best-performing Transformer global awareness embedding module are obtained. The final performance evaluation metric of this model is obtained by feeding the knowledge graph test set into the Transformer global awareness embedding module with these parameters.

[0082] Once the optimal model is obtained, it can be used to complete the knowledge graph. For each triple to be predicted, the embeddings of its head and tail entities and relations are fed into the model to obtain the score of the triple. The score is compared with the set score threshold. Triples that are higher than the threshold are the missing triples predicted by the model. After all the triples to be predicted are processed by the model, the final set of predicted triples can be obtained, thus completing the knowledge graph completion task.

[0083] Based on the aforementioned knowledge graph completion method that integrates GNN and global attention mechanisms, Python was primarily used as the programming language. The entire model was designed using the open-source deep learning framework PyTorch and the open-source graph deep learning library DGL. To evaluate the performance of the Transformer global awareness embedding module, link prediction experiments for knowledge graph completion were conducted on two knowledge graph datasets: WN18RR and FB15K237. Table 1 provides detailed information for both datasets.

[0084] Table 1 Dataset

[0085]

[0086] Table 2 presents the experimental results. This invention conducted link prediction experiments on the WN18RR and FB15K237 knowledge graph datasets and compared them with other representative completion methods. The experimental results show that, compared with existing models, this invention significantly outperforms knowledge graph completion methods based on GNN and TransE in terms of evaluation metrics. This indicates that the fusion of GNN and Transformer global awareness embedding modules can significantly improve the knowledge graph completion effect during the knowledge graph embedding representation learning process. In summary, the method proposed in this invention effectively improves the model's performance in knowledge graph completion through the fusion learning of GNN and Transformer global awareness embedding modules, and provides new ideas for further research on knowledge graph completion. This research has significant theoretical and practical value and has a positive impact on promoting the development of knowledge graph completion models.

[0087] Table 2 Experimental Results

[0088]

[0089] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.

Claims

1. A knowledge graph completion method integrating GNN and global attention mechanism, characterized in that: Includes the following steps: Step 1: Input the knowledge graph into the Comphormer model. The Comphormer model consists of a GNN structure extraction module and a Transformer global awareness embedding module. After the knowledge graph is input into the Comphormer model, it first enters the GNN structure extraction module. The GNN structure extraction module introduces reverse relation and self-loop relation types and performs initial embedding to obtain the initial representation of entities and relations. Step 2: Integrate the degree structure encoding into the entity initialization embedding; the relation initial embedding is represented by the initialized basis vectors. Step 3: Based on the selected attention operator, use the CompGAT layer in the GNN structure extraction module to aggregate and update the initial embeddings of entities and relations, and obtain entity and relation embeddings containing local structural information of entities, which are used as the output of the GNN structure extraction module. Step 4: Input the output of the GNN structure extraction module into the Transformer global perception embedding module. In this module, the multi-relation fusion degree encoding is integrated into the entity embedding, and the edge features of each edge in the knowledge graph are calculated to obtain the entity embedding with multi-relation fusion degree encoding and the edge features of each edge. Step 5: In the Transformer global awareness embedding module, calculate the spatial location encoding and edge feature encoding between each pair of entities, and add the spatial location encoding and edge feature encoding as bias terms to the attention matrix. Calculate the attention matrix for each attention head in each layer. Step 6: Based on the obtained attention matrix, update the entity embedding after multi-relation fusion degree encoding according to the data processing flow of the standard Transformer global awareness embedding module, and perform LayerNorm and residual linking; the entity embedding updated by the Transformer global awareness embedding module is used as the output of the Transformer global awareness embedding module; the entity embedding output by the Transformer global awareness embedding module and the relation embedding output by the GNN structure extraction module are used as the final entity and relation embeddings output by the Transformer global awareness embedding module. Step 7: Input the entity embeddings and relation embeddings output by the Transformer global awareness embedding module into the scoring function module, calculate the scores of triples in the knowledge graph, and evaluate the quality of the entity embeddings and relation embeddings output by the Transformer global awareness embedding module.

2. The knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method for obtaining the initial representation of entities and relations in step 1 is as follows: In the GNN structure extraction module of the Transformer global perception embedding module, reverse relation type and self-loop relation type are introduced to enrich the edge type and initialize the embedding to obtain the initial representation of entities and relations.

3. The knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method of step 2 is as follows: after initializing the knowledge graph, degree structure encoding is introduced to enrich the representation of entities, and the initial embedding of entities containing graph centrality information is obtained; the initial embedding of relations is represented by the initialized basis vectors, so that the number of parameters is reduced.

4. The knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method of step 3 is as follows: This module designs three attention operators, namely Add, Multiply, and Concatenate. Different operators can be selected according to different knowledge graphs to achieve the best effect. Based on the selected attention operator, the CompGAT layer in the GNN structure extraction module is used to aggregate and update the initial embeddings of entities and relations, so as to obtain entity embeddings and relation embeddings containing local structural information of the knowledge graph, which are used as the output of the GNN structure extraction module.

5. The knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method of step 4 is as follows: input the output of the GNN structure extraction module into the Transformer global perception embedding module, calculate the multi-relation fusion degree encoding of the entity and integrate it into the entity embedding, so as to obtain the entity embedding containing graph centrality and multi-relation centrality information; calculate the edge features of each edge for the calculation of edge feature encoding in step 5.

6. The knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method of step 5 is as follows: calculate the spatial position code between entities based on the shortest path distance and the shortest resistance distance between each pair of entities, calculate the edge feature code between entities based on the shortest path between each pair of entities and the edge features of each edge on the shortest path, and use these two codes as bias terms for the attention matrix calculation. The spatial position code enables the model to perceive the relative position information between entities, and the edge feature code can embed the relationship into the attention mechanism.

7. The knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method of step 6 is as follows: calculate the attention matrix of each attention head in each layer of Transformer, update the entity embedding according to the standard Transformer global awareness embedding module process, and perform LayerNorm and residual linking; after the Transformer global awareness embedding module updates, an entity embedding that simultaneously contains local structural information, global relative position information and relational information can be obtained.

8. The knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method of step 7 is as follows: A ConvE model is used as the scoring function. The embedded representations of entities and relations are reshaped into a two-dimensional image, and convolutional layers are used to capture local features between entities and relations. Then, a fully connected layer outputs the score of the triplet. For each positive triplet, a set of negative triplets is generated by randomly replacing the head and tail entities. The positive triplet includes (e... i , r k , e j Negative sample triples include (e i ',r k ',e j The model is trained to distinguish between real and incorrect triples by comparing the scores of positive and negative samples, and the results are used to calculate the loss function and perform backpropagation, thus completing one round of model training.

Images