Knowledge graph completion method fusing GNN and global attention mechanism

Through the knowledge graph completion method that integrates GNN and global attention mechanism, Compphormer, combined with GNN's local structure extraction and Transformer's global perception, solves the shortcomings of existing methods in complex relationship processing and information perception capabilities, and improves the knowledge graph Completion effect.

CN118966327AActive Publication Date: 2024-11-15TIANJIN POLYTECHNIC UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411190456.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-15
Estimated Expiration
2044-08-28

Smart Images

  • Figure CN118966327A_ABST
    Figure CN118966327A_ABST
Patent Text Reader

Abstract

The invention designs a knowledge graph completion method fusing GNN and a global attention mechanism, and the method comprises the steps: inputting a knowledge graph into a GNN structure extraction module of a Transform global perception embedding module, introducing a reverse relation and a self-circulation relation type, and introducing central coding to obtain the initial embedding of an entity and a relation; structural information of the mapping knowledge domain is extracted through a graph attention mechanism, and entities containing entity local structural information and relation embedding are obtained; the method comprises the following steps of: inputting an entity into a Transform global perception embedding module, calculating entity embedding after fusing multi-relation fusion degree codes, calculating spatial position codes and edge feature codes, adding the codes as offset terms into attention matrix calculation, processing according to a standard Transform global perception embedding module flow, and obtaining entity embedding representation as Transform global perception embedding module output; and the entities and the relationships processed by the Transform global perception embedding module are embedded and input into a scoring function for evaluation and complementation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of knowledge graph technology and is a knowledge graph completion method that integrates GNN and global attention mechanism. Background Art

[0002] In recent years, knowledge graphs have attracted great attention from academia and industry. Knowledge graphs are composed of entities, relationships, and semantic descriptions, and are a heterogeneous directed graph data structure. Existing knowledge graphs include Freebase, YAGO, WordNet, NELL, Wikidata, etc., which have been widely used in search engines, question-answering systems, dialogue systems, recommendation systems, knowledge reasoning, event prediction, etc. in the field of artificial intelligence. Since knowledge graphs are mostly constructed manually or semi-automatically, problems such as incomplete data and data sparsity often occur. To address these problems, the Knowledge Graph Completion (KGC) task came into being.

[0003] Knowledge graph completion aims to derive potential triples based on the existing triples in the knowledge graph, further improve the knowledge graph, and reduce the labor cost required to build the knowledge graph. Specifically, the knowledge graph completion task is currently mainly abstracted into a prediction problem, that is, predicting the missing parts of the triples, such as predicting the relationship between the head and tail entities when the head entity and the relationship are known; or predicting the tail entity when the head entity and the relationship are known; or predicting the head entity when the tail entity and the relationship are known. At present, the mainstream knowledge graph completion method is to obtain entity representations through knowledge graph embedding (KGE) and other methods, and use these representations to predict and infer missing entities and relationships. Knowledge graph embedding aims to map the elements (entities and relationships) in the knowledge graph into a low-dimensional continuous vector space, learn the embedded representation of entities and relationships, and at the same time maintain the intrinsic structure and semantic information of the knowledge graph, that is, convert the symbolic representation of the knowledge graph into a numerical representation, so as to achieve efficient knowledge graph completion.

[0004] At present, the most commonly used knowledge graph completion methods are the embedding method based on the translation distance model and the embedding method based on the graph neural network model.

[0005] The first is the translation distance model embedding method, which is based on a simple assumption: entities and relations can be matched by translating vectors. The earliest translation distance model was proposed by Bordes et al. in the paper "Translating Embeddings for Modeling Multi-relational Data" in 2013 as a knowledge representation learning model, namely the TransE model. This model maps entities and relations in the knowledge graph into a low-dimensional vector space, obtains the embedded vector representation of entities and relations, and uses the vector difference between the head and tail entities to represent the relationship between entities.

[0006] Next is the embedding method based on the graph neural network model. The graph neural network model (GNN) is a neural network that can process graph structured data and extract features by aggregating and updating the node information in the graph. The most basic GNN is the graph convolutional network GCN in the paper "Semi-Supervised Classification with Graph Convolutional Networks" proposed by Kipf and Welling et al. in 2017. GCN aggregates the data in the graph and updates the feature information of itself and the neighborhood through a mapping function. Since the basic GCN aggregates the neighborhood through a unified function, it is impossible to effectively distinguish different entities and relationships in the neighborhood. In order to process knowledge graph data of multiple relational types, scholars have successively proposed graph neural network models that comprehensively consider the entity and relationship information in the knowledge graph, such as the R-GCN model in "Modeling Relational Data with Graph Convolutional Networks" proposed by Schlichtkrull et al. in 2018 and the CompGCN model in "Composition-based Multi-Relational Graph Convolutional Networks" proposed by Vashishth et al. in 2020. Among them, R-GCN distinguishes different relations by assigning different weight matrices to each relation type, thereby separating the features of different relations and retaining relation-specific information to better capture the associations between entities; CompGCN further considers the complex interactions between entities and relations, and embeds both entities and relations in the knowledge graph.

[0007] However, both the translation distance model and the graph neural network model have obvious defects: the translation distance model has limitations in dealing with one-to-many, many-to-one and many-to-many complex relationships, is sensitive to hyperparameters, has limited scalability, cannot meet the needs of automatically adding new entities and new relationships, and is completely unable to perceive the information of non-adjacent entities; although the graph neural network model has a certain ability to capture the information of first-order and above neighbor entities of an entity, it is limited by the number of GNN layers (a maximum of 3 to 4 layers can be stacked, and too many layers will cause over-smoothing and over-squeezing problems). When aggregating, it often only cares about neighbor entities that are closer, and cannot perceive the information of neighbor entities at a greater distance, which seriously affects the effect of knowledge graph completion.

[0008] In recent years, Transformer has demonstrated strong performance in natural language processing (NLP), computer vision (CV), and artificial intelligence generated content (AIGC). Transformer has strong expressive power and the ability to capture global information, but few scholars have explored the application of Transformer in the graph field. There are two main reasons: (1) Graph data has an irregular structure and lacks natural order, which makes it difficult to directly apply Transformer to graph data. Knowledge graphs are a very complex heterogeneous graph data type, and applying Transformer to knowledge graph embedding will face huge challenges; (2) In knowledge graphs, the local structure of entities is very important information. When updating entity embeddings, Transformer pays too much attention to global information, which will result in the inability to effectively capture the information of directly adjacent entities. In other words, it has poor ability to capture local structural information of entities. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a knowledge graph completion method that integrates GNN and global attention mechanism - Comphormer. Through the fusion learning of GNN structure extraction module and Transformer global perception embedding module, GNN and Transformer can give full play to their respective advantages, use the global perception ability of Transformer to solve the defect that GNN cannot capture the information of distant neighbors, and use the powerful local structure extraction ability of GNN to solve the defect that Transformer easily loses the local structure information of entities, further optimize the model's expression ability of knowledge graph, and improve the effect of knowledge graph completion.

[0010] The present invention solves the technical problem by adopting the following technical solutions:

[0011] A knowledge graph completion method integrating GNN and global attention mechanism includes the following steps:

[0012] Step 1: Input the knowledge graph into the Comphormer model. The Comphormer model consists of a GNN structure extraction module and a Transformer global perception embedding module. After the knowledge graph is input into the Comphormer model, it first enters the GNN structure extraction module. The GNN structure extraction module introduces reverse relations and self-loop relation types, and performs initial embedding to obtain the initial representation of entities and relations.

[0013] Step 2: Incorporate the degree structure encoding into the entity initialization embedding, and the relationship initial embedding is represented by the initialized basis vector;

[0014] Step 3: According to the selected attention operator, the CompGAT layer in the GNN structure extraction module is used to aggregate and update the initial embeddings of entities and relations to obtain entity and relationship embeddings containing local structural information of entities as the output of the GNN structure extraction module;

[0015] Step 4: Input the output of the GNN structure extraction module into the Transformer global perception embedding module. In this module, the multi-relation fusion degree encoding is integrated into the entity embedding, and the edge features of each edge in the knowledge graph are calculated to obtain the entity embedding that integrates the multi-relation fusion degree encoding and the edge features of each edge.

[0016] Step 5: In the Transformer global perception embedding module, the spatial position encoding and edge feature encoding between each pair of entities are calculated, and the spatial position encoding and edge feature encoding are added as bias items to the attention matrix, and the attention matrix of each attention head in each layer is calculated;

[0017] Step 6: According to the obtained attention matrix, the entity embedding after fusion of multi-relation fusion degree encoding is updated according to the process of data processing of the standard Transformer global perception embedding module, and LayerNorm and residual link are performed; the entity embedding updated by the Transformer global perception embedding module is used as the output of the Transformer global perception embedding module; the entity embedding output by the Transformer global perception embedding module and the relation embedding output by the GNN structure extraction module are used as the entity and relation embeddings finally output by the Transformer global perception embedding module;

[0018] Step 7: Input the entity embedding and relation embedding output by the Transformer global perception embedding module into the scoring function module, calculate the score of the triples in the knowledge graph, and evaluate the quality of the entity embedding and relation embedding output by the Transformer global perception embedding module.

[0019] Moreover, the specific implementation method of obtaining the initial representation of entities and relationships in step 1 is: in the GNN structure extraction module in the Transformer global perception embedding module, reverse relationship types and self-loop relationship types are introduced to enrich the edge types, and initial embedding is performed to obtain the initial representation of entities and relationships.

[0020] Moreover, the specific implementation method of step 2 is: after initializing the knowledge graph, introducing degree structure encoding to enrich the representation of entities and obtain the initial embedding of entities containing graph centrality information; the initial embedding of relationships is represented by the initialized basis vectors to reduce the number of parameters.

[0021] Moreover, the specific implementation method of step 3 is: this module designs three attention operators, namely addition Add, multiplication Mul and concatenation Con, and different operators can be selected according to different knowledge graphs to achieve the best effect; based on the selected attention operator, the CompGAT layer in the GNN structure extraction module is used to aggregate and update the initial embeddings of entities and relationships, and the entity embedding and relationship embedding containing the local structure information of the knowledge graph can be obtained as the output of the GNN structure extraction module.

[0022] Moreover, the specific implementation method of step 4 is: input the output of the GNN structure extraction module into the Transformer global perception embedding module, calculate the multi-relation fusion degree encoding of the entity and integrate it into the entity embedding, and obtain the entity embedding containing graph centrality and multi-relation centrality information; calculate the edge features of each edge for the calculation of edge feature encoding in step 5.

[0023] Moreover, the specific implementation method of step 5 is: calculating the spatial position encoding between entities based on the shortest path distance and the shortest resistance distance between each pair of entities, calculating the edge feature encoding between entities based on the shortest path between each pair of entities and the edge feature of each edge on the shortest path, and using these two encodings as bias items for attention matrix calculation. The spatial position encoding enables the model to perceive the relative position information between entities, and the edge feature encoding can embed the relationship into the attention mechanism.

[0024] Moreover, the specific implementation method of step 6 is: calculate the attention matrix of each attention head of each layer of Transformer, update the entity embedding according to the standard Transformer global perception embedding module process, and perform LayerNorm and residual link; after the Transformer global perception embedding module is updated, an entity embedding containing local structural information, global relative position information, and relationship information can be obtained.

[0025] Moreover, the specific implementation method of step 7 is: using the ConvE model as the scoring function, reshaping the embedded representation of entities and relations into a two-dimensional image, and using a convolutional layer to capture the local features between entities and relations, and then outputting the score of the triplet through a fully connected layer, for each positive sample triplet, randomly replacing the head and tail entities to generate a set of negative sample triplets, where the positive sample triplet includes (e i , r k , e j ), the negative sample triples include (e i ',r k ',e j '), the model is trained to distinguish true triplets from false triplets by comparing the scores of positive and negative samples, and is used to calculate the loss function and perform back propagation, thus completing a round of model training.

[0026] The advantages and positive effects of the present invention are:

[0027] The present invention designs a knowledge graph completion method integrating GNN and global attention mechanism, namely Transformer global perception embedding module, which is composed of GNN structure extraction module and Transformer global perception embedding module; by inputting knowledge graph into GNN structure extraction module, introducing reverse relationship and self-loop relationship type and introducing degree structure encoding, the initialization embedding of entity and relationship is obtained; and by extracting the structural information of knowledge graph through graph attention mechanism, the entity and relationship embedding containing entity local structure information is obtained; and then input into Transformer global perception embedding module, the entity embedding after integrating multi-relation fusion degree encoding is calculated, and the spatial position encoding is calculated. The code and edge feature encoding are added as bias items in the attention matrix calculation, and the entity embedding is updated according to the standard Transformer global perception embedding module processing flow to obtain an entity embedding representation that contains local structural information, global relative position information, and relationship information at the same time, as the output of the Transformer global perception embedding module; the entity embedding output by the Transformer global perception embedding module and the relationship embedding output by the GNN structure extraction module are used as the entity and relationship embeddings finally output by the Transformer global perception embedding module; the entity and relationship embedding output by the Transformer global perception embedding module are input into the scoring function, and the embedding quality evaluation and knowledge graph completion are performed. The present invention further optimizes the deep learning model's ability to express knowledge graphs, and the Transformer global perception embedding module can simultaneously capture local structural information and global information in the knowledge graph, improving the effect of knowledge graph completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a diagram showing the overall architecture of the Transformer global perception embedding module of the present invention;

[0029] FIG2 is a diagram of the architecture of the GNN structure extraction module of the present invention;

[0030] FIG3 is a diagram of the architecture of the Transformer global perception embedding module of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below with reference to the accompanying drawings.

[0032] The knowledge graph completion method disclosed in the present invention is composed of a Transformer global perception embedding module and a scoring function, wherein the Transformer global perception embedding module is composed of a GNN structure extraction module and a Transformer global perception embedding module. The overall architecture is as follows Figure 1 shown.

[0033] For the GNN structure extraction module of the Transformer global perception embedding module, in the knowledge graph, entities and relationships are the basic units that constitute the graph structure. The present invention requires a model that can process entities and relationships at the same time. At the same time, in order to effectively extract the structural information in the knowledge graph, the present invention designs a new type of GNN based on the attention mechanism, namely CompGAT, as the basic framework of the GNN structure extraction module. In order to adapt the module to the specific needs of local structure extraction and further improve the performance of the model in the knowledge graph completion task, degree structure encoding is introduced in CompGAT, and the attention mechanism is integrated to design three different attention combination operators. CompGAT can not only encode entity information, but also jointly embed relationships, generate entity and relationship embeddings containing local structural properties, and provide feature representations for subsequent tasks. First, the present invention introduces reverse relationship and self-loop relationship types in the GNN structure extraction module, and initializes the embedding. Next, degree structure encoding is introduced, and the out-degree and in-degree of the node are used as the measure of centrality, and the centrality information is integrated into the embedding of the entity using direct addition. Next, the entity embedding encoded by the fusion structure and the relationship embedding represented by the basis vector will be input into the multi-layer CompGAT for aggregation and update. Before aggregating the entity and relationship embeddings, attention operators can be selected, which are addition Add, multiplication Mul and concatenation Con. After multi-layer CompGAT aggregation and update, the entity embedding and relationship embedding containing local structural information are obtained and used as the output of the GNN structure extraction module. The architecture of the GNN structure extraction module is as follows: Figure 2 .

[0034] For the Transformer global perception embedding module, the present invention designs and introduces three encodings for knowledge graph type data, so that the model can adapt well to knowledge graph type data. The three encodings are: multi-relation fusion degree encoding, spatial position encoding and edge feature encoding. After the output of the GNN structure extraction module is input into the Transformer entity embedding module, the multi-relation fusion degree encoding is first introduced into the entity embedding to obtain the entity embedding containing the graph centrality and multi-relation centrality information, and the entity embedding matrix after the fusion of the multi-relation fusion degree encoding is simultaneously processed as the Query, Key, and Value matrices in the Transformer input. When calculating the attention matrix, spatial distance encoding and edge feature encoding are introduced in the form of bias terms. According to the shortest path distance and shortest resistance distance between each pair of entities, the spatial position encoding is generated by using an exponential decay mask and added to the attention matrix calculation as a bias term. Edge feature encoding comprehensively considers the information of the entities linked by each edge and the type of relationship itself. First, the edge feature on each edge is calculated, which is calculated by fusion of the embedding of the relationship corresponding to the edge and the embedding of the head and tail entities connected to it; after obtaining the edge feature, the edge encoding between each entity pair is calculated. For each pair of entity pairs, a shortest path is found, and the average of the sum of the dot products of the edge features of each edge on the path and the corresponding learnable embedding is calculated to obtain the edge encoding between each pair of entities. Finally, the obtained edge encoding is added to the attention matrix calculation by means of a bias term. By introducing spatial distance encoding and edge feature encoding, a new attention matrix calculation method is obtained. Finally, the entity embedding after fusion of multi-relation fusion degree encoding is input into the Transformer global perception embedding module for processing, and the obtained entity embedding representation is used as the output of the Transformer global perception embedding module. The architecture of the Transformer global perception embedding module is as follows. Figure 3 .

[0035] As for the scoring function, in the present invention, the scoring function is responsible for calculating the score of the triples in the knowledge graph to evaluate the quality of the entity and relationship embedding output by the Transformer global perception embedding module. The present invention uses ConvE as a scoring function to map the embedding of the scoring function triple to a score that represents the possibility of the triple. For each positive sample triple, the head and tail entities are randomly replaced to generate a set of negative sample triplets, and the embeddings of the positive and negative samples are input into the ConvE model to obtain the score of each triple; the model is trained by comparing the scores of positive and negative samples to distinguish between true triplets and erroneous triplets, and used to calculate the loss function and perform back propagation, thereby completing the model training.

[0036] In this application, the structured triples of the knowledge graph are represented as , where e i It is called the head entity, e j is called the tail entity, r k Represents entity e i and e j The relationship between them is, E = {e1, e2, ..., e M} is an entity set, R = {r1, r2, ..., r N} is the relationship set between entities, entity e i , relationship k The corresponding embedding vector has dimension d.

[0037] A knowledge graph completion method integrating GNN and global attention mechanism includes the following steps:

[0038] Step 1: Input the knowledge graph into the GNN structure extraction module of the Transformer global perception embedding module and preprocess it.

[0039] The knowledge graph to be completed is divided into three parts: training set, validation set and test set. Inverse relationship type and self-loop relationship type are introduced into the knowledge graph to enrich the edge type, and the initial representation of the generated knowledge graph is initialized. The edges of reverse relationship type and self-loop relationship type are only used in the GNN structure extraction module. These edges do not really exist and are only used to pass messages between entities. In this module, entity e i and the relationship k The embedding representations are , , and their initial representations after initialization are , After initialization, the initial representation knowledge graph of the training set in the knowledge graph is input into the GNN structure extraction module of the Transformer global perception embedding module to extract the local structure information of the entity.

[0040] Step 2: Introduce degree structure encoding into entity embedding and use basis vectors to represent relation embedding.

[0041] In the GNN structure extraction module, degree structure coding is first introduced to enrich the structural representation of entities. The out-degree and in-degree of entities are used as the measure of centrality, and the degree structure coding containing centrality information is integrated into the entity embedding. Specifically, for each entity, the in-degree coding and out-degree coding are calculated according to its in-degree and out-degree respectively. Then, its in-degree coding and out-degree coding are added to the initial entity embedding vector to obtain the entity embedding containing centrality information. The initial entity embedding after integrating the degree structure coding containing centrality information can be expressed as:

[0042] (1)

[0043] in, is the initial embedding of the entity, , According to entity e i The learnable embedding vector is obtained by calculating the in-degree and out-degree of the entity. When calculating the in-degree and out-degree of the entity, the edges of the reverse relationship and self-loop relationship type introduced in step 1 are not included, and only the original edges in the knowledge graph are included. In order to expand the model as the number of relationships increases, the basis vector method is used to represent the relationship embedding, and the relationship is initially embedded as:

[0044] (2)

[0045] in, is the initial embedding of the relation, is the learnable coefficient vector, v b is the basis vector of relation embedding, and B is the number of basis vectors.

[0046] Step 3: Aggregate and update to obtain the output of the GNN structure extraction module.

[0047] Select appropriate attention operators and input entity and relation embeddings into multi-layer CompGAT for aggregation update. The embedding update formula for entities and relations in the CompGAT layer is as follows:

[0048] (3)

[0049] (4)

[0050] (5)

[0051] in, It is entity e i The embedding representation of the next layer, f(...) is the aggregation function, e j For entity e i In the relationship k The neighbor entity under e m For entity e i In the relationship n The neighbor entities below, is the neighbor entity e j The embedding representation of this layer is is the neighbor entity e j The corresponding relationship r k The embedding representation of this layer is That is, there are three types of relationships (original relationship R, reverse relationship R inv , self-circular relation R self-loop) The respective projection matrices in this layer, Ψ is a fusion operator based on the attention mechanism. The present invention designs three different attention combination mechanisms to make the combination of entity and relationship representations optional, namely, Add, Multiply and Concatenate. The specific formula is as follows:

[0052] (6)

[0053] After multi-layer CompGAT aggregation and updating, entity embedding and relationship embedding containing local structural information of the knowledge graph are obtained as the output of the GNN structure extraction module.

[0054] Step 4: Input the output of the GNN structure extraction module into the Transformer global perception embedding module, introduce multi-relation fusion degree encoding and calculate the edge features of each edge.

[0055] The output of the GNN structure extraction module is input into the Transformer global perception embedding module, and multi-relation fusion degree coding is first introduced. Taking into account the diversity of relationship types connected to each entity in the knowledge graph, the present invention proposes a multi-relation fusion degree coding, which can capture the connection pattern of entities under different relationship types. Specifically, the in-degree and out-degree of the entity under different relationships are first calculated, and a learnable in-degree embedding vector and a learnable out-degree embedding vector are assigned to each relationship, and the in-degree and out-degree coding of the entity under each relationship are calculated, and then the in-degree and out-degree coding of each relationship are weighted and averaged to obtain the final multi-relation fusion degree coding of the entity. Finally, the coding is added to the entity embedding to obtain the entity embedding containing graph centrality and multi-relation centrality information. The formula is as follows:

[0056] (7)

[0057] in, is the entity embedding vector output by the GNN structure extraction module, N is the number of relations, and According to entity e i In the relationship k The learnable embedding vector obtained by the in-degree and out-degree under Entity embedding after fusion of multiple relation fusion degree encoding.

[0058] Considering that the knowledge graph not only contains a large number of entities, but also the edges connecting these entities, that is, the relationships between entities, in order to more effectively encode the edge information in the knowledge graph in the attention layer, the present invention proposes an edge encoding method, namely edge feature encoding. Edge feature encoding comprehensively considers the embedding of head and tail entities and relationships. Edge feature encoding aims to capture the correlation between connected entities and relationships, and integrate it into the attention mechanism to improve the model's expressiveness of the knowledge graph. This method comprehensively considers the embedding of head and tail entities and relationships. First, it is necessary to calculate the edge features of each edge in the knowledge graph. Specifically, for each edge in the knowledge graph, the embedding of the relationship corresponding to the edge is combined with the embedding of the head and tail entities connected to it to calculate the edge feature d of each edge in the knowledge graph. ijk . Edge feature d ijk The calculation formula is as follows:

[0059] (8)

[0060] (9)

[0061] Among them, r k 、h i and h j They are the relations r output by the GNN structure extraction module k Relational embedding and relational k The head and tail entity embeddings of the link are encoded by multi-relation fusion, g ijk In order to integrate the fusion embedding of the head and tail entities and relations on the edge, W1 and W2 are weight matrices respectively. The obtained edge features will be used for the subsequent calculation of edge feature encoding between entities.

[0062] Step 5: Calculate the attention matrix.

[0063] When calculating the attention matrix, the present invention adds two additional encodings, namely spatial position encoding and edge feature encoding.

[0064] The first is spatial position encoding. In order to encode the spatial information of the knowledge graph in the model, the present invention uses an encoding method based on the shortest resistance distance and the shortest path distance. Specifically, for a knowledge graph G, consider a function , used to measure entity e in the knowledge graph G i and e j The spatial relationship between . Function It can be defined by the connectivity between entities in the graph. If two nodes are connected together, select for e i and e j The shortest resistance distance between for e iand e j The shortest path distance between them. If the distance does not exist, and The output of is set to a special value, namely -1. Next, and Each output value of adds an exponential decay mask that decreases exponentially as the distance between entities in the graph structure decreases. and After the two spatial positions are encoded, they are used as bias items in the attention calculation. Let A be the attention matrix, and the calculation formula is as follows:

[0065] (10)

[0066] Among them, h i and h j W is the entity embedding after fusion of multi-relation fusion degree encoding, Q and W K is the weight matrix, and are different distance encoding functions, respectively, by entity e i and e j The shortest resistance distance and the shortest path distance are obtained by adding an exponential decay mask and shared among all layers. Represents element-wise multiplication.

[0067] Next is the edge feature encoding. For each entity pair (e i , e j ), find a connection e i and e j The shortest path SP ij = (p1, p2, ..., p N ), and find the edge characteristics d of all edges on the path ijk Next, we take the average of the sum of the dot products of these edge features and the corresponding learnable embedding vectors to obtain the edge code c ij Finally, the edge code c is obtained in the attention calculation ij As a bias term. The updated attention matrix calculation formula is as follows:

[0068] (11)

[0069] in, It is SP ij The nth edge e n The edge features of is the nth weight embedding, N is the entity e i and e j The shortest path distance between them.

[0070] Step 6: Calculate the output of the Transformer global perception embedding module.

[0071] After calculating the attention matrix of each attention head in each layer, the entity embedding matrix after fusion of multi-relation fusion degree encoding is processed as the Query, Key, and Value matrices in the Transformer input according to the standard Transformer global perception embedding module processing flow, the entity embedding is updated, and LayerNorm and residual links are performed; the entity embedding updated by the Transformer global perception embedding module is used as the output of the Transformer entity embedding module; the entity embedding output by the Transformer entity embedding module and the relation embedding output by the GNN structure extraction module are used as the entity and relation embeddings finally output by the Transformer global perception embedding module.

[0072] Step 7: Input the output of the Transformer global perception embedding module into the scoring function module to evaluate and complete the knowledge graph.

[0073] The output of the Transformer global perception embedding module is input into the scoring function, which is responsible for calculating the score of the triples in the knowledge graph to evaluate the quality of the entity and relationship embedding output by the Transformer global perception embedding module. The present invention adopts the ConvE model as the scoring function, which reshapes the embedding representation of entities and relationships into a two-dimensional image, uses a convolutional layer to capture the local features between entities and relationships, and then outputs the score of the triples through a fully connected layer. The specific scoring process is as follows: the output of the Transformer global perception embedding module is input into the scoring function; for each positive sample triple (e i , r k , e j ), randomly replace the head and tail entities to generate a set of negative sample triplets (e i ',r k ',e j '), the embedding of positive and negative samples is input into the ConvE model to obtain the score of each triple; the model is trained to distinguish true triples from false triples by comparing the scores of positive and negative samples, and used to calculate the loss function and perform back propagation, thus completing a round of model training. Among them, the loss function uses binary cross entropy loss, and the optimization algorithm uses the Adam optimizer.

[0074] Next, the validation set of the knowledge graph is fed into the Transformer global perception embedding module to calculate entity and relationship embeddings, and the embedding results are fed into the evaluation function to obtain the evaluation results of each round. The evaluation function will calculate the following indicators:

[0075] Mean Reciprocal Rank (MRR): MRR is a commonly used indicator to measure model performance. It is used to evaluate the link prediction effect in the knowledge graph completion task. The factor it considers is the ranking of candidate answers. MRR calculates the mean of the reciprocal rankings of the correct answers. Specifically, for each question, all candidate answers are ranked according to the scores predicted by the model, and then the reciprocal ranking of the correct answer (i.e. 1 / ranking) is calculated. Finally, the mean of the reciprocal rankings of all questions is used as the MRR value. A higher MRR value indicates that the model performs better in ranking correct answers. The MRR calculation formula is as follows:

[0076] (12)

[0077] Among them, test is the test set, rank h and rank t They respectively represent the ranking of the test triple in the set of candidate tuples consisting of replacing the head entity and the tail entity.

[0078] Hit rate Hits@N: The Hits@N indicator measures whether the model has the correct answer in the top N candidate answers. The factor considered is the proportion of the head entity / tail entity of the effective test triple in the top N of all entity set scoring rankings. N represents a positive integer, usually 1, 3, or 10. The significance of Hits@N is to measure whether the model can rank the correct answer in the top N. A higher Hits@N value indicates a higher accuracy of the model. The calculation formula for Hits@N is as follows:

[0079] (13)

[0080] Where C (…) is a conditional function, which takes a value of 1 when the condition is met and 0 otherwise.

[0081] After obtaining the MRR index of this round of evaluation results, the model with the best effect is retained compared with the MRR index in the previous round of results. After multiple rounds of training, the best performing Transformer global perception embedding module parameters are obtained. The final effect evaluation index of this model can be obtained by sending the test set of the knowledge graph to the Transformer global perception embedding module under this parameter.

[0082] After obtaining the best model, you can use it to complete the knowledge graph. For each triple to be predicted, the embeddings of its head and tail entities and relationships are sent to the model to obtain the score of the triple. It is compared with the set score threshold. The triples above the threshold are the missing triples predicted by the model. After all the triples to be predicted are processed by the model, the final set of predicted triples can be obtained to complete the knowledge graph completion task.

[0083] According to the above-mentioned knowledge graph completion method integrating GNN and global attention mechanism, the programming language used is mainly Python, and the design of the entire model is realized based on the deep learning open source framework PyTorch and the graph deep learning open source library DGL. In order to evaluate the performance of the Transformer global perception embedding module, the present invention conducts link prediction experiments of knowledge graph completion on two knowledge graph datasets, namely WN18RR and FB15K237. Table 1 shows the detailed information of the two datasets.

[0084] Table 1 Dataset

[0085]

[0086] Table 2 is the experimental results. The present invention conducted link prediction experiments on the WN18RR and FB15K237 knowledge graph datasets, and compared them with other representative completion methods. The experimental results show that compared with the existing models, the present invention is significantly better than the knowledge graph completion methods based on GNN and TransE in terms of evaluation indicators, which shows that in the process of learning the knowledge graph embedding representation, the fusion of GNN and Transformer global perception embedding modules can significantly improve the knowledge graph completion effect. In summary, the method proposed in the present invention effectively improves the performance of the model in knowledge graph completion through the fusion learning of GNN and Transformer global perception embedding modules, and provides new ideas for further research on knowledge graph completion. The research of the present invention has important significance and value both in theory and practice, and has a positive impact on promoting the development of knowledge graph completion models.

[0087] Table 2 Experimental results

[0088]

[0089] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.

Claims

1. A knowledge graph completion method integrating GNN and global attention mechanism, characterized by: The following steps are involved: Step 1: Input the knowledge graph into the Comphormer model. The Comphormer model consists of a GNN structure extraction module and a Transformer global perception embedding module. After the knowledge graph is input into the Comphormer model, it first enters the GNN structure extraction module. The GNN structure extraction module introduces reverse relations and self-loop relation types, and performs initial embedding to obtain the initial representation of entities and relations. Step 2: Incorporate the degree structure encoding into the entity initialization embedding, and the relationship initial embedding is represented by the initialized basis vector; Step 3: According to the selected attention operator, the CompGAT layer in the GNN structure extraction module is used to aggregate and update the initial embeddings of entities and relations to obtain entity and relationship embeddings containing local structural information of entities as the output of the GNN structure extraction module; Step 4: Input the output of the GNN structure extraction module into the Transformer global perception embedding module. In this module, the multi-relation fusion degree encoding is integrated into the entity embedding, and the edge features of each edge in the knowledge graph are calculated to obtain the entity embedding that integrates the multi-relation fusion degree encoding and the edge features of each edge. Step 5: In the Transformer global perception embedding module, the spatial position encoding and edge feature encoding between each pair of entities are calculated, and the spatial position encoding and edge feature encoding are added as bias items to the attention matrix, and the attention matrix of each attention head in each layer is calculated; Step 6: According to the obtained attention matrix, the entity embedding after fusion of multi-relation fusion degree encoding is updated according to the process of data processing of the standard Transformer global perception embedding module, and LayerNorm and residual link are performed; the entity embedding updated by the Transformer global perception embedding module is used as the output of the Transformer global perception embedding module; the entity embedding output by the Transformer global perception embedding module and the relation embedding output by the GNN structure extraction module are used as the entity and relation embeddings finally output by the Transformer global perception embedding module; Step 7: Input the entity embedding and relation embedding output by the Transformer global perception embedding module into the scoring function module, calculate the score of the triples in the knowledge graph, and evaluate the quality of the entity embedding and relation embedding output by the Transformer global perception embedding module.

2. According to claim 1, a knowledge graph completion method integrating GNN and global attention mechanism is characterized in that: The specific implementation method for obtaining the initial representation of entities and relationships in step 1 is: in the GNN structure extraction module in the Transformer global perception embedding module, reverse relationship types and self-loop relationship types are introduced to enrich the edge types, and initial embedding is performed to obtain the initial representation of entities and relationships.

3. According to claim 1, a knowledge graph completion method integrating GNN and global attention mechanism is characterized in that: The specific implementation method of step 2 is: after initializing the knowledge graph, introducing degree structure encoding to enrich the representation of entities and obtain the initial embedding of entities containing graph centrality information; the initial embedding of relationships is represented by initialized basis vectors to reduce the number of parameters.

4. According to claim 1, a knowledge graph completion method integrating GNN and global attention mechanism is characterized in that: The specific implementation method of step 3 is as follows: this module designs three attention operators, namely, addition Add, multiplication Mul and concatenation Con. Different operators can be selected according to different knowledge graphs to achieve the best effect; based on the selected attention operator, the CompGAT layer in the GNN structure extraction module is used to aggregate and update the initial embeddings of entities and relationships, and the entity embedding and relationship embedding containing the local structure information of the knowledge graph can be obtained as the output of the GNN structure extraction module.

5. According to claim 1, a knowledge graph completion method integrating GNN and global attention mechanism is characterized in that: The specific implementation method of step 4 is: input the output of the GNN structure extraction module into the Transformer global perception embedding module, calculate the multi-relation fusion degree encoding of the entity and integrate it into the entity embedding, and obtain the entity embedding containing graph centrality and multi-relation centrality information; calculate the edge features of each edge for the calculation of edge feature encoding in step 5.

6. A knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method of step 5 is: calculate the spatial position encoding between entities based on the shortest path distance and the shortest resistance distance between each pair of entities, calculate the edge feature encoding between entities based on the shortest path between each pair of entities and the edge feature of each edge on the shortest path, and use these two encodings as bias items for attention matrix calculation. The spatial position encoding enables the model to perceive the relative position information between entities, and the edge feature encoding can embed the relationship into the attention mechanism.

7. A knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method of step 6 is: calculate the attention matrix of each attention head of each layer of Transformer, update the entity embedding according to the standard Transformer global perception embedding module process, and perform LayerNorm and residual link; after the Transformer global perception embedding module is updated, an entity embedding containing local structural information, global relative position information, and relationship information can be obtained.

8. The knowledge graph completion method integrating GNN and global attention mechanism according to claim 1, characterized in that: The specific implementation method of step 7 is: using the ConvE model as the scoring function, reshaping the embedding representation of entities and relations into a two-dimensional image, and using a convolutional layer to capture the local features between entities and relations, and then outputting the score of the triplet through a fully connected layer. For each positive sample triplet, randomly replace the head and tail entities to generate a set of negative sample triplets, where the positive sample triplet includes (e i , r k , e j ), the negative sample triples include (e i ',r k ',e j '), the model is trained to distinguish true triplets from false triplets by comparing the scores of positive and negative samples, and is used to calculate the loss function and perform back propagation, thus completing a round of model training.

Citation Information

Patent Citations

  • Knowledge graph complementing method based on topic keyword filtering

    CN109977234A

  • Knowledge graph long-tail relation completion method based on attention mechanism

    CN111291139A

  • Knowledge graph completion method based on bidirectional attention mechanism

    CN111949764A

  • Knowledge graph link completion method and system for multiple relationships

    CN114064627A

  • Open world knowledge graph completion method and device

    CN114444694A