Knowledge graph reasoning method based on linear attention mechanism

Abstract

The invention relates to a knowledge graph reasoning technology, and provides a knowledge graph reasoning method based on a linear attention mechanism, which comprises the following steps of: firstly, embedding nodes of a graph of a knowledge graph, inputting the nodes into a graph neural network, and obtaining node embedding containing local topological information; then, nodes containing local topological information are embedded, and a linear attention mechanism is utilized to obtain feature vectors containing global information; then, embedding the feature vector containing global information and the node containing local topological information, obtaining the weight of each piece of input information by utilizing an attention mechanism, and embedding by taking the weight sum of each part as the final node; and finally, based on the obtained node embedding and relation embedding, utilizing a scoring network, and outputting a reasoning result, so that the method can capture the long-range dependency relation in the knowledge graph, meanwhile, avoids the problem of over-smoothing of the graph neural network, and effectively improves the problem that the Hits at10 index of the existing reasoning model is relatively low.

Description

Technical Field

[0001] The present invention relates to knowledge graph reasoning technology, and specifically to a knowledge graph reasoning method based on a linear attention mechanism. Background Art

[0002] Data in a knowledge graph typically exists in the form of triples, specifically represented as (head entity, relationship, tail entity). A large number of triples form a vast network graph. Any object or concept in nature can be represented as an entity node in a knowledge graph, and any connection can be represented as an edge in the knowledge graph. For example, if a knowledge graph contains a triple (Socrates, born in, Athens), the semantics of this triple represent "Socrates was born in Athens," where "Socrates" and "Athens" are two entity nodes and "born in" is a relationship, together forming an edge in the graph structure.

[0003] Currently, knowledge graphs have become one of the primary data sources for many scientific research and application areas. For example, many existing natural language processing tasks, such as information retrieval, data mining, and data analysis, require knowledge graph data support. Knowledge graph applications have also spread across various industries, such as social networks, intelligent dialogue, personalized intelligent recommendations, intelligent question-and-answer (Q&A), and intelligent search, and are further applied in vertical fields such as finance, social security, medical services, and public opinion response. However, in actual applications, knowledge graph data automatically acquired through models or manually obtained is often redundant and incomplete, and may even contain abnormal and erroneous knowledge. This greatly reduces the usability of knowledge graph data in tasks, and the quality of the data directly affects the development and progress of related technologies.

[0004] The goal of knowledge graph reasoning is to infer missing facts based on existing facts in the graph. A typical example of this is link prediction within a knowledge graph. However, existing knowledge graph reasoning techniques based on graph neural networks face a trade-off between long-distance information compression and oversmoothing. Specifically, to obtain information from distant nodes, existing techniques require increasing model depth, which in turn leads to oversmoothing in graph neural networks.

[0005] To avoid this pair of problems, existing technologies usually adopt a more conservative strategy and are very cautious about increasing the depth of the model. This in turn causes the model to perform poorly when faced with a large number of long-range dependencies in open knowledge graphs, and even be unable to handle dependencies whose path length is greater than the model depth. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to propose a knowledge graph reasoning method based on the linear attention mechanism, which can capture the long-range dependencies in the knowledge graph while avoiding the over-smoothing problem of the graph neural network and effectively improve the problem of low Hits@10 index of the existing reasoning model.

[0007] The technical solution adopted by the present invention to solve the above technical problems is:

[0008] A knowledge graph reasoning method based on a linear attention mechanism. The architecture of the knowledge graph reasoning model includes the following steps:

[0009] A1. Based on the knowledge graph, construct a directed graph G = (V, E) of the knowledge graph, where V is the set of nodes representing the entities contained in the knowledge graph, and E is the set of edges representing the relationships between the entities contained in the knowledge graph; construct and initialize a learnable embedding matrix with a dimension of L × K, where L is the set number of iterations and K is the number of relationships contained in the knowledge graph; the value of the element in the lth row and kth column of the embedding matrix constitutes the relationship embedding of the kth relationship contained in the knowledge graph at the lth iteration;

[0010] A2. Extract N triples from the knowledge graph as positive samples T for this round of training n ; For each positive sample T n , construct M negative samples respectively

[0011] A3. For each positive sample T input n , and process them as follows:

[0012] A31. Initialize the node embeddings of the nodes contained in the directed graph G;

[0013] A32. For each node u contained in the directed graph G i , respectively, using graph neural network, based on node u i Node embeddings of neighbor nodes and node u i Relational embedding of edge ownership between nodes and their neighboring nodes For node u i The node embedding is updated; where l represents the iteration round, k j Represents node u i The index of the edge relationship between it and its j-th neighbor node, Represents the kth j The relation embedding of a relation in the first round of iteration;

[0014] A33. Use the node embeddings of each node included in the directed graph G obtained by the update in step A32 as rows to form a node embedding matrix; use the node embedding matrix as input and use the linear attention mechanism to calculate a node feature matrix containing global information; use each row of the node feature matrix as a feature vector for each node included in the directed graph G;

[0015] A34. Use the node embedding before updating in step A32 as the first feature, use the node embedding after updating in step A32 as the second feature, and use the feature vector of the node obtained in step A33 as the third feature;

[0016] For each node u contained in the directed graph G i , respectively, using the attention mechanism to calculate the node u i The attention weights of the first, second and third features of the node u are calculated; then, according to the attention weights, i The first, second and third features are weighted averaged, and the weighted sum of the output is used as the node u i Node embedding updated in this round of iteration;

[0017] A35, determine whether the set number of iterations L is reached, if so, end the iteration; otherwise, the node u obtained in step A34 is i The node embedding updated in this round of iteration is used as input and returns to step A32;

[0018] A36. Using positive sample T n The node embeddings corresponding to the entities contained in its negative samples, as well as the relation embeddings of the contained relations, calculate the cross entropy loss;

[0019] A4. Use step A2 to input each positive sample T n The cross entropy loss is calculated to calculate the total loss of this round of training, and the reverse update is performed based on the total loss;

[0020] A5. Determine whether all triples contained in the knowledge graph have been traversed. If so, end the training and obtain a trained knowledge graph reasoning model; otherwise, return to step A2.

[0021] Using the trained knowledge graph reasoning model, the reasoning process includes:

[0022] B1. Use the query head entity and query relationship, combined with the candidate tail entity set, to construct the set of triples to be inferred;

[0023] B2. For each triple to be inferred in the set of triples to be inferred, input them into the knowledge graph inference model, obtain the relation embeddings of the contained relations according to the embedding matrix, and obtain the node embeddings corresponding to the contained entities according to the method of steps A31 to A35;

[0024] B3. Use the relation embeddings of the entities and relations contained in each triple to be inferred, and obtain their scores through the scoring network;

[0025] B4. Output the reasoning result based on the score of each triple to be inferred.

[0026] Furthermore, in step A31, for the nodes included in the directed graph G, the corresponding positive samples T n The node embedding of the node of the head entity is initialized to a full 1 vector, except for the corresponding positive sample T n The node embeddings of all nodes other than the head entity are initialized to all 0 vectors

[0027] Furthermore, in step A32, for each node u included in the directed graph G i , respectively, according to the following formula, using graph neural network, based on node u i Node embeddings of neighbor nodes and node u i Relational embedding of edge ownership between nodes and their neighboring nodes For node u i The node embedding is updated:

[0028]

[0029] in, is the i-th node u in the directed graph G i Neighbor node v j A set of nodes, and the i-th node u i and its neighbor node v j The edges between them are all connected by nodes v j Points to node u i ; φ1 is the fusion function, φ1(v j ) represents the neighbor node v j Generated messages; Indicates node u in the first iteration i Updated node embedding, Indicates the node u that is input into this step in the lth iteration i The jth neighbor node v j Node embedding, Represents node u in the lth iteration i and its jth neighbor node v jRelational embedding of edge ownership; Indicates node u in the first iteration i The jth neighbor node v j The attention weight of .

[0030] Furthermore, the attention weight is calculated according to the following formula:

[0031]

[0032] in, Indicates node u in the first iteration i The jth neighbor node v j Attention score; In this step, the learnable parameter matrices of attention calculations in each iterative round are independent of each other. represents the learnable parameter matrix of the attention calculation in this step in round l, and σ is the LeakyReLU or ReLU activation function.

[0033] Furthermore, step A34 includes:

[0034] A341. Using the attention mechanism, calculate the node u according to the following formula i The first characteristic Second feature and the third characteristic The attention weight α i,s :

[0035]

[0036] In this step, the learnable parameter matrices of attention calculations in each iterative round are independent of each other. represents the learnable parameter matrix of the attention calculation in this step in round l; represents matrix transpose, σ is the LeakyReLU activation function; s is the feature number;

[0037] A342, according to the attention weight α i,s , for node u i The first characteristic Second feature and the third characteristic Perform weighted averaging and use the weighted sum of the output as the node u i Node embedding updated in this round of iteration:

[0038]

[0039] in, Represents node u i The node embeddings updated in this round of iterations.

[0040] Furthermore, step A36 includes:

[0041] A361. Using the scoring network, calculate the positive sample T according to the following formula n And the scores of its negative samples:

[0042]

[0043] Among them, T represents the triplet currently evaluated by the rating network, is the node embedding of the node corresponding to the tail entity of the current evaluation triple, r T is the relation embedding of the relation in the current evaluation triple, whose value is the value of the element corresponding to the relation in the Lth row of the embedding matrix; MLP represents multi-layer perceptron, and || represents concatenation;

[0044] A362. Calculate the cross entropy loss according to the following formula:

[0045]

[0046] in, is a positive sample T n The cross entropy loss.

[0047] Furthermore, in step A33, the node feature matrix containing global information is calculated using the linear attention mechanism with the node embedding matrix as input according to the following formula:

[0048]

[0049] Q=W Q,h E (l) ,K=W K,h E (l) ,V=W V,h E (l)

[0050] Among them, W Q,h 、W K,h and W V,h They represent the linear transformation matrices of the Query, Key, and Value channels respectively, and φ2 represents the kernel function for dimensional transformation in the linear attention mechanism; E (l) represents the node embedding matrix in the lth iteration, Represents the node feature matrix in the lth iteration; Represents matrix transpose.

[0051] Furthermore, in the linear attention mechanism, the kernel function for dimension transformation is:

[0052] φ2(x)=ELU(x)+1

[0053] Where x represents the input of the kernel function.

[0054] The beneficial effects of the present invention are:

[0055] The method of the present invention uses a shallow graph neural network to capture the neighborhood features of nodes and obtain local topological information through a limited number of message transmissions. Secondly, a linear attention mechanism is used to establish a direct connection channel between nodes in the entire graph, so that any two nodes can perform cross-layer information exchange, and global information exchange is facilitated by introducing global information. Finally, the dual-channel mechanism is combined with the attention mechanism so that nodes can dynamically select more important information. For example, when reasoning about short-range dependencies, the attention weight of local topological information is higher, while when reasoning about long-range dependencies, the attention weight of global information is higher. Therefore, the method of the present invention can capture long-range dependencies in the knowledge graph while avoiding the problem of over-smoothing of the graph neural network, effectively improving the problem of low Hits@10 index of the existing reasoning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of node embedding initialization of directed graph G of the present invention. DETAILED DESCRIPTION

[0057] This paper aims to provide a knowledge graph reasoning method based on a linear attention mechanism. It can capture long-range dependencies in knowledge graphs while avoiding the oversmoothing problem of graph neural networks, effectively improving the low Hits@10 metric of existing reasoning models. Hits@10 represents the proportion of correct triples—i.e., true relationships—that are within the top 10 in link prediction tasks and is a metric commonly used to evaluate the performance of knowledge graph embedding models. The architecture of the knowledge graph reasoning model includes the following steps:

[0058] A1. Based on the knowledge graph, construct a directed graph G = (V, E) of the knowledge graph, where V is the set of nodes representing the entities contained in the knowledge graph, and E is the set of edges representing the relationships between the entities contained in the knowledge graph; construct and initialize a learnable embedding matrix with a dimension of L × K, where L is the set number of iterations and K is the number of relationships contained in the knowledge graph; the value of the element in the lth row and kth column of the embedding matrix constitutes the relationship embedding of the kth relationship contained in the knowledge graph at the lth iteration;

[0059] A2. Extract N triples from the knowledge graph as positive samples T for this round of training n ; For each positive sample T n , construct M negative samples respectively

[0060] A3. For each positive sample T input n , and process them as follows:

[0061] A31. Initialize the node embeddings of the nodes contained in the directed graph G;

[0062] A32. For each node u contained in the directed graph G i , respectively, using graph neural network, based on node u i Node embeddings of neighbor nodes and node u i Relational embedding of edge ownership between nodes and their neighboring nodes For node u i The node embedding is updated; where l represents the iteration round, k j Represents node u i The index of the edge relationship between it and its j-th neighbor node, Represents the kth j The relation embedding of a relation in the first round of iteration;

[0063] A33. Use the node embeddings of each node included in the directed graph G obtained by the update in step A32 as rows to form a node embedding matrix; use the node embedding matrix as input and use the linear attention mechanism to calculate a node feature matrix containing global information; use each row of the node feature matrix as a feature vector for each node included in the directed graph G;

[0064] A34. Use the node embedding before updating in step A32 as the first feature, use the node embedding after updating in step A32 as the second feature, and use the feature vector of the node obtained in step A33 as the third feature;

[0065] For each node u contained in the directed graph G i , respectively, using the attention mechanism to calculate the node u i The attention weights of the first, second and third features of the node u are calculated; then, according to the attention weights, i The first, second and third features are weighted averaged, and the weighted sum of the output is used as the node u i Node embedding updated in this round of iteration;

[0066] A35, determine whether the set number of iterations L is reached, if so, end the iteration; otherwise, the node u obtained in step A34 is i The node embedding updated in this round of iteration is used as input and returns to step A32;

[0067] A36. Using positive sample T nThe node embeddings corresponding to the entities contained in its negative samples, as well as the relation embeddings of the contained relations, calculate the cross entropy loss;

[0068] A4. Use step A2 to input each positive sample T n The cross entropy loss is calculated to calculate the total loss of this round of training, and the reverse update is performed based on the total loss;

[0069] A5. Determine whether all triples contained in the knowledge graph have been traversed. If so, end the training and obtain a trained knowledge graph reasoning model; otherwise, return to step A2.

[0070] Using the trained knowledge graph reasoning model, the reasoning process includes:

[0071] B1. Use the query head entity and query relationship, combined with the candidate tail entity set, to construct the set of triples to be inferred;

[0072] B2. For each triple to be inferred in the set of triples to be inferred, input them into the knowledge graph inference model, obtain the relation embeddings of the contained relations according to the embedding matrix, and obtain the node embeddings corresponding to the contained entities according to the method of steps A31 to A35;

[0073] B3. Use the relation embeddings of the entities and relations contained in each triple to be inferred, and obtain their scores through the scoring network;

[0074] B4. Output the reasoning result based on the score of each triple to be inferred.

[0075] In the knowledge graph, the local neighborhood structure of a node contains important semantic information and plays a key role in reasoning tasks. To this end, in step A32, a shallow graph neural network is used to capture the neighborhood features of the node and obtain local topological information. The purpose of using a shallow graph neural network is to avoid the over-smoothing problem caused by deep graph neural networks, thereby giving full play to the dual information channels of the present invention.

[0076] The long-range information compression problem in graph neural networks makes it difficult for nodes to obtain information from more distant nodes, which makes it difficult for graph neural networks to handle knowledge reasoning with long-range dependencies. To address this problem, the model of the present invention introduces a global information module to facilitate global information exchange between nodes. In step A33, the global information module is sampled and a linear attention mechanism is used to establish direct connection channels between nodes in the entire graph, enabling any two nodes to exchange information across layers.

[0077] After steps A32 and A33, each node obtains three feature vectors, among which the first feature is the self-loop information feature vector, the second feature is the local topology structure feature vector, and the third feature is the global information feature vector. The three vectors have different importance for different nodes. The candidate nodes that have a long-range dependency relationship with the query head node receive more important global information, while the neighboring nodes of the head node need more local topology structure information to distinguish them from other neighboring nodes. For this reason, in step S34, the present invention introduces an attention mechanism, which dynamically adjusts the weights of each node to the three information through the attention mechanism, so that the nodes can select more important information on their own.

[0078] Finally, the node embeddings and relationship embeddings obtained through multiple iterations are input into a scoring network to obtain the model’s scores.

[0079] Therefore, the method of the present invention can capture the long-range dependencies in the knowledge graph, while avoiding the over-smoothing problem of the graph neural network, and effectively improve the problem of low Hits@10 index of the existing reasoning model.

[0080] The following is further described with reference to the embodiments.

[0081] Example:

[0082] In this embodiment, a knowledge graph reasoning method based on a linear attention mechanism is provided, and the architecture of the knowledge graph reasoning model includes the following steps:

[0083] A1. Data Preparation

[0084] In this step, a directed graph G = (V, E) of the knowledge graph is constructed based on the knowledge graph, where V is a set of nodes representing the entities contained in the knowledge graph, and E is a set of edges representing the relationships between the entities contained in the knowledge graph.

[0085] Construct and initialize a learnable embedding matrix of dimension L×K, where L is the set number of iterations and K is the number of relations contained in the knowledge graph; the value of the element in the lth row and kth column of the embedding matrix constitutes the relation embedding of the kth relation contained in the knowledge graph at the lth iteration.

[0086] A2. Extract samples

[0087] In this step, N triples are extracted from the knowledge graph as positive samples T for this round of training. n ; For each positive sample T n , construct M negative samples respectively

[0088] A3. Training

[0089] In this step, for each positive sample T input n , and process them as follows:

[0090] A31. Initialize node embedding

[0091] In this step, the node embeddings of the nodes contained in the directed graph G are initialized.

[0092] The reasoning method of the present invention is an inductive reasoning method, and its node embedding initialization method can adopt any existing embedding representation method that does not directly train each node.

[0093] In this embodiment, in order to enable fast migration within different knowledge graphs in the same field, the feature vectors of different nodes are initialized to feature vectors of all 0 or all 1 according to the reasoning problem. Specifically, for the nodes contained in the directed graph G, the corresponding positive samples T n The node embedding of the node of the head entity is initialized to a full 1 vector, except for the corresponding positive sample T n The node embeddings of all nodes except the head entity’s node are initialized to all-zero vectors.

[0094] like Figure 1 As shown in the figure, the left half describes the input knowledge graph and the query. The input knowledge graph is a family pedigree diagram, consisting of six nodes (A, B, C, D, E, and F) representing family members and several kinship relationships. The query consists of two parts: the query head node A and the query relationship "grandson", and the question requires predicting the tail node in the triple (query head node, relationship, tail node). The right half shows the result of feature vector initialization, where the feature vector of the query head node A is initialized to all 1s, and the feature vectors of other nodes are initialized to all 0s. Since the embedding representation of specific nodes is not trained, it can be quickly transferred within the same domain knowledge graph with the same relationship.

[0095] A32. Extracting local topological information

[0096] In this step, for each node u contained in the directed graph G i , respectively, using graph neural network, based on node u i Node embeddings of neighbor nodes and node u i Relational embedding of edge ownership between nodes and their neighboring nodes For node u i The node embedding is updated; where l represents the iteration round, k j Represents node u i The index of the edge relationship between it and its j-th neighbor node, Represents the kth j The relation embedding of a relation in the lth round of iteration.

[0097] In this step, specifically, for each node u contained in the directed graph G i , respectively, according to the following formula, using graph neural network, based on node u i Node embeddings of neighbor nodes and node u i Relational embedding of edge ownership between nodes and their neighboring nodes For node u i The node embedding is updated:

[0098]

[0099]

[0100] in, is the i-th node u in the directed graph G i Neighbor node v j A set of nodes, and the i-th node u i and its neighbor node v j The edges between them are all connected by nodes v j Points to node u i ; φ1 is the fusion function, φ1(v j ) represents the neighbor node v j Generated messages; Indicates node u in the first iteration i Updated node embedding, Indicates the node u that is input into this step in the lth iteration i The jth neighbor node v j Node embedding, Represents node u in the lth iteration i and its jth neighbor node v j Relational embedding of edge ownership; Indicates node u in the first iteration i The jth neighbor node v j The attention weight of .

[0101] Calculate the attention weight according to the following formula

[0102]

[0103] in, Indicates node u in the first iteration i The jth neighbor node v jAttention score; In this step, the learnable parameter matrices of attention calculations in each iterative round are independent of each other. represents the learnable parameter matrix of the attention calculation in this step in round l, and σ is the ReLU activation function. In addition to the ReLU activation function, the LeakyReLU activation function can also be sampled.

[0104] A33. Global Information Interaction

[0105] In this step, the node embeddings of each node contained in the directed graph G obtained by updating in step A32 are used as rows to form a node embedding matrix; the node embedding matrix is used as input, and the linear attention mechanism is used to calculate and obtain a node feature matrix containing global information; each row of the node feature matrix is used as a feature vector of each node contained in the directed graph G.

[0106] Specifically, according to the following formula, with the node embedding matrix as input, the linear attention mechanism is used to calculate the node feature matrix containing global information:

[0107]

[0108] Q=W Q,h E (l) ,K=W K,h E (l) ,V=W V,h E (l)

[0109] Among them, W Q,h 、W K,h and W V,h They represent the linear transformation matrices of the Query, Key, and Value channels respectively, and φ2 represents the kernel function for dimensional transformation in the linear attention mechanism; E (l) represents the node embedding matrix in the lth iteration, represents the node feature matrix in the lth iteration; T represents the matrix transpose.

[0110] The linear attention mechanism is an improved version of the attention mechanism that aims to solve the computational complexity problem of traditional attention mechanisms, such as the self-attention mechanism in Transformer, when processing long sequences. The time complexity and space complexity of the traditional attention mechanism are both O(B 2 ), where B is the length of the sequence. The linear attention mechanism decomposes the calculation of the attention score into a series of linear operations by introducing linear transformations and approximation methods, reducing the complexity to O(N), thus enabling more efficient processing of long sequence data.

[0111] The linear attention mechanism replaces the softmax function in the traditional attention mechanism with a more efficient kernel function through the kernel function, which is a key part of achieving efficient computing. Common kernel functions include exponential kernel, ReLU kernel, Sigmoid kernel, etc. In this embodiment, the linear attention mechanism specifically samples the Linear Transformer. Therefore, the kernel function for dimensionality transformation is:

[0112] φ2(x)=ELU(x)+1

[0113] Where x represents the input of the kernel function.

[0114] The ELU function, whose full name is Exponential Linear Units, is a piecewise function. The part greater than 0 is a linear function, and the part less than 0 is α(exp(x)-1). Taking ELU(x)+1 as the kernel function ensures that the kernel function is greater than 0.

[0115] A34, dual-channel fusion

[0116] In this step, the node embedding before the update in step A32 is used as the first feature, the node embedding after the update in step A32 is used as the second feature, and the feature vector of the node obtained in step A33 is used as the third feature. i , respectively, using the attention mechanism to calculate the node u i The attention weights of the first, second and third features of the node u are calculated; then, according to the attention weights, i The first, second and third features are weighted averaged, and the weighted sum of the output is used as the node u i The node embeddings updated in this round of iterations.

[0117] Specifically, in this embodiment, this step includes:

[0118] A341. Using the attention mechanism, calculate the node u according to the following formula i The first characteristic Second feature and the third characteristic The attention weight α i,s :

[0119]

[0120] In this step, the learnable parameter matrices of attention calculations in each iterative round are independent of each other. represents the learnable parameter matrix of the attention calculation in this step in round l; represents matrix transpose, σ is the LeakyReLU activation function, and s is the feature number.

[0121] A342, according to the attention weight α i,s , for node u i The first characteristic Second feature and the third characteristic Perform weighted averaging and use the weighted sum of the output as the node u i Node embedding updated in this round of iteration:

[0122]

[0123] in, Represents node u i The node embeddings updated in this round of iterations.

[0124] A35. Embedding Iteration

[0125] In this step, it is determined whether the set number of iterations L is reached. If so, the iteration is terminated; otherwise, the node u obtained in step A34 is i The node embedding updated in this round of iteration is used as input and returns to step A32;

[0126] A36. Loss calculation

[0127] In this step, we use the positive sample T n The node embeddings corresponding to the entities contained in its negative samples, as well as the relation embeddings of the contained relations, are used to calculate the cross entropy loss.

[0128] In this embodiment, specifically, this step includes:

[0129] A361. Using the scoring network, calculate the positive sample T according to the following formula n And the scores of its negative samples:

[0130]

[0131] Among them, T represents the triplet currently evaluated by the rating network, is the node embedding of the node corresponding to the tail entity of the current evaluation triple, r T is the relation embedding of the relation in the current evaluation triple, whose value is the value of the element corresponding to the relation in the Lth row of the embedding matrix; MLP represents multi-layer perceptron, and || represents concatenation.

[0132] A362. Calculate the cross entropy loss according to the following formula:

[0133]

[0134] in, is a positive sample T n The cross entropy loss.

[0135] A4. Parameter Update

[0136] In this step, use step A2 to input each positive sample T n The cross entropy loss is calculated to calculate the total loss of this round of training, and the reverse update is performed based on the total loss.

[0137] A5. Training Iterations

[0138] In this step, determine whether all triples contained in the knowledge graph have been traversed. If so, end the training and obtain a trained knowledge graph reasoning model; otherwise, return to step A2.

[0139] Using the trained knowledge graph reasoning model, the reasoning process includes:

[0140] B1. Use the query head entity and query relationship, combined with the candidate tail entity set, to construct the set of triples to be inferred;

[0141] B2. For each triple to be inferred in the set of triples to be inferred, input them into the knowledge graph inference model, obtain the relation embeddings of the contained relations according to the embedding matrix, and obtain the node embeddings corresponding to the contained entities according to the method of steps A31 to A35;

[0142] B3. Use the relation embeddings of the entities and relations contained in each triple to be inferred, and obtain their scores through the scoring network;

[0143] B4. Output the reasoning result based on the score of each triple to be inferred.

[0144] The model trained in this embodiment was tested on two datasets: WN18RR and FB15k-237.

[0145] WN18RR is a subset of WN18, which retains more of the symmetric, asymmetric, and combination relationships in the original dataset, while removing inversion relationships. WN18 is a subset of the WordNet dataset, in which the main patterns of relationships are symmetric, asymmetric, and inversion relationships. WordNet is an English dictionary established and maintained by Princeton University since 1985. WordNet groups terms according to their meaning, and each group of terms with the same meaning is called a synonym set. WordNet provides a brief summary definition for each synonym set and records the semantic relationships between different synonym sets.

[0146] The FB15k-237 dataset is a subset of the FB15k dataset. This dataset primarily retains symmetric, asymmetric, and composite relationships, while also removing inversion relationships. The FB15k dataset is a subset of the Freebase dataset. The types of relationships in this graph are primarily symmetric, asymmetric, and inversion relationships. Freebase is a large collaborative knowledge base composed of metadata. Its content primarily comes from contributions from its community members and integrates many online resources, including content from some private wiki sites.

[0147] The model trained in this embodiment is tested on the WN18RR and FB15k-237 datasets according to the method in steps B1 to B4. The results are shown in Table 1 below. The values in the table are Hits@10 index scores.

[0148] Table 1. Test results of WN18RR and FB15k-237 datasets

[0149]

[0150] The training and test sets of the WN18RR dataset exhibit significant differences in the average distance between reachable point pairs. The training set exhibits a longer average distance between reachable point pairs, while the test set likely samples only highly interconnected node clusters, such as specific semantic branches in WordNet, resulting in shorter inter-node distances. This structural characteristic results in the reasoning tasks in the test set being highly dependent on local topological information. While the global information interaction proposed in this paper can effectively capture long-range dependencies, its advantages cannot be fully demonstrated because the test set does not involve cross-cluster reasoning.

[0151] On the FB15k-237 dataset, the FB15k-237 dataset contains a large number of complex multi-hop relationships due to its coverage of knowledge in multiple fields such as social, professional, and geographical fields. This indicates that the test task has significantly increased the requirements for long-range dependency modeling capabilities. This feature is highly compatible with the global attention mechanism of the present invention. In the v4 version with the longest average distance in the test set, the Hits@10 of the present invention reached 66.6, the best among all models, which verifies the core advantage of the global interaction mechanism in complex reasoning scenarios in open domains.

[0152] Finally, it should be noted that the above embodiments are merely preferred implementations and are not intended to limit the present invention. It should be noted that those skilled in the art will be able to make modifications, equivalent substitutions, and improvements without departing from the spirit and scope of the present invention and the claims, all of which should be included within the scope of protection of the present invention.

Claims

1. A knowledge graph reasoning method based on linear attention mechanism, characterized by: The architecture of its knowledge graph reasoning model includes the following steps: A1. Based on the knowledge graph, construct a directed graph G = (V, E) of the knowledge graph, where V is the set of nodes representing the entities contained in the knowledge graph, and E is the set of edges representing the relationships between the entities contained in the knowledge graph; construct and initialize a learnable embedding matrix with a dimension of L × K, where L is the set number of iterations and K is the number of relationships contained in the knowledge graph; the value of the element in the lth row and kth column of the embedding matrix constitutes the relationship embedding of the kth relationship contained in the knowledge graph at the lth iteration; A2. Extract N triples from the knowledge graph as positive samples T for this round of training n ; For each positive sample T n , construct M negative samples respectively A3. For each positive sample T input n , and process them as follows: A31. Initialize the node embeddings of the nodes contained in the directed graph G; A32. For each node u contained in the directed graph G i , respectively, using graph neural network, based on node u i Node embeddings of neighbor nodes and node u i Relational embedding of edge ownership between nodes and their neighboring nodes For node u i The node embedding is updated; where l represents the iteration round, k j Represents node u i The index of the edge relationship between it and its j-th neighbor node, Represents the kth j The relation embedding of a relation in the first round of iteration; A33. Use the node embeddings of each node included in the directed graph G obtained by the update in step A32 as rows to form a node embedding matrix; use the node embedding matrix as input and use the linear attention mechanism to calculate a node feature matrix containing global information; use each row of the node feature matrix as a feature vector for each node included in the directed graph G; A34. Use the node embedding before updating in step A32 as the first feature, use the node embedding after updating in step A32 as the second feature, and use the feature vector of the node obtained in step A33 as the third feature; For each node u contained in the directed graph G i , respectively, using the attention mechanism to calculate the node u i The attention weights of the first, second and third features of the node u are calculated; then, according to the attention weights, i The first, second and third features are weighted averaged, and the weighted sum of the output is used as the node u i Node embedding updated in this round of iteration; A35, determine whether the set number of iterations L is reached, if so, end the iteration; otherwise, the node u obtained in step A34 is i The node embedding updated in this round of iteration is used as input and returns to step A32; A36. Using positive sample T n The node embeddings corresponding to the entities contained in its negative samples, as well as the relation embeddings of the contained relations, calculate the cross entropy loss; A4. Use step A2 to input each positive sample T n The cross entropy loss is calculated to calculate the total loss of this round of training, and the reverse update is performed based on the total loss; A5. Determine whether all triples contained in the knowledge graph have been traversed. If so, end the training and obtain a trained knowledge graph reasoning model; otherwise, return to step A2. Using the trained knowledge graph reasoning model, the reasoning process includes: B1. Use the query head entity and query relationship, combined with the candidate tail entity set, to construct the set of triples to be inferred; B2. For each triple to be inferred in the set of triples to be inferred, input them into the knowledge graph inference model, obtain the relation embeddings of the contained relations according to the embedding matrix, and obtain the node embeddings corresponding to the contained entities according to the method of steps A31 to A35; B3. Use the relation embeddings of the entities and relations contained in each triple to be inferred, and obtain their scores through the scoring network; B4. Output the reasoning result based on the score of each triple to be inferred.

2. The knowledge graph reasoning method based on the linear attention mechanism according to claim 1, characterized in that: In step A31, for the nodes included in the directed graph G, the corresponding positive samples T n The node embedding of the node of the head entity is initialized to a full 1 vector, except for the corresponding positive sample T n The node embeddings of all nodes except the head entity’s node are initialized to all-zero vectors.

3. The knowledge graph reasoning method based on the linear attention mechanism according to claim 1, characterized in that: In step A32, for each node u included in the directed graph G i , respectively, according to the following formula, using graph neural network, based on node u i Node embeddings of neighbor nodes and node u i Relational embedding of edge ownership between nodes and their neighboring nodes For node u i The node embedding is updated: in, is the i-th node u in the directed graph G i Neighbor node v j A set of nodes, and the i-th node u i and its neighbor node v j The edges between them are all connected by nodes v j Points to node u i ; φ1 is the fusion function, φ1(v j ) represents the neighbor node v j Generated messages; Indicates node u in the lth iteration i Updated node embedding, Indicates the node u that is input into this step in the lth iteration i The jth neighbor node v j Node embedding, Represents node u in the lth iteration i and its jth neighbor node v j Relational embedding of edge ownership; Indicates node u in the first iteration i The jth neighbor node v j The attention weight of .

4. The knowledge graph reasoning method based on the linear attention mechanism according to claim 3, characterized in that: Calculate the attention weight according to the following formula in, Indicates node u in the first iteration i The jth neighbor node v j Attention score; In this step, the learnable parameter matrices of attention calculations in each iterative round are independent of each other. represents the learnable parameter matrix of the attention calculation in this step in round l, and σ is the LeakyReLU or ReLU activation function.

5. A knowledge graph reasoning method based on a linear attention mechanism as claimed in claim 1, characterized in that: Step A34 includes: A341. Using the attention mechanism, calculate the node u according to the following formula i The first characteristic Second feature and the third characteristic The attention weight α i,s : In this step, the learnable parameter matrices of attention calculations in each iterative round are independent of each other. represents the learnable parameter matrix of the attention calculation in this step in round l; represents matrix transpose, σ is the LeakyReLU activation function; s is the feature number; A342, according to the attention weight α i,s , for node u i The first characteristic Second feature and the third characteristic Perform weighted averaging and use the weighted sum of the output as the node u i Node embedding updated in this round of iteration: in, Represents node u i The node embeddings updated in this round of iterations.

6. A knowledge graph reasoning method based on a linear attention mechanism as claimed in claim 1, characterized in that: Step A36 includes: A361. Using the scoring network, calculate the positive sample T according to the following formula n And the scores of its negative samples: Among them, T represents the triplet currently evaluated by the rating network, is the node embedding of the node corresponding to the tail entity of the current evaluation triple, r T is the relation embedding of the relation in the current evaluation triple, whose value is the value of the element corresponding to the relation in the Lth row of the embedding matrix; MLP represents multi-layer perceptron, and || represents concatenation; A362. Calculate the cross entropy loss according to the following formula: in, is a positive sample T n The cross entropy loss.

7. A knowledge graph reasoning method based on a linear attention mechanism according to any one of claims 1 to 6, characterized in that: In step A33, the node feature matrix containing global information is calculated using the linear attention mechanism with the node embedding matrix as input according to the following formula: Q=W Q,h E (l) ,K=W K,h E (l) ,V=W V,h E (l) Among them, W Q,h 、W K,h and W V,h They represent the linear transformation matrices of the Query, Key, and Value channels respectively, and φ2 represents the kernel function for dimensional transformation in the linear attention mechanism; E (l) represents the node embedding matrix in the lth iteration, Represents the node feature matrix in the lth iteration; Represents matrix transpose.

8. A knowledge graph reasoning method based on a linear attention mechanism as claimed in claim 6, characterized in that: In the linear attention mechanism, the kernel function for dimension transformation is: φ2(x)=ELU(x)+1 Where x represents the input of the kernel function.

Images