A Relationship Learning Method and System Based on Multi-Bouncing Attention Graph Neural Network

By using the multi-hop attention graph neural network (MAGNA) diffusion attention mechanism, the problem that graph attention networks cannot effectively capture the context of nodes that are not directly connected is solved, and more efficient node classification and knowledge graph improvement are achieved.

CN116261731BActive Publication Date: 2026-11-17BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +2
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202180065230.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-05-24
Filing Date
2021-09-23
Publication Date
2026-11-17
Estimated Expiration
2041-09-23

AI Technical Summary

Technical Problem

Existing graph attention networks cannot effectively consider the contextual information between nodes that are not directly connected, resulting in a lack of comprehensiveness and accuracy in processing graph-structured data.

Method used

Multi-hop attention graph neural network (MAGNA) is used to compute multi-hop attention between nodes through the diffusion attention mechanism. By combining graph attention and diffusion techniques, it learns the multi-hop contextual relevance between node pairs, expands the receptive field of the model and captures large-scale structural information.

Benefits of technology

It improves the accuracy of node classification and the effect of knowledge graph enhancement, achieving a 5.7% improvement in relative error reduction compared to existing technologies, and achieving state-of-the-art performance on large open graph benchmark datasets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116261731B_ABST
    Figure CN116261731B_ABST
Patent Text Reader

Abstract

A system and method for improving a knowledge graph. The system includes a computing device, which includes a processor and a storage device storing computer-executable code. The computer-executable code is configured to: provide an incomplete knowledge graph comprising a plurality of nodes and a plurality of edges, each of the plurality of edges connecting two nodes of the plurality of nodes; compute an attention matrix of the incomplete knowledge graph based on a single-hop attention between any two nodes of the plurality of nodes connected by one of the plurality of edges; compute a multi-head diffusion attention between any two nodes of the plurality of nodes according to the attention matrix; obtain an updated embedding of the incomplete knowledge graph using the multi-head diffusion attention; and update the incomplete knowledge graph based on the updated embedding to obtain an updated knowledge graph.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-referencing

[0002] Pursuant to 35 U.S.C., 119(e), this application claims priority and benefit to U.S. Provisional Patent Application Serial No. 63 / 082096, filed September 23, 2020, by Guangtao Wang, Zhitao Ying, Jing Huang, and Jurij Leskovec, entitled “Method and System for Representation Learning of Relational Structures via Graph Diffusion Transformer,” the entire contents of which are incorporated herein by reference. It should be noted that the graph diffusion transformer in the aforementioned provisional application is equivalent to the multi-hop attention graph neural network (MAGNA) discussed in this disclosure.

[0003] References are cited and discussed in the description of this disclosure, which may include patents, patent applications, and various publications. The citations and / or discussions of such references are provided solely to clarify the description of this disclosure and do not imply that any such references constitute "prior art" as disclosed herein. All references cited and discussed in the specification are incorporated herein by reference in their entirety, to the same extent as each individual reference is incorporated by reference individually. Technical Field

[0004] This disclosure generally relates to relation learning, and more specifically, to methods and systems for graph representation learning using a multi-hop attention graph neural network (MAGNA). Background Technology

[0005] The background description provided herein is intended to provide a general overview of the context of this disclosure. Within the scope of this background description, the work of the currently named inventors and descriptions that might not have been considered prior art at the time of filing are not, expressly or impliedly, acknowledged as prior art to this disclosure.

[0006] The introduction of self-attention mechanisms has driven recent advancements in many fields, including graph representation learning. Graph Attention Networks (GATs) and related models have developed attention mechanisms for Graph Neural Networks (GNNs), calculating attention scores between nodes connected by edges, allowing the model to process messages from a node's direct neighbors based on these attention scores.

[0007] However, this attention mechanism does not consider nodes that are not directly connected but provide important network context. Therefore, there is an unresolved need in the art to address the aforementioned shortcomings and deficiencies. Summary of the Invention

[0008] In some aspects, this disclosure relates to a system. In some embodiments, the system includes a computing device that includes a processor and a storage device storing computer-executable code. The computer-executable code, when executed at the processor, is configured to:

[0009] Provide an incomplete knowledge graph comprising multiple nodes and multiple edges, each of the multiple edges connecting two nodes among the multiple nodes;

[0010] The attention matrix of the incomplete knowledge graph is calculated based on the single-hop attention between any two nodes among the plurality of nodes connected by one of the plurality of edges;

[0011] Calculate the multi-head diffusion attention of any two nodes among the plurality of nodes based on the attention matrix;

[0012] The updated embedding of the incomplete knowledge graph is obtained using the multi-head diffuse attention; and

[0013] The incomplete knowledge graph is updated based on the updated embedding to obtain an updated knowledge graph.

[0014] In some embodiments, the computer-executable code is configured to compute the attention matrix in the following manner:

[0015] pass Calculate edge(v) i r k v j Attention score Where v i and v j It is node i and node j, r k It is the type of the edge between node i and node j. and These are the trainable weights shared by the l-th layer of multi-head attention. This indicates the embedding of node i in the l-th layer. This indicates the embedding of node j in the l-th layer, r k It is a trainable relation embedding of the k-th relation type, where || denotes the concatenation of embedding vectors;

[0016] pass Obtain the attention score matrix S (l) ,in It is the knowledge graph mentioned above; and

[0017] Through A (l) =softmax(S (l)Calculate the attention matrix A (l) .

[0018] In some embodiments, the computer-executable code is configured to compute the multi-head diffusion attention in the following manner:

[0019] pass Calculate the multi-hop attention matrix Where hop is a positive integer in the range of 2 to 20, and θ hop It is the attention decay factor; and

[0020] pass Calculate the multi-head diffusion attention, where Θ represents the parameter of equation (1), H (l) It is the input entity embedding of the l-th layer.

[0021] In some embodiments, the following methods are used to describe the Make a rough estimate:

[0022] Setting Z (0) =H (l) Z (k+1) =(1-α)AZ (k) +αZ (0) (Equation (5)), where 0≤k≤K, θ hop =α(1-α) hop ;as well as

[0023] Will Defined as Z (K) .

[0024] In some embodiments, hop is equivalent to K, and hop and K are positive integers in the range of 2 to 12. In some embodiments, hop and K are in the range of 3 to 10. In some embodiments, hop and K are 6, 7, or 8. In some embodiments, l is a positive integer in the range of 2 to 24. In some embodiments, l is 3, 6, 12, 18, or 24. In some embodiments, l is 3, 6, or 12.

[0025] In some embodiments, the computer-executable code is configured to obtain the updated embedding of the incomplete knowledge graph by sequentially performing a first-layer normalization and addition operation, a feedforward operation, and a second-layer normalization and addition operation on the multi-head diffusion attention.

[0026] In some embodiments, the feedforward is performed using a two-layer feedforward network. The two-layer feedforward network can be a two-layer multilayer perceptron (MLP).

[0027] In some embodiments, the computer-executable code is further configured to, after obtaining the updated embedding,: compute a loss function based on the updated embedding and the labels of nodes and edges of the incomplete knowledge, and adjust the parameters used to compute the attention matrix, compute the multi-head diffusion attention, and obtain the updated embedding.

[0028] In some embodiments, the computer-executable code is configured to: iteratively compute the attention matrix, compute the multi-head attention diffusion, obtain the updated embedding, compute the loss function and adjust the parameters, and update the incomplete knowledge graph using the updated embedding obtained after the multiple iterations.

[0029] In some embodiments, the computer-executable code is configured to update the incomplete knowledge graph by predicting new features or new edges of the plurality of nodes based on the updated embedding, and adding the new features or the new edges to the incomplete knowledge graph.

[0030] In some embodiments, the computer-executable code is further configured to recommend a product to a customer when the updated knowledge graph includes multiple customers and multiple products, provided that the products and customers are linked by edges in the updated knowledge graph, wherein the edges indicate the customer's interest in the product.

[0031] In some embodiments, the computer-executable code is further configured to: when the updated knowledge graph includes multiple customers, provide credit scores for the multiple customers based on the characteristics of the customers in the knowledge graph.

[0032] In some aspects, this disclosure relates to a method. In some embodiments, the method includes:

[0033] An incomplete knowledge graph is provided by a computing device, comprising multiple nodes and multiple edges, each of which connects two nodes.

[0034] The attention matrix of the incomplete knowledge graph is calculated by a computing device based on the single-hop attention between any two nodes among the plurality of nodes connected by one of the plurality of edges;

[0035] The computing device calculates the multi-head diffusion attention of any two nodes among the plurality of nodes based on the attention matrix;

[0036] The updated embedding of the incomplete knowledge graph is obtained by the computing device using the multi-head diffuse attention; and

[0037] The computing device updates the incomplete knowledge graph based on the updated embedding to obtain an updated knowledge graph.

[0038] In some embodiments, the step of calculating the attention matrix includes:

[0039] pass Calculate edge(v) i r k v j Attention score Where v i and v j It is node i and node j, r k It is the type of the edge between node i and node j. and These are the trainable weights shared by the l-th layer of multi-head attention. This indicates the embedding of node i in the l-th layer. This indicates the embedding of node j in the l-th layer, r k It is a trainable relation embedding of the k-th relation type, where || denotes the concatenation of embedding vectors;

[0040] pass Obtain the attention score matrix S (l) ,in It is the knowledge graph mentioned above; and

[0041] Through A (l) =softmax(S (l) Calculate the attention matrix A (l) .

[0042] In some embodiments, the step of calculating the multi-head diffuse attention includes:

[0043] pass Calculate the multi-hop attention matrix Where hop is a positive integer in the range of 2 to 12, and θ hop It is the attention decay factor; and

[0044] pass Calculate the multi-head diffusion attention, where Θ represents the parameter of equation (1), H (l) It is the input entity embedding of the l-th layer.

[0045] In some embodiments, the following methods are used to describe the Make a rough estimate:

[0046] Setting Z (0) =H (l) Z (k+1) =(1-α)AZ (k) +αZ (0) (Equation (5)), where 0≤k≤K, θhop =α(1-α) hop ;as well as

[0047] Will Defined as Z (K) .

[0048] In some embodiments, hop is equivalent to K, and hop and K are positive integers in the range of 2 to 12. In some embodiments, hop and K are in the range of 3 to 10. In some embodiments, hop and K are 6, 7, or 8. In some embodiments, l is a positive integer in the range of 2 to 24. In some embodiments, l is 3, 6, 12, 18, or 24. In some embodiments, l is 3, 6, or 12.

[0049] In some embodiments, the computer-executable code is configured to obtain the updated embedding of the incomplete knowledge graph by sequentially performing a first-layer normalization and addition operation, a feedforward operation, and a second-layer normalization and addition operation on the multi-head diffusion attention.

[0050] In some aspects, this disclosure relates to a non-transitory computer-readable medium storing computer-executable code. The computer-executable code, when executed at a processor of a computing device, is configured to perform the methods described above.

[0051] These and other aspects of this disclosure will become apparent from the following description of preferred embodiments in conjunction with the accompanying drawings and their headings, although variations and modifications therein may affect the novel conception of this disclosure without departing from it. Attached Figure Description

[0052] The accompanying drawings illustrate one or more embodiments of this disclosure and, together with the written description, serve to explain the principles of this disclosure. Where possible, the same reference numerals are used throughout the drawings to refer to the same or similar elements of the embodiments.

[0053] Figure 1A The calculation of attention weights for edges according to certain embodiments of the present disclosure is illustrated schematically.

[0054] Figure 1B The diagram schematically depicts, according to certain embodiments of the present disclosure, the use of edge attention weights to compute self-attention weights between unconnected node pairs via an attention diffusion process.

[0055] Figure 2 A multi-hop attention graph neural network (MAGNA) architecture according to certain embodiments of the present disclosure is schematically depicted.

[0056] Figure 3A A multi-hop attention graph system according to certain embodiments of the present disclosure is schematically depicted.

[0057] Figure 3B A MAGNA block according to certain embodiments of this disclosure is schematically depicted.

[0058] Figure 4 The training process of a MAGNA application according to certain embodiments of the present disclosure is illustrated schematically.

[0059] Figure 5A The diagram schematically illustrates a node classification method according to certain embodiments of the present disclosure after training with MAGNA.

[0060] Figure 5B The illustration schematically depicts a method for finding new edges according to certain embodiments of the present disclosure after training with a MAGNA application.

[0061] Figure 6 Table 1 shows the node classification accuracy on Cora, Citeseer, and PubMed using MAGNA and other related methods according to certain embodiments of this disclosure.

[0062] Figure 7 Table 2 shows the node classification accuracy on the OGBArxiv dataset using MAGNA and other related methods according to certain embodiments of this disclosure.

[0063] Figure 8 Table 3 illustrates the knowledge enhancement of WN1 8PR and FB15k-237 by MAGNA and other related methods according to certain embodiments of this disclosure.

[0064] Figure 9A The following figures illustrate the rates of processing Cora datasets using MAGNA and other related methods according to certain embodiments of this disclosure.

[0065] Figure 9B The impact of model layer depth using MAGNA and other related methods on processing the Cora dataset is illustrated in certain embodiments of this disclosure.

[0066] Figure 9C The impact of hop count using MAGNA and other related methods on processing the Cora dataset is illustrated in certain embodiments of this disclosure.

[0067] Figure 9D The effect of parameter α on processing the Cora dataset using MAGNA and other related methods according to certain embodiments of this disclosure is illustrated.

[0068] Figure 10Attention weights on the Cora dataset according to certain embodiments of this disclosure are shown. Detailed Implementation

[0069] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations therein will be apparent to those skilled in the art. Various embodiments of the present disclosure are now described in detail. Referring to the accompanying drawings, throughout the views, the same numerals indicate the same parts. Unless the context clearly specifies otherwise, the terms “a,” “an,” and “the” as used herein and throughout the claims have the meaning of the plural. Furthermore, as used in the description and claims of this disclosure, unless the context clearly specifies otherwise, “in” has the meaning of “in” and “on”. And, headings or subheadings may be used in the specification for the reader's convenience, without affecting the scope of the present disclosure. In addition, some terms used in this specification are given more specific definitions below.

[0070] The terms used in this specification generally have their ordinary meanings in the art, in the context of this disclosure, and in the specific context in which each term is used. Certain terms used to describe this disclosure are discussed below or elsewhere in the specification to provide practitioners with additional guidance regarding the description of this disclosure. It will be understood that the same thing can be expressed in more than one way. Therefore, alternative languages ​​and synonyms may be used for any one or more terms discussed herein, and have no particular significance in whether a term is elaborated or discussed herein. This disclosure provides synonyms for certain terms. The use of one or more synonyms does not preclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any terms discussed herein, is merely illustrative and in no way limits the scope or meaning of this disclosure or any exemplary terms. Likewise, this disclosure is not limited to the various embodiments given in this specification.

[0071] As described herein, the term "module" may refer to or include an application-specific integrated circuit (ASIC); electronic circuitry; combinational logic circuitry; field-programmable gate array (FPGA); a processor (shared, dedicated, or grouped) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-a-chip. The term "module" may include memory (shared, dedicated, or grouped) that stores code executed by the processor.

[0072] The term "code" as used herein can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared" as used above means that some or all of the code from multiple modules can be executed using a single (shared) processor. Furthermore, some or all of the code from multiple modules can be stored in a single (shared) memory. The term "group" as used above means that some or all of the code from a single module can be executed using a group of processors. Furthermore, a group of memories can be used to store some or all of the code from a single module.

[0073] As described herein, the term "interface" generally refers to a communication tool or device used at the interaction point between components to perform data communication between components. Generally, interfaces can be applied at both the hardware and software levels, and can be unidirectional or bidirectional. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. Components communicating with the interface can be, for example, multiple components of a computer system or peripheral devices.

[0074] Introduction to Multi-hop Graph Neural Networks (MAGNA)

[0075] In some aspects, this disclosure provides a Multi-hop Attention Graph Neural Network (MAGNA) to incorporate multi-hop contextual information into attention computation, thereby enabling remote interaction at each layer of the GNN. In some embodiments, to compute attention between nodes that are not directly connected, MAGNA diffuses attention scores across the entire network, which increases the “receptive field” of each layer of the GNN. Unlike previous methods, MAGNA uses diffusion before attention values ​​to effectively account for all paths between unconnected node pairs. This helps MAGNA capture large-scale structural information at each layer and learn more informational attention. Experimental results on node classification and knowledge graph improvement benchmarks demonstrate that MAGNA achieves state-of-the-art results: MAGNA achieves a relative error reduction of up to 5:7% compared to previous state-of-the-art techniques on Cora, Citeseer, and PubMed. MAGNA also achieves state-of-the-art performance on large open graph benchmark datasets. In terms of knowledge graph improvement, MAGNA achieves state-of-the-art progress on four different performance metrics for WN18RR and FB15k-237.

[0076] Figure 1A The calculation of attention weights for edges according to certain embodiments of this disclosure is illustrated schematically. Figure 1B The illustration schematically depicts, according to certain embodiments of the present disclosure, the computation of self-attention weights between unconnected node pairs using the attention weights of edges through an attention diffusion process. For example... Figure 1AAs shown, the knowledge graph includes nodes A, B, C, and D, and the attention weight α of the edge from node B to node A. A,B Attention weight α of the edge from node C to node B B,C Attention weight α of the edge from node D to node A A,D And the attention weight α of the edge from node B to node D. D,B Attention weight α A,B This indicates the importance of the features of node B to node A in the graph, and is expressed by α. A,B =g(v A v B The calculation is performed only on the node representations v of nodes A and B. A and v B Related. Therefore, since there is no direct edge relationship between node C and node D, α D,C =0, meaning that node C in graph g has no effect on node D. Conversely, as Figure 1B As shown, the self-attention weights between unconnected node pairs can be obtained, and the attention weights between some directly adjacent nodes can be updated. For example, the self-attention weights α′ between a pair of nodes C and D. D,C It can be done through α′ D,C =f([α B,C α D,B This is obtained by using attention weight α. B,C And attention weight α D,B The attention weights between node B and node A can be calculated using α′. A,B =f([α D,B α A,D ], [α A,B The update takes into account the attention weight α. D,B And attention weight α A,D Attention weight α A,B The impact. Figure 1B The calculation of self-attention weights between unconnected nodes is based on Figure 1A The attention weights between directly adjacent nodes are determined according to the inventive concept of certain embodiments of this disclosure.

[0077] MAGNA Preparations

[0078] set up Given a knowledge graph, where It is N n A set of nodes, It is a connection In the M-to-N node e A set of edges. Each node And each edge e∈ε is associated with its type mapping function: as well as here, and This represents a collection of node labels and edge / relation types. MAGNA applications support... Learning heterogeneous graphs with multiple elements.

[0079] General Graph Neural Network (GNN) methods learn embeddings that map node and / or edge types to a continuous vector space. (Settings...) and For node embedding and edge / relation type embedding, where d n and d r Represents the embedding dimension and edge / association type of a node, x per line. i =X[i:] represents node v i (1≤i≤N n Embedding of ) r j =R[j:] represents the relation r j (1≤j≤N r ) embedding.

[0080] MAGNA is based on GNNs and combines the advantages of graph attention and diffusion techniques. The core of MAGNA is multi-hop attribute diffusion, which is a principled approach to learn attention between arbitrary node pairs in a scalable manner, taking into account the graph structure and directly enabling multi-hop context-dependent attention.

[0081] The key challenge here is how to allow flexible yet scalable context-sensitive multi-hop attention, where any node in a single GNN layer can influence the embeddings of any other node (even if they are far apart in the underlying network). Simply learning the attention scores of all node pairs is not feasible and leads to overfitting and poor generalization.

[0082] Multi-jump attention diffusion in MGANA

[0083] In one respect, attention diffusion is introduced by directly computing multi-hop attention at each block of MAGNA based on the attention score in each MAGNA block. The input to the attention diffusion operator is a set of triples (v...) currently available in the knowledge graph. i r k v j ), where v i v j It is a node, r kIt is an edge type. MAGNA first calculates the attention score for all available edges. Then, MAGNA's attention diffusion module calculates the attention value between pairs of nodes that are not directly connected by edges, based on the edge attention scores, through a diffusion process. The attention diffusion module can then be used as a component in the MAGNA architecture.

[0084] To compute attention diffusion, in the first phase, MAGNA computes single-hop edge attention, i.e., the attention between nodes connected by an edge. MAGNA consists of one or more blocks, each of which is also called a MAGNA layer. At each MAGNA block (or layer), for each triple (v... i r k v j ) Calculate the vector message. This is to calculate v at block or layer l+1. j The expression will be related to v j All relevant messages from the triples are aggregated into a single message, which is then used for updates.

[0085] This disclosure first calculates the edge (v) in the following manner. i r k v j Attention score s:

[0086]

[0087] in and The trainable weights shared by the l-th layer, d (l) Or d is the dimension of the node, d r It is the dimension of the edge. This indicates the embedding of node i in the i-th layer. r k It is the k-th relation type r k (1≤k≤N r The trainable relation embeddings are given by α and β, where a||b represents the concatenation of embedding vectors α and b. For graphs without relation types, this disclosure treats such relation types as a degenerate class distribution with a single class.

[0088] Apply equation (1) to the graph For each edge, this disclosure obtains the attention score matrix S. (l) :

[0089]

[0090] Subsequently, this disclosure uses the scoring matrix S (l) A (l) =softmax(S (l)Perform a row-by-row softmax to obtain the attention matrix A. (l) Attention matrix A (l) For single-hop attention, the attention matrix A (l) In This represents the attention value at layer l when aggregating messages from node j to node i.

[0091] In the second stage, MAGNA is based on the single-hop edge attention matrix A of the l-th block or layer. (l) Calculate the attention diffusion of multi-hop neighbors. In this stage, this disclosure initiates attention between nodes that are not directly connected in the knowledge graph network. This is achieved through the following attention diffusion procedure, wherein this disclosure first calculates the attention scores of multi-hop neighbors through graph diffusion based on the power of the 1-hop attention matrix A:

[0092]

[0093] in And θ i >0.

[0094] here To increase attention span, A i Let A be the single-hop attention matrix (A is the single-hop attention matrix at the l-th MAGNA block). (l) The power of θ, where i is an integer equal to or greater than 0 (please distinguish the power i here from the node i disclosed elsewhere in this disclosure), θ i Let θ be the attention decay factor and θ i >θ i+1 The power of A in the attention matrix i This provides us with the number of relationship paths between two nodes of length i, increasing the receptive field of attention. For example, two-hop attention can be calculated as: Three-hop attention can be calculated as follows: Where I is the identity matrix. As shown in equation (3), this mechanism allows the attention between two nodes to depend not only on their previous layer representations but also on the path between the nodes, effectively creating attention shortcuts between nodes that are not directly connected. The attention through each path is also weighted differently according to θ and the path length i. In some embodiments, this disclosure utilizes the geometric distribution θ i =α(1-α) i , where α∈(0,1]. This choice is based on inductive bias, meaning that nodes farther apart in message aggregation should be weighted less, and nodes with different relational path lengths to the target node should be weighted independently in sequence. Furthermore, it should be noted that if we define θ0=α∈(0,1], and A i=I, then equation (3) gives the process of personalized page ranking (PPR) on a graph with attention matrix A and delivery probability α. Therefore, the diffusion attention weight This can be viewed as the influence of node j on node i.

[0095] because We can also Consider the attention value of node j to i (as defined by attention A) (l) =softmax(S (l) (and equation (3) are obtained). Then, this disclosure defines graph attention diffusion based on feature aggregation as:

[0096]

[0097] here H represents a knowledge graph. (l) Let Θ be the embedding of the node at the l-th MAGNA block, and Θ be the set of parameters used to compute the attention. Through the diffusion process defined in Equation (3), MAGNA uses the same number of parameters as if we were only computed to compute the attention between nodes connected by edges. In other words, the learnable parameters Θ are the same as the learnable parameters in Equation (1). and The same or equal. This ensures runtime efficiency and good model generalization.

[0098] The purpose of this disclosure is to obtain the attention diffusion. However, for large graphs, since the attention matrix A is calculated to the power of i, the exact attention diffusion matrix is ​​calculated using equation (3). This can be extremely expensive. To address this bottleneck, in some embodiments, this disclosure provides attention diffusion. Approximate calculations. Specifically, this disclosure sets H... (l) For the first block or layer (H) (0) =X) input entity embedding, θ i =α(1-α) i Because MAGNA only needs to go through To perform aggregation, this disclosure defines a sequence Z. (K) To make a rough estimate The sequence converges to The truth value is K→∞:

[0099] Z (0) =H (l) Z (k+1) =(1-α)AZ (k) +αZ (0) , where 0≤k≤K (5)

[0100] Proposition 1.

[0101] Replacing the calculations derived from equations (3) and (4) with the rough estimate obtained through equation (5), the computational complexity of attention with diffusion remains O(|E|), with a constant factor corresponding to the number of hops K. In some embodiments, this disclosure defines the value of K in the range of 3 ≤ K ≤ 10, which achieves good model performance. Many real-world graphs exhibit small-world properties, in which case even a small value of K is sufficient. For graphs with larger diameters, this disclosure chooses a larger K and reduces the value of α.

[0102] GNN architecture based on direct multi-hop attention

[0103] Based on the above calculation of multi-hop attention diffusion through equations (1), (2) and (5) or equations (1) to (4), this disclosure provides a GNN architecture based on direct multi-hop attention, namely MAGNA. Figure 2 A MAGNA architecture according to certain embodiments of this disclosure is schematically depicted. For example... Figure 2 As shown, the MAGNA architecture comprises L stacked blocks. Each stacked block has a multi-head attention diffusion layer 202 and a deep aggregation layer 204.

[0104] like Figure 2 As shown, the multi-head graph attention diffusion layer 202 includes multi-head attention between nodes, aggregation of diffusion attention, and layer normalization and addition operations. Multi-head attention is used to enable the model to jointly pay attention to information from different representation subspaces from different viewpoints. In equation (6), the attention diffusion of each head i is calculated separately using equation (4), and then aggregated:

[0105]

[0106] in

[0107] Here, || represents cascading, and Θ i W is the parameter of the i-th head (1≤i≤M) in equation (1). o The parameter matrix is ​​represented. Since this disclosure calculates attention diffusion recursively in equation (5), layer normalization is added, which helps stabilize the recursive calculation process. As the final step of the multi-head graph attention diffusion layer 202, through... Perform the addition operation.

[0108] like Figure 2As shown, the deep aggregation layer 204 includes fully connected feedforward sublayers. In some embodiments, the fully connected feedforward sublayers include two feedforward networks, such as two MLP layers. Furthermore, layer normalization and residual connections are added to both sublayers 202 and 204 to make the aggregation steps of each block more expressive.

[0109]

[0110] Recalling the above description, MAGNA differs from GAT; it generalizes upon GAT. MAGNA extends GAT through a diffusion process. The features in GAT aggregate into H. (l+1) =σ(AH (l) W (l) ), where σ represents the activation function. We can divide the GAT layer into two components, as shown below:

[0111]

[0112] In component (1), MAGNA removes the restriction on notifying direct neighbors, and because Since it is derived from A, no additional parameters are required. For component (2), MAGNA uses layer normalization and deep aggregation, which achieves significant gains according to the ablation studies in Table 1, described later in the experimental section of this disclosure. Compared to the “shallow” activation function elu in GAT, we can consider deep aggregation (i.e., two-layer MLP) as a learnable deep activation function because a two-layer MLP can coarsely estimate many different functions.

[0113] Analysis of attention diffusion in graphs

[0114] In this section, we investigate the advantages of MAGNA from the perspective of discrete signal processing on graphs (Sandryhaila & Moura, Discrete signal processing on graphs: graph Fourier transform, ICASSP, 2013). Our first result shows that MAGNA can better capture large-scale structural information. Our second result explores the relationship between MAGNA and personalized page ranking (PPR).

[0115] Spectral characteristics of graph attention diffusion. We treat the attention matrix A of GAT and A of MAGNA as weighted adjacency matrices, and apply graph Fourier transform and spectral analysis to show the effect of MAGNA as a graph low-pass filter, which can more effectively capture large-scale structures in the graph. Through equation (3), Or the sum of each row of A is 1. Therefore, for And A, the normalized graph Laplacian operator are respectively and L sym =IA. We can obtain the following proposition:

[0116] Proposition 2. Setting and for and L sym The i-th eigenvalue:

[0117]

[0118] We also have Equation (9) shows that when When smaller, but And for larger This relationship indicates the use of It increases smaller eigenvalues ​​and decreases larger eigenvalues. The smaller α is, the greater the low-pass effect.

[0119] The eigenvalues ​​of low-frequency signals describe the large-scale structure in graphs and have proven crucial in graph tasks. Because... and The reciprocal format in equation (9)9 amplifies the ratio of lower eigenvalues ​​to the sum of all eigenvalues. Conversely, high eigenvalues, corresponding to noise, are suppressed.

[0120] Personalized page ranking satisfies the principle of graph attention diffusion. Because And A i,j If the value is greater than 0, we can also view the attention matrix A as a graph. The random walk matrix on A. If we use parameters α∈(0,1] in a matrix with transition matrix A. Personalized PageRank (PPR) is implemented on the page, and fully personalized page ranking is defined as:

[0121] A ppr =α(I-(1-α)A) -1 (10)

[0122] Using the power series expansion of the matrix inverse, we obtain:

[0123]

[0124] With θ i =α(1-α) i Compared to the diffusion equation (3), we have the following proposition:

[0125] Proposition 3. Graph attention diffusion uses parameters α∈(0,1] in a graph with transition matrix A. The above defines personalized page ranking, that is

[0126] The parameters in MAGNA are equivalent to the transmission probability of PPR. PPR provides a good correlation score between nodes in the weighted graph (from the weights of the attention matrix A). In summary, MAGNA places PPR before the pairwise attention score of nodes: the distributed attention between node i and node j depends on the attention score on the edges of all paths between i and j.

[0127] Implementing this disclosure in a computing device

[0128] This disclosure relates to computer systems. As shown in the figures, computer components may include physical hardware components as indicated by solid boxes and virtual software components as indicated by dashed boxes. Those skilled in the art will understand that, unless otherwise instructed, these computer components may be implemented in the form of software, firmware, or hardware components, or combinations thereof, but are not limited to these forms.

[0129] The apparatus, systems, and methods described herein can be implemented by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored in a non-transient tangible computer-readable medium. The computer program may also include stored data. Non-limiting examples of non-transient tangible computer-readable media are non-volatile memory, magnetic memory, and optical memory.

[0130] This disclosure will now be described more fully below with reference to the accompanying drawings, in which embodiments of the disclosure are illustrated. However, this disclosure may be embodied in many different forms and should not be construed as limiting the embodiments described herein. Rather, these embodiments are provided to make this disclosure comprehensive and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0131] Figure 3A A multi-hop attention map system according to certain embodiments of the present disclosure is schematically depicted. For example... Figure 3A As shown, system 300 includes computing device 310. In some embodiments, computing device 310 may be a server computer, cluster, cloud computer, general-purpose computer, headless computer, or dedicated computer providing annotation classification or knowledge graph (KG) improvement services. Computing device 310 may include, but is not limited to, processor 312, memory 314, and storage device 316. In some embodiments, computing device 310 may include other hardware and software components (not shown) to perform its respective tasks. Examples of these hardware and software components may include, but are not limited to, other required memory, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0132] Processor 312 may be a Central Processing Unit (CPU) configured to control the operation of computing device 310. Processor 312 may execute the operating system (OS) or other applications of computing device 310. In some embodiments, computing device 310 may have more than one CPU as a processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. Memory 314 may be volatile memory, such as random-access memory (RAM), used to store data and information during the operation of computing device 310. In some embodiments, memory 314 may be a volatile memory array. In some embodiments, computing device 310 may run on one or more memories 314. Storage device 316 is a non-volatile data storage medium for storing the OS (not shown) and other applications of computing device 310. Examples of storage device 316 may include non-volatile memory such as flash memory, memory cards, USB drives, hard disk drives, floppy disks, optical disk drives, solid-state drives, or any other type of data storage device. In some embodiments, computing device 310 may have multiple storage devices 316, which may be the same storage devices or storage devices of different types, and applications of computing device 310 may be stored in one or more storage devices 316 in computing device 310.

[0133] In this embodiment, processor 312, memory 314, and storage device 316 are components of computing device 310, such as a server computing device. In other embodiments, computing device 310 may be a distributed computing device, and processor 312, memory 314, and storage device 316 may be shared resources from multiple computing devices in a predefined region.

[0134] Storage device 316 specifically includes a multi-hop attention graph neural network (MAGNA) application 318 and a knowledge graph 332. MAGNA application 318 is configured to train its model structure using labels from knowledge graph 332 and to make predictions to improve or refine knowledge graph 332. Knowledge graph 332 is optional for computing device 310, provided that the knowledge graph stored on other devices can be accessed by MAGNA application 318.

[0135] like Figure 3AAs shown, the MAGNA application 318 includes a data preparation module 320, one or more MAGNA blocks 322, a loss function module 324, a prediction module 326, a function module 328, and an interface 330. In some embodiments, the MAGNA application 318 may include other applications or modules required for the operation of the MAGNA application 318. It should be noted that modules 320 to 330 are each implemented as computer-executable code or instructions, data tables or databases, or a combination of hardware and software, which together form an application. In some embodiments, each of the modules may also include sub-modules. Alternatively, some of the modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code. In some embodiments, the modules may also be collectively referred to as a model, which can be trained using training data and, after training, can be used for prediction.

[0136] Data preparation module 320 is configured to prepare training or prediction samples and send the prepared training or prediction samples to the MAGNA block. Knowledge graph 332 can have over a thousand to hundreds of thousands of nodes, but the types of edges are typically limited, such as one type of edge (yes and no) or several types of edges. Node features are stored in knowledge graph 332; when a node includes a customer, the node features could be the customer's age, gender, location, education, etc. Edges or relationships between nodes are stored in knowledge graph 332; for example, customer nodes and product nodes could have a browsing or purchasing relationship. In some embodiments, knowledge graph 332 may not be a complete knowledge graph, possibly lacking node features or edges between certain related nodes. In this case, both training and prediction samples can be knowledge graph 332. Known labels of nodes and edges between nodes in knowledge graph 332 are used for training MAGNA application 218. After training, the trained MAGNA application 318 can be used to obtain features of more nodes in knowledge graph 332 or to refine knowledge graph 230. In some embodiments, the data preparation module 320 can prepare the knowledge graph 332 by embedding nodes and edges into vectors. Specifically, it sets... Given a knowledge graph 332, where v is N n A set of nodes, It is the N that connects M pairs of nodes in v. e A set of edges. Each node And each edge e∈ε is associated with its type mapping function: φ: as well as here, and This represents a set of node types (labels) and edge / relationship types. MAGNA application 318 supports this. Learning heterogeneous graphs with multiple elements.

[0137] MAGNA block 322 is configured to, upon receiving a training knowledge graph or a knowledge graph for prediction from data preparation module 320, train MAGNA block 322 and classifier or KG improvement module 326, or perform prediction using trained MAGNA block 322 and classifier or KG improvement module 326. MAGNA block 322 may include one or more MAGNA blocks 3220, each MAGNA block 3220 having the same block structure. Figure 3B A MAGNA block 3220 according to certain embodiments of this disclosure is schematically depicted. Figure 3B As shown, the MAGNA block 3220 includes a first-layer normalization module 3221, a multi-head attention diffusion module 3222, a first addition operation module 3223, a second-layer normalization module 3224, a feedforward module 3225, and a second addition operation module 3266.

[0138] At the start of training with MAGNA application 318, the embedding node H (0) The first-level normalization module 3221 and the first addition operation module 3223 can be used for the first MAGNA block. After the operation of the l-th MAGNA block, the node embedding output by the block, i.e., H... (l) This can be used in the first-level normalization module 3221 and the first addition operation module 3223 of the next MAGNA block. If the current MAGNA block is the last MAGNA block, the output node embedding is provided to the classifier or KG refinement module 326.

[0139] The first-layer normalization module 3221 is configured to embed H at the node that receives the input at the l-th block. (l) At that time, embed H for the input nodes and edges. (l) Execution layer normalization to obtain the first normalized embedding. The first normalized embedding is then sent to the multi-head attention diffusion module 3222. The first level of normalization is defined as follows:

[0140] The multi-head attention diffusion module 3222 is configured to, upon receiving the first normalized embedding At that time, calculate the attention spread of each head. i Aggregate the attention diffusion of all heads to obtain aggregated attention diffusion. The node is embedded and sent to the first addition operation module 3223 with aggregation attention diffusion. The node embedding. The attention spread of head i through The computation can be performed using the first normalized embedding. The model parameters of the multi-head attention diffusion module 3222 are calculated according to equations (1), (2), (3), and (4). In some embodiments, equation (5) is used instead of equations (3) and (4) to approximate the attention diffusion of head i. When attention diffusion for all heads is available, the attention diffusion is calculated by equation (6): cascade.

[0141] The first addition operation module 3223 is configured to, upon receiving the aggregated attention diffusion... At that time, the aggregated attention diffusion is added to the input embedding to obtain the first addition operation. The embedding, and will have the first addition operation The embedding is sent to the second-level normalization module 3224 and the second addition operation module 3226. The input embedding can be a node embedding from the preceding MAGNA block, or if the current MAGNA block 3220 is the first block of the MAGNA block, it can be from the initial embedding H. 0 The operation steps are defined as follows:

[0142] The second-level normalization module 3224 is configured to receive a first addition operation. During embedding, the embedding is normalized to obtain a second normalized embedding, and the second normalized embedding is sent to the feedforward module 3225. Layer normalization is performed by LayerNorm(H (l+1) )implement.

[0143] Feedforward module 3225 is configured to, upon receiving a first addition operation from first-layer addition module 3223, perform a first addition operation. The embedding and the second normalized embedding LayerNorm(H) (l+1) When ), feedforward is performed to obtain feedforward embedding. The feedforward embedding is then sent to the second addition operation module 3226.

[0144] The second addition operation module 3226 is configured to, upon receiving a first addition operation from the first addition operation module 3223, perform a first addition operation. Embedded and feedforward embedding from feedforward module 3225 At that time, an addition operation is performed on both to achieve the desired result. Get the updated node embedding H (l+1) This allows the update node to embed H (l+1) It can be used in the (l+1)th MAGNA block, or in the loss function module 324 when the current block is the last MAGNA block.

[0145] Return to reference Figure 3BThe loss function module 324 is configured to embed H upon receiving an update node. (l+1) At this time, the node embedding update is used to classify nodes that already have labels, or to predict edge relationships that already have labels. The classified node labels or predicted edge relationships are compared with the previous labels, and the loss function is calculated based on the comparison. The model parameters of MAGNA block 322 are adjusted, and another round of training is performed. Training can end when the predetermined number of rounds of training has been completed or the model parameters have converged. When the model is trained well, the loss function module 324 is also configured to notify the prediction module 326.

[0146] The prediction module 326 is configured to, upon receiving notification from the loss function module 324 that the model is trained, use the trained MAGNA block 322 to classify nodes that do not have a pre-classified type, or predict relationships between nodes not linked by edges, add new node classifications and / or new edges, and / or new edge relationships in the knowledge graph 332, thereby updating the knowledge graph 332 with more information. The updated knowledge graph 332 can be used by the functional module 328. In some embodiments, the prediction module 326 is a decoder for a field transformer. In some embodiments, the decoder is a classifier.

[0147] Functional module 328 is configured to perform certain functions using the updated knowledge graph 332 when the knowledge graph 332 is updated. For example, when the knowledge graph 332 is a customer and product knowledge graph, it can be used to recommend products to one or more customers when it is predicted that a customer will be interested in a product. This can be indicated by edges or relationships linking products and customers. In some embodiments, the knowledge graph 332 can be a customer knowledge graph, where each customer can be categorized as a high-credit-score customer or a low-credit-score customer. By updating the high- or low-credit-score customer classification by prediction module 326, more customer credit information becomes available, and the customer credit data can be used by lending companies.

[0148] In some embodiments, functional module 328 is configured to perform the above functions automatically or at predefined time intervals, or when triggered by an update of knowledge graph 322. For example, after knowledge graph 332 is updated, functional module 328 will look for more links between products and customers, and when the updated relationship between a customer and a product is of interest, functional module 328 will then push the product to the corresponding customer.

[0149] Interface 330 is configured to provide an interface to the administrator of MAGNA application 318 for training MAGNA block 322 and optionally training loss function module 324, and adjusting model parameters, or is configured to provide an interface for the administrator to use MAGNA application 318 to obtain and use updated knowledge graph 332 for certain functions.

[0150] It should be noted that in different contexts of this disclosure, 'i' can have different meanings. For example, 'v' i In this context, i represents the i-th node, and θ i and A i In this context, i is a positive integer representing the path length of the edge between nodes, and head... i In this context, 'i' represents the i-th head used to calculate attention spread.

[0151] Figure 4 The training process of a MAGNA application according to certain embodiments of the present disclosure is schematically depicted. In some embodiments, the training process is performed by... Figure 3A The computing device 310 shown is implemented. It should be noted that, unless otherwise stated in this disclosure, the steps of the training process or method may be arranged in a different order, and are therefore not limited to... Figure 4 The order shown.

[0152] like Figure 4 As shown, in step 402, the data preparation module 320 retrieves the knowledge graph 332 and processes the knowledge graph 332 to obtain the initial embedding H of the knowledge graph 332. 0 and the initial embedded H 0 Provided to the first MAGNA block 3320 in MAGNA block 322. Knowledge graph 332 has labeled nodes and edges, but the node labels may be incomplete, and knowledge graph 332 may be missing many edges. Initial embedding H 0 This refers to the embedding of entities or nodes in Knowledge Graph 332.

[0153] In step 404, the first-layer normalization module 3221 of the first MAGNA block 3220 performs layer normalization on the initialized embedding to obtain a normalized embedding. Right now, The normalized embedding is then sent to the multi-head attention diffusion module 3222.

[0154] In step 406, upon receiving the normalized embedding... At that time, the multi-head attention diffusion module 3222 calculates the attention score matrix, where each row of the attention score matrix is ​​the attention score of an edge. One edge (v...) i r k v j Attention score using Calculate the attention score matrix. pass To summarize, the attention score matrix includes the attention score for each edge currently present in knowledge graph 332.

[0155] In step 408, the multi-head attention diffusion module 3222 performs attention scoring on the matrix. Perform softmax to obtain the attention matrix A (0) A (0) =softmax(S (0) ).

[0156] In step 410, the multi-head attention diffusion module 3222 uses attention matrix A (0) Computational graph attention diffusion In some embodiments, graph attention diffusion is used to improve computational speed. Approximate calculations are performed. Specifically, this is done by defining... and Z (k+1) =(1-α)AZ (k) +αZ (0) Then we get Here, α is a predefined constant in the range of 0 to 0.5. In some embodiments, α is in the range of 0.05 to 0.25. In some embodiments, α is in the range of 0.1 to 0.2. In some specific embodiments, α is 0.1 or 0.15. 0 ≤ k ≤ K, and K is a positive integer in the range of 3 to 10. In some embodiments, K is in the range of 4 to 8. In some embodiments, K is 6. The values ​​of α and K can vary depending on the size and characteristics of knowledge graph 332. For example, assuming α is 0.1 and K is 6, then Z (1) =0.9AZ (0) +0.1Z (0) Z (2) =0.9AZ (1) +0.1Z (0) Z (3) =0.9AZ (2) +0.1Z (0) Z (4) =0.9AZ (3) +0.1Z (0) Z (5) =0.9AZ (4) +0.1Z (0) Z (6) =0.9AZ (5) +0.1Z (0) And Z (6) In order to spread attention, is An approximation of the value. In some embodiments, equations (3) and (4) can also be used to approximate the graph attention diffusion. Perform the calculation. However, due to Z (1) Z (2) Z (3) Z (4) Z (5), and Z (6) It is calculated recursively, so this method of calculation is much faster than using equations (3) and (4).

[0157] In step 412, steps 404 to 410 are performed for each head to obtain the graph attention diffusion of the head. Multi-head attention diffusion module 3222 summarizes the graph attention diffusion of all heads. To achieve aggregated attention diffusion Right now, In achieving convergent attention diffusion Subsequently, the multi-head attention diffusion module 3222 also sends the multi-head attention diffusion to the first addition operation module 3223.

[0158] In step 414, when a multi-head attention diffusion is received from the multi-head attention diffusion module 3222, the first addition operation module 3223 aggregates the attention diffusion. Add to the initial embedded H 0 , in order to pass Obstacles to the diffusion of attention after manipulation The attention diffusion after the addition operation is sent to the second-layer normalization module 3224.

[0159] In step 416, upon receiving the attention diffusion after the addition operation, the second-layer normalization module 3224 performs layer normalization on the attention diffusion after the addition operation to obtain normalized attention diffusion. The normalized attention diffusion is then sent to the feedforward module 3225.

[0160] In step 418, when normalized attention diffusion is received from the second-layer normalization module 3224, the feedforward module 3225 performs feedforward on the normalized attention diffusion to obtain feedforward attention, and sends the feedforward attention to the second addition operation module 3226. The feedforward attention is...

[0161] In step 420, when feedforward attention is received from feedforward module 3225, second addition module 3226 diffuses the attention after addition operation into the feedforward attention to obtain updated embedding H. (1) ,Right now: After obtaining the updated embedded H (1) Subsequently, the second addition operation module 3226 updates the embedding sent to the second block in MAGNA block 322. In some embodiments, the feedforward attention has two layers of feedforward networks, such as two layers of MLP.

[0162] In step 422, the second block in MAGNA block 322 repeats steps 404 to 420, but in step 404, the input is not the initial embedded H.(0) Instead, it is an updated embedded H (1) In some embodiments, MAGNA block 322 comprises 2 to 10 blocks. In some embodiments, MAGNA block 322 comprises 3 to 6 blocks. The number of blocks may depend on the size and features of knowledge graph 332. Steps 404 to 420 are performed for each block to update the embedding. When the number of blocks is L, the blocks are block 0 (or layer 0), block 1 (or layer 1), ..., block (L-1) (or layer (L-1)), and the final output of the embedding is H. (L) In obtaining the embedded H (L) After the output, MAGNA block 322 will also send the output embedding to loss function module 324.

[0163] In step 424, after receiving the output embedding from MAGNA block 322, loss function module 324 computes a loss function by comparing the output embedding with the ground truth labels of knowledge graph 332, and uses the loss function to adjust the parameters of MAGNA module 322. In some embodiments, the loss function is cross-entropy loss.

[0164] In step 426, MAGNA application 318 iteratively performs steps 404 to 424 using the updated embeddings from previous iterations until training is repeated a predetermined number of times, or until the model parameters converge.

[0165] Figure 5A A method 500A for node classification according to certain embodiments of the present disclosure is schematically depicted after training with MAGNA application 318. In some embodiments, the method is performed by computing device 310, particularly... Figure 3A The prediction module 326 shown is implemented. It should be noted that, unless otherwise stated in this disclosure, the steps of the training process or method may be arranged in a different order, and are therefore not limited to the one described above. Figure 5A The order is shown. Note that the training data for MAGNA application 318 can be the same knowledge graph 332.

[0166] like Figure 5A As shown, in step 426, steps 404 to 420 are iteratively executed to continuously update the embeddings of the knowledge graph until a predetermined number of iterations are completed or the model parameters converge. Then, in step 502, after obtaining the updated embeddings, the prediction module 326 uses the updated embeddings to predict the features of the nodes in the knowledge graph 332. For example, when a node is a customer, the feature could be high credit score and low credit score. In some embodiments, the prediction module 326 only predicts nodes that do not have feature values. Alternatively, the prediction module 326 can predict the features of all nodes and then filter out nodes that already have feature labels or values.

[0167] The probabilities of the predicted features of nodes can be different. In step 504, the prediction module 326 sorts the nodes from high to low based on the probabilities or confidence levels of the predicted features of the nodes.

[0168] In step 506, prediction module 326 selects the top-ranked nodes and adds the predicted features to the selected nodes. Adding feature values ​​to nodes that do not have features makes the knowledge graph 332 more complete.

[0169] Figure 5B A method 500B for finding new edges in a knowledge graph, according to certain embodiments of this disclosure, is schematically depicted after training with MAGNA application 318. In some embodiments, the method is performed by computing device 310, particularly... Figure 3A The prediction module 326 shown is implemented. It should be noted that, unless otherwise stated in this disclosure, the steps of the training process or method may be arranged in a different order, and are therefore not limited to the one described above. Figure 5B The order is shown. Note that the training data for MAGNA application 318 can be the same knowledge graph 332.

[0170] like Figure 5B As shown, in step 426, steps 404 to 420 are performed iteratively to continuously update the embeddings of the knowledge graph until a predetermined number of iterations are completed or the model's parameters converge. Then, in step 512, after obtaining the updated embeddings, the prediction module 326 uses the updated embeddings to predict new edges in the knowledge graph 332. For example, when the nodes are customers and products, and the edge type of the knowledge graph includes "interested" relationships, the new edges can link some customers to some products where there was no previously available link between the customer and the product.

[0171] The probabilities of the predicted edges can be different. In step 514, the prediction module 326 sorts the new predicted edges based on their probability or confidence level.

[0172] In step 516, the prediction module 326 selects the top-ranked new edges and adds the predicted new edges to the knowledge graph. Adding new edges that did not exist before makes the knowledge graph 332 more complete.

[0173] In some aspects, this disclosure provides methods for using a refined knowledge graph. In some embodiments, the method is a credit assessment method, which may include: refining the knowledge graph using, for example, a person's credit history, such as... Figure 5AAs shown, each person in the large group is a node in the knowledge graph, edges represent connections between people's social interactions, and node labels are people's credit categories (e.g., bad credit, good credit, excellent credit) or credit scores. Due to the application of MAGNA, the credit categories assigned to nodes are accurate. Upon receiving a person's loan request, if the person has good or excellent credit, a loan is provided, or a loan limit is determined based on the person's credit score. In some embodiments, the method is a product recommendation method, which may include: using, for example, customers' purchase history to enrich the knowledge graph, such as... Figure 5B As shown, each of the large number of customers and products is a node in the knowledge graph, and the edge between the customer node and the product node represents the probability that the customer wants to buy the product; selecting products with strong edge relationships with the customer; and sending information about the selected products to the customer. In some embodiments, the method is friend recommendation, and the recommendation method may include: using information from social networks to enrich the knowledge graph, such as... Figure 5B As shown, each user in a large number of social network users is a node in the knowledge graph, and the edge between one user node and another user node represents the probability that users will become friends if they have the opportunity to meet each other; the selection of candidate users with strong edge relationships with the target user; and sending information about candidate users to the target user.

[0174] experiment

[0175] We evaluated two classic tasks using MAGNA. (1) In node classification, we achieved an average relative error reduction of 5:7%; (2) In knowledge graph improvement, we achieved a relative improvement of 7:1% on a 1-metric basis. We compared our results with those reported in available baseline papers.

[0176] Task 1: Node classification.

[0177] Datasets. We used four benchmark datasets for node classification: (1) the standard reference network benchmarks Cora, Citeseer, and Pubmed; and (2) the Ogbn-arxiv benchmark dataset with 170K nodes and 1.2M edges from the Open Graph Benchmark. We followed standard data splitting for all datasets.

[0178] Baseline. We compared our approach with a comprehensive suite of state-of-the-art GNN methods, including: GCN, Chebyshev-based GCN, DualGCN, JKNet, LGCN, Diffusion-GCN, APPNP, Graph U-Nets (gU-Nets), and GAT.

[0179] Experimental setup. For the Cora, Citeseer, and PubMed datasets, we used six MAGNA blocks with a hidden dimension of 512 and eight attention heads. For the large-scale OGBN-ARXIV dataset, we used two MAGNA blocks with a hidden dimension of 128 and eight attention heads.

[0180] Results. We report the node classification accuracy on benchmark tests. Results are summarized in Figure 6 Table 1 and Figure 7 Table 2 shows the node classification accuracy for Cora, Citeseer, and Pubmed, while Table 1 shows the node classification accuracy for the OGB Arxiv dataset. MAGNA improves upon all methods and implements state-of-the-art techniques on all datasets.

[0181] Ablation Study. We report the model performance in Table 1 after removing each component of MAGNA (layer normalization, attention diffusion, and deep aggregation feedforward) from each layer of MAGNA. Note that this model is equivalent to GAT without these three components. We observe that diffusion and layer normalization play a crucial role in improving node classification performance across all datasets. While layer normalization alone does not benefit GNNs, its combination with the attention diffusion module significantly improves MAGNA's performance. Layer normalization is essential for ensuring training stability because MAGNA computes many attention values. We also removed layer normalization and the deep aggregation feedforward, retaining only the attention diffusion layer (see the second-to-last row of Table 1). Compared to GAT, attention diffusion allows for multi-hop attention within each layer while still benefiting node classification performance.

[0182] Task 2: Improve the knowledge graph

[0183] Datasets. We evaluate MAGNA on the standard benchmark knowledge graphs WN18RR and FB15K-237.

[0184] Baselines. We compare MAGNA with state-of-the-art baselines, including (1) translation distance-based models: TransE and its latest extensions RotatE, OTE and ROTH; (2) semantic matching-based models: ComplEx, QuatE, CoKE, convE, DistMult and TuckER; and (3) GNN-based models: R-GCN, SACN and A2N.

[0185] Training Procedure. We use the standard training procedure used in previous KG embedding models. We follow an encoder-decoder framework. The encoder applies the proposed MAGNA model to compute entity embeddings. The decoder then performs link predictions given the embeddings output from MAGNA, and can apply existing decoders from previous models. To demonstrate the exponentiation of MAGNA, we use the DistMult decoder, a simple decoder with no additional parameters.

[0186] Evaluation. We use the standard splitting of the benchmark and, given the head (tail) entities and relation types, the standard testing procedure for predicting the tail (head) entities. We fully follow the evaluation methods used in all previous works, namely Mean Reciprocal Rank (MRR), Mean Rank (MR), and hit rate at K (H@K).

[0187] Result. Figure 8 As shown in Table 3, MAGNA achieves state-of-the-art knowledge graph improvement across all four metrics. MAGNA demonstrates superior performance compared to state-of-the-art shallow embedding methods (QuatE) and deep embedding methods (SACN). It is noteworthy that when using the same decoder (DistMult), MAGNA achieves significant improvements using its own embeddings compared to using the corresponding DistMult embeddings.

[0188] MAGNA Model Analysis

[0189] Here we present (1) the spectral analysis results, (2) the influence of hyperparameters on MAGNA performance, and (3) the attention distribution analysis to demonstrate the advantages of MAGNA.

[0190] Spectral analysis. Why is MAGNA suitable for node classification? We computed the eigenvalues ​​of the graph Laplacian of the attention matrix A. and with the diffusion matrix eigenvalues ​​of the graph Laplace A comparison was made. Figure 9A The ratios on the Cora dataset are shown. Low eigenvalues ​​corresponding to large-scale structures in the figure are amplified (up to 8 times), while high eigenvalues ​​corresponding to eigenvectors with noisy information are suppressed.

[0191] MAGNA deep model. We conducted experiments on Cora by varying the number of GCN, GAT, and our MAGNA layers to 3, 6, 12, 18, and 24. Figure 9BThe results show that both deep GCNs and deep GATs (even with residual connections) suffer from performance degradation due to oversmoothing issues. In contrast, the MAGNA model consistently achieves the best results, even with 18 layers, demonstrating the robustness and expressiveness of deep MAGNA models. Note that despite having the same receptive field, a GAT with 18 layers does not outperform a MAGNA with 3 layers and K=6 hops.

[0192] The effects of K and α. Figure 9C and Figure 9D The impact of the number of hops K and the transmission probability on model performance is reported. We observed a significant performance improvement when considering information from multi-hop neighbors (K > 1). However, for K ≥ 6, increasing the number of hops K leads to diminishing returns. Furthermore, we found that the optimal K is correlated with the maximum average shortest path distance between nodes (e.g., 5.27 for Cora). This provides guidance for selecting the optimal K. Here, the average shortest path distance between nodes refers to the average path length from a given node to other nodes.

[0193] We also observed that accuracy decreased significantly when α was larger than 0.25. This is because smaller α increases the low-pass effect. Figure 9A However, if α is too small, the model will only focus on large-scale graph structures and ignore too much high-frequency information.

[0194] Attention allocation. Finally, we also analyzed the learning attention scores of GAT and MAGNA. We first assigned node v i The difference metric on the attention matrix A is defined as (Shanthamallu et al., 2020), of which U i For node v i The score is uniformly distributed. △ i This indicates the degree to which learning attention deviates from a non-informative uniform distribution. Larger Δ i This indicates a more meaningful attention score. Figure 10 The distribution of the difference measure for the first layer of MAGNA and the first head attention matrix of GAT is shown. It is observed that the attention scores learned in MAGNA exhibit greater difference. This suggests that MAGNA is more powerful than GAT in distinguishing between important and unimportant nodes and assigning attention scores accordingly.

[0195] We propose a Multi-hop Attention Graph Neural Network (MAGNA), which combines the advantages of graph attention and diffusion techniques in a single layer through attention diffusion, layer normalization, and deep aggregation. MAGNA achieves context-dependent attention between any pair of nodes in the graph in a single layer, enhancing large-scale structural information and learning a more informative attention distribution. MAGNA improves upon all state-of-the-art methods in standard tasks such as node classification and knowledge graph refinement.

[0196] The foregoing description of exemplary embodiments of this disclosure is presented for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit this disclosure to the precise form disclosed. Many modifications and variations are possible in accordance with the foregoing teachings.

[0197] The embodiments were chosen and described to explain the principles of this disclosure and its practical application, thereby enabling others skilled in the art to utilize this disclosure and various embodiments, as well as various modifications suitable for the particular intended use. Alternative embodiments will become apparent to those skilled in the art to which this disclosure pertains without departing from the spirit and scope of this disclosure. Therefore, the scope of this disclosure is defined by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

Claims

1. A system comprising a computing device, the computing device including a processor and a storage device storing computer-executable code, wherein the computer-executable code is configured to: Provide an incomplete knowledge graph comprising multiple nodes and multiple edges, each of the multiple edges connecting two nodes, the nodes including customer nodes and product nodes, the edges indicating customer interest in products; The attention matrix of the incomplete knowledge graph is calculated based on the single-hop attention between any two nodes among the plurality of nodes connected by one of the plurality of edges; Calculate the multi-head diffusion attention of any two nodes among the plurality of nodes based on the attention matrix; The updated embedding of the incomplete knowledge graph is obtained using the multi-head diffuse attention; as well as The incomplete knowledge graph is updated based on the updated embedding to obtain an updated knowledge graph, which is used to update the edges or relationships linking products and customers in order to recommend products to one or more of the customers. The computer-executable code is configured to compute the attention matrix in the following manner: pass (Equation (1)) Calculate the side Attention score ,in and It refers to nodes i and j. , , and These are the trainable weights shared by the l-th layer of multi-head attention. This indicates the embedding of node i in the l-th layer. This indicates the embedding of node j in the l-th layer. It is a trainable relation embedding of the k-th relation type, where || denotes the concatenation of embedding vectors. It is the dimension of the node. It is the dimension of the edge; pass (Equation (2)) Obtain the attention score matrix ,in It is the knowledge graph mentioned above; and pass Calculate the attention matrix .

2. The system of claim 1, wherein the computer-executable code is configured to compute the multi-head diffusion attention in such a way as: pass (Equation (3)) Calculate the multi-hop attention matrix Where hop is a positive integer in the range of 2 to 20. It is the attention decay factor; and pass (Equation (4)) calculates the multi-head diffuse attention, where Denotes the parameters of equation (1), It is the input entity embedding of the l-th layer.

3. The system according to claim 2, wherein the following methods are used to... Make a rough estimate: set up , (Equation (5)), where , , ∈(0, 1]; and Will Defined as ,in, It is defined for rough estimation sequence, It's the number of jumps.

4. The system according to claim 3, wherein K is a positive integer in the range of 2 to 12, and l is a positive number in the range of 2 to 24.

5. The system of claim 4, wherein the computer-executable code is configured to obtain the updated embedding of the incomplete knowledge graph by sequentially performing a first-layer normalization and addition operation, a feedforward operation, and a second-layer normalization and addition operation on the multi-head diffuse attention.

6. The system of claim 5, wherein the feedforward is performed using a two-layer multilayer perceptron (MLP) network.

7. The system of claim 1, wherein the computer-executable code is further configured to, after obtaining the updated embedding,: compute a loss function based on the updated embedding and the labels of nodes and edges of the incomplete knowledge, and adjust parameters used to compute the attention matrix, compute the multi-head diffusion attention, and obtain the updated embedding.

8. The system of claim 7, wherein the computer-executable code is configured to: iteratively compute the attention matrix, compute multi-head attention diffusion, obtain the updated embedding, compute the loss function and adjust the parameters, and update the incomplete knowledge graph using the updated embedding obtained after the multiple iterations.

9. The system of claim 8, wherein the computer-executable code is configured to update the incomplete knowledge graph by: predicting new features or new edges of the plurality of nodes based on the updated embedding, and adding the new features or the new edges to the incomplete knowledge graph.

10. A method comprising: An incomplete knowledge graph is provided by a computing device, comprising multiple nodes and multiple edges, each of the multiple edges connecting two nodes, the nodes including customer nodes and product nodes, and the edges indicating customer interest in a product; The attention matrix of the incomplete knowledge graph is calculated by a computing device based on the single-hop attention between any two nodes among the plurality of nodes connected by one of the plurality of edges; The computing device calculates the multi-head diffusion attention of any two nodes among the plurality of nodes based on the attention matrix; The updated embedding of the incomplete knowledge graph is obtained by the computing device using the multi-head diffuse attention; as well as The computing device updates the incomplete knowledge graph based on the updated embedding to obtain an updated knowledge graph, which is used to update the edges or relationships linking products and customers to recommend products to one or more of the customers; The steps for calculating the attention matrix include: pass (Equation (1)) Calculate the side Attention score ,in and It refers to nodes i and j. , , and These are the trainable weights shared by the l-th layer of multi-head attention. This indicates the embedding of node i in the l-th layer. This indicates the embedding of node j in the l-th layer. It is a trainable relation embedding of the k-th relation type, where || denotes the concatenation of embedding vectors. It is the dimension of the node. It is the dimension of the edge; pass (Equation (2)) Obtain the attention score matrix ,in It is the knowledge graph mentioned above; and pass Calculate the attention matrix .

11. The method of claim 10, wherein the step of calculating the multi-head diffuse attention comprises: pass (Equation (3)) Calculate the multi-hop attention matrix Where hop is a positive integer in the range of 2 to 20. It is a power of the attention matrix with hop count equal to hop. It is an attention decay factor; as well as pass (Equation (4)) calculates the multi-head diffuse attention, where Denotes the parameters of equation (1), It is the input entity embedding of the l-th layer.

12. The method of claim 11, wherein the [method / treatment] is performed in the following manner: Make a rough estimate: set up , (Equation (5)), where , , (0, 1]; and Will Defined as ,in, It is defined for rough estimation sequence, It's the number of jumps.

13. The method of claim 12, wherein K is a positive integer in the range of 2 to 12, and l is a positive number in the range of 2 to 24.

14. The method of claim 13, wherein the computer-executable code is configured to obtain the updated embedding of the incomplete knowledge graph by sequentially performing a first-layer normalization and addition operation, a feedforward operation, and a second-layer normalization and addition operation on the multi-head diffuse attention.

15. A non-transitory computer-readable medium storing computer-executable code, wherein the computer-executable code is configured to, when executed at a processor of an active computing device: Provide an incomplete knowledge graph comprising multiple nodes and multiple edges, each of the multiple edges connecting two nodes, the nodes including customer nodes and product nodes, the edges indicating customer interest in products; The attention matrix of the incomplete knowledge graph is calculated based on the single-hop attention between any two nodes among the plurality of nodes connected by one of the plurality of edges; Calculate the multi-head diffusion attention of any two nodes among the plurality of nodes based on the attention matrix; The updated embedding of the incomplete knowledge graph is obtained using the multi-head diffuse attention; as well as The incomplete knowledge graph is updated based on the updated embedding to obtain an updated knowledge graph, which is used to update the edges or relationships linking products and customers in order to recommend products to one or more of the customers. The computer-executable code is configured to compute the attention matrix in the following manner: pass (Equation (1)) Calculate the side Attention score ,in and It refers to nodes i and j. , , and These are the trainable weights shared by the l-th layer of multi-head attention. This indicates the embedding of node i in the l-th layer. This indicates the embedding of node j in the l-th layer. It is a trainable relation embedding of the k-th relation type, where || denotes the concatenation of embedding vectors. It is the dimension of the node. It is the dimension of the edge; pass (Equation (2)) Obtain the attention score matrix ,in It is the knowledge graph mentioned above; and pass Calculate the attention matrix ;and The computer-executable code is configured to compute the multi-head diffusion attention in the following manner: pass (Equation (3)) Calculate the multi-hop attention matrix Where hop is a positive integer in the range of 2 to 20. It is an attention decay factor; as well as pass (Equation (4)) calculates the multi-head diffuse attention, where Denotes the parameters of equation (1), It is the input entity embedding of the l-th layer.

Citation Information

Patent Citations

  • A knowledge graph embedding method based on a diverse graph attention mechanism

    CN109902183A

  • High-order neighborhood hybrid network representation learning method and device

    CN110991483A