Method and apparatus for performing tasks and training graph transformer using graph transformer

Abstract

The present disclosure provides a computer-implemented method for executing a task. The method comprises the following steps: obtaining a plurality of context nodes of a query node from a graph; obtaining a plurality of sequences based on the plurality of context nodes, wherein each sequence of the plurality of sequences comprises a query node and a subset of the plurality of context nodes; performing multi-sequence attention on the plurality of sequences to obtain a multi-sequence attention feature map; obtaining a message passing feature graph of the query node based on the multi-sequence attention feature graph; and executing the task based on the messaging feature map of the query node.

Description

Technical Field

[0001] In summary, this disclosure relates to artificial intelligence, and more specifically, to performing tasks using graph transformers and training graph transformers. Background Technology

[0002] Graph-structured data has been widely used in many real-world applications, such as social networks, biological networks, reference networks, Internet of Things (IoT) networks, recommender systems, financial systems, and trading systems. Node classification is an exemplary task on graphs, and node analysis is another. Graph Neural Networks (GNNs) have been widely used to learn representations of graph-structured data. However, the expressive power of GNNs is limited by message passing in graphs, which naturally introduces inductive bias.

[0003] The success of transformers in natural language processing and computer vision has fueled enthusiasm for extending transformer models to graphs. Successful graph transformers (GTs) already exist, performing graph-level tasks on small-scale graphs (e.g., molecular graphs). However, existing GTs are only applicable to small-scale graphs. While full self-attention in small graphs is computationally efficient, the quadratic time and storage complexity of full self-attention in GTs prevents them from being extended to large graphs for practical applications. Summary of the Invention

[0004] To address the aforementioned issues of existing GNNs and GTs, this disclosure proposes a Multi-sequence attention graph transformer (MGT), which introduces a strategy to help GT solve scalability problems with strong expressiveness.

[0005] According to one embodiment, this disclosure provides a computer-implemented method for performing a task. The method includes: obtaining a plurality of context nodes of a query node from a graph; obtaining a plurality of sequences based on the plurality of context nodes, wherein each of the plurality of sequences includes the query node and a subset of the plurality of context nodes; performing multi-sequence attention on the plurality of sequences to obtain a multi-sequence attention feature map; obtaining a messaging feature map of the query node based on the multi-sequence attention feature map; and performing the task based on the messaging feature map of the query node.

[0006] According to one embodiment, this disclosure provides a computer-implemented method for training a neural network (NN) model to perform a task. The method includes: obtaining a plurality of context nodes of a query node from a graph; obtaining a plurality of sequences based on the plurality of context nodes, wherein each of the plurality of sequences includes the query node and a subset of the plurality of context nodes; performing multi-sequence attention on the plurality of sequences to obtain a multi-sequence attention feature map; obtaining a message passing feature map of the query node based on the multi-sequence attention feature map; performing a task-related prediction based on the message passing feature map of the query node; generating a first loss based on the prediction and the label of the query node; and updating the NN model based on the first loss.

[0007] According to one embodiment, this disclosure provides an apparatus for performing a task. The apparatus includes: a sequence acquisition module configured to: obtain a plurality of context nodes of a query node from a graph, and obtain a plurality of sequences based on the plurality of context nodes, wherein each of the plurality of sequences includes the query node and a subset of the plurality of context nodes; a graph transformer configured to: perform multi-sequence attention on the plurality of sequences to obtain a multi-sequence attention feature map, and obtain a message passing feature map of the plurality of sequences based on the multi-sequence attention feature map; and a task relevance predictor configured to: receive the message passing feature map of the query node obtained based on the message passing feature map of the plurality of sequences, and perform the task based on the message passing feature map of the query node.

[0008] According to one embodiment, this disclosure provides a computer system comprising: one or more processors; and one or more storage devices storing computer-executable instructions, which, when executed, cause the one or more processors to perform the operations of the methods described above and to perform the operations of methods according to various aspects of this disclosure.

[0009] According to one embodiment, this disclosure provides one or more computer-readable storage media storing computer-executable instructions that, when executed, cause one or more processors to perform the operations of the methods described above and to perform the operations of methods according to various aspects of this disclosure.

[0010] According to one embodiment, a computer program product is provided, comprising computer-executable instructions that, when executed, cause one or more processors to perform the operations of the methods described above and to perform the operations of methods according to various aspects of this disclosure.

[0011] According to the proposed scalable MGT framework, this disclosure presents the graph structure as multiple short sequences of nodes and performs multi-sequence attention on these short sequences to alleviate the quadratic complexity of self-attention mechanisms in existing graph transformers. Instead of generating long and redundant sequences, this disclosure forms a set of short, representative sequences as the context for each query node, by limiting the sequence length. The context size can be increased linearly by increasing the number of sequences, thus overcoming the intolerable quadratic complexity of self-attention mechanisms. Furthermore, this disclosure proposes a multi-sequence attention mechanism that can handle these sequences simultaneously. Specifically, multi-sequence attention is achieved through newly designed inter-sequence attention across multiple sequences and intra-sequence attention within each sequence. Unlike existing graph transformers that operate on a single sequence, the MGT framework interweaves attention within each sequence and across multiple sequences to extract node representations. By introducing inter-sequence attention, the MGT framework possesses multi-sequence attention in addition to intra-sequence attention. Other advantages and enhancements are explained in the following description. Attached Figure Description

[0012] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are used for illustration and not limitation.

[0013] Figure 1 This is a schematic block diagram illustrating an exemplary GT framework for performing tasks according to various aspects of this disclosure.

[0014] Figure 2 This is a schematic diagram illustrating a simplified description of the attention matrix of a single sequence of self-focused data according to various aspects of this disclosure.

[0015] Figure 3 This is a schematic block diagram illustrating an exemplary MGT framework for performing tasks according to various aspects of this disclosure.

[0016] Figures 4A to 4C This is a schematic diagram illustrating a simplified description of a self-focused attention matrix of multiple sequences according to various aspects of this disclosure.

[0017] Figure 5 and Figure 6 This is a schematic block diagram illustrating an exemplary MGT framework for performing tasks according to various aspects of this disclosure.

[0018] Figure 7 and Figure 8 An exemplary process for training a neural network model including an MGT, according to various aspects of this disclosure, is shown.

[0019] Figure 9 This is a schematic block diagram illustrating an exemplary apparatus for performing a task according to various aspects of the present disclosure.

[0020] Figure 10 This is a schematic block diagram illustrating exemplary methods for performing tasks according to various aspects of this disclosure.

[0021] Figure 11 This is a schematic block diagram illustrating an exemplary method for training a neural network model according to various aspects of this disclosure.

[0022] Figure 12 This is a schematic block diagram illustrating an exemplary computing system according to various aspects of this disclosure. Detailed Implementation

[0023] This disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement embodiments of this disclosure, and not to impose any limitation on the scope of this disclosure.

[0024] Various embodiments will be described in detail with reference to the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References to specific examples and embodiments are for illustrative purposes and are not intended to limit the scope of this disclosure.

[0025] This disclosure describes a method and system implemented as a computer program executing on one or more computers to perform a task using a graph transformer. For example, an neural network (NN) model including a graph transformer and a task-related predictor can perform a machine learning task of classifying nodes in a graph, which may represent, for example, social networks, biological networks, reference networks, transaction networks, recommender systems, financial systems, the Internet of Things (IoT), etc. For example, an NN model including a graph transformer and a task-related predictor can perform a machine learning task of performing industrial analysis on nodes in a graph. For example, industrial analysis may be the prediction of equipment lifespan, the prediction of equipment wear and tear, the prediction of device operation in an IoT, etc., where devices are represented as nodes in a graph that includes multiple nodes corresponding to multiple devices and edges between nodes. Aspects of this disclosure can be applied to these fields, such as social networks, biological networks, reference networks, recommender systems, financial systems, IoT, smart manufacturing, industrial analysis, etc.

[0026] Figure 1 This is a schematic block diagram illustrating an exemplary GT framework for performing tasks according to various aspects of this disclosure.

[0027] The graph can be input to the GT framework 100. This graph can be a dataset containing nodes and edges. Nodes in the graph can represent entities, and edges represent connections between nodes. For example, a social network is a graph in which users, or specifically user accounts, are nodes. An edge exists when two users are connected in some way. For example, if two users are friends, share a post, have similar interests, have similar profiles, etc., then these two users can have a connection represented by an edge.

[0028] In one example, the graph can be represented as ,in Let G be the adjacency matrix. The characteristic matrix of graph G is represented by the characteristic matrix of graph G. It is a picture The number of nodes, This refers to the dimension of node features. The adjacency matrix A can represent the connections between nodes in graph G, and the feature matrix X can represent the features of each node in the graph. A node's features can include multiple feature components, the number of which is defined as the dimension of the node features. For example, for a graph in a social network, the feature components of a node can include age, gender, hobbies, occupation, and various behaviors such as shopping, reading, and listening to music. It should be understood that aspects of this disclosure are not limited to specific values ​​of the elements of the adjacency matrix and the feature matrix.

[0029] like Figure 1 As shown, for the query node (It can be the input of the GT framework 100), and can be sampled from the graph by a set of nodes. The context C consists of this group of nodes. It can be called a query node The context nodes. In the example shown, context C includes nodes 1 to 12. Various node sampling methods can be used or combined to sample the context nodes. In one embodiment, Personalized PageRank (PPR) and its variants, such as Approximate PPR (Daniel A. Spielman and Shang Hua Teng. A local clustering algorithm for massive graphs and its application to nearly linear time graph partitioning. SIAM Computing Journal, 42(1):1-26, 2013), can be used to derive context nodes from the graph. For example, with the previous M (exist Figure 1 In the example M=12) The node with the highest PPR score can be exported as a context node. ,Right now = M In another embodiment, a random walk method can be used to derive context nodes from the graph. Compared to the PPR method, the random walk method tends to capture more distant neighbor nodes, and thus captures long-distance dependencies. In another embodiment, a global sampling method such as K-nearest neighbor nodes can be used to integrate nodes under different similarity measures. It should be understood that any suitable sampling method can be used to sample the context nodes of a query node from the graph.

[0030] A single sequence S can be formed using query nodes and context nodes. In the example shown, sequence S includes query node 0 and all context nodes 1 through 12. It should be understood that the length of sequence S (i.e., the number of nodes in sequence S) in a large-scale graph can be much larger than the length shown. Sequence S can be input into a graph transformer (GT)T, which performs self-attention on sequence S and outputs the message passing feature (MPF) of the sequence. It should be understood that the self-attention process is a known technique, and variations of self-attention methods can exist, and any suitable self-attention method or its variations can be used to perform self-attention on sequence S. A task-related predictor P can perform a task based on the message passing feature (MPF) of the sequence.

[0031] Figure 2 This is a schematic diagram illustrating a simplified description of the single-sequence approach according to various aspects of this disclosure. For example... Figure 2 As shown, with Figure 1 Taking the sequence S shown as an example, for a sequence S with 13 nodes, the feature matrix of sequence S can be 1. A 13 matrix, where the dimension of node features is omitted for simplicity; that is, the sequence S can be simply represented as a 13 matrix. 13 arrays. The self-focused attention matrix 210 of sequence S can be 13. A 13-matrix is ​​used, where labels Si (i = 1 to 13) represent the i-th node in the sequence. Then, the attention matrix 210 is processed to obtain an attention sequence AS with the same size as the input sequence S. Since the size of the attention matrix 210 is the square of the sequence size, the computational requirements for self-attention in sequence S (such as computation time, memory usage, etc.) will increase quadratically with the length of sequence S. Assume there are a total of [number missing] nodes in sequence S. If there are 1 sampling node, then the computational requirement or computational complexity can be This means that the computational requirements are proportional to the square of the number of nodes. Therefore, quadratic time and memory complexity can limit the scalability of the GT framework 100 for large graph applications.

[0032] Figure 3 This is a schematic block diagram illustrating an exemplary MGT framework for performing tasks according to various aspects of this disclosure.

[0033] For reference Figure 1 The graph and query nodes described For example, graph and query node This can be the input of the MGT framework 300. It can be derived from the graph from the query nodes. A related set of nodes The context C is sampled. Context nodes can be softly partitioned into... Different sequences Each sequence Include There are a total of nodes. Each sampling node This can be used to represent the effective context size. In one embodiment, the aforementioned PPR and its variants (such as approximate PPR) can be used to derive context nodes from a graph. For example, having a preceding... M ( M The node with the highest PPR score can be exported as a context node. ,Right now = M In another embodiment, the aforementioned random walk method or global sampling method (such as K nearest neighbor nodes) can be used to derive context nodes from the graph. It should be understood that any suitable sampling method can be used to sample the context nodes of a query node from the graph. Then, the context nodes can be used to derive the context nodes. China A length of The sequence is sampled. In one embodiment, a soft segmentation strategy can be used to balance the distribution of nodes in the graph. In particular, fewer but more influential neighbor nodes have a higher sampling probability, and each node appears once in the sequence but may appear in multiple sequences. It should be understood that any suitable sampling method or segmentation method can be used to construct... A sequence.

[0034] To enable query nodes Being able to notice all sampling nodes, in each sequence Add extra query nodes In one embodiment, each sequence can be... Additional join query node at the beginning Then, the sampling sequence can be represented as: ,in This represents the context node in the sequence. Figure 3 In the example shown, context C with 12 context nodes 1 to 12 can be sampled from the graph and divided into... Length The sequence. After adding query node 0 to each sequence, multiple ( Sequences S1, S2, and S3, each sequence having multiple nodes ( +1=5).

[0035] Multiple sequences S1, S2 and S3 ( This can be input into the MGT, which is a type of GT. The MGT can be included in... Figure 3 The Chinese representation is MGT L ( )of K Transformer layers L1 to L K Each MGT layer has a similar structure. In the first MGT layer L1, feature maps of multiple sequences S1, S2, and S3 are processed to obtain output feature maps of multiple sequences S1, S2, and S3. Then, the output feature maps of multiple sequences S1, S2, and S3 from layer L1 are used as input to the next MGT layer L2. This process is executed iteratively, with the last MGT layer L... K The output feature map is used as the output of the MGT. In one embodiment, the number of MGT layers... K The number can be greater than one. In another embodiment, the number of MGT layers... K It can be equal to one.

[0036] Length The node sequence with +1 can be represented as the query node. sequence set S ,Right now First, extract the input node feature map of the sequence set from the graph. It should be understood that the term feature map can also be called a feature matrix. Input node feature map. Linear projection can be used to map into the latent space, and positional encoding can be added. That is, the first MGT layer L1 needs to process The feature maps of the sequences are: In one embodiment, the positional encoding can be initialized to a pre-trained Node2vec (AdityaGrover and Jure Leskovec. node2vec: Scalable feature learning for networks). KDD (See page 855-864, 2016) Embedded throughout graph G. It should be understood that any existing or suitable position encoding can be used to implement position encoding.

[0037] like Figure 3 As shown, MGT L1 includes an In-Sequence Attention (IaSA) module, which is used for... In one embodiment, each sequence in the sequence of nodes undergoes intra-sequence attention. Intra-sequence attention for each sequence in a sequence of nodes can be achieved by performing self-attention within each node sequence. Any suitable self-attention method or its variants can be used to perform self-attention.

[0038] Figure 4A This is a schematic diagram illustrating a simplified description of the scheme of interest for the multi-sequence approach according to various aspects of this disclosure. For example... Figure 4A As shown, with Figure 3 Taking sequences S1 to S3 as an example, for sequences S1 to S3 that each have 5 nodes, the self-focused attention matrices 411 to 413 of sequences S1 to S3 can be 5 A matrix of 5 is generated, where the labels RiCj (i=1 to 3, j=1 to 5) represent the j-th node in the i-th sequence. Then, attention matrices 411 to 413 are processed to obtain attention sequences AS1 to AS3 with the same size as the input sequences S1 to S3. Figure 2 Compared to the attention matrix 210 shown, 15 5. Focus on the non-zero elements in matrix 410, or the non-zero submatrices 411 to 413, which are limited to the diagonal blocks. Therefore, The computational requirements (such as computation time, memory usage, etc.) for the self-focus of a sequence can be: ,approximate This means that computational requirements are related to the context size. Proportional. In practical applications, for example, by adjusting the sequence length... Limit to a small value (e.g., =32), the context size will be related to the number of sequences. Proportional. Then, the sequence. The computational cost of self-attention for all sequences increases only with the number of sequences. Linear growth. This allows for a significant reduction in the reference size as the context size increases. Figure 2 The burden of quadratic complexity is discussed.

[0039] Return to reference Figure 3 Using MGT L1 as MGT L iAs an example, MGT L1 includes an Inter-Sequence Attention (IrSA) module, which is used for cross-sequence attention... Each sequence performs inter-sequence attention. It is observed that the sequence set or matrix... S The i The line represents the first i Sequences ,matrix S Each column contains a sampling node for all sequences. Based on this observation, the matrix... S The columns can be viewed as special sequences, and the IrSA module can perform self-focus on each column, where nodes... It can act as an anchor point, merging information from other sequences into the sequence. Therefore, in one embodiment, inter-sequence attention across multiple sequences can be achieved by... The sequence is used as This is achieved by performing self-attention on each column of the node matrix composed of rows. Referring back to the above embodiment, the feature map of the node matrix to be processed by the IaSA and IrSA modules of the first MGT layer L1 is... .

[0040] In each layer L of the MGT, the IaSA module of MGT L processes the input feature map. Each line performs self-attention; MGT L's IrSA module processes the input feature map. Self-attention is performed on each column, where the input feature map is omitted for simplicity. The superscript of . Self-following of rows and self-following of columns correspond to intra-sequence follow-ups and inter-sequence follow-ups, respectively, and are performed as shown in equation (1). (1) in, It is the self-awareness module, in which , and It consists of a query matrix, a key matrix, and a value matrix, which are learnable weighted matrices. and These are representations or feature maps that contain intra-sequence information and inter-sequence information, respectively.

[0041] Return to reference Figure 4A In the example shown, the feature map yes Figure 4A The feature maps of the attention sequences AS1 to AS3 shown are shown.

[0042] Figure 4B This is a schematic diagram illustrating a simplified description of the scheme of interest for the multi-sequence approach according to various aspects of this disclosure. For example... Figure 4B As shown, with Figure 3Taking the sequences S1 to S3 shown as an example, for a sequence set or matrix with sequences S1 to S3 as rows... S Matrix nodes S The columns are considered as special sequences SS1 to SS5. The self-focused attention matrices 421 to 425 of special sequences SS1 to SS5 can be 3... A matrix of 3, where the labels CjRi (i = 1 to 3, j = 1 to 5) represent the i-th node in the j-th special sequence, which is also Figure 4A The j-th node in the i-th sequence shown. Then, attention matrices 421 to 425 are processed to obtain attention-specific sequences ASS1 to ASS5 with the same size as the input special sequences SS1 to SS5.

[0043] like Figure 4C As shown, matrices 421 to 425 are mapped from matrix 420 to matrix 410. (And...) Figure 2 Compared to the attention matrix 210 shown, the non-zero elements or non-zero submatrices 421 to 425 are scattered in 15 5. Focus on matrix 410. +1) The computational requirements (such as computation time, memory usage, etc.) for the self-focus of a special sequence can be ,approximate This means that computational requirements are related to the context size. Proportional. For example Figure 4C As shown, attention matrices 421 to 425 are displayed together with attention matrices 411 to 413 in matrix 410, as shown in the figure. Figure 4A Compared to the case of only performing intra-sequence attention, inter-sequence attention increases the interaction between sequences and reduces the sparsity of the attention matrix 410. Therefore, by... Figure 3 The embodiments employ multi-sequence attention, which includes intra-sequence attention across multiple sequences and inter-sequence attention across multiple sequences. MGT can interweave attention across multiple sequences to facilitate message exchange among all sampling nodes. Inter-sequence attention also incurs additional computational cost. The overall time and memory complexity of MGT is... ,Right now It is still very efficient compared to using a single sequence, because of the small... and In context This generates a large number of samples. In particular, when the context size is large, the time and memory complexity increases. much smaller Figure 1 The time and memory complexity of the single-sequence focus method shown This allows MGT to be more easily extended to large graphs for practical applications.

[0044] Return to reference Figure 3 MGT L (such as MGT L1) includes a gating module G. In one embodiment, the gating module G can implement node-by-node gating to adaptively combine feature maps of interest within a sequence. Inter-sequence attention feature maps The gating module G packs two feature maps in the node feature dimension and projects them linearly to obtain the gating weights, as shown in equation (2): (2) in, , It is a learnable matrix.

[0045] Gating weights output by gating module G Through such Figure 3 The dot product method shown is applied to inter-sequence attention feature maps. To obtain attention feature maps between gated sequences Then With Intra-Sequence Interest Feature Map Add them together to obtain a multi-sequence attention feature map. It was observed that the intra-sequence attention feature map It plays a more important role in prediction, while the focus is on feature maps between sequences. This plays a supporting role. Therefore, in the illustrated embodiment, gating is only applied to the inter-sequence interest feature maps. This adaptively determines its contribution. Then, the feedforward network (FFN) sublayer... It is applied to the multi-sequence attention feature map, as shown in equation (3). (3)

[0046] Output feature map This can be referred to as the message passing feature map output by the MGT layer L. The message passing feature map output by the first MGT layer L1. It can be used as the input feature map for the next MGT layer L2, and similarly, by the first i MGT layer L i Output message passing feature map It can be used as the next MGT layer L i+1 The input feature map, where i Less than K , by the K MGT layer L K Output information transmission feature map It can be the final message passing feature map of MGT The final message passing feature map of MGT Including with The corresponding sequences Okay, and it can be seen from the final message passing feature map To obtain the query node Different representations ( In the example shown, the query node is placed at the first position of each sequence, from the message passing feature map. Retrieve the first item of each sequence to obtain the query node. Different representations ( It can be called the message passing feature graph of the query node, for example. Figure 3 As shown .

[0047] Then, the task-related predictor P is based on the message passing feature map of each query node. Inferring the outcome of the task Then on One prediction … Integrate the results to obtain the predicted outcomes for the task. For example, for... One prediction … Averaging is performed to obtain the prediction result for the task. In one embodiment, the predictor can be implemented by a multilayer perceptron (MLP) module, and the task can be node classification. In one embodiment, the predictor can perform a machine learning task of performing industrial analysis on nodes in a graph. For example, industrial analysis can be the prediction of equipment lifespan, the prediction of equipment wear and tear, the prediction of equipment operation, etc., where equipment is represented as nodes in a graph that includes multiple nodes corresponding to multiple devices and edges between nodes. Aspects of this disclosure can be applied to fields such as social networks, biological networks, reference networks, IoT networks, recommender systems, financial systems, smart manufacturing, industrial analysis, etc.

[0048] Figure 5 This is a schematic block diagram illustrating an exemplary MGT framework for performing tasks according to various aspects of this disclosure.

[0049] Figure 5 The same label in Figure 3 The same or corresponding elements are shown. The difference between MGT framework 500 and MGT framework 300 is that MGT framework 500 does not include the gating module G of MGT framework 500. In other words, it focuses on the inter-sequence feature maps. With Intra-Sequence Interest Feature Map Add them together to obtain a multi-sequence attention feature map. The other parts of the MGT framework are related to Figure 3 The MGT framework is the same, so it will not be described in detail again.

[0050] Figure 6 This is a schematic block diagram illustrating an exemplary MGT framework for performing tasks according to various aspects of this disclosure.

[0051] Figure 6 The same label in Figure 3 and Figure 5 The same or corresponding elements are shown. The MGT framework 600 includes an MSA module that obtains message passing feature maps for multiple sequences (e.g., S1 to S3). The MSA module performs multi-sequence attention on the multiple sequences to obtain a multi-sequence attention feature map, and obtains a message passing feature map for the multiple sequences based on the multi-sequence attention feature map. In one embodiment, the MSA module may perform intra-sequence attention only on each of the multiple sequences, without performing inter-sequence attention across the multiple sequences. Intra-sequence attention feature map This can be used as a multi-sequence attention feature map, which can be further processed by an MSA module (e.g., the FFN module of MSA) to obtain message passing feature maps for multiple sequences. In one embodiment, the MSA module can use... Figure 3 The MSA module can be implemented using the structure of the MGT framework 300, specifically including the IaSA module, IrSA module, gating module G, and FFN module. In one embodiment, the MSA module can use... Figure 5 The MSA module can be implemented using the structure described above, meaning it can include the IaSA module, IrSA module, and FFN module of the MGT framework 500. It should be understood that the MSA module of the MGT framework 600 can be implemented using other suitable structures from other embodiments.

[0052] Figure 7 An exemplary process for training a neural network model including an MGT, according to various aspects of this disclosure, is shown.

[0053] Figure 7 The same label in Figure 3 and Figure 5-6 The same or corresponding elements are shown. NN model 700 can be one of MGT frameworks 300, 500, and 600. See reference... Figure 3 , Figure 5-6 As described, the task-related predictor P is based on the message passing feature map of each query node. Inferring the outcome of the task Then on One prediction … The results are integrated to obtain the predictions for the task. During the training phase of NN model 700, One prediction … It can be used to determine the supervision loss, as shown in equation (4): (4) in, Indicates to The prediction This represents the label of the query node in the training dataset.

[0054] Then, Figure 7 The middle is shown as L1 The supervised loss is used to update the NN model 700. It is understandable that supervised loss is a common loss used to train NN models based on a training dataset. For example, the training dataset may include a graph, query nodes, and annotated labels for the query nodes. The known AdamW optimizer can be used to update the NN model 700 based on this loss.

[0055] Figure 8 An exemplary process for training a neural network model including an MGT, according to various aspects of this disclosure, is shown.

[0056] Figure 8 The same label in Figure 3 and Figure 5-7 The same or corresponding elements are shown. NN model 800 can be one of MGT frameworks 300, 500, and 600. See reference... Figure 3 , Figure 5-6 The description indicates alignment. All sequences originate from the same query node. Context They are derived from [the source]. Therefore, they [are derived from / from / the source]. A representation or feature map ( All of them partially describe the context. The semantic and structural characteristics are expected to be more similar in the semantic feature space, leading to similar downstream predictions. Based on this observation, we can encourage... Each representation ( Consistency between representations is achieved to eliminate the influence of randomness in node sampling. In particular, alignment of representations in the representation space can be enforced by optimizing the objective based on alignment loss, as shown in equation (5): (5) in, yes Each representation ( On average, Similarity measures such as cosine similarity, for example, similarity values ​​are located in [-1, 1].

[0057] Since the capacity of the transformer is larger than that of the GNN, it will increase the training difficulty. In this embodiment, the alignment loss can alleviate the overfitting phenomenon during training, improve the generalization ability, and thus improve efficiency.

[0058] Then, Figure 8 or Figure 7 The middle is shown as L1 Supervision losses and Figure 8 The middle is shown as L2 The alignment loss is used to update the NN model 700. The known AdamW optimizer can be used to update the NN model 800 based on the loss. The two losses can be fused using coefficients, as shown in equation (6): (6)

[0059] Figure 9 This is a schematic block diagram illustrating an exemplary apparatus for performing a task according to various aspects of the present disclosure.

[0060] Apparatus 900 includes a sequence acquisition module 910, a graph transformer 920, and a task-related predictor 930. The sequence acquisition module 910 is configured to acquire multiple context nodes of a query node from a graph, and to acquire multiple sequences based on the multiple context nodes. Each of the multiple sequences includes a query node and a subset of the multiple context nodes. The graph transformer 920 is configured to perform multi-sequence attention on the multiple sequences to obtain a multi-sequence attention feature map, and to obtain a message passing feature map of the multiple sequences based on the multi-sequence attention feature map. The task-related predictor 930 is configured to receive the message passing feature map of the query node obtained based on the message passing feature map of the multiple sequences, and to perform a task based on the message passing feature map of the query node.

[0061] In one embodiment, the graph transformer 920 may include an intra-sequence attention module and an inter-sequence attention module. The intra-sequence attention module is configured to perform intra-sequence attention on each of a plurality of sequences to obtain an intra-sequence attention feature map for the plurality of sequences. The inter-sequence attention module is configured to perform inter-sequence attention across the plurality of sequences to obtain an inter-sequence attention feature map for the plurality of sequences. The graph transformer 920 is configured to obtain a multi-sequence attention feature map based on the intra-sequence attention feature map and the inter-sequence attention feature map.

[0062] In one embodiment, the intra-sequence attention module is configured to perform intra-sequence attention by performing self-attention on nodes in each of a plurality of sequences. The inter-sequence attention module is configured to perform inter-sequence attention by performing self-attention on nodes in each column of a matrix consisting of a plurality of sequences, wherein each of the plurality of sequences is a row of the matrix, and nodes at the same position in the plurality of sequences constitute a column of the matrix.

[0063] In one embodiment, the graph transformer 920 may include a gating module. The gating module is configured to perform gating on the inter-sequence interest feature map to obtain a gated inter-sequence interest feature map. The graph transformer 920 is configured to obtain a multi-sequence interest feature map based on the intra-sequence interest feature map and the gated inter-sequence interest feature map. In one embodiment, the gating is a node-wise gating performed using a gating function configured to obtain gating weights for each node of the multiple sequences based on the intra-sequence interest feature map and the inter-sequence interest feature map.

[0064] In one embodiment, the apparatus 900 includes an extraction module configured to extract a message passing feature map of a query node based on message passing feature maps of multiple sequences. In one embodiment, the query node is placed at a fixed position in each of the multiple sequences, and the extraction module extracts the message passing feature map of the query node based on the fixed position. In one embodiment, the fixed position is a first position in each of the multiple sequences.

[0065] In one embodiment, the sequence acquisition module 910 is configured to obtain multiple context nodes based on a query node by using one of the Personalized Page Rank (PPR) method, a random walk method, and a K-nearest neighbor method. In another embodiment, the sequence acquisition module 910 is configured to obtain multiple sequences by softly distributing the multiple context nodes into multiple sequences, wherein a single node may appear only once in a sequence but may appear in more than one sequence.

[0066] In one embodiment, the graph transformer 920 includes K transformer layers, each iteratively performing: performing multi-sequence attention on multiple sequences, and obtaining message passing feature maps of the multiple sequences, wherein the output of one transformer layer is used as the input of the next transformer layer, and K is greater than one or K is one. The graph transformer 920 is configured to obtain a message passing feature map of a query node based on the message passing feature map of the multiple sequences output by the last of the K transformer layers.

[0067] In one embodiment, graph transformer 920 is configured to perform multi-sequence attention based on input feature maps of multiple sequences, wherein the input feature map of the first transformer layer in the K transformer layers is obtained based on graph features of nodes in the multiple sequences and positional encoding features of nodes in the multiple sequences, wherein the input feature map of the kth transformer layer in the K transformer layers is a message passing feature map of multiple sequences output by the (k-1)th transformer layer in the K transformer layers, where k is greater than one.

[0068] In one embodiment, the task-related predictor 930 is configured to perform predictions for classification or analysis on the query node. It should be understood that the task can be any industrial task that can be processed based on graph-structured data, such as the example task described above and other suitable tasks.

[0069] Figure 10 This is a schematic block diagram illustrating an exemplary computer implementation of a method for performing a task according to various aspects of this disclosure.

[0070] At step 1010, multiple context nodes of the query node can be obtained from the graph.

[0071] At step 1020, multiple sequences can be obtained based on multiple context nodes, wherein each of the multiple sequences includes a query node and a subset of the multiple context nodes. In one embodiment, the query node is placed at a fixed position in each of the multiple sequences, for example, the fixed position is the first position of each of the multiple sequences.

[0072] At step 1030, multi-sequence attention can be performed on multiple sequences to obtain a multi-sequence attention feature map.

[0073] At step 1040, the message passing feature map of the query node can be obtained based on the multi-sequence attention feature map.

[0074] At step 1050, the task can be performed based on the message passing feature map of the query node.

[0075] In one embodiment, at step 1030, intra-sequence attention can be performed on each of the multiple sequences to obtain intra-sequence attention feature maps of the multiple sequences, inter-sequence attention can be performed across the multiple sequences to obtain inter-sequence attention feature maps of the multiple sequences, and a multi-sequence attention feature map can be obtained based on the intra-sequence attention feature maps and the inter-sequence attention feature maps.

[0076] In one embodiment, at step 1030, intra-sequence attention of each of the multiple sequences can be performed by performing self-attention on nodes in each of the multiple sequences, and inter-sequence attention across the multiple sequences can be performed by performing self-attention on nodes in each column of a matrix composed of the multiple sequences, wherein each of the multiple sequences is a row of the matrix and nodes at the same position in the multiple sequences are a column of the matrix.

[0077] In one embodiment, at step 1030, gating of the inter-sequence interest feature map can be performed to obtain a gated inter-sequence interest feature map, and a multi-sequence interest feature map can be obtained based on the intra-sequence interest feature map and the gated inter-sequence interest feature map. In one embodiment, the gating is a node-by-node gating performed using a gating function configured to obtain the gating weights of each node of multiple sequences based on the intra-sequence interest feature map and the inter-sequence interest feature map.

[0078] In one embodiment, at step 1040, message passing feature maps of multiple sequences can be obtained based on multi-sequence attention feature maps, and message passing feature maps of query nodes can be obtained based on message passing feature maps of multiple sequences.

[0079] In one embodiment, at step 1010, multiple context nodes of the query node can be obtained from the graph based on the query node using one of the Personalized Page Rank (PPR) method, the random walk method, and the K nearest neighbor method.

[0080] In one embodiment, at step 1020, multiple sequences can be obtained by softly distributing multiple context nodes into multiple sequences, wherein a single node appears only once in a sequence but may appear in more than one sequence.

[0081] In one embodiment, method 1000 can be performed using a neural network (NN) model that includes a graph transformer and a task-related predictor.

[0082] In one embodiment, the graph transformer includes K transformer layers, each layer iteratively performing: performing multi-sequence attention on multiple sequences, and obtaining message passing feature maps of the multiple sequences, wherein the output of one transformer layer is used as the input of the next transformer layer, and K is greater than one or K is one. At step 1040, the message passing feature map of the query node can be obtained based on the message passing feature map of the multiple sequences output by the last transformer layer of the K transformer layers.

[0083] In one embodiment, at step 1030, multi-sequence attention of multiple sequences can be performed based on the input feature maps of multiple sequences. The input feature map of the first transformer layer in K transformer layers can be obtained based on the graph features of the nodes of multiple sequences and the position encoding features of the nodes of multiple sequences. The input feature map of the kth transformer layer in K transformer layers is the message passing feature map of multiple sequences output by the (k-1)th transformer layer in K transformer layers, where K is greater than one.

[0084] In one embodiment, at step 1050, a prediction of the classification or analysis of the query node can be performed as a task. In one embodiment, the graph represents one of a social network, a referral network, a transaction network, a financial network, and an IoT network.

[0085] Figure 11 This is a schematic block diagram illustrating an exemplary computer implementation of a method for training a neural network model according to various aspects of this disclosure.

[0086] At step 1110, multiple context nodes of the query node can be obtained from the graph.

[0087] At step 1120, multiple sequences can be obtained based on multiple context nodes, wherein each of the multiple sequences includes a query node and a subset of multiple context nodes.

[0088] At step 1130, multi-sequence attention can be performed on multiple sequences to obtain a multi-sequence attention feature map.

[0089] At step 1140, the message passing feature map of the query node can be obtained based on the multi-sequence attention feature map.

[0090] At step 1150, task-related predictions can be performed based on the message passing feature map of the query node.

[0091] At step 1160, a first loss can be generated based on the labels of the predicted and queried nodes.

[0092] At step 1170, the NN model can be updated based on the first loss.

[0093] In one embodiment, at step 1160, a second loss can be generated based on the message passing feature map of the query node. At step 1170, the NN model can be updated based on the first and second losses. The first loss can be... Figure 7 and Figure 8 The monitoring loss shown, the second loss can be Figure 8 The alignment loss is shown.

[0094] Figure 12 This is a schematic block diagram illustrating an exemplary computing system according to various aspects of the present disclosure. The computing system 1200 may include at least one processor 1210. The computing system 1200 may also include at least one storage device 1220. The storage device 1220 may store computer-executable instructions, which, when executed, cause the processor 1210 to perform actions according to... Figure 1-11 Any operation of the embodiments described in this disclosure.

[0095] Embodiments of this disclosure may be embodied in a computer-readable medium, such as a non-transitory computer-readable medium. A non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform actions according to the following combination... Figure 1-11 Any operation of the embodiments described in this disclosure.

[0096] Embodiments of this disclosure can be embodied in a computer program product including computer-executable instructions that, when executed, cause one or more processors to perform actions according to the following combination: Figure 1-11 Any operation of the embodiments described in this disclosure.

[0097] It should be understood that all operations in the above methods are merely exemplary, and this disclosure is not limited to any operation in these methods or the order of such operations, and should cover all other equivalents under the same or similar concepts.

[0098] It should also be understood that all modules in the aforementioned apparatus can be implemented using various methods. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules can be further functionally divided into sub-modules or combined together.

[0099] The foregoing description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to limit them to the aspects shown herein. All structural and functional equivalents of elements throughout the various aspects described herein that are known or will become apparent later to those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims.

Claims

1. A computer-implemented method for performing a task, comprising: Obtain multiple context nodes of the query node from the graph; Based on the plurality of context nodes, a plurality of sequences are obtained, wherein each of the plurality of sequences includes the query node and a subset of the plurality of context nodes; Perform multi-sequence attention on the multiple sequences to obtain a multi-sequence attention feature map; Based on the multi-sequence attention feature map, the message passing feature map of the query node is obtained; and The task is executed based on the message passing feature map of the query node.

2. The method according to claim 1, wherein, The multi-sequence attention process for the plurality of sequences includes: Intra-sequence attention is performed on each of the plurality of sequences to obtain an intra-sequence attention feature map of the plurality of sequences; Perform inter-sequence attention across the multiple sequences to obtain an inter-sequence attention feature map of the multiple sequences; The multi-sequence attention feature map is obtained based on the intra-sequence attention feature map and the inter-sequence attention feature map.

3. The method according to claim 2, wherein, The step of performing intra-sequence attention on each of the plurality of sequences includes: performing self-attention on nodes in each of the plurality of sequences; The step of performing inter-sequence attention across the plurality of sequences includes performing self-attention on nodes in each column of a matrix composed of the plurality of sequences, wherein each of the plurality of sequences is a row of the matrix, and nodes at the same position in the plurality of sequences are columns of the matrix.

4. The method according to claim 2, wherein, The process of obtaining the multi-sequence attention feature map includes: Gating is performed on the inter-sequence interest feature map to obtain a gated inter-sequence interest feature map; The multi-sequence attention feature map is obtained based on the intra-sequence attention feature map and the inter-gated sequence attention feature map.

5. The method according to claim 4, wherein, The gating is a node-by-node gating performed using a gating function, which is configured to obtain the gating weights of each node in the plurality of sequences based on the intra-sequence attention feature map and the inter-sequence attention feature map.

6. The method according to claim 1, wherein, The process of obtaining the message passing feature map of the queried node includes: Based on the multi-sequence attention feature map, the message passing feature map of the multiple sequences is obtained; Based on the message passing feature maps of the multiple sequences, the message passing feature map of the query node is obtained.

7. The method according to claim 1, wherein, The multiple context nodes obtained from the graph for the query node include: obtaining the multiple context nodes based on the query node by using one of the Personalized PageRank (PPR) method, the random walk method, and the K nearest neighbor method.

8. The method according to claim 1, wherein, The query node is placed at a fixed position in each of the plurality of sequences.

9. The method according to claim 1, wherein, The method is implemented using a neural network (NN) model that includes a graph transformer and a task-related predictor.

10. The method according to claim 6, wherein, The graph transformer comprises K transformer layers, each layer iteratively performing multi-sequence attention on the plurality of sequences and obtaining message passing feature maps of the plurality of sequences, wherein the output of one transformer layer is used as the input of the next transformer layer, and wherein K is greater than one or K is one. The step of obtaining the message passing feature map of the query node based on the message passing feature maps of the multiple sequences includes: obtaining the message passing feature map of the query node based on the message passing feature maps of the multiple sequences output by the last transformer layer among the K transformer layers.

11. The method according to claim 10, wherein, The multi-sequence attention process for the plurality of sequences includes: performing the multi-sequence attention based on the input feature maps of the plurality of sequences, wherein the input feature map of the first transformer layer in the K transformer layers is obtained based on the graph features of the nodes of the plurality of sequences and the position encoding features of the nodes of the plurality of sequences, wherein the input feature map of the kth transformer layer in the K transformer layers is the message passing feature map of the plurality of sequences output by the (k-1)th transformer layer in the K transformer layers, wherein k is greater than one.

12. The method according to claim 1, wherein, The execution of the task includes: performing classification or analysis predictions on the query node.

13. The method according to claim 1, wherein, The diagram represents one of the following: social networks, referral networks, transaction networks, financial networks, and Internet of Things (IoT) networks.

14. A computer-implemented method for training a neural network model to perform a task, comprising: Obtain multiple context nodes of the query node from the graph; Multiple sequences are obtained based on the multiple context nodes, wherein each of the multiple sequences includes the query node and a subset of the multiple context nodes; Perform multi-sequence attention on the multiple sequences to obtain a multi-sequence attention feature map; Based on the multi-sequence attention feature map, the message passing feature map of the query node is obtained; Based on the message passing feature map of the query node, perform predictions related to the task; Based on the prediction and the label of the query node, a first loss is generated; and The NN model is updated based on the first loss.

15. The method of claim 14, further comprising: A second loss is generated based on the message passing feature map of the query node; as well as The NN model is updated based on the first loss and the second loss.

16. An apparatus for performing a task, comprising: A sequence acquisition module is configured to: obtain multiple context nodes of a query node from a graph, and obtain multiple sequences based on the multiple context nodes, wherein each of the multiple sequences includes the query node and a subset of the multiple context nodes; A graph transformer configured to: perform multi-sequence attention on the plurality of sequences to obtain a multi-sequence attention feature map, and obtain a message passing feature map of the plurality of sequences based on the multi-sequence attention feature map; A task-related predictor configured to: receive a message passing feature map of a query node obtained based on message passing feature maps of the plurality of sequences, and execute the task based on the message passing feature map of the query node.

17. A computer system, comprising: One or more processors; as well as One or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform the operation of the method according to any one of claims 1-15.

18. One or more computer-readable storage media storing computer-executable instructions, which, when executed, cause one or more processors to perform the operation of the method according to any one of claims 1-15.

Images