Invariant graph transformer

Abstract

A computer generates, using one or more first layers of a machine learning model, a rationale graph embedding corresponding to a rationale graph and a first environment graph embedding corresponding to a first environment graph on a per node basis using a graph. The computer generates a second environment graph embedding from a second environment graph. The computer concatenates the rationale graph embedding and the first environment graph embedding to form a first combination input and concatenates the rationale graph embedding and the second environment graph embedding to form a second combination input. The computer generates, using a second layer of the machine learning model, a first classification output using the first combination input and a second classification output using the second combination input. The computer adjusts parameters of the first layers and the second layer based on a comparison of the first classification output and the second classification output.

Description

PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 INVARIANT GRAPH TRANSFORMER CROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 519,757, filed August 15, 2023, which is herein incorporated by reference in its entirety for all purposes. BACKGROUND

[0002] Rationale discovery is defined as finding a subset of input data, which maximally support the prediction of the downstream tasks. In the graph machine learning context, graph rationale is defined as locating a critical subgraph in the given graph topology. In contrast to the rationale subgraph, the remaining subgraph is named an environment subgraph.

[0003] Graph rationalization can enhance model performance since a mapping between the graph rationale and a prediction label is viewed as invariant. To ensure the discriminative power of the extracted rationale subgraphs, a technique named intervention can be utilized. During intervention, given changing environment subgraphs, the semantics from the rationale subgraph is invariant, which guarantees the correct prediction result. However, most of the existing graph rationalization methods develop their intervention strategies on the graph level, which is coarse- grained.

[0004] Embodiments of the disclosure address this problem and other problems individually and collectively. SUMMARY

[0005] One embodiment is related to a method of training. A computer system can generate using one or more first layers of a machine learning model, a rationale graph embedding corresponding to a rationale graph and a first environment graph embedding corresponding to a first environment graph on a per node basis using aPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 graph. The computer system can generate a second environment graph embedding from a second environment graph. The second environment graph embedding is generated on a per node basis. The computer system can concatenate the rationale graph embedding and the first environment graph embedding to form a first combination input. The computer system can concatenate the rationale graph embedding and the second environment graph embedding to form a second combination input. The computer system can generate, using a second layer of the machine learning model, a first classification output using the first combination input. The computer system can generate, using the second layer of the machine learning model, a second classification output using second combination input. The computer system can adjust parameters of the one or more first layers and the second layer based on a comparison of the first classification output and the second classification output. The machine learning model includes a transformer sub-model that generates an attention data structure. The parameters of the one or more first layers and the second layer are further adjusted to maximize a self attention of the rationale graph embedding in the attention data structure and minimize interactions between the rationale graph embedding and the first environment graph embedding and the second environment graph embedding in the attention data structure.

[0006] Another embodiment is related to a production method. A computer system can generate, using one or more first layers of a machine learning model, a rationale graph embedding corresponding to a rationale graph and an environment graph embedding corresponding to an environment graph on a per node basis using a graph. The computer system can concatenate the rationale graph embedding and the environment graph embedding to form a combination input. The computer system can generate, using a second layer of the machine learning model, a classification output using the combination input. The machine learning model includes a transformer sub- model that generates an attention data structure. The parameters of the one or more first layers and the second layer were trained (1) to maximize a self-attention of the rationale graph embedding and (2) to minimize interactions between the rationale graph embedding and the environment graph.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0007] Another embodiment is related to a computer system comprising: a processor; and a computer-readable medium coupled to the processor. The computer- readable medium comprising code executable by the processor for implementing a method comprising: generating, using one or more first layers of a machine learning model, a rationale graph embedding corresponding to a rationale graph and a first environment graph embedding corresponding to a first environment graph on a per node basis using a graph. The method includes generating a second environment graph embedding from a second environment graph. The second environment graph embedding is generated on a per node basis. The method includes concatenating the rationale graph embedding and the first environment graph embedding to form a first combination input. The method includes concatenating the rationale graph embedding and the second environment graph embedding to form a second combination input. The method includes generating, using a second layer of the machine learning model, a first classification output using the first combination input. The method includes generating, using the second layer of the machine learning model, a second classification output using second combination input. The method includes adjusting parameters of the one or more first layers and the second layer based on a comparison of the first classification output and the second classification output, wherein the machine learning model includes a transformer sub-model that generates an attention data structure. The parameters of the one or more first layers and the second layer are further adjusted to maximize a self attention of the rationale graph embedding in the attention data structure and minimize interactions between the rationale graph embedding and the first environment graph embedding and the second environment graph embedding in the attention data structure.

[0008] These and other embodiments of the disclosure are described in detail below. For example, other embodiments are directed to systems, devices, and computer readable media associated with methods described herein.

[0009] Further details regarding embodiments of the disclosure can be found in the Detailed Description and the Figures.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG.1 shows a block diagram of a machine learning system according to embodiments.

[0011] FIG.2 shows a block diagram of components of an analysis computer according to embodiments.

[0012] FIG.3A shows a diagram of a node-level fine-grained graph rationalization method according to embodiments.

[0013] FIG.3B shows a diagram of an alternate node-level fine-grained graph rationalization method according to embodiments.

[0014] FIG.4 shows a diagram of a virtual node-level fine-grained graph rationalization method according to embodiments.

[0015] FIG.5 shows a flowchart of a fine-grained graph rationalization method of training according to embodiments.

[0016] FIG.6 shows a flowchart of a fine-grained graph rationalization method of production according to embodiments.

[0017] FIG.7 shows a heatmap of an adjacency matrix of an intervener according to embodiments.

[0018] FIG.8 shows graphs illustrating training stability of machine learning models according to embodiments.

[0019] FIG.9 shows graphs illustrating performance of embodiments according to embodiments.

[0020] FIG.10 shows a diagram of components of a computer system according to embodiments.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 TERMS

[0021] Prior to discussing embodiments of the disclosure, some terms can be described in further detail.

[0022] The term "artificial intelligence model" or "Al model" can include a model that may be used to predict outcomes in order achieve a pre-defined goal. The Al model may be developed using a learning algorithm, in which training data is classified based on known or inferred patterns. An Al model may also be referred to as a "machine learning model" or "predictive model."

[0023] "Machine learning" can include an artificial intelligence process in which software applications may be trained to make accurate predictions through learning. The predictions can be generated by applying input data to a predictive model formed from performing statistical analyses on aggregated data. A model can be trained using training data, such that the model may be used to make accurate predictions. The prediction can be, for example, a classification of an image (e.g., identifying images of cats on the Internet) or as another example, a recommendation (e.g., a movie that a user may like or a restaurant that a consumer might enjoy).

[0024] In some embodiments, a model may be a statistical model, which can be used to predict unknown information from known information. For example, a learning module may be a set of instructions for generating a regression line from training data (supervised learning) or a set of instructions for grouping data into clusters of different classifications of data based on similarity, connectivity, and / or distance between data points (unsupervised learning). The regression line or data clusters can then be used as a model for predicting unknown information from known information. Once model has been built from learning module, the model may be used to generate a predicted output from a new request. A new request may be a request for a prediction associated with presented data. For example, a new request may be a request for classifying an image or for creating a recommendation for a user.

[0025] A “graph” can include a representation of a topological graph in a plane of distinct vertices connected by edges. The distinct vertices in a graph may be referred toPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 as “nodes.” Each node may represent specific information for an event or may represent specific information for a profile of an entity or object. The nodes may be related to one another by a set of edges, E. An “edge” may be described as an unordered pair composed of two nodes as a subset of the graph G=(V, E), where is G is a graph comprising a set V of vertices (nodes) connected by a set of edges E. For example, a graph may represent a transaction network in which a node representing a transaction may be connected by edges to one or more nodes that are related to the transaction, such as nodes representing information of a device, a user, a transaction type, etc. An edge may be associated with a numerical value, referred to as a “weight”, that may be assigned to the pairwise connection between the two nodes. The edge weight may be identified as a strength of connectivity between two nodes and / or may be related to a cost or distance, as it often represents a quantity that is required to move from one node to the next.

[0026] A “subgraph” can include a graph formed from a subset of elements of a larger graph. The elements may include vertices and connecting edges, and the subset may be a set of nodes and edges selected amongst the entire set of nodes and edges for the larger graph. For example, a plurality of subgraph can be formed by randomly sampling graph data, wherein each of the random samples can be a subgraph. Each subgraph can overlap another subgraph formed from the same larger graph.

[0027] The term “node” can include a discrete data point representing specified information. Nodes may be connected to one another in a graph by edges, which may be assigned a value known as an edge weight in order to describe the connection strength between the two nodes. For example, a first node may be a data point representing a first device in a network, and the first node may be connected in a graph to a second node representing a second device in the network. The connection strength may be defined by an edge weight corresponding to how quickly and easily information may be transmitted between the two nodes. An edge weight may also be used to express a cost or a distance required to move from one state or node to the next. For example, a first node may be a data point representing a first position of a machine, and the first node may be connected in a graph to a second node for a second position ofPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 the machine. The edge weight may be the energy required to move from the first position to the second position.

[0028] An “interaction” may include a reciprocal action or influence. An interaction can include a communication, contact, or exchange between parties, devices, and / or entities. Example interactions include a transaction between two parties and a data exchange between two devices. In some embodiments, an interaction can include a user requesting access to secure data, a secure webpage, a secure location, and the like. In other embodiments, an interaction can include a payment transaction in which two devices can interact to facilitate a payment.

[0029] “Interaction data” can include data related to and / or recorded during an interaction. In some embodiments, interaction data can be transaction data of the network data. Transaction data can comprise a plurality of data elements with data values.

[0030] A “transformer” can include a deep learning architecture that utilizes attention mechanisms. A transformers can include primary components of 1) tokenizers, which convert text into tokens, 2) embedding layers, which convert tokens into semantically meaningful representations, and 3) transformer layers, which carry out the reasoning capabilities, and can consist of Attention and MLP layers. Transformer layers can be one of two types, encoder and decoder. A transformer can include an encoder, a decoder, or both an encoder and a decoder. Bidirectional encoder representations from transformers (BERT) are an example of encoder-only models, while generative pre- trained transformers (GPT) are an example of decoder-only models.

[0031] An “encoder” can include a type of neural network that converts data from a raw format into another form. An encoder can include a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weights the encodings’ relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. These output encodings are then passed to the next encoder as its input in a looping manner to encode a full input text. The encoder can bePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 bidirectional in terms of attention, for example, attention can be placed on tokens before and after the current token.

[0032] A “decoder” can include a type of neural network that generates output from encoded representations. A decoder can include attention mechanisms (e.g., a self-attention mechanism and an attention mechanism over the encodings) and a feed- forward neural network. A decoder can function in a similar fashion to an encoder, but an additional attention mechanism is inserted, which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the encoder-decoder attention.

[0033] An “embedding” can include vector representation of data. An embedding can be a low-dimensional learned continuous vector representation of discrete variables. An embedding can be generated using a neural network that is trained to convert input data into output vector representations of the input data.

[0034] A “processor” may include a device that processes something. In some embodiments, a processor can include any suitable data computation device or devices. A processor may comprise one or more microprocessors working together to accomplish a desired function. The processor may include a CPU comprising at least one high-speed data processor adequate to execute program components for executing user and / or system-generated requests. The CPU may be a microprocessor such as AMD's Athlon, Duron and / or Opteron; IBM and / or Motorola's PowerPC; IBM's and Sony's Cell processor; Intel's Celeron, Itanium, Pentium, Xeon, and / or XScale; and / or the like processor(s).

[0035] A “memory” may be any suitable device or devices that can store electronic data. A suitable memory may comprise a non-transitory computer readable medium that stores instructions that can be executed by a processor to implement a desired method. Examples of memories may comprise one or more memory chips, disk drives, etc. Such memories may operate using any suitable electrical, optical, and / or magnetic mode of operation.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0036] A “server computer” may include a powerful computer or cluster of computers. For example, the server computer can be a large mainframe, a minicomputer cluster, or a group of servers functioning as a unit. In one example, the server computer may be a database server coupled to a Web server. The server computer may comprise one or more computational apparatuses and may use any of a variety of computing structures, arrangements, and compilations for servicing the requests from one or more client computers. DETAILED DESCRIPTION

[0037] Embodiments can provide for fine-grained graph rationalization, including well-tailored intervention strategies on graph data. Self-attention module(s) of transformer models can operate on graph data and provide interactions between input nodes. The training of an invariant graph transformer (IGT) can determine invariant nodes of a graph, which can improve a classification output of the machine learning model.

[0038] The invariant graph transformer can be implemented in various ways, e.g., a node-level IGT (IGT-N) and virtual node-level IGT (IGT-VN). The rationale environment decomposition of the IGT-N is designed on the node level and the decomposition of the IGT-VN is designed on the virtual node level. Comprehensive experiments on 6 real-world datasets showed performance advantages or competitiveness of embodiments against 12 strong baselines.

[0039] A rationale can be a subset of the input features that play a critical role in the model prediction on downstream tasks (Liu et al.2022; Wu et al.2022a). A rationale graph can be a subgraph of the input graph, which contains most of the task-relevant semantics. The application of graph rationale can provide explanation to the prediction results on which part of the input graph plays an important role to derive the final prediction. The rationale discovery can also drive the effective prediction on out-of- distribution test samples. That is because, according to the common content / style decomposition (Kaddour et al.2022), for data from different distributions, their stylePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 (e.g., environment) features are changing but their content features (e.g., rationales) are invariant.

[0040] Existing graph rationalization solutions (Liu et al., 2022; Wu et al., 2022a) employ a trainable augmenter to execute the rationale / environment decomposition. In this process, a node / edge mask is generated by the augmenter to decompose the given graph into a rationale graph and an environment graph. The idea of graph rationalization is to preserve the utility of the graph rationale even when faced with changing environment graphs. To achieve this, a technique named intervention is used, where the environment graph interacts with the rationale graph.

[0041] The intervention mechanism (e.g., intervener) can accurately represent the interaction between the rationale and the environment. The intervener can work in an adversarial behavior against the aforementioned augmenter, a point not emphasized in the existing literature. If the intervener is more powerful, it can capture more detailed interactions between the rationale and environment subgraphs. Given such a powerful intervener, the augmenter is compelled to minimize these interactions between the graph rationale and the environment, so as to obtain a purer graph rationale.

[0042] Existing works develop interveners in a coarse and non-parametric manner. After performing rationale and environment decomposition on the graph data, previous works compute graph-level embeddings for the rationale and environment subgraphs. The intervention is then designed as an interaction between these graph- level embeddings. For example, (Liu et al., 2022) adds the environment embedding into the rationale embedding as the intervened rationale embedding, while (Wu et al., 2022a) defines the intervened prediction as the Hadamard product between the predictions based on the rationale subgraph and the environment subgraph. Such a graph-level non-parametric intervention is insufficient to effectively represent the interaction between the rationale and environment graphs.

[0043] In response to this technical problem, embodiments provide for a fine- grained parametric intervention mechanism referred to as fine-grained graph rationalization. Fine-grained graph rationalization can utilize the self-attention module inPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 the transformer model, which captures interactions between input tokens. Building upon insights from transformers (Vaswani et al., 2017) and their linear variant Linformer (Wang et al., 2020), fine-grained graph rationalization can formulate the interaction between the rationale and environment subgraphs at the node-level or the virtual node level. Embodiments provide for both variants, which are referred to as IGT-N and IGT- VN. Additionally, to maximize the effectiveness of the intervention process, embodiments formulate a min-max game involving the node encoder, augmenter, intervener, and predictor, compelling the rationale subgraph to be as informative as possible.

[0044] To evaluate the proposed approach, comprehensive experiments were conducted on 7 graph-level benchmarks. IGT-N and IGT-VN were compared against 13 state-of-the-art baseline methods. The results demonstrated that methods and systems according to embodiments outperform the baseline methods, validating the superior performance of methods described herein.

[0045] As such, embodiments provide for a technical solution to the graph rationalization problem by providing a fine-grained graph rationalization model, which works at the node / virtual node level. I. PRELIMINARIES

[0046] This section introduces the notations, the classic transformer architecture with a self-attention module, and overall ideas of existing graph rationalization works. A. Notations

[0047] Bold uppercase letters are used for matrices and tensors (e.g., A), bold lowercase letters for column vectors (e.g., u), lowercase and uppercase letters in regular font for scalars (e.g., d, K), and calligraphic letters for sets (e.g., ^^). The syntax from NumPy is used to index vectors / matrices / tensors. For example, ^^^ ^^, ∶^ and ^^^: , ^^^ represent the p-th row and the q-th column of matrix A; A[p, q] represents the entry at the p-th row and the q-th column. Similarly, u[p] denotes the p-th entry of vector u. Superscript ⊤ denotes the transpose of matrices and vectors. ∘ denotes the HadamardPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 product. || denotes the concatenation operation and the concatenation dimension will be clearly mentioned according to the context.

[0048] An attributed graph can be represented as ^^ ൌ ^ ^^, ^^, ^^^, which includes an adjacency matrix ^^ ∈ ℝ^ൈ^, a node feature matrix ^^ ∈ ℝ^ൈௗ^, and an edge feature matrix ^^ ∈ ℝ^ൈ^ൈௗಶ. ^^ denotes the number of nodes, and ^^^(or ^^ா) representsthe dimensions of or edge features, respectively. It can be assumed that the node and edge feature dimensions are the same (i.e., ^^^ൌ ^^ாൌ ^^) for brevity. If the dimensions are different, then a fully-connected layer can map the values into a common feature space. The ground truth of a graph is represented by y. B. Graph Transformer

[0049] The core modules of the transformer architecture are the self-attention layer and the feed-forward network layer. Given the input as a sequence of symbol representations ^^ ∈ ℝ^ൈௗಹ, it is first transformed into the query, key, and value matrices as ^^ ൌ ^^ ^^ொ, ^^ ൌ ^^ ^^^, ^^ ൌ ^^ ^^^(1)

[0050] Where query weights, key weights, and value weights are notated as ^^ொ∈ ℝௗಹൈௗೂ, ^^^∈ ℝௗಹൈௗ^, ^^^∈ ℝௗಹൈௗೇ. For brevity, the following dimensions can be assumed ^^ுൌ ^^ொൌ ^^^ൌ ^^^ൌ ^^. Then, the output of the self-attention module is computed as: ^^ ^^^^^ ൌ ^^ ^^ ^^ ^^^ ^^^ ൌ ^^^ ^ (2a)

[0051] A choice of the non-linearity σ is a softmax function and equation (2b) shows the residual connection in the self-attention module. Then, the feed-forward network (FFN) module updates the symbol representation matrix ^^ as follows, ^^ ← ^^ ^^ ^^^ ^^^ ^ ^^ (3)PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0052] Equation 3 also includes a residual connection according to the transformer architecture. Some additional techniques can be included such as layer normalization (Ba, Kiros, and Hinton 2016), dropout (Srivastava et al.2014), and multi- head attention (Vaswani et al.2017), but they are omitted here for brevity.

[0053] As the transformer was originally tailored for sequence data or set data with positional encoding, a variety of techniques are proposed to transform the graph- structured data into set data. According to the taxonomy proposed by (Min et al.2022), most graph transformers are designed from the perspectives of (1) incorporating the topology encoding into the node features (e.g., ^^ ൌ ^^|| ^^ ^^ ^^ ^^ ^^ ^^^ ^^^) where a typical example of ^^ ^^ ^^ ^^ ^^ ^^ function is extracting the eigenvectors, (2) incorporating the gy encoding into the attention matrix (e.g., ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^^^^ ^topolo^^√ௗ, ^^^) where an example of ^^ ^^ ^^ ^^ ^^ ^^ ^^ function is Hadamard graph neuralnetworks (GNNs) (Dwivedi et al.2023) as an auxiliary module (e.g., ^^ ൌ ^^ ^^ ^^^ ^^, ^^^).

[0054] The idea of the transformer architecture is to reconstruct a weighted omplete propagation graph ^^ ൌ ^^^^ ^c^ ^^√ௗ^. C. Invarianton Graphs

[0055] The rationale graph is a subgraph that encodes most downstream task- relevant semantics. An example is a functional group in polymer graphs (Liu et al., 2022; Wu et al., 2022a), which fundamentally determines the chemical property of a polymer. Mathematically, a given graph is decomposed into a rationale graph and an environment graph: ^^ → ^^^^∪ ^^^^௩. The graph embeddings on ^^^^and ^^^^௩are computed as ℎ^^and ℎ^^௩. To ensure the rationale graph is invariant with respect to the prediction results when confronting different environments, a utility loss is minimized given the rationale embedding ℎ^^intervened by the environment embedding ℎ^^^௩(e.g.,^^ ^^ ^^ ℒ௨௧^^^ℎ^^^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ℎ^^^௩^). Here, ℎ^^^௩ could be either from the same graph (e.g.,ℎ^^^௩ൌ ℎ^^௩), or could be environment embeddings from other graphs, such as those in the batch. A key difference among existing methods lies in the intervention operation,PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 two examples include: 1) GREA (Liu et al., 2022) designs the intervention as an addingoperation (e.g., ℎ^^ ^ ℎ^^^௩) and 2) DIR (Wu et al., 2022a) designs the intervention as anelement-wisely weighting of the prediction vector (e.g., ^^^^^ௗ^ℎ^^^ ∘ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^^^ௗ^ℎ^^^௩^^, where ∘ is the Hadamard product and ^^^^^ௗis a predictor). II. SYSTEMS

[0056] In this section the systems according to embodiments will be discussed. The systems described herein can perform rationale and environment graph decomposition and intervention. A. Overview

[0057] FIG.1 shows a system 100 according to embodiments of the disclosure. The system 100 comprises an analysis computer 102, a database 104, a network processing computer 106, an authorizing entity computer 108, a transport computer 110, a resource provider computer 112, an access device 114, and a user device 116. The analysis computer 102 can include a machine learning model 118. The database 104 can store training samples 120 and production samples 122.

[0058] The analysis computer 102 can be in operative communication with the database 104 and the network processing computer 106. The network processing computer 106 can be in operative communication with the database 104, the authorizing entity computer 108, and the transport computer 110. The resource provider computer 112 can be in operative communication with the transport computer 110 and the access device 114. The access device 114 can be in operative communication with the user device 116.

[0059] For simplicity of illustration, a certain number of components are shown in FIG.1. It is understood, however, that embodiments of the invention may include more than one of each component. In addition, some embodiments of the invention may include fewer than or greater than all of the components shown in FIG.1.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0060] Messages between at least the devices in the system 100 in FIG.1 can be transmitted using a secure communications protocols such as, but not limited to, File Transfer Protocol (FTP); HyperText Transfer Protocol (HTTP); Secure Hypertext Transfer Protocol (HTTPS), SSL, ISO (e.g., ISO 8583) and / or the like. The communications network may include any one and / or the combination of the following: a direct interconnection; the Internet; a Local Area Network (LAN); a Metropolitan Area Network (MAN); an Operating Missions as Nodes on the Internet (OMNI); a secured custom connection; a Wide Area Network (WAN); a wireless network (e.g., employing protocols such as, but not limited to a Wireless Application Protocol (WAP), I-mode, and / or the like); and / or the like. The communications network can use any suitable communications protocol to generate one or more secure communication channels. A communications channel may, in some instances, comprise a secure communication channel, which may be established in any known manner, such as through the use of mutual authentication and a session key, and establishment of a Secure Socket Layer (SSL) session.

[0061] The analysis computer 102 can include a computer or server computer. The analysis computer 102 can analyze graph data to determine a rationale portion of the graph data. The analysis computer 102 can train the machine learning model 118 to classify graph properties of graphs. The analysis computer 102 can utilize the machine learning model 118 to determine whether or not a particular rationale graph is in fact invariant. The analysis computer 102 can obtain graph data from the database 104 for use in the machine learning model 118 to train the machine learning model 118 and to determine classifications for graph data. For example, the analysis computer 102 can obtain the graph data from the database 104, where the database 104 can receive node data and edge data from the network processing computer 106 in an interaction network.

[0062] During training, the analysis computer 102 can obtain the training samples 120 from the database 104. The analysis computer 102 can train the machine learning model 118 using the training samples 120. The training samples 120 can include graphs that are labeled with particular classifications. During training, the analysis computerPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 102 can adjust parameters in the machine learning model 118 to obtain more accurate classifications.

[0063] During production, the analysis computer 102 can obtain the production samples 122 from the database 104. The production samples 122 can include unlabeled graphs. The analysis computer 102 can use the trained machine learning model 118 to generate classification outputs for the production samples 122.

[0064] As an illustrative example, the analysis computer 102 can receive the training samples 120 that include data related to access requests. Each node in the graphs included in the training samples 120 can be associated with a user or a resource provider that provides resources to users. The edges in the graphs can be associated with access requests that represent a particular user requesting access to a resource from a particular resource provider. Each graph in the training samples 120 can be labelled with a classification (e.g., fraudulent network, non-fraudulent network, highly connected network, sparse network, etc.). The analysis computer 102 can train the machine learning model 118 using the training samples 120 to predict classifications from the production samples 122.

[0065] After training the machine learning model 118, the analysis computer 102 can generate classification outputs using the machine learning model 118 for the production samples 122. For example, the analysis computer 102 can obtain production samples 122 that include graphs, which include nodes representing users or resource providers and edges representing access requests. The analysis computer 102 can generate a classification output for each graph in the production samples 122. The classification output can be a classification of the graph. The analysis computer 102 can provide the classification output to a downstream device. The downstream device can utilize the classification output in any suitable manner (e.g., to make a determination related to the graph or data represented therein). For example, the analysis computer 102 can provide the classification output to the network processing computer 106.

[0066] The database 104 can include any suitable database. The database may be a conventional, fault tolerant, relational, scalable, secure database such as thosePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 commercially available from Oracle™ or Sybase™. The database 104 can store graph data that can be provided to the analysis computer 102. The database 104 can store a plurality of graphs constructed from data obtained from the network processing computer 106, for example. It is understood, however, that the database 104 can obtain graph data or data points for graph data from any suitable computer. For example, the database 104 can receive molecular compound data from a mass spectrometer or other scientific equipment. In some embodiments, a graph creation computer can be present and can receive raw data from the network processing computer 106, construct graphs from the raw data, and store the graphs in the database 104.

[0067] The user device 116 can include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), personal digital assistants (PDAs), Internet of Things (IoT) devices, and / or the like. The user device 116 can initiate interactions (e.g., access requests to electronic or physical resources) with resource provider computers and / or access devices. For example, the user device 116 can select one or more items for the interaction at a resource provider location (e.g., a grocery store). During checkout, the user can be instructed to tap (e.g., bring into near- field communication range) the user device 116 against the access device 114. The user device 116 can utilize a camera to aid the user in bringing the user device 116 into a correct location (e.g., a landing plane) using direction instructions. In some embodiments, the user device 116 can utilize machine learning models to determine the correct location. In other embodiments, the user device 116 can utilize machine identification code to determine the correct location. Once in range, the user device 116 can communicate data, such as credentials, to the access device 114 for the interaction.

[0068] The access device 114 can include a device operated by a resource provider. The access device 114, for example, can include a mobile device, a POS terminal, a laptop, etc. The access device 114 can communicate with another device (e.g., a user device 116) to perform an interaction. During the interaction, the access device 114 can receive credentials from the user device and can provide interactionPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 data to the resource provider computer 112 for authorization of the interaction. In some embodiments, the access device 114 can generate an authorization request message comprising at least the interaction data. The access device 114 can provide the authorization request message to the resource provider computer 112.

[0069] The resource provider computer 112 can include any suitable computational apparatus operated by a resource provider (e.g., a merchant). In some embodiments, the resource provider computer 112 may include one or more server computers that may host one or more websites associated with the resource provider (e.g., a merchant). In some embodiments, the resource provider computer 112 may be configured to send data to the network processing computer 106 via the transport computer 110 as part of a payment verification and / or authentication process for an access request between the user (e.g., consumer) and the resource provider. The resource provider computer 112 may also be configured to generate authorization request messages for access requests between a resource provider and a user and route the authorization request messages to the authorizing entity computer 108 for processing.

[0070] The transport computer 110 can include a server computer. The transport computer 110 may be associated with an acquirer, which may be an entity (e.g., a commercial bank) that has a business relationship with a particular merchant or other entity. Some entities can perform both issuer and acquirer functions. Some embodiments may encompass such single entity issuer-acquirers.

[0071] The network processing computer 106 can include a server computer. The network processing computer 106 may be disposed between the transport computer 110 and the authorizing entity computer 108. The network processing computer 106 may include data processing subsystems, networks, and operations used to support and deliver authorization services, exception file services, and clearing and settlement services. For example, the network processing computer 106 may comprise a server coupled to a network interface (e.g., by an external communication interface), and databases of information. The network processing computer 106 may be representativePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 of a transaction processing network. An exemplary transaction processing network may include VisaNet™. Transaction processing networks such as VisaNet™ are able to process credit card transactions, debit card transactions, and other types of commercial transactions. VisaNet™, in particular, includes a VIP system (Visa Integrated Payments system), which processes authorization requests, and a Base II system, which performs clearing and settlement services. The network processing computer 106 may use any suitable wired or wireless network, including the Internet.

[0072] In some embodiments, the network processing computer 106 can generate a classification request message. The classification request message can include a request for a classification output for a graph that is included in the production samples 122. The graph can be identified by a graph identifier that indicates where the graph is stored in the database 104. The network processing computer 106 can provide the classification request message to the analysis computer 102. The analysis computer 102 can retrieve the graph from database 104 using the graph identifier. The analysis computer 102 can determine a classification output from the graph. The analysis computer 102 can generate a classification response message comprising the classification output. The analysis 102 computer can provide the classification response message to the network processing computer 106 in response to the classification request message.

[0073] The authorizing entity computer 108 can include a server computer operated by an authorizing entity. The authorizing entity computer 108 may be associated with an authorizing entity, which may be an entity that authorizes a request. An example of an authorizing entity may be an issuer, which may typically refer to a business entity (e.g., a bank) that maintains an account for a user. An issuer may also issue and manage an account associated with the user device 116. B. Analysis Computer

[0074] FIG.2 shows a block diagram of an analysis computer 200 according to embodiments. The exemplary analysis computer 200 may comprise a processor 204. The processor 204 may be coupled to a memory 202, a network interface 206, and aPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 computer readable medium 208. The computer readable medium 208 can comprise a graph decomposition module 208A and an intervention module 208B.

[0075] The memory 202 can be used to store data and code. For example, the memory 202 can store machine learning data, weights, graph data, etc. The memory 202 may be coupled to the processor 204 internally or externally (e.g., cloud based data storage), and may comprise any combination of volatile and / or non-volatile memory, such as RAM, DRAM, ROM, flash, or any other suitable memory device.

[0076] The computer readable medium 208 may comprise code, executable by the processor 204, for performing a method of training. The training method can include the analysis computer 200 generating a rationale graph embedding and a first environment graph embedding using one or more first layers of a machine learning model. The rationale graph can correspond to a rationale graph. The first environment graph embedding can correspond to a first environment graph. Each graph embedding can be generated on a per node basis using a graph. As such, each graph embedding can include the same number of embeddings as the number of nodes in the graph. The analysis computer 200 can generate a second environment graph embedding from a second environment graph. The second environment graph embedding can also be generated on a per node basis. After generating the rationale graph embedding, the first environment graph embedding, and the second environment graph embedding, the analysis computer 200 can concatenate the rationale graph embedding and the first environment graph embedding to form a first combination input. The analysis computer 200 can also concatenate the rationale graph embedding and the second environment graph embedding to form a second combination input. After obtaining the first combination input and the second combination input, the analysis computer 200 can generate, using a second layer of the machine learning model, a first classification output using the first combination input. The analysis computer 200 can then generate, using the second layer of the machine learning model, a second classification output using second combination input. After generating the first classification output and the second classification output, the analysis computer 200 can adjust parameters of the one or more first layers and the second layer of the machine learning model. ThePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 analysis computer 200 can adjust the parameters based on a comparison of the first classification output and the second classification output. The machine learning model can includes a transformer sub-model that generates an attention data structure. The analysis computer 200 can adjust the parameters of the one or more first layers and the second layer to maximize a self attention of the rationale graph embedding and minimize interactions between the rationale graph embedding and the first environment graph embedding and the second environment graph embedding.

[0077] The computer readable medium 208 may comprise code, executable by the processor 204, for performing a production method. The analysis computer 200 can generate a rationale graph embedding and an environment graph embedding. The analysis computer 200 can generate the rationale graph embedding and the environment graph embedding using a graph and one or more first layers of a machine learning model. The rationale graph embedding can correspond to a rationale graph of the graph. The environment graph embedding can correspond to an environment graph of the graph. The analysis computer can concatenate the rationale graph embedding and the environment graph embedding to form a combination input. After generating the combination input, the analysis computer 200 can generate, using a second layer of the machine learning model, a classification output using the combination input. The machine learning model can include a transformer sub-model that generates an attention data structure. The parameters of the one or more first layers and the second layer were trained (1) to maximize a self-attention of the rationale graph embedding in the attention data structure and (2) to minimize interactions between the rationale graph embedding and the environment graph embedding in the attention data structure.

[0078] The graph decomposition module 208A may comprise code or software, executable by the processor 204, for performing graph decomposition. The graph decomposition module 208A can include an encoder and an augmenter. The encoder can encode nodes of a graph to obtain node embeddings. The encoder can encode the graph on a per node basis. The augmenter can decompose the node embeddings into two distinct subsets: 1) rationale nodes and 2) environment nodes. The graph decomposition module 208A, in conjunction with the processor 204, can obtain a graphPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 and generate a rationale graph embedding and an environment graph embedding for the graph.

[0079] The intervention module 208B can include may comprise code or software, executable by the processor 204, for performing intervention. The intervention module 208B can include an intervener that utilizes a transformer architecture to transform input data. The intervention module 208B, in conjunction with the processor 204, can transform input data (e.g., first combination inputs, second combination inputs, etc.) into transformed embeddings. The intervention module 208B, in conjunction with the processor 204, can iteratively generate transformed embeddings using an attention process in the transformer. The attention process can apply attention to the relationships between the different nodes represented in the rationale graph embedding and the environment graph embedding.

[0080] The transformed embeddings can be utilized by a predictor ^^^^^ௗto determine a classification corresponding to the graph, the rationale graph, and the environment graph.

[0081] The network interface 206 may include an interface that can allow the analysis computer 200 to communicate with external computers. The network interface 206 may enable the analysis computer 200 to communicate data to and from another device (e.g., a database 104, a downstream device, etc.). Some examples of the network interface 206 may include a modem, a physical network interface (such as an Ethernet card or other Network Interface Card (NIC)), a virtual network interface, a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, or the like. The wireless protocols enabled by the network interface 206 may include Wi-FiTM. Data transferred via the network interface 206 may be in the form of signals, which may be electrical, electromagnetic, optical, or any other signal capable of being received by the external communications interface (collectively referred to as “electronic signals” or “electronic messages”). These electronic messages that may comprise data or instructions may be provided between the network interface 206 and other devices via a communications path or channel. As noted above, anyPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 suitable communication path or channel may be used such as, for instance, a wire or cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link, a WAN or LAN network, the Internet, or any other suitable medium. III. INVARIANT GRAPH TRANSFORMER MODEL

[0082] In this section, the graph rationalization method according to embodiments is described. Embodiments can utilize a module based on transformer architecture. This section provides an overview of fine-grained graph rationalization methods, highlighting four parametric modules: an encoder, an augmenter, an intervener, and a predictor.

[0083] The encoder, denoted as ^^^^^∶ ^^ → ℝ^ൈௗ, accepts a graph data as input and produces a node embedding matrix as output. Example graph encoders include graph neural networks (GNNs) (Wu et al., 2021) and graph Transformers (Min et al., 2022). The encoder module can include any suitable graph encoder ^^^^^.

[0084] The predictor, denoted as ^^^^^ௗ∶ ℝௗ→ ℝ^takes as input a graph embedding and outputs a task-relatedFor graph regression tasks ^^ ൌ 1, while for graph classification tasks ^^ is the number of classes. The predictor can be a multi-layer perceptron (MLP) with appropriate activation functions.

[0085] Further details of the encoder and predictor in embodiments are presented below. The augmenter and intervener will be described in subsequent subsections. The designs of the augmenter and intervener are described as two variants of the fine- grained graph rationalization methods. A. Node-Level Fine-Grained Graph Rationalization

[0086] In some embodiments, the fine-grained graph rationalization method can be a node-level fine grained graph rationalization method (IGT-N). FIG.3A shows a diagram of a node-level fine-grained graph rationalization method according to embodiments. The method illustrated in FIG.3A includes a graph decomposition phase 302 and an intervention phase 304.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0087] During the decomposition phase 302, a computer can decompose a first graph 306 and a second graph 308 into environment and rationale node embedding datasets. During the intervention phase 304, the computer can perform an intervention process with the decomposed environment and rationale node embedding datasets to determine whether or not the rationale portion is invariant when introduced to an environment from the second graph 308.

[0088] The computer can receive the first graph 306 and the second graph 308 from the database 104. After receiving the first graph 306 and the second graph 308, the computer can encode the first graph 306 to obtain a first node embedding matrix ^^ 312 and can encode the second graph 308 to obtain a second node embedding matrix (not shown). The node embedding matrices can include a plurality of embeddings, where each embedding corresponds to a particular node in the originating graph.

[0089] The computer can generate the node embedding matrices in any suitable manner. For example, the computer can train a machine learning model that accepts node data as input (e.g., adjacency matrices, etc. from a graph) to generate node embeddings corresponding to the nodes.

[0090] After generating the first node embedding matrix ^^ 312 and the second node embedding matrix, the computer can process the first node embedding matrix ^^ 312 and the second node embedding matrix using an augmenter 314. 1. Node-level augmenter

[0091] The augmenter 314 is a module of the proposed fine-grained graph rationalization method. For the node-level variant, a function of the augmenter 314 can be to decompose the node set into two distinct subsets: 1) rationale nodes and 2) environment nodes. This decomposition is operated by parameterizing the node-level augmenter 314 as a learnable node partitioner, denoted by ^^^௨^ିே, ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^ ^^ ^^^ ^^, ^^^௨^ିே^^ (4)PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0092] The input is the node embedding matrix ^^ ∈ ℝ^ൈௗ312, and the output is a partition vector ^^ ∈^0, 1^^, where ^^ ^^ ^^ is a multi-layer perceptron. Each entry within ^^, such as ^^^ ^^^, signifies the probability of the i-th node being categorized as a rationale node. The computer can generate the partition vector ^^ using a learnable node partitioner machine learning model.

[0093] For the node partition vector ^^, its top-K entries are indexed as ^^ ^^ ^^^^ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^, which is used to index the rationale nodes from the node embedding matrix ^^ 312. The remaining nodes are categorized as the environment nodes whose indices are ^^ ^^ ^^^^௩ൌ ^1, ... , ^^^ െ ^^ ^^ ^^^^, where െ is a set difference operation and where ^^ is a hyperparameter whose impact is discussed below.

[0094] Using the aforementioned indices, the computer can extract rationale and environment embeddings from the node embedding matrix ^^ 312, denoted as ^^^^318 and ^^^^௩320, respectively. The computer can extract the rationale embedding ^^^^318 and the environment embedding ^^^^௩320 using equations (5a) and (5b). ^^^^ൌ ^^^ ^^ ^^ ^^^^, : ^ ∈ ℝ^ൈௗ(5a)

[0095] embedding ^^^^௩320 corresponding to the first graph 306, the computer can determine an alternate environment embedding ^^^^^322 corresponding to the second graph 308. The computer can determine the alternate environment embedding ^^^^^322 using the augmenter 314 in a similar manner to determining the environment embedding ^^^^௩320. 2. Node-level intervener

[0096] After obtaining the rationale embedding ^^^^318, the environmentembedding ^^^^௩ 320, and the alternate environment embedding ^^^^^ 322, the computercan begin the intervention phase 304.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0097] During the intervention phase 304, the computer can concatenate the rationale embedding ^^^^318 (e.g., the node-based rationale graph embedding) and the environment embedding ^^^^௩320 (e.g., the node-based first environment graph embedding) to form a first combination input 324.

[0098] The computer can also concatenate the rationale embedding ^^^^318 (e.g., the node-based rationale graph embedding) and the alternate environment embedding ^^^^^௩322 (e.g., the node-based second environment graph embedding) to form a second combination input 326.

[0099] The design of the fine-grained intervener ^^ 328 utilizes similar steps to a transformer architecture. Explicitly, the node-level intervener ^^ 328 is presented as: ^^^^௧^^, ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^^|| ^^^^௩^ (6a) where ^^ ൌ ^^ ^^ ^^ ^^^ ^^^^|| ^^^^௩^ (6b)

[0100] In this representation, the operator || concatenates along the first dimension of the matrices ^^^^and ^^^^௩. Equations (1) to (3) are referred to as Transformer and ^^ is the intermediate attention matrix from the self-attention layer (e.g., equation (6b)). Here, the self-attention module models the interactions between the rationale nodes ^^^^and the environment nodes ^^^^௩. Φ includes all the parameters of the Attn (e.g., equation (6b)) and FFN (e.g., equation (3)) modules. In some contexts, where the attention matrix ^^ is not explicitly used as an output, input and output of the intervener ϕ can be presented as ^^^^௧^^ൌ ^^^ ^^^^|| ^^^^௩^. During the attention process ( ^^ ^^ ^^ ^^), the computer can generate a key matrix, a query matrix, and a value matrix that can be used to apply attention to the node embeddings.

[0101] The computer can iteratively utilize the intervener 328 to determine attention values for the first combination input 324 and the second combination input 326. The intervener 328 can determine first transformed embeddings based on the first combination input 324 (e.g., ^^^^௧^^) and can determine second transformed embeddings based on the second combination input 326.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 3. Node-level optimization objective

[0102] After generating first transformed embeddings and second transformed embeddings using the intervener 328, the computer can perform a readout and prediction process to determine a first loss function 332 and a second loss function 334. The first loss function 332 can correspond to the situation where the rationale embedding ^^^^318 is intervened with environment of the same graph (e.g., the first graph 306). The second loss function 334 can correspond to the situation where the rationale embedding ^^^^318 is intervened with environment of a different graph (e.g., the second graph 308).

[0103] The computer can compute the utility loss term as ^^௨௧^^^ ^^^^|| ^^^^௩^ ൌ ^^௧^^^^ ^^^^^ௗ∘ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ∘ ^^^ ^^^^|| ^^^^௩^, ^^^, where ^^௧^^^is the task-specific objective. For squared error for regression tasks or thecross-entropy for classification tasks. The computer can perform invariant rationale discovery to find the graph rationale so that the utility loss attains minimization given changing environments. Thus, the total utility objective is ^^௨௧^^ ൌ ^^௨௧^^^ ^^^^|| ^^^^௩^ ^ ^^ ^^௨௧^^^ ^^^^|| ^^^^^௩^ (7)

[0104] embeddings from the changing environments. In practical implementations, the alternateenvironment embedding ^^^^^௩ 322 is the environment node embeddings from othergraphs in a mini-batch for training. Additionally, to fully utilize the rich interactions from the fine-grained intervention module, the computer can apply the following partition regularization term, ^^^^^^ ^^^^||^^^^௩^ൌ ^^ୃ^^^^^ െ ^^^^^^^ െ ^^^ୃ^^ ^^ (8)

[0105] , ^^^ ^^^ ൌ ^ 1 ^^ ^^ ^^ ^ ^^0^^ ^^ℎ ^^ ^^ ^^ ^^ ^^ ^^(9)PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0106] 0-based indexing is used so there are in total K non-zero entries (e.g., 1) in ^^. The meaning of the binary s vector is to designate whether a particular row of the rows of the matrix ^^^^|| ^^^^௩originates from the rationale nodes or the environment nodes. The underlying notion of the regularization term in equation (8) is to impose penalties on interactions between the rationale nodes and the environment nodes. Namely, the two terms 1) ^^ୃ^^^ ^^ െ ^^^ and 2) ^ ^^ െ ^^^ୃ^^ ^^ denote the total weights on the links between the rationale and environment subgraphs. To handle the changing environments, the computer can introduce an additional regularization term on thechanging environments as ^^^^^^ ^^^^|ห ^^^^^௩൯ where ^^^^^௩ is the environment nodeembeddings from another graph within the same mini-batch. Then, the total regularization term is: ^^^^^ ൌ ^^^^^^ ^^^^|| ^^^^௩^ ^ ^^ ^^ ^^ ^^^ ^^^^|| ^^^^^௩^ (10)

[0107] objective loss function can include a utility loss term and a regularization loss term. To fully harness the capabilities of the fine-grained parametric intervener 328. The behavior of the intervener 328 ϕ operates in an adversarial fashion to the other modules. As a result, it can be formulated that a min-max game can include ^^ ൌ ^ ^^^^^, ^^^௨^ିே, ^^^^^ௗ^ and ϕ as, ^^^^^௫ఏథ^^௨௧^^^ ^^ ^^^^^(11)

[0108] graph rationale by promoting interactions between the rationale nodes and the environment nodes. Conversely, the encoder, augmenter, and predictor (e.g., θ) are optimized in an opposing manner to the intervener’s objectives. As such, a first loss value can be determined using a first utility loss value and a first regularization loss value, while a second loss value can be determined using a second utility loss value and a second regularization loss value. The computer can perform a gradient decent method to optimize the loss functions to minimize ^^ and maximize ^^.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 4. Complexity of node-level fine-grained graph rationalization

[0109] As the encoder ^^^^^and the predictor ^^^^^ௗare off-the-shelf, node-level fine-grained graph rationalization introduces two new modules: the node-level augmenter, ^^^௨^ିே, and the transformer-based intervener, ϕ. Notably, despite these additions, the increase in the number of parameters remains modest. The parameters for ^^^௨^ିேoriginate from the ^^ ^^ ^^ defined in equation (4). In a configuration where the ^^ ^^ ^^ has 3 layers with a feature dimension of ^^, the parameter count is ^^^2 ^^ଶ^. The intervener ϕ, driven by the transformer layer in equation (6a), has its parameters confined to ^^^3 ^^ଶ^ 2 ^^ଶ^ ൌ ^^^5 ^^ଶ^, owing to its query, key, value projection matrices and the feed-forward net (FFN from equation (3), typically a 3-layered MLP).

[0110] Further details regarding the method illustrated in FIG.3A are illustrated in a step-by-step algorithm for node-level fine-grained graph rationalization in Algorithm 1, below. In a test phase, the output of ^^^^^ௗ∘ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ∘ ^^^ ^^^^|| ^^^^௩^ is evaluated. Algorithm 1: Node-level single training step for every training graph ^^PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 7 update ^^ via gradient descent withఈ^ ^^ೠ^^^ାఉ ^^^^^^డఏ;B. Virtual Node-Level Fine-Grained Graph Rationalization with Pre-Encoding Decomposition

[0111] In some embodiments, the fine-grained graph rationalization method can be a node-level fine grained graph rationalization method (IGT-N) and can utilize an encoding process that occurs after separation of the rationale graph and the environment graph from the graph. The method illustrated in FIG.3B includes a graph decomposition phase 352 and an intervention phase 354.

[0112] The computer can receive a first graph 356 and a second graph 358 from the database 104. During the decomposition phase 352, the computer can separate the first graph 356 into a rationale graph 362 and a first environment graph 364. The computer can use an augmenter 360 to separate the graph 356 into the rationale graph 362 and the first environment graph 364. The augmenter 360 can decompose the graph 362 into two distinct subsets of nodes from the graph 362. The augmenter 360 can be a learnable node partitioner that can be trained to identify nodes of the graph 356 as being a part of the rationale graph 362 or of the first environment graph 364. For example, the augmenter 360 can be a machine learning model that is trained to classify nodes in the graph 306 as being a part of the rationale graph 362 or of the first environment graph 364.

[0113] After determining the rationale graph 362 and the first environment graph 364, the computer can embed the nodes from the rationale graph 362 and the first environment graph 364. The computer can generate a rationale graph embedding ^^^^370 using a first encoder 366 using the rationale graph 362. The computer can generate a first environment graph embedding ^^^^௩372 using a second encoder 368 using thePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 first environment graph 368. The first encoder 366 and the second encoder 368, in some embodiments, can be the same encoder.

[0114] The computer can also determine a second environment graph embedding ^^^^^௩374 corresponding to the second graph 308 in a similar manner to determining the first environment graph embedding ^^^^௩372.

[0115] During the intervention phase 354, the computer can perform an intervention process with the decomposed first environment and rationale node embedding datasets to determine whether or not the rationale portion is invariant when introduced to an environment from the second graph 308.

[0116] The computer can generate a first combination input 376 by concatenating the rationale graph embedding ^^^^370 and the first environment graph embedding ^^^^௩. The computer can generate a second combination input 378 by concatenating the rationale graph embedding ^^^^370 and the second environment graph embedding^^^^^௩374.

[0117] After generating the first combination input 376 and the second combination input 378, the computer can generate first transformed embeddings and second transformed embeddings from the first combination input 376 and the second combination input 378, respectively, using the intervener 380.

[0118] The computer can then generate a first classification output using the first combination input or the first transformed embeddings derived therefrom. The computer can generate a second classification output using the second combination input or the second transformed embeddings derived therefrom.

[0119] The computer can compute first loss function 384 and a second loss function 386 using at least the first classification output and the second classification output. The computer can optimize the loss functions to adjust parameters of the one or more first layers and the second layer based of the machine learning model.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 C. Virtual Node-Level Fine-Grained Graph Rationalization

[0120] In some embodiments, the fine-grained graph rationalization method can be a virtual node-level fine grained graph rationalization method (IGT-VN). FIG.4 shows a diagram of a virtual node-level fine-grained graph rationalization method according to embodiments. The method illustrated in FIG.4 includes a transformer 400, a graph decomposition phase 402, and an intervention phase 404. The graph decomposition phase 402 can correspond to the graph decomposition module 208A described in reference to FIG.2. The intervention phase 404 can correspond to the intervention module 208B.

[0121] During the decomposition phase 402, a computer can decompose a first graph 406 and a second graph 408 into environment and rationale node embedding datasets. During the intervention phase 404, the computer can perform an intervention process with the decomposed environment and rationale node embedding datasets to determine whether or not the rationale portion is invariant when introduced to an environment from the second graph 408.

[0122] The computer can receive the first graph 406 and the second graph 408 from the database 104. After receiving the first graph 406 and the second graph 408, the computer can encode the first graph 406 to obtain a first node embedding matrix ^^ 412 and can encode the second graph 408 to obtain a second node embedding matrix (not shown). The node embedding matrices can include a plurality of embeddings, where each embedding corresponds to a particular node in the originating graph.

[0123] The computer can generating the node embedding matrices in any suitable manner. For example, the computer can train a machine learning model that accepts node data as input (e.g., adjacency matrices, etc. from a graph) to generate node embeddings corresponding to the nodes.

[0124] After generating the first node embedding matrix ^^ 412 and the second node embedding matrix, the computer can process the first node embedding matrix ^^ 312 and the second node embedding matrix using an augmenter 414.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 1. Virtual node-level augmenter

[0125] In the previously described node-level fine grained graph rationalization method, the augmenter 314 decomposes the nodes into rationale nodes and environment nodes via a trainable node partitioner ^^^௨^ିே. In some embodiments, the computer can extract the graph rationale at a virtual node level using the augmenter 414.

[0126] A speedup technique can be utilized in a similar yet different manner from Linformer (Wang et al., 2020). The computer can reformulate both the attention matrix and node (e.g., token for a transformer) embedding matrix to dimensions of ℝ^ൈ^and ℝ^ൈௗ, respectively. This reformulation ensures that their multiplication scales linearly with the number of nodes (tokens) ^^. Within this configuration, ^^, a pre-defined rank, is significantly smaller than ^^, and ^^ represents the feature dimension. The method illustrated in FIG.4 provides that the restructured token embedding matrix, with dimensions of ℝ^ൈௗ, can be interpreted as embeddings for ^^ virtual nodes.

[0127] Given the first node embedding matrix ^^ 412, which were determined by the encoder, the computer, using the augmenter 414, can determine the virtual node embeddings ^^^ே416 as: ^^^ேൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^ேି^ே^ ^^ (12)

[0128] Here, the row-wise applied ^^ ^^ ^^ ^^ ^^ ^^ ^^ function, along with ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^ேି^ே^ ∈ ℝ^ൈ^, yields a trainable matrix assigning n nodes to r virtual nodes, where ^^ acts as a tunable hyper-parameter. A weight matrix ^^ேି^ேcan be learned over training iterations to yield optimal adjustments to the node embeddings ^^ 412 to obtain the virtual node embeddings ^^^ே416. In experiments, a virtual node dimensionality of ^^ ൌ 8 was utilized. As all the virtual node embeddings are learned, a subset of the ^^ virtual nodes can be designated as rationale virtual nodes, whose rationality is data-driven by the intervention procedure discussed in subsequent subsections. As an example, the initial ^^ virtual nodes are deemed as rationale virtual nodes, while the last ^^ െ ^^ nodes is considered the environment virtual node. Similar toPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 the node-level fine grained graph rationalization method, here, for the virtual node-level fine grained graph rationalization method, ^^ is a hyperparameter whose impact is evaluated below.

[0129] Later, during parameter adjustment during the training, the parameter of the trainable node partitioner ^^^௨^ି^ே, which aids in determining the virtual nodes is the weight matrix ^^ேି^ே.

[0130] After determining the virtual node embeddings ^^^ே416, the computer can determine a rationale graph embedding ^^^^418 and a first environment graph embedding ^^^^௩420 based on the virtual node embeddings ^^^ே416. The rationale graph embedding ^^^^418 and a first environment graph embedding ^^^^௩420 can be determined as being particular portions of the virtual node embeddings ^^^ே416 matrix based on the hyperparameter ^^. The first ^^ node embeddings of the virtual node embeddings ^^^ே416 matrix can be included in the rationale graph embedding ^^^^418. The last ^^ node embeddings of the virtual node embeddings ^^^ே416 matrix can be included in the first environment graph embedding ^^^^௩420. For example, the computer can compute the following equations: ^^^^ൌ ^^^ே^: ^^, : ^ ∈ ℝ^ൈௗ(13a) ^^^^௩ൌ ^^^ே^ ^^: , : ^ ∈ ℝ^^ି^^ൈௗ(13b)

[0131] a first environment graph embedding ^^^^௩420, the computer can determine a second virtual node embeddings ^^^^ேbased on the second graph 408. The computer can determinethe second virtual node embeddings ^^^^ே in a similar manner to determining the virtualnode embeddings ^^^ே416 (which can be a first virtual node embedding matrix ^^^ே416).

[0132] The computer can determine a second environment graph embedding ^^^^^௩422 based on the last ^^ node embeddings of the second virtual node embeddings^^^^ே matrix.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0133] After generating the rationale graph embedding ^^^^418, a first environment graph embedding ^^^^௩420, and the second environment graph embedding ^^^^^௩422, the computer begin the intervention phase 404 and can combine the rationale graph embedding ^^^^418 and the first environment graph embedding ^^^^௩420 to form a first combination input (e.g., ^^^^|| ^^^^௩). The computer can combine the rationale graph embedding ^^^^418 and the first environment graph embedding ^^^^௩420 by concatenating the rationale graph embedding ^^^^418 and the first environment graph embedding ^^^^௩420.

[0134] The computer can combine the rationale graph embedding ^^^^418 and the second environment graph embedding ^^^^^௩422 to form a second combination input (e.g., ^^^^|| ^^^^^௩). The computer can combine the rationale graph embedding ^^^^418 and the second environment graph embedding ^^^^^௩422 by concatenating the rationalegraph embedding ^^^^ 418 and the second environment graph embedding ^^^^^௩ 422.2. Virtual node-level intervener

[0135] After generating the first combination input and the second combination input, the computer can input the first combination input and the second combination input into an intervener 428. The intervener 428 can be a virtual node-level intervener 428. The virtual node-level intervener 428 can be similar to the node-level intervener 328. The difference between the virtual node-level intervener 428 and the node-level intervener 328 is that the intervention in the virtual node-level intervener 428 functions on the virtual nodes, rather than the given real nodes.

[0136] Due to the property of the transformer 400 that it can process data sets with variable size, the design of the virtual node-level intervener ϕ is similar to the node- level intervener as ^^^^௧^^, ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^^|| ^^^^௩^ or as ^^^^௧^^ൌ ^^^ ^^^^|| ^^^^௩^ if the attention matrix ^^ is not used in some embodiments. For the virtual node-level fine grained graph rationalization method, the attention matrix ^^ ∈ ℝ^ൈ^describes the interaction among the ^^ virtual nodes.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0137] Over training time, the parameters of the one or more first layers and the second layer in the machine learning model can be adjusted to maximize a self attention of the rationale graph embedding and minimize attention interactions between the rationale graph embedding and the first environment graph embedding and the second environment graph embedding.

[0138] The computer can iteratively utilize the intervener 428 to determine attention values for the first combination input 424 and the second combination input 426. The intervener 428 can determine first transformed embeddings based on the first combination input 424 (e.g., ^^^^௧^^) and can determine second transformed embeddings based on the second combination input 426. 3. Virtual node-level optimization objective

[0139] After performing the intervention phase 404, the computer can generate, using a second layer of the machine learning model (e.g., the predictor ^^^^^ௗ), a first classification output using the first transformed embeddings obtained from the first combination input. The computer can also generate a second classification output using the second transformed embeddings obtained from the second combination input.

[0140] The predictor ^^^^^ௗtakes a combination input as input and generates a task-related vector / scalar as output. The predictor can be a multi-layer perceptron. The predictor ^^^^^ௗcan take the transformer’s output (which is the transformed combination) as input.

[0141] The output of ^^^^^ௗ∘ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ∘ ^^^^^ is used for minimizing the utilityloss ^^௨௧^^ ൌ ^^௨௧^^^ ^^^^|| ^^^^௩^ ^ ^^ ^^௨௧^^^ ^^^^|| ^^^^^௩^, where ^^௨௧^^^ ^^^^|| ^^^^௩^ ൌsimilarly. Foris the virtual node embeddings from other graphs in the mini-batch. Additionally, the previously proposed regularization termequation (8) can be extended to the virtual node-level variant: ^^^^^^ ^^^^|| ^^^^௩^ ൌ^^ୃ ^^^ ^^ െ ^^^ ^ ^ ^^ െ ^^^ୃ ^^ ^^. The total regularization term, considering the changingenvironment ^^^^^௩, is ^^^^^ ൌ ^^^^^^ ^^^^|| ^^^^௩^ ^ ^^ ^^ ^^ ^^^ ^^^^|| ^^^^^௩^. As the attention matrixPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 ^^ depicts interactions among virtual nodes, a rationale / environment indicator vector s is constructed analogously to equation (9). As such, the optimization objective of the virtual node-level fine grained graph rationalization method is^^^ ^^௫ఏథ^^௨௧^^^ ^^ ^^^^^, where ^^ ൌ ^ ^^^^^, ^^^௨^ି^ே, ^^^^^ௗ^.perform a gradient decent method to optimize the loss functions to minimize ^^ and maximize ^^. 4. Complexity of virtual node-level fine grained graph rationalization

[0143] The virtual node-level fine grained graph rationalization process can include the encoder ^^^^^, the predictor ^^^^^ௗ, the virtual node-level augmenter ^^^௨^ି^ே, and the transformer-based intervener ϕ. The parameters for the virtual node-level augmenter ^^^௨^ି^ேoriginate from the matrix ^^ேି^ே, as defined in equation (12). The number of these parameters is of the order ^^^ ^^ ^^^, where ^^ denotes the number of nodes.

[0144] In some embodiments, for implementation purposes, the number of nodes ^^ can be pre-set. For example, the number of nodes ^^ can be set to 10× average size of graphs from the dataset, where the input graphs are truncated if their size is larger than 10× the average size.

[0145] For the transformer-based intervener ϕ, the training tunable parameters originate from the transformer layer, outlined in equation (6a). The number of parameters here is ^^^5 ^^ଶ^, owing to the query, key, and value projection matrices and the feed-forward net (equation (3), typically a 3-layered MLP). The computer can adjust the parameters of the one or more first layers and the second layer based on the comparison of the first classification output and the second classification output.

[0146] Further details regarding the method illustrated in FIG.4 are illustrated in a step-by-step algorithm for the virtual node-level fine grained graph rationalizationPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01method is provided in Algorithm 2, below. In a test phase, the output of ^^^^^ௗ ∘^^ ^^ ^^ ^^ ^^ ^^ ^^ ∘ ^^^ ^^^^|| ^^^^௩^ is evaluated.Algorithm 2: virtual node-level single training step for every training graph ^^D. Invariant Graph Transformer Training Overview

[0147] FIG.5 shows a flowchart of a fine-grained graph rationalization method of training according to embodiments. The method illustrated in FIG.5 can be performed by a computer such as the analysis computer 102.

[0148] Prior to step 502, the computer can obtain a graph from a database, e.g., database 104. The graph can include a plurality of nodes and a plurality of edges. ThePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 graph can include interaction data, molecular data, or other data associated with types of relations and / or processes in physical, biological, social, and informational systems. The computer can also obtain an alternate graph (e.g., a second graph, where the aforementioned graph is a first graph) that can be utilized to intervene in the first graph.

[0149] In some embodiments, the graph can correspond to a molecule. For example, each node in the graph can indicate a different part (e.g., an atom, a collection of atoms, etc.) of a molecule. Each edge in the graph (e.g., connections between nodes) can indicate chemical bonds (covalent bond, ionic bond, etc.). The graph can include a small molecule, a larger biomolecule, etc. For example, embodiments described herein can process a graph that describes a molecule. Embodiments can determine a portion of the molecule graph that is invariant. The invariant graph of the molecule graph can indicate a portion of the molecule that may not depend on the molecular parts outside of the invariant graph. Such a portion of the molecule can be determined during drug design.

[0150] In other embodiments, the graph can correspond to interaction data (e.g., transaction data). For example, each node in the graph can represent a different entity (e.g., user, resource provider, etc.) in an interaction network. Each edge in the graph can represent an interaction between users and resource providers.

[0151] The graph can comprise a plurality of nodes and a plurality of edges. A rationale graph, derived from the graph, can include a first subset of the plurality of nodes. A first environment graph, derived from the graph, can include a second subset of the plurality of nodes. The first subset of the plurality of nodes and the second subset of the plurality of nodes may not include the same nodes.

[0152] At step 502, the computer can generate a rationale graph embedding corresponding to a rationale graph and a first environment graph embedding corresponding to a first environment graph using the graph. The rationale graph and the first environment graph can be subgraphs of the graph. The rationale graph and the first environment graph can be derived from the graph.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0153] The computer can use one or more first layers of a machine learning model to generate the rationale graph embedding and the first environment graph embedding. The computer can generate the rationale graph embedding and the first environment graph embedding on a per node basis. The rationale graph embedding can have a same number of embeddings as the number of nodes in the rationale graph. The first environment graph embedding can have a same number of embeddings as the number of nodes in the first environment graph. As such, the rationale graph embedding can include a matrix of data, where each row (or column, depending on implementation) can include a node embedding for a node in the rationale graph. Similarly, the first environment graph embedding can include a matrix of data, where each row (or column, depending on implementation) can include a node embedding for a node in the first environment graph.

[0154] The computer can generate the embeddings on a per node basis such that a separate embedding is generated for each node in the graph. For example, the computer can generate the rationale graph embedding by generating an embedding for each node in the rationale graph. The rationale graph embedding can be a matrix where each row of the matrix is an embedding corresponding to a node in the rationale graph. As another example, the computer can generate the first environment graph embedding by generating an embedding for each node in the first environment graph. The first environment graph embedding can be a matrix where each row of the matrix is an embedding corresponding to a node in the first environment graph.

[0155] At step 504, after generating the rationale graph embedding and the first environment graph embedding, the computer can generate a second environment graph embedding from a second environment graph of the second graph. The second environment graph embedding can be generated on a per node basis. The second environment graph is different than the first environment graph. The second environment graph can be derived from a different graph than the first environment graph. In some embodiments, the first environment graph and the second environment graph can originate from the same graph, but may be from different time periods of the graph. For example, the first environment graph can be derived from a first timestamp ofPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 the graph, and the second environment graph can be derived from a second timestamp of the graph.

[0156] At step 506, the computer can combine the rationale graph embedding and the first environment graph embedding to form a first combination input. For example, combining the rationale graph embedding and the first environment graph embedding can include concatenating the rationale graph embedding and the first environment graph embedding.

[0157] At step 508, the computer can combine the rationale graph embedding and the second environment graph embedding to form a second combination input. For example, combining the rationale graph embedding and the second environment graph embedding can include concatenating the rationale graph embedding and the second environment graph embedding.

[0158] At step 510, after determining the first combination input and the second combination input, the computer can generate a first classification output using the first combination input. The computer can generate the first classification output using a second layer of the machine learning model. The first classification output can be a predicted classification of the first combination input. The second layer of the machine learning model can perform a classification or a regression task to determine the first classification output.

[0159] For example, in a binary classification, for the original graph the computer could attempt to classify if the graph is a first class (e.g., good, not fraudulent, etc.) or a second class (e.g., bad, fraudulent, etc.). As an illustrative example, the computer can determine a first classification output that indicates that the graph is not fraudulent (e.g., that the interactions represented by nodes in the graph are not fraudulent).

[0160] At step 512, the computer can generate a second classification output using the second combination input. The computer can generate the second classification output using the second layer of the machine learning model.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0161] The second combination input corresponds to the situation where the environment portion of the graph is modified to a random environment (e.g., via the second environment graph embedding). If the computer determines that the second classification output is the same as the first classification output, then the rationale graph of the graph is indeed a rationale part of the graph since it the resulting classification did not change when the environment graph of the graph was substituted for another environment graph.

[0162] At step 514, the computer can adjust parameters of the one or more first layers and the second layer of the machine learning model. The computer can adjust the parameters across training iterations to optimize the machine learning model. The computer can adjust the parameters based on a comparison of the first classification output and the second classification output.

[0163] The machine learning model can include a transformer sub-model that generates an attention data structure. The computer can adjust the parameters of the one or more first layers and the second layer to maximize a self attention of the rationale graph embedding and minimize interactions between the rationale graph embedding and the first environment graph embedding and the second environment graph embedding. E. Invariant Graph Transformer Production Overview

[0164] FIG.6 shows a flowchart of a fine-grained graph rationalization method of production according to embodiments. The method illustrated in FIG.6 can be performed by a computer such as the analysis computer 102.

[0165] At step 602, the computer can receive a classification request message from a network processing computer (e.g., the network processing computer 106). The classification request message can comprise a request for a classification output and a graph identifier. The graph identifier can identify a particular graph in a database (e.g., the database 104). The graph identifier can be a numeric value that indicates a location at which the graph is stored in the database.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0166] At step 604, the computer can retrieve the graph from the database using the graph identifier. The computer can query the database for the graph that is stored in association or pointed to by the graph identifier. For example, the computer can obtain the graph from the production sample 122 in the database 104. In other embodiments, the computer can perform steps 602 and 604 in response to user input or requests from other computers.

[0167] At step 606, after obtaining the graph, the computer can generate a rationale graph embedding and an environment graph embedding from the graph on a per node basis. The rationale graph embedding can correspond to a rationale graph, which is a subgraph of the graph. The environment graph embedding can correspond to an environment graph, which is a subgraph of the graph.

[0168] For example, the computer can generate the rationale graph embedding and the environment graph embedding using an encoder (e.g., the encoder 310 or the encoder 410) and the graph. The encoder can accept graphs as input and can generate a node embedding matrix as output. The node embedding matrix can include a plurality of rows, where each row corresponds to and is an embedding of a node in the graph.

[0169] After generating the node embedding matrix for the graph, the computer can determine the rationale graph embedding and the environment graph embedding from the node embedding matrix using an augmenter (e.g., the augmenter 314 or the augmenter 414). For example, the augmenter (e.g., the augmenter 314) can include a machine learning model (e.g., a multi-layer perceptron) that is trained to generate a partition vector that is used to partition the node embeddings of the node embedding matrix into the rationale graph embedding and the environment graph embedding.

[0170] At step 608, after generating the rationale graph embedding and the environment graph embedding, the computer can concatenate the rationale graph embedding and the environment graph embedding to form a combination input.

[0171] At step 610, the computer can generate a classification output using the combination input. The computer can generate the classification output using a second layer of the machine learning model. The machine learning model can include aPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 transformer sub-model that generates an attention data structure. The parameters of the one or more first layers and the second layer were trained (1) to maximize a self- attention of the rationale graph embedding and (2) to minimize interactions between the rationale graph embedding and the environment graph embedding.

[0172] The classification output can be a classification of the graph. For example, the graph can include nodes representing users or resource providers and edges representing access requests. The classification output can be a classification of 1) a fraudulent network vs a not fraudulent network, 2) a high usage network vs a low usage network, 3) a risk score, or other classes that can relate to the users, resource providers, and / or interactions between them.

[0173] At step 612, after generating the classification output, the computer can generate a classification response message in response to the classification request message. The classification response message can include the classification output. In some embodiments, the classification response message can include the graph identifier and / or the graph.

[0174] At step 614, the computer can provide the classification response message to the network processing computer for further processing. The network processing computer can perform any suitable further processing based on the classification output. For example, if the classification output indicates that a particular graph is a fraudulent network, then the network processing computer 106 can perform fraud prevention methods. IV. EXPERIMENTS

[0175] In this section, experimental datasets and baseline methods are discussed. Subsequent subsections evaluate the comparative effectiveness of embodiments, supplemented by an ablation study and visualization of the attention matrix. Further, an efficiency study, a convergence analysis, and a sensitivity study can be found below.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 A. Datasets

[0176] 7 publicly-available real-world datasets were selected for evaluation: 1) graph classification datasets molhiv (Hu et al., 2020), moltox21 (Hu et al., 2020), molbace (Hu et al., 2020), molbbbp (Hu et al., 2020) and 2) graph regression datasets ZINC (Dwivedi et al., 2023), AQSOL (Dwivedi et al., 2023), and mollipo (Hu et al., 2020). The metrics and dataset split recommended by the given benchmarks were followed. To be concrete, area under the ROC curve (ROC-AUC) is the metric for datasets molhiv, moltox21, molbace, molbbbp; root-mean-square deviation (RMSE) is the metric for dataset mollipo; mean absolute error (MAE) is the metric of datasets ZINC and AQSOL. The details statistics of the datasets are given in Table 3, below. The average result with standard deviation in 10 runs is reported. B. Baseline Methods

[0177] The selected baseline methods include: 1) 4 graph neural network baselines: GIN (Xu et al., 2019), GAT (Velickovic et al., 2018), GATv2 (Brody et al., 2022), and GatedGCN (Bresson & Laurent, 2017) 2) 7 graph Transformers: GT (Dwivedi & Bresson, 2020), GraphiT (Mialon et al., 2021), SAN (Kreuzer et al., 2021), SAT (Chen et al., 2022), Graphormer (Ying et al., 2021), GraphTrans (Wu et al., 2022b), GPS (Rampasek et al., 2022), and 3) 2 graph rationale discovery methods: DIR (Wu et al., 2022a) and GREA (Liu et al., 2022). C. Effectiveness Study

[0178] The effectiveness comparison between the proposed IGT-N, IGT-VN, and baseline methods are provided in Table 1. To ensure a fair comparison, certain pre- trained models, such as the pre-trained Graphormer (Ying et al., 2021), are not included. Also, as DIR (Wu et al., 2022a) is designed to conduct interventions on the label prediction vectors, it cannot be naively applied to graph regression tasks. Notably, the methods according to embodiments, IGT-N and IGT-VN, consistently outperform, or are at least on par with, all the baseline methods on both the graph classification and graph regression datasets.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 D. Ablation Study

[0179] An ablation study was conducted on methods according to embodiments, IGT-N and IGT-VN. Two ablated variants were selected as baselines: (1) ^^^^^◦ ^^^^^ௗ, which was a pure composition of the encoder ^^^^^and the predictor ^^^^^ௗwithout any rationale discovery module; and (2) IGT-N without reg and IGT-VN without reg, which remove the regularization term (equation (10)) from the total objective loss function. The results highlighted that both the transformer-based intervener and the regularization term (equation (10)) substantially enhance the performance of IGT-N and IGT-VN , as shown in Table 2.

[0180] Table 1a and 1b, below, illustrate an effectiveness comparison (mean±std) with baseline methods. (↓) denotes the lower the better and (↑) denotes the higher the better. Statistics with an asterisk (*) are reported in the original papers. The best is bold and the second best is italics. N / A means the method cannot work on regression tasks. Graph Regression )PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 Graphormer 0.122±0.006* 1.265±0.025 0.911±0.015Graph Classification )PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 GPS 78.8±1.0* 75.7±0.4 79.6±1.4 69.6±1.1 *

[0181] Table 2, below, illustrates an ablation study (mean±std) of embodiments. (↓) denotes the lower the better and (↑) denotes the higher the better. Dataset Metric mollipo molbace molbbbpE. Attention Visualization

[0182] The significance of the regularization term is evaluated by visualizing the adjacency matrix ^^, effectively the attention matrix, of the intervener ϕ in FIG.7. FIG.7 shows a heatmap of an adjacency matrix of an intervener according to embodiments. FIG.7 includes a first adjacency matrix 702 that does not include a regularization term and a second adjacency matrix 704 that does include a regularization term.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0183] The IGT-VN embodiments was selected for visualization. Unlike IGT-N, which works on a variable number of nodes (from different graphs), IGT-VN maps nodes to a predetermined number of virtual nodes, simplifying the presentation in FIG. 7. Specifically, the number of virtual nodes was set to be ^^ ൌ 16 with ^^ ൌ 10, designating 10 virtual nodes to rationales and the remainder as environments. All the visualization results were obtained from the molbace dataset. The attention matrix ^^ was normalized row-wise by Softmax. From the observations, the following two insights are highlighted.

[0184] Even in the absence of the regularization term, in the first adjacency matrix 702, interactions between rationales and environments appear significantly weaker than those within the rationales themselves. One potential explanation is the changing nature of the environment. In optimizing the utility loss ^^௨௧^^ൌ^^௧^^^^ ^^^^|| ^^^^௩^ ^ ^^ ^^௧^^^^ ^^^^|| ^^^^^௩^, the everchanging environment ( ^^^^^௩) might leadthe model to minimize interactions between rationales and environments so that the utility of the rationale can be ensured.

[0185] The first observation supports the decision to introduce the regularization loss term, which aims to penalize rationale-environment interactions. When the proposed regularization term (e.g., in equation (8)) is implemented, in the second adjacency matrix 704, there is a noticeable decline in rationale-environment interactions (the off-diagonal blocks in the adjacency matrices in FIG.7). As demonstrated in the ablation study, this leads to the enhanced model performance. F. Dataset Statistics

[0186] This section describes the statistics of the datasets that are presented in Table 3. Name # # Nodes # Edges # # Split MetricPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 ZINC 12000 23.2 49.8 21 N / A 10000 / MAE (node), 1000 / PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 1. Hardware and implementations

[0187] The node-level fine-grained graph rationalization method and the virtual node-level fine-grained graph rationalization method were implemented, along with the baseline methods, in PyTorch and PyTorch-geometric. The efficiency study results were from one NVIDIA Tesla V100 SXM2-32GB GPU on a server with 96 Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz processors and 1.5T RAM. When baseline methods have pre-existing results for specific datasets, those statistics were directly used. In cases where such results were absent, the models were implemented based on either the available code or details described in the associated publications.

[0188] Shared implementations among embodiments and baseline methods will be introduced. A batch size was set as 32 and the weight decay was set as 0. The hidden dimension was set as 300. The dropout rate for both the encoder and the intervener was searched between {0, 0.5}. The pooling was searched between {mean, sum}. The normalization was searched between a batch normalization (Ioffe & Szegedy, 2015) and layer normalization (Ba et al., 2016). The learning rate was initialized as 0.0001 and decayed by a factor 14 if the validation performance was not improved in 10 epochs. Next, the implementation of embodiments and baseline methods is discussed. 2. Implementation of embodiments

[0189] The encoder was set as GPS (Rampasek et al., 2022) on ZINC, AQSOL, mollipo, molhiv, molbace, and set as GraphTrans (Wu et al., 2022b) on moltox21 and molbbbp. For the predictor module, the design was implemented as a 3-layered MLP with ReLU activation in the intermediate layers. In the implementation of embodiments,β was set as ^^ ൌ ൫2 ൈ ^^^ ൯ൗ ൫ ^^ ൈ ^ ^^ െ 1^൯ (for the node-level fine-grained graphrationalization method) or as ^^ ൌ ൫2 ൈ ^^^ ൯ൗ ൫ ^^ ൈ^^^ െ 1^൯ (for the virtual node-level fine-grained graph rationalization method). The α and ^^^ were searched between [0.2, 2],step size 0.2. In the implementation of embodiments, the K was set as ^^ ൌ ^^ ^^ ^^ ^^ ^^^ ^^^ ൈ ^^^ (for the node-level fine-grained graph rationalization method) or ^^ ൌPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 ^^ ^^ ^^ ^^ ^^^ ^^^ ൈ ^^^ (for the virtual node-level fine-grained graph rationalization method). ^^ was searched between {8, 16, 32} for the virtual node-level fine-grained graph rationalization method. A detailed sensitivity study was created to explore the bestselection of ^^^, below, which shows the best ^^^ is around 0.75.3. Implementation of baseline methods

[0190] The number of layers was searched of GIN (Xu et al., 2019), GAT (Velickovic et al., 2018), GATv2 (Brody et al., 2022), GatedGCN (Bresson & Laurent, 2017), DIR (Wu et al., 2022a), and GREA (Liu et al., 2022) between {2, 3, 5, 10} and report the best performance, considering configurations both with and without a virtual node connecting to all the given nodes.

[0191] Regarding the Transformer-based baselines (GT (Dwivedi & Bresson, 2020), GraphiT (Mialon et al., 2021), SAN (Kreuzer et al., 2021), SAT (Chen et al., 2022), Graphormer (Ying et al., 2021), GraphTrans (Wu et al., 2022b), GPS (Ramp´asek et al., 2022)), for the (absolute or relative) positional encoding, the tests adhered to the suggestions made in their original papers. The number of layers was also searched between {2, 3, 5, 10}.

[0192] The implementations of GIN, GAT, GATv2, and GatedGCN were from the PyTorch-geometric4 package. The implementations of GT 5, GraphiT 6, SAN 7, SAT 8, Graphormer 9, GraphTrans 10, GPS 11, DIR 12, GREA 13 were adapted from publicly available code. G. Number of Parameters

[0193] A detailed number of parameter comparison is presented in Table 4, where 4 typical 5-layered encoders are listed (GIN, SAT, GraphTrans, and GPS), and the herein described node- / virtual node-level augmenter, intervener modules (e.g., ^ ^^^௨^ିே, ^^^ and ^ ^^^௨^ି^ே, ^^^). Advantageously, the interveners according to embodiments do not significantly increase the size of the whole model.PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01

[0194] Table 4, below, illustrates a number of parameter comparison between the proposed augmenter, intervener, and common graph encoders. Model # Parameters 7 9 7 9 1 0H. Efficiency Study

[0195] Model efficiency (training iterations / second) of embodiments was compared in Table 5 working with different encoders (GIN and GPS). The batch size was set as 32. The inclusion of the proposed augmenter and intervener, represented as ^ ^^^௨^ିே, ^^^ or ^ ^^^௨^ି^ே, ^^^, introduces a slight reduction in training speed. That is because the proposed parametric augmenter and intervener increase the steps of the data pipeline, and enlarge the computational graph for auto-gradient tools, such as PyTorch. Advantageously, the parameter count of the parametric augmenter and intervener is low, ensuring that the overall training speed of the model is not dramatically affected. A detailed comparison about the number of parameters of the newly-introduced node- / virtual node-level augmenter ( ^^^௨^ିே / ^^^௨^ି^ே), intervener (ϕ) modules and other typical encoder modules ( ^^^^^) are presented in Table 4.

[0196] Table 5, below, illustrates wall-clock time (iterations / second) comparison of different encoder-intervener combinations. The larger value the faster thePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 performance. (↓) denotes the speed degradation compared with the vanilla encoder without intervener. Encoder Intervener mollipo molbace molbbbpI. Training Convergence

[0197] A question of concern could be about the impact of the min-max objective on the training stability of IGT-N and IGT-VN. The training losses were monitored of both IGT-N and IGT-VN across three datasets (molbace, molbbbp, mollipo) using two encoders (GIN and GPS). FIG.8 shows graphs illustrating training stability of machine learning models according to embodiments. FIG.8 includes graphs illustrating training stability of IGT-N and IGT-VN including a first graph 802, a second graph 804, a third graph 806, a fourth graph 808, a fifth graph 810, and a sixth graph 812. The first graph 802 shows training stability of IGT-N and IGT-VN using the molbace training set and the GIN encoder. The second graph 804 shows training stability of IGT-N and IGT-VN using the molbace training set and the GPS encoder. The third graph 806 shows training stability of IGT-N and IGT-VN using the molbbbp training set and the GIN encoder. The fourth graph 808 shows training stability of IGT-N and IGT-VN using the molbbbp training set and the GPS encoder. The fifth graph 810 shows training stability of IGT-N and IGT-VN using the mollipo training set and the GIN encoder. The sixth graph 812PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 shows training stability of IGT-N and IGT-VN using the mollipo training set and the GPS encoder.

[0198] The results, presented in FIG.8, demonstrate that the training remains stable even when θ (representing the encoder, augmenter, and predictor) and ϕ (representing the intervener) engage in a min-max game. J. Sensitivity Study

[0199] In this section, the impact of hyperparameter K is evaluated (from equations (5a), (5b), (13a), and (13b)). The hyperparameter K determines the ratio of the rationale and environment subgraphs. In the implementation, K was set to ^^ ൌ^^ ^^ ^^ ^^ ^^^ ^^^ ൈ ^^^ (for the node-level fine-grained graph rationalization method) or ^^ ൌ^^ ^^ ^^ ^^ ^^^ ^^^ ൈ ^^^ (for the virtual node-level fine-grained graph rationalization method). Model performance was evaluated across varying ^^^ on the molbace and molbbbp datasets, as illustrated in FIG.9. FIG.9 includes a first graph 902 and a second graph 904. The first graph 902 illustrates performance of embodiments using the molbace dataset. The second graph 904 illustrates performance of embodiments using the molbbbp dataset. It is noted that if most nodes / virtual nodes were marked as the environment component, the model performance degraded. Similar performance degradation was observed if too many nodes / virtual nodes are marked as the rationalenodes (e.g., ^^^ = 0.875). That is because for a large ^^^ (e.g., ^^^ = 1), the modeldegenerated to a vanilla graph encoder, with less intervention involved. When ^^^ was set as 0.75 or 0.675, the best performance was observed. V. RELATED WORK

[0200] Various works related to invariant learning on graphs and transformers on graphs will be described. A. Invariant Learning on Graphs

[0201] Invariant feature learning, which has seen extensive applications in many fields such as computer vision (Kaddour et al., 2022; Arjovsky et al., 2019; Ganin &PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 Lempitsky, 2015; Chang et al., 2020), is gaining more attention in the community of graph machine learning. OOD-GNN (Li et al., 2023), for instance, applies random Fourier features to eliminate the statistical dependence between relevant and irrelevant graph representations. (Bevilacqua et al., 2021) studies the graph invariant representation learning with a particular focus on the size discrepancies between the training / test graphs. DIR (Wu et al., 2022a) decomposes the given graph into a rationale subgraph and an environment subgraph; after that, it uses a graph classifier to conduct prediction on the above two graphs respectively and its intervention is via the Hadamard product between the prediction vectors. Similarly, GREA (Liu et al., 2022) conducts the rationale / environment decomposition at the node level and its intervention operation is to directly add the environment graph embedding into the rationale graph embedding. In a similar vein, GIL (Li et al., 2022b) decomposes the given graph into the invariant and variant subgraphs; based on that, it infers the environment label, as input of the invariance regularization term (Koyama & Yamaguchi, 2020). Furthermore, invariant learning can also benefit graph contrastive learning (Li et al., 2022c) to enhance the data augmentation process. B. Transformer on Graphs

[0202] The transformer architecture has achieved significant success in various domains, including natural language processing (Vaswani et al., 2017), computer vision (Han et al., 2023), and more (Xu et al., 2022). In recent years, there has been a surge of interest in enhancing graph machine learning methods by leveraging the capabilities of transformers. For example, GraphTrans (Wu et al., 2022b) concatenates the Transformer architecture after various message-passing neural networks; GPS (Rampasek et al., 2022) proposes a powerful layer, which operates both a graph convolution layer and a transformer layer in parallel; GT (Dwivedi & Bresson, 2020) generalizes the attention module on graphs via concatenating the spectral vectors with the raw node features and computing the attention weights on all the existing edges; Graphormer (Ying et al., 2021) encodes both the node centrality and node-pair shortest path into the transformer architecture. A comprehensive survey about graph transformer, from the architecture perspective, can be found in (Min et al., 2022).PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 VI. CONCLUSION

[0203] The invariant rationale discovery problem on graph data is solved by the embodiments described herein. Distinct from existing methods, embodiments (e.g., IGT- N and IGT-VN) are designed at more fine-grained levels, specifically at the node and virtual node levels, so that they can better model the interactions between the rationale and environment subgraphs. The intervener and the other model modules are formulated in a min-max game, which can significantly improve the quality of the extracted graph rationales. Comprehensive experiments on 7 real-world datasets illustrated the effectiveness of the proposed method compared to 13 baseline methods. The evaluation of the efficiency of the proposed augmenter and intervener shows that they did not considerably impact the overall training efficiency.

[0204] Embodiments of the disclosure have a number of additional advantages. For example, embodiments provide for methods for determining whether or not a portion of a graph is an invariant portion of the graph using node level embeddings rather than graph level embeddings, thus providing higher accuracy in the determined rationale portion.

[0205] Any of the computer systems mentioned herein may utilize any suitable number of subsystems. Examples of such subsystems are shown in FIG.10 in computer system 10. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be the components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components. A computer system can include desktop and laptop computers, tablets, mobile phones and other mobile devices.

[0206] The subsystems shown in FIG.10 are interconnected via a system bus 75. Additional subsystems such as a printer 74, keyboard 78, storage device(s) 79, monitor 76 (e.g., a display screen, such as an LED), which is coupled to display adapter 82, and others are shown. Peripherals and input / output (I / O) devices, which couple to I / O controller 71, can be connected to the computer system by any number of meansPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 known in the art such as input / output (I / O) port 77 (e.g., USB, FireWire®). For example, I / O port 77 or external interface 81 (e.g., Ethernet, Wi-Fi, etc.) can be used to connect computer system 10 to a wide area network such as the Internet, a mouse input device, or a scanner. The interconnection via system bus 75 allows the central processor 73 to communicate with each subsystem and to control the execution of a plurality of instructions from system memory 72 or the storage device(s) 79 (e.g., a fixed disk, such as a hard drive, or optical disk), as well as the exchange of information between subsystems. The system memory 72 and / or the storage device(s) 79 may embody a computer readable medium. Another subsystem is a data collection device 85, such as a camera, microphone, accelerometer, and the like. Any of the data mentioned herein can be output from one component to another component and can be output to the user.

[0207] A computer system can include a plurality of the same components or subsystems, e.g., connected together by external interface 81, by an internal interface, or via removable storage devices that can be connected and removed from one component to another component. In some embodiments, computer systems, subsystem, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components. In various embodiments, methods may involve various numbers of clients and / or servers, including at least 10, 20, 50, 100, 200, 500, 1,000, or 10,000 devices. Methods can include various numbers of communication messages between devices, including at least 100, 200, 500, 1,000, 10,000, 50,000, 100,000, 500,00, or one million communication messages. Such communications can involve at least 1 MB, 10 MB, 100 MB, 1 GB, 10 GB, or 100 GB of data.

[0208] Aspects of embodiments can be implemented in the form of control logic using hardware circuitry (e.g., an application specific integrated circuit or field programmable gate array) and / or using computer software stored in a memory with a generally programmable processor in a modular or integrated manner, and thus a processor can include memory storing software instructions that configure hardwarePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 circuitry, as well as an FPGA with configuration instructions or an ASIC. As used herein, a processor can include a single-core processor, multi-core processor on a same integrated chip, or multiple processing units on a single circuit board or networked, as well as dedicated hardware. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and / or methods to implement embodiments of the present disclosure using hardware and a combination of hardware and software.

[0209] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as, for example, Java, C, C++, C#, Objective-C, Swift, or scripting language such as Perl or Python using, for example, conventional or object- oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission. A suitable non-transitory computer readable medium can include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk) or Blu- ray disk, flash memory, and the like. The computer readable medium may be any combination of such devices. In addition, the order of operations may be re-arranged. A process can be terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0210] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable medium may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device (e.g., as firmware) or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer productPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 (e.g., a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0211] Any of the methods described herein may be totally or partially performed with a computer system including one or more processors, which can be configured to perform the steps. Any operations performed with a processor may be performed in real-time. The term “real-time” may refer to computing operations or processes that are completed within a certain time constraint. As examples, a time constraint may be 30 seconds, 1 minute, 10 minutes, 30 minutes, 1 hour, 4 hours, 1 day, or 7 days. Thus, embodiments can be directed to computer systems configured to perform the steps of any of the methods described herein, potentially with different components performing a respective step or a respective group of steps. Although presented as numbered steps, steps of methods herein can be performed at a same time or at different times or in a different order. Additionally, portions of these steps may be used with portions of other steps from other methods. Also, all or portions of a step may be optional. Additionally, any of the steps of any of the methods can be performed with modules, units, circuits, or other means of a system for performing these steps.

[0212] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the disclosure. However, other embodiments of the disclosure may be directed to specific embodiments relating to each individual aspect, or specific combinations of these individual aspects.

[0213] The above description of example embodiments of the present disclosure has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form described, and many modifications and variations are possible in light of the teaching above.

[0214] A recitation of "a", "an" or "the" is intended to mean "one or more" unless specifically indicated to the contrary. The use of “or” is intended to mean an “inclusivePATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 or,” and not an “exclusive or” unless specifically indicated to the contrary. Reference to a “first” component does not necessarily require that a second component be provided. Moreover, reference to a “first” or a “second” component does not limit the referenced component to a particular location unless expressly stated. The term “based on” is intended to mean “based at least in part on.”

[0215] The claims may be drafted to exclude any element which may be optional. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely”, “only”, and the like in connection with the recitation of claim elements, or the use of a “negative” limitation.

[0216] All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. None is admitted as prior art. Where a conflict exists between the instant application and a reference provided herein, the instant application shall dominate.

Claims

PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 WHAT IS CLAIMED IS:

1. A method of training comprising performing by a computer system: generating, using one or more first layers of a machine learning model, a rationale graph embedding corresponding to a rationale graph and a first environment graph embedding corresponding to a first environment graph on a per node basis using a graph; generating a second environment graph embedding from a second environment graph, wherein the second environment graph embedding is generated on a per node basis; concatenating the rationale graph embedding and the first environment graph embedding to form a first combination input; concatenating the rationale graph embedding and the second environment graph embedding to form a second combination input; generating, using a second layer of the machine learning model, a first classification output using the first combination input; generating, using the second layer of the machine learning model, a second classification output using second combination input; and adjusting parameters of the one or more first layers and the second layer based on a comparison of the first classification output and the second classification output, wherein the machine learning model includes a transformer sub-model that generates an attention data structure, and wherein the parameters of the one or more first layers and the second layer are further adjusted (1) to maximize a self attention of the rationale graph embedding in the attention data structure and (2) to minimize interactions between the rationale graph embedding and the first environment graph embedding and the second environment graph embedding in the attention data structure.

2. The method of claim 1, wherein generating the rationale graph embedding and the first environment graph embedding comprises: generating a node embedding matrix based on the graph;PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 generating, using a learnable node partitioner machine learning model, a partition vector; determining the rationale graph embedding based on the node embedding matrix and the partition vector; and determining the first environment graph embedding based on the node embedding matrix and the partition vector.

3. The method of claim 1, wherein generating the rationale graph embedding and the first environment graph embedding comprises: generating a node embedding matrix based on the graph; generating a virtual node embedding matrix on a per node basis based on the node embedding matrix using a weight matrix and a softmax function, wherein the weight matrix is a parameter of the parameters of the one or more first layers; determining the rationale graph embedding based on the virtual node embedding matrix; and determining the first environment graph embedding based on the virtual node embedding matrix.

4. The method of claim 3, wherein the rationale graph embedding includes node embeddings from a first K number of rows of the virtual node embedding matrix, wherein K is a hyperparameter in the machine learning model.

5. The method of claim 3, wherein a dimensionality of the virtual node embedding matrix is smaller than a dimensionality of the node embedding matrix.

6. The method of claim 1 further comprising: transforming, using the transformer sub-model, the first combination input into first transformed embeddings, wherein generating the first classification output uses the first transformed embeddings.

7. The method of claim 1 further comprising:PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 transforming, using the transformer sub-model, the second combination input into second transformed embeddings, wherein generating the second classification output uses the second transformed embeddings.

8. The method of claim 1, wherein the parameters are further adjusted based on optimizing a total objective loss function that includes a utility loss term and a regularization loss term, wherein the utility loss term is based on the first classification output, the first combination input, the second classification output, and the second combination input, wherein the regularization loss term is based on the attention data structure.

9. The method of claim 1, wherein the rationale graph embedding comprises a first plurality of embeddings, wherein each embedding corresponds to a node of the graph, wherein the first environment graph embedding comprising a second plurality of embedding, wherein each embedding corresponds to a node of the graph.

10. The method of claim 1 further comprising: obtaining the graph from a database, wherein the database receives node data and edge data from a network processing computer in an interaction network.

11. The method of claim 1, wherein each node of the graph represents a user or a resource provider, wherein each edge of the graph represents interactions performed between the users and the resource providers.

12. The method of claim 1, wherein the first environment graph is derived from the graph, wherein the graph is a first graph, and wherein the second environment graph is derived from a second graph.

13. The method of claim 1, wherein the first environment graph is derived from a first timestamp of the graph, and wherein the second environment graph is derived from a second timestamp of the graph.

14. A production method comprising performing by a computer system:PATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 generating, using one or more first layers of a machine learning model, a rationale graph embedding corresponding to a rationale graph and an environment graph embedding corresponding to an environment graph on a per node basis using a graph; concatenating the rationale graph embedding and the environment graph embedding to form a combination input; and generating, using a second layer of the machine learning model, a classification output using the combination input, wherein the machine learning model includes a transformer sub-model that generates an attention data structure, and wherein parameters of the one or more first layers and the second layer were trained (1) to maximize a self-attention of the rationale graph embedding in the attention data structure and (2) to minimize interactions between the rationale graph embedding and the environment graph embedding in the attention data structure.

15. The method of claim 14, wherein generating the rationale graph embedding and the environment graph embedding comprises: generating the rationale graph embedding based on each node of the rationale graph; and generating the environment graph embedding based on each node of the environment graph.

16. The method of claim 14 further comprising: receiving a classification request message comprising a request for a classification output and a graph identifier from a network processing computer; retrieving the graph from a database using the graph identifier; generating a classification response message comprising the classification output; and providing the classification response message to the network processing computer.

17. A computer system comprising: a processor; andPATENT Attorney Docket No.: 079900-1450592 Client Reference No.: 7737WO01 a computer-readable medium coupled to the processor, the computer- readable medium comprising code executable by the processor for implementing the method of any one of claims 1-16.

18. The computer system of claim 17, wherein generating the rationale graph embedding and the first environment graph embedding comprises: generating a node embedding matrix based on the graph; generating, using a learnable node partitioner machine learning model, a partition vector; determining the rationale graph embedding based on the node embedding matrix and the partition vector; and determining the first environment graph embedding based on the node embedding matrix and the partition vector.

19. The computer system of claim 17, wherein generating the rationale graph embedding and the first environment graph embedding comprises: generating a node embedding matrix based on the graph; generating a virtual node embedding matrix on a per node basis based on the node embedding matrix using a weight matrix and a softmax function, wherein the weight matrix is a parameter of the parameters of the one or more first layers; determining the rationale graph embedding based on the virtual node embedding matrix, wherein the rationale graph embedding includes node embeddings from a first K number of rows of the virtual node embedding matrix, wherein K is a hyperparameter in the machine learning model; and determining the first environment graph embedding based on the virtual node embedding matrix.

20. The computer system of claim 17, wherein the graph comprises a plurality of nodes and a plurality of edges, wherein the rationale graph includes a first subset of the plurality of nodes and the first environment graph includes a second subset of the plurality of nodes, wherein the first subset of the plurality of nodes and the second subset of the plurality of nodes do not include the same nodes.

Images