A layered explanation method based on a fully unfolded material graph neural network

Abstract

The application realizes a layered explanation method of a material graph neural network based on sufficient unfolding. Through five steps: for the complex learning task of molecular property prediction, a mainstream graph neural network model is reproduced to obtain a prediction result; for each layer structure, a feature extractor of a coarse structure is initialized to converge an encoding representation of a GNN to be explained and a structure composed of the GNN; for each layer graph structure, a weight distribution module for calculating the importance of a clustering unit is initialized; based on related concepts of representation learning, all data are trained through cyclic iteration; the trained coarse module and the weight distribution module are combined to generate a hierarchical importance ranking of an input graph structure, and finally, it is verified whether the prediction process of the GNN can reconstruct domain knowledge. The method provided by the application proposes an improved optimization framework to solve a target function, finally establishes a general process for layered explanation of the GNN and adopts an effective optimization algorithm, so as to further help users to analyze and understand the learning result of the graph neural network in multiple granularities.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and more specifically to a hierarchical interpretation method based on a fully expanded material graph neural network. Background Technology

[0002] Deep learning technology is revolutionizing predictive analytics in big data research and has already achieved significant breakthroughs in image and speech recognition, question answering, machine translation, and many other applications. Typically, deep learning methods involve training input data through multi-layered neural network structures, such as feedforward neural networks, convolutional neural networks (CNNs), and recurrent neural networks (RNNs), through multiple iterations to ultimately obtain a learning model.

[0003] In recent years, with the continuous development of machine learning, Graph Neural Networks (GNNs) have become a very promising technology. Their successful applications in various fields, such as chemistry, materials science, and finance, have made GNNs a popular tool in the machine learning toolbox. However, like many deep learning models, GNNs also face a serious problem: the "black box problem."

[0004] The "black box problem" refers to the difficulty in explaining and understanding the internal workings of machine learning models. Despite their excellent performance on many tasks, users and researchers often struggle to understand how they make specific predictions or decisions. This opacity can lead to trust issues, particularly in critical fields such as chemistry, materials science, and finance, where we need to ensure that model predictions are reliable, trustworthy, and free from potential bias. Therefore, interpreting GNNs has become a crucial topic in the field of explainable AI.

[0005] Fortunately, in recent years, the number of publications on GNN explanations has increased dramatically, providing new ideas and methods for solving the "black box problem." These methods offer explanations of GNNs at different levels, helping to build trust in the model while ensuring fairness across various applications.

[0006] Some methods focus on interpreting predictions for individual instances, i.e., interpretations specific to a particular data point. One of the most well-known methods is GNNExplainer, which analyzes the model's internal structure to reveal which features and connections the model relies on when making predictions. This instance-level interpretation is extremely helpful in understanding how the model makes decisions in specific situations.

[0007] On the other hand, some methods focus on explaining the behavior of the entire model, i.e., model-level interpretation. For example, XGNN is a model-level interpretation method that can reveal the overall characteristics of GNNs, including the patterns and regularities they learn in graph data. This helps us better understand the overall performance and potential limitations of the model.

[0008] Despite these methods' significant progress in interpreting GNNs, a persistent issue remains: they largely ignore the hierarchical clustering structure within the input graph data. In practical applications, much graph data exhibits a hierarchical structure, which is often naturally generated within a scientific or social context and is crucial for the graph's characteristics and application-level properties.

[0009] For example, a common class of compounds in materials science is called metal-organic frameworks (MOFs). These compounds are complex molecular diagrams recursively constructed from various secondary building units (SBUs). The catalytic performance of MOFs is influenced by the local atomic structure of the underlying SBUs and the connectivity between higher-level SBUs. Therefore, to fully understand the performance of MOFs, information at multiple levels needs to be considered, including the interactions between local and higher-level structures.

[0010] To address this issue, this research aims to interpret GNN models at multiple levels consistent with the hierarchical structure of the input graph. This means we need to understand not only how GNNs make decisions at the lower-level structure but also how they propagate information and influence predictions across higher-level structures. This is crucial for a deeper understanding of how the model works. While some methods, such as MotifExplainer, have focused on extracting influential high-level clusters from graph data, they neglect the interpretation of structural relationships between high-level clusters. Furthermore, traditional GNN interpretation methods often allow for subgraph interpretation based on prior knowledge of the graph, achieving multi-level interpretation. However, this approach does not adequately capture the importance of group-based features across multiple graph levels, which limits the generation and optimization of candidate subgraphs.

[0011] Generally, we represent the input graph of a GNN as G = (v, ε), where v is the set of nodes and ε is the set of edges, and its adjacency matrix is ​​represented as A ∈ 0, 1. |v|×|v| Where Aij = 1 indicates that from node v i to node v j An edge exists; otherwise, Aij = 0. The node characteristic matrix is ​​used... In other words, without loss of generality, we consider a GNN model f used for a graph classification task. This classifier uses a conditional distribution... We learn the output distribution for each input graph, where Y represents the output variable in the class set 1, ..., C. Inside the GNN model, the key is to learn the final node embeddings through convolutional layers, collectively represented as Z = f emb (G, X), where Zi is the number of nodes v i The embeddings of these nodes are read for downstream classification tasks and then classified using a multilayer perceptron (MLP). The readings and MLP layers are defined together as... Please note that f(·) = f cls (f emb (·)).

[0012] The mainstream GNN interpretation method extracts a subgraph G from each input graph. s ∈G, and a set of node features Xs∈X. To ensure that the subgraph contains the key features that the GNN model depends on, a work called GNNExplainer introduced a representative objective function, which has been widely adopted:

[0013]

[0014] in This represents the mutual information between the explanatory subgraph and the outcome variable. This represents the regularization term that imposes a sparse constraint on the subgraph interpretation. Mutual information can be viewed as a correlation score between the extracted subgraph and the result, indicating the importance of the subgraph features.

[0015] This invention addresses the technical problem of overcoming the shortcomings of existing technologies by providing a hierarchical factor visualization analysis method and apparatus based on deep neural networks with text stream input. Through the display and analysis of hierarchical factors, the relationship between input text data and model prediction results can be shown, and further, factors can be expanded level by level by adding important keywords. While existing methods have established general processes and employed effective optimization algorithms for GNN interpretation, these methods, based on the classical objective function of mutual information, are only defined at the input layer level, neglecting the crucial hierarchical graph structure in our work. Furthermore, the optimization frameworks used in existing methods depend on individual features embedded by the GNN interpretation model, which can be optimized together to achieve the objective. To overcome these shortcomings, this paper provides a hierarchical interpretation method based on a fully expanded material graph neural network, formally defining the objective function for multi-level GNN interpretation, compatible with widely accepted mutual information metrics. To establish connections between high-level and input-level graph features, we introduce the novel concept of full expansion. In addition, we propose an improved optimization framework to address the objective function; by adding additional graph coarsening and attribute modules, we can appropriately represent, aggregate, and optimize high-level graph features. Ultimately, a general process for hierarchical interpretation of GNNs was established, and effective optimization algorithms were adopted to further help users analyze and understand the results of graph neural network learning at multiple granularities. Summary of the Invention

[0016] To this end, this invention first proposes a hierarchical interpretation method based on a fully expanded material graph neural network, named STFExplainer (Stratified GNN Explanations through Sufficient Expansion), which includes five steps:

[0017] Step 1: Prepare a complex learning task for predicting molecular properties with hierarchical interpretability. Encode the hierarchical domain knowledge into a hierarchical knowledge tree to represent the hierarchical interpretation constraints of the input graph data. Reproduce the mainstream graph neural network model, train the model using the training set and validation set, and then use the trained model to make predictions on the test set to obtain the prediction results.

[0018] Step 2: Initialize a coarsening feature extractor for each layer to extract the coarsening representation of each layer. The coarsening module is a deep neural network that aggregates the encoding representations of the constituent structures of the GNN to be interpreted, and learns the features that represent the entire cluster.

[0019] Step 3: Initialize a weight allocation module for each layer of graph structure to calculate the importance of clustering units, and generate candidate hierarchical interpretable subgraphs through subgraph sampling. Subgraph sampling involves reparameterization techniques. By relaxing the subgraph search NP problem, it is transformed into a sampling distribution estimation problem. A temperature parameter is introduced to control the degree of approximation of discrete sampling.

[0020] Step 4: Based on the relevant concepts of representation learning, a fully unfolded method is designed to map the interpretable subgraph of the coarsened structure to the complete component features of the input subgraph and input them into the GNN to be interpreted to calculate the predicted changes. Then, gradient descent is used to optimize the coarsening module and weight allocation module of each layer structure. The training is carried out iteratively on all data until the results converge.

[0021] Step 5: Combining the coarsening module and weight allocation module trained in Steps 2 and 3, generate a hierarchical importance ranking of the input graph structure. Based on domain knowledge, perform visual analysis on the important components in sequence to verify whether the GNN prediction process can reconstruct domain knowledge.

[0022] The hierarchical knowledge tree mentioned in step 1 is represented as a clustering tree, consisting of a series of clustering assignment matrices S. (l) ∈ Define, where G represents the clustering graph data of the l-th layer of the graph. (l) The set of nodes, Represents the original graph node v i Cluster nodes belonging to the l-th layer graph Otherwise, it is 0.

[0023] Step 2 embeds Z = f into the nodes generated by the GNN model. emb Based on (G, X), we first learn a coarsening module. This module aggregates the representations of each high-level cluster and obtains its feature embeddings. in h is the number of clusters in the l-th level graph. * The output embedding size of the coarsening module is represented as follows:

[0024]

[0025] in It is a deep neural network parameterized by ψ1, and AGG(·) is the pooling operation.

[0026] Step 3: Design a weight allocation module Used to calculate the weight value of each level l of the edge between clusters. That is, the correlation score, specifically expressed as:

[0027]

[0028] in It is a multilayer perceptron (MLP) parameterized by ψ2, where [·; ·] is a matrix concatenation operation. Subsequently, the reparameterization technique proposed by PGExplainer is applied to achieve continuous relaxation of the side sampling process:

[0029]

[0030] Where ∈ ~Uniform(0,1) are random numbers used for sampling, and the sigmoid function σ with temperature hyperparameter β is used to assign correlation scores. Convert to edge weights Binarization is performed using β approaching 0, and the weights are aggregated to generate candidate subgraphs at level l.

[0031] Step 4 is based on the theory of sufficient representation, random variables A sufficient representation is defined as containing all expressions with... The information representation related to the target's underlying distribution or parameter inference should be used. When mapped back to the original graph, the mapping information should be consistent with the l-th level subgraph. With the same mutual information, the only candidate for this mapping is The complete expansion of each bottom-level node / edge to which the cluster / edge belongs is defined as the "fully expanded" function SE(·). Then, by maximizing the mutual information between each layer of the interpreted structure and the complete structure, and minimizing the information entropy and average weight of the output weights of the allocation module, an approximate exploration of the compact key structure is achieved. Finally, the candidate graph is maximized. Mutual information between full expansion and prediction results:

[0032]

[0033] The final objective function of the optimization framework is obtained, the supervision signal of the backpropagation algorithm is output, and then the coarsening module and weight allocation module are optimized to extract the knowledge dependencies of the GNN model.

[0034] Step 5: Based on the hierarchical importance ranking of the input structure learned in Steps 2-4, extract the high-weight structures in sequence for visualization, calculate and interpret statistical indicators of performance, and combine expert knowledge to conduct technical verification and analysis.

[0035] The technical effects to be achieved by this invention are as follows:

[0036] (1) This invention proposes a GNN interpretation target that is adapted to multi-level graph structures and is compatible with widely accepted mutual information metrics.

[0037] (2) In order to establish the connection between high-level and input-level graph features, the present invention introduces the concept of full expansion.

[0038] (3) An improved optimization framework is also proposed to address the objective function. By adding additional graph coarsening and attribute modules, high-level graph features can be appropriately represented, aggregated, and optimized. Finally, a general process for hierarchical interpretation of GNNs is established and an effective optimization algorithm is adopted to further help users analyze and understand the results of graph neural network learning at multiple granularities.

[0039] (4) Experimental data show that the proposed inventive technique outperforms other methods in almost all key accuracy and recall metrics on both real-world and synthetic data and classification / regression tasks. Specifically, it consistently ranks among the top two in six accuracy / recall metrics out of 11 explanation methods. For the other 10 alternative methods, the metric differences between them and the proposed STFExplainer were also calculated. On real and synthetic datasets, the inventive technique outperforms the best alternative methods by 12.4% and 8.7% respectively in average accuracy and recall metrics. Attached Figure Description

[0040] Figure 1 An explanatory diagram of the layered knowledge of this invention;

[0041] Figure 2 A flowchart and a schematic diagram of the internal structure of a hierarchical interpretation method based on a fully expanded graph neural network;

[0042] Figure 3 A diagram illustrating the factor hierarchy; Detailed Implementation

[0043] The following are preferred embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0044] This invention proposes a hierarchical interpretation method based on a fully expanded material graph neural network.

[0045] Includes the following steps:

[0046] (1) Prepare complex learning tasks with hierarchical interpretability (such as MOF molecular performance prediction), and encode hierarchical domain knowledge (such as functional group information, building block information, etc.) into hierarchical knowledge trees to represent the hierarchical interpretation constraints of the input graph data. Reproduce mainstream graph neural network models, train the model using training and validation sets, and then use the trained model to make predictions on the test set to obtain the prediction results; the hierarchical knowledge tree is represented as a clustering tree. This tree consists of a series of clustering assignment matrices. Define, where G represents the clustering graph data of the l-th layer of the graph. (l) The set of nodes, Represents the original graph node v i Cluster nodes belonging to the l-th layer graph Otherwise, it is 0.

[0047] (2) A coarsening feature extractor (coarsening module) is initialized for each layer to extract a coarsened representation of each layer. The coarsening module is a deep neural network that aggregates the encoded representations of the constituent structures of the GNN to be interpreted, learning features representing the entire cluster. Specifically, in the node embeddings Z = f generated by the GNN model... emb Based on (G, X), a coarsening module was first learned. This module aggregates the representations of each high-level cluster and obtains its feature embeddings. in h is the number of clusters in the l-th level graph. * This represents the output embedding size of the coarsening module. Formally, we have:

[0048]

[0049] in It is a deep neural network parameterized by ψ1, and AGG(·) is the pooling operation.

[0050] (3) Initialize a weight allocation module for each layer of graph structure to calculate the importance of clustering units, and generate candidate hierarchical interpretable subgraphs through subgraph sampling. Subgraph sampling involves reparameterization techniques, which transform the NP problem of subgraph search into a sampling distribution estimation problem through relaxation, and introduce a temperature parameter to control the approximation of discrete sampling.

[0051] Specifically, a weight allocation module was designed. Used to calculate the weight value of each level l of the edge between clusters. That is, the relevance score. Formally, we have:

[0052]

[0053] in It is a multilayer perceptron (MLP) parameterized by ψ², where [·; ·] is a matrix concatenation operation. Subsequently, this paper applies the reparameterization technique proposed by PGExplainer to achieve continuous relaxation of the side-sampling process:

[0054]

[0055] Where ∈ ~Uniform(0,1) are random numbers used for sampling, and the sigmoid function σ with temperature hyperparameter β is used to assign correlation scores. Convert to edge weights It uses β approaching 0 for binarization. Edge weights are aggregated to generate candidate subgraphs at level l.

[0056] (4) Based on the relevant concepts of representation learning, a fully unfolded method is designed to map the interpretable subgraph of the coarsened structure to the complete component features of the input subgraph and input them into the GNN to be interpreted to calculate the predicted changes. Then, gradient descent is used to optimize the coarsening module and weight allocation module of each layer. The training is performed iteratively on all data until the results converge.

[0057] This step is based on the theory of full representation, and random variables. A sufficient representation is defined as containing all expressions with... This involves representing information related to the target's underlying distribution or parameter inference. When mapped back to the original graph, the mapping information should correspond to the l-th level subgraph. They have the same mutual information. The only candidate for this mapping is... The complete expansion of each bottom-level node / edge to which the cluster / edge belongs is defined in this paper as the "fully expanded" function SE(·). Then, by maximizing the mutual information between each layer of the interpreted structure and the complete structure, and minimizing the information entropy and average weight of the output weights of the allocation module, an approximate exploration of the compact key structure is achieved. Finally, the candidate graph is maximized. Mutual information between full expansion and prediction results:

[0058]

[0059] The final objective function of the optimization framework is obtained, the supervision signal of the backpropagation algorithm is output, and then the coarsening module and weight allocation module are optimized to extract the knowledge dependencies of the GNN model.

[0060] (5) Combining the coarsening module and weight allocation module trained in (2) and (3), the hierarchical importance ranking of the input graph structure is generated. The important components are visualized and analyzed in turn according to the domain knowledge to verify whether the prediction process of GNN can reconstruct the domain knowledge.

[0061] That is, based on the hierarchical importance ranking of the input structure learned in steps (2), (3), and (4), the high-weight structures are extracted sequentially for visualization. The invention technology is verified and analyzed by calculating the statistical indicators of the explanatory performance and combining expert knowledge.

[0062] Figure 2This is a flowchart and a schematic diagram of the internal structure of a hierarchical interpretation method based on a fully expanded graph neural network according to an embodiment of the present invention.

[0063] like Figure 2 As shown, we first obtained the raw data from the embodiment, then preprocessed it into a standard graph data structure, and trained it using a standard Graph Neural Network (GNN) until the model converged, ensuring good performance on the test set. Next, we transformed the domain knowledge of the graph data into hierarchical constraints and extracted a hierarchical clustering tree for each data instance. Finally, we obtained the GNN model to be explained: a standard graph data structure and a hierarchical clustering tree derived from human-readable domain knowledge.

[0064] After obtaining the data required for the GNN interpretation method, we constructed a hierarchical interpretation framework based on a fully unfolded graph neural network. Figure 2 (As shown). In the second step, we take the encoding of the input data by the GNN model to be explained as input, and the hierarchical clustering tree transformed from human-readable domain knowledge as constraints. Through a coarsening module, we obtain the coarsened clustering feature representation of the corresponding structural level.

[0065] The third step involves using a weighting module to assign weights to the coarsened graph structure and its coarsened features, thereby obtaining the correlation between each component of the graph structure at each layer and the final predicted value.

[0066] In the fourth step, based on the reparameterization technique, we sample coarsened subgraphs according to the correlation between each component of the graph structure and the final predicted value, obtaining candidate explanatory subgraphs. Using the full expansion method, we obtain the prediction result of the GNN after fully expanding this candidate coarsened subgraph as input, and calculate the mutual information between it and the GNN prediction result of the original input structure. Combined with the regularization constraints described in the invention, we finally obtain the objective function value as the supervision signal for the backpropagation algorithm, thereby optimizing the coarsening module and the weight allocation module, and extracting the knowledge dependencies of the GNN model.

[0067] In the fifth step, we extract the trained and iterated hierarchical interpretation framework, obtain the correlation between each component of the graph structure in each layer and the final predicted value, and then sort and visualize them according to the correlation to help users analyze the model results (e.g., Figure 3 (As shown).

Claims

1. A layered explanation method based on a fully unfolded material graph neural network, characterized in that: The method comprises the following steps: Step 1, preparing a molecular property prediction complex learning task with hierarchical explainability, wherein the processed data can be a text stream, encoding the hierarchical domain knowledge as a hierarchical knowledge tree, using the training set and the validation set to train the model, and then using the trained model to predict on the test set to obtain the prediction result; Step 2, initializing a coarse structure feature extractor for each layer structure to extract the coarse representation of each layer structure, wherein the coarse structure feature extractor is a deep neural network that converges the encoding representation of the structure to be explained GNN, and learns the features representing the entire cluster; Step 3, initializing a weight distribution module for calculating the importance of each layer structure, and generating a candidate hierarchical explainable subgraph through subgraph sampling, wherein the subgraph sampling involves a reparameterization technique, which converts the NP problem of subgraph search into a sampling distribution estimation problem by relaxation, and introduces a temperature parameter to control the approximation degree of discrete sampling; Step 4, based on the related concept of representation learning, designing a sufficient expansion method to map the explainable subgraph of the coarse structure to the complete component feature in the input subgraph, and inputting the feature into the GNN to be explained to calculate the change of the prediction, and then optimizing the coarse module and the weight distribution module of each layer structure through gradient descent, and training all data through cyclic iteration until the result converges; Step 5, combining the trained coarse module and weight distribution module in steps 2 and 3 to generate a hierarchical importance ranking of the input graph structure, and visualizing and analyzing the important components according to the domain knowledge to verify whether the prediction process of the GNN can reconstruct the domain knowledge; The basis of step 4 is the sufficient representation theory, a random variable A sufficient representation of a random variable is defined as a representation that contains all the information about The target underlying distribution or parameter inference, when mapped to the original graph, its mapped information should have the same mutual information as the first Layer subgraph With the same mutual information, the only candidate for such a mapping is the full expansion of each bottom-most node / edge that the cluster / edge in Belongs to, defined as the "sufficient expansion" function Then, by maximizing the mutual information between each layer's explanation structure and the full structure, as well as minimizing the information entropy of the assigned module output weights and the average weight, the approximate exploration of the compact key structure is achieved, finally, the mutual information between the sufficient expansion of the candidate graph And the prediction result is maximized: The final objective function of the optimized framework is obtained, and the supervision signal of the back propagation algorithm is outputted, and then the coarse module and the weight distribution module are optimized to extract the knowledge dependence of the GNN model.

2. The layer-wise explanation method based on a fully unfolded material graph neural network according to claim 1, wherein: The hierarchical knowledge tree in step 1 is represented as a clustering tree by a series of clustering assignment matrices , , where represents the set of nodes of the layer clustering graph data , represents the original graph node belongs to the cluster node of the layer graph , otherwise 0.

3. The layer-wise explanation method based on a fully unfolded material graph neural network of claim 2, wherein: Step 2: Embedding nodes generated by the GNN model Building upon this foundation, we will first learn a coarsening module. This module aggregates the representations of each high-level cluster and obtains its feature embeddings. ,in It is the first The number of clusters in the hierarchical graph The output embedding size of the coarsening module is represented as follows: wherein is a deep neural network parameterized by parameters, is a pooling operation.

4. The layer-wise explanation method based on a fully unfolded material graph neural network of claim 3, wherein: Step 3 designs a weight distribution module for calculating the weight value of each level of the edge between clusters , i.e. the correlation score, specifically represented as: wherein is derived from a parameterized multi-layer perceptron (MLP), is a matrix concatenation operation, followed by the reparameterization trick proposed by PGExplainer to achieve continuous relaxation of the edge sampling process: wherein is a random number for sampling, and has a temperature hyperparameter sigmoid function for converting the relevance scores into edge weights , using binarization towards 0, the weights are aggregated to generate candidate subgraphs of the layer .

5. The layer-wise explanation method based on a fully unfolded material graph neural network according to claim 4, wherein: Step 5 extracts the high-weight structure for visualization according to the hierarchical importance ranking of the input structure learned in steps 2-4, and verifies and analyzes the technology by calculating the statistical indicators of the explanation performance and combining the expert knowledge.

Images