Network node classification method based on multilayer graph neural network model

Abstract

The invention provides a network node classification method based on a multilayer graph neural network model, and the method comprises the steps: constructing a graph according to network nodes, carrying out the preprocessing of initial node features in the graph, and obtaining an initial embedded vector; according to the initial embedding vectors of the node and the neighbor node thereof, updating the node representation by adopting a layered message passing mechanism; according to the initial embedded vectors of the nodes and the neighbor nodes thereof, the graph attention network uses an attention mechanism to calculate weights for the neighbor nodes of each node and then dynamically updates feature vectors of the nodes; a multi-head attention mechanism is adopted to splice and merge the feature vectors of the nodes output by the graph attention network and the updated node representation, and the comprehensive feature vector of each node is obtained; the comprehensive feature vectors are further optimized through graph contrast learning and a node embedding module, and optimized features are obtained; and obtaining predicted node classification according to the optimization features. According to the method, complex features of heterogeneous graph data can be effectively captured, the method has good robustness for an over-smoothing problem, more accurate heterogeneous network representation learning is realized, and the method has advanced performance and robustness in a node classification task.

Description

Technical Field

[0001] The present invention relates to the technical field of network node classification, and in particular to a network node classification method based on a multi-layer graph neural network model. Background Art

[0002] In recent years, graph neural networks (GNNs) have attracted great attention from researchers due to their powerful learning capabilities, and have been widely used in various tasks such as node classification, link prediction, graph classification, molecular property prediction, natural language processing, and node sorting. The mainstream GNN model adopts a message passing strategy to learn expressive node representations through message propagation and aggregation between adjacent nodes. In homogeneous scenarios, nodes and adjacent nodes have similar features and labels. Therefore, the cumulative aggregation of their features enhances the signal consistent with the given label, thereby improving the prediction accuracy. However, in most scenarios in real life, different nodes have different features and labels. In such heterogeneous scenarios, the continuous aggregation of data may lead to reduced signal clarity and the introduction of additional noise, resulting in poor model learning and performance degradation.

[0003] In order to break through the above limitations, researchers have focused on studying heterogeneous representation learning of GNN nodes. Most of these studies focus on the aggregation stage of message passing, such as designing symbolic messages belonging to different categories to distinguish adjacent nodes, enabling GNN to capture high-frequency signals; designing specific filters to capture bandpass signals; aggregating information of direct adjacent nodes and higher-level adjacent nodes and even embedding space. Although these methods have achieved good performance, they mainly focus on the single-round message passing process and ignore the integration of multi-hop messages.

[0004] Another type of work focuses on effectively utilizing multi-hop messages by designing different connections for message integration. Some studies proposed initial connections to prevent local information from being "eliminated" by stacking multiple convolutional layers; inspired by ResNet (deep residual) networks, some works explored the application of residual connections in GNNs; and some directly integrated the results of the intermediate convolution stage with carefully designed elements such as concat, learnable weights, signed weights, or RNN-like architectures. These works are simple and effective, but they can only model information within a few hops, cannot accurately model information in a specific order, and cannot integrate local and global heterogeneous information at the same time, resulting in feature mixing and affecting the accuracy of the model. In addition, excessive stacking of propagation layers in GNNs causes the representation of nodes to tend to converge to a specific value, resulting in over-smoothing problems, and ultimately hindering GNNs from effectively utilizing the information of higher-level neighboring nodes.

[0005] In graph neural network (GNNs) models, global information that includes node heterogeneity is crucial to ensure the accuracy of the model. First, global information promotes the interaction between any pair of nodes, regardless of their topological distance. Second, fully considering the heterogeneity of nodes helps to fully learn network representations.

[0006] Therefore, in the process of message passing, traditional graph neural networks (GNNs) suffer from poor model learning and performance degradation due to the heterogeneity of neighborhood nodes. Multiple rounds of iterations in GNNs make node representations difficult to distinguish, which affects the model's ability to absorb relationships between long-distance nodes and causes over-smoothing problems. Existing research has specific limitations in learning heterogeneous representations of GNN nodes. For example, most studies focus on the single-round message passing process and ignore the integration of multi-hop messages, or fail to integrate local and global heterogeneous information at the same time. The importance of global information in GNNs to ensure model accuracy has been overlooked.

[0007] The existing network node classification methods mainly include the following: methods based on traditional machine learning, methods based on shallow neural networks, methods based on traditional graph neural networks (GNNs), and improved methods of graph neural networks based on deep learning. The calculation of methods based on traditional machine learning is relatively simple, and they may show good performance and interpretability for small-scale data sets; they do not require a lot of computing resources and complex model architectures. However, it is difficult to process large-scale and high-dimensional graph data, and the feature extraction ability is limited; the graph structure is not fully utilized, and the complex relationship between nodes cannot be well captured. Compared with traditional machine learning methods, methods based on shallow neural networks can automatically learn features and have certain generalization capabilities; training is relatively fast, and may have good effects for some simple graph structures. However, the structure of shallow neural networks limits its ability to model complex relationships, making it difficult to capture deep-level graph features; it has limited processing capabilities for node heterogeneity and is easily affected by noise and outliers. Methods based on traditional graph neural networks can effectively utilize graph structure information, learn node representations, and have good performance for graph-related tasks; they can capture local and global relationships between nodes through message passing mechanisms. However, this method has the problem of poor model learning effect and performance degradation caused by the heterogeneity of neighborhood nodes. Multiple rounds of iterations easily lead to over-smoothing problems, making node representations difficult to distinguish. Most studies focus on the single-round message transmission process and ignore the integration of multi-hop messages, or fail to integrate local and global heterogeneous information at the same time. The improved graph neural network method based on deep learning can solve some problems of traditional GNNs by designing more complex network structures and loss functions, which can better handle node heterogeneity and over-smoothing problems and improve the accuracy and generalization ability of the model. However, this method has high model complexity and requires a lot of computing resources and time for training and optimization. It is difficult to adjust hyperparameters and has high requirements on the quality and scale of data.

[0008] In the process of message passing in traditional graph neural networks, the heterogeneity of neighborhood nodes will lead to poor model learning effect and performance degradation. Multiple rounds of iterations in graph neural networks (GNNs) will make node representations difficult to distinguish, affecting the model's ability to absorb long-distance node pairs and causing over-smoothing problems. Most existing studies focus on the single-round message passing process and ignore the integration of multi-hop messages, or fail to integrate local and global heterogeneous information at the same time. Summary of the invention

[0009] In view of the technical problem that existing network node classification methods cannot well capture the complex features of graph data and avoid over-smoothing, the present invention proposes a network node classification method based on a multi-layer graph convolutional neural network model. The multi-layer graph neural network model is used to integrate local and global data to better capture the complex features of graph data and avoid over-smoothing problems. At the same time, heterogeneous graph representation learning is adaptively performed to improve the accuracy of node classification.

[0010] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a network node classification method based on a multi-layer graph neural network model, the steps of which are as follows:

[0011] Step 1: Build a graph based on network nodes, preprocess the initial node features in the graph, and obtain the initial embedding vector;

[0012] Step 2: Based on the initial embedding vectors of the node and its neighboring nodes, a hierarchical message passing mechanism is used to update the node representation;

[0013] Step 3: Based on the initial embedding vectors of the node and its neighboring nodes, the graph attention network uses the attention mechanism to calculate the weights of each node’s neighboring nodes and dynamically updates the node’s feature vector;

[0014] Step 4: Use the multi-head attention mechanism to concatenate and merge the feature vector of the node output by the graph attention network and the updated node representation to obtain the comprehensive feature vector of each node;

[0015] Step 5: Use graph contrast learning and node embedding modules to further optimize the comprehensive feature vector to obtain optimized features;

[0016] Step 6: Obtain predicted node classification based on optimized features.

[0017] Preferably, the method for constructing a graph is: graph G = (V, E), the node set V is the set of all nodes in graph G, the edge set E is the set of edges between all pairs of nodes in graph G, and the adjacency matrix A∈R of graph G N×N Represents the connection relationship between nodes, and the node features of all nodes constitute the node feature matrix X∈R N×F, where N is the number of network nodes and F is the dimension of node features. Each node v∈V is assigned a label y v ∈Y;

[0018] The preprocessing method is: nonlinearly projecting the original node features of node v in the node feature matrix X through a multi-layer perceptron to obtain the projection feature Z v ∈R D , D is the dimension of node embedding; for the projection feature Z v Perform linear transformation and normalization to obtain the initial embedding vector of node v

[0019] Add self-loops to the adjacency matrix A to get the adjacency matrix I is the identity matrix.

[0020] Preferably, the hierarchical message passing mechanism combines the message passing mechanism in GNN with the root tree hierarchy of the root node, aligns the hierarchy of the root tree with the ordered neurons in the node embedding, controls the allocation of neurons through the ordered gating mechanism to encode subtrees of different depths for message passing, and enables GNN to accurately encode neighborhood information in a certain order.

[0021] Preferably, the method of aligning the hierarchy of the root tree with the ordered neurons in the node embedding is:

[0022] 1) Construct the nested structure of the root tree: take the node v currently being processed in the graph G as the root, and the adjacent nodes of node v as child nodes, and recursively construct the root tree; take the child nodes of the root tree as the root, and so on to construct the nested structure of the root tree;

[0023] 2) Calculate the initial node representation of node v according to the initial embedding vector of node v and the initial embedding vectors of its neighboring nodes, predict the segmentation point position according to the initial node representation of node v, and determine the gating vector according to the segmentation point position;

[0024] 3) Encode the information in the root tree in the neuron sequence according to the gating vector, so that the hierarchical structure of the root tree is aligned with the ordered neurons in the node representation.

[0025] Preferably, the method for predicting the position of the segmentation point according to the initial node representation of the node v is: inputting the initial node representation of the node v into a linear layer to obtain an output vector, performing a cumulative summation operation on the output vector, performing a Softmax function operation on the cumulative summation vector, and obtaining the probability of each position in the node representation being a segmentation point Probability of choosing a split point The maximum position is taken as the predicted segmentation point position

[0026] Preferably, the gating vector Before entries are 1, and the rest are 0;

[0027] Node v is the node representation of the kth layer. Depends only on its k-th root number The nodes within the node represent All neurons in are sorted into a sequence, and for the root tree The information in the first Neuron encoding, for the next level root tree Corresponding to the first neurons, and

[0028] The initial embedding vector of node v As the first layer input of GNN In each layer of GNN, node v updates its node representation according to the embedding of its neighbor nodes and its own embedding. The node representation of node v in the kth layer of GNN is:

[0029]

[0030] Among them, AGGREGATE (k) It is an aggregate function, COMBINE (k) is a combinatorial function; N(v) represents the set of neighbor nodes of node v; is the node representation of neighbor node u in the kth layer of GNN;

[0031] When node v is at the kth layer, the probability that the i-th position in the node representation becomes a split point is:

[0032]

[0033] Where exp represents the exponential function, Represents the element value at the j-th position represented by the node of the k-1th layer of node v, and l represents the index variable for cumulative summation.

[0034] Preferably, the method of controlling the allocation of neurons by an ordered gating mechanism to encode subtrees of different depths for message transmission is: aggregating the initial node representations of the neighboring nodes of node v to obtain the current layer aggregated context information The node v at the k-1th layer is represented by Aggregate context information with the current layer Processing to obtain the expected vector The gate vector of the previous layer is processed by a differentiable OR operator and the expected vector of the current layer Processing to obtain the expected vector Get each position in the node representation as a split point The cumulative sum of the probabilities of The node representation of the k-1th layer of node v is Aggregate context information with the current layer Perform integration to obtain updated node representation.

[0035] Preferably, the current layer aggregates context information

[0036] The expected vector

[0037] The gating vector

[0038] The updated node representation

[0039] Among them, MEAN() represents the average operation, and N(v) represents the set of neighbor nodes of node v; is the node representation of neighbor node u in the kth layer of GNN; Indicates that the nodes in the k-1th layer are represented Aggregate context information with the current layer Splicing; W (k) and b (k) They are respectively represented as the weight matrix and bias of linear projection; softmax(·) represents the normalization operation, cumsum(·) represents the cumulative sum function; SOFTEN(·) is a differentiable OR operator, ε represents element-wise multiplication, and α1 and α2 are two adaptive parameters.

[0040] Preferably, the graph attention network uses a shared attention mechanism and a Softmax function to calculate the feature vector of the graph attention network of the k-1th layer of node v. and the feature vector of the k-1th layer of the graph attention network of its neighbor node u The attention coefficient is processed and used as the weight to dynamically update the feature vector of the k-th layer graph attention network of node v;

[0041] The graph contrastive learning updates node embeddings according to the gradients calculated by the contrastive loss function through the back-propagation algorithm to optimize the model parameters;

[0042] In step six, a multi-head attention mechanism and a Softmax function are used to process the optimized features and obtain the classification prediction of the node through the output layer.

[0043] Preferably, the feature vector of the graph attention network of the kth layer of the node v is in, is the attention coefficient between node v and its neighbor node u, σ represents the nonlinear activation function, Represents the trainable weight matrix of the k-th layer of the graph attention network;

[0044] The comprehensive feature vector

[0045] in, represents the comprehensive vector generated by node v under the multi-head attention mechanism of the kth layer; P represents the number of heads of multi-head attention; tanh represents the hyperbolic tangent function; Represents the computation function of the nth attention head at the kth layer; Represents the union of the output vectors of multiple attention heads after being processed by the tanh activation function;

[0046] The conversion process of the node embedding module in the kth layer is:

[0047] Each node embedding module takes the output of the previous node embedding module as input to further refine and arrange in order: and are the trainable weight matrix and bias vector of the k-th node embedding module, respectively. It is the optimized feature obtained by graph contrast learning; They are the optimized features obtained by the node embedding modules at the k-1th and kth layers respectively;

[0048] The contrast loss function is:

[0049]

[0050] in, is the comprehensive feature vector of node v’s neighbor node u, N’(v) represents the set of non-neighboring nodes of node v, n 1 represents nodes that are not adjacent to node v; sim(·) is the similarity function, and τ represents the temperature parameter;

[0051] The implementation method of step 6 is: for the node v of the kth layer, the multi-head attention mechanism is used to obtain the combined output of all P attention heads:

[0052]

[0053] in, Represents the optimization features of each node according to the nth attention head The attention score calculated by the importance of

[0054] The Softmax function processes the probability distribution of the possible categories of each node v:

[0055]

[0056] Among them, W class is the weight matrix, b class is the bias vector of the classification layer, K represents the last layer of the attention mechanism, Represents the output of the multi-head attention mechanism.

[0057] Compared with the prior art, the beneficial effects of the present invention are as follows: a new multi-layer graph neural network (NML-GNN) model is proposed, which can effectively capture the complex features of heterogeneous graph data and has good robustness to over-smoothing problems. Through the hierarchical message passing mechanism and graph comparative learning, more accurate heterogeneous network representation learning is achieved, and the model's ability to detect and analyze important features required for downstream applications is improved, and it has advanced performance and robustness in node classification tasks.

[0058] The multi-layer graph neural network model of the present application uses a hierarchical message passing strategy to overcome the limitations of graph structural homogeneity and over-smoothing; a global method based on graph contrast learning can enhance feature representation capabilities and improve the robustness of the model by integrating hierarchical graph convolution and graph attention network (GAT) modules; a series of node embedding modules are designed to iteratively optimize feature representation, gradually improve feature space, and improve model performance and generalization capabilities; a multi-head attention mechanism running in different representation subspaces is defined to improve the model's ability to detect and analyze features that are important for downstream applications. At the same time, the present invention can bring the following effects:

[0059] 1. Solve the problem of neighborhood node heterogeneity: The NML-GNN model proposed in the present invention adopts a hierarchical message passing mechanism, combines the message passing mechanism in the graph neural network with the root number hierarchy of the root node, and realizes a more organized message passing mechanism, so that the NML-GNN model has more discrimination for different nodes, and can accurately distinguish nodes with different characteristics. By aligning the hierarchy of the root number with the ordered neurons in the node embedding, the graph neural network GNN accurately encodes the neighborhood information in a certain order, avoids the mixing of features within the hop number, and has more discrimination for different nodes, thereby solving the problem of neighborhood node heterogeneity, avoiding poor learning effect and performance degradation caused by heterogeneity, thereby improving the accuracy of node classification. The effective processing of neighborhood node heterogeneity by the present invention makes the NML-GNN model more stable when facing different types of nodes, not easily affected by abnormal nodes, and improves the robustness of node classification.

[0060] 2. Solve the over-smoothing problem: The present invention effectively maintains the orderliness of the segmentation points in each layer through a hierarchical message passing mechanism, which can retain the personalized characteristics of the nodes in the deep network and avoid the node representation tending to converge to a specific value, thereby solving the over-smoothing problem and better capturing the unique characteristics of nodes of different categories, thereby improving the accuracy of node classification; avoiding overfitting of the NML-GNN model during the training process, so that the NML-GNN model can maintain good performance when facing new data, and enhancing the robustness of node classification.

[0061] 3. Breakthrough of existing research limitations: The NML-GNN model of the present invention introduces graph contrast learning, which can maximize the mutual information of node embeddings between connected nodes and minimize the mutual information between non-connected nodes, thereby realizing the integration of local and global heterogeneous information, enhancing the feature representation ability of the model, and improving the accuracy and generalization ability of the model in downstream tasks. In addition, through the parallel processing and adaptive weighting of the multi-head attention mechanism, the representation of node features is optimized, and different parts of the node embedding can be focused on at the same time, enhancing the ability to understand in depth, enhancing the ability to learn from complex graphic data, and being able to dynamically focus on the most important parts of the graph while maintaining local and global graph structure information, thereby achieving excellent performance in tasks such as node classification. The present invention integrates local and global heterogeneous information and dynamically focuses on important parts of the graph through a multi-head attention mechanism, so that the model can learn more comprehensive and representative node features, provide a more accurate basis for node classification, and improve the accuracy of node classification.

[0062] 4. Better capture of graph data features: The present invention can integrate local and global data to better capture the complex features of graph data; the present invention can gradually improve the feature representation through iterative optimization of the node embedding module, avoid repeated calculations and invalid feature extraction, and speed up the classification process. The present invention can accurately encode the structural information of the graph through the hierarchical message passing mechanism and the node embedding module, and can adapt to different types of graph structures, so that the model can effectively classify nodes when facing various complex graph data, improving the versatility and generalization ability of the model; graph contrast learning uses the inherent structural information in the graph to enhance the feature representation ability of the model; the multi-head attention mechanism can dynamically focus on the most important part of the graph, so as to capture node features more comprehensively. The present invention enhances the feature representation ability through graph contrast learning and multi-head attention mechanism, and can extract more general features from graph data, so as to have stronger generalization ability in different data sets and tasks. The present invention can quickly focus on key nodes and features through the hierarchical message passing mechanism and the multi-head attention mechanism, reduce unnecessary calculations and information processing, and improve the efficiency of node classification.

[0063] 5. Improve model performance and robustness: In the node classification task, the NML-GNN model of the present invention has advanced performance and robustness. Through the synergy of hierarchical message passing mechanism, graph contrast learning and multi-head attention mechanism, the present invention can effectively handle complex interactions between nodes and make accurate predictions, thereby improving the performance in tasks such as node classification, and can obtain more accurate node classification results, with stronger generalization ability, better robustness and more efficient classification process. The present invention integrates the synergy of multiple mechanisms, has stronger anti-interference ability when facing noise, data missing and other problems, and improves the stability and reliability of node classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 It is a data processing schematic diagram of the present invention.

[0066] Figure 2 Schematic diagram of node alignment.

[0067] Figure 3 Schematic diagram of the classification accuracy of nodes at different depths in different data sets according to the present invention and the baseline model, where (a) is Cora, (b) is CiteSeer, and (c) is PubMed. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] like Figure 1As shown, a network node classification method based on a multi-layer graph neural network model is implemented using a multi-layer graph convolutional neural network (NML-GNN) model. The multi-layer graph convolutional neural network model includes a graph attention network (GAT), a layered information transfer layer, namely a LayeredGCN module, a graph contrast learning module, a node embedding module and a multi-head attention mechanism. The multi-head attention mechanism includes a first multi-head attention mechanism and a second multi-head attention mechanism. The graph attention network and the LayeredGCN module are all connected to the first multi-head attention mechanism. The outputs of the graph attention network, the layered information transfer layer and the first multi-head attention mechanism are all connected to the graph contrast learning module, the graph contrast learning module is connected to the node embedding module, and the node embedding module is connected to the second multi-head attention mechanism. The layered message passing mechanism and the node embedding module of the LayeredGCN module of the present application are the basis, which provide an orderly message passing and node representation method for the multi-layer graph convolutional neural network model; the graph contrast learning module further enhances the feature representation ability of the model based on the layered representation integration of graph contrast learning; the multi-head attention mechanism enhances the graph structure representation so that the multi-layer graph convolutional neural network model can better handle the relationship between nodes and improve the performance of the model. In the LayeredGCN module, the node representation of each layer is modified by integrating the feature vectors of its neighboring nodes. At the same time, GAT applies an attention coefficient to weigh the features of each neighboring node, and calculates the attention coefficient between two nodes in each layer through a shared attention mechanism to dynamically adjust the node representation. In the NML-GNN model, for a node v of a certain layer, the results of the GAT and LayeredGCN modules are first merged to create a coherent feature vector, namely a comprehensive feature vector. Then, through the first multi-head attention mechanism, each attention head processes the connected features separately, and the outputs of all attention heads are aggregated to form the final representation of the node v of this layer. The graph contrast learning module maximizes the mutual information between the node embeddings of connected nodes while minimizing the mutual information between non-connected nodes. The node embedding module further refines the output of the previous module and arranges it in order. The second multi-head attention mechanism calculates the attention scores of the nodes and classifies them. The specific implementation steps of the present invention are:

[0070] Step 1: Data preprocessing: Build a graph based on network nodes, preprocess the initial node features in the graph and input them into the LayeredGCN module to facilitate the subsequent use of a hierarchical message passing mechanism to combine the graph structure information with the node features to update the node representation.

[0071] In practical applications, the information of network nodes is stored in the form of data sets, including adjacency lists, edge files, initial node feature files, etc., and graph processing libraries (such as NetworkX, DGL, or PyTorch Geometric) are used to construct the graph G.

[0072] The specific process of preprocessing the initial node features of the present invention is as follows:

[0073] Consider and input an unweighted and self-loop-free graph G = (V, E), where the node set V is the set of all nodes in the graph G, the edge set E is the set of edges between all pairs of nodes in the graph, and the adjacency matrix A∈R of the graph G N×N According to the edge set E, which is filled and represents the connection relationship between nodes, the node features of all nodes form the node feature matrix X∈R N×F , where N is the number of network nodes and F is the dimension of node features, i.e. the length of the feature vector of each node. Each node v∈V is assigned a label y v ∈Y. For example, in a literature citation network, a node may represent a paper, and a label may be the academic subject category of the paper (such as computer science, biology, etc.); in a Web network, a node may be a web page, and a label may be the content category of the web page (such as news, entertainment, technology, etc.). These labels provide clear target categories for subsequent node classification tasks.

[0074] The data preprocessing method is:

[0075] 1. The original node feature X of node v v ∈X is nonlinearly projected through a multi-layer perceptron (MLP) to obtain the projection feature is the weight matrix, is the bias term, σ is the ReLU nonlinear activation function, D0=F, D n1 =D, D i1 represents the dimension of the i1th layer of the multilayer perceptron, n1 represents the number of layers of the multilayer perceptron. D is the dimension of the node embedding, and the projection feature Z v ∈R D .

[0076] 2. Projection feature Z after projection v Perform linear transformation and normalization: Linear transformation and normalization are two key preprocessing steps used to adjust the scale and distribution of node features for subsequent learning and processing. Here, the initial embedding vector of node v is obtained: Among them, Linear represents the linear transformation function, and LayerNorm represents the normalization function.

[0077] 3. The normalized feature is the initial embedding vector of node v As the first layer input of GNN, it is recorded as Used for subsequent message passing and node update operations.

[0078] The graph structure information preprocessing process of the present invention is: adding self-loops to the adjacency matrix A to ensure that the characteristics of each node itself are considered during the information transmission process, which is expressed as:

[0079] Step 2: Based on the pre-processed node features of neighboring nodes and itself, a hierarchical message passing mechanism is used to update the node representation.

[0080] The NML-GNN model proposed in the present invention adopts a hierarchical message passing mechanism (named as LayeredGCN module), combines the message passing mechanism in GNN with the root tree hierarchy of the root node, and aligns the hierarchy of the root tree with the ordered neurons in the node embedding, so that GNN can accurately encode the neighborhood information in a certain order, avoid feature mixing within the hop number, and realize a more organized message passing mechanism, so that the model has more discrimination for different nodes, thereby solving the heterogeneity of graph structured data and the over-smoothing problem that may be caused by traditional graph neural networks.

[0081] The core idea of ​​the LayeredGCN module is to align the hierarchical structure of the root tree with the ordered neurons in the node embedding, and control the allocation of neurons through an ordered gating mechanism to encode subtrees of different depths for message transmission. The ordered gating mechanism is implemented by an adaptive gating unit, which controls information transmission through a gating vector. The specific process is as follows:

[0082] The way to align the hierarchy of the rooted tree with the ordered neurons in the node embeddings is:

[0083] 1) Construct a nested structure of the root tree: Take the node v currently being processed in the graph G as the root, and the adjacent nodes of node v as child nodes, and recursively construct the root tree; take the child nodes of the root tree as the root, and so on to construct the nested structure of the root tree.

[0084] Construct a nested structure of the root tree: take the currently processed node v in graph G as the root node, and its k-order root number The first-order neighbors are taken as child nodes, and then recursively for each child node, its neighbor nodes are taken as child nodes except the child nodes that are the ancestors of the node, until the k-order neighbors are formed; since there is only one round of message passing in each GNN layer, stacking multiple layers of GNN corresponds to multiple rounds of message passing, and the node v in the node representation of the k-th layer Depends only on its k-th root number The goal is to correctly encode the information within the k-th root into a fixed-size node representation. middle.

[0085] Using the nested structure between root trees, assuming the model has K layers, then in Corresponding to the neurons that represent the node's own information. As k grows, the number of roots Larger and more complex, more neurons are needed to encode its information, so it is assumed that The neurons that transmit messages should be A subset of .

[0086] 2) Calculate the initial node representation of node v according to the initial embedding vector of node v and the initial embedding vectors of its neighboring nodes, predict the segmentation point position according to the initial node representation of node v, and determine the gating vector according to the segmentation point position.

[0087] First, make a segmentation point prediction:

[0088] (1) In each layer of GNN, node v updates its node representation according to the embedding of its neighbor nodes and its own embedding. The updating process can be expressed as:

[0089]

[0090] Among them, AGGREGATE (k) is an aggregate function, such as sum, average, or maximum, COMBINE (k) is a combinatorial function, such as concatenation or weighted sum. represents the node representation of node v in the kth layer of GNN, and N(v) represents the set of neighbor nodes of node v. is the node representation of neighbor node u in the kth layer of GNN.

[0091] (2) The node representation of the node of the kth layer after fusion Input into a linear layer to get an output vector, which is cumulatively summed, that is, accumulated from right to left; finally, the Softmax function is performed on the cumulative summed vector to get the probability of each position in the node representation being a segmentation point Specifically, when node v is at the kth layer, the probability that the i-th position in the node representation becomes a split point is:

[0092]

[0093] in, Represents the element value at the j-th position represented by the node of the k-1th layer of node v, and l represents the index variable for cumulative summation.

[0094] (3) Based on the probability of the split point To predict the exact split point location: select the location with the highest probability of the split point as the predicted split point location Represented as a gate vector The gate vector Before entries are 1, and the rest are 0. Figure 2 As shown, the bottom rooted tree of graph G is Schematic diagram of the alignment of the hierarchical structure of the root node v with the node embedding of the root node v, where k = 3. The node v and its neighboring nodes within k hops naturally form a subtree with a depth of k. The messages passed from the nodes in the subtree to the node v are limited to the previous nodes in the node representation of the node v. The method of mapping different depths of adjacent nodes to other neurons is: through the gating vector Keep the previous node represented by the corresponding entries, and the rest are 0. In this way, the information of subtrees of different depths is controlled at different positions of node embedding.

[0095] 3) Encode the information in the root tree in the neuron sequence according to the gating vector, so that the hierarchical structure of the root tree is aligned with the ordered neurons in the node representation. All neurons in are sorted into a sequence, and for the root tree The information in the first Neuron encoding, for the next level root tree Involving more neurons, corresponding to the first neurons, and Use a D-dimensional gating vector Indicates the split point On its left entries are 1, and the rest are 0. In the combination phase, for the node representation Center left neurons, due to the gating vector The corresponding gating vectors are 1, which suppress the newly aggregated context information Only inherit the output of the previous layer The opposite is true for the neuron on the right.

[0096] 2. Split point gating

[0097] The gate vector is used to control the node representation of the k-1th layer and the aggregated message of the current layer Integration in the combination stage. That is, the gate vector is determined based on the predicted segmentation point probability That is, after the location of the split point, the node representation of the central node k-1 layer is performed in the combination stage Aggregate context information with the current layer The specific process is expressed by the formula:

[0098]

[0099] in, The aggregated context information of the current layer is obtained by aggregating the feature vectors of the neighboring nodes of node v. In the LayeredGCN module of the present invention, the feature vectors of all neighboring nodes of node v are averaged to obtain the aggregated context information of the current layer. MEAN represents the average operation.

[0100] In formula (2) In order to maintain the differentiability of the model, the present invention adopts a "softening" gating method, that is, by predicting the expected value of the gating vector, the expected vector Specifically, the expected vector It is through The expected vector obtained by cumulative sum and softmax operation Indicates that each position in the node embedding becomes a split point The cumulative sum of the probabilities, the cumulative direction adopted by the cumulative sum function is from right to left. Indicates that the nodes in the k-1th layer are represented Aggregate context information with the current layer Make a splice. (k) and b (k) are respectively represented as the weight matrix and bias of the linear projection. Softmax(·) represents the normalization operation, which converts the input vector into a probability distribution. This probability distribution represents each position in the node representation as a segmentation point. The probability of . cumsum(·) represents the cumulative sum function. The expected vector obtained is Used to control the updating of node representations in the model.

[0101] In formula (3) represents the gated vector after the SOFTEN operation. Specifically, in order to maintain the order of the segmentation points in the node embedding, a mechanism is needed to ensure that the newly calculated expected vector With the previous expectation vector It has consistency, that is, it satisfies the corresponding nested relationship. However, the "OR" operator (OR) that directly applies bitwise operations is discontinuous in this case, and is therefore not suitable for deep learning models that require continuous gradients. In order to solve this problem, the present invention introduces a differentiable softened version of the "OR" operation, which ensures that the size of the split point conforms to the nested structure of the root number through a differentiable OR operator (SOFTEN). This design allows the model to still retain the personalized characteristics of the nodes in the deep network, so that the model can process graph data in a structured and orderly manner.

[0102] The present invention is to calculate the expected vector Perform the SOFTEN operator (softening is achieved through the SOFTEN(·) function) to obtain the gate vector The SOFTEN operator passes the gate vector of the previous layer and the expected vector of the current layer A differentiable soft version of the OR operation is implemented by performing an element-wise weighted sum, where the weights are determined by the gate vector The complement of is given. In this way, the SOFTEN operator ensures the relative size relationship of the split points, thereby ensuring the order of the split points, while ensuring the differentiability of the model, allowing end-to-end training through optimization algorithms such as gradient descent. By introducing the SOFTEN operator, the NML-GNN model of the present application can effectively maintain the order of the split points in each layer, which is crucial to maintaining the alignment between the root tree structure and the node embedding, and also provides the model with better expressiveness and performance.

[0103] Formula (4) represents the gate vector Used to control the update of subsequent node representations. Specifically, Figure 1 The adaptive gating unit in the Control the information flow and dynamically adjust the information transmission according to the node characteristics. The present invention adds two adaptive parameters α1 and α2 in the step of updating the node representation, which can learn some small offsets during the training process, so as to more finely adjust the gate vector The specific implementation is as follows: The adaptive parameters α1 and α2 are initialized through torch.zeros((1)) as the learnable parameters of the model. During the training process, the adaptive parameters α1 and α2 are automatically updated through back propagation. The design allows the model to automatically find the optimal fine-tuning parameters during the training process without manually setting fixed offsets to optimize model performance.

[0104] Step 3: The graph attention network uses the attention mechanism to dynamically calculate the weights of each node’s neighbor nodes and dynamically update the node’s feature vector.

[0105] First, the graph attention network obtains the attention coefficient through the shared attention mechanism and the Softmax function and normalizes it to ensure that the sum of the neighbor weights is 1. The specific calculation process of the attention coefficient between the node v in the kth layer and its neighbor node u is expressed as follows:

[0106]

[0107] Among them, exp represents the exponential function, and LeakyReLu() represents an activation function that can be used to introduce nonlinearity and avoid gradient disappearance; a (k)Represents a learnable parameter vector, which is one of the key parameters used in the k-th layer of the graph attention network to adjust and control the attention distribution between different nodes; Represents the trainable weight matrix in the k-th layer graph attention network; The feature vector of the graph attention network representing the kth layer of node v, represents the feature vector of the k-1th layer of the graph attention network of node u, and || represents the concatenation operation of the matrix.

[0108] Here the eigenvector The update rule is in is the adjacency matrix of the graph G with self-loops added, I is the identity matrix, is the adjacency matrix The diagonal matrix of . Represents the trainable weight matrix for updating the initial representation of nodes in the k-th layer of the graph attention network, is the bias term, and σ is the nonlinear activation function ReLU. When k=1, the node represents N(v) is the set of neighbor nodes of node v.

[0109] Next, the attention coefficient is used as the weight to dynamically update the feature vector of node v:

[0110]

[0111] in, is the attention coefficient between node v and node u. σ represents a nonlinear activation function, such as the ReLU function.

[0112] The present invention introduces a dual-module architecture, in which the GAT module focuses on dynamically weighting the features of neighbor nodes through a shared attention mechanism, so that the model can dynamically discriminate the importance of relationships. The LayeredGCN module incorporates a hierarchical graph convolution layer, which enables the model to capture local and global structural information. Each layer of the hierarchical graph convolution layer updates the node representation by averaging the features of its neighbors, thereby improving the granularity of capturing information from different graph depths.

[0113] Step 4: Use the multi-head attention mechanism to convert the feature vector of the node output by GAT And the node representation output by the LayeredGCN module Perform concatenation and merging to generate a comprehensive feature vector for each node

[0114] Specifically, in order to enhance the model stability and expressiveness, the present invention adopts a multi-head attention mechanism to weigh and combine the representations from the hierarchical information transfer layer and the graph attention network, which can operate in different representation subspaces and generate a comprehensive feature vector by integrating the outputs of different modules:

[0115]

[0116] in, represents the comprehensive vector generated by node v under the multi-head attention mechanism of the kth layer; P represents the number of multi-head attention heads; tanh represents the hyperbolic tangent function, which is used to perform nonlinear transformation on the output of the attention head; Represents the computation function of the nth attention head at the kth layer; Represents the union of the output vectors of multiple attention heads after being processed by the tanh activation function. Each attention head calculates the node combination embedding Then the results of all P attention heads are merged to form the final comprehensive feature vector of each node v in the kth layer.

[0117] Each attention head is expressed as in, is the weight matrix associated with the nth attention head, D is the dimension of the node embedding to ensure proper scaling, and the softmax function represents the normalization of the attention scores; Represents the input node embedding vector, which is the combined embedding of nodes output by the GAT module and the LayeredGCN layer and The combined result of Represents the weight matrix The transpose of H T represents the transpose of the node embedding vector H.

[0118] The multi-head attention mechanism uses multiple attention heads to capture various patterns in the data from different angles, thereby enriching the feature space. This multiplicity allows for a nuanced synthesis of node embeddings, greatly improving the ability to learn from complex graph structures. The present invention integrates the LayeredGCN module and the GAT module through a multi-head attention mechanism, which helps to represent the graph more comprehensively and more variedly, thereby improving the performance of the model in downstream tasks such as node classification, link prediction, and graph classification.

[0119] Step 5: Use graph contrast learning and node embedding modules to transform the comprehensive feature vector Further optimization is performed to obtain optimized features to improve the representation that is more suitable for downstream tasks.

[0120] Specifically, the NML-GNN model of the present invention introduces a graph contrast learning module, which aims to enhance the commonality of the data of connected nodes in the comprehensive feature vector and minimize the mutual information between unconnected nodes. The specific method is expressed as follows:

[0121]

[0122] Among them, L contrast is the contrast loss function, is the comprehensive feature vector of node v’s neighbor node u, N’(v) represents the set of non-neighbor nodes of node v, and n represents nodes that are not adjacent to node v. sim(·) is a similarity function that can use the dot product or cosine similarity between two vectors, and τ represents a temperature parameter used to scale the similarity score.

[0123] Through the back-propagation algorithm, the node embedding is updated according to the gradient calculated by the contrast loss function to optimize the model parameters. For a pair of positive samples (v,u) and a set of negative samples (v,n), by comparing the positive samples with the negative samples, the model is guided to learn embeddings with closer distances between similar nodes and farther distances between different nodes. contrast The numerator represents the similarity of the positive sample pair, and the denominator represents the sum of the similarities of the neighboring nodes of node v as negative samples and the sum of the similarities of non-neighboring nodes as negative samples. Graph contrastive learning guides the model to learn embeddings that are closer between similar nodes and farther between different nodes by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. Specifically, the model adjusts the node embedding vectors so that the similarity scores of positive sample pairs are higher and the similarity scores of negative sample pairs are lower.

[0124] The present invention uses graph contrastive learning to further refine the feature representation, which enhances the distinguishability of the features and makes them more suitable for downstream tasks.

[0125] In addition, the node embedding module, the conversion process of each module in the kth layer is expressed as: Each node embedding module takes the output of the previous node embedding module as input to further refine and arrange them in sequence. The sequential operation is described as follows: This is the initial comprehensive information passing output. Through this iterative process, node embeddings are continuously refined in successive modules, with each stage improving the model's ability to capture and utilize relevant features. and are the trainable weight matrix and bias vector of the k-th node embedding module, respectively. It is the optimized feature obtained by graph contrast learning. They are the output features obtained by the node embedding modules of the k-1th and kth layers respectively.

[0126] During the training of the model, the comprehensive feature vector of each layer They all contain the information learned by node v in the previous layer, as well as the structure of the graph and the information of neighboring nodes. Through continuous transmission and integration of this information, the model can better capture the relationship between nodes and the characteristics of the graph, thereby improving the performance of the model in tasks such as node classification or graph segmentation.

[0127] The global method based on graph contrastive learning integrates layered graph convolution and graph attention network (GAT) modules, extracts local neighborhood and structural features through layered graph convolution, and combines graph attention network to allocate weights to capture fine-grained relationships and global information, enhances feature representation capabilities, improves model robustness, and improves downstream task performance. It has good learning and processing capabilities for graph structure and node relationships. The node embedding module transforms node features, gradually improves the feature space, enhances task relevance, improves model performance and generalization capabilities, and can be flexibly adjusted according to task requirements to mine node potential features. The combination of the two can learn the features of graph data more comprehensively from the perspective of graph structure and node relationship, as well as from the perspective of node characteristics. This enables the model to learn richer, more accurate and highly task-related feature representations, thereby further improving the performance, robustness and generalization capabilities of the model, and achieving better results in downstream tasks such as node classification and graph segmentation.

[0128] Step 6: Use the multi-head attention mechanism and Softmax function to handle the complex interactions between nodes and obtain the classification prediction of the node through the output layer.

[0129] For each node v in the kth layer, the multi-head attention mechanism is expressed as follows:

[0130]

[0131] in, is the combined output of all P attention heads of node v, Represents the nth attention head. Each attention head calculates its attention score according to the importance of each node feature, and its expression is:

[0132]

[0133] here, and is to apply trainable weights to the output features The query matrix, key matrix and value matrix generated after that, d kis the dimension of the key vector, and the softmax function is applied to each node to obtain the normalized attention score. The softmax function is used in the last layer of the model to convert the summarized node features into probabilities, which is crucial for tasks such as node classification. The softmax output of each node v is calculated as:

[0134]

[0135] in, is the probability distribution of the possible categories of node v, and the probability distribution of the possible categories of node v predicted by the model The category with the highest probability will be used as the model's predicted category for node v. This predicted category is combined with the pre-assigned label y of node v. v If the predicted category is the same as the true label y v If the values ​​of the nodes are the same, it means that the model has correctly classified the node; otherwise, it means that the classification is wrong. By comparing a large number of nodes in this way, performance indicators such as the classification accuracy of the model can be calculated to evaluate the performance of the model in the node classification task. class is the weight matrix, b class is the bias vector of the classification layer. K represents the last layer of the attention mechanism. Represents the output of the multi-head attention mechanism. The softmax function ensures that the output probabilities sum to 1, allowing the model to make clear classification decisions. The class with the highest probability is usually selected as the model's prediction for the node label. Through these two key features - multi-head attention mechanism and softmax, the model not only enhances the ability to learn from complex graph data, but also consolidates the ability to perform effective reasoning in downstream tasks.

[0136] In order to prove the universal applicability of the model, the experiment of the present invention selected three types of data sets, namely, homogeneous data sets, heterogeneous data sets and large data sets.

[0137] For homogeneous data, three literature citation network datasets were used, namely Cora, CiteSeer and PubMed. Five heterogeneous network datasets, namely Texas, Cornell, Wisconsin, Squirrel and Chameleon, were used to evaluate the performance of the model proposed in this invention in heterogeneous environments. In the literature citation network, each node represents a scientific paper, the edge represents the citation relationship, and the node feature is a representation based on word vectors. In the Web network, nodes and edges represent web pages and hyperlinks, and the node feature is a bag-of-words representation of the web page. For large datasets, we selected ogbn-arxiv from the paper [Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J.:'Open graph benchmark: Datasets for machine learning on graphs', Advances in neural information processing systems, 2020, 33, pp. 22118-22133] and arXiv-year from the paper [Lim, D., Li, X., Hohne, F., and Lim, S.-N.:'New benchmarks for learning on non-homophilous graphs', arXiv preprint arXiv:2104.01404, 2021]. Both datasets have nodes and edges representing papers and citations, respectively. In ogbn-arxiv, node labels represent the academic topics of the papers, which tend to be homogeneous settings; while in arXivyear, node labels represent the publication year of the papers, thus showing more heterogeneous label patterns. The edge homogeneity score of each dataset is Hom.h, which represents the homogeneity level. The closer its value is to 1, the more the dataset tends to be homogeneous, and vice versa, it tends to be more heterogeneous. Table 1 summarizes the statistics of the dataset.

[0138] Table 1 Dataset statistics

[0139]

[0140] For the baseline models, including GCN---from the literature [Kipf, TN, and Welling, M.:'Semi-supervised classification with graph convolutional networks', arXiv preprintarXiv:1609.02907, 2016], GAT---from the literature [Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y.:'Graph attention networks', stat, 2017, 1050, (20), pp.10-48550] and GraphSAGE---from the literature [Hamilton, W., Ying, Z., and Leskovec, J.:'Inductive representation learning on large graphs', Advances in neural information processing systems, 2017, 30] and other classic GNN models; also includes GGCN---from the literature [Yan, Y., Hashemi, M., Swersky, K., Yang, Y., and Koutra, D.:'Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks', in Editor (Ed.) ^ (Eds.):'Book Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks'(IEEE, 2022, edn.), pp. 1287-1292] and other methods designed specifically for heterogeneity in recent years; GCNII---from the literature [Chen, M., Wei, Z., Huang, Z., Ding, B., and Li, Y.:'Simple and deep graph convolutional networks', in Editor (Ed.) ^ (Eds.):'Book Simple and deep graph convolutional networks' (PMLR, 2020, edn.), pp.1725-1735]、Geom-GCN---The latest model on the homogeneous citation network dataset from the literature [Pei,H.,Wei,B.,Chang,KC-C.,Lei,Y.,and Yang,B.:'Geom-gcn:Geometric graph convolutional networks',arXiv preprint arXiv:2002.05287,2020]. The present invention also includes models that utilize information from different hops: Mixhop---from the literature [Abu-El-Haija, S., Perozzi, B., Kapoor, A., Alipourfard, N., Lerman, K., Harutyunyan, H., Ver Steeg, G., and Galstyan, A.:'Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing', in Editor (Ed.)^(Eds.):'Book Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing'(PMLR, 2019, edn.), pp. 21-29], Jk-net---from the literature [Xu, K., Li, C., Tian, ​​Y., Sonobe, T., Kawarabayashi, K.-i., and Jegelka, S.:'Representation learning on graphs with jumping knowledge networks', in Editor (Ed.)^(Eds.):'Book Representation learning on graphs with jumping knowledge networks'(PMLR, 2018, edn.), pp.5453-5462]. For baselines with multiple architecture variants, the best one is selected for each dataset and the model is indicated with a small asterisk '*'. .

[0141] The experiments of the present invention use the Adam optimizer, apply dropout and L2 regularization to prevent overfitting, and insert a LayerNorm layer every other layer; the hidden dimension is set to 256 and the block size factor is set to 64. For each data set, unless otherwise specified, an 8-layer model structure is used (which is deeper than most popular GNN models), and grid search is performed on all models to adjust hyperparameters.

[0142] The model proposed in the present invention is evaluated on the 7 data sets in Table 2. For all data sets, the nodes of each category are randomly divided into 60%, 20% and 20% for training, verification and testing, respectively. The results in Table 2 show the average classification accuracy and standard deviation of all models on the test set after 10 random splits. All data are evaluated on the same data split set as the present invention.

[0143] The experimental results show that the NML-GNN model proposed in the present invention consistently achieves the current best technology on both homogeneous and heterogeneous datasets. On homogeneous datasets, the model of the present invention surpasses the performance of the current best technology GCNII model. However, GCNII performs poorly on heterogeneous datasets, with a gap of more than 20% on the Squirrel dataset. The amazing performance on this dataset may be related to the "over-compression" problem. In addition, the model of the present invention achieves the current best technology on heterogeneous datasets, and its performance on all datasets is consistently better than the baseline. The performance of node classification on different datasets proves the generality of the model of the present invention.

[0144] Table 2 Experimental results of node classification on homogeneous and heterogeneous datasets

[0145]

[0146]

[0147] In order to evaluate the ability of the model proposed in the present invention to alleviate the problem of over-smoothing, the experiment set the layer depth to 2, 4, 8, 16, and 32 to evaluate the performance of the model. The experiment selected a benchmark model as a control group: IncepGCN+Drop from the document [Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint arXiv: 1907.10903, 2019.] uses the advanced training technique DropEdge, which is developed to alleviate over-smoothing. The results of the above document are used for all benchmark methods except GAT. For GAT, a separate grid search is performed for each depth to adjust its hyperparameters, and the search space is the same as that of the present invention for fair comparison.

[0148] The performance of the model at different depths is shown in Figure 3 As shown. Figure 3 It can be seen that the model of the present invention achieves the best performance on all datasets and all layers compared to the baseline model. For each dataset, the performance of the model of the present invention does not drop significantly like some baselines. Note that this is achieved without using training techniques such as DropEdge. It can be seen that the hierarchical message passing mechanism and adaptive parameter design can effectively maintain the distinction between nodes, especially in deep networks. This design better alleviates the over-smoothing problem, which shows that the model of the present invention can well alleviate the over-smoothing problem.

[0149] Large datasets present complex linking patterns. To evaluate the model's ability to scale on large datasets, the proposed model is compared with a baseline model and the state-of-the-art model UniMP from the literature [Shi, Y., Huang, Z., Feng, S., Zhong, H., Wang, W., and Sun, Y.:'Masked label prediction: Unified messagepassing model for semi-supervised classification', arXiv preprint arXiv:2009.03509, 2020]. Among them, GCNII is used to process ogbn-arxiv, LINK---from the literature [Lim, D., Li, X., Hohne, F., and Lim, S.-N.:'New benchmarks for learning on non-homophilous graphs', arXiv preprint arXiv:2104.01404, 2021] and H2GCN---from the literature [Zhu, J., Yan, Y., Zhao, L., Heimann, M., Akoglu, L., and Koutra, D.:'Beyond homophilyin graph neural networks: Current limitations and effective designs', Advances in neural information processing systems, 2020, 33, pp. 7793-7804] are used to process arXiv-year. Features, links, and labels are inherited from the ogbn-arxiv dataset, and all models follow the same split data as the ogbn-arxiv and arXiv-year datasets. The benchmark results come from the literature where the two datasets ogbn-arxiv and arXiv-year are located and GGCN. The results are shown in Table 3.

[0150] Table 3 Classification accuracy results of different models on large datasets

[0151]

[0152]

[0153] As can be seen from Table 3, for the heterogeneous dataset Ogbn-arrxiv, the proposed model outperforms all baseline methods except UniMP. This is explainable because UniMP is a large-scale model based on graph Transformer, which uses additional node label information. The performance on these two large datasets demonstrates the scalability and expressiveness of the proposed model in capturing complex patterns in large networks.

[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A network node classification method based on a multi-layer graph neural network model, characterized in that: The steps are as follows: Step 1: Build a graph based on network nodes, preprocess the initial node features in the graph, and obtain the initial embedding vector; Step 2: Based on the initial embedding vectors of the node and its neighboring nodes, a hierarchical message passing mechanism is used to update the node representation; Step 3: Based on the initial embedding vectors of the node and its neighboring nodes, the graph attention network uses the attention mechanism to calculate the weights of each node’s neighboring nodes and dynamically updates the node’s feature vector; Step 4: Use the multi-head attention mechanism to concatenate and merge the feature vector of the node output by the graph attention network and the updated node representation to obtain the comprehensive feature vector of each node; Step 5: Use graph contrast learning and node embedding modules to further optimize the comprehensive feature vector to obtain optimized features; Step 6: Obtain predicted node classification based on optimized features.

2. The network node classification method based on the multi-layer graph neural network model according to claim 1 is characterized in that: The method for constructing a graph is: graph G = (V, E), the node set V is the set of all nodes in the graph G, the edge set E is the set of edges between all pairs of nodes in the graph G, and the adjacency matrix A∈R of the graph G N×N Represents the connection relationship between nodes, and the node features of all nodes constitute the node feature matrix X∈R N×F , where N is the number of network nodes and F is the dimension of node features. Each node v∈V is assigned a label y v ∈Y; The preprocessing method is: nonlinearly projecting the original node features of node v in the node feature matrix X through a multi-layer perceptron to obtain the projection feature Z v ∈R D , D is the dimension of node embedding; for the projection feature Z v Perform linear transformation and normalization to obtain the initial embedding vector of node v Add self-loops to the adjacency matrix A to get the adjacency matrix I is the identity matrix.

3. The network node classification method based on the multi-layer graph neural network model according to claim 1 or 2, characterized in that: The hierarchical message passing mechanism combines the message passing mechanism in GNN with the root tree hierarchy of the root node, aligns the root tree hierarchy with the ordered neurons in the node embedding, and controls the allocation of neurons through the ordered gating mechanism to encode subtrees of different depths for message passing, so that GNN can accurately encode neighborhood information in a certain order.

4. The network node classification method based on the multi-layer graph neural network model according to claim 3 is characterized in that: The method to align the hierarchy of the root tree with the ordered neurons in the node embedding is: 1) Construct the nested structure of the root tree: take the node v currently being processed in the graph G as the root, and the adjacent nodes of node v as child nodes, and recursively construct the root tree; Take the child nodes of the root tree as the root, and build the nested structure of the root tree in this way; 2) Calculate the initial node representation of node v according to the initial embedding vector of node v and the initial embedding vectors of its neighboring nodes, predict the segmentation point position according to the initial node representation of node v, and determine the gating vector according to the segmentation point position; 3) Encode the information in the root tree in the neuron sequence according to the gating vector, so that the hierarchical structure of the root tree is aligned with the ordered neurons in the node representation.

5. The network node classification method based on the multi-layer graph neural network model according to claim 4 is characterized in that: The method for predicting the position of the segmentation point according to the initial node representation of the node v is as follows: the initial node representation of the node v is input into a linear layer to obtain an output vector, a cumulative summation operation is performed on the output vector, and a Softmax function operation is performed on the cumulative summation vector to obtain the probability of each position in the node representation being a segmentation point. Probability of choosing a split point The maximum position is taken as the predicted segmentation point position 6. The network node classification method based on the multi-layer graph neural network model according to claim 5 is characterized in that: The gating vector Before entries are 1, and the rest are 0; Node v is the node representation of the kth layer. Depends only on its k-th root number The nodes within the node represent All neurons in are sorted into a sequence, and for the root tree The information in the first Neuron encoding, for the next level root tree Corresponding to the first neurons, and The initial embedding vector of node v As the first layer input of GNN In each layer of GNN, node v updates its node representation according to the embedding of its neighbor nodes and its own embedding. The node representation of node v in the kth layer of GNN is: Among them, AGGREGATE (k) It is an aggregate function, COMBINE (k) is a combinatorial function; N(v) represents the set of neighbor nodes of node v; is the node representation of neighbor node u in the kth layer of GNN; When node v is at the kth layer, the probability that the i-th position in the node representation becomes a split point is: Where exp represents the exponential function, Represents the element value at the j-th position represented by the node of the k-1th layer of node v, and l represents the index variable for cumulative summation.

7. The network node classification method based on a multi-layer graph neural network model according to any one of claims 4 to 6, characterized in that: The method of controlling the allocation of neurons through an ordered gating mechanism to encode subtrees of different depths for message transmission is: aggregating the initial node representations of the neighboring nodes of node v to obtain the current layer aggregated context information Denote the node at the k-1th layer of node v Aggregate context information with the current layer Processing to obtain the expected vector The gate vector of the previous layer is processed by a differentiable OR operator and the expected vector of the current layer Processing to obtain the expected vector Get each position in the node representation as a split point The cumulative sum of the probabilities of The node representation of the k-1th layer of node v is Aggregate context information with the current layer Perform integration to obtain updated node representation.

8. The network node classification method based on the multi-layer graph neural network model according to claim 7 is characterized in that: The current layer aggregates context information The expected vector The gating vector The updated node representation Among them, MEAN() represents the average operation, and N(v) represents the set of neighbor nodes of node v; is the node representation of neighbor node u in the kth layer of GNN; Indicates that the nodes in the k-1th layer are represented Aggregate context information with the current layer Splicing; W (k) and b (k) They are respectively represented as the weight matrix and bias of linear projection; softmax(·) represents the normalization operation, cumsum(·) represents the cumulative sum function; SOFTEN(·) is a differentiable OR operator, ε represents element-wise multiplication, and α1 and α2 are two adaptive parameters.

9. The network node classification method based on the multi-layer graph neural network model according to claim 8 is characterized in that: The graph attention network uses a shared attention mechanism and a Softmax function to calculate the feature vector of the k-1th layer of the graph attention network of node v. and the feature vector of the k-1th layer of the graph attention network of its neighbor node u The attention coefficient is processed and used as the weight to dynamically update the feature vector of the k-th layer graph attention network of node v; The graph contrastive learning updates node embeddings according to the gradients calculated by the contrastive loss function through the back-propagation algorithm to optimize the model parameters; In step six, a multi-head attention mechanism and a Softmax function are used to process the optimized features and obtain the classification prediction of the node through the output layer.

10. The network node classification method based on the multi-layer graph neural network model according to claim 9 is characterized in that: The feature vector of the k-th layer of the graph attention network for node v in, is the attention coefficient between node v and its neighbor node u, σ represents the nonlinear activation function, Represents the trainable weight matrix of the k-th layer of the graph attention network; The comprehensive feature vector in, represents the comprehensive vector generated by node v under the multi-head attention mechanism of the kth layer; P represents the number of heads of multi-head attention; tanh represents the hyperbolic tangent function; Represents the computation function of the nth attention head at the kth layer; Represents the union of the output vectors of multiple attention heads after being processed by the tanh activation function; The conversion process of the node embedding module in the kth layer is: Each node embedding module takes the output of the previous node embedding module as input to further refine and arrange in order: and are the trainable weight matrix and bias vector of the k-th node embedding module, respectively. It is the optimized feature obtained by graph contrast learning; They are the optimized features obtained by the node embedding modules at the k-1th and kth layers, respectively; The contrast loss function is: in, is the comprehensive feature vector of node v’s neighbor node u, N’(v) represents the set of non-neighboring nodes of node v, n 1 represents nodes that are not adjacent to node v; sim(·) is the similarity function, and τ represents the temperature parameter; The implementation method of step 6 is: for the node v of the kth layer, the multi-head attention mechanism is used to obtain the combined output of all P attention heads: in, Represents the optimization features of each node according to the nth attention head The attention score calculated by the importance of The Softmax function processes the probability distribution of the possible categories of each node v: Among them, W class is the weight matrix, b class is the bias vector of the classification layer, K represents the last layer of the attention mechanism, Represents the output of the multi-head attention mechanism.

Images