A node classification method combining high-order structure and network representation with attention mechanism

Abstract

The present invention belongs to the field of network representation learning, and discloses a node classification method for network representation that combines high-order structures and attention mechanisms. Considering the applicability of different high-order structures in the network to the network, different types of network data can be adaptively learned, alleviating the suboptimal performance problem caused by not considering the applicability of high-order structures in network representation learning. First, data preparation and data preprocessing are performed; second, the attribute features and structural features of the data are extracted respectively using the graph convolutional network layer; then, the similarity of the attribute features and structural features is learned using the attention mechanism, and the representation of each node neighborhood in the network is aggregated according to the similarity; then, the graph convolution operation is used to convert it into the representation of the downstream task; finally, the loss function is used to continuously update the parameters of the algorithm model until the optimal algorithm model is obtained, and an efficient network representation is obtained, which can be applied to spam recognition, fake news detection and disease prediction.

Description

Technical Field

[0001] The present invention belongs to the field of network representation learning, and relates to a node classification method of network representation combining high-order structure and attention mechanism, which can be used for node classification and link prediction in social networks. Background Art

[0002] Many complex systems can be represented in the form of networks, such as social networks, biological networks, and transportation networks. Network representation learning learns a low-dimensional dense representation vector for each node and applies it to common network analysis tasks, such as node classification and link prediction. These network analysis tasks have shown great potential in application areas such as spam identification, fake news detection, and disease prediction.

[0003] The current mainstream network representation learning algorithms in social networks can be divided into the following categories:

[0004] Network representation learning algorithm based on matrix decomposition. For example, Yang et al. proposed TADW in 2015, which added text feature learning network representation based on matrix decomposition; Cao et al. proposed GraRep in 2015, which can handle weighted networks and integrate the global structural information of the network. The main idea of ​​the network representation learning algorithm based on matrix decomposition is to decompose the matrix into a combination of multiple simple matrices. These simple matrices have low dimensions and can be used to represent the original information of the network, such as the relationship between nodes, and can be used as node representations of the network. However, when the matrix is ​​too large, the matrix decomposition algorithm requires a lot of memory. In addition, the matrix decomposition algorithm is not suitable for supervised and semi-supervised tasks.

[0005] Network representation learning algorithms based on random walks. For example, the DeepWalk algorithm proposed by Perozzi et al. in 2014 treats nodes as words, generates random sequences of nodes, and uses the Word2vec model to learn the representation of nodes; in Node2vec proposed by Grover and Leskovec in 2016, the network is sampled by adjusting the breadth-first and depth-first related parameters. These algorithms all use random walk methods to generate node sequences in the network structure while maintaining the connection between nodes. At the same time, feature vectors of vertices can be generated so that downstream tasks can mine network information in low-dimensional space. Random walks play an important role in data dimensionality reduction. However, this method relies too much on the walking strategy, which will produce uncertain node relationships and affect the stability of the network representation learning effect. Although some random walk methods can retain local and global information of the network, they cannot effectively adjust parameters to adapt to different types of networks.

[0006] Network representation learning algorithms based on deep learning. For example, Kipf and Welling proposed the graph convolutional network GCN in 2017, which applied the convolution operator to network data through first-order approximation; Velickovic et al. proposed the graph attention network GAT in 2018, which used the self-attention mechanism to assign different weights to the neighbors of nodes, effectively aggregating neighborhood features. It has become a trend to extend existing deep learning models to network data. In recent years, many network representation learning models based on deep learning have been proposed, such as graph convolutional networks, graph attention networks, graph autoencoders, etc. Deep learning enhances the applicability and performance of network representation learning algorithms, but is easily affected by overfitting and over-smoothing problems.

[0007] Network representation learning algorithms that incorporate high-order structures. For example, Rossi et al. proposed HONE in 2018, which used different network motifs to learn network representations; Xu et al. proposed MORE in 2020, which combined network motifs and attribute information to achieve good performance in downstream tasks. Network representation methods based on high-order structures model structures such as subgraphs. Subgraphs can represent high-order structural granularity information, thereby enhancing the learned network representation features. However, existing methods process different high-order structural information in the same way, and fail to fully tap the role of different types of network high-order information. Summary of the invention

[0008] Most existing network representation learning algorithms do not consider the useful information contained in the connection patterns of high-order structures, which limits the quality of learned node representations. In the methods that consider using high-order connection patterns, they ignore the fact that the importance of high-order structures to network representation learning varies in different scenarios, and fail to bring into play the useful information of these high-order patterns.

[0009] In view of the problems existing in the prior art, the purpose of this invention is to overcome the suboptimal performance problem caused by not considering the applicability of high-order structures in network representation learning, and propose a network representation learning algorithm that combines high-order structures and attention mechanisms. It can effectively use the attention mechanism to adaptively learn the high-order connection patterns existing in the network to adapt to the differences in high-order connectivity patterns in different scenarios, thereby generating effective network representations for downstream classification tasks. We learn information about network structure and attributes, and then adaptively fuse them together through the attention mechanism.

[0010] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0011] A node classification method based on a network representation that combines high-order structures and an attention mechanism. The network representation learning algorithm first preprocesses the data to obtain the attribute information matrix and the structure information matrix of the data; secondly, the attribute features and structure features of the nodes in the network data are extracted using the graph convolution formula respectively; then, the attention mechanism is used to calculate the scores and aggregate the node neighborhoods to obtain the aggregated new node representation, which is converted into a probabilistic form that meets the downstream classification task using the graph convolution operation; finally, the loss function is used to continuously update the parameters of the algorithm model until the optimal algorithm model is obtained and used for the downstream classification task; the steps are as follows:

[0012] Step (1): Data preparation and data preprocessing, after processing, the attribute information matrix X of the data is obtained a and the structural information matrix X s ;

[0013] 1) Check whether the attribute information of the data set exists. If it exists, it remains unchanged. Otherwise, the unit matrix is ​​used. As attribute information of network data Where N is the number of nodes in the network data, d a is the dimension of attribute information;

[0014] 2) For the topological information of the network, the motif node degree formula is used to calculate the 2-4 order motif node degree value of each node and use it as the structural information matrix of the network data set

[0015] 3) For the attribute information matrix X a and the structural information matrix X s , respectively, and row standardization is performed; the attribute information matrix standardization operation is Where X a [i,j] represents X a The value of the i-th row and j-th column in the structure information matrix is Where X s [i,j] represents X s The value of row i and column j in ;

[0016] Step (2): Extract the attribute feature matrix H from the network data a And the structural feature matrix H s ;

[0017] 1) Get the data adjacency matrix A ij =1 means node i and node j are connected, and 0 means they are not connected. The adjacency matrix after adding self-loop connections to each node is in is the unit matrix, representing the self-loop connection of the node, and the degree matrix of the node of the graph is obtained in

[0018] 2) Use the graph convolution layer to extract the network’s attribute feature matrix H a , the graph convolution formula is as follows:

[0019]

[0020] Where σ(·) represents a nonlinear function, and ReLU(x)=max{0,x} is used as the function. a Represents the trainable weights of the graph convolution layer.

[0021] Use graph convolutional layers to extract the structural feature matrix H of the network s , the graph convolution formula is as follows:

[0022]

[0023] Where σ(·) represents a nonlinear function, and ReLU(x)=max{0,x} is used as the function. s Represents the trainable weights of the graph convolution layer;

[0024] Step (3): Use the attention mechanism to aggregate the representation of the node neighborhood to obtain a new node representation matrix H att ;

[0025] 1) For node i, the cosine similarity function is used to calculate the attention score of the neighborhood:

[0026] e ij =βcos(H a [i,:],H s [j,:])

[0027] in, is a learnable scaling factor;

[0028] 2) For the attention score e ij Perform normalization operation:

[0029]

[0030] in, Represents the set of neighboring nodes of node i;

[0031] 3) Use the attention score to aggregate the domain of node j to obtain a new node representation H att [i,:], the specific operations are:

[0032]

[0033] Step (4): Use graph convolution operations to convert into the representation matrix H of the downstream task out ;

[0034]

[0035] Among them, σ(·) uses ReLU(x)=max{0,x}, W out Represents the trainable weights of the graph convolution layer;

[0036] Step (5): Use the loss function to continuously update the parameters W of the algorithm model until the optimal algorithm model is obtained;

[0037] 1) Using the softmax function, the downstream task network nodes are represented by H out The probability function of node classification converted to

[0038]

[0039] 2) The loss function used in the optimization process is cross entropy loss

[0040]

[0041] in, is the set of labeled data indexes in the training set, Y is the real label of the data, and there are C categories in total; the weight parameters are updated through training to achieve continuous optimization of the model, and the training loss is After convergence, the optimal algorithm model is obtained.

[0042] Compared with the existing algorithms, the present invention has the following beneficial effects: the present invention mainly uses network motifs to describe the high-order structure in the network as structural information, uses the attention mechanism to effectively learn the correlation between node attributes and structures in the network, and learns efficient network node representation, showing superior performance in downstream classification tasks. It overcomes the suboptimal performance problem caused by not considering the applicability of high-order structures in network representation learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the basic framework of the present invention.

[0044] Figure 2 This is a schematic diagram of the structural attention mechanism in the present invention. DETAILED DESCRIPTION

[0045] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0046] In order to make the technical problems solved by the present invention, the technical solutions adopted and the technical effects achieved clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all the contents.

[0047] It is divided into five steps: (1) data preparation and data preprocessing; (2) extracting attribute features and structural features of network data; (3) using the attention mechanism to aggregate node neighborhood representations; (4) converting the learned node representations into a form suitable for downstream tasks; (5) using the loss function to repeatedly optimize until the optimal algorithm model is obtained;

[0048] The first step is data preparation and data preprocessing to make the data distribution more suitable for the needs of the task.

[0049] 1) Check whether the attribute information of the data set exists. If so, normalize the attributes. Otherwise, use the diagonal unit matrix I as the attribute information X of the network data. a

[0050] 2) For the topological information of the network data, the motif node degree formula is used to calculate the 2-4 order motif node degree value of each node and use it as the structural information X of the network data set. s .

[0051] 3) For attribute information X a and structural information X s , and perform row standardization respectively. The attribute information normalization operation is The structural information normalization operation is

[0052] The second step is to extract the attribute features and structural features of the network data for subsequent calculation of the attention mechanism.

[0053] 1) Using graph convolutional layers Extract attribute features.

[0054] 2) Using graph convolutional layers Extract structural features.

[0055] The third step is to aggregate node neighborhood representations using the attention mechanism.

[0056] 1) Calculate the neighborhood attention score e of node i using the cosine similarity formula and scaling factor ij =β·cos(H a [i,:],H s [j,:]).

[0057] 2) Use the softmax function to normalize the attention score

[0058] 3) Aggregate node domains using scores calculated based on the structural attention mechanism

[0059] The fourth step is to transform the learned node representation into a form suitable for downstream tasks.

[0060] Using the graph convolution formula Transform node representations into a form suitable for downstream tasks.

[0061] The fifth step is to use the loss function to perform repeated optimization until the optimal algorithm model is obtained.

[0062] 1) In the node classification task, the downstream task node representation is converted into a probability function of node classification

[0063] 2) Set the loss function of the node classification task to the cross entropy function

[0064] 3) Continuously train and update the model weights to obtain the optimal algorithm model after the loss function converges.

[0065] In combination with the scheme of the present invention, the experimental analysis is carried out as follows:

[0066] There are many high-order structures in the network represented by social network. Compared with binary connections, ternary or even multivariate complex connection patterns are widely present in social networks. This connection pattern is widely used in scenarios such as anomaly detection and social and academic network analysis, and is of great significance in spam classification, anomaly detection and other fields. Therefore, experiments and related comparisons can be conducted on social network datasets to prove the effectiveness of this model.

[0067] (1) Introduction to the use of network motifs and social network datasets

[0068] At present, a large number of high-order connection patterns with different structures have been found in the research of social networks, academic networks, transportation networks, etc. This connection pattern plays an important role in the network. In social networks, in addition to the connection between two users, there is a ternary closure composed of the mutual connection between three users, namely the M31 type motif (Table 1), which means that these three users know each other and form a small group with a close relationship. This connection pattern can reflect the network structure of individuals in social networks and is common in social networks. Network motifs appear frequently in real networks. In real networks and random networks with the same number of nodes and edges, the frequency of network motifs in the former is much higher than that in the latter. The basic situation of the network motifs used in the experiment is shown in Table 1.

[0069] Table 1 Network motif information used

[0070]

[0071] This model is tested on four public social network datasets: Cora, Citationv1, DBLPv7, and ACMv9. The task is node classification. The detailed information of the datasets is shown in Table 2.

[0072] Table 2 Dataset statistics

[0073]

[0074]

[0075] The Cora dataset is a paper citation network. Nodes are published articles, edges are citation links, and the feature of the node is a 1433-dimensional one-hot vector, whose value represents whether each word generation exists in the paper. All nodes in the network are divided into 7 categories with a total of 2708 samples. Citationv1, DBLPv7 and ACMv9 are 3 citation networks extracted from the AMiner dataset, from Microsoft, DBLP and ACM respectively. In these datasets, each node represents an article, and each edge represents the citation link of the article. Among them, Citationv1 has a total of 8935 nodes, DBLPv7 has a total of 5484 nodes, and ACMv9 contains 9360 nodes. These three datasets all contain 5 categories.

[0076] (2) Experimental results comparing this method with other excellent methods

[0077] The comparison of the experimental results of this method with other excellent models in the social network node classification task in the field of network representation learning is shown in Table 3. Among them, MLP is a classic neural network model in deep learning, which completes various complex tasks by stacking multiple linear and nonlinear transformations; GCN is a graph convolutional network model for semi-supervised learning, which learns the representation of the network by aggregating the information of neighborhood nodes. SGC removes unnecessary complex and redundant calculations in GCN, which speeds up the calculation speed while effectively reducing the performance degradation. GAT is a graph neural network model that uses the attention mechanism to aggregate the features of neighborhood nodes. Its efficient performance has achieved the best results in many tasks. MORE uses simple operators to aggregate the structural information and attribute information of nodes to learn the representation of nodes.

[0078] Table 3 Node classification accuracy results on the dataset

[0079]

[0080] By comparing the experimental results, it can be seen that this method uses network motifs to describe the high-order connection patterns that exist in large numbers in the network as structural information, uses the attention mechanism to effectively learn the correlation between node attributes and structures in the network, and learns efficient network node representations, showing good performance in downstream classification tasks.

[0081] The above-described embodiments merely express the implementation methods of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A node classification method based on network representation combining high-order structure and attention mechanism, characterized in that: For social network node classification tasks, nodes are published articles, edges are citation links, and node features are one-hot vectors, whose values ​​represent whether each word exists in the paper. First, the data is preprocessed to obtain the attribute information matrix and structure information matrix of the data. Second, the graph convolution formula is used to extract the attribute features and structure features of the nodes in the network data. Then, the attention mechanism is used to calculate the scores and aggregate the node neighborhoods to obtain the aggregated new node representation, which is converted into a probabilistic form that meets the downstream classification task using the graph convolution operation; finally, the loss function is used to continuously update the parameters of the algorithm model until the optimal algorithm model is obtained and used for downstream classification tasks; the steps are as follows: Step (1): Data preparation and data preprocessing, after processing, the attribute information matrix X of the data is obtained a and the structural information matrix X s ; 1) Check whether the attribute information of the data set exists. If it exists, it remains unchanged. Otherwise, the unit matrix is ​​used. As attribute information of network data Where N is the number of nodes in the network data, d a is the dimension of attribute information; 2) For the topological information of the network, the motif node degree formula is used to calculate the 2-4 order motif node degree value of each node and use it as the structural information matrix of the network data set 3) For the attribute information matrix X a and the structural information matrix X s , respectively, and row standardization is performed; the attribute information matrix standardization operation is Where X a [i,j] represents X a The value of the i-th row and j-th column in the structure information matrix is Where X s [i,j] represents X s The value of row i and column j in ; Step (2): Extract the attribute feature matrix H from the network data a And the structural feature matrix H s ; 1) Get the data adjacency matrix A ij =1 means node i and node j are connected, and 0 means they are not connected. The adjacency matrix after adding self-loop connections to each node is in is the unit matrix, representing the self-loop connection of the node, and the degree matrix of the node of the graph is obtained in 2) Use the graph convolution layer to extract the network’s attribute feature matrix H a , the graph convolution formula is as follows: Where σ(·) represents a nonlinear function, and ReLU(x)=max{0,x} is used as the function. a Represents the trainable weights of the graph convolution layer; Use graph convolutional layers to extract the structural feature matrix H of the network s , the graph convolution formula is as follows: Where σ(·) represents a nonlinear function, and ReLU(x)=max{0,x} is used as the function. s Represents the trainable weights of the graph convolution layer; Step (3): Use the attention mechanism to aggregate the representation of the node neighborhood to obtain a new node representation matrix H att ; 1) For node i, the cosine similarity function is used to calculate the attention score of the neighborhood: e ij =β·cos(H a [i,:],H s [j,:]) in, is a learnable scaling factor; 2) For the attention score e ij Perform normalization operation: in, Represents the set of neighboring nodes of node i; 3) Use the attention score to aggregate the domain of node j to obtain a new node representation H att [i,:], the specific operations are: Step (4): Use graph convolution operations to convert into the representation matrix H of the downstream task out ; Among them, σ(·) uses ReLU(x)=max{0,x}, W out Represents the trainable weights of the graph convolution layer; Step (5): Use the loss function to continuously update the parameters W of the algorithm model until the optimal algorithm model is obtained; 1) Using the softmax function, the downstream task network nodes are represented by H out The probability function of node classification converted to 2) The loss function used in the optimization process is cross entropy loss; in, is the set of labeled data indexes in the training set, Y is the real label of the data, and there are C categories in total; the weight parameters are updated through training to achieve continuous optimization of the model, and the training loss is After convergence, the optimal algorithm model is obtained.

Images