Self-adaptive multi-channel deep map neural network representation learning method
By employing an adaptive multi-channel deep graph neural network representation learning method, coarse- and fine-grained receptive fields are constructed and explicit dimensional constraints are introduced. This solves the oversmoothing problem of deep graph neural network models, improves the uniqueness and robustness of node representations, and enhances the model's representation ability in complex networks.
Patent Information
- Application Number
- CN202310588670.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
Existing deep graph neural network models suffer from oversmoothing in network representation learning, causing node representations to become uniform and failing to effectively capture high-order information of complex networks.
An adaptive multi-channel deep graph neural network representation learning method is adopted. By constructing coarse-grained and fine-grained receptive fields, introducing explicit attribute dimension correlation constraints, and combining β weight coefficients, the representation output of different channels is balanced to overcome the oversmoothing problem.
The uniqueness and robustness of node representations are improved under deep model conditions, thereby enhancing the model's representation ability and generalization performance in complex scenarios.
Smart Images

Figure CN119026642A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of network representation learning, and proposes an adaptive multi-channel deep graph neural network representation learning method that alleviates the over-smoothing problem of deep models, with the aim of improving the representation learning performance of the model under deep conditions. Specifically, it involves a multi-channel modeling method for different granularity perspectives of the network and a discriminative aggregation method for different channel information. Background Art
[0002] Network structures, also known as graph data, are ubiquitous in life. From the star networks in the universe to the Internet, social networks, and transportation networks that people use daily, to the biological networks of proteins and neurons in the human body. Network structures make the transmission of information and resources more efficient, and also make people's lives more convenient. As one of the analytical methods for complex networks, network representation learning can capture the similarities between nodes and higher-order information contained in the network, such as semantic relationships and structural relationships. The main purpose of network embedding is to learn low-dimensional vector representations of nodes and capture semantic and structural relationships in the original network. In recent years, with the development of artificial neural networks, graph neural networks have shown strong advantages in the field of network representation learning. However, the classic graph neural network representation learning architecture usually achieves the best results below three layers and cannot be extended to deeper levels. However, deep models can learn complex high-dimensional data representations and have strong generalization capabilities. For example, GCNII proves that the deep model is better than the shallow model. In addition, the scale of graph structure data in real scenarios is increasing. Extending the depth of the model is to better capture the complex structure and hierarchical information of graph data, thereby improving its representation ability and generalization performance in real complex scenarios. Therefore, how to design a deep network representation learning framework is a very important issue at present.
[0003] In response to the over-smoothing problem of deep graph neural network models, the present invention proposes an adaptive multi-channel deep graph neural network representation learning method. Existing graph neural network architectures can usually only be expanded to two or three layers. When the number of layers continues to increase, the node representations will gradually tend to be consistent, which is the over-smoothing problem of deep graph neural network models. The present invention pays attention to the different importance of each dimension within the attribute vector of the node, so each dimension is used to construct a fine-grained receptive field of the node so that the node can learn more unique information. At the same time, the present invention introduces explicit attribute dimension decorrelation constraints to solve the problem of over-correlation between node dimensions caused by the convolution operation of the graph neural network itself. Through the above new perspectives and methods, the present invention allows different nodes to aggregate unique deep information and overcomes the over-smoothing problem of deep models. The present invention compares the model with other representative network representation models on the node classification task to verify the effectiveness and robustness of the model. Summary of the invention
[0004] The present invention aims to alleviate the over-smoothing limitation of graph neural network in the process of network representation learning, and proposes an adaptive multi-channel deep graph neural network representation learning method. The method starts from the two perspectives of coarse-grained and fine-grained, and constructs the best receptive field for each node at each granularity. In the fine-grained channel, by analyzing the importance of each dimension in the attribute vector relative to the complete attribute vector, each dimension attribute will learn a unique retention score, so that each dimension attribute plays a different role and plays its unique role, weakening the impact of the overall change of the complete attribute vector on each dimension. In addition, when the graph neural network is stacked to a deep layer, the over-correlation phenomenon between attribute dimension pairs is often accompanied by the over-smoothing problem, which is an inevitable result of the convolution operation of the graph neural network itself. Therefore, the attribute dimension correlation constraint is also introduced in the fine-grained channel. This explicit decorrelation regularization method avoids the situation where the attribute dimensions are completely correlated. The fine-grained perspective is more biased towards the characteristic information of the node attribute vector, and will ignore part of the original topological information of the network. Therefore, the coarse-grained channel is used to retain the node commonality information contained in the network itself. Finally, the representations of the two channels are combined together using the β weight coefficient. The present invention includes the following specific steps:
[0005] Step 1: Initial attribute conversion preserves the initial features of the nodes. In order to prevent the initial features from being smoothed during the propagation and aggregation process, the present invention uses a multi-layer perceptron to preserve the initial attribute features of the nodes in the network structure:
[0006] Z=MLP(X)
[0007] Among them, Z represents the converted node attributes, MLP represents the multi-layer perceptron, and X represents the initial node attributes.
[0008] Step 2: Construct a fine-grained channel and assign a retention score to each dimension. In the fine-grained channel, the present invention adaptively selects the receptive field based on each dimension of the node representation. For each attribute vector, the present invention assigns an adaptive retention score to each dimension, which is obtained by performing a dot multiplication operation on the learnable projection vector set S and the node representation. The present invention constructs the score result into a diagonal matrix, in which the main diagonal elements are weights and the other positions are 0. The fine-grained channel expression is:
[0009]
[0010]
[0011]
[0012] Where S is a set of trainable projection vectors, is the weight of the d-dimensional vector of layer l, |d|=m. Multiply each dimension of the node attribute vector by the set The node representation is obtained by the corresponding weight in Z. fine Represents node representation under fine-grained channels.
[0013] Step 3: Explicit correlation constraints between dimensions. Assigning different weights to each dimension of the node has overcome the over-smoothing problem to a certain extent, but there are some cases where the fine-grained channel effect is not ideal. After experimental analysis, the present invention starts from the relationship between over-correlation and over-smoothing, and finds that when over-smoothing reaches a certain level, over-correlation will reach the highest value, and at this time the fine-grained channel will be affected. Therefore, the present invention adds explicit attribute dimension decorrelation constraints. Specifically, given a set of attribute dimensions {x1, x2, ..., x d}, Aim to minimize the following loss function:
[0014]
[0015] Among them, the purpose of minimizing the first term is to reduce the covariance between different attribute dimensions. When the first term becomes 0, the different attribute dimensions are uncorrelated. The purpose of minimizing the second term is to make the norm of each attribute dimension become 1. Rewrite the loss function in matrix form as follows:
[0016]
[0017] in And||·|| F is the Frobenius norm.
[0018] Step 4: Construct a coarse-grained channel and assign a retention score to each layer of representation. The present invention uses the complete vector representation of the node as a unit to provide a learnable adaptive weight for the node representation of each layer. In this way, the present invention can adaptively control the contribution of each layer of propagation. These learned weights also reflect the topological information of the graph at a coarse granularity. The coarse-grained channel is described as:
[0019]
[0020] Among them, h (l) Represents the representation vector of the node at layer l, α l represents the adaptive weight of layer l, Z caarse Represents the node representation under the node coarse-grained channel.
[0021] Step 5: The outputs of the two convolutional channels are aggregated using hyperparameters. The present invention chooses to use hyperparameters to balance the results of the two channels. In multi-channel aggregation, the use of hyperparameters instead of adaptive methods is to more stably control the behavior of the model in large-scale learning tasks under deep conditions. If an adaptive aggregation strategy is used, such as adaptive merging weights, the final result will be unpredictable and difficult to control. The use of hyperparameters can control the behavior of the model to a certain extent, making the model more stable. Since it has been proved in the experimental part that the representation under the two channels can effectively overcome the problem of over-smoothing, the present invention pays more attention to the stability and robustness of the model in terms of combination strategy. The specific approach of the present invention is to use hyperparameters to combine the representations under the two channels in a certain proportion, thereby comprehensively utilizing the node representations under different receptive fields of the two channels. The final output can be expressed as:
[0022] Z final =(1-β)Z coarse +βZ fine
[0023] Among them, β is a hyperparameter that balances the results of the two channels. Through experiments and previous experience, the optimal value of β is 0.681.
[0024] Step 6: The present invention focuses on the task of semi-supervised node classification. For this task, the present invention uses the simplest classifier, compares the soft labels obtained by Softmax activation with the true labels, and adopts the standard cross quotient loss function. In addition, there is also a constraint loss for attribute dimension decorrelation. The overall optimization goal of the model is:
[0025]
[0026] Among them, the true label of each node and predicted labels All are represented by One-Hot Embedding. When c = y i Time ic =1, and Y i All other elements in L are zero. D is the loss of attribute dimension correlation constraint, l D is the loss function in the form of a normalized attribute matrix.
[0027] The advantages and beneficial effects of the present invention are:
[0028] 1. By constructing the receptive field of the network from two perspectives, coarse-grained and fine-grained, different dimensions play different roles, which makes up for the limitation of existing solutions that treat attribute vectors as a whole.
[0029] 2. By analyzing the correlation between dimensions and the smoothness of deep node representation, it is proved that the two often occur simultaneously. By adding dimension correlation constraints, the model is well suited to data scenarios under different complex conditions.
[0030] 3. Node classification experiments based on multiple real data sets have verified the effectiveness of the present invention, that is, the new model proposed in the present invention is not only well suited for deep model scenarios, but also able to maintain competitiveness in shallow model scenarios.
[0031] 4. The present invention has broad application prospects in the fields of social network analysis, recommendation system, risk control, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a flow chart of the method of the present invention;
[0033] Figure 2 A comparison chart of node classification accuracy of the present invention on eight real data sets;
[0034] Figure 3 This is a performance diagram of the present invention on the node visualization task; DETAILED DESCRIPTION
[0035] The following is combined Figure 1 The embodiments of the present invention are provided to enable those skilled in the art to better understand the present invention, and do not limit the present invention in any way.
[0036] like Figure 1 As shown, a graph convolutional network representation learning method based on multi-channel modeling includes the following steps:
[0037] Step 1: Initial attribute conversion preserves the initial features of the nodes. In order to prevent the initial features from being smoothed during the propagation and aggregation process, the present invention uses a multi-layer perceptron to preserve the initial attribute features of the nodes in the network structure:
[0038] Z=MLP(X)
[0039] Step 2: Construct a fine-grained channel and assign a retention score to each dimension. The present invention constructs the score result into a diagonal matrix, where the main diagonal elements are weights and the other positions are 0. The fine-grained channel expression is:
[0040]
[0041]
[0042]
[0043] Step 3: Explicit correlation constraints between dimensions. Specifically, given a set of attribute dimensions {x1, x2, ..., x d}, Aim to minimize the following loss function:
[0044]
[0045] Rewrite the loss function into matrix form as follows:
[0046]
[0047] Step 4: Construct a coarse-grained channel and assign a retention score to each layer of representation. The coarse-grained channel is described as:
[0048]
[0049] Step 5: The outputs of the two convolution channels are hyper-parameterized and the final output can be expressed as:
[0050] Z final =(1-β)Z coarse +βZ fine
[0051] Step 6: The present invention focuses on the semi-supervised node classification task. For this task, the present invention adopts the standard cross quotient loss function, in addition to the constraint loss of attribute dimension decorrelation. The overall optimization goal of the model is:
[0052]
[0053] Step 7: In the testing phase, the trained graph convolutional network is used to test the node classification accuracy and visualization effect on seven real-world datasets. The specific implementation details are as follows:
[0054] Table 1 gives the detailed description of seven real-world open source datasets obtained from the Internet, namely Cora, Pubmed, Coauthor-CS, Coauthor-Phisics, Amazon-Computer, Amazon-Photo and Citeseer. Among them: Cora: A citation network with 2708 nodes and 10556 edges. Each node represents a scientific paper. The edge represents the citation relationship between papers. The node attribute is a 1433-dimensional multi-hot word bag, which is divided into 7 different research fields. Citeseer: A citation network with 3327 nodes and 4228 edges. Each node represents a published paper from 6 fields. The edge represents the citation relationship between papers. The node attribute is a 3703-dimensional multi-hot vector. Pubmed: A citation network with 19717 nodes and 44324 edges. Each node represents a scientific publication from three categories. The node attribute is a 500-dimensional real-valued vector. Coauthor: Contains two co-author datasets, Coauthor-CS and Coauthor-Phisics. Coauthor-CS contains 18,333 nodes and 81,894 edges, and Coauthor-Phisics contains 34,493 nodes and 247,962 edges. Each node represents an author in a category, and the edge represents the cooperative relationship between authors. The node attribute is the 6,805-dimensional keyword frequency. Amazon: Contains two collaborative purchase networks, Amazon-Computer and Amazon-Photo. Amazon-Computer contains 13,752 items and 245,861 edges, and Amazon-Photo contains 7,650 items and 119,081 edges. The edges represent the association between items, and the node attribute is a multi-hot word bag from product reviews.
[0055] For each data instance, the training set, validation set, and test set are divided into 38%, 50%, and 12% respectively. The invented model is selected to be compared with several representative baseline models on the test set. All parameters of the baseline method are set to the parameters used by the original model author. In terms of training optimizer, this method uses the widely used Adam as the optimization method of the model, and uses the Pytorch deep learning framework to build the overall model.
[0056] Table 1: Test data
[0057]
[0058] First, we briefly introduce several baseline models used in this method:
[0059] GCN: Fusion of topological structure and node features by aggregating neighborhood information;
[0060] GAT: distinguishes neighbors of different importance through the attention mechanism;
[0061] SGC: Simplified Graph Neural Network;
[0062] GraphSAGE - various aggregators.
[0063] Frameworks designed for the over-smoothness restriction: APPNP, DropEdge, JKnet, and DAGNN.
[0064] Considering that some methods have multiple variants, the present invention selects the best variant model of each method for comparison, and generates experimental results based on the initial settings of each model.
[0065] Figure 2 The accuracy results of the present invention and 9 currently representative baseline methods in node classification experiments are shown. For the convenience of comparison, the best results are shown in bold and the suboptimal results are shown in underline. The present invention achieved the best performance on 7 out of 7 datasets under deep conditions, the best performance on 6 out of 7 datasets under general conditions, and achieved suboptimal performance on the remaining dataset.
[0066] Figure 3 The present invention is compared with four representative baseline methods in the visualization of nodes. The node representation of each model before softmax activation is selected, and t-distributed Stochastic Neighbor Embedding (t-SNE) is used as a dimensionality reduction method, and the nodes after dimensionality reduction are projected into a two-dimensional coordinate plane. Four methods are selected for visualization comparison, namely GCN, GAT, JKnet, APPNP and AMD-GNN proposed in this paper. Different colors represent different types. Due to the over-smoothing problem, the nodes of GCN and GAT have become indistinguishable, and the node category color is only one. The classification effect of JKnet and APPNP is better, because it can be observed that nodes of different colors are divided into different groups. However, JKnet and APPNP still have the phenomenon of blurred category boundaries, far distances between nodes of the same type, and scattered distribution. The visualization results of the method AMD-GNN of the present invention show that nodes of different color categories are almost divided into independent block structures, and the nodes in the block are relatively close, achieving the goal of relatively close nodes of the same type and relatively far away nodes of different types. The visualization effect is consistent with the classification result, which verifies the effectiveness of the present invention from another perspective.
[0067] The above description is only a preferred embodiment of the present invention, which is used to help understand the method and core concept of the present invention. For those skilled in the art, according to the concept of the present invention, there will be changes in the specific implementation method and application scope, so the content of this specification should not be understood as a limitation of the present invention.
Claims
1. The present invention aims to solve the over-smoothing problem of deep graph neural networks in the process of network representation learning, and proposes an adaptive multi-channel deep graph neural network representation learning method. The invention includes the following steps: Firstly, based on the original network structure, a multi-layer perceptron is used to perform attribute conversion for each node, and two adaptive channels are constructed: a fine-grained channel and a coarse-grained channel. Secondly, using the node attributes that retain the initial features, in the fine-grained channel, each dimension of the attribute vector is taken as a unit, and a learnable projection vector set is used to assign a retention score to each dimension, and an explicit correlation constraint between dimensions is added. In the coarse-grained channel, the complete attribute vector is taken as a unit, and the node embedding of each layer is used to construct an adaptive residual link. The fine-grained and coarse-grained encodings obtained from the two channels are passed to the aggregation space, and the output results of the two convolution channels are aggregated through hyperparameters. In each round of training, the aggregation results are compared with the true labels, and the learnable parameters of the four types of convolution channels are updated with the goal of minimizing the cross entropy loss. Finally, in the testing phase, the trained model is used to predict the test set data and compared with the true labels of the test set to calculate the classification accuracy as the evaluation indicator of the model.
2. The adaptive multi-channel deep graph neural network representation learning method as claimed in claim 1, characterized in that The initial features of the nodes are retained. The present invention uses a multi-layer perceptron to perform attribute conversion. The conversion method is as follows: In order to prevent the initial features from being smoothed during the propagation and aggregation process, the present invention uses a multi-layer perceptron to retain the initial attribute features of the nodes in the network structure: Z=MLP(X) Among them, Z represents the converted node attributes, MLP represents the multi-layer perceptron, and X represents the initial node attributes.
3. The adaptive multi-channel deep graph neural network representation learning method as claimed in claim 1, characterized in that Each node in the network is used as the target node to construct a fine-grained channel and assign a retention score to each dimension. The construction method is as follows: In the fine-grained channel, the present invention adaptively selects the receptive field in units of each dimension of the node representation. For each attribute vector, the present invention assigns an adaptive retention score to each dimension, which is obtained by performing a dot multiplication operation on the learnable projection vector set S and the node representation. The present invention constructs the score result into a diagonal matrix, where the main diagonal elements are weights and the other positions are 0. The fine-grained channel expression is: Where S is a set of trainable projection vectors, is the weight of the d-dimensional vector of layer l, |d|=m. Multiply each dimension of the node attribute vector by the set The node representation is obtained by the corresponding weight in Z. fine Represents node representation under fine-grained channels.
4. The adaptive multi-channel deep graph neural network representation learning method as claimed in claim 1, characterized in that Each node in the network is used as the target node to construct a fine-grained channel and to impose explicit correlation constraints on the dimensions. The constraints are as follows: Assigning different weights to each dimension of the node has overcome the over-smoothing problem to a certain extent, but there are also some cases where the fine-grained channel effect is not ideal. After experimental analysis, the present invention starts from the relationship between over-correlation and over-smoothing and finds that when over-smoothing reaches a certain level, over-correlation will reach the highest value, and at this time the fine-grained channel will be affected. Therefore, the present invention adds explicit attribute dimension decorrelation constraints. Specifically, given a set of attribute dimensions {x1, x2, ..., x d }, Aim to minimize the following loss function: Among them, the purpose of minimizing the first term is to reduce the covariance between different attribute dimensions. When the first term becomes 0, the different attribute dimensions are uncorrelated. The purpose of minimizing the second term is to make the norm of each attribute dimension become 1. Rewrite the loss function in matrix form as follows: in And||·|| F is the Frobenius norm.
5. The adaptive multi-channel deep graph neural network representation learning method as claimed in claim 1, characterized in that Take each node in the network as the target node, construct a coarse-grained channel, and assign a retention score to each layer of representation. The construction method is as follows: The present invention uses the complete vector representation of the node as a unit to provide a learnable adaptive weight for the node representation of each layer. In this way, the present invention can adaptively control the contribution of each layer of propagation. These learned weights also reflect the topological information of the graph at a coarse-grained level. The coarse-grained channel is described as: Among them, h (l) Represents the representation vector of the node at layer l, α l represents the adaptive weight of layer l, Z caarse Represents the node representation under the node coarse-grained channel.
6. The adaptive multi-channel deep graph neural network representation learning method according to claim 1, characterized in that The outputs of the two convolution channels are hyper-aggregated in the following way: The present invention chooses to use hyperparameters to balance the results of the two channels. In multi-channel aggregation, the use of hyperparameters instead of adaptive methods is to more stably control the behavior of the model in large-scale learning tasks under deep conditions. If an adaptive aggregation strategy is used, such as adaptive merging weights, the final result will be unpredictable and difficult to control. The use of hyperparameters can control the behavior of the model to a certain extent, making the model more stable. Since it has been proved in the experimental part that the representation under two channels can effectively overcome the over-smoothing problem, the present invention pays more attention to the stability and robustness of the model in the combination strategy. The specific approach of the present invention is to use hyperparameters to combine the representations under the two channels in a certain proportion, thereby comprehensively utilizing the node representations under different receptive fields of the two channels. The final output can be expressed as: Z final =(1-β)Z coarse +βZ fine Among them, β is a hyperparameter that balances the results of the two channels. Through experiments and previous experience, the optimal value of β is 0.
681.
7. The adaptive multi-channel deep graph neural network representation learning method as claimed in claim 1 focuses on the semi-supervised node classification task. For this task, this paper uses the simplest classifier, compares the soft labels obtained by Softmax activation with the true labels, and adopts the standard cross quotient loss function. In addition, there is a constraint loss for attribute dimension decorrelation. The overall optimization goal of the model is: in, The true label of each node and predicted labels All are represented by One-Hot Embedding. When c = y i Time ic =1, and Y i All other elements in L are zero. D is the loss of attribute dimension correlation constraint, l D is the loss function in the form of a normalized attribute matrix.
8. According to the adaptive multi-channel deep graph neural network representation learning method as described in claim 1, in the testing phase, the test data is input into the trained model, the label prediction results are output, and the node classification accuracy is calculated as a measure of model performance.
