A graph neural network link prediction method oriented to node multi-dimensional representation

Abstract

A graph neural network link prediction method for node multi-dimensional representation belongs to the technical field of recommendation systems. First, the node attribute feature information is segmented in Chinese by using a Chinese corpus to expand the semantic representation of the node. At the same time, a pre-training direct inference graph convolution algorithm is established to obtain another dimension of node representation. Then, the splicing of the two node feature vectors is realized by self-defined initialization weight coefficients, and the node semantic vector and the pre-training vector are mapped in the user side and the item side vector space. The inductive graph convolution neural network algorithm SAGE is used to realize neighbor node sampling and aggregation, and the information of node degree is added in the SAGE framework to improve the probability of sampling of nodes with small degree and reduce the probability of sampling of nodes with large degree. The machine learning classification algorithm LGB is used as a downstream classification task to realize the multi-modal cross-domain link prediction task.

Description

Technical Field

[0001] This invention belongs to the field of recommendation system technology, and particularly relates to knowledge graphs, graph convolutional networks, deep learning link prediction methods, etc. Background Technology

[0002] In recent years, many researchers have developed numerous methods based on graph convolutional networks (GNNs) for applications such as graph representation learning, node classification, and link prediction. Graph (network) structured data can capture information between nodes and their edges by modeling the nodes and edges in the graph. Graph structured data has been widely applied in many fields, such as social networks, knowledge graphs, and biosciences. The goal of graph neural networks (GNNs) is to enable each vertex in the graph to learn an embedding containing information about its neighborhood. For example, to predict whether there is a connection between users and items in a social network, a graph convolutional network (Conv and ReLU) is executed on the social network to obtain user vectors and item vectors. Finally, the link probability of the edges is predicted using vector similarity calculations. Therefore, graph convolutional networks have received widespread attention for solving the problem of node link prediction in social networks.

[0003] Previous research has mainly focused on designing a graph kernel function to compute the relationships between edges in a graph. While graph kernel functions can achieve good results, they do not fully utilize the semantic relationships between user features and item features. Network methods lack effective utilization of local feature information and neglect the expressive power of node features. It is well known that Graph Convolutional Networks (GCNs) represent the information of a node by aggregating its local feature information. Therefore, effectively extracting the local expressive power of nodes is key to improving the link prediction performance of GCNs. To improve the representational power of node features, Kanavos and Livieris constructed a user relationship graph based on user-attention feature information to compute the user's influence on a social platform. Ning et al. proposed a Heterogeneous Graph Network (HGCN) that learns node representations of users and keyword nodes by learning heterogeneous information between user nodes and key nodes, enhancing information exchange between different types of nodes. Fan et al. proposed a structured self-attention architecture similar to Graph Attention Networks (GAT) for graph structure learning. In this architecture, a node-centric attention mechanism aggregates features from different neighboring nodes with learnable weights, bringing together important features from different nodes and depths into the model's output. However, existing algorithms lack user and item-side attribute features, such as age, occupation, and income, and fail to mine potential information from user characteristics and item descriptions. Furthermore, they do not consider statistical and machine learning models as downstream classification tasks to enhance the model's generalization ability. Summary of the Invention

[0004] The technical problem this invention aims to solve is a graph neural network link prediction method oriented towards multidimensional node representation. First, Chinese word segmentation is performed on node attribute feature information using a Chinese corpus to amplify the semantic representation of nodes. Simultaneously, a pre-trained transductive graph convolution algorithm is established to obtain another dimension of node representation. Then, by custom initializing weight coefficients, the feature vectors of the two nodes are concatenated, completing the mapping of node semantic vectors and pre-trained vectors in the user-side and item-side vector spaces. The inductive graph convolutional neural network algorithm SAGE is used to sample and aggregate neighbor nodes. Node degree information is added to the SAGE framework, increasing the probability of sampling nodes with lower degrees and decreasing the probability of sampling nodes with higher degrees. The machine learning classification algorithm LGB is used as a downstream classification task, thereby achieving a multimodal cross-domain link prediction task.

[0005] This invention presents a graph neural network link prediction method oriented towards multidimensional node representation, proposing a computational framework based on the graph convolutional network SAGE. In the data preprocessing stage, attribute information from both the user and item sides is introduced, including user registration information, interests, and Chinese names of items. Chinese word segments representing key node information are extracted from the text corpus using a word segmentation system. Then, a deep learning word segmentation framework is used to obtain a node word vector table. Simultaneously, a directive algorithm (node2vec) captures the distance relationships between nodes, obtaining the node traversal sequence through a biased random walk rule. The node sequence is then trained using word2vec to obtain the node vector representation. Since the node semantic vector and the pre-trained vector undergo adaptive weight concatenation during model training, the word vector representation of each node in all training samples includes not only local attributes (node ​​semantic vector) but also the association information between nodes (pre-trained vector). Then, a graph embedding algorithm samples the neighbor information of each node. By aggregating the information of neighboring nodes, each vertex can learn the attribute features of neighboring nodes and the nonlinear relationships in the graph convolutional network. As the structural complexity of graphs increases over time, machine learning algorithms are used to train the SAGE algorithm to fit the output of the last fully connected layer. This effectively solves the problems of local optima and high time complexity of the SAGE algorithm, while fully obtaining accurate node representations to achieve the best prediction results.

[0006] To achieve the above objectives, this invention adopts the following technical solution: To better implement the entire method, Python is chosen as the foundation, and PyTorch is used as the deep learning framework for writing graph neural networks. To better identify the Chinese meaning of sentences within nodes, the NLPIR deep learning word segmentation framework system is first used to accurately segment the Chinese semantics of the nodes. Then, the BERT model proposed by Google is used to train word vectors for text segmentation. node2vec is used as a pre-trained model to obtain word vectors containing the association information between nodes. The initial weights for concatenating the word vectors are set to [0.7, 0.3], and an adaptive aggregation method is used... i =concat(w1*emb1, w2*emb2) is used to update weights. The information propagation (Message) and aggregation (Aggregation) stages of the graph convolutional neural network algorithm are implemented using the open-source deep learning framework PyTorch. The LightGBM machine learning model is used to predict links between nodes. A classifier learns the parameters of the last layer of the graph convolutional network, resulting in a graph neural network link prediction method, SAGE-LGB, oriented towards multi-dimensional node representations. This is achieved through 10-fold training on the dataset. The parameters of the SAGE-LGB model are as follows: input layer node vector dimension is 128, pre-trained model is node2vec, random walk parameters p = 0.25, q = 0.25, initial weight coefficients weight = [0.7, 0.3], number of sampling layers k = 3, output layer node dimension is 64, aggregation function is Sum-Pooling, dropout to prevent overfitting = 0.1, learning rate = 0.2, number of iterations epochs = 100, and maximum tree depth is 8.

[0007] A graph neural network link prediction method oriented towards node multidimensional representation includes the following steps:

[0008] Step 1: Obtain sample data from the data source and use the NLPIR word segmentation system to perform accurate Chinese word segmentation on the sample attribute file;

[0009] Step 2: Obtain the word vector representation of each Chinese word segment using BERT.

[0010] Step 3: Model pre-training. The node vector representation is obtained using the transductive learning algorithm node2vec and concatenated with the Chinese word vector obtained in Step 2. Adaptive weight parameters are defined, and the concatenated vectors are used to obtain the input representation of the node.

[0011] Step 4: Model building. Use the custom inductive algorithm SAGE framework to learn graph embeddings, and use the word vectors obtained in Step 3 as the input to the model.

[0012] Step 5: Based on the node vectors obtained in Step 4, use the LightGBM machine learning algorithm model to predict downstream links, perform 10-fold cross-validation on the dataset, and select the best prediction result according to the score.

[0013] Preferably, step 3 specifically includes the following steps:

[0014] Step 3.1: Model pre-training. The node2vec algorithm is used to pre-train the constructed network topology graph to obtain the preliminary vector representation of the nodes. Adding a pre-trained model can effectively improve the accuracy of the downstream model and make the model more likely to converge.

[0015] Step 3.2: Word vector concatenation. The sum of the Chinese word vectors of each node is then concatenated with the pre-trained node vectors obtained in Step 3.1 using adaptive weights (concat(w1*emb1, w2*emb2)). The word vectors from Chinese word segmentation enrich the node attribute information, while the pre-trained node vectors expand the link information between nodes, thus improving prediction accuracy.

[0016] Preferably, step 4 specifically includes the following steps:

[0017] Step 4.1: Sample the neighbors of the node. The study found that the model with a sampling layer of 2 in the training set and the validation set has the best prediction effect, indicating that the central node can aggregate the node information of its second-order domain into the embedding representation of its own node.

[0018] Step 4.2: Aggregate the feature information of the sampled neighbors. For each node, the features of the neighboring nodes can be aggregated by the aggregation function, and then concatenated with the features of the previous layer to obtain the feature information of the current layer.

[0019] Step 4.3: Adjust the probability of negative sampling of nodes in the network graph. Add node degree information to the SAGE framework to increase the probability of sampling nodes with low degree and decrease the probability of sampling nodes with high degree.

[0020] Compared with the prior art, the present invention has the following significant advantages:

[0021] This invention utilizes the transductive algorithm node2vec for pre-training and uses NLPIR to automatically segment and label domain-specific terms. This allows us to obtain a set of segmented terms for explanatory nodes. The BERT model is then used to pre-train the Chinese word segmentation of nodes to obtain word vector representations. These word vectors are then concatenated with the node vectors obtained from the transductive algorithm as input to the downstream model. Adding semantic information to nodes allows us to capture differences in node words, thus increasing the model's representational ability and improving prediction performance. An adaptive concatenation method is used to improve prediction accuracy. The SAGE graph convolutional network framework is constructed and implemented to extract the embedding representations of nodes. By obtaining information about each node's neighboring nodes and aggregating the feature information of the sampled neighbors, the vector representation of the node in the current layer is updated. During model parameter initialization, adding the node word vectors obtained from the pre-trained model efficiently achieves rapid learning of node information, accelerating model convergence. Simultaneously, adding the pre-trained model effectively reduces the number of parameters in the downstream decoder stage, improving model training speed. Adding a downstream machine learning classifier effectively increases the model's classification accuracy. The downstream part of the model incorporates LightGBM machine learning as a classifier, which significantly improves the prediction of node pairs. Overall, the method of this invention can predict network node links relatively efficiently and accurately, and has the advantages of being suitable for massive amounts of data and having strong generalization ability, making it widely applicable to recommendation systems and social networks. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method involved in the present invention;

[0023] Figure 2 This is a schematic diagram of the pre-trained modeling structure of the present invention;

[0024] Figure 3 This is a schematic diagram illustrating the word vector representation concatenation at the input layer in this invention;

[0025] Figure 4 This is a diagram showing the word vector weight splicing ratio in this invention;

[0026] Figure 5 This is a network schematic diagram of the SAGE involved in this invention;

[0027] Figure 6 This is a diagram of the network layer parameters of the SAGE designed in this invention;

[0028] Figure 7 This is a structural diagram of the SAGE-LGB proposed in this invention. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0030] The hardware used in this invention includes one PC and one 1080 graphics card.

[0031] like Figure 1 As shown, this invention provides a graph neural network link prediction method oriented towards multidimensional node representation, specifically including the following steps:

[0032] Step 1: Segment the Chinese semantics of the sample.

[0033] Step 1.1: Obtain the attribute data of the sample nodes.

[0034] Obtain social network datasets in relevant fields, and perform data cleaning to obtain feature data describing user attributes, such as gender, age, and personal signature, as well as feature data describing item attributes, such as item name and usage instructions.

[0035] Step 1.2: Perform Chinese word segmentation using NLPIR technology.

[0036] NLPIR automatically segments and annotates terms in professional fields, supports the processing of Chinese sentences, and performs precise segmentation on the semantic features of the description node attributes after obtaining sample data from the data source.

[0037] Step 2: Obtain the word vector table representing the nodes.

[0038] BERT is used to map Chinese word segments to a 128-dimensional vector space through word vectors. The cosine value in the vector space is used as the similarity in text semantics. The word vector of a node is represented by sum pooling of all the token embeddings describing the node.

[0039] Step 3: Use the direct-transduction model for pre-training to obtain preliminary node representations.

[0040] Step 3.1: node2vec, as a transductive algorithm, is used to pre-train the constructed network topology graph. First, the walking strategy is controlled by defining hyperparameters p = 0.25 and q = 0.25 to generate a random walk node sequence. Then, the word2vec word vector model is used to train the sequence and obtain the 128-dimensional vector representation of the node.

[0041] like Figure 2 The diagram shown is a schematic representation of the pre-trained modeling structure of this invention.

[0042] Step 3.2: Concatenate the node semantic vector and the pre-trained vector.

[0043] emb1=Sum Pooling{token1, token2,..., tokeni}, i∈1...n

[0044] emb-node=concat(w1*emb1,w2*emb2)

[0045] Here, emb1 represents the node semantic vector, which is the word vector table token obtained in step 2. i The result of sum pooling (i∈1..n). `emb2` represents the node pre-training vector, calculated using the graph neural network transductive algorithm `node2vec`, with initialization parameters `w1 = 0.7` and `w2 = 0.3` for vector concatenation. During model training, parameters `w1` and `w2` are iteratively trained along with the model.

[0046] like Figure 3 The diagram shown is a schematic of word vector representation splicing, which serves as the overall feature input for a multimodal graph embedding link prediction method based on semantic representation.

[0047] like Figure 4 The diagram shown is a weight ratio diagram of the node semantic vector and the pre-trained vector initialization. The parameters will be fed into the model for iterative update and training along with the model.

[0048] Step 4: Model building. A custom inductive algorithm SAGE framework is used to learn graph embeddings. The word vectors obtained in Step 3 are used as the input of the model, and the optimal node representations in the training network layers are extracted through inductive learning methods.

[0049] Step 4.1: Sample the neighbors of the node and select the number of outer recurrent network layers k as the order of neighbors that can be aggregated, which also represents the number of aggregations. Here, the value of k is chosen as 2, indicating that the central node can aggregate the node information of its second-order neighborhood into its own embedding representation at most.

[0050] Step 4.2: Aggregate the feature information of the sampled neighbors. In the inner loop, for each node v, first use the (k-1)-layer embedding representations of v's neighboring nodes to generate the k-layer representation h of v's neighboring nodes through the aggregation function Mean. N(v) Then h N(v) The embedding representation of node v at the (k-1)th layer The data is concatenated and then subjected to a nonlinear transformation to generate the k-th layer embedded representation of v. The formula for a simple one-layer graph neural network can be expressed as: H (k+1) =f(H (k) ,A)=σ(AH (k) W (k) ).

[0051] Step 4.3: Add information about the degree of each node. Since nodes with higher degrees are more easily sampled by their neighbors, it is difficult for nodes with lower degrees to learn their representations through sampling and aggregation inductively. Therefore, node weights d are added here. i The calculation formula is: Deg i This represents the sum of the out-degree and in-degree of node i. For example, in an undirected graph, if a node has an out-degree of 3 and an in-degree of 2, the weight of the sampled node can be calculated using the formula: The sampling weights of nodes are inversely proportional to their degree. Essentially, this increases the probability of sampling nodes with lower degree and decreases the probability of sampling nodes with higher degree. This ensures that during the training of graph convolutional networks, high-degree nodes are not sampled too frequently, which could lead to problems such as the inability to learn the vectors of low-degree nodes.

[0052] like Figure 5 The diagram shows the design of the SAGE network. The SAGE network is a two-stage network structure, consisting of a "Message" stage and an "Aggregation" stage. The network input layer is the concatenation of pre-trained word vectors and text semantic vectors. Furthermore, the SAGE network considers the degree of nodes when performing negative sampling, improving prediction accuracy.

[0053] Step 5: Train the parameters of the last layer of the SAGE network designed in Step 4 using a machine learning model. Construct sample pairs on the sample data from the data source. If there is an edge from node i to j, then A... ij =1 otherwise 0. Various machine learning algorithms were used as downstream classifiers. Among them, the LightGBM algorithm performed best on small sample datasets. The dataset was subjected to 10-fold cross-validation. The machine learning model had a learning rate of 0.2, 100 iterations, and a maximum depth of 5. The best prediction result was selected based on the score.

[0054] like Figure 6 As shown, the parameters of a SAGE forward propagation network layer are as follows: the input node dimension is 350×478, representing the adjacency matrix formed by the number of nodes in the graph; the conv convolution kernel size is 64×64 and 32×32; after two layers of dropout = 0.1, the final output is a 32-dimensional vector.

[0055] like Figure 7 The diagram shows the structure of SAGE-LGB. It combines the graph convolution algorithm SAGE and the machine learning model LightGBM to achieve end-to-end link prediction.

[0056] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A graph neural network link prediction method for node multi-dimensional representation, characterized by: Step 1, word segmentation of sample Chinese semantics; Step 1.1, obtaining attribute data of sample nodes; Obtain social network data set, and clean the data to obtain feature data describing user attributes and feature data describing item attributes; Step 1.2, Chinese word segmentation using NLPIR technology; NLPIR can automatically segment and label professional domain words, support Chinese sentence processing, and after obtaining sample data from data source, accurately segment the semantic features describing node attributes; Step 2, obtaining word vector table representing nodes; Using BERT to map Chinese word segmentation to 128-dimensional vector space through word vector, using cosine value on vector space as similarity of text semantics, sum pooling all token embedding representing nodes to represent word vector of nodes; Step 3, using a direct inference model for pre-training and obtaining initial node representation; Step 3.1, node2vec as a direct inference algorithm for pre-training the constructed network topology graph; first, define hyperparameters p=0.25 and q=0.25 to control the strategy of walking, generate node sequence of random walk, and then use word2vec word vector model to train the sequence and obtain 128-dimensional vector representation of nodes; Step 3.2, concatenate node semantic vector and pre-training vector; emb1 = Sum Pooling {token1, token2,..., token i}, i ∈ 1... n emb_node=concat(w1*emb1,w2*emb2) Wherein, emb1 represents the node semantic vector, which is the word vector table token obtained in step 2 i emb2 represents the node pre-training vector, which is calculated by the graph neural network direct inference algorithm node2vec, and the vector splicing is defined by the initialization parameters w1=0.7 and w2=0.3; In the model training stage, the parameters w1 and w2 will be iteratively trained with the model; Step 4, model building, self-defined inductive algorithm SAGE framework for graph embedding learning, using the word vector obtained in step 3 as the input of the model, and using inductive learning method to extract the optimal node representation in the training network layer; Step 4.1, sampling the neighbors of the node, selecting the outer loop network layer number k as the neighbor order that can be aggregated, which also represents the aggregation times; here, the value of k is 2, which means that the center node can aggregate the node information of its second-order domain to its own embedding representation at most; Step 4.2: Aggregate the feature information of the sampled neighbors. In the inner loop, for each node v, first use the (k-1)-layer embedding representations of v's neighboring nodes to generate the k-layer representation h of v's neighboring nodes through the aggregation function Mean. N(v) Then h N(v) The embedding representation of node v at the (k-1)th layer The data is concatenated and then subjected to a nonlinear transformation to generate the k-th layer embedded representation of v. The formula for a simple one-layer graph neural network is: H (k+1) =f(H (k) ,A)=σ(AH (k) W (k) ); Step 4.3, add information of each node degree, since the larger the node degree, the easier it is to be sampled by the neighbor node, which makes it difficult for the node with small degree to learn the representation of the node through sampling and aggregation induction, here the node weight d is added i The calculation formula is: Deg i The sum of the out-degree and the in-degree of node i Step 5, using the last layer network parameters of SAGE network, constructing sample pairs on sample data of data source, if there is an edge between node i to j, A ij = 1, otherwise 0; 10-fold cross-validation is performed on the data set, the learning rate of the machine learning model is 0.2, the iteration is 100 times, the maximum depth is 5; and the optimal prediction result is selected according to the score; A SAGE forward propagation network layer parameter, the input node dimension is 350*478, which represents the adjacency matrix formed by the number of nodes in the graph, the conv convolution kernel size is 64*64, 32*32, after two layers of dropout=0.1, finally output a 32-dimensional vector.

Images