Method for Constructing Scientific Research User Portrait Based on Transformer and GNN Fusion Model

Abstract

The present invention discloses a method for constructing a scientific research user portrait based on a fusion model of Transformer and GNN. Based on paper co-authorship and citation data information, a scientific research network graph dataset is constructed; a simplified Transformer model is used to calculate the attention of nodes to global nodes, and the node features are updated according to the attention scores; then the updated node features are input into a graph neural network to aggregate node neighborhood information; finally, the model output is combined with the graph structure information to obtain the final node features, and the model parameters are adjusted according to a custom loss function to predict the user portrait labels of the nodes. The present invention solves the problem that existing graph neural network methods are difficult to be directly used for constructing scientific research user portraits. In view of the characteristics of scientific research networks, the scalability and prediction accuracy of the model are improved, and the user portrait labels of scientific research users can be effectively predicted.

Description

Technical Field

[0001] The present invention belongs to the technical field of scientific research user portraits, and specifically relates to a method for constructing a scientific research user portrait based on a fusion model of Transformer and GNN. Background Art

[0002] With the development of science and technology and the continuous increase of scientific research achievements, academic users in the big data era need to face a vast amount of academic resources. As a tool that can efficiently utilize big data, user portraits can be applied to the scientific research field to enable users to obtain effective information efficiently and greatly reduce the retrieval cost. The scientific research user portrait forms labels or features by analyzing and modeling the characteristics of the scientific research user group, providing decision-making support for information service institutions and academic platforms. The research on scientific research user portraits mainly focuses on the construction method of user portraits based on interests. At present, the scientific research network has increasingly become an important carrier of information in the scientific research field. Therefore, when constructing a user portrait of the scientific research network, it is necessary to extract both individual and network structure information simultaneously.

[0003] In recent years, graph deep learning has shown unique advantages in the feature extraction of network structure data types. Graph neural networks based on the message passing mechanism can handle various graph learning tasks in a large number of applications, including the construction of user portraits with network structures as data. However, with the continuous increase in the volume of the scientific research network and its unique network characteristics, existing graph neural network methods are difficult to be directly used for the construction of scientific research user portraits. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for constructing a scientific research user portrait based on a fusion model of Transformer and GNN to solve the above problems.

[0005] To achieve the above purpose, the present invention provides the following technical solution: A method for constructing a scientific research user portrait based on a fusion model of Transformer and GNN.

[0006] S1: Based on the information of paper co-authorship and citation data, taking scholars or papers as nodes and the co-authorship relationship or paper citation relationship between scholars as edges, construct a scientific research network graph, process the node features into feature vectors, and convert the labels of known nodes into label vectors;

[0007] S2: Using the node information and structure information of the scientific research network graph, first calculate the similarity between the central node and all nodes in the whole by using a simplified Transformer module. After obtaining the attention score matrix, linearly transform the attention score matrix with the original node features to obtain the updated node features;

[0008] S3: Use the output obtained by the attention calculation module as the input of the model fusion module. After aggregating the global information, obtain the neighborhood information of the nodes through the graph neural network model;

[0009] S4: Use the node information and structure information of the scientific research network graph to obtain the graph topology information through the graph neural network model, combine it with the node features obtained according to the attention calculation module and the model fusion module to obtain the final node features, and predict the labels based on the features of the nodes;

[0010] Further, the process of obtaining the scientific research network graph dataset in step S1 includes the following steps:

[0011] S11: Construct the scientific research network graph dataset: Based on the co-authorship and citation data information of papers, use scholars or papers as nodes, and the co-authorship relationship between scholars or the paper citation relationship as edges to construct the scientific research network graph G, satisfying G=(V, A, X, Y), where V represents the node set, representing scholars or papers; A∈RN×N represents the adjacency matrix, representing the co-authorship relationship between scholars or the citation relationship between papers, where N is the total number of nodes; X∈RN×F represents the node feature matrix, and the keywords of each author's paper are represented by the node features, where F is the feature dimension of each node; Y∈RN×C represents the label matrix, and the label represents the most active research field label of the scholar, where C is the number of categories;

[0012] S12: Data preprocessing: Classify the dataset according to whether it contains labels, and define the labeled nodes as the training set; Divide the training set into the training set, validation set and test set according to the ratio of 6:2:2; Among them, the training set is used to train the model, the validation set is used to adjust the hyperparameters of the model, and the test set is used to evaluate the final performance of the model.

[0013] Further, in the process of updating the node features according to the attention in step S2, a simplified Transformer method is adopted to achieve a linear increase in the computational complexity of the model as the number of nodes increases. Specifically, it includes the following steps:

[0014] S21: Data input: First, use a neural network layer to map the input feature X∈RN×F to the node features in the latent space, that is, H(0) = f(X), where f is a single-layer MLP. The node feature H(0)∈R N×d will be used for subsequent attention calculation and propagation.

[0015] S22: After obtaining the initial node feature matrix H(0), it needs to be projected into three high-dimensional spaces Q, K, and V. The specific formula is as follows:

[0016]

[0017] where Q, K, and V respectively represent the query, key, and value matrices in the attention mechanism, and W Q , W K , W V respectively represent their respective projection matrices;

[0018] S23: According to the obtained Q, K, and V feature matrices, calculate the attention scores between the central node and the global node, and update the node features:

[0019] Attention mechanism: The original attention calculation formula is as follows:

[0020]

[0021] According to the definition of the original attention mechanism, first, QKT needs to be calculated, and its complexity is O(n2d). Subsequently, multiplying softmax(QKT) by V also requires O(n2d) complexity. Thus, it can be seen that the key factor restricting the performance of Attention is the softmax operation in the definition. If the softmax operation is removed, then QKTV satisfies the matrix multiplication associative law. Therefore, KTV can be calculated first, and then the result can be obtained by left-multiplying with Q. The complexity required for both steps is O(nd2). In the dataset, since d << n, the overall calculation complexity can be regarded as O(n).

[0022] Simplified attention score calculation: The original attention calculation formula is equivalently rewritten as:

[0023]

[0024] This formula can be understood as taking the calculated attention scores to perform a weighted average on vj. Among them, the e operation ensures the non-negativity of the result. Therefore, if a new attention needs to be defined, while retaining the original similar distribution characteristics, the non-negativity of the result needs to be ensured.

[0025] This patent proposes a simplified attention score calculation and feature update method: First, normalize the node vector l2 to limit it between [-1, 1], then calculate the similarity between two node vectors through dot product similarity, and finally add one to the result to ensure the non-negativity of the attention scores. Taking node i as an example, after calculating its similarity with the global node, the node features are updated. The improved formula is as follows:

[0026]

[0027] Among them, the subscripts i and j represent nodes i and j, q and k represent the feature vectors of the nodes, qTk represents the similarity calculation between nodes i and j, ||·|| represents l2 normalization of the feature vector to ensure that the result of the vector dot product is ≥ -1, and adding 1 ensures the non-negativity of the result. The final result Hi represents the updated feature of node i;

[0028] The two summation terms in the numerator and denominator are shared for all samples, that is, in actual calculation, only O(N) is needed to calculate once, and then only O(1) is needed for the update of each sample. Therefore, the total complexity of updating N samples is O(N).

[0029] The feature update is represented in matrix form, and the specific formula is as follows:

[0030]

[0031] Among them 、 represents the normalized matrix, and H represents the updated feature matrix;

[0032] Furthermore, in the model fusion process of step S3, a model that combines Transformer and graph neural network is proposed. With the help of the attention mechanism, the graph neural network can expand the receptive field to all nodes globally, even if these nodes are far from the central node. On the other hand, the graph neural network can help the model efficiently aggregate relevant information in more neighborhoods, specifically including the following steps:

[0033] S31: Residual connection: To alleviate the problems of gradient disappearance and explosion, a residual connection is introduced in the simplified Transformer module. This also helps to retain some information and increase the expressive performance of the simplified Transformer module:

[0034]

[0035] Among them, β is the hyperparameter of the residual connection, and H combines the pairwise attention information and self-loop information on N nodes. The former allows the model to capture the influence of other nodes, while the latter retains the information of the central node.

[0036] S32: Graph neural network layer: Taking the updated node matrix obtained by the Transformer module as the input, the graph neural network iteratively updates the features of the nodes by combining the features of neighboring nodes and the features of the nodes themselves, helping the model to effectively capture the information of neighboring relevant nodes; Each layer of the graph neural network has two important functions. For k = 1, 2, …, K, the specific calculation process is as follows:

[0037]

[0038]

[0039] Among them, AGGREGATE represents an aggregation function, and its function is to aggregate information from the neighbor nodes of each node; N(v) represents the set of neighbor nodes of the v-th node, which represents the sum of the information of node v. COMBINE represents a combination function, and its function is to update the features of the node by combining the aggregated information from the neighbor nodes and the current node features. The node feature matrix HK of the last layer can be regarded as the final node feature matrix.

[0040] Furthermore, in the process of combining graph topology information and predicting node labels in step S4, a simple and effective scheme is adopted to combine the graph topology information with the node features obtained from the previous two modules to obtain the final node features, and finally predict the labels based on the features of the nodes. The specific steps are as follows:

[0041] S41: Structural information: To improve the accuracy of the final prediction, it is necessary to add the structural information of graph G to the model; different from most existing schemes that use positional encoding, here the output features of the model are combined with the propagation embedding of the GNN in the output layer through weighted summation. The specific formula is as follows:

[0042]

[0043] where α is a weight hyperparameter, and the GNN module can be a simple GNN architecture (for example, GCN), H out is the final node feature matrix.

[0044] S42: Label prediction: The final node features are obtained through attention calculation, model fusion, and structural information calculation, and the labels of the nodes are predicted. The softmax function can be used for prediction. The specific formula is as follows:

[0045]

[0046] S43: Loss function: After label prediction, it is necessary to set a loss function to measure the difference between the predicted value and the actual value of the model, and optimize the model parameters by minimizing the loss function. The specific formula is as follows:

[0047]

[0048] where represents the predicted label of node i, n represents the number of label nodes, loss represents the cross-entropy loss function, and the entire model can obtain the optimal model parameters by minimizing the objective function O through backpropagation.

[0049] Technical effects and advantages of the present invention: Applying the algorithm based on the graph neural network to the problem of constructing a scientific research user portrait fills the gap in the lack of methods in this field. Compared with the existing methods for constructing a scientific research user portrait, the present invention constructs a scientific research network graph and extracts user information and network structure information simultaneously. By adopting a simplified Transformer model, the computational complexity increases linearly with the increase in the number of nodes. While obtaining global information, the model can be extended to large datasets. After obtaining global information, by fusing the Transformer with the graph neural network model, the advantages of the two models are complementary. Finally, adding graph structure information further improves the accuracy. The invention can effectively predict the labels of the scientific research user portrait, which is of great help to improving the retrieval and decision-making efficiency in the scientific research field. Description of the Drawings

[0050] Figure 1 It is a schematic flow chart of the method of the present invention.

[0051] Figure 2 It is a schematic diagram of the overall model framework of the present invention.

[0052] Figure 3 It is a schematic diagram of the operation process of node feature update of the present invention. Detailed Embodiments

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0054] The present invention provides a method for constructing a scientific research user portrait based on a fusion model of Transformer and GNN as shown in Figures 1 - 3 :

[0055] S1: Based on the information of co-authorship and citation data of papers, taking scholars or papers as nodes and the co-authorship relationship or paper citation relationship between scholars as edges, construct a scientific research network graph, process the node features into feature vectors, and convert the labels of known nodes into label vectors;

[0056] The process of obtaining the scientific research network graph dataset includes the following steps:

[0057] S11: Construct a scientific research network graph dataset: Based on co - authoring and citation data information, taking scholars or papers as nodes and the co - authoring relationship between scholars or the paper citation relationship as edges, construct a scientific research network graph G, satisfying G=(V, A, X, Y), where V represents the node set, representing scholars or papers; A∈RN×N represents the adjacency matrix, representing the co - authoring relationship between scholars or the citation relationship between papers, where N is the total number of nodes; X∈RN×F represents the node feature matrix, with the node features representing the keywords of each author's paper, where F is the feature dimension of each node; Y∈RN×C represents the label matrix, and the labels represent the most active research field labels of scholars, where C is the number of categories.

[0058] S12: Data pre - processing: Classify the dataset according to whether it contains labels, and define the labeled nodes as the training set; divide the training set into a training set, a validation set, and a test set according to the ratio of 6:2:2; among them, the training set is used to train the model, the validation set is used to adjust the hyperparameters of the model, and the test set is used to evaluate the final performance of the model.

[0059] S2: Use the node information and structure information of the scientific research network graph. First, use a simplified Transformer module to calculate the similarity between the central node and all global nodes to obtain the attention score matrix. After that, linearly transform the attention score matrix with the original node features to obtain the updated node features. The overall update process is as Figure 3 shown;

[0060] Updating the node features according to attention includes the following steps:

[0061] S21: Data input: First, use a neural network layer to map the input feature X∈RN×F to the node features in the latent space, that is, H(0)=f(X), where f is a single - layer MLP. The node features H(0)∈R N×d will be used for subsequent attention calculation and propagation.

[0062] S22: After obtaining the initial node feature matrix H(0), it needs to be projected into three high - dimensional spaces Q, K, V. The specific formulas are as follows:

[0063]

[0064] Among them, Q, K, V respectively represent the query, key, and value matrices in the attention mechanism, and W Q 、W K 、W V represent their respective projection matrices;

[0065] S23: Attention Score Calculation and Feature Update: Centering on node i, calculate its similarity with the global node and then update the node features. Different from traditional methods, without applying the softmax operator, while preserving the distribution characteristics similar to the original method and the non-negativity of the results, the improved formula is as follows:

[0066]

[0067] Among them, the subscripts i and j represent node i and node j, q and k represent the feature vectors of the nodes, qTk represents the similarity calculation between nodes i and j, ||·|| represents l2 normalization of the feature vector to ensure that the result of the vector dot product ≥ -1, and adding 1 ensures the non-negativity of the result. The final result Hi represents the updated feature of node i;

[0068] Calculated using the overall feature matrix, the specific formula is as follows:

[0069]

[0070] Among them 、 represent the normalized matrix, H represents the updated feature matrix, and diag represents generating a diagonal matrix according to the given vector;

[0071] S3: Take the output obtained by the attention calculation module as the input of the model fusion module, and after aggregating the global information, obtain the neighborhood information of the nodes through the graph neural network model;

[0072] Obtaining neighborhood information through the model fusion module includes the following steps:

[0073] S31: Residual Connection: To alleviate the problems of gradient vanishing and explosion, a residual connection is introduced in the simplified Transformer module. This also helps to retain some information and increase the expressive performance of the simplified Transformer module:

[0074]

[0075] Among them, β is the hyperparameter of the residual connection, and H combines the pairwise attention information and self-loop information on N nodes. The former allows the model to capture the influence of other nodes, while the latter retains the information of the central node.

[0076] S32: Graph Neural Network Layer: Take the updated node matrix obtained by the Transformer module as the input. The graph neural network iteratively updates the node features by combining the features of neighbor nodes and the features of the node itself, helping the model to effectively capture the information of neighborhood-related nodes; Each layer of the graph neural network has two important functions. For k = 1, 2, …, K, the specific calculation process is as follows:

[0077]

[0078]

[0079] Among them, AGGREGATE represents an aggregation function, and its function is to summarize information from the neighbor nodes of each node; N(v) represents the set of neighbor nodes of the v-th node, which represents the sum of the information of node v. COMBINE represents a combination function, and its function is to update the features of the node by combining the aggregated information from neighbor nodes and the current node features. The node feature matrix HK of the last layer can be regarded as the final node feature matrix.

[0080] S4: Use the node information and structure information of the scientific research network graph, obtain the graph topology information through the graph neural network model, combine it with the node features obtained according to the attention calculation module and the model fusion module to obtain the final node features, and predict the label based on the features of the node;

[0081] Predicting the node label includes the following steps:

[0082] S41: Structure information: In order to improve the accuracy of the final prediction, it is necessary to add the structure information of graph G to the model; different from most existing solutions that use positional encoding, here we adopt the combination of the model output features and the propagation embedding of the GNN in the output layer. The specific formula is as follows:

[0083]

[0084] where α is a weight hyperparameter, and the GNN module can be a simple GNN architecture (for example, GCN), H out is the final node feature matrix, and A represents the adjacency matrix of the scientific research network.

[0085] S42: Label prediction: Obtain the final node features through attention calculation, model fusion, and structure information calculation, and perform node label prediction. The softmax function can be used for prediction. The specific formula is as follows:

[0086]

[0087] D3: Loss function: After performing label prediction, it is necessary to set a loss function to measure the difference between the model prediction value and the actual value, and optimize the model parameters by minimizing the loss function. The specific formula is as follows:

[0088]

[0089] where denotes the predicted label of node i, n denotes the number of label nodes, loss denotes the cross-loss function, and the entire model can obtain the optimal model parameters by minimizing the objective function O through backpropagation;

[0090] Experimental verification:

[0091]

[0092] Table 1 shows the experimental results of all models. We choose accuracy, a commonly used evaluation metric in multi-classification problems, to measure the model performance. The results show that our model significantly outperforms the standard GNN models (GCN, GAT). This indicates that our model has high expressive power while achieving linear complexity. Compared with other advanced GNN models applying Transformer (NAGphormer and SGFormer), our model is also highly competitive. We also find that there is a certain overfitting phenomenon in NAGphormer, which may be caused by its relatively complex architecture and the model architecture based on node sampling. In contrast, in the case of limited dataset samples, the lightweight architecture enables our model to have better generalization ability.

[0093] To analyze the importance of model fusion and structural information, a series of ablation studies were conducted on multiple datasets, and the results are summarized in Table 2.

[0094]

[0095] Model fusion: To study the effectiveness of model fusion, we compared the model without fusing with GNN and the default model. The experimental results show that model fusion can significantly improve the model performance, demonstrating the necessity of complementarity between the two models. This is because for the research network with sparse edges, the neighborhood information of central nodes is very important. Using only Transformer will cause the model to treat all nodes globally equally, resulting in a decrease in accuracy. While GNN can aggregate the neighborhood information of central nodes through the message passing mechanism, so we adopt the strategy of model fusion to fuse Transformer and GNN. The results show that the accuracy of the model can be improved after the two are fused.

[0096] Structural information: To study the effectiveness of structural information, we compared the default model with the model without adding structural information. The experimental results show that adding structural information can significantly improve the model performance, demonstrating the necessity of structural information in node-level tasks. This is because for graph data, structural information also has a significant impact on node representation, and adding structural information can improve the accuracy.

[0097] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for constructing a scientific research user portrait based on a fusion model of Transformer and GNN, characterized in that: The specific steps are as follows: S1. Based on the information of co-authorship and citation data of papers, construct a scientific research network graph G with scholars or papers as nodes and the co-authorship relationship between scholars or the paper citation relationship as edges; S2. Use a simplified Transformer module to calculate the similarity between the central node and all global nodes in the node information and structural information of the scientific research network graph G to obtain an attention score matrix, and then linearly transform the attention score matrix with the original node features to obtain updated node features, forming an updated node feature matrix; S3. Use the updated node feature matrix obtained in S2 as the input of the GNN fusion model. The GNN fusion model is specifically a model that fuses Transformer and graph neural network, and iteratively updates the node features by combining the features of neighbor nodes and the features of the node itself through the graph neural network; S31: Residual connection: To alleviate the problems of gradient vanishing and explosion, a residual connection is introduced in the simplified Transformer module; this also helps to retain some information and increase the expressive performance of the simplified Transformer module: ； Among them, β is the hyperparameter of the residual connection, and H combines the pairwise attention information and self-loop information on N nodes; the former allows the model to capture the influence of other nodes, while the latter retains the information of the central node; S32: Graph neural network layer: Use the updated node matrix obtained by the Transformer module as the input. The graph neural network iteratively updates the node features by combining the features of neighbor nodes and the features of the node itself, helping the model to effectively capture the information of neighborhood-related nodes; each layer of the graph neural network has two important functions. For k = 1, 2,..., K, the specific calculation process is as follows: ； ； Among them, AGGREGATE represents an aggregation function, and its function is to aggregate information from the neighbor nodes of each node; N(v) represents the set of neighbor nodes of the v-th node, which represents the sum of the information of node v; COMBINE represents a combination function, and its function is to update the features of a node by combining the aggregated information from neighbor nodes and the current node features. The node feature matrix HK of the last layer can be regarded as the final node feature matrix; S4. Use the node information and structural information of the scientific research network graph, obtain the graph topology information through the graph neural network model, combine it with the node features obtained according to the simplified Transformer module and S3, and perform accurate label prediction using the softmax function based on the final node features; S41: Structural information: To improve the accuracy of the final prediction, the structural information of graph G needs to be added to the model; different from most existing solutions that use positional encoding, here the output features of the model are combined with the propagation embedding of the GNN in the output layer through weighted summation, and the specific formula is as follows: ； where α is a weight hyperparameter, the GNN module is a simple GNN architecture, and H out is the final node feature matrix; S42: Label prediction: Obtain the final node features through attention calculation, model fusion, and structural information calculation, and perform label prediction on the nodes. The softmax function can be used for prediction, and the specific formula is as follows: ； S43: Loss function: After performing label prediction, a loss function needs to be set to measure the difference between the predicted value and the actual value of the model, and the model parameters are optimized by minimizing the loss function. The specific formula is as follows: ； where represents the predicted label of node i, n represents the number of label nodes, and loss represents the cross-loss function. The entire model can obtain the optimal model parameters by minimizing the objective function O through backpropagation.

2. The method for constructing a scientific research user portrait based on the Transformer and GNN fusion model according to claim 1, wherein: S1 also includes processing the node features into feature vectors and converting the labels of known nodes into label vectors.

3. The method for constructing a scientific research user portrait based on the Transformer and GNN fusion model according to claim 1, characterized in that: S2 specifically includes: S21: Data Input: Use a neural network layer to map the input node features \(X\in\mathbb{R}^{N\times F}\) to node features in the latent space, i.e., \(H^{(0)} = f(X)\), where \(f\) is a single-layer MLP; the node features \(H^{(0)}\in\mathbb{R}^{N\times d}\) will be used for subsequent attention calculation and propagation; S22: After obtaining the initial node feature matrix \(H^{(0)}\), it is necessary to project it into three high-dimensional spaces \(Q\), \(K\), and \(V\). The specific formulas are as follows: ； where Q, K, and V represent the query, key, and value matrices in the attention mechanism, respectively, and W Q , W K , and W V represent their respective projection matrices, respectively; S23: According to the obtained feature matrices \(Q\), \(K\), and \(V\), calculate the attention scores between the central node and the global node, and update the node features.

Images