Academic research theme trend prediction method based on time sequence diagram attention network

Abstract

The invention discloses an academic research theme trend prediction method based on a time sequence diagram attention network. The academic research theme trend prediction method comprises the steps of collecting metadata of a plurality of papers and preprocessing the metadata; constructing a data set by taking keywords as academic research themes for the preprocessed metadata; constructing a time sequence diagram attention network model, and training and optimizing model parameters on the training data set; and inputting the test data set into the trained time sequence diagram attention network model to obtain an academic research theme trend prediction result. According to the method, a multi-head graph attention mechanism and time sequence convolution are combined to comprehensively extract time sequence features and associated features of an academic research subject, and the limitation that a traditional neural network academic research subject trend prediction method cannot utilize the academic research subject associated features can be effectively overcome; the method can improve the precision of academic research theme trend prediction, and can be applied to academic trend analysis and paper recommendation modules of journal paper databases.

Description

Technical Field

[0001] The present invention relates to academic trend prediction technology, and in particular to an academic research topic trend prediction method based on a temporal graph attention network. Background Art

[0002] Predicting the future trends of academic research topics has important guiding significance for scholars' research work and journal publishing plans. Common methods for predicting the trends of academic research topics include methods based on statistical models and methods based on neural networks. Traditional methods based on statistical models mainly include autoregression analysis and sliding average methods, which require manual extraction of statistical features and use of autoregression formulas to calculate the prediction results. Such methods are highly subjective and inefficient, and are gradually being replaced by methods based on neural networks.

[0003] Existing neural network-based methods mainly include long short-term memory networks, gated recurrent units, and temporal convolutional networks. However, the trend of academic research topics not only has the characteristics of temporal changes, but also has the characteristics of mutual correlation. The popularity of an academic research topic is often accompanied by the popularity of related academic research topics. Existing neural network-based methods often only consider the temporal characteristics of the academic research topic itself, but ignore the correlation characteristics between academic research topics, resulting in low actual prediction accuracy. Based on the above problems, the existing academic research topic trend prediction methods still have a lot of room for improvement. Summary of the invention

[0004] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide a method for predicting academic research topic trends based on a temporal graph attention network.

[0005] Technical solution: The present invention provides a method for predicting academic research topic trends based on a time-series graph attention network, comprising the following steps: Step 1: Collect metadata of multiple papers, including paper titles, abstracts, keywords, publication dates, and citation counts, and preprocess the metadata; Step 2: Use the preprocessed metadata to construct a dataset using keywords as academic research topics, and divide the dataset into a training dataset and a test dataset in chronological order; Step 3, construct a time-series graph attention network model, and train and optimize the model parameters on the training data set; the time-series graph attention network model includes a time-series convolution module, a multi-head graph attention module, and a prediction output module which are set in sequence; Step 4: Input the test dataset into the trained temporal graph attention network model to obtain the results of academic research topic trend prediction.

[0006] Furthermore, the metadata after preprocessing is used as keywords to construct a data set of academic research topics, including: Take academic research topics as nodes, the number of times different academic research topics appear in a paper as the weight of the edge connecting the nodes, and store the weights of all edges in an adjacency matrix In the example, the adjacency matrix Denoted as a graph structure; is the number of nodes; Count the number of citations of papers published on each academic research topic in each year and store them in a sequence matrix middle, is the total number of time steps. Each column of the sequence matrix represents an academic research topic, and each row represents a year. Recorded as time series; The graph structure and time series form a dataset.

[0007] Furthermore, a time-series graph attention network model is constructed, and training and optimization of model parameters on the training dataset include: Input the data in the training dataset into the time series convolution module to obtain the time series characteristics of the academic research topic itself , Represents the time step The characteristics are expressed as: , In the formula, Represents the time step Features, is the batch size, is the number of observation time steps, is the number of nodes, is the feature number, is the size of the convolution kernel, is the expansion coefficient, For the The weight of the convolution kernel, Represents the time step Features; The timing characteristics Input to the multi-head graph attention module to obtain the correlation features between academic research topics ; Associating Features Input to the prediction output module to obtain the feature matrix ; The prediction output module includes a first fully connected layer, a ReLU activation function layer and a second fully connected layer which are arranged in sequence.

[0008] Furthermore, the time series features Input to the multi-head graph attention module to obtain the correlation features between academic research topics include: The timing characteristics After the scaled dot product attention operation, the attention score of each attention head is calculated, and the calculation formula is: , In the formula, is the attention score, and are the linear transformation matrices of query and key respectively, is the scaling factor, is the hidden layer feature dimension, Represents a matrix transpose operation; After obtaining the attention score, according to the adjacency matrix Attention score Perform masking processing, the calculation formula is: , In the formula, Indicates the last two dimensions of the attention score Line The value of the column, Represents the adjacency matrix Line The value of the column; Aggregate the neighbor node features of each attention head to obtain the associated features , the calculation formula is as follows: , In the formula, is the number of attention heads, , Respectively The linear transformation matrix and bias matrix of the values ​​of the attention heads, The function is used to normalize the attention scores. Indicates The attention scores calculated by the attention heads.

[0009] Furthermore, dividing the data set into a training data set and a test data set in chronological order includes: Set the time step to The observation window and time step are The prediction window represents the use of the past Years of historical information to predict the future The trend of the year is calculated by sliding the adjacent values ​​along the time sequence on the time series, and the value in the observation window is used as the feature to obtain the feature matrix , taking the values ​​in the prediction window as labels to get the label matrix ; The feature label pairs are divided into training data sets and test data sets in order according to the ratio.

[0010] Furthermore, the metadata is preprocessed including removing null value items, converting to plain text format, and removing punctuation marks.

[0011] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. The present invention can comprehensively consider the temporal characteristics of the academic research topics themselves and the correlation characteristics between them, thereby improving the accuracy of the trend prediction of academic research topics; 2. The present invention uses a time series convolution module to obtain a larger receptive field with a smaller neural network depth, which is more advantageous than traditional methods in processing long-term time series features; 3. The present invention dynamically adjusts the weights of different neighbor nodes in the graph structure through a multi-head graph attention module, so as to capture complex correlation features more accurately; 4. The present invention can improve the accuracy of academic research topic trend prediction and can be applied to academic trend analysis and paper recommendation modules of journal paper databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of a method for predicting academic research topic trends based on a temporal graph attention network; Figure 2 It is a schematic diagram of the adjacency matrix and graph structure; Figure 3 It is a schematic diagram of time series and data sets; Figure 4 This is the architecture diagram of the temporal graph attention network model; Figure 5 This is the schematic diagram of the dilated causal convolution in the temporal convolution module; Figure 6 It is a curve chart of the predicted value and label value of the R&D investment theme by the model of the present invention and the most advanced model TCN; Figure 7 It is a curve diagram of the predicted value and label value of the model of the present invention and the current most advanced model TCN for the supply chain coordination topic; Figure 8 It is a curve chart of the predicted value and label value of the model of the present invention and the most advanced model TCN for the earnings management theme; Fig. 9 It is a curve chart of the prediction value and label value of the model of the present invention and the current most advanced model TCN for the organizational pattern topic. DETAILED DESCRIPTION

[0013] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0014] The present embodiment describes a method for predicting academic research topic trends based on a time series graph attention network, the flow chart of which is as follows: Figure 1 As shown, the method at least includes the following steps 1 to 4.

[0015] Step 1: Collect metadata of multiple papers, including paper titles, abstracts, keywords, publication dates, and number of citations, and preprocess the metadata.

[0016] For example, paper metadata is collected from any journal database such as Wanfang Academic Journal Database by using web crawler technology, and the collected paper metadata is preprocessed, including removing empty value items, converting to plain text format, and removing punctuation marks.

[0017] Step 2: Use the preprocessed metadata to construct a dataset using keywords as academic research topics, and divide the dataset into a training dataset and a test dataset in chronological order.

[0018] Furthermore, the metadata after preprocessing is used as keywords to construct a data set of academic research topics, including: Take academic research topics as nodes, the number of times different academic research topics appear in a paper as the weight of the edge connecting the nodes, and store the weights of all edges in an adjacency matrix In the example, the adjacency matrix Denoted as a graph structure; is the number of nodes; Count the number of citations of papers published on each academic research topic in each year and store them in a sequence matrix middle, is the total number of time steps. Each column of the sequence matrix represents an academic research topic, and each row represents a year. Recorded as time series; The graph structure and time series form a dataset.

[0019] For example, all keywords in the field of economics are extracted from the paper metadata and their respective frequencies are counted. The frequency threshold is set to 20, and keywords with frequencies below the threshold are removed to reduce the data size and space complexity. The remaining keywords are used as academic research topics.

[0020] The specific process of constructing the graph structure is as follows: using academic research topics as nodes in the graph structure, counting the number of times different academic research topics appear in a paper as the weight of the edges connecting the nodes, and storing the weights of all edges in an adjacency matrix. middle, is the number of nodes, that is, the number of topics, and we get Figure 2 If Topic 1 and Topic 2 appear together 13 times in a paper, 13 is used as the weight of the edge connecting Topic 1 and Topic 2 and stored in the adjacency matrix Similarly, the number of times that Topic 2 and Topic 3 appear together in a paper is 27 times, the number of times that Topic 2 and Topic 4 appear together in a paper is 86 times, the number of times that Topic 3 and Topic 4 appear together in a paper is 112 times, and the number of times that Topic 4 and Topic N appear together in a paper is 116 times. Then store 27, 86, 112, and 116 in the adjacency matrix respectively. at the corresponding position.

[0021] For example, the specific process of constructing a time series is: Count the number of citations of papers published on each academic research topic in each year and store them in a sequence matrix Among them is the total number of time steps, is the number of nodes, each column of the matrix represents an academic research topic, and each row represents a year. Figure 3 The time series shown on the left.

[0022] Furthermore, dividing the data set into a training data set and a test data set in chronological order includes: Set the time step to The observation window and time step are The prediction window represents the use of the past Years of historical information to predict the future The trend of the year is calculated by sliding the adjacent values ​​along the time sequence on the time series, and the value in the observation window is used as the feature to obtain the feature matrix , taking the values ​​in the prediction window as labels to get the label matrix ; The feature label pairs are divided into training data sets and test data sets in order according to the ratio.

[0023] For example, the specific process of dividing the data set is: like Figure 3As shown on the right, an observation window with a time step of 4 and a prediction window with a time step of 1 are set, which means that the historical information of the past 4 years predicts the trend of the next year. The front and back adjacent windows slide along the time sequence on the time series, and the values ​​in the observation window are used as features to obtain the feature matrix , is the batch size, is the number of observation time steps, is the number of nodes, 1 is the number of features, and the values ​​in the prediction window are used as labels to obtain the label matrix , is the batch size, is the number of prediction time steps, is the number of nodes, and 1 is the number of features. A split ratio of 7:3 is set to divide the feature-label pairs into training data sets and test data sets in order.

[0024] Step 3, construct a temporal graph attention network model, and train and optimize the model parameters on the training data set; the temporal graph attention network model includes a temporal convolution module, a multi-head graph attention module and a prediction output module which are set in sequence.

[0025] like Figure 4 As shown in the figure, the temporal graph attention network model includes the following modules: The temporal convolution module is used to extract the temporal features of the academic research topic itself from the input data; A multi-head graph attention module, which is used to extract correlation features between academic research topics from the data output by the temporal convolution module; The prediction output module is used to convert the data with time series features and correlation features output by the multi-head graph attention module into prediction values ​​for output.

[0026] Among them, the temporal convolution module utilizes the principle of dilated causal convolution. The temporal convolution kernel parameters of the temporal convolution module are weight normalized, and then output after residual connection, layer normalization and ReLU activation function.

[0027] Among them, causal convolution means that the features of the current time step are only convolved with the features of the past time step, and are not affected by the features of the future time step. Figure 5 As shown in the figure, the dilated causal convolution introduces spacing between the elements in the convolution kernel of the causal convolution, so that the convolution operation can span a larger time step. By adjusting the dilation factor from 1 to 2 and then to 4, the temporal convolution module can achieve a larger receptive field in fewer layers to effectively capture long-term temporal features.

[0028] The multi-head graph attention module utilizes the scaled dot product attention and multi-head graph attention mechanisms. It first passes through the multi-head graph attention, and then goes through the residual connection, layer normalization and ReLU activation function before outputting.

[0029] The output of the multi-head graph attention module is first subjected to convolution with a kernel size of 1, residual connection and ReLU activation function before the features are input into the prediction output module.

[0030] The prediction output module includes a first fully connected layer, a ReLU activation function layer, and a second fully connected layer which are arranged in sequence.

[0031] Furthermore, a time-series graph attention network model is constructed, and training and optimization of model parameters on the training dataset include: Input the data in the training dataset into the time series convolution module to obtain the time series characteristics of the academic research topic itself , Represents the time step The characteristics are expressed as: , In the formula, Represents the time step Features, is the batch size, is the number of observation time steps, is the number of nodes, is the feature number, is the size of the convolution kernel, is the expansion coefficient, For the The weight of the convolution kernel, Represents the time step Features; The timing characteristics Input to the multi-head graph attention module to obtain the correlation features between academic research topics ; Associating Features Input to the prediction output module to obtain the feature matrix ; The prediction output module includes a first fully connected layer, a ReLU activation function layer and a second fully connected layer which are arranged in sequence.

[0032] Furthermore, the time series features Input to the multi-head graph attention module to obtain the correlation features between academic research topics include: The timing characteristics After the scaled dot product attention operation, the attention score of each attention head is calculated, and the calculation formula is: , In the formula, is the attention score, and are the linear transformation matrices of query and key respectively, is the scaling factor, is the hidden layer feature dimension, Represents a matrix transpose operation; After obtaining the attention score, according to the adjacency matrix Attention score Perform masking processing, the calculation formula is: , In the formula, Indicates the last two dimensions of the attention score Line The value of the column, Represents the adjacency matrix Line The value of the column; Aggregate the neighbor node features of each attention head to obtain the associated features , the calculation formula is as follows: , In the formula, is the number of attention heads, , Respectively The linear transformation matrix and bias matrix of the values ​​of the attention heads, The function is used to normalize the attention scores. Indicates The attention scores calculated by the attention heads.

[0033] Associating Features Input to the prediction output module to obtain the feature matrix include: The prediction output module consists of two fully connected layers and a ReLU activation function in the middle to associate the features The input to the prediction output module will be connected through residual connection to obtain the feature matrix: ,in is the batch size, 1 is the number of observation time steps, is the number of nodes, is the feature number, and the feature matrix finally obtained by the prediction output module is , is the batch size, is the number of prediction time steps, is the number of nodes, and 1 is the final number of features.

[0034] In this example, the hyperparameters of the temporal graph attention network model training process are set to: The number of training rounds is 3000, and the early stopping strategy is adopted. If the loss does not decrease in 50 consecutive training rounds, the training is stopped to prevent overfitting. The loss function uses the mean square error loss MSE, the optimizer uses the Adam optimizer, the learning rate is 1E-3, and the weight decay coefficient is 1E-5. The number of research topics is the number of graph nodes The total number of time steps is 366 From 1999 to 2021, a total of 23 years, the number of observation time steps is For 4 years, the number of forecast time steps is The number of stacked layers is determined by the number of observation time steps. Before entering the prediction output module, the stacked layers are convolved with a convolution kernel size of 1 and a residual connection and ReLU activation function before being input into the prediction output module. In the first layer of the stack, a 1×1 convolution layer is used to align the data. The convolution kernel size of the temporal convolution is 2. Expansion coefficient The number of attention heads in the multi-head attention module is doubled from 1 layer by layer, and the number of attention heads in the multi-head attention module is 4. The batch size and the hidden feature dimension of each module need to be continuously adjusted and selected based on the training results.

[0035] Step 4: Input the test dataset into the trained temporal graph attention network model to obtain the results of academic research topic trend prediction.

[0036] The root mean square error (RMSE), mean absolute error (MAE) and determination coefficient (R²) were used as evaluation indicators to evaluate the prediction performance of the proposed method and the existing methods.

[0037] The root mean square error RMSE is defined as the square root of the mean of the sum of the squares of the prediction errors, and the formula is as follows: , In the formula, Indicates The label value or true value of a sample, Indicates The predicted value or output value of a sample is Represents the total number of samples. RMSE can reflect the overall level of prediction error. The smaller the value, the higher the prediction accuracy of the model.

[0038] The mean absolute error (MAE) is defined as the mean of the absolute errors between the predicted value and the true value. The formula is as follows: , MAE directly reflects the average difference between the predicted value and the true value. The smaller the value, the higher the prediction accuracy of the model.

[0039] Coefficient of determination It is used to measure the degree of fit of the model to the data and is defined as follows: , The value range is [0, 1]. The closer the value is to 1, the stronger the model's ability to explain the data.

[0040] Table 1 shows the performance comparison of the proposed model and the existing models in predicting the trend of academic research topics on the Wanfang academic dataset. The table lists the comparison of different models with a prediction time step of 1 and observation time steps of 4, 6, and 8, respectively. The existing models include the long short-term memory network LSTM, the gated recurrent unit GRU, and the temporal convolutional network TCN. In addition, Figures 6 to 9 The curves of the prediction value and label value of the model of the present invention and the current most advanced model TCN for four randomly selected research topics are given. Figure 6 The theme is R&D investment. Figure 7 The theme is supply chain coordination, Figure 8 The theme is earnings management. Fig. 9 The theme is organizational pattern, where the observation time step is 4 and the prediction time step is 1. The results show that the model prediction error of the proposed method is smaller than that of the existing method, while the prediction accuracy is higher than that of the existing method.

[0041] Table 1 Performance comparison of the proposed model and existing models in predicting academic research topic trends

Claims

1. A method for predicting academic research topic trends based on a temporal graph attention network, characterized in that: The steps include: Step 1: Collect metadata of multiple papers, including paper titles, abstracts, keywords, publication dates, and citation counts, and preprocess the metadata; Step 2: Use the preprocessed metadata to construct a dataset using keywords as academic research topics, and divide the dataset into a training dataset and a test dataset in chronological order; Step 3, construct a time-series graph attention network model, and train and optimize the model parameters on the training data set; the time-series graph attention network model includes a time-series convolution module, a multi-head graph attention module, and a prediction output module which are set in sequence; Step 4: Input the test dataset into the trained temporal graph attention network model to obtain the results of academic research topic trend prediction.

2. According to claim 1, a method for predicting academic research topic trends based on a time-series graph attention network is characterized in that: The preprocessed metadata is used as keywords to construct a data set of academic research topics, including: Take academic research topics as nodes, the number of times different academic research topics appear in a paper as the weight of the edge connecting the nodes, and store the weights of all edges in an adjacency matrix In the example, the adjacency matrix Denoted as a graph structure; is the number of nodes; Count the number of citations of papers published on each academic research topic in each year and store them in a sequence matrix middle, is the total number of time steps. Each column of the sequence matrix represents an academic research topic, and each row represents a year. Recorded as time series; The graph structure and time series form a dataset.

3. According to claim 2, a method for predicting academic research topic trends based on a time-series graph attention network is characterized in that: Constructing a time-series graph attention network model, training it on a training dataset, and optimizing model parameters include: Input the data in the training dataset into the time series convolution module to obtain the time series characteristics of the academic research topic itself , Represents the time step The characteristics are expressed as: ， In the formula, Represents the time step Features, is the batch size, is the number of observation time steps, is the number of nodes, is the feature number, is the size of the convolution kernel, is the expansion coefficient, For the The weight of the convolution kernel, Represents the time step Features; The timing characteristics Input to the multi-head graph attention module to obtain the correlation features between academic research topics ; Associating Features Input to the prediction output module to obtain the feature matrix ; The prediction output module includes a first fully connected layer, a ReLU activation function layer and a second fully connected layer which are arranged in sequence.

4. According to claim 3, a method for predicting academic research topic trends based on a time-series graph attention network is characterized in that: The timing characteristics Input to the multi-head graph attention module to obtain the correlation features between academic research topics include: The timing characteristics After the scaled dot product attention operation, the attention score of each attention head is calculated, and the calculation formula is: ， In the formula, is the attention score, and are the linear transformation matrices of query and key respectively, is the scaling factor, is the hidden layer feature dimension, Represents a matrix transpose operation; After obtaining the attention score, according to the adjacency matrix Attention score Perform masking processing, the calculation formula is: ， In the formula, Indicates the last two dimensions of the attention score Line The value of the column, Represents the adjacency matrix Line The value of the column; Aggregate the neighbor node features of each attention head to obtain the associated features , the calculation formula is as follows: ， In the formula, is the number of attention heads, , Respectively The linear transformation matrix and bias matrix of the values ​​of the attention heads, The function is used to normalize the attention scores. Indicates The attention scores calculated by the attention heads.

5. According to claim 1, a method for predicting academic research topic trends based on a time-series graph attention network is characterized in that: Dividing the data set into training data set and test data set in chronological order includes: Set the time step to The observation window and time step are The prediction window represents the use of the past Years of historical information to predict the future The trend of the year is calculated by sliding the adjacent values ​​along the time sequence on the time series, and the value in the observation window is used as the feature to obtain the feature matrix , taking the values ​​in the prediction window as labels to get the label matrix ; The feature label pairs are divided into training data sets and test data sets in order according to the ratio.

6. According to claim 1, a method for predicting academic research topic trends based on a time-series graph attention network is characterized in that: The metadata is preprocessed, including removing empty value items, converting to plain text format, and removing punctuation marks.

Images