A citation intent classification method based on multi-task bilateral branch network

Abstract

The present invention discloses a citation intent classification method based on a multi-task bilateral branch network. In order to alleviate the impact of the dataset imbalance problem on the model performance, a bilateral branch network is adopted as the basic network structure. During the training process, the original dataset is first obtained, and the oversampling technology is used to generate an inverted dataset and construct an auxiliary dataset. In order to be able to introduce the information of the auxiliary dataset, a multi-task learning method is adopted. When constructing the model, the SciBert model is used as the embedding layer to extract the feature vector of the input citation text. In addition, the attention mechanism is used to construct an attention layer to improve the model's attention to important words in the citation text and reduce the attention to other words. Finally, the model shifts the model's attention during the training process through an adaptive parameter α. The present invention improves the accuracy of citation intent classification, so that the intent category of the citation can be predicted more accurately.

Description

Technical Field

[0001] The present invention relates to a method for classifying citation intent, and more particularly, to a classification method for citation intent datasets with imbalanced categories. Background Art

[0002] In recent years, the number of papers published annually in many academic fields has been increasing year by year. For example, top conferences in the field of deep learning in computer science have almost always accepted more papers each year than the previous year. Whether you're a seasoned researcher or just starting out, you need to read papers to track the development of your research field. Faced with such a massive volume of papers, researchers need to spend a considerable amount of time sifting through and reading them. Traditional citation analysis judges the quality of a document based solely on the frequency of citations. Citations are cited for different purposes: some serve as research background, while others serve as methods. Therefore, understanding the purpose of citations can help researchers better screen papers and analyze scientific work.

[0003] Existing methods all construct different feature information from content such as the citation sentence, citation title, and cited title, and then combine different classifiers to achieve citation intent classification. Due to the different ways of constructing feature information, many classification methods have been proposed, some of which extract feature information through manually set rules, while others use deep learning methods to generate feature information. These existing citation intent classification methods assume that the different intent categories in the citation dataset are balanced. However, there is a significant data imbalance in the dataset of this invention.

[0004] In real-world scenarios, class imbalance in datasets is a common problem. Class imbalance often leads to poor model performance on categories with a small number of samples. The citation intent categories of the present invention are divided into 6 categories, of which most citation texts belong to the same category, and the remaining citation texts belong to the remaining 5 categories. The model tends to predict samples from other categories as the category with a large number of samples. Therefore, the present invention uses a bilateral branch network to improve the performance of the model under data imbalance problems, and also introduces a multi-task learning method to construct a citation intent classification model based on a multi-task bilateral branch network. Summary of the Invention

[0005] One purpose of the present invention is to address the deficiencies of the existing technology and provide a citation intent classification method based on a multi-task bilateral branch network, which uses a bilateral branch network as the basis of the model and utilizes a multi-task learning method to construct the model.

[0006] The method adopted by the present invention is as follows:

[0007] Step 1: Obtain citation text data with intent classification labels as the original dataset D o .

[0008] Step 2: Use oversampling method to adjust D o The training set is sampled to generate the reverse data set D r .

[0009] Step 3: Obtain citation text data whose classification labels are chapter names of scientific literature and construct auxiliary dataset D a .

[0010] Step 4: Build a multi-task bilateral branch network model and use the training set of the original dataset, the reversed dataset, and the auxiliary dataset for training.

[0011] Step 5: Use the trained multi-task bilateral branch network model for verification and testing.

[0012] Step 6: Use the tested multi-task bilateral branch network model to classify the citation intent of the citation text.

[0013] Another object of the present invention is to provide a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor implements the above method when executing the executable code.

[0014] The technical solution provided by the present invention has the following beneficial effects:

[0015] The present invention introduces external information by adopting a multi-task learning approach, expands the features that the model can learn, and improves the generalization ability of the model. In addition, the present invention uses a bilateral branch network to improve the classification performance of the model on categories with a small number of samples, alleviating the impact of data imbalance on model performance.

[0016] The present invention adopts the SciBert model as the shared part in the bilateral branch network because it is a model trained on a large amount of scientific literature and has more appropriate prior knowledge; in addition, the use of the attention mechanism enables the model to focus on words that can help express the meaning of the sentence, while reducing the influence of other words on the semantic extraction of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart involved in the present invention;

[0018] Figure 2 is the model structure diagram; DETAILED DESCRIPTION

[0019] The specific embodiments of the present invention are described in further detail below with reference to the accompanying drawings. Figure 1 As shown, where:

[0020] Step 1: Obtain citation text data with intent classification labels as the original dataset D o ; Divide the entire original data set into the training set, validation set, and test set of the original data set according to a certain ratio. For example, D o 60% of the data is used as the training set, 15% of the data is used as the validation set, and the remaining 25% of the data is used as the test set.

[0021] The intent is the purpose of citing the text. For example, citation intent can be divided into six categories: background, contrast, expansion, future, motivation, and use.

[0022] Step 2: Use oversampling method to adjust D o The training set is sampled to generate the reverse data set D r .

[0023] The specific operations are as follows:

[0024] 2-1 Calculate the ratio of each category in the training set of the original data set according to formula (1)

[0025]

[0026] in Indicates c m The number of samples in each category; N max Indicates the number of samples corresponding to the category with the largest number of samples;

[0027] 2-2 Using the ratio of each category Calculate the sampling probability of each category

[0028]

[0029] where n c The number of categories representing intent.

[0030] 2-3 Iteration Sampling:

[0031] 2-3-1 Initialization: Use the library function to randomly obtain a value num, the value range of num is [0,1); at the same time, initialize the variable sum to 0;

[0032] 2-3-2 Traverse each category in the training set of the original data set in turn

[0033] (1) For the traversed categories, use the sampling probability of the current category Update the value of sum, that is

[0034] (2) Determine whether the random number num ≤ sum is satisfied. If so, randomly select a citation text data from the current category and add it to the reverse dataset; if not, do nothing.

[0035] (3) Traverse the next category and repeat steps (1)-(2) until all categories are traversed.

[0036] 2-3-3 Determine whether the number of elements in the reverse dataset is the same as the training set of the original dataset. If so, end; if not, return to step 2-3-1.

[0037] For example, the number of citation texts in the constructed reverse dataset is 2400. Then 2400 iterative loops are required.

[0038] Assume that the training set in the original dataset has a total of 6 categories, and the sampling probabilities of the 6 categories are respectively category 0 = 0.45, category 1 = 0.14, category 2 = 0.15, category 3 = 0.13, category 4 = 0.1, category 5 = 0.03; traverse each category in the training set of the original dataset starting from category 0 in sequence.

[0039] For example, in the first iteration, the randomly generated num = 0.4, the initialized sum = 0. When traversing to category 0, first update sum = 0 + 0.45, and then judge that num < sum is satisfied, then randomly select a citation text from category 0 and add it to the reverse dataset, and jump out of this category traversal.

[0040] In the second iteration, the randomly generated num = 0.5, and sum is re-initialized to 0. When traversing to category 0, first update sum = 0 + 0.45. Then judge that num < sum is not satisfied, so continue to traverse the next category. When traversing to category 1, first update sum = 0.45 + 0.14. Then judge that num < sum is satisfied, then randomly select a citation text from category 1 and add it to the reverse dataset, and jump out of this category traversal.

[0041] Step 3: Obtain the citation text data whose classification label is the chapter name of scientific literature, and construct the auxiliary dataset D a . The auxiliary dataset D a The citation text data in it is not repeated with the citation text data in the original dataset.

[0042] For example, if there are 12,000 citation text data with the classification label being the chapter name of a scientific document, they are first randomly shuffled, and then the citation text of the first part of the shuffled citation text data is taken. The number of data in this part of the data is consistent with the number of training set samples in the original data set.

[0043] Step 4: Build a multi-task bilateral branch network model and use the training set of the original dataset, the reversed dataset, and the auxiliary dataset for training.

[0044] The multi-task bilateral branch network model includes an input layer, an embedding layer, an attention layer, an auxiliary task module, a main task module, and an output layer;

[0045] The input layer is used to receive the text data in the training set, the reversed data set, and the auxiliary data set of the original data set, and then perform preprocessing operations on the text data;

[0046] The preprocessing operation of the text data is specifically as follows: since the text data in the training set of the original data set and the reversed data set contain the #AUTHOR_TAG tag, the tag in each citation text is deleted; the word segmentation tool is used to divide the entire citation text into individual words, and some punctuation marks and some stop words are removed.

[0047] For example, the input citation text data is: In addition, MoCo(#AUTHOR_TAG) showed that differentiating the model for anchors and positive / negative samples is effective.

[0048] After preprocessing, it becomes a divided word sequence:

[0049] X=[in,addition,moco,showed,that,differentiating,model,for,anchors,and,positive,negative,samples,is,effective]

[0050] The embedding layer uses the SciBert model to convert text data into a vector matrix H a 、H o 、H r ;

[0051] The SciBert model is used because it contains prior knowledge in the field of scientific literature. i, for \(i\in[1,\ldots,n]\), convert it to its corresponding representation vector \(h\) i :

[0052] h i = SciBert(x i ) Formula (3)

[0053] \(i\) is the serial number of a word in a citation text after preprocessing, and \(n\) is the number of real words in a citation text after preprocessing. Then, form the vector matrix \(H\) of the citation text with the representation vectors of each word:

[0054] \(H = [h_1,\ldots,h\) n ,\ldots,h\) L Formula (4)

[0055] is the vector matrix of the citation text. \(d\) is the length of the representation vector of each word, and \(L\) is the number of words in the longest citation text in a set of data. When \(n < L\), \(h\) n+1 ,\ldots,h\) L are all zero vectors.

[0056] For example, encode the word sequence \(X\) obtained by preprocessing to get the representation matrix \(H\):

[0057] \(H = SciBert(X)\)

[0058] The generated \(H\) is \(768\times15\), where 768 is the length of the representation vector of each word, and 15 is the number of words in the word sequence \(X\).

[0059] The attention layer includes a first text attention module and a second text attention module; the first text attention module receives \(H\) output by the embedding layer a 、\(H\) o , and for each word representation vector \(h\) a 、\(H\) o in the input vector matrices \(H\) i first calculate the corresponding weight \(\beta\) i , and then use the obtained weight \(\beta\) i to perform weighted summation on the input vector matrices \(H\) a 、\(H\) o to obtain the final representation vector \(s\) a 、\(s\) o of the citation text; the second text attention module receives \(H\) output by the embedding layer r , and for each word representation vector \(h\) r in the input vector matrix \(H\) i first calculate the corresponding weight \(\beta\) i , and then use the obtained weight \(\beta\) i to perform weighted summation on the input vector matrix \(H\) rThe weighted summation is used to obtain the final representation vector s of the citation text r ;

[0060] The first and second text attention modules are both implemented using the linear function, tanh activation function, and Softmax function of the deep learning framework Pytorch;

[0061] Each word in the input vector matrix H represents a vector h i The weight β is calculated through the linear function, activation function tanh and Softmax function i :

[0062] (1) Get the hidden state u i :

[0063] u i =tanh(W w h i +b w ) Formula (5)

[0064] Where W w 、b w They represent the attention weight matrix and attention offset vector respectively, and i is the sequence number of the word in the citation text after input layer preprocessing.

[0065] (2) Use the Softmax function to normalize the hidden state of each word in the citation text to obtain the weight β i :

[0066]

[0067] in is to u i Perform the transposition operation, i is the sequence number of the word in the quoted text after input layer preprocessing, exp(.) represents the exponential function with the natural constant e as the base; u w is a randomly initialized vector.

[0068] For example

[0069]

[0070] Among them, 0.01, 0.02, 0.009, …, 0.005, and 0.02 in β are the weight values of each word vector respectively.

[0071] Reuse the weight β of each word i The weighted summation is used to obtain the representation vector s of the citation text:

[0072] s=∑ i β i hi Formula (7)

[0073] The auxiliary task module includes a linear layer and a Softmax function layer;

[0074] (1) The linear layer represents the high-dimensional citation text vector s output by the attention layer a Transformed into an output vector Z with the same dimension as the number of categories in the auxiliary dataset a ,

[0075] Z a =W a s a +b a Formula (8)

[0076] where s a The representation vector representing the citation text, W a represents the weight matrix, b a represents the offset vector, n a Indicates the number of categories in the auxiliary dataset.

[0077] (2) Softmax function layer outputs vector Z a Perform normalization and calculate the probability distribution of each category in the prediction results

[0078]

[0079] where c a ∈[0,…,n a ], c a Indicates that the current category belongs to the cth category in the auxiliary dataset a categories.

[0080] For example

[0081]

[0082] As above a The probability value of index 0 in the probability distribution is the largest, so the model predicts that the citation text data is category 0.

[0083] The main task module includes a connection layer, a linear layer, and a Softmax function layer;

[0084] (1) The connection layer outputs s of the first text attention module o and s output by the second text attention module r Perform weighted splicing to get the splicing vector s m :

[0085] s m =[αso ,(1-α)s r ] Formula (10)

[0086] Where α is an adaptive parameter, and its calculation formula is as follows:

[0087]

[0088] Where T total is the total number of iterations required for model training, and T is the Tth iteration the model is currently in.

[0089] For example, the total number of iterations T total =150, the current iteration is the 31st, then

[0090] (2) The obtained splicing vector s m Input into the linear layer to get the output vector Z m ,

[0091] Z m =W m s m +b m Formula (12)

[0092] Where W m represents the main task weight matrix, b m Represents the main task offset vector, n c Indicates the number of categories in the original dataset.

[0093] (3) Softmax function layer outputs vector Z m Perform normalization and calculate the probability distribution of each category in the prediction results

[0094]

[0095] where c m ∈[0,…,n c ], c m Indicates that the current category belongs to the cth category in the original dataset m categories.

[0096] For example

[0097]

[0098] As above m The probability value of index 3 in the probability distribution is the largest, so the model predicts that the citation text data is category 3.

[0099] The output layer receives the output of the auxiliary task module and the main task module;

[0100] The output layer calculates the total loss of the model and uses it for backpropagation to update the model parameters. The cross entropy loss function is used to calculate the loss of the model, and its formula is:

[0101]

[0102] In the formula, E represents the number of samples, e represents the e-th sample, n represents the number of categories, c represents the c-th category, and y c Represents the true category of the sample. c Represents the probability that the sample is predicted to be category c.

[0103] Calculate the loss of the input of the training set of the original dataset, the auxiliary dataset and the reversed dataset respectively

[0104]

[0105] where y o ,y a ,y r The corresponding label values of each input citation text for the training set of the original dataset, the auxiliary dataset, and the reversed dataset respectively.

[0106] Final total loss The calculation is as follows:

[0107]

[0108] Where α is the adaptive parameter in the connection layer and λ is the model hyperparameter.

[0109] Step 5: Use the trained multi-task bilateral branch network model for verification and testing.

[0110] During the iterative training of the model, each citation text in the validation set is used as the x o and x r Input the iteratively trained model for prediction. (1) First define the variable val_f1 and initialize the variable best_val_f1 to 0. (2) Then use macro-F1 to calculate a value for the model's prediction results of all citation texts in the validation set and their corresponding true labels. And assign this value to the variable val_f1. (3) Determine whether val_f1>best_val_f1 is true. If so, save the model parameters obtained from this iterative training. And use the value of val_f1 to update the value of best_val_f1. If not, continue to repeat the (2) (3) process.

[0111] The verified multi-task bilateral branch network model is used to test the test set and the evaluation standard marco-F1 is used to judge its actual test results; each citation text in the test set is used as the x in the input layer. o and x r .

[0112] Step 6: Use the tested multi-task bilateral branch network model to classify the citation intent of the citation text.

[0113] This paper combines the public citation intent dataset (ACT) to perform citation intent classification analysis:

[0114] Table 1 Data volume of citation intent dataset

[0115] Intent Categories #of instances BACKGROUND 1648 COMPARES_CONTRASTS 368 EXTENSION 171 FUTURE 62 MOTIVATION 276 USES 475

[0116] From the table above, we can see that the number of samples belonging to the BACKGROUND category is much greater than that of the FUTURE category.

[0117] Table 2 Prediction experimental results of the present invention on the citation intention dataset

[0118] Model Public Private TextCNN 0.16538 0.14954 BiLSTM+Attention 0.21719 0.19254 RCNN 0.24926 0.19830 UFMG 0.19944 0.21516 SciCite 0.24926 0.22334 MTBBN 0.25452 0.25972

[0119] This task is publicly available on the Kaggle website, and the competition ranking is based on the Private score. The Public score is calculated using approximately 50% of the data in the test set, while the remaining data is used to calculate the Private score.

[0120] In the above citation intent prediction experimental results table, TextCNN, RCNN, and BiLSTM+Attention are classic text classification models. UFMG is the best-performing model on the same dataset as our present invention, and SciCite is the best-performing model on other similar research datasets. MTBBN is the multi-task bilateral branch network model used in our present invention. The macro-F1 score was used as the performance evaluation metric for citation intent classification.

[0121] The F1-score is a statistical metric used to measure the accuracy of binary classification models. It's used to measure the accuracy of imbalanced data, taking into account both the precision and recall of the classification model. In multi-classification problems, there are two variants of the F1-score: micro-F1 and macro-F1. However, micro-F1 is susceptible to data imbalance. Therefore, macro-F1 is used as the classification performance evaluation metric in this paper, as it is less susceptible to data imbalance.

[0122] The F1-score evaluation indicator formula is as follows:

[0123]

[0124] Precision and recall represent the classification accuracy and recall, respectively. Precision calculates the proportion of predicted positive samples that are actually positive, while recall calculates the proportion of predicted positive samples among the actual positive samples. As can be seen from the above formula, the F1-score is an evaluation metric that combines the precision and recall of a classifier.

[0125] Since F1-score is applicable to binary classification problems, the citation intent in the experiment is a multi-classification problem, so macro-F1 is used as the evaluation indicator. Macro-F1 counts the TP, FP, TN, and FN of each category, calculates their respective precision and recall, and obtains the F1 value of each category. Then, the average is taken to obtain macro-F1, that is:

[0126]

[0127] Among them, F1-score1, F1-score2, …, F1-score n They represent the F1-score of the 0th, 1st, ... n-1th categories respectively, and n is the number of categories.

Claims

1. A citation intent classification method based on a multi-task bilateral branch network, characterized by The following steps are involved: Step 1: Obtain citation text data with intent classification labels as the original dataset D o ; At the same time, the entire original data set is divided into the training set, validation set, and test set of the original data set; Step 2: Use oversampling method to adjust D o The training set is sampled to generate the reverse data set D r ; The details are as follows: 2-1 Calculate the ratio of each category in the training set of the original data set according to formula (1) in Indicates c m The number of samples in each category; N max Indicates the number of samples corresponding to the category with the largest number of samples; 2-2 Using the ratio of each category Calculate the sampling probability of each category where n c the number of categories representing intent; 2-3 Iteration Sampling: 2-3-1 Initialization: Use the library function to randomly obtain a value num, the value range of num is [0,1); at the same time, initialize the variable sum to 0; 2-3-2 Traverse each category in the training set of the original data set in turn (1) For the traversed categories, use the sampling probability of the current category Update the value of sum, that is (2) Determine whether the random number num≤sum is satisfied. If so, randomly select a citation text data from the current category and add it to the reversed data set. Otherwise, do nothing. (3) Traverse the next category and repeat steps (1)-(2) until all categories are traversed; 2-3-3 Determine whether the number of elements in the reversed dataset is the same as the training set of the original dataset. If so, end; otherwise, return to step 2-3-1; Step 3: Obtain citation text data whose classification labels are chapter names of scientific literature and construct auxiliary dataset D a ; Step 4: Build a multi-task bilateral branch network model and train it using the training set of the original dataset, the reversed dataset, and the auxiliary dataset; The multi-task bilateral branch network model includes an input layer, an embedding layer, an attention layer, an auxiliary task module, a main task module, and an output layer; The input layer is used to receive the text data in the training set, the reversed data set, and the auxiliary data set of the original data set, and then perform preprocessing operations on the text data; The embedding layer uses the SciBert model to convert text data into a vector matrix H a 、H o 、H r ; The attention layer includes a first text attention module and a second text attention module; The auxiliary task module includes a linear layer and a Softmax function layer; The main task module includes a connection layer, a linear layer, and a Softmax function layer; The output layer receives the output of the auxiliary task module and the main task module; Step 5: Use the trained multi-task bilateral branch network model for verification and testing; Step 6: Use the tested multi-task bilateral branch network model to classify the citation intent of the citation text.

2. A citation intent classification method based on a multi-task bilateral branch network according to claim 1, characterized in that The preprocessing operation of the text data based on the input layer in the multi-task bilateral branch network model is specifically as follows: if the text data in the training set of the original data set and the reversed data set contains the #AUTHOR_TAG tag, then the tag in each citation text is deleted; then the word segmentation tool is used to divide the entire citation text into individual words, while removing punctuation marks and stop words.

3. The citation intent classification method based on a multi-task bilateral branch network according to claim 1 is characterized in that The SciBert model vectorizes the words corresponding to the citation text data in the original dataset, reversed dataset, and auxiliary dataset input into the input layer, and then obtains the vector matrix corresponding to the citation text in the original dataset, reversed dataset, and auxiliary dataset; Specifically: For each word x i , i∈[1,…n] is converted into its corresponding representation vector h i : h i = SciBert(x i ) Formula (3) Where i is the index of the word in the quoted text, and n is the total number of words in the quoted text; Then the representation vector of each word is combined into the vector matrix H of the citation text: H=[h1,…,h n ,…,h L ] Formula (4) in is the vector matrix of the citation text; d is the length of the representation vector of each word, and L is the longest word number of the citation text.

4. The citation intent classification method based on a multi-task bilateral branch network according to claim 3 is characterized in that The first text attention module receives the H output by the embedding layer a 、H o , for the matrix H a 、H o Each word in the vector h i First calculate the corresponding weight β i Then use the weight β i For the matrix H a 、H o The weighted summation is used to obtain the representation vector s of the original dataset and the citation text in the reversed dataset a 、s o ; Among them H a 、H o Represent the vector matrices corresponding to the citation texts in the original dataset and the reversed dataset respectively; The second text attention module receives the H output by the embedding layer r , for the matrix H r Each word in the vector h i First calculate the corresponding weight β i Then use the weight β i For the matrix H r The weighted summation is used to obtain the representation vector s of the citation text in the auxiliary dataset r ; Among them H r A vector matrix representing the corresponding citation text in the auxiliary dataset.

5. The citation intent classification method based on a multi-task bilateral branch network according to claim 4 is characterized in that The linear layer in the auxiliary task module converts the citation text representation vector s in the original dataset output by the attention layer into a Transformed into an output vector Z with the same dimension as the number of categories in the auxiliary dataset a , Z a =W a s a +b a Formula (5) where s a Represents the representation vector of the citation text in the original dataset, W a represents the weight matrix of the auxiliary task, b a represents the offset vector of the auxiliary task, n a Indicates the number of categories in the auxiliary dataset; The Softmax function layer in the auxiliary task module outputs the vector Z a Perform normalization and calculate the probability distribution of each category in the prediction results where c a ∈[0,…,n a ].

6. The citation intent classification method based on a multi-task bilateral branch network according to claim 5 is characterized in that The connection layer in the main task module outputs s of the first text attention module o and s output by the second text attention module r Perform weighted splicing to get the splicing vector s m : s m = [αs o , (1 - α)s r Equation (7) Where α represents the adaptive parameter, and its calculation formula is as follows: Where T total is the total number of iterations of model training, and T is the current number of iterations; The linear layer is used to receive the concatenated vector s m , then output vector Z m , Z m =W m s m +b m Formula (9) Where W m represents the main task weight matrix, b m Represents the main task offset vector, n c Indicates the number of categories in the original dataset; The Softmax function layer outputs the vector Z to the linear layer m Perform normalization and calculate the probability distribution of each category in the prediction results where c m ∈[0,…,n c ].

7. The citation intent classification method based on a multi-task bilateral branch network according to claim 6 is characterized in that The loss of the model uses the cross entropy loss function, and its formula is: Where E represents the number of samples, e represents the e-th sample, n represents the number of categories, c represents the c-th category, and y c represents the true category of the sample, z c Represents the probability that the sample is predicted to be category c; Calculate the loss of the input of the training set of the original dataset, the auxiliary dataset and the reversed dataset respectively where y o ,y a ,y r are the corresponding label values of each input citation text of the training set of the original dataset, the auxiliary dataset, and the reversed dataset respectively; Final total loss The calculation is as follows: Where λ is a model hyperparameter.

8. The citation intent classification method based on a multi-task bilateral branch network according to claim 7 is characterized in that The first and second text attention modules are both implemented using the linear function, tanh activation function, and Softmax function of the deep learning framework Pytorch; Each word in the input vector matrix H is represented by a vector h i The weight β is calculated through the linear function, activation function tanh and Softmax function i : (1) Get the hidden state u i : u i =tanh(W w h i +b w ) Formula (14) Where W w 、b w Represent the attention weight matrix and attention offset vector respectively; (2) Use the Softmax function to normalize the hidden state of each word in the citation text to obtain the weight β i : in is to u i Perform the transposition operation, exp(.) represents the exponential function with the natural constant e as the base; u w is a randomly initialized vector; Reuse the weight β of each word i The weighted summation is used to obtain the representation vector s of the citation text: s = ∑ i β i h i Equation (16).

9. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 8 is implemented.

Images