Methods for determining the impact of a paper

Abstract

The present invention discloses a method for determining the influence of a paper, comprising: obtaining the full text of the citing documents of the paper to be analyzed, performing data preprocessing on the citing documents to obtain the citation texts in the citing documents, and annotating some of the citation texts to obtain a label category set for the citation texts; using the label category set of the citation texts obtained in step 1 as a training set to train a first DeepMoji model, and using the trained first DeepMoji model to perform sentiment classification on unannotated citation texts; obtaining social media mentions of the paper to be analyzed, repeating the above steps to obtain a trained second DeepMoji model, and using the second DeepMoji model to perform sentiment classification on unannotated social media mentions; and calculating the paper's influence index by assigning different weights to different emotions in the citation texts and social media mentions. The present invention achieves a more fine-grained influence calculation method that can distinguish different citation content and social media mention content.

Description

Technical Field

[0001] The present invention belongs to the technical field of paper information measurement, and particularly relates to a method for determining the influence of a paper. Background Art

[0002] Traditional methods for calculating paper influence based on citation frequency have been widely used in academic evaluation and research performance management, resulting in a series of influential metrics such as the Journal Impact Factor (IF) and the H-index. However, these methods treat all citations equally and fail to effectively distinguish the specific differences between different citations. Furthermore, with the rise of online scholarly communication and publishing, more and more academic work is being disseminated and cited online. Traditional methods for evaluating academic papers are increasingly unable to meet the needs of comprehensive, timely, and objective evaluation. Incorporating altmetrics (social influence indicators) into evaluation systems has gradually become a focus of researchers. However, current methods for automatically identifying sentiment in citation text still primarily rely on constructing sentiment dictionaries, which have low accuracy. Research on sentiment analysis of text mentions of academic papers on social media is also limited. Therefore, how to effectively identify sentiment in academic texts and integrate them with social influence indicators to achieve a more granular assessment of paper influence has become a critical issue. Summary of the Invention

[0003] The purpose of the present invention is to address the shortcomings of the existing technology and provide a method for determining the influence of a paper. The method achieves a more accurate automatic classification effect of specific text emotions, and calculates the influence index by assigning different weights to different emotions, thereby realizing a more fine-grained influence calculation method that can distinguish different citation content and social media mentions.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A method for determining the influence of a paper includes the following steps:

[0006] Step 1: Obtain the full text of the citing articles of the paper to be analyzed, perform data preprocessing on them to obtain the citation texts in the citing articles, and annotate some of the citation texts to obtain the label category set of the citation texts;

[0007] Step 2: Use the label category set of the citation text obtained in step 1 as a training set to train the first DeepMoji model, and use the trained first DeepMoji model to perform sentiment classification on the unlabeled citation text;

[0008] Step 3: Obtain social media mentions of the paper to be analyzed, annotate some of the social media mentions to obtain a label category set for the social media mentions, use this label category set to train a second DeepMoji model, and use the trained second DeepMoji model to perform sentiment classification on unannotated social media mentions.

[0009] Step 4: Calculate the influence index of the paper by assigning different weights to different emotions in the citation text obtained in step 2 and the social media mention text in step 3.

[0010] Furthermore, step 1 specifically includes the following sub-steps:

[0011] S1.1. Obtain the full text of the cited articles and the corresponding citing articles of the papers to be analyzed from the database, and delete the full text of the cited articles and citing texts for which the full text cannot be obtained;

[0012] S1.2. Convert the full-text PDF of the cited article to TXT format, preserving the citation marks;

[0013] S1.3. Use regular expression to match citation tags to obtain all citation texts in citing documents;

[0014] S1.4. Construct label categories for citation objects, citation motivations, and citation emotions, as well as corresponding classification criteria;

[0015] S1.5. Manually label the cited object, citation motivation, and citation sentiment of the citation text according to the determined classification criteria, and convert the citation text into a citation text feature vector to obtain a labeled set of citation texts.

[0016] Furthermore, the label categories of the reference objects include background, terminology, methods, and conclusions.

[0017] Furthermore, the label categories included in the citation motivation include background, based on, support, criticism, and different.

[0018] Furthermore, the label categories of reference sentiment include positive, neutral, and negative.

[0019] Furthermore, step 2 specifically includes the following sub-steps:

[0020] S2.1 Initialize the model parameters of the first DeepMoji model, and input the citation text feature vector obtained in step 1 into the first Embedding layer of the first DeepMoji model to obtain the embedding feature vector q of the citation text i ;

[0021] S2.2, embedding feature vector q of citation text obtained in S2.1 i Input the second and third BiLSTM layers of the first DeepMoji model to obtain the feature vector q′ i and q″ i ;

[0022] S2.3, the embedding feature vector q obtained in S2.1 i , the eigenvector q′ obtained in S2.2 i ,q″ i Splice and get the feature vector p i And the eigenvector p i Input into the third layer of the Attention of the first DeepMoji model to obtain the high-dimensional feature vector p″ of the citation text i ;

[0023] S2.4, p″ i Input the fourth Softmax layer of the first DeepMoji model and output the classification labels of the citation sentiment, citation emotion and citation motivation of the citation text;

[0024] S2.5. Training the model parameters of the first DeepMoji model using the annotated label categories obtained in step 1 and the classification labels of the citation sentiment, citation emotion, and citation motivation of the citation text obtained in step 2.4. During the training process, establishing a loss function and optimizing the model parameters through backpropagation until the loss function converges, thereby obtaining the optimized model parameters of the first DeepMoji model.

[0025] S2.6. Use the trained first DeepMoji model to classify the citation sentiment of unlabeled citation text.

[0026] Furthermore, the loss function established in step 2.5 is the cross entropy loss function L, whose formula is:

[0027]

[0028] Where, is the reference sentiment label output by the Softmax layer, is the output reference object label, is the output citation motivation label; y i is the manually annotated sentiment label of the reference in the training set, y′ i is the reference object label manually annotated in the training set, y i are manually annotated citation motivation labels in the training set; i = 1, 2, …, N, where N is the total number of samples; λ1 and λ2 are both model parameters.

[0029] Furthermore, step 3 specifically includes the following sub-steps:

[0030] S3.1. Search the social media platforms by the title or DOI number of the papers mentioned in the analysis to obtain the social media mention text;

[0031] S3.2. Determine the tag categories for mentioning motivations and emotions and their corresponding classification criteria, and annotate the tag categories of some social media mentions based on the constructed tag categories and their classification criteria;

[0032] S3.3. Convert the social media mention text into a social media mention text feature vector. According to the label categories of some social media mention texts obtained in step S3.2 and their corresponding feature vectors, obtain a labeled set. Input the feature vectors of the social media mention texts in the set layer by layer into the second DeepMoji model to train its model parameters. Use the trained second DeepMoji model to classify the mention sentiment of unlabeled social media mention texts.

[0033] Furthermore, the tag categories mentioning motivation include summarizing or citing the article's findings, criticizing or questioning the article's content or findings, endorsing or recommending the article, and applying it to real life, and the tag categories mentioning emotion include neutral, positive, and negative.

[0034] Furthermore, the influence index of the paper constructed in step 4 is:

[0035]

[0036] Among them, w1 is the weight of positive citations or mentions, w2 is the weight of neutral citations or mentions, and w3 is the weight of negative citations or mentions; c i is the number of positive citations in the cited text, c′ i is the number of neutral citations in the cited text, c″ i is the number of negative citations in the cited text, c i +c′ i +c″ i =N;m i is the positive mention in the social media mention text, m′ i For neutral mentions in social media mentions, m″ i For negative mentions in social media mentions, m i +m′ i +m″ i =N′.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. This method combines traditional citation analysis indicators with emerging Altmetrics indicators, reflecting both the academic influence of a paper and its public attention and social influence, and can reveal the actual influence of a paper in a more comprehensive and profound way.

[0039] 2. This paper uses manually annotated labels to fine-tune the DeepMoji sentiment classification model, improving the DeepMoji model's feature extraction capability for specific text classification tasks, obtaining a more accurate semantic feature vector representation, and achieving a more accurate automatic classification effect of specific text sentiment;

[0040] 3. This paper constructs a multi-task learning framework for the DeepMoji model. In the citation text sentiment classification task, two auxiliary tasks, namely, citation object classification and citation motivation classification, are used to help the model train. In the social media mention text sentiment classification task, the auxiliary task of mention motivation is used to help the model train, making it more robust.

[0041] 4. The present invention uses the DeepMoji model to realize automatic sentiment recognition of citation texts and texts that mention papers on social media, and calculates the influence index by assigning different weights to different emotions, thus realizing a more fine-grained influence calculation method that can distinguish different citation contents and social media mentions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a block diagram of an implementation method for determining the influence of a paper according to an embodiment of the present invention;

[0043] Figure 2 Flowchart for the implementation of the method for determining the influence of a paper according to an embodiment of the present invention;

[0044] Figure 3 A DeepMoji model diagram for sentiment classification of citation text according to an embodiment of the present invention;

[0045] Figure 4 A diagram of the DeepMoji model for sentiment classification of social media mentions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0048] The present invention will be further described below with reference to specific examples, but they are not intended to limit the present invention.

[0049] like Figure 1 and Figure 2 As shown, the embodiment of the present invention discloses a method for determining the influence of a paper, comprising the following steps:

[0050] Step 1: Obtain the full text of the citing articles of the paper to be analyzed, perform data preprocessing on them to obtain the citation texts in the citing articles, and annotate some of the citation texts to obtain the label category set of the citation texts; this step includes the following sub-steps:

[0051] S1.1. Obtain the full text of the cited articles and the corresponding citing articles of the papers to be analyzed from the database, and delete the cited articles and citing articles for which the full text cannot be obtained;

[0052] S1.2. Convert the full-text PDF of the cited article into a TXT format text, retaining the citation marks. When a cited article appears in a citing article, it is usually marked in the format of "[number]". For example, the introduction section of the cited article "SARS-CoV-2 variants of concern and vaccine escape, from Alpha to Omicron and beyond" in its citing article "Antibody persistence and safety after heterologous boosting with orallyaerosolised Ad5-nCoV in individuals primed with two-dose CoronaVac previously: 12-month analyses of a randomized controlled trial" is marked as "[3]", where "[3]" is the citation mark and the sentence containing "[3]" is the citation sentence. Use Python language to convert all the full-text PDFs of the cited articles obtained in S1.1 into TXT format text, and retain all the citation marks "[]" in the text;

[0053] S1.3. Use Python's regular expression to match the citation mark "[]" and obtain the citation text in the citing article. Here, the citation text is defined as the citation sentence containing the citation mark, as well as the two preceding and two following sentences. A citing article may match multiple citation marks, so all citation texts must be extracted.

[0054] S1.4. Construct label categories for the cited object, citation motivation, and citation sentiment of the citation text, as well as the corresponding classification criteria for the label categories. See Tables 1, 2, and 3. Table 1 lists the label categories for the cited object, Table 2 lists the label categories for the citation motivation, and Table 3 lists the label categories for the citation sentiment.

[0055] Table 1 shows the label categories of the reference objects

[0056]

[0057]

[0058] Table 2 shows the label categories of citation motivations

[0059]

[0060] Table 3 shows the label categories of reference emotions

[0061]

[0062]

[0063] S1.5. Manually label the cited texts by citation object, citation motivation, and citation sentiment according to the determined classification criteria. CountVectorizer and TfidfVectorizer functions in the Python language sklearn module are used to convert all citation texts into citation text feature vectors to obtain a labeled set of citation texts. Among them, x i represents the citation text feature vector in the i-th sample; y i represents the reference sentiment label in the i-th sample; y i Indicates the reference object label; y′ i Indicates the citation motivation label, all labels are manually annotated; i = 1, 2, ..., N, where N is the total number of samples.

[0064] Step 2: Use the label category set of the citation text obtained in step 1 as the training set to train the first DeepMoji model, and use the trained first DeepMoji model to perform sentiment classification on the unlabeled citation text. Figure 3 ; This step specifically includes the following sub-steps:

[0065] S2.1 Take the citation text vector x in the labeled set obtained in step 1 i Input the first Embedding layer of the first DeepMoji model. This layer can embed each word in the text into the vector space and compress the embedding dimension to [-1, 1] through the tanh activation function. This can output the embedding feature vector of the citation text, which is expressed as:

[0066] q i =tahn(Embedding(x i ));

[0067] Among them, q i is the embedding feature vector of the citation text, with a feature dimension of 256, x i is the citation text feature vector, tahn is the activation function hyperbolic tangent function, and Embedding is the embedding layer in the first DeepMoji model;

[0068] S2.2, embedding feature vector q of citation text obtained in S2.1 i Input the second and third layers of the first DeepMoji model. Both layers are BiLSTM layers, i.e., bidirectional LSTM layers (long short-term memory recurrent neural network). Each layer is composed of a forward LSTM and a backward LSTM. LSTM in both directions is used to model context information. Each layer has 1024 hidden units, 512 in each direction. The embedding feature vector q i After inputting into the second and third layers, the feature vectors q′i and q″ are obtained respectively. i :

[0069] q′ i =relu(BiLSTM(s i ))

[0070] q″ i =relu(BiLSTM′(s′ i ))

[0071] Among them, BiLSTM is the first BiLSTM layer of the first DeepMoji model, q′ i is the citation text embedding feature vector s i The feature vector obtained after inputting the first BiLSTM layer has a feature dimension of 1024; BiLSTM′ is the second BiLSTM layer of the first DeepMoji model, q″i is q′ i The feature vector obtained after inputting the second BiLSTM layer has a feature dimension of 1024; relu is the activation function;

[0072] S2.3, the embedding feature vector q obtained in S2.1 i , the eigenvector q′ obtained in S2.2 i and q″ i Perform concatenation to obtain the feature vector:

[0073] p i =concat(q i ,q′ i ,q″ i )

[0074] Among them, p i The feature dimension is 2304;

[0075] By using skip-connections (to alleviate the gradient explosion and gradient disappearance caused by the increase in the number of network layers), p i Input the third Attention layer of the DeepMoji model and output a high-dimensional feature vector representation of the citation text:

[0076] p′ i =Dropout(Attention(p i ))

[0077] p″ i =Flatten(Dropout(tahn(Dense(p′ i ))));

[0078] Among them, p i is the concatenated 2304-dimensional feature vector, Attention is the attention layer in the DeepMoji model, and Dropout is a function set to prevent overfitting of the neural network; p″ i is the citation text feature vector output by the Attention layer, Dense is the fully connected layer, tahn is the activation function, and Flatten is the dimensionality reduction function that flattens the feature vector into one dimension;

[0079] S2.4, p″ i Input the fourth Softmax layer of the first DeepMoji model and output the classification labels of citation sentiment, citation emotion, and citation motivation:

[0080]

[0081]

[0082]

[0083] Among them, Softmax is an activation function commonly used in multi-classification problems. It can normalize a numerical vector into a probability distribution vector, and the sum of each probability is 1; is the output reference sentiment label, is the output reference object label, is the output citation motivation label;

[0084] S2.5. Initialize the model parameters of the first DeepMoji model. Train the model parameters of the first DeepMoji model using the manual annotation label categories obtained in step 1 combined with the classification labels of the citation sentiment, citation emotion, and citation motivation of the citation text obtained in step 2.4. During the training process, establish a loss function and optimize the model parameters through backpropagation until the loss function converges to obtain the optimized model parameters of the first DeepMoji model. The loss function is the cross entropy loss function L:

[0085]

[0086] Where, is the reference sentiment label output by the Softmax layer, is the output reference object label, is the output citation motivation label; y i is the manually annotated sentiment label of the reference in the training set, y′ i is the reference object label manually annotated in the training set, y i are the manually annotated citation motivation labels in the training set; i = 1, 2, …, N, where N represents the total number of samples; λ1 and λ2 are both model parameters, and their values are both 0.05.

[0087] S2.6. Use the trained first DeepMoji model to perform sentiment classification on unlabeled citation text.

[0088] Step 3: Obtain social media mentions of the paper to be analyzed, annotate some of the social media mentions to obtain a label category set for the social media mentions, use this label category set to train a second DeepMoji model, and use the trained second DeepMoji model to perform sentiment classification on unannotated social media mentions. This step specifically includes the following substeps:

[0089] S3.1. Altmetrics, short for altmetrics, represents the social dissemination of scholarly work. Its website, Altmetrics.com, captures records of papers being shared and cited on social networking platforms. Searching Altmetrics.com by the title or DOI of the paper being analyzed yields text mentions of the paper on various social media platforms, referred to here as "social media mentions."

[0090] Since some social networking platforms are not widely used and have relatively little data, we selected texts from three representative platforms based on existing research: Twitter texts, Facebook texts, and Blog texts. We removed non-English texts and texts with incomplete links to obtain the social media mentions of the papers to be analyzed on the three platforms.

[0091] S3.2. Determine the label categories for mentions of motivation and emotion and their corresponding classification criteria. Manually annotate the label categories of some social media mentions based on the constructed label categories and their classification criteria. Furthermore, convert the social media mentions into feature vectors to obtain their label sets. See Table 4 for a table of label categories for mentions of motivation and emotion.

[0092] Table 4 shows the label categories of mentioned motivations and mentioned emotions

[0093]

[0094] S3.3. Use the CountVectorizer and TfidfVectorizer functions in the Python language sklearn module to convert all social media mention texts into social media mention text feature vectors. According to the label categories of some social media mention texts obtained in step S3.2 and their corresponding feature vectors, a labeled set is obtained. Among them, s i represents the social media mention text feature vector in the i-th sample; z i represents the mentioned sentiment label in the i-th sample, z′ i Indicates the motivation label mentioned, all labels are manually annotated; i = 1, 2, ..., N', N' represents the total number of samples;

[0095] S3.4, initialize the model parameters of the second DeepMoji model, and set s i The second DeepMoji model is input layer by layer to train its model parameters, and the trained second DeepMoji model is used to classify the emotions of unlabeled social media mentions, such as Figure 4 shown.

[0096] S4. Calculate the influence index of the paper by assigning different weights to different emotions in the citation text obtained in step 2 and the social media mention text obtained in step 3. The influence index is specifically:

[0097]

[0098] Among them, w1 is the weight of positive citations or mentions, which is 1; w2 is the weight of neutral citations or mentions, which is 0.5; w3 is the weight of negative citations or mentions, which is 0.25; c i is the number of positive citations in the cited text, c′ i is the number of neutral citations in the cited text, c″ i is the number of negative citations in the cited text, c i +c′ i +c″ i =N;m i is the positive mention in the social media mention text, m′ i For neutral mentions in social media mentions, m″ i For negative mentions in social media mentions, m i +m′ i +m″ i =N′.

[0099] The above are only preferred embodiments of the present invention and do not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of the present invention specification should be included in the protection scope of the present invention.

Claims

1. A method for determining the influence of a paper, characterized by: The steps include: Step 1: Obtain the full text of the citing articles of the paper to be analyzed, perform data preprocessing on them to obtain the citation texts in the citing articles, and annotate some of the citation texts to obtain the label category set of the citation texts; Step 2: Use the label category set of the citation text obtained in step 1 as a training set to train the first DeepMoji model, and use the trained first DeepMoji model to perform sentiment classification on the unlabeled citation text; Step 3: Obtain social media mentions of the paper to be analyzed, annotate some of the social media mentions to obtain a label category set for the social media mentions, use this label category set to train a second DeepMoji model, and use the trained second DeepMoji model to perform sentiment classification on unannotated social media mentions. Step 4: Calculate the paper's influence index by assigning different weights to the different sentiments in the citation texts obtained in step 2 and the social media mention texts obtained in step 3; Wherein, step 1 specifically includes the following sub-steps: S1.

1. Obtain the full text of the cited articles and the corresponding citing articles of the papers to be analyzed from the database, and delete the cited articles and citing articles for which the full text cannot be obtained; S1.

2. Convert the full-text PDF of the cited article to TXT format, preserving the citation marks; S1.

3. Use regular expression to match citation tags to obtain all citation texts in citing documents; S1.

4. Construct label categories for citation objects, citation motivations, and citation emotions, as well as corresponding classification criteria; S1.

5. Manually label the cited object, citation motivation, and citation sentiment of the citation text according to the determined classification criteria, and convert the citation text into a citation text feature vector to obtain a labeled set of citation texts; The label categories for the cited objects include background, terminology, methods, and conclusions; The label categories included in the citation motivation include background, based on, support, criticism, and different.

2. The method for determining the influence of a paper according to claim 1, characterized in that: The label categories of reference sentiment include positive, neutral, and negative.

3. The method for determining the influence of a paper according to claim 1, characterized in that: Step 2 specifically includes the following sub-steps: S2.1 Initialize the model parameters of the first DeepMoji model and input the citation text feature vector obtained in step 1 into the first Embedding layer of the first DeepMoji model to obtain the embedding feature vector of the citation text ; S2.2, embedding feature vector of citation text obtained in S2.1 Input the second and third BiLSTM layers of the first DeepMoji model to obtain the feature vectors and ; S2.3, embedding feature vector obtained in S2.1 , the feature vector obtained in S2.2 、 Splice and get the feature vector And the feature vector Input into the third layer of the Attention of the first DeepMoji model to obtain the high-dimensional feature vector of the citation text ; S2.4, Input the fourth Softmax layer of the first DeepMoji model and output the classification labels of the citation sentiment, citation emotion and citation motivation of the citation text; S2.

5. Training the model parameters of the first DeepMoji model using the annotated label categories obtained in step 1 and the classification labels of the citation sentiment, citation emotion, and citation motivation of the citation text obtained in step 2.

4. During the training process, establishing a loss function and optimizing the model parameters through backpropagation until the loss function converges, thereby obtaining the optimized model parameters of the first DeepMoji model. S2.

6. Use the trained first DeepMoji model to classify the citation sentiment of unlabeled citation text.

4. The method for determining the influence of a paper according to claim 3, characterized in that: The loss function established in step 2.5 is the cross entropy loss function L, whose formula is: Where, is the reference sentiment label output by the Softmax layer, is the output reference object label, is the output citation motivation label; is the manually annotated sentiment label of the reference in the training set, is the reference object label manually annotated in the training set, Manually annotated citation motivation labels in the training set; represents the total number of samples; and are all model parameters.

5. The method for determining the influence of a paper according to claim 1, characterized in that: Step 3 specifically includes the following sub-steps: S3.

1. Search the social media platforms by the title or DOI number of the papers mentioned in the analysis to obtain the social media mention text; S3.

2. Determine the tag categories for mentioning motivations and emotions and their corresponding classification criteria, and annotate the tag categories of some social media mentions based on the constructed tag categories and their classification criteria; S3.

3. Convert the social media mention text into a social media mention text feature vector. According to the label categories of some social media mention texts obtained in step S3.2 and their corresponding feature vectors, obtain a labeled set. Input the feature vectors of the social media mention texts in the set layer by layer into the second DeepMoji model to train its model parameters. Use the trained second DeepMoji model to classify the mention sentiment of unlabeled social media mention texts.

6. The method for determining the influence of a paper according to claim 5, characterized in that: The tag categories that mentioned motivation included summarizing or citing the findings of the article, criticizing or questioning the content or findings of the article, endorsing or recommending the article, and applying it to real life. The tag categories that mentioned emotion included neutral, positive, and negative.

7. The method for determining the influence of a paper according to claim 1, characterized in that: The impact index of the paper constructed in step 4 is: in, is the weight of positive citations or mentions, is the weight of a neutral reference or mention, The weight of negative citations or mentions; is the number of positive citations in the cited text, is the number of neutral citations in the cited text, is the number of negative citations in the cited text, ; For positive mentions in social media mentions, For social media mentions of neutral mentions in text, Negative mentions in text for social media mentions, .

Images