Scientific paper academic value measurement method based on semantic fusion and reference behavior analysis

Abstract

The invention discloses a scientific paper academic value measurement method based on semantic fusion and reference behavior analysis. The method comprises the following steps: determining a literature database according to a target research field, constructing a field keyword list, and retrieving and collecting an initial literature data set; the internal innovation value is quantified; quantizing an external approved value; the step of constructing the academic value comprehensive measurement model comprises the sub-steps of applying an entropy weight method to distribute index weights; calculating an academic value comprehensive score through linear weighting; checking and evaluating through a Spearman correlation coefficient; according to the method, a comprehensive evaluation system based on multi-dimensional semantics and behavior characteristics is constructed, internal knowledge contribution and external recognition value of the papers are systematically fused, objectivity and precision of scientific research paper academic value evaluation can be effectively improved, and therefore the problems of evaluation time lag, reference behavior deviation and subjective weighting existing in a traditional method are solved, and the scientific research paper academic value evaluation efficiency is improved. And the method has good universality and popularization value.

Description

Technical Field

[0001] The present invention relates to the technical fields of data representation and computer systems, and particularly to a method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis. Background Art

[0002] In the early stage, the evaluation of academic value often adopted the method of "evaluating papers by journals" and using the number of citations to replace the value, which was objectively reasonable to a certain extent. However, the disadvantages of the above evaluation methods are also very obvious. The journal impact factor reflects the average level of the papers published in the journal. Regarding the average level as the level of a specific paper violates basic statistical common sense; the number of citations reflects the application situation of the paper by other studies. On the one hand, simple counting cannot reflect how it is applied, and on the other hand, it simply equates influence with quality. This not only leads to the narrowing of the evaluation dimension but also causes the distortion of value judgment. With the iterative upgrade of the scientific metrology paradigm, research on scientific evaluation gradually uses fine-grained content mining technology to conduct in-depth analysis from citation content such as citation sentiment and citation intensity. Currently, based on semantic fusion and citation behavior analysis, it is possible to effectively identify the internal innovation value from knowledge evolution and the external recognition value from the consensus of the academic community in scientific papers. However, the previous methods for measuring the academic value of papers often only relied on explicit measurement indicators and failed to fully consider the integration of internal knowledge and external explicit measurement indicators, resulting in problems such as delayed value manifestation and too coarse analysis granularity.

[0003] Currently, the existing technology uses natural language processing methods to optimize the parsing of citation content to reduce evaluation lag, or realizes the tracking of the influence propagation path through citation network analysis to improve the accuracy and applicability of the evaluation model, thereby enhancing the overall sensitivity of academic value recognition. However, it only starts from a single dimension of the statistical characteristics of the paper text or network topology indicators, without considering the fusion characteristics of semantics and citation relationships, and lacks exploration and evidence collection in aspects such as innovation quality - contribution impact. Some existing technologies also adopt citation sentiment classification and identify method innovation in scientific papers, and manually identify and label high-value papers with the support of expert experience. However, this semi-automated method requires a large amount of domain knowledge annotation and cannot expand the evaluation scenario. Or the existing value representation dimension is relatively single, it is difficult to analyze the knowledge contribution gradient in fine granularity, and there is a lack of an academic value measurement method that constructs an internal innovation quality - external academic influence from the perspective of the integration of innovation quality and contribution impact. Summary of the Invention

[0004] Objective of the Invention: The objective of the present invention is to provide a method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis. By integrating an innovation recognition model based on full-text semantic analysis and an external recognition modeling mechanism based on citation network behavior patterns, and supported by a comprehensive index system of citation sentiment, citation intensity, and citation relevance, a citation network integrating semantic features is constructed and analyzed in a fine-grained manner to solve the technical problems of single value representation dimension, insufficient timeliness, and strong subjective dependence in existing technical methods, reduce the dependence on expert knowledge and manual labor, and improve the intelligence, objectivity, and scalability of value evaluation.

[0005] Technical Solution: A method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis according to the present invention includes the following steps: (1) Determine a literature database according to the target research field, construct a list of domain keywords, and retrieve and collect an initial literature dataset; (2) Quantify the internal innovation value; (3) Quantify the external recognition value; (4) Construct a comprehensive measure model for academic value, including: applying the entropy weight method to allocate index weights; calculating the comprehensive score of academic value by linear weighting; (5) Evaluate through Spearman correlation coefficient test.

[0006] Further, step (1) includes the following steps: (11) Filter out irrelevant literature through a text matching algorithm; (12) Obtain the full text of the paper, and use text structure parsing technology to extract the title, abstract, keywords, text, and references, and generate a standardized full-text dataset; (13) Extract citation content and citation relationship data based on reference marks, and generate a citation dataset including citation content, literature information, and citation network.

[0007] Further, step (2) includes the following steps: (21) Select the paper title, abstract, and keywords to construct a training corpus, and perform cleaning, stemming, and low-frequency word filtering through the NLTK tool; (22) Train a domain-specific word embedding model Word2vec_AM and verify its semantic fidelity; (23) Calculate the semantic distance between the target literature and historical literature: ; where represents that there are papers before the publication year of literature p; is the semantic vector representation of literature p.

[0008] Further, the word embedding model training includes generating domain word vectors using the Word2vec algorithm and verifying the semantic fidelity of the model through cosine similarity; only training with the target domain keyword content is adopted to exclude the interference of cross-domain texts.

[0009] Further, step (3) includes the following steps: (31) Extract the citation content and citation relationship network, including citation location and citation sentiment annotation; among them, the citation locations include: introduction, method, result, discussion; (32) Train the sentiment classification model, that is, adopt the BOW model as the potential feature items of the sentiment text, multiple machine learning models such as XGBoost, Logistic Regression, Multinomial NB, and RandomForest, and select the XGBoost model with the best F1 performance on the validation set as the citation content sentiment classification model to calculate the sentiment value: ; Among them, are the citation frequencies of the citing document j for the cited document i with different sentiments respectively, k represents the type of sentiment, is the weight assignment coefficient for the sentiment k from expert experience, N = 3, which is the maximum number of sentiment types; (33) Calculate the citation intensity value: ; Among them, are the citation frequencies of the citing document j for the cited document i at different chapter positions respectively, k represents the type of chapter, is the weight assignment coefficient for the citation intensity provided by the expert experience for the chapter k, N = 4, which is the maximum number of chapter types; (34) Calculate the citation relevance value: ; In the formula, is the word embedding vector of the reference document i, is the word embedding vector of the citation content r of the citing document j, ={ , , …, } is the set of all citation contents of the citing document j for the reference document i, and N is the citation frequency of the citing document j for the reference document i.

[0010] Further, the citation sentiment classification rules include: the sentiment types are divided into three categories: positive, negative, and objective, and the annotation rule is based on the citation purpose; the Kappa consistency test is adopted to ensure the reliability of the annotation results of multiple people, and the machine learning algorithm with the highest F1 value is preferably selected for the classification model.

[0011] Further, step (4) includes the following steps: (41) Analyze the representativeness of each index in the evaluation of scientific papers through the entropy weight method. The calculation formula for the entropy weight of the value index is as follows: ; ; In the formula, is the characteristic proportion of the i-th citation relationship under the j-th index; is the observed data of the j-th index in the i-th citation relationship, where i = 1, 2,..., n; j = 1, 2,…, m; n is the number of citation relationships, and m is the number of indices; is the entropy value of the j-th index; ; In the formula, is the entropy weight of the j-th evaluation index; (42) Through the linear weighted comprehensive evaluation method, calculate the comprehensive score of the academic value of scientific papers by applying the obtained weights; the calculation formula for the academic value is as follows: ; In the formula, is the normalized score of the evaluation index i; is the weight of the i-th evaluation index; (43) Calculate the academic value score after standardizing the literature in the target field;

[0012] An electronic device according to the present invention includes a memory, a processor, and a computer program stored on the memory. When the program is executed by the processor, any of the above methods is implemented.

[0013] A computer-readable storage medium according to the present invention stores a computer program. When the program is executed by the processor, any of the above methods is implemented.

[0014] Beneficial effects: Compared with the prior art, the present invention has the following remarkable advantages: By measuring the internal innovation value and external recognition value of scientific papers, the present invention can improve the accuracy of academic value evaluation and enhance the reliability of scientific research achievement recognition. This multi-dimensional fusion evaluation mechanism makes scientific research evaluation more systematic and intelligent, and can meet the needs of scientific research management decision-making and talent selection evaluation; by integrating natural language processing and manual coding means, the diversity of internal and external feature data improves the comprehensiveness and accuracy of academic value evaluation; by systematically constructing a semantic fusion and in-depth analysis model of citation content, it can more effectively identify the true influence of paper contributions, thereby reducing the uncertainty of single measurement indicators and expert experience judgments. Description of the Drawings

[0015] Figure 1 It is a schematic flow diagram of the present invention. Specific embodiments

[0016] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0017] As Figure 1 shown, an embodiment of the present invention provides a method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis, including the following steps: S1. Select a target research field, collect the full text of scientific papers and their citation data, and perform data cleaning and format standardization processing; including the following steps: S11. Select a typical scientific research field as the target field. The requirement for the target field is that there has been a large accumulation of scientific paper collections before. Based on the selected target field, the scientific paper data mainly comes from the Web of Science (WOS) database. Construct a retrieval formula as TI = ((Keywords1) OR (Keywords2) OR…OR (Keywords n )), DT = Article, PY = (Start Time - Now), and remove duplicates and clean the papers through the unique identifier WOS, and finally obtain the initial literature dataset of the target field; S12. Remove irrelevant literature, and use a text matching algorithm to screen out the literature irrelevant to the target field; S13. Obtain the full text PDF data of the literature; use the GROBID (GeneRation Of BIbliographic Data) deep learning model to parse the full text PDF data, extract the structured content of the paper, and generate a full text dataset including the paper title, abstract, keywords, text, and references, etc.; S14. Identify the reference markers through a rule matching method, and based on the matching of the marker symbols and the reference sections, extract the citation content and reference objects of the paper, and generate a citation dataset including the citation content, literature information, and reference information.

[0018] S2. Based on the scientific paper collection obtained in step S1, use a semantic modeling method to calculate the semantic distance between the target literature and the previous literature collection in the same field, quantify the internal innovation value of the target paper, and obtain the magnitude of the internal innovation value; specifically as follows: S21. Select the title, abstract, and keywords of the scientific paper, use the NLTK (Natural LanguageToolkit) tool for text cleaning and stemming processing, and set a word frequency threshold to remove low-frequency noise terms, and construct a training corpus for the Word2Vec word embedding model; Based on the training corpus of papers in the target field, a domain-specific word embedding model, the Word2vec_AM model, is trained on the basis of the Word2Vec basic word embedding model; the effectiveness of the model is evaluated using the annotated corpus to examine its semantic fidelity; the "title, keywords, abstract" content of the literature in the target field is vectorized using the Word2vec_AM model; S23. Obtain the vector results of the "title, keywords, abstract" content of the domain scientific papers from the earliest year to year t-1 and the "title, keywords, abstract" content of the target document p in year t; calculate the cosine similarity semantic distance between the vector of document p and the vectors of all documents in the domain in year t-1 to represent the innovation degree of document p; the calculation formula for its innovation is as follows: ; Among them, indicates that there are papers before the publication year of document p; is the semantic vector representation of document p; S3. This step provides a calculation scheme for quantifying the external recognition value index of the target document; it includes the following steps: S31. Obtain the citation content of the scientific papers in the target field, extract the in-text sentences where the reference markers are located from the full text of each document, with a citation window of 1, to form a citation content dataset including reference identification, citation frequency, citation location, and citation content text; extract the corresponding reference information from the reference section to construct a citation relationship network of the document identification, citation content, citation location, and reference identification of the scientific papers in the target field; S32. Participated by multiple people, according to the determined sentiment annotation rules as shown in Table 1, the extracted citation content is annotated with three sentiment polarities: positive, negative, and objective, and the annotation results meet the Kappa consistency test; the corpus is segmented into an 80% training set, a 10% validation set, and a 10% test set, and the BOW (Bag of Words) model is used as the potential feature item of the sentiment text to train multiple machine learning models, and the model with the best F1 performance on the validation set is selected as the citation content sentiment classification model; the sentiment classification model is used to classify the sentiment of all the extracted citation content; calculate the multi-dimensional citation sentiment value of the target document obtained from the citing documents according to the normalization processing of the vectors; the calculation formula for its citation sentiment value is as follows:

[0019] Among them, are the citation frequencies of the citing document j for different sentiments of the cited document i respectively, k represents the type of sentiment, is the weight assignment coefficient for sentiment k from expert experience, N = 3, which is the maximum number of sentiment types.

[0020] Table 1 Citation Emotion Annotation Rule Table ; S33. Identify and quantify the citation intensity. Extract the rule for the section information where the reference marker is located in each scientific paper in the target field, and classify it according to "Introduction, Method, Result, Discussion"; count the frequency according to the number of times the reference marker appears in each section; calculate the citation intensity value of the target document p by aggregating and calculating according to the normalization processing of the vector; the calculation formula of its citation intensity value is as follows: ; Among them, are the citation frequencies of the citing document j for different section positions of the cited document i, k represents the type of section, is the weight distribution coefficient of the citation intensity provided by expert experience for section k, N = 4, which is the maximum number of section types.

[0021] S34. Calculate and quantify the citation relevance value. Based on the citation relationship, calculate the relevance between the "title, abstract, keywords" of each scientific paper in the target field and the corresponding cited content in its citing documents; if there are multiple citations in a citing document, select the highest relevance value; this relevance value will be used as the citation relevance value from its citing document;

[0022] In the formula, is the word embedding vector of reference i, is the word embedding vector of the cited content r of the citing document j, ={ , ,…, } is the set of all cited contents of the citing document j for the reference i, and N is the citation frequency of the citing document j for the reference i.

[0023] S4. Construct an intelligent evaluation model for scientific papers: Objectively assign weights to the internal innovation value index, external recognition value index, and secondary external recognition indicators through the entropy weight method, and construct a measurement model that can be applied to practical calculations; specifically as follows: S41. Analyze the representativeness of each index in the evaluation of scientific papers through the entropy weight method. The calculation formula of the entropy weight of the value index is as follows: ; ; In the formula, is the characteristic proportion of the i-th citation relationship under the j-th index; is the observed data of the j-th index in the i-th citation relationship (i = 1, 2,..., n; j = 1, 2,…, m), where n is the number of citation relationships and m is the number of indices; is the entropy value of the j-th index; ; In the formula, is the entropy weight of the j-th evaluation index; S42. Through the linear weighted comprehensive evaluation method, calculate the comprehensive score of the academic value of scientific papers by applying the obtained weights; the calculation formula of the academic value is as follows: ; In the formula, is the normalized score of the evaluation index i; is the weight of the i-th evaluation index; S43. Calculate the academic value score of the literature in the target field after standardization; S5. By comparing with the traditional citation metrology method, evaluate the comprehensive performance advantages of this method in terms of index dimension coverage, academic value recognition accuracy, evaluation robustness, etc.; including the following steps: S51. Calculate the statistical characteristics of the academic value index, external recognition value index, internal innovation value index and traditional measurement indexes (number of citations, number of mentions) of the literature in the target field; S52. Use the Spearman correlation coefficient test to calculate the correlation characteristics between the academic value results of the literature in the target field and the traditional measurement structure, so as to evaluate the comprehensive performance of this method in terms of index dimension coverage, academic value recognition accuracy, evaluation robustness, etc.

Claims

1. A method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis, characterized in that It includes the following steps: (1) Determine the literature database according to the target research field, construct a list of domain keywords, and retrieve and collect the initial literature dataset; (2) Quantify the internal innovation value; (3) Quantify the external recognition value; (4) Construct an integrated measurement model of academic value, including: applying the entropy weight method to allocate index weights and calculating the comprehensive score of academic value by linear weighting; (5) Evaluate through the Spearman correlation coefficient test.

2. The method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis according to claim 1, characterized in that, Step (1) includes the following steps: (11) Filter out irrelevant literature through a text matching algorithm; (12) Obtain the full text of the paper, use text structured parsing technology to extract the title, abstract, keywords, text, and references, and generate a standardized full text dataset; (13) Extract citation content and citation relationship data based on reference markers, and generate a citation dataset including citation content, literature information, and citation network.

3. A method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis according to claim 1, characterized in that, Step (2) includes the following steps: (21) Select the paper title, abstract, and keywords to construct a training corpus, and perform cleaning, stemming, and low-frequency word filtering through the NLTK tool; (22) Train the domain-specific word embedding model Word2vec_AM and verify its semantic fidelity; (23) Calculate the semantic distance between the target literature and historical literature. The formula is as follows: ； Among them, indicates that there are papers before the publication year of document p; is the semantic vector representation of document p.

4. A method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis according to claim 3, characterized in that The word embedding model training includes generating domain word vectors using the Word2vec algorithm and verifying the semantic fidelity of the model through cosine similarity; only the content of keywords in the target domain is used for training to exclude cross-domain text interference.

5. A method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis according to claim 1, characterized in that Step (3) includes the following steps: (31) Extract citation content and citation relationship network, including citation location and citation sentiment annotation; among them, the citation location includes: introduction, method, result, discussion; (32) Train the sentiment classification model, that is, use the BOW model as the potential feature item of the sentiment text, train multiple machine learning models such as XGBoost, Logistic Regression, Multinomial NB, and RandomForest, and select the XGBoost model with the best F1 performance on the validation set as the citation content sentiment classification model to calculate the sentiment value. The formula of the sentiment classification model is as follows: ； Among them, are the citation frequencies of the cited document i with different emotions by the citing document j, k represents the type of emotion, is the weight distribution coefficient of emotion k from expert experience, N = 3, which is the maximum number of emotion types; (33) Calculate the citation intensity value. The formula is as follows: ； Among them, are the citation frequencies of cited document i at different chapter positions by citing document j, k represents the type of chapter, is the weight distribution coefficient of the citation intensity provided by expert experience for chapter k, N = 4, which is the maximum number of chapter types; (34) Calculate the citation relevance value. The formula is as follows: ； In the formula, is the word embedding vector of reference i, is the word embedding vector of the cited content r of citing document j, ={ , ,…, } is the set of all cited contents of citing document j for reference i, and N is the citation frequency of citing document j for reference i.

6. The method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis according to claim 5, characterized in that The citation sentiment classification rules include: the sentiment type is divided into three categories: positive, negative, and objective, and the annotation rule is based on the citation purpose; the Kappa consistency test is used to ensure the reliability of the annotation results of multiple people, and the machine learning algorithm with the highest F1 value is preferred for the classification model.

7. A method for measuring the academic value of scientific papers based on semantic fusion and citation behavior analysis according to claim 1, characterized in that Step (4) includes the following steps: (41) Analyze the representativeness of each index in the evaluation of scientific papers through the entropy weight method. The calculation formula of the entropy weight of the value index is as follows: ； ； In the formula, is the characteristic proportion of the i-th citation relationship under the j-th index; is the observed data of the j-th index in the i-th citation relationship, where i = 1, 2,..., n; j = 1, 2,…, m; n is the number of citation relationships, and m is the number of indices; is the entropy value of the j-th index; ； In the formula, is the entropy weight of the j-th evaluation index; (42) Through the linear weighted comprehensive evaluation method, calculate the comprehensive score of the academic value of scientific papers by applying the obtained weights; the calculation formula of the academic value is as follows: ； Wherein, is the normalized score of evaluation index i; is the weight of the i-th evaluation index; (43) Calculate the standardized academic value score of the literature in the target field.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1-7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the described program is executed by a processor, it implements the method according to any one of claims 1-7.

Images