Method for constructing consistency score of manuscript review report

Abstract

The invention provides a method for constructing a consistency score of a manuscript review report, and relates to the technical field of manuscript review. The method for constructing the consistency score of the manuscript review report specifically comprises the following steps: S1, constructing a consistency score index; s2, data preprocessing and feature extraction; and S3, model training and performance evaluation. According to the method, the multi-level scoring index system is constructed, text classification and feature extraction are combined, comprehensive evaluation of the quality of the review report is achieved, the evaluation precision and objectivity are improved, secondly, the model is trained through LSTM, the classification accuracy is as high as 95%, the review classification accuracy is remarkably improved, in addition, the method has multidisciplinary applicability, and the method is suitable for popularization and application. The method can be applied in a cross-domain mode, the problem that a traditional model is limited to a single domain is solved, finally, through comprehensive analysis of linguistic and emotional features, the detailed degree of text analysis is remarkably improved, and powerful support is provided for manuscript review report quality evaluation.

Description

Technical Field

[0001] The present invention relates to the field of manuscript review technology, and in particular to a method for constructing a consistency score for a manuscript review report. Background Art

[0002] A review report is a detailed evaluation and suggestion provided by the reviewer on the advantages and disadvantages, academic value and whether the paper should be accepted for publication during the peer review process of an academic paper. It is an important part of ensuring the quality of scientific research and academic integrity in the scientific publishing process. The existing review report process usually includes the following steps: After the author submits the paper, the editor will invite experts in multiple fields to serve as reviewers to conduct an independent evaluation of the paper. The reviewer writes a detailed review report based on the paper's innovation, research methods, experimental results and conclusions, and makes suggestions for revisions or decides whether to recommend it for publication. The editor makes the final decision based on these comments.

[0003] The current application of technology in review reports mainly focuses on two aspects: one is the extraction of text features, such as single sentence length, total word count and sentiment score; the other is text classification, which is used to judge the quality of the report or whether it meets specific standards. However, these technologies have obvious shortcomings: automated tools may not be able to fully capture the potential bias and subjective judgment of human reviewers. In text analysis, indicators such as sentiment scores may be affected by text context and cultural background; although natural language processing technology has made great progress, machines still have limited understanding of domain-specific context and complex semantics, making it difficult to accurately capture the potential meaning in review reports; with the development of generative AI, automatically generated review reports may lack personalized insights, reduce the quality of the report, and make it difficult to reflect the human insight and experience when writing manually; existing automated analysis tools are insufficient in identifying real academic innovations in papers and review reports, and can only evaluate superficial language features, but it is difficult to judge the actual contribution of research work. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method for constructing a consistency score for review reports, which solves the problems of bias and limitations in the prior art.

[0005] To achieve the above objectives, the present invention is implemented by the following technical solution: a method for constructing a consistency score for a review report, specifically comprising the following steps: S1. Constructing consistency scoring indicators We randomly collected journal review report text datasets from different disciplines, and divided the texts into seven categories based on sentence semantics: Method, Result, Contribution, Conclusion, Introduction, Presentation (ie, Format) and References (Refs); S2. Data preprocessing and feature extraction Use the Python library BeautifulSoup to remove HTML tags from the original data of the review report to generate readable text and extract multiple linguistic and sentiment features; S3. Model training and performance evaluation The processed dataset was trained using the LSTM model, with 90% of the sentences as the training set and 10% as the test set, to achieve model classification and improve the accuracy of consistency scoring.

[0006] Preferably, the step S1 of constructing a consistency scoring index includes marking each sentence and constructing a multi-level scoring structure.

[0007] Preferably, the data preprocessing and feature extraction in step S2 extracts a variety of linguistic features including review scores, review length, average sentence length, number of unique words, number of sentences and Dale-Chall readability score.

[0008] Preferably, the sentiment feature in the data preprocessing and feature extraction in step S2 is to analyze the subjectivity and polarity of the text using the TextBlob library.

[0009] Preferably, in the model training and performance evaluation in step S3, the similarity between the review comments and the article abstract is evaluated by using the SPECTRE model to evaluate the consistency between the review content and the research topic.

[0010] Preferably, in the step S3 model training and performance evaluation, an XGBoost model may be used to predict the extracted text features to achieve the prediction and evaluation of the consistency score (ConformityScore).

[0011] The present invention provides a method for constructing a consistency score for a review report. It has the following beneficial effects: The present invention provides a method for constructing a consistency score for review reports. The present invention collects review reports from multiple disciplines, and through semantic classification and feature extraction, ensures that the model can be applicable to review reports of different disciplines, solves the limitations of traditional models applicable to a single field, and has wide versatility. Through the annotation and classification of 7762 sentences, combined with the training of the LSTM model, the test results show that the accuracy of the model of the present invention in the classification of review reports is as high as 95%. This significant improvement in accuracy solves the technical bias in the prior art for insufficient accuracy in the classification of review reports, and ensures the reliability of the model in practical applications.

[0012] The present invention provides a method for constructing a consistency score for a review report. The present invention not only classifies by keywords, but also extracts a variety of linguistic features (such as review scores, sentence length, number of unique words, etc.) and sentiment features (such as text subjectivity, sentiment polarity), thereby greatly improving the model's ability to conduct detailed analysis of texts. In particular, excellent results have been achieved in using the SPECTER model to evaluate the similarity between review reports and article abstracts, effectively improving the accuracy and objectivity of review report consistency evaluation. The multi-level scoring structure and classification mechanism of this technology solve the complexity and inaccuracy problems of review report analysis in the prior art. Combined with experimental data and model performance, the technical solution of the present invention not only effectively improves the ability to evaluate review quality, but also has significant interdisciplinary adaptability, achieving unexpected technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the method steps of the present invention; Figure 2 It is a schematic diagram for explaining various text feature variables of the present invention; Figure 3 This is a schematic diagram showing an example of the semantic classification results of the review report of the present invention; Figure 4 It is a schematic diagram of the definition of Conformity Score of the present invention; Figure 5 This is a schematic diagram of the present invention for predicting consistency scores using various text feature variables. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] like Figure 1-5As shown, the embodiment of the present invention provides a method for constructing a consistency score of a review report, which specifically includes the following steps: S1. Constructing consistency scoring indicators We randomly collected journal review report text datasets from different disciplines, and divided the texts into seven categories based on sentence semantics: Method, Result, Contribution, Conclusion, Introduction, Presentation (ie, Format) and References (Refs). We annotated each sentence and constructed a multi-level scoring structure. S2. Data preprocessing and feature extraction Use the Python library BeautifulSoup to remove HTML tags from the raw data of the review report to generate readable text, and extract multiple linguistic features (such as review score, review length, average sentence length, number of unique words, number of sentences, and Dale-Chall readability score) and sentiment features (such as using the TextBlob library to analyze the subjectivity and positivity of the text); S3. Model training and performance evaluation The processed data set is trained using the LSTM model, with 90% of the sentences used as the training set and 10% as the test set to achieve model classification and improve the accuracy of consistency scoring. The SPECTRE model is used to evaluate the similarity between the review comments and the article abstract, and to assess the consistency between the review content and the research topic. The XGBoost model can be used to predict the extracted text features to achieve the prediction and evaluation of the consistency score (ConformityScore).

[0016] Specifically: The present invention randomly collected journal review report texts from different disciplines and extracted and constructed the original data set, which contained 7762 sentences in total. According to the semantics of each sentence, it was divided into 7 categories: Method, Result, Contribution, Conclusion, Introduction, Presentation (ie, Format) and References (Refs). Figure 3 shown.

[0017] The LSTM model was used to train the processed data set, using 90% of the sentences as the training set and 10% as the test set. The final model achieved a classification accuracy of 95%. According to these classification definitions, the sentences in each category were annotated and a multi-level scoring structure was constructed, in which the "method" category sentences accounted for the largest proportion, followed by the "contribution" and "display" category sentences, and the "conclusion" category sentences accounted for the least. Figure 4 As shown in Figure 2. Each category of sentences usually contains specific keywords, and is automatically classified based on these keywords, and the consistency score (CS) is calculated.

[0018] Data preprocessing and feature extraction After establishing the consistency score, it is necessary to verify the rationality of this newly defined indicator and extract and analyze the text features of the review report from multiple dimensions. Therefore, using the original data of the review report, the Python library BeautifulSoup is used to remove HTML tags and generate pure readable text. Based on the processed text, a variety of linguistic and sentiment features are extracted, including: i) Linguistic features: review score, review length (raw and log-transformed), average sentence length, number of unique words, number of sentences, and Dale-Chall readability score.

[0019] ii) Sentiment features: Use the TextBlob library to analyze the subjectivity and positivity of the text and obtain the sentiment tendency of the review report.

[0020] iii) Interaction features: The SPECTRE model is used to measure the similarity between the review comments and the article abstract to assess the consistency between the review content and the research topic. Figure 2 shown.

[0021] The correlation between Conformity Score and text features The consistency score proposed based on text classification is predicted by the extracted text features. Figure 5 The actual observed audit report consistency scores and the consistency scores predicted by the XGBoost model (with various text features as input) are shown.

[0022] The technology of the present invention improves the limitations of traditional models through semantic classification and feature extraction of multidisciplinary data, and is applicable to review reports in multiple fields. Through the combination of a multi-level scoring structure and a machine learning model, the test results show the high accuracy of the model in classifying review reports. In addition to keyword classification, the invention also enhances the analysis depth and accuracy of the model through comprehensive consideration of linguistic and sentiment features. The present invention shows unexpected results in the analysis of interdisciplinary review reports, improving the efficiency and accuracy of evaluation.

[0023] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a consistency score for a review report, characterized in that: The specific steps include: S1. Constructing consistency scoring indicators We randomly collected journal review report text datasets from different disciplines, and divided the texts into seven categories based on sentence semantics: Method, Result, Contribution, Conclusion, Introduction, Presentation (ie, Format) and References (Refs); S2. Data preprocessing and feature extraction Use the Python library BeautifulSoup to remove HTML tags from the raw data of the review report to generate readable text, and extract multiple linguistic features (such as review score, review length, average sentence length, number of unique words, number of sentences, and Dale-Chall readability score) and sentiment features (such as using the TextBlob library to analyze the subjectivity and positivity of the text); S3. Model training and performance evaluation The processed dataset was trained using the LSTM model, with 90% of the sentences as the training set and 10% as the test set, to achieve model classification and improve the accuracy of consistency scoring.

2. A method for constructing a consistency score for a review report according to claim 1, characterized in that: The step S1 of constructing the consistency scoring index includes marking each sentence and constructing a multi-level scoring structure.

3. A method for constructing a consistency score for a review report according to claim 1, characterized in that: The step S2 of data preprocessing and feature extraction extracts a variety of linguistic features including review score, review length, average sentence length, number of unique words, number of sentences and Dale-Chall readability score.

4. A method for constructing a consistency score for a review report according to claim 1, characterized in that: The sentiment features in the data preprocessing and feature extraction in step S2 are to analyze the subjectivity and positivity of the text using the TextBlob library.

5. A method for constructing a consistency score for a review report according to claim 1, characterized in that: In the S3 step, the SPECTRE model is used to evaluate the similarity between the review comments and the article abstracts in the model training and performance evaluation, and the consistency between the review content and the research topic is evaluated.

6. A method for constructing a consistency score for a review report according to claim 1, characterized in that: In the step S3 model training and performance evaluation, the XGBoost model may be used to predict the extracted text features to achieve the prediction and evaluation of the consistency score (ConformityScore).

Images