Method for paper review by using LLM

Abstract

The invention discloses a method for automatically reviewing papers by using an LLM (Logistics Language Model). The method comprises the following steps of: inputting a review question and a to-be-reviewed papers into the LLM; generating a simulated answer from the LLM; acquiring a plurality of pieces of paper node data from a plurality of knowledge graph nodes of a paper knowledge graph of a paper knowledge graph library; identifying a plurality of paper contents related to the plurality of simulation answers from the to-be-reviewed paper; multiple combinations are input into the LLM, and each combination in the multiple combinations is composed of one of the multiple review questions, one of the multiple pieces of paper node data and one of the multiple pieces of paper content from the to-be-reviewed paper; and generating review opinions for the to-be-reviewed paper through each combination by using the LLM.

Description

Technical Field

[0001] This disclosure relates to Large Language Models (LLMs), and more specifically to methods for paper review using LLMs. Background Technology

[0002] In academia, peer review is a crucial procedure in academic publishing. To enhance the quality and reputation of academic journals, editors must carefully screen and initially evaluate submitted papers. Papers that meet the initial evaluation criteria then undergo further peer review. However, due to the sheer volume of submissions, editors and peer reviewers often expend significant time and effort. Currently, LLMs (Limited Learning Professionals) trained in specific academic fields can effectively read and understand journal articles. Therefore, automated review solutions can leverage LLMs to automatically read, analyze, and understand submitted papers and provide reviewer reports. Simultaneously, interactive human-computer Q&A methods can improve the quality and speed of paper review. This can effectively reduce the overall workload and time investment of journal editors and peer reviewers in the review process, thereby further simplifying the peer review process and ultimately improving the quality of journal publication.

[0003] Furthermore, the publication of academic papers is crucial to a researcher's academic career. Researchers, as authors, are frequently expected to publish academic papers. However, authors with limited academic publishing experience must consider various factors to successfully complete the publication process and overcome numerous obstacles, including writing barriers. Therefore, authors can improve their papers before submission by using automated peer review systems to check them, refine them based on the system's feedback, add missing details, and select appropriate journals for submission. When integrated with external knowledge bases in specific fields, LLMs can encompass a broad range of domain-specific knowledge across disciplines; therefore, automated review solutions based on such models can provide authors with relevant feedback, improve the quality of their papers, and increase the likelihood of acceptance.

[0004] Currently, academic papers still rely on manual review, which suffers from problems such as inefficiency, inability to analyze the semantic content of papers to be reviewed, and unsatisfactory review results. Therefore, there is a need to develop an automated review solution to at least partially address these issues. Significant technical challenges must be addressed in understanding and reviewing multimodal data (text, tables, images) in order to design and train pre-trained LLMs in a targeted manner. Summary of the Invention

[0005] To address the limitations of inefficiency and unsatisfactory review results, this disclosure provides a method for facilitating automated paper review by applying an LLM integrated with a knowledge graph, a method for fine-tuning a pre-trained LLM for paper review tasks, and a method for generating paper-specific knowledge graphs.

[0006] This disclosure provides a method for automating paper review using an LLM, the method comprising:

[0007] Input multiple review questions and papers to be reviewed into the LLM.

[0008] Based on the multiple review questions and the papers to be reviewed, LLM generates multiple simulated responses.

[0009] Multiple paper node data are obtained from multiple knowledge graph nodes in a paper knowledge graph database. These multiple paper node data are related to multiple simulated answers. The paper knowledge graph is a graph of knowledge covering everything from the hierarchical structure of a paper to the publicly disclosed conceptual relationships within the paper.

[0010] Multiple paper contents related to the simulated responses were identified from the papers awaiting review.

[0011] Multiple combinations are input into the LLM, where each combination consists of one of the multiple review questions, one of the multiple paper node data, and one of the multiple paper content from the papers to be reviewed.

[0012] LLM is used to generate review comments for papers to be reviewed by each combination.

[0013] In some embodiments, the plurality of review questions are extracted from a pre-defined review question library or from user input.

[0014] In some embodiments, the method further includes generating a review decision based on the review comments.

[0015] In some embodiments, obtaining multiple paper node data includes comparing the cosine similarity between the embedding of each of the multiple simulated answers and the embedding of each of the multiple knowledge graph nodes of the paper knowledge graph to obtain each of the multiple knowledge graph nodes, wherein the embedding that presents the highest cosine similarity is taken as the paper node data, and the paper node data is combined for all the multiple simulated answers as the multiple paper node data.

[0016] In some embodiments, identifying multiple pieces of paper content from a paper to be reviewed includes: comparing the cosine similarity between the embedding of each of the multiple simulated responses and the embedding of each of the multiple pieces of paper content from the paper to be reviewed, identifying each of the multiple pieces of paper content from the paper to be reviewed whose embedding exhibits the highest cosine similarity as the paper content from the paper to be reviewed, and combining the paper content from all the multiple simulated responses as the multiple pieces of paper content from the paper to be reviewed.

[0017] In some embodiments, the LLM is fine-tuned using a training sample set that includes paper content of various data types, as well as a set of review questions and comments.

[0018] In some embodiments, the training sample set includes multiple real or artificially constructed paper samples, which include text paragraphs, tabular data, images and other content, as well as corresponding review questions and comments prepared by editors and peer reviewers.

[0019] In some embodiments, the method also includes deleting all original data associated with the paper to be reviewed after the review comments are generated.

[0020] This disclosure serves journal editors and peer reviewers by reducing the workload of journal article publication and improving the processing efficiency. Furthermore, it serves authors by helping them improve paper quality and publication success rates based on reviewer feedback. The automated review process using LLM (Limited Learning Model) in this disclosure constructs a knowledge graph of the paper to enhance the LLM's understanding and analysis of the paper's semantic content. During the review process of submitted papers, the automated review method according to this disclosure also protects authors' undisclosed intellectual property rights, ensuring that any innovative content and data from authors are not disclosed. Attached Figure Description

[0021] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. The drawings and corresponding detailed descriptions are provided only to offer a better understanding of the present disclosure and in no way constitute any limitation on the scope of the disclosure as defined in the claims. Specifically:

[0022] Figure 1A A flowchart of a paper knowledge graph generation method according to certain embodiments of the present disclosure is shown.

[0023] Figure 1B A preset review question library for general academic papers is shown according to certain embodiments of this disclosure.

[0024] Figure 1C A preset review question library for literature review papers according to certain embodiments of this disclosure is shown.

[0025] Figure 1D A flowchart of a method for generating review comments according to certain embodiments of the present disclosure is shown.

[0026] Figure 1E An exemplary paper review decision table according to certain embodiments of this disclosure is shown.

[0027] Figure 1F A flowchart illustrating the review decision generated according to certain embodiments of this disclosure is shown.

[0028] Figure 1G A schematic diagram of a paper review system according to certain embodiments of the present disclosure is shown.

[0029] Figure 1H A flowchart is shown illustrating a method for paper review using LLM according to certain embodiments of this disclosure.

[0030] Figure 2A A flowchart illustrating the fine-tuning of an LLM according to certain embodiments of this disclosure is shown.

[0031] Figure 2B A schematic diagram of an exemplary LLM according to certain embodiments of the present disclosure is shown.

[0032] Figure 3A A schematic diagram illustrating the hierarchy and content of an exemplary paper according to certain embodiments of this disclosure is shown.

[0033] Figure 3B A schematic diagram of an exemplary paper knowledge graph according to certain embodiments of the present disclosure is shown.

[0034] Figure 3C A schematic diagram of a paper review generation method utilizing LLM according to certain embodiments of the present disclosure is shown.

[0035] Figure 4A A schematic diagram of an exemplary system applicable according to certain embodiments of the present disclosure is shown.

[0036] Figure 4B A flowchart of an automated paper review system according to certain embodiments of the present disclosure is shown.

[0037] Figure 5 A schematic diagram of an electronic device according to certain embodiments of the present disclosure is shown. Detailed Implementation

[0038] The present disclosure will now be described in more detail with reference to the accompanying drawings. However, the present disclosure may be implemented in many different forms and should not be construed as being limited to the embodiments described herein.

[0039] To help those skilled in the art better understand the technology involved in this application, a description of embodiments of this disclosure is provided below. These descriptions constitute part, but not all, of the functionality of this disclosure. All other embodiments obtained by others based on the embodiments disclosed in this application should fall within the protection scope of this application.

[0040] This disclosure provides an embodiment of a method for automated paper review using an LLM (Library Management System), specifically, for example, for automated review of journal articles. The method includes: inputting multiple review questions and the paper to be reviewed into an LLM; having the LLM generate multiple simulated answers based on the multiple review questions and the paper; obtaining multiple paper node data from multiple knowledge graph nodes of a paper knowledge graph in a paper knowledge graph library, wherein the multiple paper node data are related to the multiple simulated answers, wherein the paper knowledge graph is a graph covering knowledge from the hierarchical structure of the paper to the publicly disclosed conceptual relationships within the paper; identifying multiple pieces of content related to the multiple simulated answers from the paper to be reviewed; inputting a combination of the multiple review questions, the multiple paper node data, and the multiple pieces of content from the paper to be reviewed into the LLM, and using the LLM to generate review comments for the paper to be reviewed.

[0041] The knowledge graph of a paper can be obtained from a knowledge graph database of papers, and includes precise and structured knowledge, domain-specific knowledge, etc. Figure 1A A schematic diagram is shown of a method for generating a paper knowledge graph library using a sample LLM. The method includes the following steps.

[0042] In Block 101, the intelligent document parsing module is configured to generate a paper knowledge graph and related node data. This paper knowledge graph encompasses knowledge ranging from the paper's hierarchical structure to conceptual relationships. A large amount of knowledge related to the research field is also stored in a pre-defined paper knowledge graph repository. To generate the paper knowledge graph and extract related node data, the process involves preprocessing published open-access journal articles. First, the internal information of the published open-access journal articles is stored, such as the paper document, review comments, chapter text, titles, abstracts, and other relevant sections. Then, the paper document is parsed, and text mining techniques are used to extract the required chapters. The extracted information is then organized into a structured format suitable for constructing the knowledge graph. The specific extraction method can depend on the specific requirements and the available tools or libraries for text processing and analysis.

[0043] At block 102, node data (text, tables, and images) are encoded into embedding vectors.

[0044] If the paper is used as a training sample, the generated paper knowledge graph and the embedding vectors of the node data are added to the paper knowledge graph library at block 103; if the paper is reviewed, the original paper document is deleted at block 104, and the generated paper knowledge graph data is not added to the paper knowledge graph library. Relevant knowledge can be quickly located by matching the embedding vectors of the nodes.

[0045] Review questions can be obtained from a pre-set review question library or can be entered by the user. The pre-set review question library can cover common dimensions of paper review and evaluation, including originality, theoretical importance, experimental design, data analysis, paper writing, and research significance. It also allows users to enter their own review questions for specific papers or contexts.

[0046] Figure 1B and Figure 1C As two illustrative examples, a list of two preset review questions according to this embodiment is shown. Figure 1B As shown, the questions in the general academic paper pre-set review question bank may include:

[0047] How does this study differ significantly from other studies in the literature?

[0048] Is the holistic research method novel?

[0049] Does this research add significant insights to the theoretical controversies in the literature?

[0050] Are the theoretical assumptions of this study clearly stated?

[0051] Is this research incremental, or is it a major breakthrough for the future?

[0052] Are the methods used sound and rigorous?

[0053] Are the participants suitable for the research objectives?

[0054] Is the sample size sufficient for the research question?

[0055] Does the research design allow for adequate testing of the hypotheses?

[0056] Are there any defects in the materials or procedures used?

[0057] Is the results analysis appropriate for the research question?

[0058] "Was the statistical analysis conducted correctly according to on-site standards?"

[0059] "Is the expected effect statistically significant?"

[0060] Does an insignificant effect conflict with the research hypothesis?

[0061] "Based on existing literature, is the interpretation of the results correct?"

[0062] How do these findings align with or contradict previous research? If they contradict each other, what are the reasons?

[0063] Are these findings generalizable, or are they limited to current research?

[0064] Are these conclusions based on the results, or are they the author's own speculations?

[0065] Did the authors discuss the main limitations of the study?

[0066] Is the overall writing style of the article clear?

[0067] Questions from a pre-designed review question bank for academic review papers, such as Figure 1C As shown, it may include:

[0068] Did the author explain their reasons for writing a review paper in this field?

[0069] Does this review paper provide a good overview of the development of this field and offer insights into its future development?

[0070] "Does the author adequately represent the most relevant and up-to-date developments in these fields?"

[0071] "Is the submitted review detailed enough to allow for reproducibility and / or reproducibility (e.g., a summary methodology described for each specific study, publicly available search strategies, inclusion criteria, and risk of bias assessment)?"

[0072] Are the statistical summarization methods (e.g., meta-analysis, meta-regression) and their reporting (e.g., p-value, 95% CLS, etc.) appropriate and well-described?

[0073] "Does the structure, flow, or writing of your review paper need improvement (e.g., adding numbered subheadings, shortening the text, reorganizing chapters, or moving details from one chapter to another. Follow the PRISMA guidelines)?"

[0074] Can the manuscript be improved through language editing?

[0075] Figure 1HThis document illustrates a currently disclosed embodiment of a method for paper review using an LLM (Library of Letters). The method includes inputting review questions and a paper to be reviewed into an LLM, generating simulated responses from the LLM, and retrieving multiple paper node data from multiple knowledge graph nodes of a paper knowledge graph in a paper knowledge graph library, wherein the multiple paper node data are associated with multiple simulated responses. The paper knowledge graph is a graph encompassing knowledge from the hierarchical structure of a paper to the conceptual relationships disclosed in the paper, and wherein a paper knowledge graph is selected from the paper knowledge graph library based on the multiple simulated responses. The method then further includes identifying multiple paper content items from the paper to be reviewed that are associated with the multiple simulated responses. The method includes inputting multiple combinations into the LLM, wherein each of the multiple combinations consists of one of the multiple review questions, one of the multiple paper node data items, and one of the multiple paper content items from the paper to be reviewed, and generating review comments for the paper to be reviewed using an LLM, such as GPT-4o or Claude 3.5Sonnet, for each combination.

[0076] like Figure 1H As shown, step 1 involves inputting multiple review questions and the paper to be reviewed into the LLM. In one embodiment, text content from the paper to be reviewed can be extracted, for example, through optical character recognition (OCR) used for scanning documents or open-source methods for processing paper document files (e.g., sciencebeam-parser, python-docx). Reviewer comments (if any), submission, title, authors, organization, and text information such as abstract and methodology can be extracted. Multiple review questions are extracted from a list of review questions from a pre-defined review question library, for example, such as... Figure 1B and Figure 1C The example shown uses a pre-set review question library for general academic papers or academic review papers, or it can be extracted from user input. In one embodiment, the question vector that shows the highest cosine similarity to sentences in simulated responses is considered the most relevant. If multiple question vectors are relevant to the review question, the question vector with the highest cosine similarity can be selected, or any other suitable similarity metric can be used. The specific selection criteria for the question vectors will depend on the context of the application and the desired results.

[0077] An example of the above description is as follows. First, the review question "What are the limitations of the proposed method?" is extracted from a review question database or user input. Then, the above review question and the paper to be reviewed are input into the LLM to obtain a simulated answer, such as "The limitations of the proposed method include scalability issues and the need for further validation." By calculating the cosine similarity between the sentence vectors in the simulated answer and relevant blocks of the document text (e.g., from the first 100 sentences), the "Limitations and Future Work" section appears most frequently. "Limitations and Future Work" is identified as the most relevant block discussing the limitations of the proposed method.

[0078] In such Figure 1H In step 2, as shown, the multiple simulated answers can be generated from the LLM. The purpose of the simulated answers is to provide context for comparison. Then, the paper knowledge graph of the paper to be reviewed is obtained from a paper knowledge graph library. For example, step 2 includes converting each sentence in the simulated answers into an embedding, such as using a Sentence Transformer, and then comparing the cosine similarity between the paper content in the paper knowledge graph and the embeddings of each sentence in the paper to be reviewed with the embeddings in the multiple simulated answers. Then, multiple paper node data, where these multiple paper node data are relevant to the review question, are identified from multiple knowledge graph nodes in the paper knowledge graph library based on the multiple simulated answers. Then, multiple pieces of content from the paper to be reviewed, where these multiple pieces of content are relevant to the review question, are identified based on the multiple simulated answers. The Sentence Transformer can process each sentence in the paper content to obtain a vector representation.

[0079] Preferably, the multiple paper node data are most relevant to the multiple simulated answers, and multiple pieces of content from the paper to be reviewed are most relevant to the multiple simulated answers. In one example, the step includes: embedding the simulated answers to generate their quantifiable vectors, comparing the quantifiable vectors of the simulated answers with the quantifiable vectors of the paper blocks in the paper knowledge graph, obtaining the quantifiable vector of the paper block most relevant to the quantifiable vector of the simulated answers, and using the paper block with the most relevant text as the most relevant text content.

[0080] In such Figure 1H In step 3, after obtaining the relevant text in the previous step, a combination of the text of the paper to be reviewed, review questions, and the most relevant text content can be generated, or a pair consisting of the text of the paper to be reviewed, review questions, and the most relevant text content. This combination or pair is then input into the LLM (Library Management System), and the LLM generates reviewer comments based on this combination or pair. The same LLM can be used for both steps 1 and 3.

[0081] Figure 1DA flowchart of a method for generating review comments according to another embodiment of this disclosure is shown. To protect the author's intellectual property rights, all original data related to the submitted document has been removed. Figure 1D As shown, the review comment generation method includes the following steps. First, in step 141, the user's review question is input. Then, in step 142, the user's review question and the paper to be reviewed are input into the LLM (Limited Language Graph). Then, in step 143, the LLM generates simulated answers to the review question and embeddings for each sentence. Then, in step 144, the similarity between the embeddings of each sentence is calculated, and the most similar content is obtained from the paper to be reviewed and the paper content from the knowledge graph. In step 145, a (paper text to be reviewed, review question, most relevant text content) pair is created and input into the LLM. Finally, in step 146, all original data related to the paper to be reviewed is deleted.

[0082] Furthermore, for multiple review questions, the LLM can generate a set of review comments. Then, based on this set of review comments generated by the LLM, it can generate a review decision and an optional reviewer report. The reviewer report includes the review decision and the set of review comments.

[0083] Figure 1E An exemplary paper review decision table according to certain embodiments of the present disclosure is shown, wherein the review decision includes the following options: accept, major revision, minor revision, reject, and reject without review.

[0084] Figure 1F A flowchart illustrating a review decision generation method based on certain embodiments of this disclosure is shown. Figure 1F As shown, the method for generating opinions includes the following steps.

[0085] First, in step 161, review questions are input from a pre-defined review question database. Then, in step 162, the review questions and the paper to be reviewed are input into the LLM (Limited Language Module). Next, in step 163, the LLM generates simulated answers to the review questions and embeddings for each sentence. Then, in step 164, the similarity between the embeddings of each sentence is calculated, and the most similar content is obtained from the paper to be reviewed and the paper content from the knowledge graph. In step 165, a (paper text, review question, most relevant text content) pair is created and input into the LLM to generate review comments and, based on these comments, a review decision. Finally, in step 166, all original data related to the paper to be reviewed is deleted.

[0086] Figure 1G A schematic diagram of a paper review system according to certain embodiments of the present disclosure is shown. This paper review system can be used for journal articles or other papers requiring review. Figure 1G As shown, the paper review system comprises five main modules.

[0087] The paper review system includes an intelligent document parsing module, which is configured to extract structured and hierarchical information and content from documents and output the extracted information and content.

[0088] The paper review system also includes a paper knowledge graph module. This module comprises a paper knowledge graph library module for storing pre-generated paper knowledge graphs, and a paper knowledge graph generation module. The paper knowledge graph generation module is configured to receive structured hierarchical information and content from the intelligent document parsing module, generate a paper knowledge graph corresponding to the paper, and send the pre-generated paper knowledge graph to the paper knowledge graph library module for storage.

[0089] The paper review system also includes a training sample module, which comprises a training sample library module for storing preprocessed training samples of paper data of different data types, and an LLM training module. The training sample library module provides preprocessed training samples of paper data of different data types to the LLM training module for training the language model. The training sample library module also provides preprocessed training samples of paper data of different data types to the paper knowledge graph generation module.

[0090] The paper review system also includes a review engine module. The review engine module comprises a knowledge retrieval module, a knowledge ranking module, and an LLM (Local Level Management). The knowledge retrieval module is configured to receive a pre-generated paper knowledge graph from the paper knowledge graph module. The review engine module is configured to convert review questions into corresponding vectors, retrieve multiple paper node data from the paper knowledge graph, extract multiple pieces of content related to the multiple simulated answers from the papers to be reviewed, and use the knowledge ranking module. The LLM is configured to generate review comments for the questions based on the multiple paper node data and the multiple pieces of content from the papers to be reviewed. Optionally, the LLM is also configured to generate a review decision for the submitted papers based on the review comments.

[0091] The paper review system also includes a user interface module, which comprises a reviewer report module and a user interaction module. The reviewer report module is configured to generate a reviewer report based on review comments and optional review decisions. The user interface module is configured to facilitate the output of paper review reports and enable human-computer interaction. The review report includes review comments and review decisions.

[0092] This disclosure also provides an LLM fine-tuning method for automated paper review. Figure 2A This is a flowchart of an LLM fine-tuning method for automated paper review according to certain embodiments of this disclosure. The method includes steps 201 and 202.

[0093] Step 201: First, obtain a collection of published academic papers as a training sample set, or extract it from existing paper samples. The obtained training sample set or existing paper samples contain paper content of different data types, as well as a set of review questions and comments for training the language model review task. Paper content refers to the textual information contained in research papers or journal articles. Paper content includes various components, including: title, abstract, introduction, methods, results, discussion, conclusion, and any other chapters or sections that constitute the main body of the written document. Paper content includes the text that conveys the research findings, analyses, arguments, and supporting evidence presented in the paper. A large amount of training sample set or existing paper sample content can be collected, including text paragraphs, tabular data, figures, and other content, as well as corresponding review questions and comments prepared by editors and peer reviewers.

[0094] Step 202: Train an LLM capable of outputting review opinions based on various data types (e.g., text, tables, images, and review questions / opinions) in the training sample set. Apply multimodal training to the LLM to obtain an LLM capable of outputting review opinions and decisions. By combining different training data types and employing multimodal learning, this enables the language model to process multiple data inputs simultaneously and further generate review opinion outputs for review questions. The LLM is then fine-tuned based on review opinions, review questions, and the training sample set. The LLM can be any suitable model, such as autoregressive and transform-based decoder models (e.g., Meta Llama).

[0095] Multimodal training implemented in LLM includes the following steps.

[0096] First, an open-source journal database is introduced to provide a source of paper samples for training. Existing paper samples from the database are processed using open-source methods (e.g., sciencebeam-parser) to extract text from different data types within the paper samples, such as titles, figures, tables, and images. Title-image pairs (i.e., figures, tables, and images) are collected as training data. In some embodiments, optical character recognition (OCR) can be used to extract text from the paper content. Open-source methods (e.g., sciencebeam-parser, python-docx) can be used to extract paper content from scanned documents to process the paper document files. In some embodiments, the extracted paper content includes submission, title, authors, organization, and textual information (e.g., abstract, methods, etc.), as well as review questions and reviewer comments.

[0097] Image encoders (e.g., ViT-L / 14 or ViT-g / 14) are used in LLMs. Image encoders can be cohesive units within an LLM. These visual encoders are kept frozen to ensure high-quality visual representations without the computational cost of retraining them. They can then be fine-tuned using an LLM (e.g., a Meta Llama model). LLMs can also be kept frozen to leverage their powerful generative capabilities in certain fine-tuning steps. In an LLM, the processing steps are summarized as follows. Fine-tuning methods can be implemented using prepared training data, image-text contrastive learning that maximizes the mutual information between image (e.g., charts, tables, and common images) and text representations, image-based text generation (which forces the encoder to generate text from the visual input), and image-text matching that improves fine-grained alignment by predicting whether image-text pairs match. Fine-tuning an LLM involves techniques such as supervised fine-tuning, where a pre-trained model is trained on a task-specific labeled dataset to improve performance for a specific task, such as GPT-4o fine-tuning for text summarization. For fine-tuning an LLM, a language modeling loss is used. Using multimodal training, LLM is fine-tuned in two phases—first focusing on visual-language representation learning, and then on visual-to-language generative learning. Finally, the model is evaluated on various visual-language tasks to assess how effectively it can use visual inputs (such as charts, tables, or images) to generate relevant text output.

[0098] In some embodiments, LLM includes autoregressive and transform decoder-based models.

[0099] Figure 2BSchematic diagrams of exemplary LLMs according to certain embodiments of this disclosure are shown, illustrating an LLM design for paper review. The LLM design may employ a decoder architecture and also includes a visual encoder capable of processing recalled tables and images. The inputs to the LLM include task cue text, recalled text from a knowledge graph, recalled tables and images, and question text. The visual encoder of the LLM is configured to accept tables and / or images, extract semantic features from the recalled tables and images, and output the corresponding text as input to the LLM decoder. Specifically, the visual encoder is configured to accept, encode, and fuse tables and images to generate text through multimodal embedding and cross-modal attention mechanisms. The extracted semantic features are embedded to introduce visual understanding capabilities. The LLM decoder is configured to decode the task cue text, recalled text from the knowledge graph, semantic features of the recalled tables and images, and question text. In the above method, the LLM decoder can generate simulated answers, retrieve relevant content from the paper knowledge graph, and identify content from the paper to be reviewed. For example, knowledge graph query methods (such as Cypher on Neo4j) can be used to retrieve relevant content from a paper knowledge graph. The LLM then processes multiple data inputs simultaneously and further generates opinion outputs for review questions. The type of extracted semantic feature output depends on the image type, as shown below.

[0100] From the chart:

[0101] Chart type: Indicates whether it is a bar chart, line chart, pie chart, etc.

[0102] Data points and trends: Extract values ​​and trends, such as patterns of increase or decrease.

[0103] Axis information: Captures the axis label, unit, and scale.

[0104] Legends and explanations: Understand what each color or symbol in the chart represents.

[0105] Note: Extract any textual annotations from the sections explaining the charts.

[0106] From the table:

[0107] Title: Identifying column and row headers to understand data dimensions.

[0108] Data type: Identify data types (e.g., numbers, text, categories).

[0109] Cell value: Extracts the content of a single cell.

[0110] Relationships: Understand the relationships that may be displayed in the table, such as grouping, sums, or averages.

[0111] From the image:

[0112] Objects and entities: Identify and label objects in images.

[0113] Relationships and Actions: Understanding the interactions between the objects or actions depicted.

[0114] Scene background: Extract or set background information.

[0115] The decoder's output is connected to the frozen LLM for fine-tuning. The fully connected layer adjusts the fine-tuned LLM's output to match the model's input dimension.

[0116] This disclosure also provides a method for fine-tuning an LLM using a knowledge graph to better grasp various academic papers. The LLM can be fine-tuned using existing datasets, and can be an existing model such as MetaLlama. The method includes the following steps.

[0117] Step 3.1: Utilize the extracted paper hierarchical structure to construct a paper knowledge graph based on the paper hierarchical structure and content. Figure 3A A schematic diagram illustrating the hierarchical structure and content of a paper according to an embodiment of this disclosure is shown. The hierarchical structure and content include authors, organization, abstract, keywords, chapters, references, etc.

[0118] Step 3.2: By extracting relevant content from individual paper knowledge graphs, construct a paper knowledge graph library that covers specific knowledge in related fields. Figure 3B A schematic diagram of a paper knowledge graph according to an embodiment of the present disclosure is shown.

[0119] Step 3.3: Integrating the paper knowledge graph and paper knowledge graph library with LLM will enhance LLM's ability to understand papers and provide supplementary knowledge, thereby ultimately improving LLM's comprehension ability.

[0120] After fine-tuning the LLM, a pre-trained LLM is obtained. The pre-trained LLM refers to the LLM described above. For example... Figure 3C As shown, for example, paper review is performed by inputting review questions (one or more) and the paper to be reviewed into a pre-trained LLM. The LLM then generates simulated responses. Based on the simulated responses, relevant content is retrieved from a paper knowledge graph. The paper knowledge graph includes precise and structured knowledge, domain-specific knowledge, etc. Simultaneously, relevant content is retrieved from the paper to be reviewed based on the simulated responses. Then, based on the relevant content from the paper knowledge graph and the relevant content from the paper to be reviewed, constructed prompts and images are obtained. Next, the constructed prompts and images are input into the pre-trained LLM to obtain review comments.

[0121] This disclosure also provides an automated paper review method and an automated paper review system for generating review decisions and reports. The automated paper review method is based on the aforementioned review system.

[0122] Figure 4A An automated paper review system according to certain embodiments of this disclosure is illustrated. The automated paper review system may include a cloud server, a communication network, and / or one or more user devices. The cloud server is configured to receive instructions from the user devices and perform the automated paper review tasks as described above. Review comments and reports for the target paper are sent back to the user devices. The communication network may be any suitable combination of one or more wired and / or wireless networks. The user devices may include any device suitable for interaction with a user.

[0123] Figure 4B A flowchart of an automated paper review system according to certain embodiments of the present disclosure is shown. This embodiment outlines the steps of an automated paper review solution for generating review decisions and reports through an automated paper review method. Users involved include, but are not limited to, journal editors, peer reviewers, and authors. The system applying the automated paper review method includes the following steps 4.1-4.5.

[0124] Step 4.1: The user submits the document to the cloud server through the system client. The cloud server includes an intelligent paper parsing module, a paper knowledge graph module, and a review engine module. After uploading the paper to be reviewed and the review questions from the client device, the intelligent paper parsing module automatically begins to analyze the paper. Specifically, it extracts structured information from the paper, such as the title, abstract, body text, tables, images, and other content.

[0125] Step 4.2: The system automatically constructs the corresponding paper knowledge graph through the paper knowledge graph module. By classifying the structured information and content of the papers obtained through analysis, the corresponding paper knowledge graph can be easily constructed. Original documents can be deleted in this step.

[0126] Step 4.3: The system converts questions from the pre-set review question database into question feature vectors within the review engine module. These question vectors are then matched and classified with the feature vectors of nodes in the generated paper knowledge graph. This process is performed in conjunction with the paper knowledge graph database. The purpose of the matching process is to obtain the content of the most relevant knowledge nodes. Figure 4B The content of three knowledge nodes is displayed in the middle.

[0127] Step 4.4: The system inputs the review questions and related content (such as ranking node data and paper content from the papers to be reviewed) into the pre-trained LLM to generate corresponding review comments. This step will be repeated to obtain the review comments corresponding to all preset review questions, which will be combined into a review comment set. The pre-trained LLM generates the corresponding review decisions based on the preset review comment set. The review decisions and the set of review comments together constitute the review report.

[0128] Step 4.5: The generated paper review report is output from the cloud server to the client as a text string. It is then formatted into a standard report for user reading on the client device. At this stage, all data related to the user submission, including the knowledge graph and review report, is deleted from the cloud server system.

[0129] This disclosure also provides a computer-readable storage medium having a computer program stored thereon. Figure 5 A schematic diagram of an electronic device according to certain embodiments of the present disclosure is shown. The electronic device includes a processor, which, under the instruction of executable instructions in a program, performs the following steps: 1) obtaining a review question and its corresponding vector from a communication interface; 2) determining a target node matching the vector from a paper knowledge graph; 3) generating review comments on the review question based on the content corresponding to the matching node using a pre-trained LLM; 4) making a review decision using a language model and preset review questions; 5) generating a paper review report; and 6) storing the program in a computer-readable storage medium. When the processor executes the stored steps / instructions, the above-described automated paper review solution can be implemented.

[0130] Based on the foregoing description, a first embodiment of this disclosure provides an automated peer review process for journal articles, comprising the following steps: First, it receives each review question as input from journal editors, peer reviewers, or authors participating in the peer review process, and then obtains the embedding vectors corresponding to these review questions. Second, it generates a knowledge graph of the target journal article. Third, it matches the vectors derived from the review questions with the specific knowledge graph of the target article and the overall knowledge graph library. This step aims to extract the most relevant content from the library to provide rich background knowledge for the target article. Fourth, the extracted content and the corresponding review questions are used as input to an LLM (Local Level Management Model), which generates and automatically outputs corresponding review comments for each review question related to the target article.

[0131] A second embodiment of this disclosure provides a pre-trained LLM for automated paper review and a training method thereof. The system acquires a training sample set containing paper content of various data types (text, tables, and images) for the review task, and a set of review questions / comment texts for training the pre-trained LLM. Based on different types of training sample data, the system performs multimodal training on the pre-trained LLM to generate a model capable of outputting review comments and decisions.

[0132] A third embodiment of this disclosure provides a method for enhancing the ability of a pre-trained LLM to understand journal articles using knowledge graphs. This primarily involves constructing knowledge graphs for individual articles to simplify their content and structure. Individual knowledge graphs of articles within the same journal are synthesized, and relationships between these articles are extracted to construct a journal-specific knowledge graph library. Finally, synthesizing knowledge graphs from articles from different journals within the same field creates a domain-specific knowledge graph library. The knowledge graphs from individual articles, the article knowledge graph library, and the pre-trained LLM are interconnected to implement the method described in the first part. This results in a pre-trained LLM enhanced by knowledge graphs.

[0133] The fourth embodiment of this disclosure provides a solution for protecting authors' undisclosed intellectual property during automated paper peer review. This includes not retaining or saving the original complete documents submitted in the automated digital platform. The paper knowledge graph generated from the submitted documents is used only to implement the methods described in the first embodiment. The generated paper knowledge graph is not updated in the paper knowledge graph repository. After automated review is completed, all data related to the submitted paper is deleted.

[0134] The fifth embodiment of this disclosure provides an automatic paper review system, including an intelligent document parsing module, a paper knowledge graph module, a review engine module, a training sample module, and a user interaction module. (1) The intelligent document parsing module extracts structured hierarchical information and paper content from the document. (2) The paper knowledge graph module generates a knowledge graph corresponding to the paper. This module also includes a paper knowledge graph library for storing pre-generated paper knowledge graphs. (3) The review engine module converts review questions into corresponding vectors and combines them with the paper knowledge graph. It determines the content of target document nodes that match the vectors in the stored knowledge graph and obtains the corresponding paper content data. The pre-trained LLM described in the second aspect generates review opinions on the review questions based on the content data that matches the review vectors. It also provides review decisions for submitted papers based on typical review questions of academic papers and a review question library. (4) The training sample module provides pre-processed paper data training samples of different data types for training the LLM. (5) The user interaction module generates a paper review report and provides human-computer interaction. The content of the review report includes review opinions and decisions generated from the review questions.

[0135] A sixth embodiment of this disclosure provides a computer storage medium storing a computer program. When executed by a processor, the program implements the solutions or systems described in the five embodiments above.

[0136] One advantage of the automated paper peer review solution proposed in this disclosure relates to the paper publication process from the perspectives of editors, reviewers, and authors. First, the solution can improve the quality of the paper peer review process and reduce turnaround time, effectively saving the workload and time of journal editors and peer reviewers, thereby improving the efficiency of paper publication. Second, the proposed automated paper peer review solution can provide authors with targeted review comments, improving paper quality and thus increasing the publication success rate. These two benefits contribute to the development and progress of the entire academic paper publication field. Furthermore, compared to general pre-trained LLMs, the pre-trained LLM used in this disclosure is trained with specialized scientific knowledge data, rather than with training data from everyday life. During automated paper peer review, the knowledge graph can analyze and represent paper knowledge at multiple levels. The paper knowledge graph provides the most relevant content to the LLM, avoiding illusions, preventing irrelevant or incomprehensible review comments, and improving its quality. This disclosure also addresses a key problem of previous automated review solutions: the inability to analyze the semantic content of papers. Another advantage is that this disclosure considers intellectual property and data privacy issues. It does not retain related data, such as the original submitted documents and the generated paper knowledge graph, and the pre-trained LLM does not use any content from the submitted documents for secondary training. It protects the authors' unpublished intellectual property and their rights in the automated paper review process.

[0137] The embodiments or elements shown in this disclosure, including specific illustrations and materials used in the examples, are intended to be illustrative and not restrictive. They allow for various changes, adjustments, or adaptations consistent with the basic concepts of this disclosure. It should be clarified that all depicted diagrams are for illustrative purposes only; they are neither drawn to scale nor exact replicas of actual devices.

[0138] Unless otherwise specified, individual embodiments or aspects and features relating to the accompanying drawings may be combined or interchanged with each other without limiting or expanding the scope of the disclosure, provided that such combination or interchange is meaningful and consistent with the meaning of this disclosure. Advantages described with respect to specific embodiments of this disclosure or with respect to specific drawings are also advantages of other embodiments of this disclosure, wherever applicable.

Claims

1. A method for automatically reviewing papers using a Large Language Model (LLM), the method comprising: Input multiple review questions and papers to be reviewed into the LLM. Based on the multiple review questions and the papers to be reviewed, the LLM generates multiple simulated responses. Multiple paper node data are obtained from multiple knowledge graph nodes in a paper knowledge graph database. These multiple paper node data are related to multiple simulated answers. The paper knowledge graph is a graph encompassing knowledge from the hierarchical structure of the paper to the publicly disclosed conceptual relationships within the paper. Multiple paper contents related to the multiple simulated responses were identified from the papers to be reviewed. Multiple combinations are input into the LLM, wherein each of the multiple combinations consists of one of the multiple review questions, one of the multiple paper node data, and one of the multiple paper content from the paper to be reviewed. The LLM is used to generate review comments for the paper to be reviewed for each combination.

2. The method according to claim 1, wherein, The multiple review questions are extracted from a pre-set review question library or from user input.

3. The method according to claim 1, further comprising: A review decision is generated based on the review comments.

4. The method according to claim 1, wherein, Obtaining multiple paper node data includes comparing the cosine similarity between the embedding of each of the multiple simulated answers and the embedding of each of the multiple knowledge graph nodes of the paper knowledge graph, obtaining each of the multiple knowledge graph nodes with the embedding that presents the highest cosine similarity as paper node data, and combining the paper node data of all the multiple simulated answers as the multiple paper node data.

5. The method according to claim 1, wherein, Identifying multiple paper contents from the paper to be reviewed includes: comparing the cosine similarity between the embedding of each of the multiple simulated responses and the embedding of each of the multiple paper contents from the paper to be reviewed; obtaining each of the multiple paper contents from the paper to be reviewed with the embedding that presents the highest cosine similarity as the paper content from the paper to be reviewed; and combining the paper contents for all the multiple simulated responses as the multiple paper contents from the paper to be reviewed.

6. The method according to claim 1, wherein, The LLM has been fine-tuned using a training sample set that includes paper content and review questions / comments of various data types.

7. The method according to claim 6, wherein, The training sample set includes multiple real or artificially constructed paper samples, which include text paragraphs, tabular data, graphics and other content, as well as corresponding review questions and comments prepared by editors and peer reviewers.

8. The method of claim 1, further comprising deleting all original data related to the paper to be reviewed after generating the review comments.

Images