Scientific literature review generation method and device based on large language model

Abstract

The invention relates to a scientific literature review generation method and device based on a large language model. The method comprises the following steps: giving a literature subject and a reference, and sequentially generating a title, a chapter title and an abstract of a literature review by utilizing a large language model; and selecting a reference subset for each chapter by using a large language model, and generating a section title and section content according to the generated title, chapter title, abstract and the selected reference subset. According to the method, the scientific literature review meeting the requirements can be gradually generated on the basis of the given literature subject and the reference literature, the continuity and correlation of the generated scientific literature review can be improved, meanwhile, the efficiency of scientific research is greatly improved, and the use cost of a large language model API is remarkably reduced.

Description

Technical Field

[0001] The present invention belongs to the fields of information technology and artificial intelligence technology, and specifically relates to a method and device for generating a scientific literature review based on a large language model. Background Art

[0002] Literature review is an indispensable part of scientific research. It is a systematic compilation and summary of existing research results in a certain field or topic. It is the starting point for researchers to enter new fields and conduct new research. Through literature review, researchers can fully understand the research status, main viewpoints, controversial issues and future development trends in the field, thereby providing theoretical support and direction guidance for their own research. However, in recent years, the number of various types of literature has grown exponentially, and researchers have spent a lot of time and energy on literature review, resulting in low research efficiency. Therefore, it is of great significance to automatically generate scientific literature reviews with the help of large language models.

[0003] However, due to practical factors such as the length of literature review texts and the high cost of using large models, few researchers have been able to provide a systematic approach to solve the problem of automatic generation of literature reviews. For example, AutoSurvey[1], a work that was developed at the same time as the present invention, proposed to equip large models with initial retrieval and outline generation functions to automatically generate literature reviews; the literature[2] wrote a review with the help of a large model. Different from the above work, this proposal focuses on the NLPCC2024Scientific Literature Survey Generation evaluation task[3], which is the first public evaluation task on automatic generation of literature reviews.

[0004] In addition, in the field of generating long texts with the help of large language models, reference [4] explored a new attention mechanism to effectively model the generation of long-context texts; reference [5] used a detailed outline to improve the coherence of long story generation; reference [6] proposed a pipeline including outline generation, information retrieval enhancement, enhanced outline generation, and long blog content generation. Different from the above work, this scheme is mainly aimed at the field of automatic generation of scientific literature reviews, and adopts an outline-based literature generation strategy to gradually manage the long context required to generate literature reviews.

[0005] The above-mentioned references [1] to [6] are as follows:

[0006] [1]Wang,Y.,Guo,Q.,Yao,W.,Zhang,H.,Zhang,X.,Wu,Z.,Zhang,M.,Dai,X.,Zhang,M.,Wen,Q.,et al.:Autosurvey:Large language models can automaticallywrite surveys.arXiv preprint arXiv:2406.10252(2024)

[0007] [2]Schulhoff,S.,Ilie,M.,Balepur,N.,Kahadze,K.,Liu,A.,Si,C.,Li,Y.,Gupta,A.,Han,H.,Schulhoff,S.,et al.:The prompt report:A systematic survey ofprompting techniques.arXiv preprint arXiv:2406.06608(2024)

[0008] [3]Tian,Y.,Gu,X.,Li,A.,Zhang,H.,Xu,R.,Li,Y.,Liu,M.:Overview of thenlpcc2024shared task6:Scientific literature survey generation.In:Proceedingsof the NLPCC 2024Conference(2024)

[0009] [4]Xiao,G.,Tian,Y.,Chen,B.,Han,S.,Lewis,M.:Efficient streaminglanguage models with attention sinks.In:ICLR(2023)

[0010] [5]Yang,K.,Klein,D.,Peng,N.,Tian,Y.:Doc:Improving long storycoherence with detailed outline control.arXiv preprint arXiv:2212.10077(2022)

[0011] [6]Lee,Y.,Ka,S.,Son,B.,Kang,P.,Kang,J.:Navigating the path of writ-ing:Outline-guided text generation with large language models.arXiv preprintarXiv:2404.13919(2024)

[0012] Automatically generating scientific literature reviews has gradually attracted the attention of researchers because of its advantages such as improving research efficiency and reducing the time cost of scientific researchers. However, the existing work on automatically generating scientific literature reviews faces some challenges. First, literature reviews are usually very long. Advanced large models such as GPT-4 and Claude 3 are constrained by the size of 4k-8k when outputting tokens (words), so they cannot generate a complete literature review in one output; and trying to sequentially generate the content of the entire review based on all the previously generated content may also have certain problems, because the large model may forget the early information or fail to maintain a consistent structure without an overall plan. Secondly, scientific literature reviews cover a lot of information. In terms of the use of large language model APIs, it may be very expensive to pass this information to the large model and generate the corresponding review, which reduces its actual use value. Therefore, few researchers can currently provide a systematic approach to solve the problem of automatic generation of literature reviews. Summary of the invention

[0013] In view of the above-mentioned shortcomings and to improve research efficiency, the present invention proposes a method and device for generating a scientific literature review based on a large language model, which guides the large language model to automatically generate the required literature review in a step-by-step prompting manner.

[0014] The technical solution adopted by the present invention is as follows:

[0015] A method for generating a scientific literature review based on a large language model comprises the following steps:

[0016] Given a document topic and references, use a large language model to sequentially generate the title, chapter title, and abstract of the literature review;

[0017] A large language model is used to select a subset of references for each chapter, and then section titles and section contents are generated based on the generated title, chapter title, abstract, and the selected subset of references.

[0018] Furthermore, the task of generating a scientific literature review is decomposed into six steps, and corresponding prompts are designed for the six steps respectively; the six steps include: title generation, chapter title generation, abstract generation, selection of a reference subset for each chapter, section title generation, and section content generation.

[0019] Furthermore, the title generation includes:

[0020] Use semicolons and line breaks in the prompt to connect multiple literature topics and references respectively;

[0021] Instruct the large language model in the prompt to generate the title using the specific token "Title:" in the last line;

[0022] Validates the output format and regenerates the header using the same prompt if a specific tag is not found.

[0023] Furthermore, the chapter title generation includes:

[0024] Generate chapter titles using prompts while preserving previous information, including references, literature topics, titles, and possible explanations;

[0025] Instruct the big model to generate multiple chapter titles, and regenerate chapter titles only when the number of titles generated by the big model is not within the set range;

[0026] Use line breaks to connect the chapter headings to form an outline for your literature review.

[0027] Furthermore, the summary generation includes:

[0028] Use the prompts to generate a paper abstract based on the given literature topic, generated title and outline.

[0029] Further, the selecting of a reference subset for each chapter includes:

[0030] For each generated chapter title, a prompt is used to instruct the large language model to select a subset of relevant references, and the selected subset of relevant references is used to subsequently generate section titles and section contents for the specific chapter.

[0031] Furthermore, the section title generation includes:

[0032] The prompt is fed into a large language model, which generates section headings for a given chapter title based on the title, outline, and a selected subset of references;

[0033] The generated outline is extended by merging all section headings to provide a complete and detailed outline for the next step of generating section content.

[0034] Furthermore, the section content generation includes:

[0035] Provide an outline consisting of all chapter and section titles, as well as a selected subset of relevant references, and guide the large language model to appropriately cite these references through examples to generate section content;

[0036] Emphasize the section title several times in the prompt to ensure that the large model stays focused on the task of generating section content.

[0037] A scientific literature review generation device based on a large language model, comprising:

[0038] The outline generation module is used to sequentially generate the title, chapter title and abstract of the literature review using a large language model given the literature topic and references;

[0039] The section title and section content generation module is used to select a reference subset for each chapter using a large language model, and then generate a section title and section content based on the generated title, chapter title, abstract and the selected reference subset.

[0040] The beneficial effects of the present invention are as follows:

[0041] Given the topic and references of the literature, the present invention designs a series of prompts, adopts an outline-based generation strategy, and guides the large language model to sequentially generate the title, abstract, chapter title, and main content of the literature review. In this process, by breaking down the task of generating a scientific literature review into manageable steps, the large model can maintain a high-level perspective while generating the title, thereby improving the coherence and relevance of the generated review. In addition, after the title is generated, the generation of the main content part can be conditioned on the chapter structure instead of all the previously generated content, thereby significantly reducing the cost of using the large language model API.

[0042] The present invention can gradually generate a scientific literature review that meets the requirements based on given literature topics and references, which can greatly improve the efficiency of scientific research. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the scientific literature review generation task.

[0044] Figure 2 It is a scientific literature review that gradually generates an overall framework. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments and drawings.

[0046] The present invention designs a method and device for generating scientific literature reviews based on a large language model.

[0047] (1) Task Description

[0048] This paper mainly focuses on the NLPCC 2024 Scientific Literature Survey Generation evaluation task, which focuses on generating literature reviews based on given literature topics and references, such as Figure 1 shown.

[0049] Specifically, given a reference list R = {r1, r2, ..., r n} and the literature subject list S = {s1,s2,…,s m}, the goal of this invention is to generate a well-structured literature review<T,A,M> The output specifically includes the title T, the abstract A and the body M = {(h1,c1),(h2,c2),…(h k ,c k )}, where h k Indicates titles of different levels, such as chapter title, section title, and further subsection title, etc. k Indicates the corresponding content.

[0050] (2) Overall framework

[0051] The framework of the scientific literature review generation method based on a large language model proposed in this invention is as follows: Figure 2 As shown, it mainly consists of six steps and is divided into two main stages.

[0052] The first stage is the outline generation stage. This stage mainly instructs the large language model to sequentially generate the title, chapter titles, and abstract of the literature review. Each step in this stage is conditioned on all previous outputs to ensure that the planning of the entire literature review is coherent.

[0053] The second stage is the section title and section content generation stage. The present invention selects a reference subset for each chapter, and then generates section titles and section content based on the generated outline and the selected reference subset. This design ensures that the generation of each section can understand the overall structure of the literature review, while also reducing the input length and saving the cost of using the large language model API.

[0054] (3) Prompt design

[0055] This program is Figure 2 The six steps in the code are designed with corresponding prompts, which are as follows:

[0056] 1) Step 1: Title generation

[0057] Based on the provided subject {subject} and references {references}, this solution guides the large language model to generate a suitable title for the literature review using the prompt Prompt-1 shown below. Here, a semicolon (;) and a line break (\n) are used to connect multiple subject and references, respectively.

[0058]

[0059] The Chinese form of the above Prompt-1 is:

[0060]

[0061] To highlight, the model is instructed to generate the title using a specific token "Title:" in the last line. In addition, the output format is verified, and if the specific token is not found, the title is regenerated using the same prompt as Prompt-1, but with an additional prefix: The response format is incorrect. Note that only one guess is required. The title should start with "Title:". The Chinese version is: The format just output is incorrect, note that only one title is output, and the output title should start with "Title:" as the starting token.

[0062] 2) Step 2: Chapter title generation

[0063] Generate chapter titles using prompt-2 as shown below, while retaining the previous information, including references, literature topics, titles, and possible explanations. Here, the large model is instructed to generate 6-10 chapter titles. A small number error is acceptable. Chapter titles will only be regenerated when the number of chapter titles generated by the large model is not within the set range (for example, 3-25). Use line breaks (\n) to connect these chapter titles to form the outline of the literature review.

[0064]

[0065] The Chinese form of the above Prompt-2 is:

[0066]

[0067] 3) Step 3: Summary Generation

[0068] This solution generates a paper abstract based on the given document subject {subject}, generated title {title} and outline {outline}, using the prompt Prompt-3 shown below.

[0069]

[0070] The Chinese form of the above Prompt-3 is:

[0071]

[0072] 4) Step 4: Select a subset of references for each chapter

[0073] Using the complete reference list for subsequent generation steps will result in a high cost of using the Big Language Model API. In addition, always using the same references will also lead to similarities between chapters. Therefore, before generating the specific structure and content of each chapter, this solution first selects some references for each chapter as the document subset of each chapter. Specifically, for each chapter title generated in step 2, use the prompt Prompt-4 shown below to instruct the Big Language Model to select the relevant reference subset. The selected reference subset will be used to subsequently generate the section title and section content of the specific chapter.

[0074]

[0075] The Chinese form of the above Prompt-4 is:

[0076]

[0077] 5) Step 5: Section title generation

[0078] This solution uses the prompt Prompt-5 shown below to generate section headings. The large language model generates section headings for a given chapter heading based on the title {title}, outline {outline}, and the selected reference subset {subset of references}; then the outline generated in step 2 is expanded by merging all section headings to provide a complete and detailed outline for the next step of section content generation.

[0079]

[0080] The Chinese form of the above Prompt-5 is:

[0081]

[0082] 6) Step 6: Section content generation

[0083] This solution uses prompt-6 as shown below to generate detailed content for each section. To ensure the coherence of the content and the consistency of the structure, this solution provides an outline consisting of all chapter titles and section titles, and also provides a subset of relevant references generated by step 4, and guides the large language model to appropriately cite these reference subsets through examples to generate section content.

[0084]

[0085]

[0086] The Chinese form of the above Prompt-6 is:

[0087]

[0088] In step 6, the prompt may become very long, causing the large language model to lose focus on the task of generating section content. To address this problem, this scheme emphasizes the section title {subsec heading} multiple times in Prompt-6 to ensure that the large model maintains focus.

[0089] The key points of the present invention are:

[0090] 1) The overall process design and implementation of this method focuses on the coordination of two stages and six steps.

[0091] 2) Design and implementation of chapter content generation strategy based on the outline.

[0092] 3) Design and implementation of six prompts for each of the six steps.

[0093] The present invention uses the Qwen-long version of the Alitong Yiqianwen model to implement a scientific literature review generation system, and uses the dataset provided in the NLPCC 2024 Scientific Literature Survey Generation evaluation task to automatically generate scientific literature reviews. In this scientific literature review generation evaluation task, the system won the third place in the competition, with an overall score only 0.03% lower than the second place team. An example of the generated results is shown in Table 1.

[0094] As can be seen from Table 1, the system can successfully capture the theme of the paper "Interpretability of Neural Networks" and generate relevant titles. For the generation of the outline, it also successfully captures the key aspects of the review, such as explainable artificial intelligence, black box models, etc. For the abstract, content, and citations, it can be seen that the generated review has some similarities with the references (bold in Table 1), resulting in appropriate abstracts and citations. In short, the system can gradually generate scientific literature reviews that meet the requirements based on the given literature topics and references, which can greatly improve the efficiency of scientific research.

[0095] Table 1. Example of scientific literature review generation

[0096]

[0097] Another embodiment of the present invention provides a scientific literature review generation device based on a large language model, comprising:

[0098] The outline generation module is used to sequentially generate the title, chapter title and abstract of the literature review using a large language model given the literature topic and references;

[0099] The section title and section content generation module is used to select a reference subset for each chapter using a large language model, and then generate a section title and section content based on the generated title, chapter title, abstract and the selected reference subset.

[0100] The division of the above modules is only an example. In actual applications, the above functions can be assigned to different functional modules as needed to complete all or part of the functions described in the above method. The specific working process of each module can refer to the corresponding process in the above method embodiment, which will not be repeated here.

[0101] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0102] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0103] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. A method for generating scientific literature reviews based on a large language model, characterized in that: The following steps are involved: Given a document topic and references, use a large language model to sequentially generate the title, chapter title, and abstract of the literature review; A large language model is used to select a subset of references for each chapter, and then section titles and section contents are generated based on the generated title, chapter title, abstract, and the selected subset of references.

2. The method according to claim 1, characterized in that: The task of generating a scientific literature review is decomposed into six steps, and corresponding prompts are designed for the six steps respectively; the six steps include: title generation, chapter title generation, abstract generation, selection of a reference subset for each chapter, section title generation, and section content generation.

3. The method according to claim 2, characterized in that The title generation includes: Use semicolons and line breaks in the prompt to connect multiple literature topics and references respectively; Instruct the large language model in the prompt to generate the title using the specific token "Title:" in the last line; Validates the output format and regenerates the header using the same prompt if a specific tag is not found.

4. The method according to claim 2, characterized in that: The chapter title generation includes: Generate chapter titles using prompts while preserving previous information, including references, literature topics, titles, and possible explanations; Instruct the big model to generate multiple chapter titles, and regenerate chapter titles only when the number of titles generated by the big model is not within the set range; Use line breaks to connect the chapter headings to form an outline for your literature review.

5. The method according to claim 2, characterized in that: The summary generation includes: Use the prompts to generate a paper abstract based on the given literature topic, generated title and outline.

6. The method according to claim 1, characterized in that The subset of references selected for each chapter includes: For each generated chapter title, a prompt is used to instruct the large language model to select a subset of relevant references, and the selected subset of relevant references is used to subsequently generate section titles and section contents for the specific chapter.

7. The method according to claim 6, characterized in that The section header generation includes: The prompt is fed into a large language model, which generates section headings for a given chapter title based on the title, outline, and a selected subset of references; Expand the generated outline by merging all section headings to provide a complete and detailed outline for the next step of section content generation; The section content generation includes: Provide an outline consisting of all chapter and section titles, as well as a selected subset of relevant references, and guide the large language model to appropriately cite these references through examples to generate section content; Emphasize the section title several times in the prompt to ensure that the large model stays focused on the task of generating section content.

8. A scientific literature review generation device based on a large language model, characterized in that: include: The outline generation module is used to sequentially generate the title, chapter title and abstract of the literature review using a large language model given the literature topic and references; The section title and section content generation module is used to select a reference subset for each chapter using a large language model, and then generate a section title and section content based on the generated title, chapter title, abstract and the selected reference subset.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images