A paragraph-level long text generation method based on supervised fine-tuning and reinforcement learning
By combining supervised fine-tuning and reinforcement learning, explicit reasoning and reflection mechanisms with interaction trajectory data are used to optimize the language model, which solves the problem of insufficient structural planning in the generation of long articles at the chapter level, and achieves high-quality outline generation and article writing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-02-12
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to effectively plan the structure for generating long articles at the chapter level, resulting in unclear chapter hierarchy, incomplete content coverage, and difficulty in achieving high-quality generation under resource constraints.
We employ a method based on supervised fine-tuning and reinforcement learning. An initial draft outline is generated through a teacher language model. Explicit reasoning and reflection mechanisms are introduced for structural revision. Interaction trajectory data is used for quality assessment and screening. Alignment optimization is performed using reinforcement learning of GRPO to improve the model's structural planning capabilities.
Under resource constraints, it significantly improves the ability of small-parameter language models to plan long-form outlines. The generated outlines are clear, hierarchical, and fully cover the content. They also have transferable structure optimization strategies and support the generation of high-quality explanatory articles.
Smart Images
Figure CN122133616A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of text-level long text generation technology, and in particular to a text-level long text generation method based on supervised fine-tuning and reinforcement learning. Background Technology
[0002] With the development of natural language processing technology, deep learning-based language models have been widely used in text generation tasks, especially in scenarios such as news writing, knowledge introduction, and explanatory article generation, demonstrating good language fluency and expressive ability. Existing large-scale pre-trained language models are usually trained on large-scale corpora and can automatically generate coherent text content based on the input topic, achieving certain results in short text generation and paragraph-level text generation tasks.
[0003] However, for long text generation tasks such as expository texts that require chapter-level structural planning, relying solely on the autoregressive generation capabilities of language models can easily lead to problems such as chaotic text structure, unclear chapter hierarchy, and incomplete content coverage. Expository articles typically have clear chapter divisions and logical hierarchical relationships, placing higher demands on overall structural consistency and content organization capabilities. Existing technologies still have limited stability and controllability in this type of task.
[0004] To alleviate structural issues in long text generation, some existing technologies have introduced a two-stage approach of outline generation followed by rewriting: first, an outline is generated, and then the main text is generated based on the outline. This approach improves the structural rationality of the generated text to some extent, but the quality of the generated outline directly affects the quality of the final article. If the outline itself has structural flaws, the subsequent article generated based on the outline will still struggle to meet the requirements of a full-text structure.
[0005] In existing outline generation techniques, mainstream methods typically rely on large-scale language models to directly generate document outlines through supervised fine-tuning or cue engineering. While these methods can achieve good results with sufficient data, they often depend on continuously expanding model size, resulting in high training and inference costs and making it difficult to stably achieve high-quality document-level structure planning on models with limited parameter sizes. Furthermore, training methods based solely on supervised learning cannot explicitly guide the model to learn the structure optimization process, and the model's self-correction ability for structural errors is limited.
[0006] In recent years, reinforcement learning methods have been introduced into language model alignment and optimization processes to improve the quality of generated content under specific evaluation metrics. However, existing reinforcement learning methods mostly focus on the scoring at the output level, with insufficient modeling of the structural evolution path during the generation process. Furthermore, in the absence of high-quality initial strategies, reinforcement learning training is prone to instability or convergence difficulties.
[0007] Furthermore, existing technologies typically utilize only static samples consisting of inputs and final outputs for training data construction, lacking explicit modeling of the reasoning, reflection, and revision trajectories during the generation process. This makes it difficult to effectively characterize the evolutionary process from low-quality to high-quality structures. Consequently, the model struggles to learn generalizable structural optimization strategies when faced with complex textual structure tasks.
[0008] Therefore, existing technologies still have the following shortcomings in the task of generating expository texts at the chapter level: first, insufficient modeling ability for chapter structure planning and hierarchical consistency; second, insufficient learning of structural revision and optimization paths during the generation process; and third, difficulty in balancing generation quality and training stability under limited parameter scale. These problems limit the further application of existing technologies in expository text generation scenarios. Summary of the Invention
[0009] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for generating long texts at the chapter level based on supervised fine-tuning and reinforcement learning.
[0010] The objective of this invention is achieved through the following technical solution: a method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning, the method comprising: Based on a single expository text topic input, the teacher's language model is invoked to obtain a chapter-level initial draft outline. A corresponding target outline is constructed based on the initial draft outline, and the initial draft outline and target outline are stored in pairs as early experience data. Explicit reasoning and reflection mechanisms are introduced into early empirical data to analyze the text structure and revise the structure of the initial draft outline; the reasoning process, reasoning results, and revision results are recorded as an interaction trajectory. Perform quality assessment and screening of interaction trajectory data; The language model is subjected to supervised fine-tuning cold start training based on the selected data, and alignment optimization is performed using GRPO-based reinforcement learning. The trained language model is used to generate a target chapter-level long-form outline, and a complete explanatory article is generated based on the target outline.
[0011] Furthermore, the initial draft outline contains a multi-level heading structure, and the initial planning of the topic is generated by calling the teacher model under the Logic interactive planning framework.
[0012] Furthermore, the target outline is generated by the teacher's language model under the guidance of the Logic method, and it is more reasonable than the initial draft outline in terms of chapter setting, hierarchical division, or content organization.
[0013] Furthermore, the introduction of explicit reasoning and reflection mechanisms into early experience data for text structure analysis includes: the teacher's language model refers to the target outline and performs a layer-by-layer structural check on the initial draft outline to identify problems such as missing chapters, disordered hierarchy, logical jumps, or insufficient content coverage; the teacher's language model explicitly generates the reasoning process text in natural language form.
[0014] Furthermore, the structural revision of the initial draft outline includes: The reasoning process text is used to generate structured feedback information, indicating the chapter positions that need to be adjusted, the adjustment methods, and the adjustment goals. The structure is then revised based on the structured feedback.
[0015] Furthermore, the quality assessment and screening of the interaction trajectory data includes: By comparing and analyzing the revised outline with the corresponding initial draft outline, we can determine the degree of improvement in terms of the completeness of the chapter structure, the consistency of the hierarchy, and the rationality of the content coverage. Interaction trajectory data with unclear reasoning process, inconsistent feedback and revision results, or insignificant structural optimization effects are removed.
[0016] Furthermore, the supervised fine-tuning cold start training of the language model based on the screened data includes: using the initial draft outline as the model input condition, and using the corresponding reasoning process, structured feedback and revision results as supervision signals to perform supervised fine-tuning training on the language model to be trained.
[0017] Furthermore, the alignment optimization using GRPO-based reinforcement learning includes: generating multiple candidate chapter-level long text outlines using a cold-start strategy model, and performing relative quality evaluation on the candidate results based on structural legality, hierarchical consistency, and content integrity indicators; and updating the model parameters using the GRPO method according to the relative evaluation relationship between the candidate results.
[0018] On the other hand, a device for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the aforementioned method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning.
[0019] On the other hand, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the aforementioned method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning.
[0020] The beneficial effects of this invention are: 1. It can significantly improve the ability of a small-parameter language model to plan the structure of long outlines under resource-constrained conditions; 2. By explicitly modeling the "criticism-revision" process, the model possesses a transferable and reusable structural optimization strategy; by combining supervised fine-tuning and reinforcement learning, the problem of insufficient generalization or local optima caused by a single training method is avoided. 3. The generated outline can directly support the writing of expository articles and has high practical application value. Attached Figure Description
[0021] Figure 1 This is an overall flowchart of the method provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the apparatus provided in an embodiment of the present invention. Detailed Implementation
[0022] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0023] like Figure 1 As shown, the invention constructs a training process around the task of generating long outlines, consisting of early experience collection, supervised learning cold start, and reinforcement learning alignment optimization. First, within the Logic interactive planning framework, a teacher model with strong reasoning and structural planning capabilities is used to sequentially generate a draft outline, structured feedback, and an optimized outline for a given topic, thereby collecting early experience interaction trajectories including the structural revision process. Subsequently, these interaction trajectories are standardized into a learnable data structure, and the corresponding reasoning and reflection processes during defect identification and structural revision are further made explicit to fully characterize the decision-making path from draft to optimized outline. Based on this, an automatic scoring mechanism is used to screen the reasoning and reflection data for quality, constructing a highly consistent training sample set. Next, supervised fine-tuning is used to cold start train a small-parameter language model, enabling it to initially master the ability to optimize outline structure based on feedback. Then, a reinforcement learning method based on group relative optimization is introduced to align and optimize the model's generation strategy, further improving the stability of the generated outlines in terms of structural legality and content quality. Finally, the trained model is used to generate long-form outline-driven explanatory articles, and the usability of the outlines is validated through a closed-loop process of actual writing. The specific steps are as follows: Step one: First, the expository text topic is received as input. This topic can be a single keyword, phrase, or a brief description, used to define the writing target and content scope of the expository article to be generated. Within the Logic method framework, the teacher's language model is invoked to perform chapter-level structural planning on the expository text topic, generating a corresponding initial draft outline. This initial draft outline contains multiple chapter titles and their hierarchical relationships, used to describe the overall structural arrangement of the expository text. However, this initial draft outline typically only reflects the model's initial planning capabilities and may have shortcomings in structural completeness, hierarchical rationality, or content coverage. Based on the generated initial draft outline, a target outline corresponding to it is further constructed. This target outline is generated by the teacher's language model under the guidance of the Logic method, and it has higher rationality in terms of chapter setting, hierarchical division, or content organization compared to the initial draft outline. The initial draft outline and the corresponding target outline are stored in pairs to construct early empirical data, used to characterize the evolutionary relationship of the expository text structure from the initial state to the optimized state.
[0024] Step Two: Introducing Explicit Reasoning and Reflection Mechanisms to Generate Interactive Trajectory Data. Based on the early experience data generated in Step One, which includes the initial draft outline and the target outline, explicit reasoning and reflection mechanisms are introduced to analyze its text structure. Specifically, the teacher's language model, referring to the target outline, performs a layer-by-layer structural check on the initial draft outline to identify issues such as missing chapters, disorganized hierarchy, logical jumps, or insufficient content coverage. During this analysis, the teacher's language model explicitly generates reasoning process text in natural language to explain the basis for its judgment that the initial draft outline has defects, thereby transforming the originally implicit judgment process into observable and learnable reasoning information. Based on this reasoning process, structured feedback information is further generated. This structured feedback clearly indicates the chapter positions that need adjustment, the adjustment methods, and the adjustment goals, providing guidance for subsequent structural revisions. Subsequently, based on the structured feedback, the initial draft outline is structurally revised, generating a revised outline. The structural revisions include, but are not limited to, adding, deleting, merging, splitting, or rearranging chapters. The reasoning process, structured feedback, and revision results are recorded in a unified manner to form complete interaction trajectory data, which is used to characterize the intermediate decision-making information in the process of optimizing the structure of expository texts.
[0025] Step 3: Quality assessment and screening of the interaction trajectory data. In this step, the interaction trajectory data generated in Step 2 is assessed for quality. This quality assessment mainly includes whether the reasoning process is complete and reasonable, whether the structured feedback is consistent with the reasoning process, and whether the revised results achieve structural quality improvement compared to the initial draft outline.
[0026] Specifically, by comparing and analyzing the revised outline with the corresponding initial draft outline, the degree of improvement in terms of chapter structure completeness, hierarchical consistency, and content coverage rationality is determined. Interaction trajectory data with unclear reasoning processes, inconsistent feedback with the revised results, or insignificant structural optimization effects are discarded. Simultaneously, the final output is scored using a 1-5 point rating scale (maximum 5 points per dimension, total 35 points) across seven dimensions: interest, organization, relevance, content coverage, logic, content breadth, and content depth. The final output is required to score higher than the initial draft to ensure significant quality improvement. Samples violating basic structural constraints (such as invalid hierarchical transitions or missing key header information) or with insufficient overall quality (total score not exceeding 32 points) are discarded. Only interaction trajectory data that clearly reflects the evolution of the expository text structure from a low-quality state to a high-quality state are retained, thus constructing a high-quality reasoning reflection dataset.
[0027] Step 4: Supervised Fine-tuning Cold Start Training Based on High-Quality Reasoning Reflection Data. Based on the high-quality reasoning reflection data selected in Step 3, supervised learning samples for language model training are constructed. Using the initial draft outline as the model input, and the corresponding reasoning process, structured feedback, and revision results as supervision signals, supervised fine-tuning training is performed on the language model to be trained. Through this supervised fine-tuning process, the language model learns discourse-level structural planning capabilities and masters the generation pattern of structural revision based on feedback information, thereby obtaining a cold start model with basic expository text structure optimization capabilities. The input template for supervised fine-tuning training is: "Draft outline: {draft outline}, Feedback: {feedback}, Outline: {outline}. Based on the draft outline and given feedback, generate reflective reasoning about how to provide feedback for the draft outline. You need to select one sentence from the feedback dimension to improve the draft outline, and then optimize the draft outline into the final outline based on the feedback."<reflective_critique> Within the tags, generate reflective reasoning about how to provide feedback for the draft outline;<iterative_refinement> Within the tags, generate reflective reasoning about how to iterate and optimize from the draft outline to the final outline. Please strictly adhere to the following format: Example format:<reflective_critique> {reflective_critique}< / reflective_critique> Feedback<iterative_refinement> {iterative_refinement}< / iterative_refinement> {outline}.
[0028] Step 5: Alignment Optimization Based on GRPO Reinforcement Learning. After supervised fine-tuning training, the cold-start model is used as the initial policy model, and a GRPO-based reinforcement learning method is introduced to further optimize the model's generation strategy. Under the condition of the same expository text topic or initial draft outline, multiple candidate chapter-level long text outlines are generated using the initial policy model, and the relative quality of the candidate results is evaluated based on indicators such as structural validity, hierarchical consistency, and content completeness. Based on the relative evaluation relationships between candidate results, the GRPO method is used to update the model parameters. Simultaneously, we designed a multiplicative reward function to guide the model from two aspects: structural reward and quality reward. The structural reward penalizes empty outlines and outputs with irregular structures, while the quality reward is calculated based on the average score and total score of seven dimensions: interest, organization, relevance, content coverage, logic, content breadth, and content depth. When the outline score reaches or exceeds 30 points, the reward increases exponentially. This exponential scaling incentivizes the model to focus more on generating high-quality outlines, thereby guiding the model to gradually shift towards generating higher-quality chapter-level long-text outlines, resulting in a language model further optimized in terms of structural quality.
[0029] Step Six: Generate a complete explanatory article based on the target chapter-level long-form outline. Based on the target chapter-level long-form outline generated in Step Five, determine the overall chapter structure and hierarchical order of the explanatory article. Guide the general-purpose language model to generate the corresponding explanatory article content sequentially according to the chapter divisions determined in the target chapter-level long-form outline, maintaining semantic and functional consistency between the chapter content and the outline entries during the generation process. Finally, integrate the explanatory article content generated from each chapter to obtain a complete explanatory article text that is structurally complete, logically coherent, and meets the requirements of explanatory writing.
[0030] To verify the effectiveness of the method of this invention, comparative experiments were conducted on the FreshWiki dataset and the WikiOutline dataset, and the experimental results are shown in Tables 1 and 2. The method of this invention was compared with existing methods such as Direct Gen, STORM, OmniThink, and Logic under the same experimental settings. As can be seen from Tables 1 and 2, compared with direct generation methods, the method of this invention achieves significant improvements in key structural quality indicators such as organization, logic, and content coverage in the task of generating chapter-level outlines. The generated outline structure is clearer, the hierarchy is more reasonable, and it can more completely cover the core content related to the topic of the expository text. These advantages are particularly evident under conditions of small-parameter language models.
[0031] Furthermore, by introducing GRPO-based reinforcement learning alignment optimization on the basis of the method of this invention, the model achieves stable improvements in multiple indicators such as title soft recall rate, title entity recall rate, organization, logic, and content depth, as shown in Tables 1 and 2. Experimental results demonstrate that through reinforcement learning alignment optimization, the small-parameter language model significantly approaches or even reaches the generation level of large-parameter language models in terms of the quality of generating long-form outlines at the chapter level. In summary, the experimental results fully verify the effectiveness of the method of this invention in expository text chapter structure planning and long-form text generation tasks, indicating its good versatility, stability, and practical application value.
[0032] Table 1. Experimental results using the FreshWiki dataset.
[0033] Table 2 Experimental results of the WikiOutline dataset
[0034] Corresponding to the aforementioned embodiment of a text-level long text generation method based on supervised fine-tuning and reinforcement learning, the present invention also provides an embodiment of a text-level long text generation device based on supervised fine-tuning and reinforcement learning.
[0035] See Figure 2 The present invention provides a document-level long text generation device based on supervised fine-tuning and reinforcement learning, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a document-level long text generation method based on supervised fine-tuning and reinforcement learning in the above embodiment.
[0036] The present invention provides an embodiment of a document-level long text generation device based on supervised fine-tuning and reinforcement learning, which can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 2 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is a document-level long text generation device based on supervised fine-tuning and reinforcement learning provided by the present invention. (Except for...) Figure 2In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0037] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0038] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0039] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a method for generating long-text articles based on supervised fine-tuning and reinforcement learning as described in the above embodiments.
[0040] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0041] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for generating long-text articles based on supervised fine-tuning and reinforcement learning.
[0042] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.
[0043] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning, characterized in that, The method includes: Based on a single expository text topic input, the teacher's language model is invoked to obtain a chapter-level initial draft outline. A corresponding target outline is constructed based on the initial draft outline, and the initial draft outline and target outline are stored in pairs as early experience data. Explicit reasoning and reflection mechanisms are introduced into early empirical data to analyze the text structure and revise the structure of the initial draft outline; the reasoning process, reasoning results, and revision results are recorded as an interaction trajectory. Perform quality assessment and screening of interaction trajectory data; The language model is subjected to supervised fine-tuning cold start training based on the selected data, and alignment optimization is performed using GRPO-based reinforcement learning. The trained language model is used to generate a target chapter-level long-form outline, and a complete explanatory article is generated based on the target outline.
2. The method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning according to claim 1, characterized in that, The initial draft outline contains a multi-level heading structure, and the initial planning of the topic is generated by calling the teacher model under the Logic interactive planning framework.
3. The method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning according to claim 1, characterized in that, The target outline is generated by the teacher's language model under the guidance of the Logic method, and it is more reasonable than the initial draft outline in terms of chapter setting, hierarchical division or content organization.
4. The method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning according to claim 1, characterized in that, The method of introducing explicit reasoning and reflection mechanisms into early experience data to analyze the text structure includes: the teacher's language model refers to the target outline and performs a layer-by-layer structural check on the initial draft outline to identify problems such as missing chapters, disordered hierarchy, logical jumps, or insufficient content coverage; the teacher's language model explicitly generates the reasoning process text in natural language form.
5. The method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning according to claim 4, characterized in that, The structural revision of the initial draft outline includes: The reasoning process text is used to generate structured feedback information, indicating the chapter positions that need to be adjusted, the adjustment methods, and the adjustment goals. The structure is then revised based on the structured feedback.
6. The method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning according to claim 1, characterized in that, The quality assessment and screening of the interaction trajectory data includes: By comparing and analyzing the revised outline with the corresponding initial draft outline, we can determine the degree of improvement in terms of the completeness of the chapter structure, the consistency of the hierarchy, and the rationality of the content coverage. Interaction trajectory data with unclear reasoning process, inconsistent feedback and revision results, or insignificant structural optimization effects are removed.
7. The method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning according to claim 1, characterized in that, The supervised fine-tuning cold start training of the language model based on the filtered data includes: using the initial draft outline as the model input condition, and using the corresponding reasoning process, structured feedback and revision results as supervision signals to perform supervised fine-tuning training on the language model to be trained.
8. The method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning according to claim 1, characterized in that, The alignment optimization using GRPO-based reinforcement learning includes: generating multiple candidate chapter-level long text outlines using a cold-start strategy model, and performing relative quality evaluation on the candidate results based on structural legality, hierarchical consistency, and content integrity indicators; and updating the model parameters using the GRPO method according to the relative evaluation relationship between the candidate results.
9. A device for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning as described in any one of claims 1-8.
10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a method for generating expository texts at the chapter level based on supervised fine-tuning and reinforcement learning as described in any one of claims 1-8.