A method and system for constructing an instruction fine-tuning dataset for large model long text generation

Abstract

This invention relates to the field of dataset construction technology, and more particularly to a method and system for constructing a fine-tuning instruction dataset for generating long texts from large models. The method includes: hierarchically decomposing documents from semi-structured or structured open-source knowledge bases; extracting structured content according to three levels: topic, outline, and paragraph; removing noise to form structured units; constructing a search query for each structured unit; retrieving several document entries from the entire internet, a specified knowledge base, and / or a retrieval system; selecting the document entries with the highest similarity to construct corresponding reference summaries; generating fine-grained question-answer pairs based on each structured unit and its corresponding reference summaries; and performing quality filtering and confidence evaluation on all fine-grained question-answer pairs to form a structured question-answer dataset. This invention effectively improves the quality and efficiency of generated data through a multi-agent collaboration mechanism and a hierarchical task decomposition strategy.

Description

Technical Field

[0001] This invention relates to the field of dataset construction technology, and in particular to a method and system for constructing instruction-based fine-tuning datasets for generating long texts from large models. Background Technology

[0002] With the rapid development of deep learning and natural language processing technologies, long-form article generation (LFAG) tasks based on large pre-trained language models (such as GPT, BERT, and LLaMA) have achieved remarkable results in applications such as intelligent writing, content generation, and automatic question answering. Long-form article generation is a complex task that requires the generated content to not only have logical consistency and contextual coherence, but also to cover rich knowledge content and structured information.

[0003] Long text generation datasets are used to further train existing text generation models, enhancing their ability to generate long texts. Most existing long text generation datasets lack a clear hierarchical structure and fine-grained annotations, leading to defects in logical reasoning, content consistency, and structural completeness in the generated text. For example, traditional text generation methods often rely on a single information source, lacking multi-dimensional evaluation and verification of the generated content. This is particularly pronounced in long article generation, where the generated content often exhibits incoherence and loose structure, failing to meet the demands of high-quality generation tasks.

[0004] Secondly, the citation information involved in long text generation tasks often lacks accurate annotation. Annotation of citation information is crucial to ensuring the accuracy and reliability of the generated content. However, in existing datasets, the accuracy and completeness of citations are often not given sufficient attention, directly affecting the credibility and academic rigor of the generated content. Furthermore, existing datasets often lack support for multi-stage training, making it impossible to effectively decompose the task into multiple stages for processing and optimization. The model often relies on a single fixed input during the generation process, lacking adaptability and flexibility to the needs of different stages, leading to information omissions or logical inconsistencies in the generated content.

[0005] Currently, although several generative models have made some progress in the field of long text generation, most existing training data rely on traditional text datasets (such as Wikipedia and news reports). These datasets typically lack detailed annotations and hierarchical support for the structure of long texts, often providing only simple paragraph and content structures, which are insufficient to cover the complex structures and diverse content required for generating long articles. Although some existing text generation methods have considered the overall structure of the article, they still suffer from problems such as over-reliance on a single data source and insufficient annotation during data processing, resulting in limitations in the diversity and accuracy of the generated articles.

[0006] In recent years, semi-structured data (such as Wikipedia, technical documents, and XML documents) has become an important data source for data generation tasks. Semi-structured data is a data type that falls between structured and unstructured data. It has a certain organizational structure but does not fully conform to the traditional relational database model. Common semi-structured data includes XML, JSON files, HTML documents, and log files. Semi-structured data typically contains rich hierarchical structural information, such as titles, paragraphs, and chapters. By parsing these structured tags, it is possible to effectively extract article summaries, keywords, and other structured information, thereby improving the ability to perform tasks such as text generation, summary generation, or question answering. The advantage of semi-structured data lies in its hierarchical structure, which provides rich contextual clues for generation tasks, helping the generation model understand the relationships between content and improving the accuracy and efficiency of dataset construction. However, processing semi-structured data is challenging; data parsing, information extraction, and cleaning are very tedious and require significant resources for in-depth mining and structuring.

[0007] With the surge in data volume, semi-structured data has far surpassed structured data in scale, becoming a valuable resource for long text generation tasks. However, due to its high data complexity and stringent processing requirements, the effective development and utilization of semi-structured data still faces significant challenges. Currently, many datasets fail to fully mine and apply this hierarchical information, resulting in a large amount of potential data value remaining untapped. How to efficiently extract key information from semi-structured data and transform it into a high-quality dataset supporting long text generation tasks is a pressing technical challenge that needs to be addressed. Summary of the Invention

[0008] This invention addresses significant shortcomings in existing methods for constructing long text generation datasets, including insufficient structured annotation, missing citation information, and inadequate task decomposition. Existing datasets often rely on a single information source and lack hierarchical, fine-grained annotations specific to long text generation tasks, resulting in deficiencies in logical consistency, information completeness, and citation accuracy. The purpose of this invention is to overcome these shortcomings by proposing a method for constructing a fine-tuning instruction dataset for large-scale long text generation models.

[0009] In view of this, the present invention proposes a method for constructing an instruction fine-tuning dataset for generating long texts in large models, comprising:

[0010] Step 1: Decompose the documents of semi-structured or structured open-source knowledge bases hierarchically, extract structured content according to three levels: topic, outline, and paragraph, remove noise, and form structured units;

[0011] Step 2: For each structured unit, construct a search query, retrieve several document entries from the entire network, the specified knowledge base and / or search system, select the document entries with the highest similarity, and construct the corresponding reference abstract;

[0012] Step 3: Generate fine-grained question-answer pairs based on each structured unit and its corresponding reference abstract;

[0013] Step 4: Perform quality filtering and confidence evaluation on all fine-grained question-answer pairs to form a structured question-answer dataset.

[0014] Preferably, step 1 includes:

[0015] Automated scripts are used to extract data from documents, with the extraction rule being Hierarchy(D). i )for:

[0016] Hierarchy(D i ) = {Topic, {Outline ij},{Paragraph ijk}}

[0017] Among them, D i This represents the i-th document, Topic represents the topic, and Outline represents the content. ij Paragraph represents the j-th outline heading of the i-th document. ijk This represents the content of the k-th paragraph in the i-th document, which belongs to the j-th outline.

[0018] Delete blank paragraphs and incorrectly formatted paragraphs;

[0019] Merge short paragraphs that share a consistent theme;

[0020] Standardize hierarchical relationships to ensure a reasonable number of outlines and paragraphs under the same theme;

[0021] D i Decompose into triples: U ijk =(Topic) i Outline ij Paragraph ijk )

[0022] Repeat the above steps until you obtain triples for all documents, which then form a set U of structured units:

[0023] U={U ijk}

[0024] Preferably, the automated script includes: regular expression matching, an HTML / XML parser, and a structure extraction algorithm based on heading level changes.

[0025] Preferably, step 2 includes:

[0026] For each structured unit U in set U ijk Topic i Sub-topic Outline ij Paragraph content ijk Keyword extraction, stop word removal, and phrase recombination were performed to construct the retrieval query Q. ijk :

[0027] Q ijk =f(Topic i Outline ij Paragraph ijk )

[0028] By using keyword enhancement, domain limitation, and phrase recognition, Q... ijk Enhance;

[0029] For the enhanced search query Q′ ijk Perform a search operation on the entire internet, a specified knowledge base, and / or a search system to obtain a search result set R. ijk :

[0030] R ijk ={r1,r2,r3,...r m}

[0031] Where, r m This represents the m-th retrieved document entry, including its title, abstract, and a portion of the text.

[0032] U is calculated according to the following formula. ijk With Rijk Each document entry r m Similarity Sim(U ijk ,r m ), and select the document entries with the highest similarity from them;

[0033] Sim(U ijk ,r m =cosine_similarity(U ijk ,r m )

[0034] Wherein, cosine_similarity represents cosine similarity;

[0035] For each selected document entry, record the document title, authors, publication year, source, and excerpt, forming a structured unit U. ijk Citation ijk :

[0036] Citation ijk =(Title,Authors,Year,Source,Excerpt).

[0037] Preferably, step 3 includes:

[0038] For each structured content unit U ijk and its corresponding reference abstracts Citation ijk The system pre-sets a multi-dimensional problem template set T = {t1, t2, ..., t} l}, where each template t l Indicates a type of question;

[0039] Connect the template set with U ijk Semantic combination automatically generates a candidate question set Q ijk ={t1(U ijk ),t2(U ijk ),…};

[0040] For each candidate problem q p ijk ∈Q ijk Based on structured content U ijk Reference Abstract Citation ijk The GenAns module, an instruction-based question-and-answer generation module, is invoked to generate the answer a. p ijk :

[0041] a pijk =GenAns(q p ijk U ijk Citation ijk )

[0042] All generated results are formatted to output a structured question-and-answer (QA) set. ijk ={(q 1 ijk ,a 1 ijk ),(q 2 ijk ,a 2 ijk ),...}

[0043] Each question and answer will be accompanied by a reference summary (Citation). ijk This forms fine-grained question-and-answer pairs.

[0044] Preferably, step 4 includes:

[0045] Each fine-grained question-and-answer pair generated in step 3 undergoes quality filtering, including content completeness, answer accuracy, and language standardization.

[0046] The confidence level is evaluated by combining the output probability or confidence scoring mechanism of the language model. If the confidence level is lower than the threshold θ, or if there is a logical contradiction, language incomprehension and / or an unfounded answer, the corresponding fine-grained question and answer is marked as a low-quality sample and removed.

[0047] Using semantic similarity calculation and content hash matching, duplicate or similar question and answer items are identified. Duplicate content is merged or high-confidence versions are retained, while the rest are removed.

[0048] The retained fine-grained question-and-answer data undergoes unified structure validation to ensure field completeness, valid citations, and that the content character set and encoding format conform to JSON / CSV standards. Simultaneously, it checks for format anomalies, empty fields, and illegal symbols. All validated data is then stored as a standardized sample set.

[0049] D QA ={(q i ,a i Citation i )}.

[0050] On the other hand, the present invention provides a system for constructing instruction fine-tuning datasets for generating long texts from large models, comprising:

[0051] The hierarchical decomposition module is used to hierarchically decompose documents from semi-structured or structured open-source knowledge bases. It extracts structured content according to three levels: topic, outline, and paragraph, removes noise, and forms structured units.

[0052] The retrieval module is used to construct a retrieval formula for each structured unit, retrieve several document entries from the entire network, a specified knowledge base and / or retrieval system, select the document entries with the highest similarity, and construct the corresponding reference abstract;

[0053] The question-answer pair generation module is used to generate fine-grained question-answer pairs based on each structured unit and its corresponding reference abstract;

[0054] The dataset building module is used to perform quality filtering and confidence evaluation on all fine-grained question-answer pairs to form a structured question-answer dataset.

[0055] Compared with the prior art, the advantages of the present invention are:

[0056] This invention provides a method for constructing a fine-tuning dataset for generating long text in large models. This method utilizes a multi-agent collaboration mechanism and a hierarchical task decomposition strategy, leveraging structured decomposition and task decoupling to efficiently transform semi-structured data sources into structured information, thereby generating a dataset that supports long text generation tasks. The method designs a collaborative workflow with clearly defined modules, achieving end-to-end management from raw data parsing, outline construction, content generation to citation extraction, thus improving the efficiency and accuracy of data construction.

[0057] This invention presents a multi-stage annotation method that supports citation tracking and structural consistency. This method introduces layer-by-layer annotation of structured elements such as outlines, paragraph content, and references during the data construction process, ensuring that the generated data possesses a clear structure and accurate citation information. This method effectively improves structural control and content consistency in long text generation tasks, and is particularly suitable for academic writing tasks with high requirements for citation consistency. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of a hierarchical decoupling and fine-grained annotation-based long text generation dataset construction method;

[0059] Figure 2 This is a flowchart of a hierarchical decoupling and fine-grained annotation-based long text generation dataset construction method. Detailed Implementation

[0060] This invention selects semi-structured data, represented by Wikipedia, as the data source, fully leveraging the advanced capabilities of large language models in text processing and knowledge extraction. Through multiple steps including data mining, citation retrieval, question-answering annotation, and data cleaning, this invention can efficiently extract useful information from semi-structured data and generate datasets suitable for long text generation tasks. Compared with traditional dataset construction methods, this invention significantly enhances the model's ability to handle complex information sources, improves the handling of structure and citations in long text generation tasks, and thus effectively improves the quality and diversity of the dataset.

[0061] Furthermore, by introducing a hierarchical task decomposition and staged training strategy, this invention enables the construction of long text generation datasets to be broken down into multiple sub-tasks, significantly improving the training efficiency and effectiveness of the generation model. Through fine-grained annotation and multi-stage training support, this invention provides a more flexible and efficient way to enhance the performance of long text generation models in various application scenarios, thus offering a new technical path for the development of the long text generation field.

[0062] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0063] Example 1

[0064] This invention proposes a method for constructing a fine-tuning dataset for generating long texts in large models, comprising a content decoupling module, a literature retrieval and summarization module, a fine-grained question-answering annotation module, and a data cleaning and quality control module. The method includes the following steps:

[0065] Step 1) Decompose the original semi-structured Baidu Encyclopedia open source data hierarchically, extract structured content according to three levels: topic, outline, and paragraph, and form a multi-level content framework to ensure the integrity and hierarchy of the literature knowledge system.

[0066] Step 2) For each content unit, the retrieval system automatically searches for relevant authoritative references, selects the literature fragment with the highest similarity from the search results as the reference abstract for that content unit, and records the citation information to support subsequent generation.

[0067] Step 3) Based on the hierarchical content and corresponding reference information, generate multi-dimensional fine-grained question-and-answer pairs for each knowledge unit, covering multiple knowledge points such as definition, principle, application, and limitation, to ensure the breadth and depth of the question-and-answer content.

[0068] Step 4) Perform quality audits on all generated data, including content consistency checks, citation matching assessments, and question-and-answer quality scoring, removing low-quality samples to ensure the accuracy and high-quality standards of the final dataset.

[0069] In the above technical solution, step 1) specifically includes:

[0070] Step 101) Obtain a semi-structured knowledge document dataset: This is the foundation for building a long text generation dataset. In this step, it is necessary to collect and organize various semi-structured or structured knowledge document sets (denoted as set K), such as data from open-source knowledge bases like Wikipedia.

[0071] Specifically, let the original dataset be:

[0072] K = {D1,D2,D3,...D} i}

[0073] Among them, D i This represents the i-th original document. Each document may contain multiple levels of content structure, such as document topic, headings, subheadings, and body paragraphs. Through unified processing and pre-screening, we ensure that the original documents have basic hierarchical potential, preparing them for subsequent structured decomposition.

[0074] Step 102) Processing highly structured knowledge document datasets using scripts: For the initially filtered data, an automated script is used for preliminary hierarchical parsing. During the parsing process, based on the structured tags in the text (such as HTML tags, Markdown tags, explicit outline numbering, etc.), specific hierarchical information is extracted: 1. First level: Topic 2. Second level: Outline 3. Third level: Paragraph. The extraction rules can be expressed as:

[0075] Hierarchy(D i ) = {Topic, {Outline ij},{Paragraph ijk}}

[0076] Among them, Outline ij Paragraph represents the j-th outline heading. ijk This indicates content belonging to the k-th paragraph under the j-th outline. The processing script includes, but is not limited to, techniques such as regular expression matching, HTML / XML parsers, and structure extraction algorithms based on heading level changes.

[0077] Step 103) Clean up noisy content and abnormal structure: Since open source knowledge bases may have problems such as disordered formatting, unordered paragraphs, empty paragraphs or incorrect annotations, data cleaning is required in this step, including: 1. Deleting blank paragraphs and paragraphs with incorrect formatting; 2. Merging small paragraphs with the same continuous topic; 3. Standardizing hierarchical relationships to ensure that the number of outlines and paragraphs under the same topic is reasonable.

[0078] Define the noise filtering function as follows:

[0079] Clean(D i )=D′ i

[0080] Where D′ i This is the cleaned-up annotated document.

[0081] Step 104) After cleaning is complete, each document is formally decomposed into structured units, specifically defined as triples:

[0082] U ijk =(Topic) i Outline ij Parallelgraph ijk )

[0083] gather:

[0084] U={U ijk}

[0085] This will serve as the basic input for subsequent steps such as retrieval, question and answer generation, and annotation.

[0086] In the above technical solution, step 2) specifically includes:

[0087] Step 201) Constructing the retrieval query: After obtaining set U from step one, for each content unit U... ijk This requires automatically generating search queries to retrieve relevant literature from external knowledge bases. Search Query Q ijk Build it in the following way:

[0088] Q ijk =f(Topic i Outline ij Paragraph ijk )

[0089] The function f(·) performs keyword extraction, stop word removal, and phrase recombination on the topic, subtopics, and paragraph content to generate a concise and representative query. To improve search performance, this step can employ the following techniques: 1. Keyword Expansion: Expanding core terms with synonyms. 2. Domain Restriction: If supported by the system, adding domain-specific terms (such as physics, finance, medicine, etc.). 3. Phrase Mining: Retaining proper nouns, technical terms, and other key expressions.

[0090] The formulaic representation of the search query is enhanced as follows:

[0091] Q′ ijk =Enhance(Q) ijk )

[0092] Step 202) Perform a literature search. Use search query Q′ ijk Perform a search operation on the entire internet or a specified knowledge base or retrieval system (such as a self-built academic database, open-source literature repository, or API interface such as Google Serper API or Bing API), and return a set of search results:

[0093] R ijk ={r1,r2,r3,...r m}

[0094] Where, r m This represents the m-th retrieved document entry, typically including basic information such as title, abstract, and excerpt from the text. To improve search accuracy, the system supports: 1. multi-round searches; 2. exclusion of irrelevant documents.

[0095] Step 203) Filtering highly relevant document fragments based on similarity matching: For the search result set R ijk It is necessary to evaluate the relationship between each document and the original content unit U. ijk The semantic similarity is used to select the segment with the highest similarity as the reference summary. The similarity function is:

[0096] Sim(U ijk ,r t =cosine_similarity(U ijk ,r m )

[0097] Here, cosine_similarity represents the cosine similarity.

[0098] Step 204) Record references and corresponding citation information: For each selected highly relevant fragment r * The following citation information needs to be standardized and recorded: 1. Title; 2. Authors; 3. Publication Year; 4. Source (Journal / Conference / Web); 5. Excerpt. The citation metadata record is defined as follows:

[0099] Citation ijk =(Title,Authors,Year,Source,Excerpt)

[0100] Finally, for each unit U ijk Link a corresponding reference abstract (Citation)ijk This serves as an important support for subsequent question-and-answer generation and fine-grained annotation.

[0101] In the above technical solution, step 3) specifically includes:

[0102] Step 301) Construct fine-grained question-answering templates and generate candidate questions: For each structured content unit U ijk and its corresponding reference abstract r * The system has a pre-set collection of multi-dimensional question templates:

[0103] T = {t1, t2, ..., t} l}

[0104] Each template t l This indicates a question type, such as definition, principle, application, or advantages and disadvantages. By combining the template with the semantics of the content unit, a set of candidate questions is automatically generated.

[0105] Q ijk ={t1(U ijk ),t2(U ijk ),...}

[0106] For example, questions such as "What is a convolutional neural network?" and "How does it work?" can be generated for the topic of "convolutional neural network", thus covering key knowledge dimensions.

[0107] Step 302) Generate answer text based on content and reference summary: For each candidate question q p ijk ∈Q ijk The system is based on structured content U ijk Reference Abstract Citation ijk The command-based question-and-answer generation module is invoked to generate the answer:

[0108] a p ijk =GenAns(q p ijk U ijk Citation ijk )

[0109] The generation process comprehensively extracts relevant information about the question through a language model, ensuring that the answer is accurate, the content is derived from the references, the language is natural and standard, and that it is traceable and consistent with the context.

[0110] Step 303) Standardize the output of fine-grained question-and-answer pairs: The system formats all generated results and outputs a set of structured questions and answers.

[0111] QA ijk ={(q1 ijk ,a 1 ijk ),(q 2 ijk ,a 2 ijk ),...}

[0112] Each question and answer includes an accompanying reference summary (Citation). ijk This forms a complete triple (q, a, Citation). Finally, it is exported in JSON or CSV format, forming a high-quality, fine-grained question-answering dataset with a unified structure, clear source, and direct usability for fine-tuning instructions.

[0113] In the above technical solution, step 4) specifically includes:

[0114] Step 401) Low-quality question-answer pair screening and confidence assessment: The system first performs quality checks on each question-answer pair generated in step 3), mainly including automatic evaluation of dimensions such as content completeness, answer accuracy, and language standardization. It also combines the output probability of the language model or a confidence scoring mechanism to determine its credibility. If the confidence of a question-answer pair is lower than the threshold θ, or if there are logical contradictions, incoherent language, or unfounded answers, it is marked as a low-quality sample and removed.

[0115] (q,a,Citation)→discard,ifConf(a)<θ

[0116] This process ensures that only semantically clear, structurally sound, and supported question-and-answer samples are retained for subsequent stages.

[0117] Step 402) Removing duplicate questions and answers and redundant content: In the entire question and answer data, the system uses semantic similarity calculation and content hash matching to identify duplicate or similar question and answer items. Duplicate content is merged or high-confidence versions are retained, while the rest are removed. The semantic duplicate determination function can be expressed as:

[0118]

[0119] Where δ is the similarity threshold. This process can significantly improve the uniqueness and information density of question-and-answer samples, and avoid training data bias.

[0120] Step 403) Final Data Validation and Format Standardization: After screening and deduplication, the system performs unified structure validation on the retained question-and-answer samples to ensure that fields are complete, references are valid, and the character set and encoding format conform to JSON / CSV specifications; it also checks for format anomalies, empty fields, or illegal symbols. All data that passes validation is stored as a standardized sample set.

[0121] D QA ={(q i ,a i Citation i )}

[0122] This step ensures that the final output data has a stable structure, is highly readable during training, and can be directly used as input for high-quality instruction fine-tuning.

[0123] Example 2

[0124] like Figure 1 As shown, the decomposition, decoupling, and fine-grained annotation-based long text generation dataset construction method of the present invention mainly includes the following steps:

[0125] Step 1: Prepare the data source

[0126] The original dataset comes from open-source knowledge repositories such as Wikipedia. The data miner will extract content that meets the requirements from these documents, specifically including:

[0127] 1. Topic

[0128] 2. Outline

[0129] 3. Paragraph content

[0130] This step first ensures the document has hierarchical potential and prepares it for subsequent structured decomposition. Redundant parts of the text should be removed to ensure a clear and operable hierarchical structure. Automated scripts (such as HTML tags, Markdown tags, etc.) are used for layered extraction to ensure standardized document structure for easier subsequent processing.

[0131] Table 1. Example of a term index

[0132] Index field Field Explanation Title Article Title Topic The overall theme of the document, as the highest level heading. Outline Structured headings for each chapter Paragraph Detailed paragraph content under each chapter Reference Detailed information about cited references, including title, URL, snippet, etc.

[0133] Step 2: Citation retrieval and abstract extraction

[0134] Within each content unit, the retrieval system automatically searches for relevant authoritative references. The system searches based on the content, selects abstracts of relevant literature, generates content summary pairs, and records citation information. During this process, multiple retrieval tools (such as Google Serper API, Bing API, etc.) participate in multiple rounds of filtering, using similarity algorithms to select the most relevant literature, ensuring the authority and accuracy of the citation information.

[0135] Table 2 Examples of Citation Retrieval and Abstract Extraction

[0136]

[0137] Step 3: Fine-grained question and answer annotation

[0138] Based on the information from steps 1 and 2, the question-and-answer annotation tool designs question-and-answer pairs according to the document content. Each question-and-answer pair not only covers multiple knowledge points such as definitions, principles, and applications, but also ensures the breadth and depth of the content. Question-and-answer generation is not limited to simple questions and answers, but also includes in-depth analysis of each paragraph or knowledge point. The answer to each question should be detailed, ensuring coverage of all aspects from definition to application.

[0139] Table 3. Sample Question-Answer Pair Data

[0140]

[0141] Step 4: Data Cleaning and Quality Control

[0142] In this step, all generated question-and-answer data undergoes rigorous quality screening from multiple perspectives. This includes: 1. Content consistency check; 2. Citation matching assessment; and question-and-answer quality scoring.

[0143] Data cleaning is not merely about removing low-quality samples; it's also about ensuring that the generated content meets academic standards and can efficiently support the generation of training data. Quality review should cover content consistency, accuracy of citations, and whether it meets the expected depth and breadth.

[0144] Figure 2 This is a flowchart of the method of the present invention.

[0145] Example 3

[0146] Embodiment 3 of the present invention provides a system for constructing instruction fine-tuning datasets for generating long texts from large models, based on the method of Embodiment 1, including:

[0147] The hierarchical decomposition module is used to hierarchically decompose documents from semi-structured or structured open-source knowledge bases. It extracts structured content according to three levels: topic, outline, and paragraph, removes noise, and forms structured units.

[0148] The retrieval module is used to construct a retrieval formula for each structured unit, retrieve several document entries from the entire network, a specified knowledge base and / or retrieval system, select the document entries with the highest similarity, and construct the corresponding reference abstract;

[0149] The question-answer pair generation module is used to generate fine-grained question-answer pairs based on each structured unit and its corresponding reference abstract;

[0150] The dataset building module is used to perform quality filtering and confidence evaluation on all fine-grained question-answer pairs to form a structured question-answer dataset.

[0151] It is worth noting that in the embodiments of the above system, the modules included are divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional module are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0152] Innovation points:

[0153] This invention proposes a method for constructing a fine-tuned dataset for long text generation instructions that integrates structured task decomposition and multi-functional module collaboration, possessing multi-level structural modeling capabilities and citation information tracking capabilities. This method ensures the structural integrity and content accuracy of the generated data by dividing it into sub-tasks such as outline construction, paragraph generation, citation retrieval, and quality control. Possible alternatives are methods based on direct extraction or rewriting of large models, which lack task controllability and information traceability mechanisms, and are prone to structural confusion and factual errors.

[0154] This invention designs a question-answering generation mechanism that combines outline-driven approach with fine-grained citation embedding. It supports the generation of highly readable and verifiable training samples from multiple dimensions, including structural hierarchy, key content points, and citation relationships. Compared to question-answering construction methods based solely on the original text, the mechanism proposed in this invention significantly enhances the organization and knowledge traceability of the generated content, improving the model's ability to model complex structural content.

[0155] This invention establishes a multi-stage collaborative process consisting of outline generation, citation retrieval, paragraph question-and-answer generation, and quality filtering. It integrates language model generation capabilities with rule-based processing mechanisms, enabling the continuous and stable production of high-quality datasets with clear structure and explicit citations. A possible alternative is to use end-to-end imperative generation methods, but these methods often lack quality control and structural consistency guarantees, making it difficult to meet the stringent requirements of long text generation tasks regarding training data.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for constructing an instruction fine-tuning dataset for generating long text in large models, comprising: Step 1: Decompose the documents of semi-structured or structured open-source knowledge bases hierarchically, extract structured content according to three levels: topic, outline, and paragraph, remove noise, and form structured units; Step 2: For each structured unit, construct a search query, retrieve several document entries from the entire network, the specified knowledge base and / or search system, select the document entries with the highest similarity, and construct the corresponding reference abstract; Step 3: Generate fine-grained question-answer pairs based on each structured unit and its corresponding reference abstract; Step 4: Perform quality filtering and confidence evaluation on all fine-grained question-answer pairs to form a structured question-answer dataset; Step 1 includes: Automated scripts are used to extract data from documents, and the extraction rules are as follows: for: ； in, Indicates the first i This document Indicate the theme, Indicates the first i The j-th outline title of the document, Indicates the first i The content of the kth paragraph under the jth outline in this document; Delete blank paragraphs and incorrectly formatted paragraphs; Merge short paragraphs that share a consistent theme; Standardize hierarchical relationships to ensure a reasonable number of outlines and paragraphs under the same theme; Will Decompose into triples: ; Repeat the above steps until you obtain triples for all documents, thus forming a set of structured units. : 。 2. The method for constructing a fine-tuning dataset for generating long text in large models according to claim 1, characterized in that, The automated script includes: regular expression matching, an HTML / XML parser, and a structure extraction algorithm based on heading level changes.

3. The method for constructing a fine-tuning dataset for generating long texts in large models according to claim 1, characterized in that, Step 2 includes: For sets Each structured unit , on the topic Subtopics With paragraph content Keyword extraction, stop word removal, and phrase recombination are performed to construct the search query. : ； Through keyword enhancement, domain limitation, and phrase recognition, Enhance; Enhanced search query Perform a search operation across the entire internet, a specified knowledge base, and / or a search system to obtain a set of search results. : ； in, Indicates the first m Each retrieved document entry includes its title, abstract, and excerpt from the text. Calculate according to the following formula and Each document entry similarity From these, the document entries with the highest similarity are selected; ； in, Indicates cosine similarity; For the selected document entries, record the document titles. ,author Publication year Source and fragment text , forming structured units References Abstract : 。 4. The method for constructing a fine-tuning dataset for generating long text in large models according to claim 1, characterized in that, Step 3 includes: For each structured content unit and its corresponding reference abstracts The system pre-sets a multi-dimensional problem template set T={ , ,..… }, where each template Indicates a type of question; template set and Semantic combination automatically generates a set of candidate questions. ={ ), ,…}; For each candidate problem Based on structured content Reference Abstract The GenAns module, an instruction-based question-and-answer generation module, is invoked to generate answers. : =GenAns( ); All generated results are formatted to output a structured question-and-answer set. = ; Each question and answer will be accompanied by a summary of references. This forms fine-grained question-and-answer pairs.

5. The method for constructing a fine-tuning dataset for generating long text in large models according to claim 1, characterized in that, Step 4 includes: Each fine-grained question-and-answer pair generated in step 3 undergoes quality filtering, including content completeness, answer accuracy, and language standardization. Confidence assessment is performed by combining the output probability or confidence scoring mechanism of the language model; if the confidence level is below a threshold... If there are logical contradictions, language barriers, or unfounded answers, the corresponding fine-grained questions and answers will be marked as low-quality samples and removed. Using semantic similarity calculation and content hash matching, duplicate or similar question and answer items are identified. Duplicate content is merged or high-confidence versions are retained, while the rest are removed. The retained fine-grained question-and-answer data undergoes unified structure validation to ensure field completeness, valid citations, and that the content character set and encoding format conform to JSON / CSV standards. Simultaneously, it checks for format anomalies, empty fields, and illegal symbols. All validated data is then stored as a standardized sample set. ={( , , )}.

6. A system for constructing a fine-tuning dataset for generating long texts from large models, based on the method described in claim 1, characterized in that, include: The hierarchical decomposition module is used to hierarchically decompose documents from semi-structured or structured open-source knowledge bases. It extracts structured content according to three levels: topic, outline, and paragraph, removes noise, and forms structured units. The retrieval module is used to construct a retrieval formula for each structured unit, retrieve several document entries from the entire network, a specified knowledge base and / or retrieval system, select the document entries with the highest similarity, and construct the corresponding reference abstract; The question-answer pair generation module is used to generate fine-grained question-answer pairs based on each structured unit and its corresponding reference abstract; and The dataset building module is used to perform quality filtering and confidence evaluation on all fine-grained question-answer pairs to form a structured question-answer dataset.

Images