Large-scale diversified synthetic data generation method oriented to large language model pre-training

Abstract

The invention discloses a large-scale diversified synthetic data generation method oriented to large language model pre-training. The method comprises the steps that a dynamic topic network is automatically constructed from multi-source data through unsupervised hierarchical density clustering to serve as a knowledge base; based on the network, a mass of differential text generation instructions are automatically generated through dynamic combination of parameterized prompt templates, and a prompt generation strategy is continuously optimized by using a data-driven experimental optimization loop; when the large language model carries out content synthesis according to a text generation instruction, a generation process is constrained through factual guidance enhanced by integrated retrieval and a structure and quality synchronous guidance mechanism; and performing multi-dimensional quantitative evaluation on the generated preliminarily screened synthetic data after semantic-level deduplication and reference decontamination, forming an optimization instruction according to an evaluation result, and feeding back the optimization instruction to each upstream link to realize closed-loop self-optimization. According to the method, full-process automation from knowledge discovery to evaluation optimization is realized, and large-scale, high-quality and diversified synthetic data can be systematically generated.

Description

Technical Field

[0001] This invention relates to the field of large language model training data synthesis technology, specifically to a method for generating large-scale, diverse synthetic data for large language model pre-training. Background Technology

[0002] In recent years, with the rapid development of Large Language Models (LLMs), obtaining high-quality, large-scale training data has become a key bottleneck restricting further improvements in model performance. Utilizing synthetic data generated by the LLM itself, due to its good scalability and controllability, has gradually become an important way to supplement and enrich training data and alleviate the data scarcity problem. Currently, a large number of instruction fine-tuning synthetic datasets (such as Alpaca and Self-Instruct) based on advanced models (such as the GPT series and Llama) have emerged. The related technical processes are becoming increasingly mature and have achieved significant results in improving the model's instruction following and dialogue capabilities.

[0003] However, existing mainstream synthetic data generation methods are mainly geared towards the supervised fine-tuning or alignment stage of the model, and their data scale is usually limited to thousands to hundreds of thousands of samples, which is far from meeting the massive data requirements of billions or even trillions of word units needed to pre-train a large language model from scratch. Extending the application scenarios of synthetic data to the pre-training scale faces the following unique technical challenges: (1) Insufficient data diversity: Existing methods mostly use a limited number of similar prompt templates, resulting in high repetition of generated content in terms of language style, narrative structure and topic angle, narrow coverage, which seriously restricts the generalization ability of pre-trained models; (2) Limited topic and knowledge coverage: Usually relying on manually selected or limited topic sets, it is difficult to achieve systematic coverage of multiple fields and long tail parts of human knowledge, resulting in blind spots in the knowledge scope of the model; (3) Poor scalability of prompt engineering: Relying on manually writing high-quality prompts is inefficient and it is difficult to automatically generate millions or even hundreds of millions of diverse and high-quality text generation instructions, which restricts the explosive growth of data scale; (4) Difficulty in quality and reliability control: The generation model itself may produce factual errors, logical contradictions or content illusions, and there is a lack of efficient and automated mechanisms to ensure the overall credibility, factual accuracy and internal consistency of massive synthetic data.

[0004] While existing research has demonstrated that pre-training models using synthetic data can achieve competitive performance, the specific technical details and complete synthetic datasets are often not fully disclosed, making it difficult to reproduce, verify, and iterate. Current research largely focuses on small-scale instruction generation, while systematic methods for generating diverse, high-quality synthetic data required for large-scale pre-training remain scarce. Existing technical approaches, such as seed rewriting, topic-guided methods, or dataset reconstruction, all have significant limitations in addressing the data requirements of pre-training. Specifically, large-scale synthetic data generation methods for pre-training large language models suffer from the following main drawbacks: (1) The knowledge system is static and rigid, making it difficult to achieve automated expansion and balanced coverage: Existing solutions mostly rely on static topic lists that are predefined manually or obtained through simple clustering, which cannot automatically discover, organize and expand the knowledge structure from massive and dynamically growing network data. At the same time, the automation level for associating and selecting high-quality "seed" texts for each topic is low, resulting in uneven knowledge coverage, making it difficult to effectively incorporate emerging fields and long-tail knowledge, and failing to provide a solid and dynamic knowledge base for large-scale generation.

[0005] (2) The generation of prompts relies on manual templates, making it difficult to ensure content diversity on a large scale: The existing prompt generation solutions are mainly based on manually written, limited fixed templates, or can only perform simple dimension replacements. There is a lack of a systematic method for parameterizing and automating the synthesis of massive amounts of differentiated instructions, resulting in low generation efficiency, making it difficult to support the data production needs of hundreds of millions of data points, and even more difficult to carry out systematic and controllable content diversity design in terms of audience, style, logical depth and other dimensions.

[0006] (3) The generation process lacks proactive constraints, resulting in lagging quality control and high costs: Existing methods mainly rely on post-generation filtering and manual spot checks, which is a "post-processing" mode. It is impossible to simultaneously complete fact verification, logical error correction, and information density assurance during the content generation process. This design, which lacks process constraints and real-time calibration mechanisms, leads to high data cleaning costs and uncontrollable overall quality risks.

[0007] (4) The system process is in an open-loop state and lacks self-optimization capability based on results: Existing solutions usually regard each stage of data generation (such as topic mining, prompt generation, content generation, and quality filtering) as an open-loop process. There is a lack of automated evaluation and feedback adjustment mechanism based on the final synthetic data effect (such as downstream model training performance and multi-dimensional quality indicators) between each stage. The system cannot dynamically optimize upstream strategies and parameters according to the objective performance of the output results, which makes it difficult to achieve self-iteration and adaptive improvement in the continuous large-scale production process, resulting in insufficient intelligence and engineering level.

[0008] Therefore, there is an urgent need for a new and systematic technical solution to achieve the automated generation of large-scale, high-quality, and highly diverse synthetic data, so as to provide a reliable data foundation for the pre-training of the next generation of large language models. Summary of the Invention

[0009] The purpose of this invention is to address the core problems existing in the prior art, such as static and rigid knowledge coverage, reliance on manual prompt generation, passive and lagging quality control, and lack of self-optimization capabilities in the process, and to provide a method for generating large-scale and diverse synthetic data for pre-training of large language models.

[0010] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a method for generating large-scale, diverse synthetic data for pre-training large language models, comprising the following steps: S1. Based on the original multi-source data, a structured dynamic topic network is generated through multi-source data vectorization representation, hierarchical density clustering, and topic knowledge association. S2. Based on a dynamic topic network, multi-dimensional parameters are decoupled and dynamically combined through parameterized prompt templates to automatically generate text generation instructions. The prompt generation strategy is continuously adjusted and optimized through a data-driven experimental optimization loop. S3, when the large language model synthesizes content according to the text generation instructions, it generates preliminary synthesized data through a constraint generation process that integrates factual guidance and structure and quality synchronous guidance mechanisms. S4. Perform semantic-level deduplication and baseline decontamination on the initial screening synthetic data to obtain a clean synthetic dataset. S5. Perform multi-dimensional quantitative evaluation on the purified synthetic dataset, and generate optimization instructions based on the evaluation results, feeding them back to at least one of steps S1-S4 to adjust the corresponding parameters or strategies.

[0011] Furthermore, step S1 specifically includes the following sub-steps: S11. Acquire raw text data from multiple independent data sources in parallel, and use a pre-trained text embedding model to encode the title and core content summary of each document into a dense semantic vector to convert unstructured text into a set of structured semantic vectors. S12. Based on the semantic vector set, unsupervised clustering is performed using a hierarchical density clustering algorithm; by analyzing the semantic similarity and hierarchical inclusion relationship between each topic cluster in the clustering results, a dynamic hierarchical topic tree is constructed. S13. For each topic cluster in the dynamic hierarchical topic tree, obtain the corresponding topic name, topic description, knowledge point list and core seed text based on its original document; based on the topic description and knowledge point list of each topic cluster, obtain the extended seed samples of the topic cluster through efficient vector similarity retrieval; finally, output a structured dynamic topic network.

[0012] Furthermore, the data sources include: deduplicated, cleaned, and quality-filtered web crawled text, open-licensed academic paper abstracts, books, technical documents, and encyclopedia entries.

[0013] Furthermore, step S13 specifically includes: For each topic cluster in the dynamic hierarchical topic tree, a preset number of representative text fragments are extracted from its original documents based on the maximum marginal relevance criterion, serving as the core seed samples for that topic cluster. Based on the core seed samples, the topic name and topic description are automatically generated, and a list of knowledge points corresponding to that topic cluster is generated through keyword extraction and concept summarization. The topic description and knowledge point list of each topic cluster are encoded into a comprehensive topic semantic vector through a pre-trained text embedding model; the original text corpus is divided into blocks and encoded into the same vector space to obtain text block vectors; the cosine similarity between each comprehensive topic semantic vector and each text block vector is calculated through efficient vector similarity retrieval, and the top K text blocks with the highest cosine similarity are recalled as extended seed samples for the topic cluster. The final output is a structured dynamic topic network, where each node contains a topic name, topic description, list of knowledge points, core seed samples, and extended seed samples.

[0014] Furthermore, step S2 specifically includes the following sub-steps: S21. Quality pre-filtering: The original core seed text and extended seed text in the dynamic topic network are scored in real time by a pre-trained quality assessment model, and seed texts with scores higher than a preset threshold are selected as input materials for subsequent instruction generation. S22. Construct a prompt template containing multiple structured parameter slots: The parameter slots shall include at least the following: knowledge topic slot, knowledge element slot, content seed slot, audience characteristic slot, content depth slot, genre style slot, and structure requirement slot. S23. Layered combination to generate differentiated instructions: A layered combination mechanism is adopted. Based on the prompt template and according to the preset prompt generation strategy, content elements are selected from the input materials and the text is automatically synthesized to generate instructions. S24. Data-driven generation strategy optimization: Sample a representative set of topics from the dynamic topic network as an experimental set; adjust the sampling weights and combination rules of key dimensions in the prompt template, and generate multiple synthetic data subsets with different style configurations based on the same experimental set; use each synthetic data subset to quickly fine-tune the benchmark language model, and evaluate the performance of each fine-tuned benchmark language model in parallel on the downstream task evaluation set; identify the prompt style combination pattern that produces the best performance based on the evaluation results, and transform this pattern into the dynamic sampling weight distribution and combination rules of each dimension in the prompt template to update the prompt generation strategy.

[0015] Furthermore, step S23 specifically includes: The parameter slots are organized into at least two logical layers, including a content planning layer consisting of audience characteristics and content depth, and an expression form layer consisting of genre style and structural requirements; parameterized sub-templates are configured for each parameter option in each layer; according to the preset prompt generation strategy, sub-templates are dynamically selected from each layer and instantiated, filled, and spliced ​​with knowledge topics, knowledge elements, and content seeds to generate text generation instructions.

[0016] Furthermore, step S3 specifically includes the following sub-steps: S31. Factual Guidance: When the large language model generates text according to the text generation instructions, it analyzes the generated text fragments in real time. When it identifies text containing potential factual descriptions, data citations, or explanations of technical terms, it automatically triggers a retrieval process: using the currently generated text as a query vector, it retrieves the most relevant evidence fragments from the authoritative knowledge vector library; and dynamically inserts the evidence fragments into the current generation context of the large language model in a structured form to guide the continuation and elaboration of the text. S32. Synchronous guidance of structure and quality: Before the generation begins, a structured outline is provided to the large language model according to the task type to guide the generation of text that conforms to the target logical structure; during the generation process, a real-time quality evaluator is run synchronously to score the quality of the generated text in real time; if the score is lower than the preset threshold, the generation of the current text is stopped or rewriting is triggered, otherwise the generation process continues. S33. Distributed Constraint Generation: A distributed asynchronous task queue is used for efficient processing of generation tasks. The text generation instructions are placed in the task queue. A group of generation worker nodes, which integrate retrieval enhancement generation and quality assessment components, pull tasks in parallel and execute a constraint generation process that integrates factual guidance, structure and quality synchronization guidance mechanisms. The generated text is written to a distributed file system. The task scheduler performs overall load balancing, fault tolerance processing and progress monitoring to finally obtain the initial screening and synthesized data.

[0017] Furthermore, step S4 specifically includes the following sub-steps: S41. Semantic Deduplication: Each initial screening composite data is represented as a set of words or continuous word sequences, and multiple hash functions are used to generate the minimum hash signature; the initial screening composite data with similar minimum hash signatures are divided into the same hash bucket using the Locality Sensitive Hashing algorithm; the Jaccard similarity between any two initial screening composite data in each hash bucket is calculated, and for initial screening composite data pairs with a Jaccard similarity exceeding a preset threshold, only one of them is retained; S42. Baseline Desaturation: The initial screening synthetic data after semantic-level deduplication is compared with at least one public evaluation set, wherein the public evaluation set contains evaluation questions and corresponding standard answers; a combination of substring matching and fuzzy hash comparison is used to identify and remove samples in the initial screening synthetic data that have content duplication with the evaluation questions or the standard answers, thereby obtaining a purified synthetic dataset.

[0018] Furthermore, step S5 specifically includes the following sub-steps: S51. Conduct a multi-dimensional quantitative evaluation of the purified synthetic dataset, wherein the multi-dimensional quantitative evaluation includes knowledge coverage, content diversity, comprehensive quality distribution and factual consistency sampling; and generate a structured diagnostic report based on the evaluation results; S52. Automatically parse the structured diagnostic report into optimization instructions and feed them back to at least one of the steps S1-S4 to dynamically adjust the corresponding parameters or strategies.

[0019] A second aspect of this invention provides an apparatus for implementing the above-described method for generating large-scale, diverse synthetic data for pre-training large language models, comprising: The dynamic topic network construction module is used to generate a dynamic topic network based on original multi-source data through multi-source data vectorization representation, hierarchical density clustering, and topic knowledge association. The network includes topic names, topic descriptions, knowledge point lists, and associated core seed samples and extended seed samples. The automated prompt generation and optimization module is used to automatically generate text generation instructions based on a dynamic topic network, by decoupling and dynamically combining multi-dimensional parameters through parameterized prompt templates, and continuously adjusting and optimizing the prompt generation strategy through a data-driven experimental optimization loop. The guided content generation module is used to generate initial synthesized data by integrating factual guidance and structure and quality synchronization guidance mechanisms when the large language model synthesizes content according to text generation instructions. The synthetic data purification module is used to perform semantic-level deduplication and baseline decontamination on the initial screening synthetic data to obtain a purified synthetic dataset. The multi-dimensional evaluation and closed-loop control module is used to perform multi-dimensional quantitative evaluation of the purified synthetic dataset, and generate optimization instructions based on the evaluation results. These instructions are then fed back to at least one of the following modules: dynamic topic network construction module, automated prompt generation and optimization module, guided content generation module, and synthetic data purification module, to adjust the corresponding parameters or strategies.

[0020] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention designs an automated synthetic data generation architecture that integrates dynamic knowledge mining, intelligent instruction synthesis, process-guided generation and closed-loop optimization. Through the collaboration and data integration of five core functional modules, this architecture realizes the fully automated production process from raw data input to high-quality synthetic data output, and forms a self-iterative optimization closed loop based on multi-dimensional effect evaluation. It systematically solves the problems of insufficient data diversity, limited knowledge coverage, uncontrollable quality and unsystematic process in large-scale pre-training.

[0021] (2) This invention addresses the problem that existing technologies rely on static topic lists, resulting in incomplete coverage and difficulty in incorporating emerging and long-tail knowledge. It automatically constructs a dynamic knowledge system based on unsupervised hierarchical clustering and precise association of dual-path seeds. By using unsupervised hierarchical density clustering algorithm to automatically mine and construct a hierarchical topic network from multi-source data, and innovatively implements a dual mechanism of core seed extraction and semantic retrieval-based extended seed precise association for each topic node, it realizes the dynamic discovery and expansion of knowledge structure and the large-scale and balanced supply of high-quality training materials, and provides high-quality training materials for large-scale association of each topic, thereby ensuring the breadth, balance and timeliness of the knowledge coverage of synthetic data.

[0022] (3) This invention addresses the problems of existing technologies relying on manual templates, limited diversity of prompts, and limited generation efficiency. It is based on multi-dimensional parameterized template combination and data-driven strategy to automatically generate and optimize intelligent prompts. By constructing a prompt template library that supports flexible combination of multi-dimensional parameters, a hierarchical combination mechanism is adopted to automatically synthesize massive differentiated text generation instructions. Through the embedded lightweight experimental optimization loop, the prompt generation strategy is adaptively adjusted according to the training effect of downstream models. This realizes the leap from manual experience to automated and intelligent decision-making in prompt engineering, and realizes the paradigm shift from fixed rules to data-driven and self-optimized generation, breaking through the diversity and scale bottleneck of large-scale data production.

[0023] (4) This invention addresses the problem that existing technologies often filter data after generation and that factual errors and logical contradictions are difficult to prevent. By integrating proactive process constraints with a closed-loop control mechanism based on evaluation feedback, this invention introduces enhanced factual calibration, structured logical guidance, and real-time quality assessment during the content generation process, thereby bringing quality control forward and embedding key quality assurance measures into the source of content generation. At the same time, a multi-dimensional quantitative evaluation system is established, and the evaluation results are automatically converted into optimization instructions for each upstream link, driving the entire generation device to continuously self-adjust parameters and strategies, ensuring long-term quality and system adaptability in large-scale production. This proactive process constraint mechanism significantly improves the factual accuracy, logical coherence, and information density of the synthesized data, and greatly reduces the cost of subsequent cleaning, realizing the transformation of quality control from passive processing to proactive process constraints.

[0024] (5) In view of the problems of fragmented processes and lack of effect-based self-optimization in existing technologies, this invention constructs a complete generation-evaluation-iteration data-driven closed loop by integrating multi-dimensional quantitative evaluation and closed-loop control modules. It can automatically transform the evaluation results of the output data into specific optimization instructions for each upstream link, drive the continuous optimization of strategies such as topic mining, prompt generation, and process constraints, and enable the entire system to have self-diagnosis and adaptive evolution capabilities, ensuring long-term quality stability and efficiency improvement in large-scale production. Attached Figure Description

[0025] Figure 1 This is a flowchart of a method for generating large-scale, diverse synthetic data for pre-training large language models; Figure 2 This is the architecture flowchart of the dynamic topic network building module; Figure 3 This is the architecture flowchart of the automated prompt generation and optimization module; Figure 4 This is the architecture flowchart of the guided content generation module. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0029] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0030] This invention provides a complete, automated, and closed-loop-optimizable method for generating large-scale, diverse synthetic data for pre-training large language models, systematically addressing the shortcomings of existing technologies to generate large-scale, high-quality, and diverse synthetic data. The core of this synthetic data generation method lies in its data-driven, closed-loop-optimized automated synthesis device. This method achieves this through the synergy of five core functions: dynamic topic network construction, automated prompt generation and optimization, guided content generation, synthetic data cleansing, and multi-dimensional evaluation and closed-loop control. This forms a complete automated process from raw data to high-quality synthetic data, and continuous self-optimization is achieved through evaluation feedback. This systematically solves the problems faced by existing technologies in large-scale scenarios, such as diversity bottlenecks, limited knowledge coverage, uncontrollable quality, and unsystematic processes.

[0031] like Figure 1 As shown, the synthetic data generation method specifically includes the following steps: S1. Constructing a Dynamic Topic Network: From massive, unstructured raw multi-source data, through multi-source data vectorization, hierarchical density clustering, and topic knowledge association, a dynamically evolving, hierarchical, structured topic network is automatically mined and organized, containing topic names, topic descriptions, knowledge point lists, and associated core seed samples and large-scale expanded seed samples. Figure 2 As shown, this dynamic topic network can systematically represent topic knowledge and its relationships, forming the knowledge base and raw material library for the entire synthetic data generation process.

[0032] S11. Multi-source data vectorization processing: Obtain raw text data from multiple independent data sources in parallel, including but not limited to: web crawled text that has been deduplicated, cleaned and quality filtered (such as RefinedWeb), open-licensed academic paper abstracts, books, technical documents and encyclopedia entries, etc.; use pre-trained text embedding models, such as the BGE-Large model, to encode the title and core content summary of each document into high-dimensional dense semantic vectors, so as to transform unstructured text into a computable set of structured semantic vectors.

[0033] S12. Adaptive Construction of Hierarchical Topic System: Based on a set of semantic vectors, unsupervised clustering is performed using a hierarchical density clustering algorithm (such as HDBSCAN). This algorithm can automatically identify multi-granularity topic clusters based on the spatial density distribution of semantic vectors without pre-setting the number of topics. By analyzing the semantic similarity and hierarchical inclusion relationships among topic clusters in the clustering results, a dynamic hierarchical topic tree is automatically constructed. In this tree structure, the root node represents a broad macro-level subject area, the intermediate nodes represent subdivided topics, and the leaf nodes represent specific sub-topics, thus forming a dynamic topic network that can represent the internal connections and evolutionary paths of knowledge.

[0034] It should be noted that hierarchical density clustering algorithms can generate multi-granularity topic clusters, the granularity of which determines its depth in the hierarchical topic tree. For example, the coarse-grained layer might contain the topic clusters "Computer Science" and "Physics"; the "Computer Science" topic cluster in the medium-granularity layer can be further subdivided into the "Artificial Intelligence" and "System Software" topic clusters; "Artificial Intelligence" can be further subdivided into "Machine Learning" and "Computer Vision"; "System Software" can be further subdivided into "Operating System" and "Database"; and "Machine Learning" in the finest-grained layer can be further subdivided into "Deep Learning" and "Reinforcement Learning". The system automatically constructs a dynamic hierarchical structure by analyzing the semantic similarity and inclusion relationships between these topic clusters.

[0035] Furthermore, the semantic similarity between topic clusters can be calculated using any of the following methods: First, based on cluster centroid vectors, the average of the document vectors of each cluster is calculated as the centroid, and then the cosine similarity between the centroids is calculated. Second, based on inter-cluster documents, representative documents are sampled from each cluster, and the average or maximum similarity of all cross-cluster document pairs is calculated as the inter-cluster similarity.

[0036] S13. Topic Knowledge Generation and Association: This step aims to obtain the corresponding topic name, topic description, knowledge point list, and core seed text for each topic node (topic cluster) in the dynamic hierarchical topic tree based on its original document, establishing a complete structured knowledge description. Based on deep semantic association, it accurately matches core and extended training seed samples. Specifically, based on the topic description and knowledge point list of each topic cluster, it obtains extended seed samples for that topic cluster through efficient vector similarity retrieval (such as the FAISS library), ultimately generating a structured dynamic topic network that serves as the knowledge base and raw material library. Specifically, the following sub-steps are executed: (a) Core Seed Sample Extraction and Topic Knowledge Generation: For each topic cluster in the dynamic hierarchical topic tree, firstly, based on the maximum marginal relevance criterion, a preset number (e.g., 100) of the most representative and information-dense text fragments are automatically extracted from the original document set with semantic association, serving as the core seed samples for that topic cluster. Subsequently, based on the core seed samples, a concise and accurate topic name and topic description are automatically generated using a large language model. Simultaneously, combining keyword extraction techniques (e.g., TF-IDF, TextRank) and the conceptual induction capabilities of the large language model, a list of knowledge points corresponding to that topic cluster is generated.

[0037] (b) Semantic Similarity-Based Extended Seed Sample Association: To further expand the training materials for each topic, the following precise association process is executed. First, the topic descriptions and knowledge point lists generated for each topic cluster are encoded into a unified comprehensive topic semantic vector using a pre-trained text embedding model (such as the BGE-Large model). Simultaneously, the massive original text corpus (such as deduplicated FineWeb data) is divided into blocks and encoded into the same vector space to obtain text block vectors, thus forming a set of text block vectors. Subsequently, through efficient vector similarity retrieval (such as the FAISS library), the cosine similarity between each comprehensive topic semantic vector and all text block vectors is calculated, and the top K (e.g., Top-1000) text blocks with the highest cosine similarity are recalled as extended seed samples for large-scale association of that topic cluster.

[0038] (c) Finally, this step outputs a structured dynamic topic network. This network uses a dynamic hierarchical topic tree as its skeleton, with each node containing a topic name, topic description, a list of knowledge points, core seed samples, and extended seed samples. This network constitutes the knowledge base and raw material library that can be directly accessed in subsequent synthetic data generation processes.

[0039] By employing the aforementioned multi-source data vectorization, adaptive hierarchical clustering, and two-stage semantic association (core extraction and extended retrieval), this invention achieves automated and structured construction of dynamic topic networks. It boasts significant advantages, including accurate topic discovery, comprehensive knowledge coverage, fine-grained association granularity, and strong overall scalability.

[0040] S2. Automated Prompt Generation and Optimization: Based on a dynamic topic network, multi-dimensional parameters are decoupled and dynamically combined through parameterized prompt templates to automatically generate diverse and executable text generation instructions. A lightweight, closed-loop, data-driven experimental optimization loop is used to evaluate and adaptively adjust the prompt generation strategy, thereby continuously improving the effectiveness of subsequent synthesized data for training large language models. Figure 3 As shown.

[0041] It should be understood that the automated generation and optimization of prompts aims to achieve automated synthesis of large-scale, high-quality, and diverse text generation instructions through closed-loop optimization using parameterized prompt templates and data-driven experimental optimization loops. This addresses the problems of traditional prompt engineering, such as reliance on manual labor, difficulty in scaling, and unstable results. By decoupling, parameterizing, and dynamically combining multi-dimensional parameters such as knowledge content, audience characteristics, presentation depth, genre style, and text structure, and introducing a lightweight data-driven experimental optimization loop, a leap from fixed rules to adaptive intelligent generation is achieved.

[0042] S21. Quality Pre-filtering: Before generating text instructions, a pre-trained quality assessment model (such as a pre-trained classifier based on fineweb-edu) scores the original core seed text and extended seed text in the dynamic topic network in real time. Seed text with scores higher than a preset threshold (e.g., 2.8 points) is selected as input material for subsequent instruction generation and proceeds to the next step, executing the subsequent prompt template filling and instruction generation process. Otherwise, the next original core seed text and extended seed text are scored in real time. This quality pre-filtering mechanism avoids low-quality content from entering the subsequent processing stage from the source, preventing low-quality input through the quality assessment model.

[0043] S22. Structured Multi-Dimensional Parameter Slot Design: Each prompt template contains a series of structured parameter slots to receive specific information from the upstream network, ensuring the accuracy and richness of the text generation instructions. Therefore, it is necessary to construct prompt templates containing multiple structured parameter slots.

[0044] The parameter slots include at least: ① a knowledge topic slot, used to receive the names and descriptions of topics and subtopics from the dynamic topic network, anchoring the core knowledge domain of the generated content; ② a knowledge element slot, used to receive a list of knowledge points from the dynamic topic network, guiding the professional boundaries and key coverage of content generation; ③ a content seed slot, used to receive key text fragments extracted from high-quality core seed text and large-scale expanded seed text, providing specific and reliable generation references and context for the large language model; ④ an audience characteristic slot, used to define the target audience, such as {university students}, {middle school students}, {primary school students}, etc., with each audience corresponding to different language complexity, case selection, and knowledge presentation requirements; and ⑤ a content depth slot, used to control the depth level of knowledge presentation, including emphasis. The content is divided into several categories: ⑥ **Genre Style Slot:** This slot defines the text genre, such as academic textbook style, narrative textbook style, technical blog style, and encyclopedia entry style. Each genre style corresponds to a specific text structure, narrative method, and expression standard. ⑦ **Structure Requirements Slot:** This slot defines a specific organizational structure framework, including {including positive and negative examples} (requiring both positive and negative examples for comparative analysis), {using a question-and-answer format to enhance interactivity and relevance}, and {problem-analysis-solution structure} (developing content in a logical order of problem description, cause analysis, and solution).

[0045] S23. Layered combination to generate differentiated instructions: Using a layered combination mechanism, based on prompt templates and according to preset prompt generation strategies, content elements are selected from the input materials to automatically synthesize text to generate instructions.

[0046] Specifically, firstly, the parameter slots are organized into at least two logical layers: a content planning layer composed of audience characteristics and content depth, and an expression form layer composed of genre style and structural requirements. The content planning layer, based on audience characteristics and content depth, determines the target audience and the depth of explanation for the generated content, while the expression form layer, based on genre style and structural requirements, defines the overall paradigm and micro-organizational logic of the text. Each layer provides multiple optional, parameterized sub-templates, configuring parameterized sub-templates for each parameter option within each layer. Based on this, a dynamic layered combination mechanism is run: according to a preset prompt generation strategy, sub-templates are dynamically selected from each layer and instantiated, filled, and spliced ​​with knowledge topics, knowledge elements, and content seeds, ultimately generating a complete, coherent, and executable text generation instruction, which can then be directly input into a large language model. Through this hierarchical combination mechanism, a single topic and seed text can automatically generate dozens or even hundreds of highly customized instruction variants (i.e. prompt words), such as "a popular science blog on quantum computing concepts for middle school students using a question-and-answer format" or "a frontier review report on deep learning for university students, including positive and negative examples," which fundamentally ensures the enormous diversity of synthetic data in terms of content planning and expression.

[0047] S24. Data-Driven Generation Strategy Optimization: To ensure that the prompt generation strategy continuously produces synthetic data most beneficial to the training of the large language model, a lightweight, closed-loop data-driven experimental optimization loop is introduced to achieve parameter adaptation and effect-oriented iterative optimization of the prompt generation strategy. Its core workflow is as follows: First, a small batch of representative topics is sampled from the dynamic topic network as an experimental set. Then, the sampling weights and combination rules of key dimensions (such as audience characteristics, content depth, genre style, and structural requirements) in the prompt template are systematically adjusted, generating multiple small-scale synthetic data subsets with different style configurations based on the same experimental set. Subsequently, each synthetic data subset is used to quickly fine-tune a small baseline language model, and the performance of each fine-tuned baseline language model is evaluated in parallel on a compact evaluation set covering multiple downstream tasks (such as question answering and reasoning). When evaluating the performance of each fine-tuned baseline language model, the comprehensive performance of multiple downstream tasks such as question answering and reasoning is used for evaluation, employing metrics such as accuracy and completion. Finally, based on the evaluation results, by analyzing the characteristic patterns of the best prompt generation strategy and considering the combination effect between various dimensions, the prompt style combination pattern that produces the best performance is identified, and this pattern is transformed into the dynamic sampling weight distribution (i.e., dynamic sampling probability distribution, used to guide subsequent generation) and combination rules of each dimension in the prompt template, so as to update the prompt generation strategy in step S23 and realize the continuous self-evolution of the prompt generation strategy.

[0048] It should be noted that sampling weights are numerical values ​​that control the probability of different parameter options being selected. Sampling weights enable the system to shift from uniform random generation to data-driven selection, dynamically adjusting the strategy based on experimental results and concentrating computational resources on the most effective combination of prompts. A small benchmark language model is selected as a proxy to quickly evaluate the effectiveness of the prompt generation strategy. After finding the optimal prompt generation strategy, these strategies are then used to generate high-quality synthetic data for training large language models.

[0049] This invention achieves high controllability and diversity in content planning, knowledge depth and expression form of synthetic data through dynamic combination of parameterized prompt templates and data-driven strategy optimization, and can adaptively generate data distributions that are more effective for model training.

[0050] S3. Generating Guided Content: When the large language model synthesizes content based on the text generation instructions from step S2, it constrains the generation process by integrating factual guidance and structure and quality synchronization guidance mechanisms, generating initial screening synthesis data, such as... Figure 4 As shown.

[0051] It should be understood that the core of this step lies in the pre-positioning and integration of the quality control mechanism into the generation process. By introducing constraints on the generation process, a shift from passive processing to proactive guidance is achieved, thereby ensuring the credibility, logic, and content quality of the synthesized data at the source of generation.

[0052] S31. Factual Guidance Mechanism: When the large language model generates text content according to text generation instructions, it analyzes the generated text content fragments in real time. When it identifies text content containing potential factual descriptions, data citations, or explanations of technical terms, it automatically triggers a retrieval process: using the currently generated text as the query vector, it retrieves the most relevant evidence fragments from an offline, verified authoritative knowledge vector library (such as a vector index built from Wikipedia or a textbook fragment library) (based on the cosine similarity between the query vector and the authoritative document vector). Subsequently, these evidence fragments are dynamically inserted into the current generation context of the large language model in a structured form (such as "Reference: [Evidence Text]") to guide the large language model to continue and elaborate on the subsequent text based on more reliable information sources (i.e., evidence fragments), thereby effectively suppressing factual errors and illusions.

[0053] It should be noted that real-time analysis of text content fragments is conducted by detecting verifiable objective statements, descriptions containing specific details (such as time, location, and data), or explicit or technical explanations. Triggering the retrieval enhancement process typically requires meeting the following conditions: ① Formal matching: The text conforms to a predefined factual expression pattern (such as containing specific keywords, numbers, or sentence structures); ② Semantic analysis: The content is objectively verifiable; ③ Confidence level: The confidence score of the analysis model exceeds a preset threshold (e.g., 0.7); ④ Contextual fit: It is suitable for fact-checking in the current generation stage and topic.

[0054] S32. Synchronous Guidance Mechanism for Structure and Quality: Implements a dual guidance strategy, including structured guidance and real-time quality assessment. On one hand, before generation begins, a clear structured outline (e.g., "problem-cause-solution") is provided to the large language model based on the task type to guide it in generating logically coherent and structurally complete text content that conforms to the target logical structure. On the other hand, during generation, a lightweight real-time quality evaluator runs synchronously. This evaluator scores the text content being generated by the large language model in real time based on preset rules (e.g., basic grammar checking, information density assessment) or a fine-tuned classification model. If the quality score is below a preset threshold, the generation of the current low-quality text content is stopped or a rewriting process is triggered; otherwise, the generation process continues. This mechanism can proactively identify and intercept logically chaotic or low-information-density outputs during the initial generation stage.

[0055] S33. Distributed Constraint Generation: To efficiently handle massive generation tasks, a distributed asynchronous task queue architecture is adopted. Text generation instructions from step S2 are placed in a task queue, from which multiple parallel generation worker nodes pull and process data. Each worker node integrates retrieval-enhanced generation and quality assessment components to execute a constraint generation process with factual guidance and structural and quality synchronization guidance mechanisms, and outputs the generated text to a distributed file system. A task scheduler performs overall load balancing, fault tolerance, and progress monitoring of the generation worker nodes to ensure the efficient and stable execution of large-scale, highly complex constraint generation tasks, ultimately producing preliminary synthetic data.

[0056] This step effectively addresses the challenge of preventing factual errors and logical inconsistencies in synthetic data by shifting quality control from a passive, post-processing approach to a proactive constraint on the generation process.

[0057] S4. Cleaning the Synthetic Data: Automated and systematic semantic-level deduplication and baseline decontamination are performed on the initial screening synthetic data to remove noise, duplication, and potential biases, eliminate redundancy and potential risks of evaluation data leakage, and obtain a cleaned synthetic dataset. This prepares a high-quality, high-purity synthetic dataset for subsequent accurate evaluation and model training.

[0058] S41. Semantic Deduplication: To avoid redundancy within the dataset and ensure information density, refined semantic deduplication is performed on all initial screening synthetic data. The specific process is as follows: First, each initial screening synthetic data is represented as a set of words or continuous word sequences (n-grams), and a minimum hash signature is generated for it using multiple hash functions. Then, efficient approximation algorithms such as Locality Sensitive Hashing (MinHash LSH) are used to group initial screening synthetic data with similar minimum hash signatures into the same hash bucket. Finally, the Jaccard similarity between any two initial screening synthetic data is calculated within each bucket. If the Jaccard similarity exceeds a preset threshold (e.g., 0.9), it is considered highly similar, and only one initial screening synthetic data is retained. This method can efficiently identify and remove highly similar or duplicate initial screening synthetic data from massive amounts of initial screening synthetic data.

[0059] S42. Benchmark Destaining: To ensure the fairness of downstream tasks evaluating large language models and prevent accidental leakage of evaluation questions from synthetic data leading to evaluation distortion, proactive benchmark destaining is performed on the initially screened synthetic data after semantic-level deduplication. Specifically, this includes: comparing the semantically deduplicated initially screened synthetic data with at least one mainstream public evaluation set (such as MMLU, C-Eval), where the public evaluation set contains evaluation questions and corresponding standard answers; using a combination of substring matching and fuzzy hash comparison to identify and remove samples in the initially screened synthetic data that have content duplication with evaluation questions or standard answers, ultimately obtaining a purified synthetic dataset. Further, the method combining substring matching and fuzzy hash comparison specifically includes: (1) Standardization of evaluation data: ① Collect the corresponding evaluation sets (such as MMLU, C-Eval, etc.); ② Extract the complete text of each question, including: question description, option content (for multiple choice questions), reference answer, problem-solving ideas, etc.; ③ Standardize all evaluation texts: unify the encoding format, unify the capitalization, and remove extra spaces and punctuation marks.

[0060] (2) Preprocessing of preliminary screening data: Perform the same standardization process on the preliminary screening data to ensure that its format is consistent with that of the evaluation data.

[0061] (3) Substring matching detection (fast filtering layer): Construct an evaluation text dictionary and use multi-pattern matching algorithms such as Aho-Corasick for fast scanning. If the synthesized text contains evaluation text substrings with a length exceeding a preset value (e.g., 20 characters), it is directly judged as a high-probability leak and deleted; if the substring length is within a certain range (e.g., 10-20 characters), proceed to the next step of precise verification.

[0062] (4) Fuzzy hash comparison (precise verification layer): The ssdeep or tlsp algorithm is used to calculate the fuzzy hash value for the evaluation text and the synthetic data text to be verified, and the similarity is compared. If the similarity exceeds the threshold (e.g., 80%), it is judged as a possible leak and is removed.

[0063] This invention effectively improves the diversity, information density, and security and fairness of purified synthetic datasets for model evaluation by semantic-level deduplication and benchmark decontamination.

[0064] S5. Multi-dimensional evaluation and closed-loop control: Perform automated and comprehensive multi-dimensional quantitative evaluation on the purified synthetic dataset, and generate optimization instructions based on the evaluation results. Feedback is sent to at least one step of steps S1-S4 to adjust the corresponding parameters or strategies, forming an intelligent closed loop that drives the system to continuously self-optimize.

[0065] It should be understood that this step is the decision-making center that constitutes the complete closed loop of "generation-purification-evaluation-iteration". It aims to systematically evaluate the purified synthetic data and generate optimization instructions based on the evaluation results, driving the continuous self-optimization of the entire process.

[0066] S51. Multi-dimensional quantitative assessment and diagnosis: Perform multi-dimensional quantitative assessment on the purified synthetic dataset, which includes at least the following indicators, and generate a structured diagnostic report based on the assessment results: ① Knowledge Coverage: Statistically measure the number and proportion of topic nodes in the dynamic topic network covered by the purified synthetic dataset to quantify the completeness and balance of its knowledge system.

[0067] ② Content Diversity: The richness of language styles and expression patterns in the dataset is quantified from both surface lexical and deep semantic levels by comprehensively calculating the perplexity differences of large language models based on n-grams and the average cosine distance between documents based on semantic embedding vectors (such as using a BGE model). The specific evaluation process is as follows: Lexical level (Perplexity difference analysis): First, each synthesized text is preprocessed, segmented into word sequences, and words not present in the vocabulary of the baseline model (n-gram language model) are replaced with special markers (such as "unknown words"). Next, the baseline n-gram language model is used to predict the probability of each word sequence word-by-word: for each word in the sequence, the model predicts the conditional probability of the current word based on the context of the preceding n-1 words. Then, the perplexity of the text is calculated based on the conditional probabilities of all words: the logarithm of the conditional probability of each word is taken, the average of these logarithmic values ​​is calculated, and the negative of this average is then exponentially calculated to obtain the final perplexity value. By analyzing the distribution (variance, range) of the perplexity of all texts in the dataset or its relative difference from the perplexity of the reference corpus, the diversity of texts in lexical usage and surface sentence structure is quantified.

[0068] Semantic Level (Mean Cosine Distance Analysis): First, a pre-trained semantic encoding model (such as BGE) is used to generate a semantic embedding vector for each text in the dataset. Then, the cosine distance between each pair of document vectors in the dataset is calculated, and their average value is taken. The larger this average cosine distance, the more dispersed the text distribution in the semantic space, and the richer the semantic topics and viewpoints it covers, thus quantifying the diversity of content from a deep semantic perspective.

[0069] By integrating the quantitative indicators from the two levels mentioned above, this method achieves a comprehensive and multi-dimensional assessment of the diversity of synthetic data content.

[0070] ③ Overall quality distribution: Use a dedicated synthetic data quality scorer (such as the fineweb-edu scorer) to score all documents in the cleaned synthetic dataset in batches, count the proportion of data in the high, medium and low quality ranges, and evaluate the overall quality level.

[0071] ④ Fact Consistency Sampling: Through random sampling, the samples are automatically verified using retrieval-enhanced fact-checking pipelines or methods such as FActScore (Factual Consistency Score), the average fact consistency score is calculated, and the overall fact accuracy level of the cleaned synthetic dataset is evaluated.

[0072] S52. Generation and Feedback of Closed-Loop Optimization Instructions: The structured diagnostic report generated in step S51 is automatically parsed into specific and operable optimization instructions for each upstream step, and then fed back to at least one of steps S1-S4. The corresponding parameters or strategies are dynamically adjusted to drive the continuous iteration and closed-loop optimization of the entire synthetic data generation process. This is the core step for realizing the evolution of the entire synthetic data generation process.

[0073] For example, if the evaluation finds that the data quality score for a certain genre (such as "critical discussion") under a specific topic (such as "machine learning") is low, the system will automatically generate instructions to feed back to step S2, dynamically reducing the weight of this (topic, genre) combination in parameter sampling, or associating it with more detailed structural constraints. If the "topic coverage" evaluation shows insufficient coverage of certain emerging fields (such as "quantum computing"), the feedback will guide step S1 to strengthen the mining of relevant data sources or adjust clustering parameters. If the "fact consistency" score is poor, the feedback will be fed back to step S3 to increase the trigger frequency of its retrieval enhancement calibration, or prompt step S4 to introduce additional factual filtering rules.

[0074] For example, if the evaluation finds that the data quality score for a certain genre (such as "critical discussion") under a specific topic (such as "machine learning") is low, the system will automatically generate instructions and feed them back to step S2 to dynamically reduce the weight of this (topic, genre) combination in parameter sampling, or associate it with more detailed structural constraints. If the "Topic Coverage" report shows insufficient coverage of certain emerging fields (such as "quantum computing"), the feedback will guide step S1 to strengthen the mining and inclusion of relevant data sources, or adjust clustering parameters to identify more granular emerging topic clusters. If the "Fact Consistency" score is poor, the feedback will be fed back to step S3 to increase the trigger frequency of its retrieval enhancement calibration, and step S4 will be prompted whether additional factual filtering rules need to be introduced.

[0075] This invention forms a data-driven closed-loop iterative mechanism through multi-dimensional quantitative evaluation and diagnosis, as well as closed-loop optimization instruction generation and feedback. Based on the objective evaluation results of the final output data, it can continuously perform automated diagnosis and dynamic optimization of the entire process of synthetic data generation.

[0076] In summary, this invention addresses the problems of static and rigid knowledge systems, difficulties in automated expansion, and uneven coverage. It proposes a dynamic topic network construction and expansion method based on unsupervised clustering and semantic retrieval. Specifically, it automatically discovers and organizes knowledge topics from massive multi-source data through hierarchical density clustering, and employs a dual mechanism of core seed extraction and precise association of extended seeds based on vectorized retrieval to dynamically associate each topic with massive amounts of high-quality text. This overcomes the core shortcomings of existing technologies, such as reliance on manual static topic definition, inefficient seed association, and uneven coverage, thus constructing a dynamic and solid knowledge foundation for generating synthetic data with broad knowledge coverage and balanced content. This invention addresses the problems of reliance on manual templates in prompt generation and the difficulty in scalable assurance of deep content diversity. It proposes an intelligent prompt generation technology based on multi-dimensional parametric templates and data-driven optimization. Specifically, it constructs a prompt template library that supports flexible combinations of multiple dimensions such as theme, audience, style, and depth, and embeds a lightweight experimental optimization loop that automatically adjusts the prompt generation strategy based on the fine-tuning effect of downstream models. This overcomes the bottlenecks of traditional methods that rely on fixed templates and have limited generation efficiency and diversity, achieving automated synthesis of massive, differentiated, high-quality text generation instructions, which is key to ensuring the inherent diversity of synthesized data. This invention also addresses the problems of a lack of proactive constraints in the generation process, passive and lagging quality control, and high costs. It designs a process constraint generation mechanism that integrates retrieval-enhanced generation, real-time evaluation, and structured guidance. This mechanism synchronously introduces authoritative knowledge sources for factual calibration during content generation and utilizes structured guidance and a lightweight scorer to ensure logical coherence and perform preliminary quality filtering. This mechanism transforms quality control from passive post-processing to proactive generation process constraints, effectively addressing the quality control challenges of preventing factual errors, logical inconsistencies, and low-information-density text in synthesized data from the source, significantly reducing post-processing cleaning costs and quality risks. This invention addresses the problem of open-loop system processes and the lack of effect-based self-optimization capabilities. It constructs an automated data synthesis framework encompassing dynamic topic mining, intelligent suggestion synthesis, process constraint generation, and evaluation feedback iteration. This framework integrates purification and a multi-dimensional evaluation-based intelligent control system. Through the collaboration and closed-loop feedback of core modules, this framework dynamically optimizes the strategies and parameters of each upstream stage based on the multi-dimensional effect evaluation results (such as knowledge coverage, diversity, and quality) of the final synthesized data. From an engineering perspective, it solves the pain points of existing solutions' fragmented processes and inability to adaptively and continuously improve, ensuring the system's self-iteration and stable, high-quality output during large-scale production.

[0077] It is worth mentioning that this invention also provides a large-scale, diverse synthetic data generation device for pre-training large language models, used to implement the large-scale, diverse synthetic data generation method for pre-training large language models described in the above embodiments. This device includes a dynamic topic network construction module, an automated prompt generation and optimization module, a guided content generation module, a synthetic data cleansing module, and a multi-dimensional evaluation and closed-loop control module, such as... Figure 1 As shown, this device realizes a complete, automated pipeline from raw multi-source data to high-quality synthesized data, and forms an intelligent closed loop for self-optimization of the drive unit. Figure 1 As shown, in this device, the data flow begins with the original multi-source dataset, sequentially flowing through the dynamic topic network construction module, the automated prompt generation and optimization module, and the guided content generation module to generate initial synthetic data. This data is then purified and refined by the synthetic data purification module, ultimately producing a high-quality synthetic dataset that can be directly used for training large models. Simultaneously, the multi-dimensional evaluation and closed-loop control module quantitatively evaluates the refined data and transforms the generated diagnostic report into optimization instructions, feeding them back to the upstream dynamic topic network construction module, automated prompt generation and optimization module, guided content generation module, and synthetic data purification module, driving iterative optimization of their parameters and strategies. This device fundamentally integrates fragmented processing steps into an organic whole, achieving a fully automated closed-loop process from dynamic knowledge mining to intelligent instruction synthesis, process-guided generation, data purification, and effect evaluation feedback, ensuring the systematic nature, scalability, and continuous optimization capabilities of large-scale data production.

[0078] In this embodiment, the dynamic topic network construction module is used to automatically mine and organize a dynamically evolving, hierarchical, structured dynamic topic network from massive, unstructured raw multi-source data. This network is generated through multi-source data vectorization, hierarchical density clustering, and topic knowledge association. The network includes topic names, topic descriptions, knowledge point lists, and associated core seed samples and large-scale expanded seed samples. Figure 2 As shown.

[0079] In this embodiment, the automated prompt generation and optimization module is used to automatically generate diverse and executable text generation instructions based on a dynamic topic network by decoupling and dynamically combining multi-dimensional parameters through parameterized prompt templates. A lightweight, closed-loop, data-driven experimental optimization loop is used to evaluate and adaptively adjust the prompt generation strategy, thereby continuously improving the effectiveness of the synthesized data for training the large language model. Figure 3 As shown.

[0080] In this embodiment, the guided content generation module is used to constrain the generation process by integrating factual guidance and simultaneous structural and quality guidance mechanisms when the large language model synthesizes content according to text generation instructions. That is, factual guidance and simultaneous structural and quality guidance mechanisms are introduced into the generation process to constrain the generation process and generate initial screening synthesis data, such as... Figure 4 As shown.

[0081] In this embodiment, the synthetic data purification module is used to perform automated and systematic semantic-level deduplication and benchmark decontamination on the initial screening synthetic data, so as to remove noise, duplication and potential bias in the initial screening synthetic data, eliminate redundancy and potential evaluation data leakage risks, and obtain a purified synthetic dataset, which prepares a high-quality and high-purity synthetic dataset for subsequent accurate evaluation and model training.

[0082] In this embodiment, the multi-dimensional evaluation and closed-loop control module is used to perform automated and comprehensive multi-dimensional quantitative evaluation of the purified synthetic dataset, and generate optimization instructions based on the evaluation results. These instructions are then fed back to at least one of the following modules: dynamic topic network construction module, automated prompt generation and optimization module, guided content generation module, and synthetic data purification module, to adjust the corresponding parameters or strategies and achieve continuous iteration and self-optimization of the device.

[0083] In summary, this invention, through its five interconnected core modules that form a closed loop of evaluation and feedback, collectively creates a complete, data-driven, automated synthetic data generation entity. Specifically, the dynamic topic network construction module constructs a dynamic knowledge network using unsupervised clustering and semantic retrieval, addressing the static nature of knowledge systems; the automated prompt generation and optimization module achieves intelligent prompt synthesis through parameterized templates and data-driven optimization, overcoming the scalability limitations of manual prompts; the guided content generation module enhances generation and real-time evaluation process constraints through fusion retrieval, placing quality control at the forefront; the synthetic data purification module ensures data purity through meticulous cleaning; and the multi-dimensional evaluation and closed-loop control module drives iterative development across the entire device through multi-dimensional evaluation and targeted feedback. This invention systematically solves the core problems faced by existing technologies, such as single data patterns, limited knowledge coverage, uncontrollable quality, and unsystematic processes, providing a complete and scalable method and apparatus for achieving large-scale, high-quality, and diversified automated synthetic data production.

[0084] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating large-scale, diverse synthetic data for pre-training large language models, characterized in that, Includes the following steps: S1. Based on the original multi-source data, a structured dynamic topic network is generated through multi-source data vectorization representation, hierarchical density clustering, and topic knowledge association. S2. Based on a dynamic topic network, multi-dimensional parameters are decoupled and dynamically combined through parameterized prompt templates to automatically generate text generation instructions. The prompt generation strategy is continuously adjusted and optimized through a data-driven experimental optimization loop. S3, when the large language model synthesizes content according to the text generation instructions, it generates preliminary synthesized data through a constraint generation process that integrates factual guidance and structure and quality synchronous guidance mechanisms. S4. Perform semantic-level deduplication and baseline decontamination on the initial screening synthetic data to obtain a clean synthetic dataset. S5. Perform multi-dimensional quantitative evaluation on the purified synthetic dataset, and generate optimization instructions based on the evaluation results, feeding them back to at least one of steps S1-S4 to adjust the corresponding parameters or strategies.

2. The method for generating large-scale, diverse synthetic data for pre-training large language models according to claim 1, characterized in that, Step S1 specifically includes the following sub-steps: S11. Acquire raw text data from multiple independent data sources in parallel, and use a pre-trained text embedding model to encode the title and core content summary of each document into a dense semantic vector to convert unstructured text into a set of structured semantic vectors. S12. Based on the semantic vector set, unsupervised clustering is performed using a hierarchical density clustering algorithm; by analyzing the semantic similarity and hierarchical inclusion relationship between each topic cluster in the clustering results, a dynamic hierarchical topic tree is constructed. S13. For each topic cluster in the dynamic hierarchical topic tree, obtain the corresponding topic name, topic description, knowledge point list and core seed text based on its original document. Based on the topic description and knowledge point list of each topic cluster, an extended seed sample for that topic cluster is obtained through efficient vector similarity retrieval; finally, a structured dynamic topic network is output.

3. The method for generating large-scale, diverse synthetic data for pre-training large language models according to claim 2, characterized in that, The data sources include: deduplicated, cleaned, and quality-filtered web crawled text, open-licensed academic paper abstracts, books, technical documents, and encyclopedia entries.

4. The method for generating large-scale, diverse synthetic data for pre-training large language models according to claim 2, characterized in that, Step S13 specifically includes: For each topic cluster in the dynamic hierarchical topic tree, a preset number of representative text fragments are extracted from its original documents based on the maximum marginal relevance criterion, serving as the core seed samples for that topic cluster. Based on the core seed samples, the topic name and topic description are automatically generated, and a list of knowledge points corresponding to that topic cluster is generated through keyword extraction and concept summarization. The topic description and knowledge point list of each topic cluster are encoded into a comprehensive topic semantic vector through a pre-trained text embedding model; the original text corpus is divided into blocks and encoded into the same vector space to obtain text block vectors; the cosine similarity between each comprehensive topic semantic vector and each text block vector is calculated through efficient vector similarity retrieval, and the top K text blocks with the highest cosine similarity are recalled as extended seed samples for the topic cluster. The final output is a structured dynamic topic network, where each node contains a topic name, topic description, list of knowledge points, core seed samples, and extended seed samples.

5. The method for generating large-scale, diverse synthetic data for pre-training large language models according to claim 1, characterized in that, Step S2 specifically includes the following sub-steps: S21. Quality pre-filtering: The original core seed text and extended seed text in the dynamic topic network are scored in real time by a pre-trained quality assessment model, and seed texts with scores higher than a preset threshold are selected as input materials for subsequent instruction generation. S22. Construct a prompt template containing multiple structured parameter slots: The parameter slots shall include at least the following: knowledge topic slot, knowledge element slot, content seed slot, audience characteristic slot, content depth slot, genre style slot, and structure requirement slot. S23. Layered combination to generate differentiated instructions: A layered combination mechanism is adopted. Based on the prompt template and according to the preset prompt generation strategy, content elements are selected from the input materials and the text is automatically synthesized to generate instructions. S24. Data-driven generation strategy optimization: Sample a representative set of topics from the dynamic topic network as an experimental set; adjust the sampling weights and combination rules of key dimensions in the prompt template, and generate multiple synthetic data subsets with different style configurations based on the same experimental set; use each synthetic data subset to quickly fine-tune the benchmark language model, and evaluate the performance of each fine-tuned benchmark language model in parallel on the downstream task evaluation set; identify the prompt style combination pattern that produces the best performance based on the evaluation results, and transform this pattern into the dynamic sampling weight distribution and combination rules of each dimension in the prompt template to update the prompt generation strategy.

6. The method for generating large-scale, diverse synthetic data for pre-training large language models according to claim 5, characterized in that, Step S23 specifically includes: The parameter slots are organized into at least two logical layers, including a content planning layer consisting of audience characteristics and content depth, and an expression form layer consisting of genre style and structural requirements; parameterized sub-templates are configured for each parameter option in each layer; according to the preset prompt generation strategy, sub-templates are dynamically selected from each layer and instantiated, filled, and spliced ​​with knowledge topics, knowledge elements, and content seeds to generate text generation instructions.

7. The method for generating large-scale, diverse synthetic data for pre-training large language models according to claim 1, characterized in that, Step S3 specifically includes the following sub-steps: S31. Factual Guidance: When the large language model generates text according to the text generation instructions, it analyzes the generated text fragments in real time. When it identifies text containing potential factual descriptions, data citations, or explanations of technical terms, it automatically triggers a retrieval process: using the currently generated text as a query vector, it retrieves the most relevant evidence fragments from the authoritative knowledge vector library; and dynamically inserts the evidence fragments into the current generation context of the large language model in a structured form to guide the continuation and elaboration of the text. S32. Synchronous guidance of structure and quality: Before the generation begins, a structured outline is provided to the large language model according to the task type to guide the generation of text that conforms to the target logical structure; During the generation process, a real-time quality evaluator runs synchronously to score the quality of the generated text in real time. If the score is lower than the preset threshold, the generation of the current text will be stopped or a rewriting will be triggered; otherwise, the generation process will continue. S33. Distributed Constraint Generation: A distributed asynchronous task queue is used for efficient processing of generation tasks. The text generation instructions are placed in the task queue. A group of generation worker nodes, which integrate retrieval enhancement generation and quality assessment components, pull tasks in parallel and execute a constraint generation process that integrates factual guidance, structure and quality synchronization guidance mechanisms. The generated text is written to a distributed file system. The task scheduler performs overall load balancing, fault tolerance processing and progress monitoring to finally obtain the initial screening and synthesized data.

8. The method for generating large-scale, diverse synthetic data for pre-training large language models according to claim 1, characterized in that, Step S4 specifically includes the following sub-steps: S41. Semantic Deduplication: Each initial screening composite data is represented as a set of words or continuous word sequences, and multiple hash functions are used to generate the minimum hash signature; the initial screening composite data with similar minimum hash signatures are divided into the same hash bucket using the Locality Sensitive Hashing algorithm; the Jaccard similarity between any two initial screening composite data in each hash bucket is calculated, and for initial screening composite data pairs with a Jaccard similarity exceeding a preset threshold, only one of them is retained; S42. Baseline Desaturation: The initial screening synthetic data after semantic-level deduplication is compared with at least one public evaluation set, wherein the public evaluation set contains evaluation questions and corresponding standard answers; a combination of substring matching and fuzzy hash comparison is used to identify and remove samples in the initial screening synthetic data that have content duplication with the evaluation questions or the standard answers, thereby obtaining a purified synthetic dataset.

9. The method for generating large-scale, diverse synthetic data for pre-training large language models according to claim 1, characterized in that, Step S5 specifically includes the following sub-steps: S51. Conduct a multi-dimensional quantitative evaluation of the purified synthetic dataset, wherein the multi-dimensional quantitative evaluation includes knowledge coverage, content diversity, comprehensive quality distribution and factual consistency sampling; and generate a structured diagnostic report based on the evaluation results; S52. Automatically parse the structured diagnostic report into optimization instructions and feed them back to at least one of the steps S1-S4 to dynamically adjust the corresponding parameters or strategies.

10. An apparatus for implementing the method for generating large-scale diverse synthetic data for pre-training large language models according to any one of claims 1-9, characterized in that, include: The dynamic topic network construction module is used to generate a dynamic topic network based on original multi-source data through multi-source data vectorization representation, hierarchical density clustering, and topic knowledge association. The network includes topic names, topic descriptions, knowledge point lists, and associated core seed samples and extended seed samples. The automated prompt generation and optimization module is used to automatically generate text generation instructions based on a dynamic topic network, by decoupling and dynamically combining multi-dimensional parameters through parameterized prompt templates, and continuously adjusting and optimizing the prompt generation strategy through a data-driven experimental optimization loop. The guided content generation module is used to generate initial synthesized data by integrating factual guidance and structure and quality synchronization guidance mechanisms when the large language model synthesizes content according to text generation instructions. The synthetic data purification module is used to perform semantic-level deduplication and baseline decontamination on the initial screening synthetic data to obtain a purified synthetic dataset. The multi-dimensional evaluation and closed-loop control module is used to perform multi-dimensional quantitative evaluation of the purified synthetic dataset, and generate optimization instructions based on the evaluation results. These instructions are then fed back to at least one of the following modules: dynamic topic network construction module, automated prompt generation and optimization module, guided content generation module, and synthetic data purification module, to adjust the corresponding parameters or strategies.

Images