Iterative supervision fine tuning data generation method and system

Abstract

The invention discloses an iterative supervision fine tuning data generation method and system, and relates to artificial intelligence and data processing technologies, and the method comprises the steps: generating a basic training data pair based on a pre-training language model, and the basic training data pair comprises an instruction-response pair; performing model training based on the basic training data, and determining training performance feedback data according to the performance of the model on the verification set and the task index; analyzing a model performance index according to the determined training performance feedback data; predicting and optimizing an optimization strategy of training data and training parameters based on the model performance index and the basic training data pair; and performing model training according to the optimized training data and the training parameters so as to execute AI model training platform data generation by using a strategy output by the model after training. Model performance feedback, automatic task mining and sample optimization mechanisms are introduced, training data are dynamically adjusted and expanded, and closed-loop iteration and performance-oriented optimization of data generation are achieved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and data processing technology, and in particular to an iterative supervised fine-tuning data generation method and system. Background Technology

[0002] Existing supervised fine-tuning data generation methods often rely on static datasets and lack dynamic optimization targeting model weaknesses. The generated data is significantly insufficient in terms of diversity, coverage, and relevance, resulting in limited fine-tuning effects and difficulty in significantly improving the performance of large models on complex tasks.

[0003] Traditional methods typically involve a one-time data generation and training process, lacking a cyclical feedback and update mechanism based on model performance, making it difficult to continuously optimize training data to adapt to improvements in model capabilities and changes in the target task.

[0004] Existing technologies often rely on extensive manual selection or design of high-quality supervised data, which is not only inefficient and costly but also prone to introducing subjective bias, thus limiting the scalability of large-scale supervised fine-tuning. Some supervised data may be concentrated on certain high-frequency tasks or patterns, leading to insufficient learning of low-frequency patterns or marginal tasks during training, thus limiting generalization performance. Current AI model training platforms generally face the following technical problems: First, supervised fine-tuning data mainly relies on static datasets, lacking dynamic optimization mechanisms targeting model weaknesses, resulting in insufficient data diversity, coverage, and relevance, and limited fine-tuning effects; second, traditional methods are mostly one-time training processes, lacking cyclical updates based on model performance feedback, making it difficult to adapt to the evolution of model capabilities and changes in task requirements; third, high-quality data still relies on manual selection and design, which is inefficient, costly, and prone to introducing subjective bias, limiting large-scale applications; fourth, the training data structure is unbalanced, tending to favor high-frequency tasks while neglecting marginal tasks, weakening the model's generalization ability.

[0005] The technical solutions most similar to this application mainly include the following categories: 1. Instruction Tuning-Based Approach Methods such as FLAN and Alpaca fine-tune pre-trained models by constructing diverse instruction-response pairs. These methods typically rely on manual or template-based instruction generation, offering a certain breadth of task coverage, but lack a feedback adjustment mechanism for the model's actual capabilities. The generation process is static, one-off, and lacks iterative optimization, making it difficult to form an adaptive closed loop.

[0006] 2. Methods for selecting training samples using manual evaluation or heuristic scoring. For example, the Self-Instruct method first generates instruction-response pairs using an existing large model, and then uses manual rules or evaluation models to filter high-quality data. This type of method initially introduces the model's self-supervised capability, but the scoring mechanism it relies on lacks task adaptability, and it still mainly relies on single-round data generation, lacking dynamic control over data quality.

[0007] 3. Reinforcement Learning with Human Feedback (RLHF) methods, such as those used in the training of InstructGPT and ChatGPT. Although a feedback mechanism is introduced, this method focuses more on optimizing the output policy than on data generation itself. Furthermore, it has high training costs and significant human feedback costs, making it unsuitable for large-scale, low-cost SFT data generation. Summary of the Invention

[0008] This application provides an iterative supervised fine-tuning data generation method and system, which introduces model performance feedback, automatic task mining and sample optimization mechanisms to dynamically adjust and expand training data, thereby achieving closed-loop iteration and performance-oriented optimization of data generation.

[0009] This application provides an iterative supervised fine-tuning data generation method, applied to AI model training platform data generation scenarios, including: Basic training data pairs are generated based on a pre-trained language model, and the basic training data pairs include instruction-response pairs. The model is trained based on the basic training data, and the training performance feedback data is determined based on the model's performance on the validation set and task metrics. Based on the identified training performance feedback data, analyze the model performance metrics; Based on model performance metrics and basic training data pairs, predict and optimize training data and training parameters optimization strategies; The model is trained based on optimized training data and training parameters, and the AI ​​model training platform data is generated after training by using the strategy output by the model.

[0010] This application provides an iterative supervised fine-tuning data generation system, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the iterative supervised fine-tuning data generation method described above.

[0011] This application's embodiments introduce model performance feedback, automatic task mining, and sample optimization mechanisms to dynamically adjust and expand training data, thereby achieving closed-loop iteration of data generation and performance-oriented optimization.

[0012] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0013] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the basic process of the iterative supervised fine-tuning data generation method in an embodiment of this application.

[0014] Figure 2 This is a schematic diagram of the structure of the iterative supervised fine-tuning data generation system according to an embodiment of this application. Detailed Implementation

[0015] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0016] To address the technical bottlenecks in existing AI model training platforms, such as reliance on static data for supervised fine-tuning data generation, lack of performance feedback and dynamic optimization, imbalanced data structure, and high costs of manual screening, this application proposes an iterative supervised fine-tuning data generation system. This system utilizes the collaborative operation of a processor and memory to execute a computer program containing modules for model performance feedback analysis, automatic task mining, and data and parameter optimization, enabling dynamic generation and updating of training data. This application generates initial data pairs based on a pre-trained language model and extracts model weaknesses by combining validation set performance. Performance metrics drive task reconstruction and data augmentation, automatically expanding low-frequency or marginal style data to improve data diversity and coverage. Furthermore, the system employs a closed-loop iterative approach to continuously optimize training data and policy parameters, avoiding the solidification problem of one-time training and significantly improving the model's adaptability and generalization performance to complex tasks. This method effectively solves the problems of delayed data updates, uncontrollable quality, and excessive manual intervention in traditional AI assistant training processes, possessing higher training efficiency and scalability, and is suitable for large-scale model fine-tuning needs across multiple tasks and scenarios. Specifically, this application provides an iterative supervised fine-tuning data generation method applied to AI model training platform data generation scenarios, including: In the AI ​​model training platform data generation scenario, in step S101, basic training data pairs are generated based on a pre-trained language model. These basic training data pairs include instruction-response pairs. Specifically, preliminary supervised fine-tuning data can be generated using a pre-trained large model based on an existing static dataset. In some embodiments, generating basic training data pairs based on a pre-trained language model includes: generating data of different styles, difficulties, and task types based on a pre-trained language model (such as LLaMA or Qwen) through conditional control parameters. This is achieved using Prompt template generation, coupled with rule filters and confidence level screening (such as BLEU scores and duplicate detection) to complete conditional control, remove low-quality text, and support the fusion of hand-crafted and synthetic data.

[0017] In step S102, model training is performed based on the basic training data, and training performance feedback data is determined according to the model's performance on the validation set and task metrics. The model training and evaluation module is a key link connecting data generation and performance feedback in this invention. It mainly performs systematic training on the supervised fine-tuning data generated in each iteration and produces evaluation results to feed back into the data optimization process. By integrating multiple technical mechanisms, the model training is ensured to be efficient, stable, and highly adaptable.

[0018] In step S103, based on the determined training performance feedback data, the model performance metrics are analyzed, such as accuracy, recall, F1 score, BLEU score, training and validation loss, generalization error, and other metrics.

[0019] In step S104, based on model performance metrics and basic training data pairs, an optimization strategy for training data and training parameters is predicted and optimized. In step S105, the model is trained based on the optimized training data and training parameters. After training, the AI ​​model training platform generates data using the strategy output by the model. Specifically, during the iterative training process, a data version control mechanism ensures that each round of sample generation is traceable and rollbackable. Each set of data automatically generates a version number, along with metadata such as task type, sample source, and generation rules. It also supports incremental storage and sample difference tracking based on a Git-like structure, ensuring the controllability of data updates and comparative analysis capabilities.

[0020] By accessing the performance metrics of the training model (such as accuracy, loss variation, generalization error, etc.), we can dynamically analyze the contribution of the current data to the model training effect, locate the task dimensions or redundant samples with declining performance, and thus drive the adjustment of the next round of task mining and sample generation strategies.

[0021] Set a timing control strategy for training and data updates, such as automatically starting data updates every N training rounds, and adjusting the sample ratio structure, task popularity weights and style coverage according to the model training trend, to build an adaptive scheduling scheme that gradually shifts from "diversity-driven" to "accuracy-driven".

[0022] It can also provide a full-process display of data evolution, including version switching, sample quality score distribution, task coverage heatmap, etc., and has an anomaly alarm function to alert potential risks such as sample imbalance and model stagnation.

[0023] This application's embodiments introduce model performance feedback, automatic task mining, and sample optimization mechanisms to dynamically adjust and expand training data, achieving closed-loop iteration of data generation and performance-oriented optimization. Current AI model training platforms generally face the following technical problems: First, supervised fine-tuning data mainly relies on static datasets, lacking dynamic optimization mechanisms targeting model weaknesses, resulting in insufficient data diversity, coverage, and relevance, and limited fine-tuning effects; second, traditional methods are mostly one-time training processes, lacking cyclical updates based on model performance feedback, making it difficult to adapt to model capability evolution and changes in task requirements; third, high-quality data still relies on manual selection and design, which is inefficient, costly, and prone to introducing subjective bias, limiting large-scale applications; fourth, the training data structure is unbalanced, tending to favor high-frequency tasks while neglecting marginal tasks, weakening the model's generalization ability.

[0024] In some embodiments, model training based on basic training data includes: employing a multi-stage supervised fine-tuning mechanism, with the training process divided into multiple stages: warm-up, main training, and fine-tuning, coupled with different learning rate strategies. This ensures that the model remains stable when adapting to new data while fully learning the characteristics of new tasks, effectively suppressing catastrophic forgetting. Regarding training control, a dynamic hyperparameter adjustment mechanism is introduced to monitor training curves (such as loss and accuracy fluctuations) and automatically adjust parameters such as learning rate and batch size to improve training efficiency and stability. The module has a built-in comprehensive performance evaluation system covering metrics such as accuracy, F1, and BLEU, and incorporates zero-shot and few-shot generalization ability testing, robustness verification, and task coverage graph analysis, outputting standardized reports as feedback input to the data iteration management module.

[0025] The performance feedback analysis in this embodiment is used to perform in-depth analysis and intelligent analysis of the multidimensional performance data output by the model training and evaluation module, guide subsequent task mining and sample optimization, and promote the dynamic iteration and precise upgrading of supervised fine-tuning data.

[0026] Specifically, in some implementations, based on the determined training performance feedback data, the model performance metrics are analyzed, including: accuracy, recall, F1 score, BLEU score, training and validation loss, and generalization error. Clustering is performed based on the analyzed performance metrics, and anomaly detection is performed based on the clustering results to identify the model's performance under different tasks and data versions, thereby accurately identifying model bottlenecks and overfitting risks. This application proposes a sample contribution evaluation and gradient influence modeling mechanism. By evaluating sample contribution and importance, combined with sample weight and gradient contribution analysis, it determines which samples have the greatest impact on improving model performance, effectively guiding sample optimization and expansion, and avoiding resource waste. In some embodiments, the training performance feedback data is determined based on the model's performance on the validation set and task metrics, including: For any pair of basic training data Calculate its gradient contribution to the model parameter update during backpropagation:

[0027] Here, GCS represents the gradient contribution value, indicating that the stronger the influence of this sample on model parameter tuning, the better. For loss function, These are the model parameters.

[0028] The importance of a sample is measured using the following comprehensive evaluation function:

[0029] It represents the semantic novelty of a basic training data pair and measures its redundancy with existing basic training data pairs. This indicates that the high loss value of the basic training data has been retained for a long time during training, representing whether it has been fully learned by the model. The parameters λ1, λ2, and λ3 are task-adjustable hyperparameters.

[0030] According to importance The data is sorted, high-value samples are selected for sample synthesis, task construction, incremental training sampling, and data cleaning and redundancy removal.

[0031] Also includes: based on importance The samples are sorted, and those with low importance but high long-term residual loss values ​​are used for anomaly detection or mislabeling. In other words, samples with low importance but high loss residuals enter the "re-examination zone" for anomaly detection or mislabeling.

[0032] In some embodiments, optimization strategies for predicting and optimizing training data and training parameters based on model performance metrics and basic training data pairs include: Based on the performance of the identified model across different tasks and data versions, the types of weak tasks are determined. In some embodiments, the types of weak tasks are determined in the following manner: The analysis examines the model's performance across different tasks, identifies model weaknesses based on a multi-dimensional indicator modeling vulnerability identification mechanism, and extracts multi-dimensional evaluation indicators at the task level (such as accuracy, robustness, response consistency, and inference chain integrity). Historical tasks are categorized by semantic tags or functional categories; Based on the segmentation results, vulnerability is calculated, and the task vulnerability factor is defined as:

[0033] in Indicates task category i Intra-performance variance This indicates the proportion of samples that failed to be generated.

[0034] After identifying the weak task types, historical instruction patterns are extracted using semantic clustering techniques to construct task templates. Task variants are then generated based on these templates to expand the expressive diversity of training instructions. For example, multiple task variants can be generated through word order substitution and semantic fuzzification to broaden the expressive diversity of training instructions.

[0035] Further scoring and ranking are performed based on factors such as vulnerability, coverage difference, and novelty. High-value tasks are prioritized and sent to the sample optimization module for processing. The scoring and ranking satisfy the following conditions:

[0036] in, This indicates the vulnerability of task i. This indicates the degree of missing information for task i in the training data. This indicates the novelty of task i compared to existing tasks. , , This represents the corresponding weighting coefficient.

[0037] In some embodiments, the method further includes: generating candidate samples through a multi-source generation mechanism (e.g., calling a large model for generation, knowledge base filling, dialogue transcription, etc.), and using a lightweight scoring model to score and filter samples from the dimensions of syntax, semantics, and task completion to obtain high-scoring samples.

[0038] It also includes a style and structure enhancement process for high-quality samples, using text transfer and structural transformation techniques to generate diverse versions, while introducing style tags to improve the model's understanding capabilities. The entire process has automatic feedback capabilities; low-quality samples can be sent back to the task mining module, forming a data closed-loop optimization mechanism. Text clustering algorithms (such as K-Means on BERTembeddings) are applied to identify semantically repetitive samples and then fused or pruned them.

[0039] This application's method significantly improves the quality and relevance of supervised fine-tuning data generation by constructing an iterative closed-loop mechanism based on performance feedback. Traditional methods rely on static datasets, which struggle to effectively cover model weaknesses, limiting fine-tuning effectiveness. This application, however, introduces automatic task mining and sample optimization, dynamically identifying model performance deficiencies on different tasks and samples, and specifically expanding and optimizing training data. This dynamic adjustment not only enhances data diversity and coverage but also strengthens the model's generalization ability to complex and marginal tasks, solving the problems of poor generalization and uneven sample distribution inherent in traditional methods.

[0040] Furthermore, this application achieves intelligent management and continuous optimization of the training process through performance feedback analysis and data iteration management, reducing reliance on manual selection and design of high-quality supervised data, significantly saving labor costs, and reducing subjective bias. Multi-stage training and dynamic parameter adjustment techniques improve training stability and efficiency, further driving model performance improvement. This application realizes closed-loop iteration and performance-oriented optimization of supervised fine-tuning data generation, effectively overcoming the bottlenecks of existing technologies in data quality, training efficiency, and model generalization, meeting the practical needs of large-scale, efficient, and accurate fine-tuning.

[0041] This application also proposes an iterative supervised fine-tuning data generation system, including a processor and a memory, wherein the memory stores a computer program, the specific structure of which is as follows: Figure 2 As shown, when the computer program is executed by the processor, it implements the steps of the iterative supervised fine-tuning data generation method as described above.

[0042] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0043] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0044] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0045] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. An iterative supervised fine-tuning data generation method, characterized in that, Applications include data generation scenarios for AI model training platforms, including: Basic training data pairs are generated based on a pre-trained language model, and the basic training data pairs include instruction-response pairs. The model is trained based on the basic training data, and the training performance feedback data is determined based on the model's performance on the validation set and task metrics. Based on the identified training performance feedback data, analyze the model performance metrics; Based on model performance metrics and basic training data pairs, predict and optimize training data and training parameters optimization strategies; The model is trained based on optimized training data and training parameters, and the AI ​​model training platform data is generated after training by using the strategy output by the model.

2. The iterative supervised fine-tuning data generation method as described in claim 1, characterized in that, The basic training data pairs generated based on the pre-trained language model include: Based on a pre-trained language model, data of different styles, difficulties and task types are generated by controlling the parameters. The data is generated using Prompt templates and combined with rule filters and confidence scores to complete the condition control.

3. The iterative supervised fine-tuning data generation method as described in claim 1, characterized in that, Model training based on basic training data includes: A multi-stage supervised fine-tuning mechanism is adopted, and the training process is divided into multiple stages: warm-up, main training, and fine-tuning, in conjunction with different learning rate strategies.

4. The iterative supervised fine-tuning data generation method as described in claim 3, characterized in that, Based on the model's performance on the validation set and task metrics, the training performance feedback data includes: For any pair of basic training data Calculate its gradient contribution to the model parameter update during backpropagation: Where GCS represents the gradient contribution value, For loss function, These are model parameters; The importance of a sample is measured using the following comprehensive evaluation function: It represents the semantic novelty of a basic training data pair and measures its redundancy with existing basic training data pairs. This represents the long-term residual loss value of the basic training data during training, and the parameters λ1, λ2, and λ3 are task-adjustable hyperparameters. According to importance The samples are sorted, and high-value samples are selected for sample synthesis and task construction.

5. The iterative supervised fine-tuning data generation method as described in claim 4, characterized in that, Based on the model's performance on the validation set and task metrics, the training performance feedback data also includes: According to importance The samples are sorted, and those with high long-term residual loss values ​​but lower importance than a preset threshold are used for anomaly detection or label misidentification.

6. The iterative supervised fine-tuning data generation method as described in claim 3, characterized in that, Based on the identified training performance feedback data, the analytical model performance metrics include: The analysis includes performance metrics such as accuracy, recall, F1 score, BLEU score, training and validation loss, and generalization error. Clustering is performed based on parsed performance metrics, and anomaly detection is performed based on the clustering results to identify the model's performance under different tasks and data versions.

7. The iterative supervised fine-tuning data generation method as described in claim 6, characterized in that, Optimization strategies based on model performance metrics and basic training data pairs, which predict and optimize training data and training parameters, include: Based on the performance of the identified model under different tasks and data versions, determine the types of weak tasks; Based on the type of weak task, semantic clustering technology is used to extract historical instruction patterns, construct task templates, and generate task variants based on the task templates to expand the expressive diversity of training instructions. Tasks are scored and ranked based on vulnerability, coverage difference, and novelty, with high-value tasks prioritized and sent to the sample optimization module for processing. The scoring and ranking satisfy the following conditions: in, This indicates the vulnerability of task i. This indicates the degree of missing information for task i in the training data. This indicates the novelty of task i compared to existing tasks. , , This represents the corresponding weighting coefficient.

8. The iterative supervised fine-tuning data generation method as described in claim 7, characterized in that, The types of weak tasks are determined in the following way: Extract multi-dimensional evaluation metrics at the task level; and, Historical tasks are categorized by semantic tags or functional categories; Based on the segmentation results, vulnerability is calculated, and the task vulnerability factor is defined as: in Indicates task category i Intra-performance variance This indicates the proportion of samples that failed to be generated.

9. The iterative supervised fine-tuning data generation method as described in claim 7, characterized in that, Also includes: Candidate samples are generated through a multi-source generation mechanism, and a lightweight scoring model is used to score and filter samples from the dimensions of syntax, semantics and task completion to obtain high-scoring samples.

10. An iterative supervised fine-tuning data generation system, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the steps of the iterative supervised fine-tuning data generation method as described in any one of claims 1 to 9.

Images