Method for double-layer optimization of AI training data based on large model

Abstract

The invention provides a method for double-layer optimization of AI training data based on a large model, and relates to the field of artificial intelligence, and the method comprises the steps: generating a plurality of homologous answers for each question and instruction in historical data through high-randomness hyper-parameter configuration by using the randomness generation capability of the large language model; based on the plurality of homologous answers, scoring is carried out under low-randomness hyper-parameter configuration through a pre-trained teacher model, preference data pairs are generated, and answers with scores higher than a threshold value are screened out to serve as high-quality supervision fine adjustment data; and S2, enabling the homologous answers generated in the step S1 to correspond to the corresponding scoring results in the step S2, constructing primary preference data for direct preference optimization, and forming first-level optimization data in combination with the screened high-quality SFT data. Through a double-layer optimization architecture and a dynamic adjustment mechanism, the AI training efficiency is improved on the premise of ensuring the data quality, and reliable data support is provided for large-scale model training.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for optimizing AI training data at two levels based on a large model. Background Art

[0002] Current AI training requires large amounts of data and places greater emphasis on data quality. However, manual labeling (such as RLHF, Reinforcement Learning from Human Feedback) faces high costs and low efficiency. Using AI to improve data quality has gradually become an industry option. Existing AI data enhancement methods, such as RLAIF (Reinforcement Learning from AI Feedback), are often limited to the capabilities of the model itself and lack a hierarchical approach to quality improvement. Furthermore, RLAIF methods are often limited to optimizing data generated from a single round of tasks. We combine RLAIF with model reflection technology to first construct primary preference data that is more consistent with the original data distribution. Then, based on this primary preference data, we use Reflection to construct secondary preference data with higher quality but more significant differences from the original data distribution. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method for two-level optimization of AI training data based on large models. Through the innovative two-level optimization architecture and dynamic adjustment mechanism, the efficiency of AI training is improved while ensuring data quality, providing a reliable data foundation support for large-scale model training.

[0004] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0005] In a first aspect, a method for optimizing AI training data at two levels based on a large model is provided, the method comprising:

[0006] S1: Leveraging the random generation capabilities of large language models, for each question and instruction in historical data, multiple homologous answers are generated through highly random hyperparameter configurations.

[0007] S2: Based on multiple homologous answers, a pre-trained teacher model is used to score them under a low-randomness hyperparameter configuration to generate preferred data pairs, and answers with scores above a threshold are selected as high-quality supervised fine-tuning data;

[0008] S3: Match the homologous answers generated in step S1 with the corresponding scoring results in step S2 to construct primary preference data for direct preference optimization, and combine them with the screened high-quality SFT data to form the first-level optimization data;

[0009] S4: Based on high-quality SFT data, a multi-dimensional quality analysis is conducted on it through a reflective model to generate second-level optimization data containing reflective trajectories. The reflective trajectories include the correction logic and quality improvement path for the original answers.

[0010] S5: Mix the first-level optimized data and the second-level optimized data to generate a comprehensive training set, where the second-level data carries auxiliary information of the reflection trajectory to enhance the model's understanding and reasoning ability for complex tasks;

[0011] S6: Based on the comprehensive training set, historical context information is injected into each interaction round of the multi-round dialogue. Based on the optimization process of the above steps, a tree-like divergent dialogue path is generated to enhance the logical coherence and diversity of the multi-round dialogue data.

[0012] S7: When the semantic similarity is detected to be lower than the preset threshold during the online service process, the error dialogue chain is extracted and the error type is analyzed. The homologous data generation strategy of step S1, the scoring threshold of step S2, and the reflection model configuration of step S4 are adjusted in reverse to generate a closed-loop mechanism for iterative optimization of cross-level data quality.

[0013] Furthermore, S1: Leveraging the random generation capabilities of the large language model, for each question and instruction in the historical data, multiple homologous answers are generated through highly random hyperparameter configurations, including:

[0014] Based on each question and instruction in the historical data, adjust the generation hyperparameters of the large language model to a high randomness mode to generate at least 3 homologous answers;

[0015] Homologous answers must meet the cosine similarity of ≥ 0.75 with the original data answers, and semantic consistency must be ensured through fixed instruction keywords;

[0016] The repetition penalty mechanism is disabled during the generation process, and the generated length is limited to no more than 150% of the original answer.

[0017] Furthermore, S2: Based on multiple homologous answers, a pre-trained teacher model is used to score them under a low-randomness hyperparameter configuration to generate preferred data pairs, and answers with scores above a threshold are selected as high-quality supervised fine-tuning data, including:

[0018] Input the same source answers into the pre-trained teacher model, switch to low randomness mode, and score each answer based on the preset scoring template;

[0019] The scoring template scores based on three dimensions: accuracy, coherence, and information content, generating a normalized score of 0-100;

[0020] The answer pairs with a score difference of ≥15 points were screened to form the primary DPO preference data, and the answers with a score of ≥85 under a single question were marked as the first-level SFT data.

[0021] Furthermore, S3: the homologous answers generated in step S1 are matched with the corresponding scoring results in step S2 to construct primary preference data for direct preference optimization, and combined with the screened high-quality SFT data to form the first-level optimization data, including:

[0022] The primary DPO data was denoised to remove answer pairs with a repetition rate of ≥80% for the same question;

[0023] The first-level SFT data is sorted by score, the top 30% answers are selected, and data enhancement is performed through synonym replacement and sentence reorganization to generate an expanded first-level optimized dataset.

[0024] Furthermore, S4: Based on the high-quality SFT data, a multi-dimensional quality analysis is performed on it through a reflection model to generate second-level optimization data containing reflection trajectories. The reflection trajectories include the correction logic and quality improvement path for the original answers, including:

[0025] Inputting the first-level SFT data into the reflection model, loading a multi-round reflection prompt template, wherein the template includes error recognition instructions, correction path generation instructions, and high-quality answer generation instructions;

[0026] Generate optimized data containing reflection track text and second-level answers, where the reflection track and answers are bound by special separators, and the second-level answers must pass the teacher model verification score improvement of ≥10%.

[0027] Furthermore, S5: the first-level optimized data and the second-level optimized data are mixed to generate a comprehensive training set, where the second-level data carries auxiliary information of the reflection trajectory to enhance the model's understanding and reasoning ability for complex tasks, including:

[0028] The first-level optimized data and the second-level optimized data are mixed in a ratio of 6:4 to generate a comprehensive training set;

[0029] A dynamic attention mask mechanism is used during training to force the model to analyze the content of the reflection trajectory before generating the answer;

[0030] If the KL divergence between the second-level data and the original data is ≥ 2.0, it is downsampled to 50% of the original data volume, and the feature distribution is aligned through the adversarial training module.

[0031] Furthermore, S6: Based on the comprehensive training set, historical context information is injected into each interaction round of the multi-round dialogue. Based on the optimization process of the above steps, a tree-like divergent dialogue path is generated to enhance the logical coherence and diversity of the multi-round dialogue data, including:

[0032] Based on the comprehensive training set, in a multi-round dialogue scenario, the historical dialogue context is injected into the question generation phase of step S1 to generate questions containing related references;

[0033] For each round of generated tree-like dialogue paths, the top two reflection trajectory branches are retained, and the logical dependencies between the paths are modeled using a graph neural network.

[0034] The tree path data is iteratively enhanced through the two-level optimization process of steps S2-S5, and ultimately a multi-round dialogue training set with long-range coherence is generated.

[0035] Furthermore, in step S7, when the semantic similarity is detected to be lower than a preset threshold during the online service process, the error dialogue chain is extracted and the error type is analyzed. The homologous data generation strategy of step S1, the scoring threshold of step S2, and the reflection model configuration of step S4 are adjusted in reverse order, thereby generating a closed-loop mechanism for iterative optimization of cross-level data quality, including:

[0036] When the online service detects that the semantic similarity of user feedback is lower than the threshold, it extracts the reflection trajectory data of the error dialogue chain and analyzes the distribution of error types;

[0037] According to the error type, the number of homologous answers generated, the scoring threshold and the reflection model configuration are dynamically adjusted to form a closed-loop iterative mechanism for cross-level data optimization.

[0038] The second aspect is a system for optimizing AI training data at two levels based on large models, including:

[0039] The acquisition module leverages the random generation capabilities of the large language model to generate multiple homologous answers for each question and instruction in the historical data using high-random hyperparameter configurations. Based on these multiple homologous answers, the pre-trained teacher model is used to score them using low-random hyperparameter configurations to generate preferred data pairs. Answers with scores above a threshold are selected as high-quality supervised fine-tuning data.

[0040] The first-level optimization module is used to match the homologous answers generated in step S1 with the corresponding scoring results in step S2, construct primary preference data for direct preference optimization, and combine it with the screened high-quality SFT data to form the first-level optimization data;

[0041] The second-level optimization module is used to perform multi-dimensional quality analysis on high-quality SFT data through a reflection model to generate second-level optimization data containing reflection trajectories, wherein the reflection trajectories include correction logic and quality improvement paths for the original answers;

[0042] The processing module is used to mix the first-level optimized data and the second-level optimized data to generate a comprehensive training set, in which the second-level data carries auxiliary information of the reflection trajectory to enhance the model's understanding and reasoning ability of complex tasks; based on the comprehensive training set, historical context information is injected into each round of interaction in multiple rounds of dialogue, and a tree-like divergent dialogue path is generated based on the optimization process of the aforementioned steps to enhance the logical coherence and diversity of the multi-round dialogue data; when the semantic similarity is detected to be lower than the preset threshold during the online service process, the error dialogue chain is extracted and the error type is analyzed, and the homologous data generation strategy of step S1, the scoring threshold of step S2 and the reflection model configuration of step S4 are reversely adjusted to generate a closed-loop mechanism for iterative optimization of cross-level data quality.

[0043] According to a third aspect, a computing device includes:

[0044] one or more processors;

[0045] The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0046] In a fourth aspect, a computer-readable storage medium stores a program, which implements the method when executed by a processor.

[0047] The above solution of the present invention includes at least the following beneficial effects:

[0048] Highly random hyperparameters are used to generate homologous answers, ensuring that the distribution of new data is similar to that of the original data. The pre-trained teacher model uses a low-randomness scoring model to reduce uncertainty. This two-pronged approach ensures stable data distribution and quality, mitigates the risk of training fluctuations, and makes model training more reliable and predictable. The first layer integrates primary preference data with high-quality SFT data through denoising and data augmentation. The second layer utilizes a reflective model to deeply explore and correct answer questions, achieving breakthrough quality. The blending of these two layers of data creates a comprehensive training set that is both stable and high-quality, providing high-quality resources for model training. A dynamic attention masking mechanism forces the model to analyze the reflective trajectory and learn the logic for answer optimization, improving logical reasoning and problem-solving capabilities. Historical context is injected into multi-round conversations to generate a tree-like path, enhancing conversational coherence and diversity. A closed-loop iterative optimization mechanism adjusts strategies in real time based on online service error feedback, continuously optimizing data, driving the model to output more accurate and fluent answers, and improving the user experience. A standardized and automated data optimization process spans the entire process from data generation, scoring, optimization, and iterative adjustments, reducing reliance on manual labeling and optimization. Flexibly adjust data generation and optimization strategies based on error types to adapt model training to single-round and multi-round tasks, as well as different types of problems and data distributions. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flowchart of a method for optimizing AI training data at two levels based on a large model provided by an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of a system for optimizing AI training data at two levels based on a large model provided by an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram of a solution for two-level optimization of AI training data based on a large model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0053] like Figure 1 As shown, an embodiment of the present invention proposes a method for optimizing AI training data at two levels based on a large model, the method comprising the following steps:

[0054] Step S1: Utilizing the random generation capability of the large language model, for each question and instruction in the historical data, multiple homologous answers are generated through highly random hyperparameter configuration;

[0055] Step S2: Based on multiple homologous answers, the pre-trained teacher model is used to score them under a low-randomness hyperparameter configuration to generate preference data pairs, and the answers with scores above the threshold are selected as high-quality supervised fine-tuning data;

[0056] Step S3: Match the homologous answers generated in step S1 with the corresponding scoring results in step S2 to construct primary preference data for direct preference optimization, and combine them with the screened high-quality SFT data to form the first-level optimization data;

[0057] Step S4: Based on the high-quality SFT data, a multi-dimensional quality analysis is performed on it through the reflection model to generate second-level optimization data containing reflection trajectories, wherein the reflection trajectories include the correction logic and quality improvement path of the original answers;

[0058] Step S5: Mix the first-level optimized data and the second-level optimized data to generate a comprehensive training set, where the second-level data carries auxiliary information of the reflection trajectory to enhance the model's understanding and reasoning ability for complex tasks;

[0059] Step S6: Based on the comprehensive training set, historical context information is injected into each interaction round of the multi-round dialogue. Based on the optimization process of the above steps, a tree-like divergent dialogue path is generated to enhance the logical coherence and diversity of the multi-round dialogue data.

[0060] Step S7: When the semantic similarity is detected to be lower than the preset threshold during the online service process, the error dialogue chain is extracted and the error type is analyzed. The homologous data generation strategy of step S1, the scoring threshold of step S2, and the reflection model configuration of step S4 are adjusted in reverse to generate a closed-loop mechanism for iterative optimization of cross-level data quality.

[0061] In an embodiment of the present invention, homologous answers are generated through a highly random configuration, and high-quality data is screened in combination with a teacher model to ensure stable data distribution. Multi-level optimized data is then constructed on this basis, and the mixed training set significantly improves the model training effect. The second-level data is accompanied by reflection trajectories, and with the help of a dynamic attention masking mechanism, the model is helped to understand the answer thinking path and improve the understanding and reasoning ability of complex tasks. Based on the comprehensive training set, historical context is injected to generate diversified dialogue paths, enhancing the coherence and richness of multi-round dialogue data. The closed-loop mechanism detects data problems in real time, dynamically adjusts generation strategies and configurations, continuously optimizes data quality, and adapts to different application requirements. The automated process reduces manual labeling and efficiently processes large amounts of data, significantly reducing costs and improving data processing efficiency in cold start and high-manpower consumption scenarios.

[0062] In a preferred embodiment of the present invention, the above step S1: utilizing the random generation capability of the large language model to generate multiple homologous answers for each question and instruction in the historical data through highly random hyperparameter configuration may include:

[0063] Step S11, based on each question and instruction in the historical data, adjust the generation hyperparameters of the large language model to a high randomness mode to generate at least three homologous answers;

[0064] Step S12: The homologous answer must meet the cosine similarity of ≥ 0.75 with the original data answer, and the semantic consistency must be ensured by fixed instruction keywords;

[0065] Step S13: disable the repetition penalty mechanism during the generation process, and limit the generated length to no more than 150% of the original answer.

[0066] In this embodiment, a highly randomized model is used to generate homologous answers, ensuring similar data distribution, reducing the risk of training instability, and improving training reliability. Cosine similarity and keyword constraints ensure semantic consistency while maintaining answer diversity, thus increasing the value of data training. Multiple answer generation and the disabling of duplicate penalties control answer length, enriching language expression and problem-solving ideas. Clearly defined parameters and rules form a standardized process, reducing manual intervention.

[0067] In an embodiment of the present invention, the specific steps include:

[0068] In step S11, the historical data used for model training is collected and organized, and questions and instructions are extracted to form a dataset to be processed. The hyperparameter setting module that controls the generation process in the large language model is located and the generation hyperparameters are adjusted to a high-stochastic mode. Each question and instruction in the historical dataset is fed into the large language model in sequence, and each model run generates at least three answers. During the generation process, all answers corresponding to each question and instruction are recorded to form a preliminary set of homologous answers.

[0069] Step S12: Use a text similarity calculation tool (such as a word vector-based calculation method) to calculate the cosine similarity between each homologous answer and the original data answer to obtain a similarity value between each answer and the original answer; check whether each homologous answer contains fixed instruction keywords. If not, the answer is determined to be semantically inconsistent and is eliminated; filter out homologous answers with a cosine similarity ≥ 0.75 and consistent semantics, remove answers that do not meet the conditions from the preliminary set, retain answers that meet the quality requirements, and form a filtered answer set.

[0070] In step S13, in the generation settings for the large language model, find the option related to duplicate penalty and disable it to ensure that the model is not inhibited by duplicate content when generating answers. The length of each answer in the filtered answer set is measured one by one and compared with the length of the original answer. Answers that exceed 150% of the original length are pruned to meet the required length. If the answer cannot be pruned to the required length, it is removed from the set. After this process, the final set of usable homologous answers is obtained.

[0071] In a preferred embodiment of the present invention, the above step S2: based on multiple homologous answers, scoring them using a pre-trained teacher model under a low-randomness hyperparameter configuration to generate preference data pairs, and screening out answers with scores above a threshold as high-quality supervised fine-tuning data, may include:

[0072] Step S21: Input the same source answers into the pre-trained teacher model, switch to the low randomness mode, and score each answer based on the preset scoring template;

[0073] Step S22: The scoring template scores based on three dimensions: accuracy, coherence, and information content, generating a normalized score from 0 to 100.

[0074] Step S23 , screen the answer pairs with a score difference of ≥15 points to form the primary DPO preference data, and mark the answers with a score of ≥85 under a single question as the first-level SFT data.

[0075] In an embodiment of the present invention, homologous answers are input into a pre-trained teacher model and scored in a low-randomness mode combined with a preset template to reduce interference; scoring based on three key dimensions: accuracy, coherence, and information content can comprehensively measure the quality of the answers, accurately screen high-quality data, and lay a solid foundation for model training. DPO data is constructed by screening answer pairs with a score difference of more than 15 points to highlight the comparison between advantages and disadvantages, helping the model to quickly learn preference relationships; answers with a score of ≥85 are marked as first-level SFT data to enhance specific task processing capabilities and improve training efficiency; primary DPO and first-level SFT data work together to optimize preference strategies and specific task processing, respectively, enhancing the accuracy, logical coherence, and information richness of the model's answers to meet application requirements.

[0076] In an embodiment of the present invention, the specific steps include:

[0077] In step S21, select a suitable pre-trained teacher model capable of evaluating answer quality, such as a language model with excellent performance in related fields and a parameter scale suitable for the current task. Enter the parameter setting interface of the pre-trained teacher model and switch the model generation mode to low randomness mode to ensure the stability and consistency of the scoring process. A template for answer scoring should be prepared in advance, clarifying the various scoring indicators and rules in the template to ensure the uniformity and repeatability of the scoring standards. Homologous answers should be input into the pre-trained teacher model in sequence according to the corresponding relationship between questions or instructions. Each answer should be scored using the preset scoring template, and the initial scoring result for each answer should be recorded.

[0078] In step S22, based on a preset scoring template, the initial score for each answer is broken down into three dimensions: accuracy, coherence, and information content. Each dimension is evaluated separately. For each dimension, a score is assigned based on the answer's specific performance in that dimension, combined with the scoring criteria. For example, accuracy is scored based on how well the answer matches the correct knowledge and requirements; coherence is scored based on the logic of the sentence and the context; and information content is scored based on the amount of valid information provided by the answer. The scores for the three dimensions are normalized to a uniform range of 0-100. Appropriate normalization methods, such as linear transformation, can be used to ensure comparability between different answers. The normalized scores for the three dimensions are then aggregated to form a final 0-100 normalized score for each answer, which is then recorded and saved.

[0079] Step S23 traverses all answers. For answers to the same question or instruction, the difference between any two answers is calculated. Answer pairs with a score difference of 15 or more are selected as primary DPO preference data and stored. Similarly, for answers to the same question or instruction, answers with a score of 85 or more for a single question are selected. These answers are marked as first-level SFT data and stored separately. The selected primary DPO preference data and the marked first-level SFT data are organized into a specific format and output as a data file for subsequent model training to ensure data standardization and usability.

[0080] In a preferred embodiment of the present invention, the above step S3: matching the homologous answers generated in step S1 with the corresponding scoring results in step S2, constructing primary preference data for direct preference optimization, and combining the screened high-quality SFT data to form the first-level optimization data, may include:

[0081] Step S31, performing denoising on the primary DPO data, removing answer pairs with a repetition rate of ≥80% for the same question;

[0082] In step S32, the first-level SFT data is sorted by score, the top 30% of answers are selected, and data enhancement is performed through synonym replacement and sentence reorganization to generate an expanded first-level optimized data set.

[0083] In an embodiment of the present invention, the primary DPO data is denoised to remove answer pairs with a repetition rate of ≥80% for the same question, effectively eliminating redundant information. The first-level SFT data is sorted by score and the top 30% of answers are selected, ensuring that the selected data are high-quality samples and laying a good foundation for model supervision fine-tuning. On this basis, data enhancement is performed through synonym replacement and sentence reorganization, increasing the diversity of data expression while retaining the core semantics of the answers. The optimized primary DPO data and the expanded first-level SFT data work together to provide higher-quality data for model training. Together, they help the model achieve more precise optimization during the training process, improving the model's answer accuracy, logical rationality, and overall performance in dealing with complex tasks, enabling it to complete tasks more stably in practical applications.

[0084] In an embodiment of the present invention, the specific steps include:

[0085] In step S31, the primary DPO data is grouped by question or instruction, ensuring that data within the same group corresponds to the same question or instruction. For each data group, the repetition rate between answer pairs is calculated. This is determined by comparing the text content of the answers and counting the percentage of identical text fragments. Each data group is then traversed to identify answer pairs with a repetition rate of 80% or more. These answer pairs are removed from the primary DPO data. After the removal process is complete, the remaining primary DPO data is organized for subsequent use.

[0086] In step S32, all answers are sorted from high to low based on the score of each answer in the first-level SFT data. Based on the sorting results, the top 30% of answers are selected as high-quality samples in the first-level SFT data. For the selected high-quality answers, the text content is analyzed sentence by sentence to identify replaceable words. Using a synonym dictionary or related tools, the original words are replaced with synonyms to make the answers more diverse. For the answers that have undergone synonym replacement, the sentence structure is readjusted, for example, by changing active sentences to passive sentences, splitting long sentences into short sentences, or merging short sentences into long sentences, further changing the expression of the answers. The data that has undergone synonym replacement and sentence reorganization is integrated with the original retained high-quality answers to generate an expanded first-level optimized data set.

[0087] In a preferred embodiment of the present invention, step S4 above: based on the high-quality SFT data, a multi-dimensional quality analysis is performed on it through a reflection model to generate second-level optimization data including a reflection trajectory. The reflection trajectory includes the correction logic and quality improvement path for the original answer, which may include:

[0088] Step S41: Input the first-layer SFT data into the reflection model and load multiple rounds of reflection prompt templates, wherein the templates include error recognition instructions, correction path generation instructions, and high-quality answer generation instructions;

[0089] Step S42: Generate optimized data containing the reflection track text and the second-level answer, where the reflection track and the answer are bound by a special separator, and the second-level answer needs to pass the teacher model verification score improvement of ≥10%.

[0090] In an embodiment of the present invention, the first-level SFT data is input into the reflection model and multiple rounds of reflection prompt templates are loaded. Through error recognition instructions, correction path generation instructions, and high-quality answer generation instructions, data logic errors and expression defects are systematically mined and corrected, high-quality answers are generated, and accurate data is provided for model training. The optimized data contains reflection trajectories and high-verification score answers, which help the model learn problem-solving logic and improve its ability to handle complex tasks. The second-level answers are verified by the teacher model, and the score improvement of ≥10% is used as the screening standard. This can objectively and accurately evaluate the quality improvement effect of the optimized data, avoid the limitations or deviations that may exist in data generated by relying solely on the reflection model, ensure that the data entering the next stage of training has high validity and reliability, reduce the risk of poor model training results due to poor data quality, and ensure the stability and reliability of the entire model training process. The combination of reflection trajectories and second-level answers provides an important basis for the iterative upgrade of the model.

[0091] In an embodiment of the present invention, the specific steps include:

[0092] Step S41, confirm that the first-layer SFT data that has undergone preliminary processing is complete and in a standardized format, and organize it into a form suitable for input into the reflection model, such as dividing it by batches or classifying it by question type, determine the currently available reflection model with the best performance, such as GPT-4o, etc., and check whether the model operating environment is stable; in the operation interface or data processing platform of the reflection model, find the template loading function module, and import a multi-round reflection prompt template containing error recognition instructions, correction path generation instructions, and high-quality answer generation instructions; input the prepared first-layer SFT data into the reflection model with the loaded template in batches or one by one, start the model running program, and start processing the data.

[0093] Step S42, wait for the reflection model to finish running, obtain the optimized data containing the reflection track text and the second-level answer, ensure that the reflection track and the answer are bound by a pre-set special delimiter, so as to facilitate the subsequent splitting and processing of the data; use the special delimiter to separate the reflection track text and the second-level answer in the optimized data, and store them in different data areas or files respectively, so as to facilitate the subsequent verification and evaluation of the answer. The separated second-level answer is input into the pre-trained teacher model, and the answer is scored according to the preset scoring criteria, and the original score of each answer before verification is recorded. The teacher model's score for the second-level answer is compared with the original score of the answer in the first-level SFT data, and the score improvement of each answer is calculated; the second-level answers with a score improvement of ≥10% and their corresponding reflection track text are screened out, and the data that does not meet the conditions are eliminated. The qualified reflection track text and the second-level answer are reintegrated in the original binding format to generate the final usable optimization data, and are properly stored.

[0094] In a preferred embodiment of the present invention, the above step S5: mixing the first-level optimized data and the second-level optimized data to generate a comprehensive training set, wherein the second-level data carries auxiliary information of the reflection trajectory to enhance the model's understanding and reasoning ability for complex tasks, may include:

[0095] Step S51, mixing the first-level optimized data and the second-level optimized data in a ratio of 6:4 to generate a comprehensive training set;

[0096] Step S52: A dynamic attention mask mechanism is used during training to force the model to analyze the reflection trajectory content before generating the answer;

[0097] In step S53, if the KL divergence between the second-level data and the original data is ≥ 2.0, the data is downsampled to 50% of the original data volume, and the feature distribution is aligned through the adversarial training module.

[0098] In this embodiment, the first and second levels of optimized data are mixed in a 6:4 ratio, integrating the stability of the first-level data with the high-quality advantages of the second-level data to create a rich and balanced data structure, ensuring model training stability, increasing the training limit, and avoiding training inefficiencies. A dynamic attention masking mechanism is used to force the model to analyze the reflection trajectory before generating an answer, helping the model learn the answer optimization logic. This breaks the limitation of traditional methods that only learn results, mining key information, and improving logical reasoning and problem-solving capabilities. This allows the model to draw on reflection experience and output higher-quality answers when faced with complex tasks. KL divergence is calculated to determine data distribution differences. Data with large differences is downsampled to 50% of the original data volume and feature distributions are aligned to avoid training instability. A dynamic adaptation strategy ensures that the model processes the appropriate data, improving training stability and reliability, enhancing the model's adaptability to different data distributions, and broadening its application scenarios. The three steps, from data integration, training mechanism enhancement, to data distribution adaptation, form a complete training optimization system. The comprehensive training set provides high-quality data, dynamic masking facilitates deep learning, and data adaptation ensures stability, all of which improve model accuracy, generalization, and robustness.

[0099] In an embodiment of the present invention, the specific steps include:

[0100] Step S51: Check the first-level optimized data and the second-level optimized data that have been processed. According to the mixing ratio of 6:4, the total amount of the first-level optimized data and the second-level optimized data is counted, and the amount of data to be used in the mixing process is calculated according to the ratio. For example, if the total amount of the first-level optimized data is 1000 and the total amount of the second-level optimized data is 800, then 600 first-level optimized data and 400 second-level optimized data are used for mixing. The calculated amount of data is extracted from the corresponding storage location, and the first-level optimized data and the second-level optimized data are mixed together by row-by-row or batch merging to generate a comprehensive training set. During the mixing process, pay attention to maintaining the original order and structure of the data to avoid data confusion. The generated comprehensive training set is spot-checked to check whether the data mixing ratio is accurate, whether the data content is complete, and whether there is any duplication or loss. If any problems are found, the mixing process is adjusted in time and the comprehensive training set is regenerated.

[0101] Step S52, in the model training platform or framework, find the setting options related to the attention mechanism, confirm that the training environment supports the operation of the dynamic attention mask mechanism, install and configure the required plug-ins or library files; according to the characteristics of the reflection trajectory content and the model training requirements, formulate specific rules for the dynamic attention mask, for example, set it before the model generates the answer, forcing the attention to be focused on specific keywords, key sentences or paragraphs in the reflection trajectory text, and the model will give priority to parsing this part of the content. The defined dynamic attention mask rules are embedded in the training code or configuration file of the model. During the training process, the mechanism can automatically take effect and regulate the attention allocation of the model. The comprehensive training set is input into the model configured with the dynamic attention mask mechanism, and the model training is started according to the preset training parameters and number of iterations. During the training process, the model's parsing of the reflection trajectory content and the training effect are monitored in real time. If necessary, the mask rules can be fine-tuned.

[0102] Step S53, using a special statistical tool or algorithm, calculate the KL divergence value between the second-level data and the original data. During the calculation process, the format and dimension of the data meet the calculation requirements, and the KL divergence result is accurately obtained; the calculated KL divergence value is compared with the threshold value 2.0. If the KL divergence is ≥2.0, it is determined that the second-level data needs to be downsampled; if the KL divergence is <2.0, the downsampling step is skipped and the feature alignment link is directly entered. When downsampling is required, the second-level data is randomly sampled according to 50% of the original data volume or downsampled according to specific rules (such as stratified sampling by data category) to ensure that the downsampled data can retain the main features and distribution patterns of the original data. The downsampled second-level data (or the un-downsampled second-level data) is feature aligned with the original data through the adversarial training module. During the adversarial training process, the parameters of the adversarial training module are adjusted so that the feature distribution of the second-level data gradually approaches the feature distribution of the original data until a satisfactory alignment effect is achieved.

[0103] In a preferred embodiment of the present invention, step S6: injecting historical context information into each interaction round of the multi-round dialogue based on the comprehensive training set, and generating a tree-like divergent dialogue path based on the optimization process of the aforementioned steps to enhance the logical coherence and diversity of the multi-round dialogue data, may include:

[0104] Step S61: Based on the comprehensive training set, in a multi-round dialogue scenario, inject the historical dialogue context into the question generation process of step S1 to generate questions containing associated references;

[0105] Step S62: For each round of generated tree-like dialogue paths, retain the top two reflection trajectory branches and use a graph neural network to model the logical dependencies between the paths.

[0106] In step S63, the tree path data is iteratively enhanced through the two-level optimization process of steps S2-S5, and finally a multi-round dialogue training set with long-range coherence is generated.

[0107] In this embodiment of the present invention, historical conversation context is injected into the question generation process, closely linking newly generated questions with past conversations. This builds a coherent conversational logic chain, improves the coherence and naturalness of the conversation, and enables the model to better understand the context during multi-turn interactions and generate answers that better meet user expectations. By retaining the top two scoring reflection trajectory branches, the most valuable conversation extension directions are screened, preventing redundant paths from interfering with model learning and improving data validity and relevance. Using a graph neural network to model the logical dependencies between paths, it is possible to deeply explore the underlying logical structure within the conversation paths and strengthen the model's logical reasoning capabilities. The tree-structured path data is iteratively enhanced through a two-level optimization process in steps S2-S5. This refines the data through multiple steps, including answer scoring screening, reflection optimization, and data blending adjustment. Through multiple rounds of iteration, the quality and diversity of the data are continuously improved, providing higher-quality samples for model training. High-quality training data can improve the model's performance in multi-turn conversation tasks, enabling it to handle various complex multi-turn conversation scenarios in real-world applications, output accurate, fluent, and logical answers, and enhance user experience and model practicality.

[0108] In an embodiment of the present invention, the specific steps include:

[0109] Step S61 confirms the completion of the comprehensive training set. The system familiarizes itself with the business requirements and application characteristics of multi-round conversation scenarios, organizes historical conversation context data, and integrates and injects it one by one according to established rules. For example, for each newly generated question scenario, the conversation content from previous relevant rounds is integrated into the question generation input using specific identifiers or formats. The question generation program is initiated. Based on the input after injecting the historical conversation context, a large language model or related algorithm is used to generate questions containing related references. During the generation process, the relevance and rationality of the questions with the historical conversation context are checked in real time, and any questions that do not meet the requirements are adjusted or regenerated.

[0110] In step S62, in each round of dialogue, a tree-like dialogue path is generated using a model or algorithm based on the generated questions containing related references. This ensures that each path covers multiple possible answers and extension directions, simulating the diversity found in real conversation scenarios. For each generated tree-like dialogue path, the reflection trajectories in each branch are scored according to a preset scoring criteria. The top two reflection trajectories are selected from all branches and retained, while the remaining branches are discarded. Using graph neural network technology, the retained reflection trajectories are used as nodes, and the logical relationships between the branches are analyzed. Edges connecting the nodes are constructed to model the logical dependencies between the paths.

[0111] In step S63, the tree path data is sequentially substituted into the two-level optimization process of steps S2-S5. First, the answers in the tree path data are scored in terms of accuracy, coherence, and information content according to the scoring criteria of step S2. Then, according to the rules of step S3, the answer pairs whose score differences meet the requirements are screened to form the primary DPO preference data, and the high-quality first-level SFT data are marked. Then, the reflection model and Prompt template of step S4 are used to deeply optimize the data. Finally, according to the method of step S5, the data is mixed, the dynamic attention mask mechanism is applied, and the data is adjusted according to the KL divergence.

[0112] The two-level optimization process is repeated to perform multiple rounds of iterative enhancement on the tree path data. After each round of iteration, the quality improvement effect of the data is evaluated, and the parameters and strategies in the optimization process are adjusted based on the evaluation results. After multiple rounds of iterative optimization, when the tree path data reaches the expected quality standards and has long-term coherence, it is organized and output to generate the final multi-round dialogue training set for subsequent model training and optimization.

[0113] In a preferred embodiment of the present invention, the above step S7: when it is detected during the online service that the semantic similarity is lower than a preset threshold, extracting the error dialogue chain and analyzing the error type, reversely adjusting the homologous data generation strategy of step S1, the scoring threshold of step S2, and the reflection model configuration of step S4, to generate a closed-loop mechanism for iterative optimization of cross-level data quality, may include:

[0114] Step S71: When the online service detects that the semantic similarity of user feedback is lower than a threshold, it extracts the reflection trajectory data of the error dialogue chain and analyzes the error type distribution;

[0115] Step S72, based on the error type: the number of homologous answers generated, the scoring threshold and the reflection model configuration are dynamically adjusted to form a closed-loop iterative mechanism for cross-level data optimization.

[0116] In this embodiment of the present invention, by monitoring the semantic similarity of user feedback, it is possible to detect instances where model outputs do not align with user expectations. When the semantic similarity falls below a threshold, the model promptly extracts the reflection trajectory data for the error dialogue chain and analyzes the distribution of error types, accurately locating specific issues such as logical errors and factual errors in the model's answering process. Based on the error type distribution, the number of homologous answers generated, the scoring threshold, and the reflection model configuration are dynamically adjusted. This flexible adjustment mechanism based on actual error conditions allows model training and optimization strategies to better adapt to different types of questions. Steps S71 and S72 collaborate to form a closed-loop iterative mechanism for cross-level data optimization, achieving a complete process from problem detection and analysis to strategy adjustment and optimization. In each iteration, the model is improved based on the issues discovered in the previous round. Through this continuous cycle, the quality of the comprehensive training set is continuously improved. As iterations progress, the model's accuracy in online services gradually increases, providing users with more accurate and more targeted answers, improving service quality and user experience.

[0117] In an embodiment of the present invention, the specific steps include:

[0118] Step S71: During the operation of the online service, user input and model output are continuously collected. Natural language processing algorithms are used to calculate the semantic similarity between user feedback and model responses in real time. When the semantic similarity is detected to be below a preset threshold, the error dialogue chain processing program is immediately initiated, extracting all relevant reflection trajectory data from the dialogue chain from the service log or database, including intermediate results and correction records during the answer generation process. Using a pre-trained error classification model, the extracted reflection trajectory data is analyzed sentence by sentence and paragraph by paragraph, classifying error types into logical errors, factual errors, unclear expressions, and other categories. The distribution of each error type in the entire error dialogue chain is then calculated.

[0119] In step S72, based on the error type distribution results obtained in step S71, the preset adjustment rule table is compared to identify the specific parameters that need to be adjusted. The number of generated homologous answers is increased according to the rules. For example, for every 10% increase in the error rate, two answers are added to the original number of generated answers. Regarding step S2, the scoring threshold is modified based on the error type. For example, if the proportion of logical errors is high, the weight of logical coherence in the scoring criteria is increased to 50%, and the weights of other dimensions are adjusted accordingly. Based on the error situation, the reflection model is reconfigured. For example, if there are many factual errors, the external knowledge base search function is enabled, or the prompt template content is adjusted.

[0120] After completing the parameter adjustment, the updated parameters are applied to the subsequent data generation and optimization process, and the homologous answers are regenerated starting from step S1. The entire data optimization process is repeated. In the new round of iterations, indicators such as online service accuracy are continuously monitored to evaluate the adjustment effect. If the expected effect is not achieved, steps S71 and S72 are executed again to continuously optimize the parameters and form a closed-loop iterative mechanism for continuous improvement.

[0121] like Figure 2 As shown, an embodiment of the present invention further provides a system 20 for optimizing AI training data at two levels based on a large model, comprising:

[0122] Acquisition module 21 is configured to utilize the random generation capability of the large language model to generate multiple homologous answers for each question and instruction in the historical data using a high-random hyperparameter configuration; based on the multiple homologous answers, a pre-trained teacher model is used to score them under a low-random hyperparameter configuration to generate preferred data pairs, and answers with scores above a threshold are selected as high-quality supervised fine-tuning data;

[0123] The first-level optimization module 22 is used to match the homologous answers generated in step S1 with the corresponding scoring results in step S2, construct primary preference data for direct preference optimization, and combine it with the screened high-quality SFT data to form the first-level optimization data;

[0124] A second-level optimization module 23 is configured to perform a multi-dimensional quality analysis on the high-quality SFT data through a reflection model, and generate second-level optimization data including a reflection trajectory, wherein the reflection trajectory includes a correction logic for the original answer and a quality improvement path;

[0125] The processing module 24 is used to mix the first-level optimized data and the second-level optimized data to generate a comprehensive training set, wherein the second-level data carries auxiliary information of the reflection trajectory to enhance the model's understanding and reasoning ability of complex tasks; based on the comprehensive training set, historical context information is injected into each round of interaction in the multi-round dialogue, and a tree-like divergent dialogue path is generated based on the optimization process of the aforementioned steps to enhance the logical coherence and diversity of the multi-round dialogue data; when the semantic similarity is detected to be lower than the preset threshold during the online service process, the error dialogue chain is extracted and the error type is analyzed, and the homologous data generation strategy of step S1, the scoring threshold of step S2 and the reflection model configuration of step S4 are reversely adjusted to generate a closed-loop mechanism for iterative optimization of cross-level data quality.

[0126] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for optimizing AI training data at two levels based on a large model, characterized in that: The method comprises: S1: Leveraging the random generation capabilities of large language models, for each question and instruction in historical data, multiple homologous answers are generated through highly random hyperparameter configurations. S2: Based on multiple homologous answers, a pre-trained teacher model is used to score them under a low-randomness hyperparameter configuration to generate preferred data pairs, and answers with scores above a threshold are selected as high-quality supervised fine-tuning data; S3: Match the homologous answers generated in step S1 with the corresponding scoring results in step S2 to construct primary preference data for direct preference optimization, and combine them with the screened high-quality SFT data to form the first-level optimization data; S4: Based on high-quality SFT data, a multi-dimensional quality analysis is conducted on it through a reflective model to generate second-level optimization data containing reflective trajectories. The reflective trajectories include the correction logic and quality improvement path for the original answers. S5: Mix the first-level optimized data and the second-level optimized data to generate a comprehensive training set, where the second-level data carries auxiliary information of the reflection trajectory to enhance the model's understanding and reasoning ability for complex tasks; S6: Based on the comprehensive training set, historical context information is injected into each interaction round of the multi-round dialogue. Based on the optimization process of the above steps, a tree-like divergent dialogue path is generated to enhance the logical coherence and diversity of the multi-round dialogue data. S7: When the semantic similarity is detected to be lower than the preset threshold during the online service process, the error dialogue chain is extracted and the error type is analyzed. The homologous data generation strategy of step S1, the scoring threshold of step S2, and the reflection model configuration of step S4 are adjusted in reverse to generate a closed-loop mechanism for iterative optimization of cross-level data quality.

2. The method for optimizing AI training data based on a large model at two levels according to claim 1, characterized in that: S1: Leveraging the randomness generation capabilities of large language models, we generate multiple homologous answers for each question and instruction in historical data through highly random hyperparameter configurations, including: Based on each question and instruction in the historical data, adjust the generation hyperparameters of the large language model to a high randomness mode to generate at least 3 homologous answers; Homologous answers must meet the cosine similarity of ≥ 0.75 with the original data answers, and semantic consistency must be ensured through fixed instruction keywords; The repetition penalty mechanism is disabled during the generation process, and the generated length is limited to no more than 150% of the original answer.

3. The method for optimizing AI training data based on a large model at two levels according to claim 2, characterized in that: S2: Based on multiple homologous answers, a pre-trained teacher model is used to score them under a low-randomness hyperparameter configuration to generate preferred data pairs. Answers with scores above a threshold are selected as high-quality supervised fine-tuning data, including: Input the same source answers into the pre-trained teacher model, switch to low randomness mode, and score each answer based on the preset scoring template; The scoring template scores based on three dimensions: accuracy, coherence, and information content, generating a normalized score of 0-100; The answer pairs with a score difference of ≥15 points were screened to form the primary DPO preference data, and the answers with a score of ≥85 under a single question were marked as the first-level SFT data.

4. The method for optimizing AI training data based on a large model at two levels according to claim 3, characterized in that: S3: Match the homologous answers generated in step S1 with the corresponding scoring results in step S2 to construct primary preference data for direct preference optimization, and combine it with the screened high-quality SFT data to form the first-level optimization data, including: The primary DPO data was denoised to remove answer pairs with a repetition rate of ≥80% for the same question; The first-level SFT data is sorted by score, the top 30% answers are selected, and data enhancement is performed through synonym replacement and sentence reorganization to generate an expanded first-level optimized dataset.

5. The method for optimizing AI training data based on a large model at two levels according to claim 4, characterized in that: S4: Based on high-quality SFT data, a multi-dimensional quality analysis is performed on it through a reflection model to generate second-level optimization data containing reflection trajectories. The reflection trajectories include the correction logic and quality improvement path for the original answers, including: Inputting the first-level SFT data into the reflection model, loading a multi-round reflection prompt template, wherein the template includes error recognition instructions, correction path generation instructions, and high-quality answer generation instructions; Generate optimized data containing reflection track text and second-level answers, where the reflection track and answers are bound by special separators, and the second-level answers must pass the teacher model verification score improvement of ≥10%.

6. The method for optimizing AI training data based on a large model at two levels according to claim 5, characterized in that: S5: Mix the first-level optimized data with the second-level optimized data to generate a comprehensive training set. The second-level data carries auxiliary information of the reflection trajectory to enhance the model's understanding and reasoning ability for complex tasks, including: The first-level optimized data and the second-level optimized data are mixed in a ratio of 6:4 to generate a comprehensive training set; A dynamic attention mask mechanism is used during training to force the model to analyze the content of the reflection trajectory before generating the answer; If the KL divergence between the second-level data and the original data is ≥ 2.0, it is downsampled to 50% of the original data volume, and the feature distribution is aligned through the adversarial training module.

7. The method for optimizing AI training data based on a large model at two levels according to claim 6, characterized in that: S6: Based on the comprehensive training set, historical context information is injected into each interaction round of the multi-round dialogue. Based on the optimization process of the above steps, a tree-like divergent dialogue path is generated to enhance the logical coherence and diversity of the multi-round dialogue data, including: Based on the comprehensive training set, in a multi-round dialogue scenario, the historical dialogue context is injected into the question generation phase of step S1 to generate questions containing related references; For each round of generated tree-like dialogue paths, the top two reflection trajectory branches are retained, and the logical dependencies between the paths are modeled using a graph neural network. The tree path data is iteratively enhanced through the two-level optimization process of steps S2-S5, and ultimately a multi-round dialogue training set with long-range coherence is generated.

8. A system for optimizing AI training data at two levels based on a large model, the system implementing the method according to any one of claims 1 to 7, characterized in that: include: The acquisition module is used to leverage the random generation capabilities of the large language model to generate multiple homologous answers for each question and instruction in the historical data through highly random hyperparameter configuration; Based on multiple homologous answers, a pre-trained teacher model is used to score them under a low-randomness hyperparameter configuration to generate preferred data pairs, and answers with scores above a threshold are selected as high-quality supervised fine-tuning data. The first-level optimization module is used to match the homologous answers generated in step S1 with the corresponding scoring results in step S2, construct primary preference data for direct preference optimization, and combine it with the screened high-quality SFT data to form the first-level optimization data; The second-level optimization module is used to perform multi-dimensional quality analysis on high-quality SFT data through a reflection model to generate second-level optimization data containing reflection trajectories, wherein the reflection trajectories include correction logic and quality improvement paths for the original answers; A processing module is used to mix the first-level optimized data and the second-level optimized data to generate a comprehensive training set, where the second-level data carries auxiliary information of the reflection trajectory to enhance the model's understanding and reasoning ability for complex tasks; Based on a comprehensive training set, historical context information is injected into each round of interaction in multi-round dialogues, and a tree-like divergent dialogue path is generated based on the optimization process of the aforementioned steps to enhance the logical coherence and diversity of multi-round dialogue data. When the semantic similarity detected in the online service process is lower than the preset threshold, the error dialogue chain is extracted and the error type is analyzed. The homologous data generation strategy in step S1, the scoring threshold in step S2, and the reflection model configuration in step S4 are adjusted in reverse, generating a closed-loop mechanism for iterative optimization of cross-level data quality.

9. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images