A method for double-level optimization of AI training data based on a large model

Abstract

The application provides a method for double-level optimization of AI training data based on a large model, and relates to the field of artificial intelligence, which comprises the following steps: using the randomness generation capability of a large language model, generating multiple homologous answers for each question and instruction in historical data through high-randomness hyperparameter configuration; based on the multiple homologous answers, scoring through a pre-trained teacher model under low-randomness hyperparameter configuration to generate preference data pairs, and screening answers with scores higher than a threshold as high-quality supervised fine-tuning data; corresponding the homologous answers generated in step S1 to the corresponding scoring results in step S2 to construct primary preference data for direct preference optimization, and combining the screened high-quality SFT data to form first-level optimization data. Through the double-level optimization architecture and the dynamic adjustment mechanism, the application improves the AI training efficiency under the premise of ensuring data quality, and provides reliable data support for large-scale model training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a method for double-level optimization of AI training data based on a large model. BACKGROUND

[0002] Nowadays, AI training requires a large amount of data, and more emphasis is placed on data quality. However, manual annotation (such as RLHF, Reinforcement Learning from Human Feedback) faces the problems of high cost and low efficiency, and the use of AI to improve data quality has gradually become an optional solution for the industry. The existing methods for improving data using AI, such as RLAIF (Reinforcement Learning from AI Feedback), are still limited to the capabilities of the model itself, lack of quality improvement levels, and the RLAIF method is still limited to the optimization of single-round task data. We combine Reflection with RLAIF based on model reflection technology, first construct primary preference data consistent with the original data distribution, and then further construct secondary preference data with better quality but more obvious differences from the original data distribution based on the primary preference data. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a method for double-level optimization of AI training data based on a large model, which improves the AI training efficiency under the premise of ensuring the data quality and provides a reliable data foundation support for large-scale model training through the innovative double-level optimization architecture and dynamic adjustment mechanism.

[0004] To solve the above technical problems, the technical solution of the present application is as follows:

[0005] In a first aspect, a method for double-level optimization of AI training data based on a large model is provided, the method comprising:

[0006] S1: using the randomness generation capability of a large language model, generating multiple homologous answers for each question and instruction in the historical data through high-randomness hyperparameter configuration;

[0007] S2: based on the multiple homologous answers, scoring through a pre-trained teacher model under low-randomness hyperparameter configuration to generate preference data pairs, and selecting answers with scores higher than a threshold as high-quality supervised fine-tuning data;

[0008] S3: corresponding the homologous answers generated in step S1 to the corresponding scoring results in step S2, constructing primary preference data for direct preference optimization, and combining the filtered high-quality SFT data to form first-level optimization data;

[0009] S4: Based on high-quality SFT data, multi-dimensional quality analysis is performed on it through a reflection model to generate second-level optimization data containing reflection trajectories, which include correction logic and quality improvement paths for the original answers;

[0010] S5: The first and second levels of optimization data are mixed to generate a comprehensive training set, where the second level of data carries auxiliary information of reflection trajectories to enhance the model's understanding and reasoning ability for complex tasks;

[0011] S6: Based on the comprehensive training set, historical context information is injected in each round of interaction in the multi-round dialogue, and a tree-like divergent dialogue path is generated based on the optimization process of the foregoing steps to enhance the logical coherence and diversity of multi-round dialogue data;

[0012] S7: When the semantic similarity is detected to be lower than the preset threshold in the online service process, the error dialogue chain is extracted and the error type is analyzed, and the homologous data generation strategy of step S1, the scoring threshold of step S2 and the reflection model configuration of step S4 are adjusted reversely to generate a closed-loop mechanism for iterative optimization of cross-level data quality.

[0013] Further, S1: Utilizing the randomness generation capability of large language models, for each question and instruction in historical data, generate multiple homologous answers through high-randomness hyperparameter configuration, including:

[0014] Based on each question and instruction in historical data, adjust the generation hyperparameters of the large language model to the high-randomness mode to generate at least 3 homologous answers;

[0015] Homologous answers need to meet the cosine similarity ≥0.75 with the original data answers, and ensure semantic consistency through fixed instruction keywords;

[0016] Disable the repetition penalty mechanism during generation, and limit the generation length to no more than 150% of the original answer.

[0017] Further, S2: Based on multiple homologous answers, score through a pre-trained teacher model under low-randomness hyperparameter configuration to generate preferred data pairs, and select answers with scores higher than the threshold as high-quality supervised fine-tuning data, including:

[0018] Input the homologous answers into the pre-trained teacher model, switch to the low-randomness mode, and score each answer based on the preset scoring template;

[0019] The scoring template scores from three dimensions of accuracy, coherence, and information quantity to generate a normalized score of 0-100;

[0020] The answer with a score difference of ≥ 15 points is used to constitute the primary DPO preference data, and the answer with a score of ≥ 85 under a single question is marked as the first-level SFT data.

[0021] Further, S3: corresponding the homologous answers generated in step S1 to the corresponding scoring results in step S2, constructing the primary preference data for direct preference optimization, and combining the screened high-quality SFT data to form the first-level optimization data, including:

[0022] The primary DPO data is denoised to remove answer pairs with a repetition rate of ≥ 80% under the same question.

[0023] The first-level SFT data is sorted by score, and the top 30% answers are selected, and data augmentation is performed through synonym replacement and sentence restructuring to generate an expanded first-level optimization data set.

[0024] Further, S4: based on high-quality SFT data, a multi-dimensional quality analysis is performed on it through a reflection model to generate second-level optimization data containing reflection tracks, and the reflection tracks include correction logic and quality improvement paths for original answers, including:

[0025] The first-level SFT data is input into the reflection model, and multiple rounds of reflection Prompt templates are loaded, which contain error identification instructions, correction path generation instructions, and high-quality answer generation instructions.

[0026] Generate optimization data containing reflection track text and second-level answers, where the reflection track and the answer are bound by a special separator, and the second-level answer needs to be verified by a teacher model with a score improvement of ≥ 10%.

[0027] Further, S5: mix the first-level optimization data and the second-level optimization data to generate a comprehensive training set, where the second-level data carries auxiliary information of the reflection track to enhance the model's understanding and reasoning ability for complex tasks, including:

[0028] Mix the first-level optimization data and the second-level optimization data in a ratio of 6:4 to generate a comprehensive training set.

[0029] During training, a dynamic attention mask mechanism is used to force the model to analyze the reflection track content before generating the answer.

[0030] If the KL divergence of the second-level data and the original data is ≥ 2.0, then downsample it to 50% of the original data amount, and align the feature distribution through the adversarial training module.

[0031] Further, S6: Based on the comprehensive training set, historical context information is injected in each round of interaction in the multi-round dialogue, and the tree-like divergent dialogue path is generated based on the optimization process of the foregoing steps to enhance the logical coherence and diversity of the multi-round dialogue data, including:

[0032] Based on the comprehensive training set, the historical dialogue context is injected into the question generation link in step S1 in the multi-round dialogue scenario, and a question containing a related reference is generated;

[0033] For each round of generated tree-like dialogue path, the top 2 reflection track branches are retained, and the logical dependency relationship between the paths is modeled through a graph neural network;

[0034] The tree-like path data is iteratively enhanced through the double-level optimization process of steps S2-S5, and finally a multi-round dialogue training set with long-range coherence is generated.

[0035] Further, S7: When the semantic similarity is detected to be lower than the preset threshold in the online service process, the error dialogue chain is extracted and the error type is analyzed, and the homologous data generation strategy of step S1, the score threshold of step S2 and the reflection model configuration of step S4 are adjusted reversely, to generate a closed-loop mechanism for iterative optimization of data quality across levels, including:

[0036] When the online service detects that the semantic similarity of user feedback is lower than the threshold, the reflection track data of the error dialogue chain is extracted, and the error type distribution is analyzed;

[0037] According to the error type: the number of homologous answer generation, the score threshold and the reflection model configuration are dynamically adjusted, and a closed-loop iterative mechanism for cross-level data optimization is formed.

[0038] The second aspect is a system for double-level optimization of AI training data based on a large model, including:

[0039] The acquisition module is configured to utilize the randomness generation capability of the large language model to generate multiple homologous answers for each question and instruction in the historical data through high-randomness hyperparameter configuration; based on the multiple homologous answers, the pre-trained teacher model is used to score under low-randomness hyperparameter configuration to generate preferred data pairs, and the answers with scores higher than the threshold are selected as high-quality supervised fine-tuning data;

[0040] The first-level optimization module is configured to correspond the homologous answers generated in step S1 to the corresponding score results in step S2, construct primary preference data for direct preference optimization, and combine the filtered high-quality SFT data to form first-level optimization data;

[0041] The second-level optimization module is configured to perform multi-dimensional quality analysis on the high-quality SFT data by using a reflection model, and generate second-level optimization data containing a reflection track, wherein the reflection track includes a correction logic for the original answer and a quality improvement path.

[0042] The processing module is configured to mix the first-level optimization data and the second-level optimization data to generate a comprehensive training set, wherein the second-level data carries auxiliary information of the reflection track, and is used to enhance the understanding and reasoning ability of the model for complex tasks; based on the comprehensive training set, historical context information is injected in each round of interaction of the multi-round dialogue, and a tree-like divergent dialogue path is generated based on the foregoing optimization processes, so as to enhance the logical coherence and diversity of the multi-round dialogue data; when the semantic similarity is detected to be lower than a preset threshold in the online service process, an error dialogue chain is extracted and an error type is analyzed, and the homologous data generation strategy of step S1, the score threshold of step S2 and the reflection model configuration of step S4 are adjusted reversely, so as to generate a closed-loop mechanism for iterative optimization of cross-level data quality.

[0043] In a third aspect, a computing device includes:

[0044] one or more processors;

[0045] a memory device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0046] In a fourth aspect, a computer-readable storage medium stores a program, when the program is executed by a processor, the method is implemented.

[0047] The above-mentioned scheme of the present application at least has the following beneficial effects:

[0048] The high-randomness super parameter is used to generate homologous answers, ensure that the new data is similar to the original data distribution, pre-train the teacher model in a low-randomness mode to reduce uncertainty, and ensure the stability of data distribution and quality by two methods to reduce the risk of training fluctuation, making the model training more reliable and predictable. The first level integrates the primary preference data and high-quality SFT data through denoising and data enhancement, and the second level uses the reflection model to deeply mine and correct the answer problems to achieve quality breakthrough. After mixing the two layers of data, a comprehensive training set with stability and high quality is formed to provide high-quality resources for model training. The dynamic attention mask mechanism forces the model to analyze the reflection trajectory, learn the answer optimization logic, and improve the logical reasoning and problem solving ability; the history context generation tree is injected in the multi-round dialogue to enhance the coherence and diversity of the dialogue; the closed-loop iterative optimization mechanism adjusts the strategy in real time according to the online service error feedback, continuously optimizes the data, and promotes the model to output more accurate and smooth answers to improve user experience. The standardized and automated data optimization process runs through the whole process from data generation, scoring, optimization to iterative adjustment, reducing the dependence on manual annotation and manual optimization. According to the error type, the data generation and optimization strategy is flexibly adjusted to adapt the model training to single-round and multi-round tasks, as well as different types of problems and data distribution. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flowchart of a method for double-level optimization of AI training data based on a large model provided by an embodiment of the present application.

[0050] Figure 2 is a schematic diagram of a system for double-level optimization of AI training data based on a large model provided by an embodiment of the present application.

[0051] Figure 3 is a schematic diagram of a scheme for double-level optimization of AI training data based on a large model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0053] As Figure 1 shown, an embodiment of the present application proposes a method for double-level optimization of AI training data based on a large model, which includes the following steps:

[0054] Step S1: using the randomness generation capability of the large language model, for each question and instruction in the historical data, a plurality of homologous answers are generated through high-randomness hyperparameter configuration;

[0055] Step S2: based on the plurality of homologous answers, scoring is performed through the pre-trained teacher model under low-randomness hyperparameter configuration to generate preference data pairs, and answers with scores higher than a threshold value are selected as high-quality supervised fine-tuning data;

[0056] Step S3: the homologous answers generated in step S1 are corresponded to the corresponding scoring results in step S2 to construct primary preference data for direct preference optimization, and the high-quality SFT data selected are combined to form first-level optimization data;

[0057] Step S4: based on the high-quality SFT data, multi-dimensional quality analysis is performed on the high-quality SFT data through a reflection model to generate second-level optimization data containing a reflection track, and the reflection track includes a correction logic and a quality improvement path for the original answer;

[0058] Step S5: the first-level optimization data and the second-level optimization data are mixed to generate a comprehensive training set, wherein the second-level data carries auxiliary information of the reflection track, and is used to enhance the understanding and reasoning ability of the model for complex tasks;

[0059] Step S6: based on the comprehensive training set, historical context information is injected in each round of interaction of the multi-round dialogue, and a tree-shaped divergent dialogue path is generated based on the optimization processes of the foregoing steps, so as to enhance the logical coherence and diversity of the multi-round dialogue data;

[0060] Step S7: when detecting that the semantic similarity is lower than a preset threshold value in the online service process, an error dialogue chain is extracted and an error type is analyzed, the homologous data generation strategy of step S1, the scoring threshold of step S2 and the reflection model configuration of step S4 are adjusted reversely, and a closed-loop mechanism for iterative optimization of data quality across levels is generated.

[0061] In the embodiment of the present application, homologous answers are generated through high-randomness configuration, high-quality data is selected in combination with a teacher model, data distribution is ensured to be stable, multi-level optimization data is constructed on this basis, and the mixed training set significantly improves the model training effect; the second-level data carries a reflection track, and with the aid of a dynamic attention mask mechanism, the model is helped to understand the answer thinking path, and the understanding and reasoning ability for complex tasks is improved; based on the comprehensive training set, historical context is injected to generate diversified dialogue paths, and the coherence and richness of the multi-round dialogue data are enhanced. The closed-loop mechanism detects data problems in real time, dynamically adjusts the generation strategy and configuration, continuously optimizes data quality, adapts to different application requirements, reduces manual annotation through an automated process, efficiently processes a large amount of data, greatly reduces costs in cold start and high labor cost scenarios, and improves data processing efficiency.

[0062] In a preferred embodiment of the present application, the above step S1: using the randomness generation capability of the large language model, for each question and instruction in the historical data, generating multiple homologous answers through high randomness hyperparameter configuration, can include:

[0063] Step S11, based on each question and instruction in the historical data, adjusting the generation hyperparameters of the large language model to a high randomness mode, generating at least 3 homologous answers;

[0064] Step S12, the homologous answers need to satisfy the cosine similarity ≥ 0.75 with the original data answers, and ensure semantic consistency through fixed instruction keywords;

[0065] Step S13, disable the repetition penalty mechanism during generation, and limit the generation length to no more than 150% of the original answer.

[0066] In an embodiment of the present application, by generating homologous answers through high randomness mode, the data distribution is ensured to be similar, the training instability risk is reduced, and the training reliability is improved; by cosine similarity and keyword constraint, while maintaining the diversity of answers, the semantic consistency is ensured, and the data training value is improved. Through multi-answer generation and disabling repetition penalty, the answer length is controlled, the language expression and problem solving ideas are enriched, and the standardized process is formed by clear parameters and rules, reducing manual intervention.

[0067] In an embodiment of the present application, the specific steps include:

[0068] Step S11, collect and organize historical data for model training, extract questions and instructions from them separately to form a data set to be processed; find the hyperparameter setting module in the large language model that controls the generation process, and adjust the generation hyperparameters to a high randomness mode. For each question and instruction in the historical data set, input the large language model in turn, and generate at least 3 answers each time the model runs. During the generation process, record all answers corresponding to each question and instruction to form a preliminary homologous answer set.

[0069] Step S12, use a text similarity calculation tool (such as a word vector-based calculation method) to calculate the cosine similarity between each homologous answer and the original data answer, and obtain the similarity value of each answer to the original answer; check whether each homologous answer contains a fixed instruction keyword, if not, it is determined that the answer is not consistent in semantics and is removed; select homologous answers with cosine similarity ≥ 0.75 and consistent semantics, remove answers that do not meet the conditions from the preliminary set, retain answers that meet the quality requirements, and form a screened answer set.

[0070] Step S13, in the generation setting of the large language model, find the option related to the repetition penalty, turn it off to ensure that the model will not be inhibited by repeated content when generating answers; measure the length of each answer in the answer set after screening one by one, and compare it with the original answer length. For answers whose length exceeds 150% of the original answer, delete the processing to make the length meet the requirements; if it cannot be deleted to the specified length, the answer is removed from the set. After the above processing, the final available homologous answer set is obtained.

[0071] In a preferred embodiment of the present application, the above step S2: based on a plurality of homologous answers, scoring by a pre-trained teacher model under a low randomness hyperparameter configuration, generating preference data pairs, and screening answers with scores higher than a threshold as high-quality supervised fine-tuning data, can include:

[0072] Step S21, input the homologous answers into the pre-trained teacher model, switch to the low randomness mode, and score each answer based on the preset scoring template;

[0073] Step S22, the scoring template scores from three dimensions of accuracy, coherence, and information amount to generate a normalized score of 0-100;

[0074] Step S23, screen the answer pairs with a score difference ≥15 to constitute primary DPO preference data, and mark the answers with a score ≥85 under a single question as first-level SFT data.

[0075] In the embodiment of the present application, inputting the homologous answers into the pre-trained teacher model and scoring in the low randomness mode combined with the preset template reduces interference; scoring from three key dimensions of accuracy, coherence, and information amount can comprehensively measure the quality of answers, accurately screen high-quality data, and lay a solid foundation for model training. By screening answer pairs with a score difference exceeding 15 to construct DPO data, the advantages and disadvantages are highlighted to help the model quickly learn the preference relationship; marking answers with a score ≥85 as first-level SFT data strengthens the processing capability of specific tasks and improves training efficiency; the primary DPO cooperates with the first-level SFT data to optimize the preference strategy and the processing of specific tasks respectively, enhances the accuracy, logical coherence, and information richness of the model's answers, and meets the application requirements.

[0076] In the embodiment of the present application, the specific steps include:

[0077] Step S21, a suitable pre-trained teacher model is selected, which should have the ability to evaluate the quality of the answer, for example, a language model with excellent performance in the relevant field and a parameter size suitable for the current task; the parameter setting interface of the pre-trained teacher model is entered, the generation mode of the model is switched to a low randomness mode to ensure the stability and consistency of the scoring process. The template for answer scoring is prepared in advance, and the scoring indicators and rules in the template are clearly defined to ensure the uniformity and repeatability of the scoring standard; the homologous answers are input into the pre-trained teacher model in turn according to the corresponding relationship of the questions or instructions, and each answer is scored using the preset scoring template, and the initial scoring result of each answer is recorded.

[0078] Step S22, according to the preset scoring template, the initial score of each answer is split into three dimensions of accuracy, coherence and information quantity, and the performance of each dimension is evaluated; for each dimension, according to the specific performance of the answer in the dimension, combined with the scoring standard, the corresponding score is given, for example, for accuracy, the matching degree of the answer content with the correct knowledge and requirements is scored; the coherence is scored according to the sentence logic and context connection; the information quantity is scored according to the amount of effective information provided by the answer. The scores of the three dimensions are normalized to make them uniform in the score interval of 0-100, and appropriate normalization methods such as linear transformation can be used to ensure the comparability of the scores of different answers; the normalized scores of the three dimensions are summarized to form the final 0-100 normalized score of each answer, and are recorded and saved.

[0079] Step S23, all answers are traversed, and for the answers under the same question or instruction, the score difference between any two answers is calculated, and the answer pair with a score difference ≥15 points is selected as the primary DPO preference data and is sorted and saved; similarly, for the answers under the same question or instruction, the answers with a score ≥85 under a single question are selected, and these answers are marked as first-level SFT data and are separately classified and stored. The selected primary DPO preference data and the marked first-level SFT data are sorted according to a certain format, and a data file is output for subsequent model training, ensuring the standardization and usability of the data.

[0080] In a preferred embodiment of the present application, the above step S3: corresponding the homologous answers generated in step S1 with the corresponding scoring results in step S2, constructing the primary preference data for direct preference optimization, and combining the screened high-quality SFT data to form the first-level optimization data, can include:

[0081] Step S31, denoising processing is performed on the primary DPO data to remove the answer pairs with a repetition rate ≥80% under the same question;

[0082] Step S32, the first level SFT data is sorted according to the score, the top 30% answers are selected, and data enhancement is carried out through synonym replacement and sentence restructuring to generate an expanded first level optimized data set.

[0083] In the embodiment of the application, the primary DPO data is denoised, and the answer pairs with a repetition rate of ≥80% under the same question are removed, which can effectively eliminate redundant information. After the first level SFT data is sorted according to the score, the top 30% answers are selected, which ensures that the selected data is a high-quality sample, and lays a good foundation for model supervision fine-tuning. On this basis, data enhancement is carried out through synonym replacement and sentence restructuring, which retains the core semantics of the answers while increasing the diversity of data expression. The optimized primary DPO data and the expanded first level SFT data work together to provide better data for model training, and the two work together to help the model achieve more accurate optimization in the training process, improve the comprehensive performance of the model in answering accuracy, logicality and dealing with complex tasks, and make it more stable in completing tasks in practical applications.

[0084] In the embodiment of the application, the specific steps include:

[0085] Step S31, the primary DPO data is grouped according to the question or instruction to ensure that the data in the same group corresponds to the same question or instruction; for each group of data, the repetition rate between the answer pairs is calculated, which can be determined by comparing the text content of the answers and counting the proportion of the same text segment. After traversing each group of data, the answer pairs with a repetition rate of ≥80% are found and removed from the primary DPO data. After the screening and removal operation is completed, the remaining primary DPO data is arranged to prepare for subsequent use.

[0086] Step S32, according to the score of each answer in the first level SFT data, all answers are sorted from high to low, and according to the sorting result, the top 30% answers are selected as high-quality samples in the first level SFT data; for the selected high-quality answers, the text content is analyzed sentence by sentence to find replaceable words, and the original words are replaced with synonyms using a synonym dictionary or related tools to make the answer expression more diverse. The sentence structure of the answer after synonym replacement is adjusted, such as changing the active sentence to the passive sentence, splitting the long sentence into short sentences, merging short sentences into long sentences, etc., to further change the expression of the answer; the data after synonym replacement and sentence restructuring is integrated with the original retained high-quality answers to generate an expanded first level optimized data set.

[0087] In a preferred embodiment of the present application, the step S4: based on high-quality SFT data, multi-dimensional quality analysis is performed on the data by a reflection model to generate second-level optimization data containing reflection tracks, which include correction logic and quality improvement paths for the original answers, which can include:

[0088] Step S41, input the first layer SFT data into the reflection model, load the multi-round reflection Prompt template, which contains error identification instructions, correction path generation instructions and high-quality answer generation instructions;

[0089] Step S42, generate optimization data containing reflection track text and second-level answers, wherein the reflection track and the answer are bound by a special separator, and the second-level answer needs to be verified by a teacher model with a score improvement of ≥10%.

[0090] In the embodiment of the present application, the first layer SFT data is input into the reflection model and the multi-round reflection Prompt template is loaded, and through the error identification instructions, the correction path generation instructions and the high-quality answer generation instructions, the logical errors and expression defects of the data are systematically mined and corrected to generate high-quality answers to provide accurate data for model training. The optimization data contains reflection tracks and high-verification-score answers, which help the model learn problem-solving logic and improve complex task processing capability; the second-level answers are verified by a teacher model, and a score improvement of ≥10% is used as the screening standard, which can objectively and accurately evaluate the quality improvement effect of the optimized data, avoid the limitations or biases of the data generated by the reflection model alone, ensure that the data entering the next stage of training has high effectiveness and reliability, reduce the risk of poor model training effect caused by poor data quality, and ensure the stability and reliability of the entire model training process; the combination of reflection tracks and second-level answers provides an important basis for the iterative upgrade of the model.

[0091] In the embodiment of the present application, the specific steps include:

[0092] Step S41, confirm that the first layer SFT data processed in the previous step is complete and has a standard format, and arrange it into a form suitable for input into the reflection model, such as dividing by batch or classifying by problem type, determine the currently available performance-optimal reflection model, such as GPT-4o, etc., check whether the model running environment is stable; in the operation interface or data processing platform of the reflection model, find the template loading function module, import the multi-round reflection Prompt template containing error identification instructions, correction path generation instructions and high-quality answer generation instructions; input the prepared first layer SFT data into the reflection model with the loaded template batch by batch or one by one, start the model running program, and begin processing the data.

[0093] Step S42, waiting for the reflection model to run to completion, obtaining the optimization data containing the reflection track text and the second-level answer, ensuring that the reflection track and the answer are bound through the special separator set in advance, facilitating subsequent data splitting and processing; using the special separator, separate the reflection track text and the second-level answer in the optimization data, and store them in different data areas or files, so as to verify and evaluate the answer subsequently. The separated second-level answer is input into the pre-trained teacher model, and the answer is scored according to the preset scoring standard, and the original score of each answer before verification is recorded. The score of the second-level answer given by the teacher model is compared with the original score of the answer in the first layer SFT data, and the score improvement amplitude of each answer is calculated; the second-level answer whose score improvement is greater than or equal to 10% and the corresponding reflection track text are screened out, and the data that does not meet the condition is excluded, the qualified reflection track text and the second-level answer are re-integrated according to the original binding format, and the final available optimization data is generated and stored properly.

[0094] In a preferred embodiment of the present application, the step S5 of mixing the first-level optimization data and the second-level optimization data to generate a comprehensive training set, wherein the second-level data carries auxiliary information of the reflection track, which is used to enhance the understanding and reasoning ability of the model for complex tasks, can include:

[0095] Step S51, mixing the first-level optimization data and the second-level optimization data in a ratio of 6:4 to generate a comprehensive training set;

[0096] Step S52, using a dynamic attention mask mechanism during training to force the model to analyze the reflection track content before generating the answer;

[0097] Step S53, if the KL divergence of the second-level data and the original data is greater than or equal to 2.0, then down-sample the second-level data to 50% of the original data amount, and align the feature distribution through the adversarial training module.

[0098] In the embodiment of the present application, the first-level and second-level optimization data are mixed in a ratio of 6:4, the stability of the first-level data and the high-quality advantage of the second-level data are integrated, a rich and balanced data structure is constructed, the model training stability is ensured, the training upper limit is improved, and training inefficiency is avoided; the dynamic attention mask mechanism is used to force the model to analyze and reflect the trajectory before generating the answer, help the model learn the answer optimization logic, break through the limitations of traditional learning results, mine key information, improve logical reasoning and problem solving ability, so that the model can refer to the reflection experience when facing complex tasks, and output better answers. The KL divergence is calculated to judge the difference in data distribution, the data with large difference is reduced to 50% of the original data amount and aligned with the feature distribution to avoid unstable training, and the dynamic adaptation strategy ensures that the model processes adaptive data, improves the training stability and reliability, enhances the adaptability of the model to different data distribution, and widens the application scenarios. Three-step cooperation, from data integration, training mechanism strengthening to data distribution adaptation, builds a complete training optimization system, provides high-quality data from the training set, dynamically masks to help deep learning, data adaptation ensures stability, and jointly improves the accuracy, generalization and robustness of the model.

[0099] In the embodiment of the present application, the specific steps include:

[0100] In step S51, the first-level and second-level optimization data that have been completed are checked, the total amount of the first-level and second-level optimization data is counted according to the mixing ratio of 6:4, the data amount that should be taken in the mixing process is calculated according to the ratio, for example, if the total amount of the first-level optimization data is 1000 and the total amount of the second-level optimization data is 800, 600 first-level optimization data and 400 second-level optimization data are taken for mixing. The calculated data amount is extracted from the corresponding storage location, and the first-level and second-level optimization data are mixed together in a row-by-row or batch merging manner to generate a comprehensive training set. During the mixing process, attention is paid to maintaining the original order and structure of the data to avoid data confusion. The generated comprehensive training set is spot-checked to check whether the data mixing ratio is accurate, the data content is complete, and there is no repetition or loss. If problems are found, the mixing process is adjusted in time and the comprehensive training set is regenerated.

[0101] Step S52: In the model training platform or framework, locate the attention mechanism-related settings options, confirm that the training environment supports the operation of the dynamic attention mask mechanism, and install and configure the required plugins or library files. Based on the characteristics of the reflection trajectory content and the model training requirements, formulate specific rules for the dynamic attention mask. For example, before the model generates the answer, it can force attention to specific keywords, key sentences, or paragraphs in the reflection trajectory text, prioritizing the parsing of these parts. Embed the defined dynamic attention mask rules into the model's training code or configuration file. During training, this mechanism will automatically take effect and regulate the model's attention allocation. Input the comprehensive training set into the model configured with the dynamic attention mask mechanism, and start model training according to the preset training parameters and iteration count. During training, monitor the model's parsing of the reflection trajectory content and the training effect in real time. If necessary, the mask rules can be fine-tuned.

[0102] Step S53: Using specialized statistical tools or algorithms, calculate the KL divergence value between the second-level data and the original data. During the calculation, the data format and dimensions meet the calculation requirements to accurately obtain the KL divergence result; compare the calculated KL divergence value with a threshold of 2.0. If the KL divergence ≥ 2.0, it is determined that downsampling of the second-level data is required; if the KL divergence < 2.0, skip the downsampling step and proceed directly to the feature alignment stage. When downsampling is required, randomly sample 50% of the original data volume of the second-level data or perform downsampling according to specific rules (such as stratified sampling by data category) to ensure that the downsampled data retains the main features and distribution patterns of the original data. Use an adversarial training module to perform feature alignment between the downsampled second-level data (or the un-downsampled second-level data) and the original data. During adversarial training, adjust the parameters of the adversarial training module to gradually bring the feature distribution of the second-level data closer to the feature distribution of the original data until a satisfactory alignment effect is achieved.

[0103] In a preferred embodiment of the present invention, step S6: injecting historical context information into each round of interaction in a multi-turn dialogue based on a comprehensive training set, and generating a tree-like divergent dialogue path based on the aforementioned optimized process to enhance the logical coherence and diversity of the multi-turn dialogue data, may include:

[0104] Step S61: Based on the comprehensive training set, in a multi-turn dialogue scenario, the historical dialogue context is injected into the question generation stage of step S1 to generate questions containing related references.

[0105] Step S62: For each round of generated tree-like dialogue paths, retain the top 2 reflected trajectory branches with the highest scores, and model the logical dependencies between paths using a graph neural network.

[0106] Step S63, the tree path data is iteratively enhanced through the double-level optimization process of steps S2-S5, and finally a multi-turn dialogue training set with long-range coherence is generated.

[0107] In the embodiments of the present application, the historical dialogue context is injected into the question generation link, so that the newly generated question is closely related to the past dialogue, a coherent dialogue logic chain is constructed, the coherence and naturalness of the dialogue are improved, the model better understands the context in multi-turn interaction, and answers more in line with user expectations are generated; by retaining the top 2 reflection track branches, the most valuable dialogue extension direction is selected, redundant path interference is avoided, and the effectiveness and pertinence of the data are improved. The logical dependency relationship between paths is modeled using a graph neural network, which can deeply mine the potential logical structure in the dialogue path and strengthen the logical reasoning ability of the model. Through the double-level optimization process of steps S2-S5, the tree path data is iteratively enhanced, and the data is finely processed from answer scoring screening, reflection optimization to data mixing adjustment, etc. Through multiple iterations, the quality and diversity of the data are continuously improved, providing better samples for model training. High-quality training data can improve the performance of the model in multi-turn dialogue tasks, so that it can handle various complex multi-turn dialogue scenarios in practical applications, output accurate, smooth and logical answers, and improve user experience and the practicality of the model.

[0108] In the embodiments of the present application, the specific steps include:

[0109] Step S61, confirm that the integrated training set has been completed, understand the business requirements and application characteristics of the multi-turn dialogue scene, sort out the historical dialogue context data, and inject the historical dialogue context data one by one according to the established rules; for example, for each newly generated question scene, integrate the dialogue content of the previous relevant turns into the input of the question generation in a specific identifier or format. Start the question generation program, generate questions containing associated references based on the input after injecting the historical dialogue context, using large language models or related algorithms. During the generation process, the association and rationality of the question and the historical dialogue context are checked in real time, and the questions that do not meet the requirements are adjusted or regenerated.

[0110] Step S62, in each round of dialogue, based on the generated question containing the associated reference, the tree-shaped dialogue path is generated by the model or algorithm, ensuring that each path covers multiple possible answers and extension directions to simulate the diversity in real dialogue scenarios. For each generated round of tree-shaped dialogue path, according to the preset scoring standard, the reflection track in each branch is scored, and the reflection track branches with the top 2 scores are selected and retained, and the other branches are discarded; using graph neural network technology, the retained reflection track branches are used as nodes, the logical relationship between the branches is analyzed, and the connection edges between the nodes are constructed, thereby modeling the logical dependency relationship between the paths.

[0111] Step S63, the tree-shaped path data is sequentially substituted into the double-level optimization process of steps S2-S5. First, according to the scoring standard of step S2, the accuracy, coherence, information amount, etc. of the answers in the tree-shaped path data are scored; then, according to the rules of step S3, the answer pairs with a score difference meeting the requirements are selected to constitute the primary DPO preference data, and the high-quality first-level SFT data is marked; then, the reflection model and Prompt template of step S4 are used to optimize the data in depth; finally, according to the method of step S5, the data is mixed, the dynamic attention mask mechanism is applied, and the data is adjusted according to the KL divergence.

[0112] The double-level optimization process is repeatedly executed to perform multiple rounds of iterative enhancement on the tree-shaped path data. After each round of iteration, the quality improvement effect of the data is evaluated, and the parameters and strategies in the optimization process are adjusted according to the evaluation results. After multiple rounds of iterative optimization, when the tree-shaped path data reaches the expected quality standard and has long-range coherence, it is arranged and output to generate the final multi-round dialogue training set for subsequent model training and optimization.

[0113] In a preferred embodiment of the present application, the above step S7: when the semantic similarity is detected to be lower than the preset threshold value in the online service process, the error dialogue chain is extracted and the error type is analyzed, and the homologous data generation strategy of step S1, the scoring threshold of step S2 and the reflection model configuration of step S4 are adjusted reversely, to generate a closed-loop mechanism for iterative optimization of cross-level data quality, which can include:

[0114] Step S71, when the online service detects that the semantic similarity of the user feedback is lower than the threshold value, the reflection track data of the error dialogue chain is extracted, and the error type distribution is analyzed.

[0115] Step S72, dynamically adjust according to the error type: the number of homologous answer generation, the scoring threshold and the reflection model configuration, to form a closed-loop iterative mechanism for cross-level data optimization.

[0116] In the embodiment of the present application, by monitoring the semantic similarity of user feedback, the situation that the model output does not match the user's expectation can be captured, and when the semantic similarity is lower than the threshold, the reflection track data of the error dialogue chain is extracted in time and the error type distribution is analyzed, and the specific problems such as logical errors and factual errors existing in the answering process of the model are accurately located. According to the error type distribution, the number of homologous answer generation, the scoring threshold and the reflection model configuration are dynamically adjusted. This flexible adjustment mechanism according to the actual error situation can make the model training and optimization strategy better adapt to different types of problems. Steps S71 and S72 cooperatively form a closed-loop iteration mechanism of cross-level data optimization, realizing a complete process from problem detection and analysis to strategy adjustment and optimization. In each iteration, the model can be improved based on the problems found in the last round. Through continuous circulation of this process, the quality of the comprehensive training set is continuously improved; with the advancement of iteration, the accuracy of the model in online service is gradually improved, which can provide more accurate and more demand-oriented answers for users, and improve the service quality and user experience.

[0117] In the embodiment of the present application, the specific steps include:

[0118] Step S71, during the running of online service, continuously collect user input and model output content, and use natural language processing algorithm to calculate the semantic similarity between user feedback and model answer in real time; when it is monitored that the semantic similarity is lower than the preset threshold, immediately start the error dialogue chain processing program, extract all related reflection track data in the dialogue chain from the service log or database, including intermediate results in the answer generation process, correction records and other information. Use the pre-trained error classification model to analyze the extracted reflection track data sentence by sentence and section by section, divide the error types into categories such as logical errors, factual errors and unclear expressions, and count the distribution proportion of each type of error in the entire error dialogue chain.

[0119] Step S72, according to the error type distribution result obtained in step S71, compare with the preset adjustment rule table to determine the specific parameters that need to be adjusted, increase the number of homologous answer generation according to the rules, for example, if the error rate increases by 10%, then increase 2 answers based on the original generation number; for step S2, modify the scoring threshold according to the error type. For example, if the proportion of logical errors is high, the weight of logical coherence in the scoring standard is increased to 50%, and the weights of other dimensions are adjusted accordingly; according to the error situation, reconfigure the reflection model, such as enabling external knowledge base retrieval function or adjusting the Prompt template content when there are more factual errors.

[0120] After the parameter adjustment is completed, the updated parameters are applied to the subsequent data generation and optimization process, and the homologous answer is regenerated from step S1, and the whole data optimization process is repeated. In a new round of iteration, the online service accuracy and other indicators are continuously monitored to evaluate the adjustment effect. If the expected effect is not achieved, steps S71 and S72 are executed again to continuously optimize the parameters, forming a closed-loop iterative mechanism for continuous improvement.

[0121] As shown in Figure 2 The embodiment of the application also provides a system 20 for double-level optimization of AI training data based on a large model, comprising:

[0122] The acquisition module 21 is configured to utilize the randomness generation capability of the large language model to generate a plurality of homologous answers for each question and instruction in the historical data through high-randomness hyperparameter configuration; based on the plurality of homologous answers, the pre-trained teacher model is used to score under low-randomness hyperparameter configuration to generate preference data pairs, and the answers with scores higher than a threshold are selected as high-quality supervised fine-tuning data.

[0123] The first-level optimization module 22 is configured to correspond the homologous answers generated in step S1 to the corresponding scoring results in step S2, to construct primary preference data for direct preference optimization, and to combine the filtered high-quality SFT data to form first-level optimization data.

[0124] The second-level optimization module 23 is configured to perform multi-dimensional quality analysis on the high-quality SFT data through the reflection model to generate second-level optimization data containing reflection tracks, wherein the reflection tracks include correction logic and quality improvement paths for the original answers.

[0125] The processing module 24 is configured to mix the first-level optimization data and the second-level optimization data to generate a comprehensive training set, wherein the second-level data carries auxiliary information of the reflection tracks to enhance the understanding and reasoning ability of the model for complex tasks; based on the comprehensive training set, historical context information is injected in each round of interaction of the multi-round dialogue, and a tree-like divergent dialogue path is generated based on the foregoing optimization process to enhance the logical coherence and diversity of the multi-round dialogue data; when the semantic similarity is detected to be lower than a preset threshold during the online service process, the error dialogue chain is extracted and the error type is analyzed, the homologous data generation strategy of step S1, the scoring threshold of step S2 and the reflection model configuration of step S4 are adjusted reversely, and a closed-loop mechanism for iterative optimization of cross-level data quality is generated.

[0126] The above is the preferred embodiment of the application. It should be noted that those skilled in the art can make several improvements and refinements without departing from the principles of the application. These improvements and refinements should also be considered within the scope of the application.

Claims

1. A method for double-layer optimization of AI training data based on a large model, characterized in that, The method comprises: S1: using the randomness generation capability of a large language model, generating multiple homologous answers for each question and instruction in historical data through high randomness hyperparameter configuration; S2: based on the multiple homologous answers, scoring through a pre-trained teacher model under low randomness hyperparameter configuration to generate preference data pairs, and screening answers with scores higher than a threshold as high-quality supervised fine-tuning data; S3: corresponding the homologous answers generated in step S1 to the corresponding scoring results in step S2, constructing primary preference data for direct preference optimization, and combining the screened high-quality SFT data to form first-level optimization data; S4: based on the high-quality SFT data, performing multi-dimensional quality analysis on the SFT data through a reflection model loaded with multiple rounds of reflection Prompt templates, the multiple rounds of reflection Prompt templates including error identification instructions, correction path generation instructions and high-quality answer generation instructions, to generate second-level optimization data containing reflection tracks, the reflection tracks including correction logic and quality improvement path for the original answers; S5: mixing the first-level optimization data and the second-level optimization data to generate a comprehensive training set, wherein the second-level data carries auxiliary information of the reflection tracks, for enhancing the understanding and reasoning ability of the model for complex tasks; S6: based on the comprehensive training set, injecting historical context information in each round of interaction of the multi-round dialogue, and generating a tree-like divergent dialogue path based on the optimization processes of the foregoing steps, to enhance the logical coherence and diversity of the multi-round dialogue data; S7: when detecting that the semantic similarity is lower than a preset threshold in the online service process, extracting an error dialogue chain and analyzing the error type, reversely adjusting the homologous data generation strategy of step S1, the scoring threshold of step S2 and the reflection model configuration of step S4, to generate a closed-loop mechanism for iterative optimization of data quality across levels.

2. The method for big model-based double-level optimization AI training data according to claim 1, characterized in that, S1: using the randomness generation capability of a large language model, generating multiple homologous answers for each question and instruction in historical data through high randomness hyperparameter configuration, comprising: Based on each question and instruction in the historical data, adjusting the generation hyperparameters of the large language model to a high randomness mode to generate at least 3 homologous answers; The homologous answers need to satisfy a cosine similarity ≥0.75 with the original data answers, and ensure semantic consistency through fixed instruction keywords; Disable the repetition penalty mechanism during generation, and limit the generation length to no more than 150% of the original answer.

3. The method for big model-based double-level optimization AI training data according to claim 2, characterized in that, S2: based on the multiple homologous answers, scoring through a pre-trained teacher model under low randomness hyperparameter configuration to generate preference data pairs, and screening answers with scores higher than a threshold as high-quality supervised fine-tuning data, comprising: Input the homologous answers into the pre-trained teacher model, switch to a low randomness mode, and score each answer based on a preset scoring template; The scoring template scores from three dimensions of accuracy, coherence and information quantity to generate a normalized score of 0-100; Screening answers with a score difference ≥15 to constitute primary DPO preference data, and marking answers with a score ≥85 under a single question as first-level SFT data.

4. The method for big model-based double-level optimization AI training data according to claim 3, characterized in that, S3: Correspond the homologous answers generated in step S1 to the corresponding scoring results in step S2, construct primary preference data for direct preference optimization, and combine the screened high-quality SFT data to form the first-level optimization data, including: Denoising processing of primary DPO data to remove answers with a repetition rate of ≥80% under the same question; Sort the first-level SFT data by score, select the top 30% answers, and perform data enhancement through synonym replacement and sentence restructuring to generate an expanded first-level optimization data set.

5. The method for big model-based double-level optimization AI training data according to claim 4, characterized in that, S4: Based on high-quality SFT data, a reflection model loaded with multiple rounds of reflection Prompt templates is used to perform multi-dimensional quality analysis, the multiple rounds of reflection Prompt templates include error identification instructions, correction path generation instructions, and high-quality answer generation instructions, thereby generating second-level optimization data containing reflection trajectories, the reflection trajectories include correction logic and quality improvement paths for original answers, including: Input the first-level SFT data into the reflection model loaded with multiple rounds of reflection Prompt templates to generate optimization data containing reflection trajectory text and second-level answers, where the reflection trajectory and the answer are bound by a special separator, and the second-level answer needs to be verified by a teacher model with a score improvement of ≥10%.

6. The method for big model-based double-level optimization AI training data according to claim 5, characterized in that, S5: Mix the first-level optimization data and the second-level optimization data to generate a comprehensive training set, where the second-level data carries auxiliary information of reflection trajectories to enhance the understanding and reasoning ability of the model for complex tasks, including: Mix the first-level optimization data and the second-level optimization data in a ratio of 6:4 to generate a comprehensive training set; Use a dynamic attention mask mechanism during training to force the model to analyze the reflection trajectory content before generating the answer; If the KL divergence of the second-level data and the original data is ≥2.0, downsample it to 50% of the original data volume, and align the feature distribution through the adversarial training module.

7. The method for big model-based double-level optimization AI training data according to claim 6, characterized in that, S6: Based on the comprehensive training set, inject historical context information in each round of interaction in the multi-round dialogue, and generate tree-like divergent dialogue paths based on the optimization process of the previous steps to enhance the logical coherence and diversity of multi-round dialogue data, including: Based on the comprehensive training set, inject historical dialogue context into the question generation link in step S1 in the multi-round dialogue scenario to generate questions containing associated references; For each generated tree-like dialogue path, retain the top 2 reflection trajectory branches, and model the logical dependency between paths through a graph neural network; The tree-like path data is iteratively enhanced through the double-level optimization process of steps S2-S5, and finally generates a multi-round dialogue training set with long-range coherence.

8. A system for double-level optimization of AI training data based on a large model, the system implementing the method of any one of claims 1 to 7, characterized in that, Including: An acquisition module for utilizing the randomness generation capability of a large language model to generate multiple homologous answers for each question and instruction in historical data through high-randomness hyperparameter configuration; Based on multiple homologous answers, score through a pre-trained teacher model under low-randomness hyperparameter configuration to generate preference data pairs, and select answers with scores higher than a threshold as high-quality supervised fine-tuning data; The first-level optimization module is configured to correspond the homologous answers generated in step S1 to the corresponding scoring results in step S2, to construct primary preference data for direct preference optimization, and to combine the high-quality SFT data after screening to form first-level optimization data; The second-level optimization module is configured to perform multi-dimensional quality analysis on the high-quality SFT data by loading a reflection model with multiple rounds of reflection Prompt templates, the multiple rounds of reflection Prompt templates including error identification instructions, correction path generation instructions, and high-quality answer generation instructions, to generate second-level optimization data containing reflection tracks, the reflection tracks including correction logic and quality improvement paths for the original answers; The processing module is configured to mix the first-level optimization data and the second-level optimization data to generate a comprehensive training set, wherein the second-level data carries auxiliary information of the reflection tracks, and is used to enhance the understanding and reasoning ability of the model for complex tasks; Based on the comprehensive training set, historical context information is injected in each round of interaction of the multi-round dialogue, and a tree-like divergent dialogue path is generated based on the foregoing optimization process to enhance the logical coherence and diversity of the multi-round dialogue data; when the semantic similarity is detected to be lower than a preset threshold in the online service process, an error dialogue chain is extracted and the error type is analyzed, the homologous data generation strategy in step S1, the scoring threshold in step S2, and the reflection model configuration in step S4 are adjusted reversely, and a closed-loop mechanism for iterative optimization of cross-level data quality is generated.

9. A computing device, comprising: comprise: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program which, when executed by a processor, implements the method as claimed in any one of claims 1 to 7.

Images