Model training method and device, equipment, storage medium and program product

Abstract

The invention provides a model training method and device, equipment, a storage medium and a program product. The method comprises the steps of training a to-be-trained model based on target training data to obtain a target training model; the target training data is constructed based on the original problem data and the target reply data; the target reply data is determined based on scoring results of multiple pieces of first reply data corresponding to the original question data, and the scoring results of the multiple pieces of first reply data are determined based on multiple language models.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a model training method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, large deep learning models that require a large amount of training data are widely used. At the same time, the training effect of these large models is crucial.

[0003] Related technologies collect raw data from public data sources or private data platforms, filter low-quality data through rules such as keyword matching and duplicate data deduplication, and integrate the remaining data into training data according to source weight or frequency of occurrence to train large models. However, the training effect of large models is poor. Summary of the Invention

[0004] This application provides a model training method, apparatus, device, storage medium, and program product that can generate high-quality training data and enhance the training effect of large models.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a model training method, the method including: The target training model is obtained by training the model to be trained based on the target training data; the target training data is constructed based on the original question data and the target response data; the target response data is determined based on the scoring results of multiple first response data corresponding to the original question data, and the scoring results of multiple first response data are determined based on multiple language models.

[0006] This application provides a model training apparatus, the apparatus comprising: The training unit is used to train the model to be trained based on the target training data to obtain the target training model. The target training data is constructed based on the original question data and the target response data. The target response data is determined based on the scoring results of multiple first response data corresponding to the original question data, and the scoring results of multiple first response data are determined based on multiple language models.

[0007] This application provides a computer device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the model training method provided in this application.

[0008] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the model training method provided in this application when executed by a processor.

[0009] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the model training method provided in this application.

[0010] The embodiments of this application have the following beneficial effects: Multiple language models are used to score the first responses to the original question data, thereby quantifying the quality of the first responses. Based on the scoring results of these first responses, target responses are selected to construct the target training data. In this way, through multi-model collaborative evaluation, the quality of different responses can be dynamically identified, improving the quality of the final target training data. This, in turn, enhances the training effect of the large model when training the model to be trained based on the target training data. Attached Figure Description

[0011] Figure 1 This is a schematic flowchart of a model training method provided in an embodiment of this application; Figure 2 This is a first structural schematic diagram of a training data generation system provided in an embodiment of this application; Figure 3 This is a second structural schematic diagram of a training data generation system provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure.

[0016] Figure 1 This is a flowchart illustrating a model training method provided in an embodiment of this application. The following will be combined with... Figure 1 The steps shown are explained as follows: Figure 1 As shown, the method includes the following step S101: Step S101: Train the model to be trained based on the target training data to obtain the target training model.

[0017] The target training data is constructed based on the original question data and the target response data. The original question data is the portion of the questions extracted from the original training data. In some implementations, the original question data can be the question data within the original training data. In other implementations, the original question data can be questions with a certain degree of ambiguity or multiple solutions that have been preprocessed and filtered. Specifically, it can be ambiguous or multi-solution questions filtered using a preset filter. Examples include open-ended questions such as: How to learn Python?, What is blockchain?

[0018] The target response data is determined based on the scoring results of multiple first response data corresponding to the original question data. In some implementations, the first response data can be response data to the original question data generated by a language model. In other implementations, the first response data can be response data corresponding to the original question data in the original training data. The original response data can be one or multiple. In some implementations, the target response data can be determined directly from the multiple first response data based on their scoring results. In other implementations, the first response data can be optimized firstly based on their scoring results, and then the target response data can be determined from the optimized first response data.

[0019] The scoring results for multiple first-response data are determined based on multiple language models. In some implementations, the first-response data can be scored based on multiple language models, resulting in multiple scoring results for each first-response data. In other implementations, the first-response data can be scored based on multiple language models from multiple dimensions, resulting in scoring results for each first-response data under each dimension.

[0020] A language model refers to a natural language processing model built on a deep learning architecture. It can generate semantically logical responses from input text, can be trained on large-scale text data, and possesses capabilities such as answer generation and inference chain generation. In this application, multiple language models are used, and these models can come from different model categories, such as base models, instruction fine-tuning models, and multimodal models, to ensure the diversity and complementarity of the generated response data. The language models selected here are relatively large and powerful, such as the LLaMA series, Deepseek series, and Qwen series models, while also considering the actual deployment or invocation costs.

[0021] In some implementations, the target training model can be one of multiple language models. For example, from the models that participated in generating the first response data, the model with better basic performance and an architecture adapted to the training scenario can be selected as the model to be trained, and fine-tuned and optimized directly using the target training data to obtain the target training model. In other implementations, the target training model can be a language model independent of multiple language models. That is, it is not limited to the range of models participating in the generation of the first response data, and a more suitable model architecture can be selected as the model to be trained based on actual business needs (such as deployment computing power limitations), and fine-tuned and optimized using the target training data to obtain the target training model.

[0022] In some implementations, supervised fine-tuning is first used to allow the model to master the basic mapping between the original question data and the target response data. Then, the reward mechanism in reinforcement learning is used as a reward signal to optimize the training process, resulting in the target training model. In other implementations, the original question data and target response data can be used directly as supervision signals to complete the training of the model in a single stage, resulting in the target training model.

[0023] In some implementations, the trained target model can be applied to scenarios requiring complex multi-turn dialogues and coverage of multi-domain knowledge. For example, in intelligent question-answering scenarios, the target model could be a large language model. In other implementations, the trained target model can also be applied to scenarios with low-computing-power device integration and high real-time requirements. For example, in enterprise service consultation scenarios, the target model could be a transformer. Of course, it can also be applied to other scenarios, and this application does not impose any limitations.

[0024] Multiple language models are used to score the first responses to the original question data, thereby quantifying the quality of the first responses. Based on the scoring results of these first responses, target responses are selected to construct the target training data. In this way, through multi-model collaborative evaluation, the quality of different responses can be dynamically identified, improving the quality of the final target training data. This, in turn, enhances the training effect of the large model when training the model to be trained based on the target training data.

[0025] In some embodiments, the process of constructing target training data proposed in this application may include the following steps S111 to S114: Step S111: Input the original question data into multiple language models to obtain the first response data output by each language model.

[0026] The first response data refers to the response data obtained after the original question data is input into a language model. In some implementations, the first response data may include: the answer and the thought process chain. The answer and the thought process chain provide joint supervision signals needed for training the large model, providing structured supervision signals for the training of the large model. Different language models may generate first response data with different structures and contents for the same original question data. For example, the first language model may generate a relatively concise answer, the second language model may provide a detailed step-by-step explanation, and the third language model may also include relevant charts or multimedia content.

[0027] In some implementations, multiple language models may include: an LLaMA-based base model, a Deepseek-based instruction fine-tuning model, and a Qwen-based multimodal model. The original question data is input into the LLaMA-based base model, the Deepseek-based instruction fine-tuning model, and the Qwen-based multimodal model, respectively, to obtain the first response data output by each language model.

[0028] In some implementations, for models with reasoning capabilities, such as the Deepseek-r1 instruction-based fine-tuning model, both thought chains and answers are generated simultaneously. For the Qwen3 model with a reasoning switch, the reasoning switch needs to be enabled (e.g., enable_thinking=True) for both thought chains and answers to be generated simultaneously.

[0029] Step S112: Based on multiple language models, score each first response data to obtain the score result for each first response data.

[0030] In some implementations, the multiple language models can be all the models currently generating the first response data, or they can be other models besides the language model that generated the first response data, or they can be other scoring models besides all the models currently generating the first response data.

[0031] In some implementations, multiple language models can be two language models, which can reduce computational power consumption. In other implementations, multiple language models can be more than two language models, which can improve the accuracy of scoring. The number of multiple language models can be set according to actual business needs and hardware conditions.

[0032] In some implementations, the rating result of the first response data can be the rating result on multiple rating dimensions. That is, it can be the rating result on each rating dimension obtained by using multiple language models to score the first response data on multiple rating dimensions.

[0033] In some implementations, the first response data is input into a language model, and the output of the language model is determined as the score result for the first response data. In other implementations, the language model scores each first response data based on the scoring dimensions and scoring descriptions, thus obtaining a score result for each first response data.

[0034] Step S113: Determine the target response data based on the scoring results of each first response data.

[0035] In some implementations, the target response data can be determined directly from multiple first response data sets based on the scoring results of each first response data set. In other implementations, the first response data can be optimized based on the scoring results of the first response data sets, and then the target response data can be determined based on the optimized first response data sets.

[0036] In some implementations, the first response with the highest score among all the first response data is selected as the target response data. For example, for the same original question data, if three language models (language model A, language model B, and language model C) generate three first response data respectively, and the first response data generated by language model B has the highest score, then the first response data generated by language model B will be determined as the target response data. In other implementations, the score results on each scoring dimension are sorted to obtain sorted score results, and the target response data is determined based on the sorted score results according to the importance of the scoring dimensions.

[0037] Step S114: Construct target training data based on the original question data and target response data.

[0038] The target training data consists of training samples composed of the original question data and the target response data. The original question data is the question part, and the target response data is the response part. The original question data and the target response data are combined to form a complete training dataset.

[0039] The target training data can be in either a question-and-answer pair format or an extended format with a thought chain. The question-and-answer pair format contains only the original question data and the answer portion of the target response data, suitable for standard generation tasks; the extended format includes the original question data, the answer portion of the target response data, and the thought chain, suitable for tasks requiring reasoning ability. The two formats can be selected based on training needs.

[0040] In some implementations, the original question data is determined as the training samples, the target response data is determined as the sample labels corresponding to the training samples, and the target training data is constructed based on the training samples and sample labels.

[0041] The process of constructing target training data involves pairing raw question data with target response data and storing them in a standardized format. For example, each training data entry can contain a raw question data field and an answer portion field. The raw question data field stores the original question data, and the answer portion field stores the target response data. Alternatively, each training data entry can contain a raw question data field, an answer portion field, and a thought chain field. The raw question data field stores the original question data, the answer portion field stores the answer portion of the target response data, and the thought chain field stores the thought chain from the target response data. Training data constructed in this way can be directly used to train large models, improving the model's generation quality and inference capabilities.

[0042] Based on the aforementioned technical methods, firstly, multiple language models respond to the same original question data to obtain first-response data from multiple perspectives; secondly, multiple language models score each first-response data, thereby quantifying the quality of the first-response data; finally, target response data is selected based on the scoring results to construct target training data. In this way, through multi-model collaborative generation and evaluation, the quality of different response data is dynamically identified, thereby improving the quality of the final target training data and enhancing the training effect of the large model.

[0043] In some embodiments, prior to step S201, the model training method provided in this application further includes steps A1 to A3: Step A1: Determine at least one second response data corresponding to each of the multiple training question data.

[0044] Training problem data refers to a set of problems that have not yet been filtered and optimized, and it serves as the basic input for generating high-quality raw problem data.

[0045] Each of the multiple training question data corresponds to at least one second response data, that is, each training question data is associated with at least one second response data.

[0046] In some implementations, when the training data includes training question data and training response data, the training response data corresponding to the training question data is determined as the second response data. In other implementations, the training question data is input into the aforementioned multiple language models to obtain multiple second response data. In still other implementations, the training question data is input into other language models besides the aforementioned multiple language models to obtain at least one second response data. For example, it could be multiple second response data obtained by inputting the training question data into multiple other language models. Alternatively, it could be multiple second response data obtained by inputting the training question data into other language models multiple times.

[0047] Step A2: For any training question data, determine the semantic similarity between every two second response data and the perplexity of the training question data.

[0048] Semantic similarity is a quantitative indicator that measures how close two second responses are in meaning. The higher the semantic similarity value, the more similar the content of the two second responses; if the semantic similarity is low, it means that the content of the two second responses is more different.

[0049] In some implementations, for each training question data, it is necessary to calculate the semantic similarity between all possible pairs of second response data for that training question data. For example, if the training question data has three second response data A, B, and C, then the semantic similarity between A and B, A and C, and B and C needs to be calculated.

[0050] In some implementations, the second response data can be converted into vector representations using semantic embedding models (such as BERT, Sentence-BERT, CLIP, etc.), and then semantic similarity can be calculated based on cosine similarity or other distance functions. Specifically, readily available NLP libraries (such as HuggingFace Transformers) can be used to load pre-trained semantic embedding models, encode each pair of second response data, and calculate semantic similarity. In other implementations, the second response data can be converted into vector representations using word frequency statistics and importance weighting, and then the cosine of the angle between the vector representations can be calculated to measure semantic similarity. A smaller angle indicates higher semantic similarity.

[0051] Perplexity characterizes the predictive complexity of the training question data, that is, it represents the difficulty faced by the language model in generating response data corresponding to the training question data. Specifically, the lower the predictive complexity, the simpler the training question data itself is, and the easier it is to generate accurate responses. It is relatively simple for large models, the confidence of the answer is high, and there are prominent candidate data elements. The higher the predictive complexity, the more complex the training question data itself is, and the more difficult it is to generate accurate responses. It is relatively difficult for large models, the confidence of the answer is low, and the probability distribution of the selection of a specific data element is relatively even across the entire data element table, with no clearly prominent candidate data elements. In some implementations, data elements may include phrases, words, or characters, which are not limited herein. For example, data elements may include phrases, words, or characters from the third response data.

[0052] In some implementations, the perplexity of the response data corresponding to the question data to be trained is determined based on the probability distribution of the language model. For example, the language model is a baseline language model (such as Qwen3-8B). A probability distribution is generated for the response data corresponding to each question data to be trained using the baseline language model. The perplexity of the response data corresponding to the question data to be trained is obtained by calculating this probability distribution.

[0053] Specifically, the perplexity of the response data corresponding to the question data to be trained can be obtained by calling the pre-trained language model API. This prediction complexity can also be weighted by combining other features (such as the length of the question data to be trained, keyword density, contextual information, etc.) to more comprehensively evaluate the perplexity of the response data.

[0054] Step A3: Based on the semantic similarity and perplexity of multiple training question data, filter the multiple training question data to obtain the original question data.

[0055] In some implementations, the training question data can be first filtered based on semantic similarity, and then filtered again based on perplexity to obtain the original question data. Firstly, semantic similarity can eliminate training question data with highly repetitive second response content, thereby reducing redundancy and increasing data diversity. Secondly, perplexity can identify ambiguous, high-value training question data—that is, training question data that is both challenging and capable of generating high-quality second responses.

[0056] In other implementations, the training question data can be first filtered based on perplexity, and then filtered based on semantic similarity to obtain the original question data.

[0057] Based on the aforementioned technical means, training problem data can be filtered according to semantic similarity and perplexity, thereby filtering out problem data with multiple solutions and ambiguity. This not only reduces computing power consumption and enables efficient processing and optimization of large-scale data, but also helps to retain more challenging and valuable data, thus improving the training efficiency of large models.

[0058] In some embodiments, the step A3 above, "filtering multiple training question data based on semantic similarity and perplexity to obtain original question data", can be achieved through the following steps A31 and A32: Step A31: The training question data with semantic similarity less than the similarity threshold is identified as the question data to be determined.

[0059] The problem data to be determined can be problem data with multiple solutions, which means that the problem data to be trained has multiple different second response data, and the similarity of the second response data is low.

[0060] The similarity threshold is a preset value used to distinguish between question data with high similarity and low similarity. Only when the semantic similarity exceeds the similarity threshold is the question data to be trained considered to have the characteristic of multiple solutions, and its perplexity is worth further evaluation.

[0061] Step A32: Identify the problem data to be determined that has a perplexity greater than the perplexity threshold as the original problem data.

[0062] A perplexity greater than the perplexity threshold indicates that, for the language model, the probability distribution of each data element in the second response data is relatively uniform across the entire data element table, meaning the confidence level is not high.

[0063] The perplexity threshold is a preset value used to distinguish between easy and difficult problem data. Only when the perplexity is higher than the perplexity threshold is the problem data to be determined considered ambiguous, challenging, and suitable as a source of high-quality training data.

[0064] Based on the above technical means, the semantic similarity threshold is used to ensure that the problem has multiple solutions, and the perplexity threshold is used to ensure that the problem itself is ambiguous. This can effectively filter out simple or highly repetitive problems and retain truly challenging data for subsequent generation processes.

[0065] In some embodiments, the "determining the perplexity of the second response data corresponding to the training question data" in step A2 above can be achieved through the following steps A21 and A22: Step A21: Input the question data to be trained into the benchmark language model to obtain the third response data output by the benchmark language model and the predicted probabilities corresponding to the data elements in the third response data.

[0066] The baseline language model can be one of several language models, or it can be a different language model from the other language models. Furthermore, the selection of the baseline language model should consider both model capability and efficiency. If the method presented in this paper is used to generate training data for a specific language model such as Qwen3-32B, the baseline language model should ideally be a small model from the same language model family, as it can better represent the output of the baseline language model.

[0067] A baseline language model is a language model used to generate language output and calculate element-level predicted probabilities. Specifically, the main function of a baseline language model is to perform language modeling on the input question data, generate corresponding responses, and provide a probability value for each generated data element, representing the likelihood of each generated data element appearing in the context. In some implementations, the baseline language model can be a Transformer architecture model, a MoE model, or other language models.

[0068] Prediction probability refers to the confidence level of the model in predicting each word when generating third-party response data. Numerically, it is a probability value between 0 and 1. The higher the prediction probability, the more confident the model is in the reasonableness of the predicted word in the current context.

[0069] Step A22: Determine the perplexity of the training question data based on the predicted probability corresponding to each word.

[0070] Low perplexity indicates that the answers to the training question data are relatively clear and are not suitable as high-quality training data; high perplexity indicates that the training question data has multiple interpretations and is suitable as high-quality training data, belonging to high-value seed questions.

[0071] In some implementations, the logarithmic predicted probabilities of the data elements in the third response data are averaged to obtain the average log probability, and then the exponentiation of the average log probability is taken to obtain the final perplexity.

[0072] Based on the aforementioned technical means, by introducing a benchmark language model to calculate the perplexity of the third response data corresponding to the question data to be trained, ambiguous questions can be screened more accurately, the quality of training data can be optimized, and the performance of large models on complex tasks can be further improved.

[0073] In some embodiments, the step S112 above, "scoring each first response data based on multiple language models to obtain the scoring result corresponding to the first response data", can be implemented through the following steps S121 and S122: Step S121: Based on the second language model, score the first response data output by the first language model from multiple scoring dimensions to obtain the sub-scoring results corresponding to each scoring dimension.

[0074] The first language model can be any one of the multiple language models. The second language model can be any one of the multiple language models other than the first language model, and there must be at least one second language model.

[0075] Scoring dimensions refer to the dimensions used to evaluate the quality of the first response data generated by the first language model from multiple aspects. In some implementations, scoring dimensions may include: accuracy, logicality, and comprehensiveness. Accuracy is used to assess whether the first response data conforms to known facts at the factual, data, and conceptual levels, and whether it contains incorrect information or misleading statements. Logicality is used to assess whether the structure of the first response data is clear and well-organized, whether the reasoning between its parts is smooth and reasonable, and whether it clearly guides the reader's understanding. Comprehensiveness is used to assess whether the first response data adequately covers all key aspects of the user's question and whether it considers relevant background information, potential details, or different perspectives.

[0076] Each rating dimension has multiple rating values ​​and corresponding rating descriptions; that is, each rating dimension corresponds to a set of rating descriptions. The rating value refers to the specific numerical range within each rating dimension, usually using a segmented system, such as 0-5 points. Each rating value corresponds to a specific rating description, making the rating more objective and quantifiable. For example, under the accuracy dimension, the rating values ​​and rating descriptions could include: 5 points: Completely accurate, with no factual errors or misleading information; 4 points: Minor inaccuracies exist, but do not affect the overall understanding or the correctness of the core information; 3 points: Contains some obvious factual errors, or inaccurate important information, which may have some impact on understanding; 2 points: Contains numerous factual errors, or seriously inaccurate core information, making the answer essentially unreliable; 1 point: The answer is entirely based on incorrect information, or all the information provided is false / misleading; 0 points: The answer is irrelevant to the question, or its accuracy cannot be assessed.

[0077] Sub-scoring results are obtained by scoring the first response data based on a second language model within a specific scoring dimension. In some implementations, sub-scoring results may include a target score and a scoring description. The scoring description explains the determination of the target score.

[0078] In some implementations, based on the second language model, the first response data output by the first language model is scored from three scoring dimensions: accuracy, logicality, and comprehensiveness, to obtain sub-scoring results corresponding to each scoring dimension.

[0079] Step S122: Based on multiple sub-scoring results, determine the scoring result of the first response data output by the first language model.

[0080] In some implementations, the average of multiple sub-scores is taken to obtain the score of the first response data.

[0081] In other implementations, the highest and lowest scores from multiple sub-scoring results are removed, and then the average score is taken to obtain the scoring result of the first response data.

[0082] In some other implementations, a weighted summation of multiple sub-scores is performed to obtain the score result of the first response data.

[0083] Based on the aforementioned technical means, responses are quantitatively evaluated through multiple scoring dimensions, such as accuracy, logic, and comprehensiveness, to ensure that the scoring covers key quality indicators. In addition, the use of multi-model cross-scoring, with different models participating in the scoring, can improve the objectivity and comprehensiveness of the scoring, thereby enhancing the overall quality of the training data.

[0084] In some embodiments, the step S121 above, "scoring the first response data output by the first language model from multiple scoring dimensions based on the second language model, and obtaining the sub-scoring results corresponding to each scoring dimension," can be achieved through the following steps S131 to S133: Step S131: For any rating dimension, based on the second language model, determine the target rating description that matches the first response data output by the first language model from among the multiple rating descriptions corresponding to the rating dimension.

[0085] The target rating description refers to the process of determining which of the multiple rating descriptions best matches the first response data output by the first language model.

[0086] In some implementations, the second language model, based on its own knowledge base and reasoning mechanism, determines the rating description that is closest to the first response data output by the first language model and uses it as the target rating description.

[0087] Taking accuracy as the scoring dimension as an example, the first response data includes: installing the Python environment, learning basic syntax, and practicing projects. If other language models determine that the first response data (installing the Python environment, learning basic syntax, and practicing projects) has minor inaccuracies, but these do not affect the correctness of the core information, then the target scoring description is determined to be: "There are minor inaccuracies, but they do not affect the overall understanding or the correctness of the core information." For scoring dimensions such as logicality and comprehensiveness, the approach is similar to the accuracy example above, and will not be elaborated upon further here.

[0088] Step S132: Based on the pre-defined correspondence between rating descriptions and rating values, determine the target rating value that matches the target rating description.

[0089] The target score refers to the specific numerical value obtained after scoring the response data generated by the first language model using different scoring dimensions.

[0090] In some implementations, the target score is determined based on the score corresponding to the target score description. Specifically, when other language models determine that the first response data output by the first language model belongs to a score description with minor inaccuracies, based on the above correspondence between score descriptions and score values, it can be known that there are minor inaccuracies, but they do not affect the overall understanding or the correctness of the core information, which corresponds to 4 points. Therefore, the target score is 4 points, that is, 4 points is the score of the first response data output by the first language model in the accuracy score dimension.

[0091] Step S133: Based on the target score, determine the sub-score results of the first response data output by the first language model under the score dimension.

[0092] Sub-scores refer to the specific scores given to the first response data output by the first language model under each scoring dimension. In some implementations, sub-scores are derived by directly referencing or converting the target score value. For example, if the score is 4 points under the accuracy dimension, then the sub-score of the first response data output by the first language model under the accuracy dimension is 4 points.

[0093] Based on the above technical means, by matching the first response data output by the first language model with the preset rating description and determining the rating value accordingly, the objectivity and consistency of the rating can be ensured, and the rating process can be automated, making the rating process more standardized and facilitating large-scale automated processing.

[0094] In some embodiments, the step S122 above, "determining the scoring result of the first response data output by the first language model based on multiple sub-scoring results", can be achieved through the following steps S141 and S142: Step S141: Determine the weight corresponding to each target score value.

[0095] The weights corresponding to the target scores are parameters that reflect the relative importance of each scoring dimension in the overall scoring result. For example, the importance of each scoring dimension in the overall scoring result can be different, meaning the weights corresponding to the target scores are different. If accuracy is more critical to the current task, the weights corresponding to the target scores can be set higher. Conversely, the importance of each scoring dimension in the overall scoring result can be the same, meaning the weights corresponding to the target scores are the same.

[0096] In some implementations, the weights corresponding to the target score values ​​can be preset.

[0097] In some implementations, the weight for accuracy can be 3, the weight for logic can be 3, and the weight for comprehensiveness can be 2.

[0098] Step S142: Based on the weight corresponding to each target score value, perform a weighted average of the target score values ​​corresponding to multiple score dimensions to obtain the score result of the first response data output by the first language model.

[0099] The weighted average method involves multiplying the target score for each rating dimension by its corresponding weight, summing the results, and then dividing by the sum of the weights corresponding to the target scores to arrive at the final comprehensive score. The weighted average method allows for flexible adjustment of the importance of different dimensions, making the final score more closely reflect actual needs.

[0100] For example, if the target score for accuracy is 4, the target score for logic is 5, and the target score for comprehensiveness is 3, and the weights for accuracy, logic, and comprehensiveness are 0.5, 0.3, and 0.2 respectively, then the weighted average score is 4 × 0.5 + 3 × 0.3 + 2 × 0.2 = 3.1. As another example, if the target score for accuracy is 4, the target score for logic is 5, and the target score for comprehensiveness is 3, and the weights for accuracy, logic, and comprehensiveness are 3, 3, and 2 respectively, then the weighted average score is 3 × 4 / 5 + 3 × 5 / 5 + 2 × 3 / 5 = 6.6.

[0101] Based on the aforementioned technical means, by assigning weights to different scoring dimensions, the importance of each dimension in practical applications can be reflected more accurately, thereby improving the rationality of the scoring results.

[0102] In some embodiments, the step S113 above, "determining the target response data based on the scoring results of each first response data", can be achieved through the following steps S151 to S153: Step S151: Based on the first language model, the second language model, the first response data, and the scoring results, fill in multiple fields to be filled in the predefined prompt template to obtain prompt words.

[0103] The first response data can include the answer and the thought process. The answer refers to the response text generated by the language model for a given original question, while the thought process refers to the reasoning path or thinking process behind the answer, including steps such as question understanding, knowledge retrieval, logical deduction, and conclusion formation.

[0104] A prompt template is a structured text framework used to guide a model in generating output that conforms to a specific format. In some implementations, the prompt template contains multiple fields to be filled in, such as the current model field, the model to be rated field, the rating model field, the original user question field, the response to be evaluated field, the evaluation result field, and the overall evaluation field.

[0105] In some implementations, the values ​​of the current model item are filled based on the first language model, the values ​​of the model item to be scored are filled based on the first language model, and the values ​​of the scoring model item are filled based on the second language model. Of course, the values ​​of the evaluation result item can also be filled based on the evaluation result, or the values ​​of the overall evaluation item can be filled based on the overall evaluation.

[0106] Step S152: Based on the prompt words, improve the first response data to obtain improved first response data.

[0107] The quality of the first response data directly impacts user comprehension and acceptance, making its optimization crucial. When the first response data includes both the answer and the thought process chain, the quality of the answer directly influences user understanding and acceptance, thus optimizing it is equally important. The value of the thought process chain lies in providing deeper supervisory signals to the language model, enabling it to understand the possibility of why a particular answer is correct, rather than simply knowing the answer itself.

[0108] Improvement refers to the adjustment and optimization of the original answer and thought process, with the aim of enhancing their accuracy, logic, and comprehensiveness. This improvement process may include correcting factual errors, supplementing detailed descriptions, reconstructing the logical structure, and introducing new perspectives or evidence.

[0109] In some implementations, improvement is a process of modifying and refining the original answer and thought process based on the prompts. For example, if an answer is found to have a logical jump, intermediate reasoning steps will be added during the improvement process; if a thought process is deemed incomplete, relevant information or different perspectives will be added during the improvement process.

[0110] For example, the original answer was: Install Python environment, learn basic syntax, practice project; the thought process was: Install Python environment - Learn basic syntax - Practice project. Improving this answer and thought process, the improved answer is: Install Python environment, study official documentation, take online courses, practice project; the improved thought process is: Install Python environment - Study official documentation - Take online courses - Practice project.

[0111] Step S153: Determine the target response data from the improved first response data.

[0112] In some implementations, the target response data can be determined from the improved first response data based on model trust levels. For example, the improved first response data corresponding to the language model with the highest model trust level can be determined as the target response data. In other implementations, the improved first response data is scored based on multiple language models to obtain a score result for the improved first response data. The target response data is then determined from the improved first response data based on the score result. For example, the improved first response data with the highest score result can be determined as the target response data.

[0113] Based on the aforementioned technical methods, by combining scoring results with prompt templates, the model can be guided to optimize its output based on feedback, further improving the quality of the target response data. In this way, on the one hand, the prompt words provide the model with clear directions for improvement, enabling it to perform targeted optimization for specific problems; on the other hand, this method supports iterative improvement of the model based on existing responses, thereby enhancing the diversity and depth of the training data.

[0114] In some embodiments, the step S152 above, "improving the first response data based on the prompt words to obtain improved first response data", may include: in response to an improvement instruction, improving the first response data based on the prompt words and the improvement strategy carried by the improvement instruction to obtain improved first response data.

[0115] An improvement instruction is a command used to instruct the language model to optimize the initial response data. In some implementations, the improvement instruction is a user-triggered instruction, and the improvement strategy carried by the improvement instruction may be content included within the prompt template.

[0116] Improvement instructions can be expressed in natural language, such as "Please explain the reasoning process in more detail" or "Please provide a real-world example." Improvement instructions typically include specific optimization directions, such as improving accuracy, enhancing logic, or expanding overall scope.

[0117] An improvement strategy is a specific optimization method or operational suggestion carried in an improvement instruction. For example, if the improvement instruction is to improve accuracy, the corresponding improvement strategy might include specific operational paths such as checking the reliability of the sources of key facts, comparing multiple authoritative sources, and verifying the consistency of timelines. The role of the improvement strategy is to guide the model on which aspects should be focused on during the optimization process and to provide actionable implementation suggestions.

[0118] For example, an improvement strategy could include: Now, as the language model A to be scored, carefully analyze the potential shortcomings in your previous answer generation based on the evaluation results provided by the scoring language model B, and propose an improved answer. Focus on the following aspects and reflect them in your "improved answer": For low scores in "accuracy": If the scoring language model B points out factual errors, you need to strengthen fact-checking when generating the answer, prioritize the use of the latest and most reliable information sources, or explicitly perform internal verification when providing specific facts. For example, the answer is: Install Python environment, learn basic syntax, practice project; the thought chain is: Install Python environment - Learn basic syntax - Practice project. In response to the improvement instruction, due to insufficient accuracy in learning basic syntax, the answer and thought chain are improved, resulting in the improved answer: Install Python environment, study official documentation, practice project; the improved thought chain is: Install Python environment - Study official documentation - Practice project.

[0119] Based on the aforementioned technical means, improvement strategies can cover multiple aspects such as fact-checking, logical streamlining, and detail supplementation, making the model output more rigorous. By introducing improvement strategies, the model can be guided to follow specific logic or rules during the optimization process, thereby improving the efficiency and quality of the improvement.

[0120] In some embodiments, the step S152 above, "improving the first response data based on the improvement strategy carried by the prompt words and improvement instructions to obtain improved first response data", may further include the following steps S161 and S162: Step S161: Based on the improved strategy and prompts, the thought process is obtained.

[0121] The thought process represents the prompts used to improve the initial response data. The thought process is a detailed processing flow constructed by the model based on its existing knowledge base and reasoning ability after receiving prompts and improvement strategies. In some implementations, where the initial response data includes the answer and a thought chain generated based on the answer, the thought process includes multiple stages such as the model's understanding of the question, evaluation of existing answers, identification of missing information, and organization of new content. The thought process not only determines the quality of the final output answer but can also serve as part of the training data for subsequent model learning and optimization.

[0122] Step S162: Based on the thought process, the first response data is improved to obtain the improved first response data.

[0123] Given that the first response data includes the answer and the thought process generated based on the answer, the answer and the thought process are improved based on the thought process to obtain an improved answer and an improved thought process.

[0124] In practical applications, when a first language model receives an improvement instruction (e.g., please improve the logic of the answer), it first analyzes the improvement strategy within the instruction (e.g., adding intermediate reasoning steps). Then, based on the improvement strategy and prompts, the first language model constructs a detailed thought process. For example, it first identifies the core argument of the problem, then lists key evidence supporting that argument, and finally connects the evidence to the conclusion through a causal chain. Next, based on the constructed detailed thought process, the first language model modifies the answer, adding necessary intermediate reasoning steps and updating the thought chain to reflect the new reasoning path.

[0125] Based on the aforementioned technical means, the thought process is generated by improving the strategy and prompt words, and then the first response data is improved, thereby significantly improving the quality of the training data. This provides more accurate and structured supervision signals for the large model, and further effectively improves the performance of the large model in complex tasks.

[0126] In some embodiments, the model training method provided in this application may further include the following steps S171 and S172: Step S171: Determine the original question data as training samples and the target response data as the sample labels corresponding to the training samples.

[0127] Training samples refer to high-quality problem data that are retained after peer review and scoring by multiple models. These problems are typical problems that are ambiguous or have multiple possible solutions, such as open-ended questions like how to learn Python.

[0128] The target response data can be either the initial response or an improved version of the initial response. Conversely, the target response data can be either the answer and the thought process, or an improved version of the answer and the thought process.

[0129] Sample labels describe the standard answer and standard thought process associated with each training sample. The answer obtained during sample labeling is the highest-scoring response after inter-model peer review; the thought process is a complete set of structured reasoning processes matching the answer, including key knowledge point references and logical chains, used to help large models understand the thought path behind the answer. The improved answer obtained during sample labeling is the highest-scoring response after inter-model peer review and optional self-optimization; the improved thought process is a complete set of structured reasoning processes matching the improved answer, including key knowledge point references and logical chains, used to help large models understand the thought path behind the answer.

[0130] Step S172: Construct target training data based on training samples and sample labels.

[0131] Target training data refers to the data format ultimately used to train a large model. In some implementations, target training data may include three parts: the input question, the output answer, and the intermediate reasoning process (i.e., the thought process). In other implementations, target training data may include only two parts: the input question and the output answer.

[0132] Each training sample is paired with its corresponding sample label and organized into training data entries according to a standardized format. For example, a training data entry might include the following structure: Install the Python environment - study the official documentation - participate in online courses - practice project development; Install the Python environment; Study the official documentation; Take online courses; Practical projects.

[0133] Based on the aforementioned technical means, by constructing training data that includes original question data and target response data, more comprehensive supervision signals can be provided, thereby improving the model's performance on complex reasoning tasks and enhancing the logical reasoning and knowledge transfer capabilities of large models.

[0134] The model training method provided in this application embodiment will be described in detail below with reference to specific application scenarios.

[0135] The training data for large models is mainly integrated through manual screening, rule-based filtering, or by weighting / frequency of occurrence based on data sources. For example, in the training of general-purpose large language models (such as the GPT series, LLaMA, etc.), data sources usually include internet text, question-answer pairs, books, etc. During training, responses with high frequency or high weight in the dataset are directly used as standard answers, but the quality of different responses is not dynamically evaluated and distinguished.

[0136] This application provides a model training method that dynamically identifies the quality of different responses through multi-model collaborative generation and evaluation, improving the accuracy and logic of training data; it preserves the thought processes corresponding to high-quality responses, providing structured supervision signals for large-scale model training; and it reduces computational consumption through preset filters, achieving efficient processing and optimization of large-scale data. Specifically, it addresses the problems of indistinguishable response quality, lack of thought process data, and high computational consumption in existing large-scale model training data. By using multi-model collaborative response generation, peer evaluation and scoring, and optional self-optimization, it generates high-quality training data with thought processes, and combines filters to reduce computational requirements, thereby improving the efficiency of large-scale model training.

[0137] Reference Figure 2 , Figure 2 The schematic diagram of a first structure of a training data generation system shown includes: raw data input 201, filter 202, multi-model response generation module 203, peer review and scoring module 204, self-optimization switch 205, model self-optimization submodule 206, high-quality data screening 207, training dataset 208, and large model 209. The specific steps for interaction between them may include: Step S211, Seed data filtering. Use the preset filter 202 to filter the problem data (i.e., the training problem data mentioned above) in the large-scale original dataset.

[0138] Step S212: Output high-value seed questions. Filter 202 selects ambiguous or multi-solution questions as seed data (i.e., the original question data mentioned above). Implementation: By extracting question features (such as semantic similarity calculation of answers from different models, perplexity ppl of a baseline model's response, etc.), unambiguous or single-answer questions are filtered out, while high-value seed questions that may have multiple responses are retained.

[0139] Step S213: Parallel generation of responses and thought chains using multiple models. The multi-model response generation module 203 inputs each seed question into multiple large models of different categories (such as models with different architectures or training data, e.g., model A is a basic Transformer model, model B is a MoE model, model C is a Diffusion model, etc.). Each model generates corresponding response text and thought chains (such as reasoning steps, references to key knowledge points, and logical chains).

[0140] Step S214, cross-scoring between models. The peer review scoring module 204 performs cross-scoring between models. That is, each model scores the responses of other models. Scoring dimensions include: accuracy (whether the answer is correct, 0-5 points); logicality (whether the reasoning steps are coherent, 0-5 points); comprehensiveness (whether it covers multiple aspects of the question, 0-5 points); and calculating the average score of each response (removing the highest / lowest score to reduce the impact of outliers).

[0141] Step S215: Optimize responses and thought processes. Self-optimization switch 205 determines whether to enable the self-optimization process. If enabled, the model self-optimization submodule 206 optimizes responses and thought processes. That is, before peer review, each model refers to the responses and scores of other models to optimize its own responses and thought processes (such as correcting erroneous reasoning steps and supplementing key information). If disabled, peer review proceeds directly, resulting in a higher degree of matching between the generated data's thought processes and the subsequent model training prompt (without reference answers).

[0142] Step S216: Calculate the average score. Peer review scoring module 204 calculates the average score.

[0143] Step S217: Output the optimal response and thought chain. High-quality data filtering 207: Based on the average score, filter out the highest-scoring response and its corresponding thought chain (i.e., the target response data mentioned above), and use it as optimized training data (i.e., the target training data mentioned above), storing it in the training dataset.

[0144] Step S218 is used for large model training. Large model 209 is trained based on the optimized training data.

[0145] Reference Figure 3 , Figure 3 The schematic diagram of a second structure of a training data generation system shown includes: a multi-model response generation module 203, a peer review and scoring module 204, and an average score calculation module 301. The specific steps for their interaction may include: In step S311, the multi-model response generation module 203 generates response A and thought chain A through model A (LLaMA model), response B and thought chain B through model B (DeepSeek model), and response C and thought chain C through model C (Qwen model).

[0146] In step S312, the multi-model response generation module 203 inputs response A and thought chain A, response B and thought chain B, and response C and thought chain C into the peer review scoring module 204.

[0147] In step S313, the peer review scoring module 204 scores A2 and A3 through scoring device 1 model A to obtain the scoring results, scores A1 and A3 through scoring device 2 model B to obtain the scoring results, and scores A1 and A2 through scoring device 3 model C to obtain the scoring results.

[0148] In step S314, the peer review scoring module 204 summarizes the scores of each person into the average score calculation 301.

[0149] Step S315, Average Score Calculation 301: Calculate the average score using an average score calculator, and then filter high-quality data based on the average score using a high-quality data filter.

[0150] The following is a detailed description of Embodiment 1 proposed in this application: 11. Seed Data Screening. The original data contains 1 billion question-answer pairs. 5 million questions are selected based on the output similarity of different small LLMs (0.5B to 3B). The output perplexity ppl of the baseline LLM (e.g., Qwen 3-8B) is calculated, and approximately half are filtered out using dynamic thresholding, retaining 3 million multi-solution questions as seed data. Multi-model selection: Three different types of models are selected: Model A (based on LLaMA), Model B (based on Deepseek's instruction fine-tuning model), and Model C (based on Qwen's multimodal model).

[0151] 12. Response Generation. Each model takes a seed question as input and generates a response (e.g., a response to "How to learn Python") and a thought process chain (e.g., "Step 1: Install Python environment → Step 2: Learn basic syntax → Step 3: Practice project").

[0152] 13. Peer evaluation and scoring. Each model scores the responses of the other two models (maximum score for each item (weight): accuracy 3 points, logic 3 points, comprehensiveness 2 points, total score 8 points). For example, Model A scores Model B as follows: accuracy 4 points, logic 5 points, comprehensiveness 3 points (average score is 3×4 / 5+3×5 / 5+2×3 / 5=6.6 points).

[0153] Prompt Sample Template - Peer Review and Scoring.

[0154] Objective: To evaluate the answers generated by Model B and give scores based on three dimensions: accuracy, logicality, and comprehensiveness.

[0155] User input (Model B's question): [Paste the question answered by Model B here].

[0156] The answer from Model B: [Paste the answer generated by Model B here].

[0157] Scoring instructions: Please carefully read the answer to Model B and rate it according to the following three dimensions. Please ensure that your rating is objective, fair, and based on your understanding of the user's question.

[0158] Explanation of scoring dimensions: Accuracy - Total score 5 points.

[0159] Definition: Whether the answer is consistent with known facts at the factual, data, and conceptual levels, and whether it contains incorrect information or misleading statements.

[0160] Scoring criteria: 5 points: Completely accurate, with no factual errors or misleading information.

[0161] 4 points: There are minor inaccuracies, but they do not affect the overall understanding or the correctness of the core information.

[0162] 3 points: Contains some obvious factual errors or inaccurate important information, which may affect comprehension.

[0163] 2 points: Contains numerous factual errors or seriously inaccurate core information; the answer is essentially unreliable.

[0164] 1 point: The answer is based entirely on incorrect information, or the information provided is entirely false / misleading.

[0165] 0 points: The answer is irrelevant to the question or its accuracy cannot be assessed.

[0166] After rating, please briefly explain the main inaccuracies (if any), for example: "Deduct 1 point for accuracy because Model B incorrectly assigned the year of the XYZ event to 2023, when it should actually be 2022." Logicality - Total score 5 points.

[0167] Definition: Whether the structure of the answer is clear and well-organized, whether the reasoning process between each part is smooth and reasonable, and whether it can clearly guide the reader to understand.

[0168] Scoring criteria: 5 points: Clear structure, rigorous argumentation, flawless reasoning process, and smooth logic.

[0169] 4 points: The structure is generally clear and the logic is smooth, but some reasoning steps could be more concise or explained more fully.

[0170] 3 points: There are some logical jumps or disjointed connections, some reasoning may not be rigorous enough, and the overall understanding is slightly difficult.

[0171] 2 points: The logic is chaotic, there is a lack of clear connections between paragraphs and viewpoints, the reasoning process has obvious flaws, and it is difficult to understand.

[0172] 1 point: The answer is completely illogical, disorganized, and the reasoning is chaotic.

[0173] 0 points: The answer lacks structure and cannot be logically evaluated.

[0174] After scoring, please briefly explain the main problems (if any), for example: "1 point deducted for logical reasoning, because when explaining the ABC theory, model B jumps directly to the conclusion without explaining the process of deriving Z from Y." Comprehensiveness - Total score 5 points.

[0175] Definition: Does the answer adequately cover all key aspects of the user's question, taking into account relevant background information, potential details, or different perspectives?

[0176] Scoring criteria: 5 points: The question was answered fully and comprehensively, taking into account all relevant aspects, and the necessary details and background were considered.

[0177] 4 points: It covers most of the key aspects, but may have missed a minor detail or a less common perspective.

[0178] 3 points: The answer addresses the core of the question but omits important relevant information, details, or perspectives, resulting in a lack of depth.

[0179] 2 points: The answer missed several key information points, had very limited coverage of the question, and failed to fully address the user's intent.

[0180] 1 point: The answer is very brief, only scratches the surface of the problem, and provides almost no useful information.

[0181] 0 points: The answer is completely irrelevant to the question, or is merely a brief repetition.

[0182] After rating, please briefly explain which aspects could be more comprehensive (if any), for example: "Deduct 1 point for comprehensiveness. Model B explains the basic principles of X, but does not mention the specific differences of X in different application scenarios, nor does it discuss the relevant limitations." Overall rating: After completing the scoring of the above three dimensions, please summarize the overall performance of Model B in one or two sentences.

[0183] For example: "Model B's answer is excellent in terms of logic, but there is still room for improvement in terms of accuracy and comprehensiveness." Final output format: Please output your rating results in the following format: User input: [Pasted user question].

[0184] The answer for Model B: [Paste the answer for Model B].

[0185] score: Accuracy: [X] / 5 points. [Brief explanation (if available)].

[0186] Logical reasoning: [X] / 5 points. [Brief explanation (if applicable)].

[0187] Comprehensiveness: [X] / 5 points. [Brief explanation (if applicable)].

[0188] Overall assessment: [A one-sentence summary].

[0189] In addition, after the scoring is completed, low-scoring responses can be manually reviewed to further improve data quality.

[0190] 14. Self-optimization (enabled).

[0191] Model A references Model B's "Step 2: Study the official documentation" and Model C's "Step 3: Take online courses" and optimizes its own thought process to "Step 1: Install the Python environment → Step 2: Study the official documentation → Step 3: Take online courses → Step 4: Practice the project".

[0192] Prompt Example Template - Self-Iterative Optimization Based on Reference Scoring Results.

[0193] Objective: To guide large model A to analyze and improve the Prompt used to generate the answer based on the evaluation results of the previous model B.

[0194] Current model: Model A.

[0195] Model to be scored: Model A.

[0196] Evaluation model: Model B.

[0197] Original user question: [Paste here the original user question answered by Model B].

[0198] The previous output of Model A (the evaluated answer): [Paste the answer generated by Model A in the previous round here].

[0199] Evaluation results of Model B: Accuracy: [X] / 5 points. [A brief explanation of accuracy from Model B, for example: "Model A incorrectly assigned the year of the XYZ event to 2023, when it should actually be 2022."]

[0200] Logicality: [X] / 5 points. [A brief explanation of the logicality of Model B, for example: "When explaining the ABC theory, Model B jumps directly to the conclusion without explaining the intermediate process of deriving Z from Y."]

[0201] Comprehensiveness: [X] / 5 points. [A brief explanation of comprehensiveness in Model B, for example: "Model B explains the basic principles of X, but does not mention the specific differences of X in different application scenarios, nor does it discuss the relevant limitations."]

[0202] Overall evaluation: [Overall evaluation of Model B].

[0203] Improvement instructions: Now, as Model A, please carefully analyze the shortcomings of your previous answer generation based on the evaluation results provided by Model B, and propose an improved answer.

[0204] Please focus on the following aspects and reflect them in your "improved answer": Regarding the low score in "accuracy": If Model B points out a factual error, you need to strengthen fact-checking when generating the answer, prioritize the latest and most reliable sources of information, or explicitly perform internal verification when providing specific facts.

[0205] Regarding the low score in the "Logicality" section: If Model B indicates poor reasoning and disorganized structure, you need a clearer structured organization (e.g., outline first, then fill in the content), a rigorous argumentation process, smooth paragraph transitions, and, if necessary, explanations of the steps.

[0206] Regarding the low score in the "Comprehensiveness" category: If Model B points out that important information, details, or perspectives are missing, you need to fully cover all key aspects of the user's problem and proactively consider relevant background information, potential details, and different perspectives (such as advantages and disadvantages, application scenarios, limitations, etc.).

[0207] Please output: Analysis of the previous round of answer generation process: (Briefly explain, based on the evaluation of Model B, speculate on possible instruction deviations or shortcomings in the previous round of answer generation).

[0208] Assuming you haven't seen the scoring content of Model B, what thought process leads to the improved answer? Please output this thought process.

[0209] <cot> Improved thought process: < / cot> <answer> Improved answer: < / answer> 15. High-quality data output.

[0210] 151. Mixed Dataset: For a given question, Model B's response has the highest average score (11.5 points). Its response and optimized thought process chain are stored in the training dataset for subsequent training of larger models. (Each question has a different model source, ensuring diversity in responses and greater learning potential, but also higher learning difficulty).

[0211] 152. The optimized thought process and answer of each model can also be made into alternative datasets (dataset-A / B / C) for training the same series of models (the model source for each question is the same, which is suitable for training small models of the same series).

[0212] It should be noted that the above Embodiment 1 is merely an exemplary illustration of the model training method provided in this application, and the application of the model training method is not limited to the above scenario. Embodiment 1 uses a multi-model peer review process to score the first response generated by the first language model based on other language models. Then, through a self-optimization process, the first response is optimized based on the scoring results of other models to obtain an optimized first response. Finally, target training data is constructed based on the seed question and the optimized response to train the model to be trained.

[0213] The following is a detailed description of Embodiment 2 proposed in this application.

[0214] Example 2 disables self-optimization, which is suitable for scenarios with limited computing power. It only retains the multi-model peer review process, where each model directly generates responses and scores them. The thought process of the generated data is completely matched with the prompt (without reference answers) for subsequent model training.

[0215] This application uses reinforcement learning to replace peer review scoring, guiding the model to generate high-quality responses through a reward mechanism. The reward mechanism, a concept unique to reinforcement learning algorithms, does not provide the model with a text sequence of the standard answer, but rather scores the answers generated by the model (for example, a simple approach is to have another model score the generated answers). The optimization goal of the original model is to increase this score, rather than striving for the generated results to be as similar as possible to the standard answer. This method allows the model to output according to its own preferences, only constraining the final model preference score / human preference score, thus expanding the model's search freedom.

[0216] The beneficial effects of the technical solution proposed in this application are as follows: 31. Improve training data quality: Through multi-model cross-validation and scoring, effectively filter low-quality responses and retain answers with better accuracy, logic, and comprehensiveness.

[0217] 32. Preserve the supervisory signal of the thought chain: The generated training data contains "answer + thought chain", which helps large models understand the reasoning process and improves their ability to respond to complex tasks (such as mathematical reasoning and logical analysis).

[0218] 33. Reduce computing power consumption: By filtering seed data, only "high-value problems" are processed, reducing the amount of invalid data processing by 90%, which is suitable for large-scale datasets.

[0219] 34. High flexibility: Supports "enable / disable self-optimization", which can balance data quality and generation efficiency according to computing resources.

[0220] Based on the above embodiments, this application also provides a model training device. Figure 4 This is a schematic diagram of the structure of the model training device 40 provided in the embodiments of this application, as shown below. Figure 4 As shown, the model training device 40 includes: Training unit 401 is used to train the model to be trained based on the target training data to obtain the target training model; the target training data is constructed based on the original question data and the target response data; the target response data is determined based on the scoring results of multiple first response data corresponding to the original question data, and the scoring results of multiple first response data are determined based on multiple language models.

[0221] In some embodiments, the model training device 40 further includes: The first output unit is used to input the original question data into multiple language models and obtain the first response data output by each language model; The scoring unit, based on multiple language models, scores each first response data point to obtain a score result for each first response data point; The first determining unit is used to determine the target response data based on the scoring results of each first response data. The building unit is used to construct target training data based on the original question data and target response data.

[0222] In some embodiments, the model training apparatus further includes: The second determining unit is used to determine at least one second response data corresponding to each of the multiple training question data; The third determining unit is used to determine the semantic similarity between every two second response data corresponding to any training question data and the perplexity of the training question data for any training question data; the perplexity represents the prediction complexity of the training question data. The filtering unit is used to filter multiple training question data based on their semantic similarity and perplexity to obtain the original question data.

[0223] In some embodiments, the filtering unit is further configured to: determine the training question data with semantic similarity less than a similarity threshold as question data to be determined; and determine the question data to be determined with perplexity greater than a perplexity threshold as original question data.

[0224] In some embodiments, the third determining unit is further configured to input the question data to be trained into a benchmark language model to obtain the third response data output by the benchmark language model and the prediction probability corresponding to the data element in the third response data; the benchmark language model is one of a plurality of language models, or a language model different from the plurality of language models; and determine the perplexity of the question data to be trained based on the prediction probability corresponding to the data element.

[0225] In some embodiments, the scoring unit is further configured to score the first response data output by the first language model from multiple scoring dimensions based on the second language model, to obtain sub-scoring results corresponding to each scoring dimension; each scoring dimension has multiple scoring values ​​and scoring descriptions corresponding to the scoring values; the first language model is any one of the multiple language models; the second language model is a language model other than the first language model among the multiple language models; and based on the multiple sub-scoring results, determine the scoring result of the first response data output by the first language model.

[0226] In some embodiments, the scoring unit is further configured to, for any scoring dimension, based on a second language model, determine a target scoring description that matches the first response data output by the first language model from multiple scoring descriptions corresponding to the scoring dimension; determine a target scoring value that matches the target scoring description based on a pre-defined correspondence between scoring descriptions and scoring values; and determine a sub-scoring result of the first response data output by the first language model under the scoring dimension based on the target scoring value.

[0227] In some embodiments, the scoring unit is further configured to determine the weight corresponding to each target score value; and to perform a weighted average of the target score values ​​corresponding to multiple scoring dimensions based on the weight corresponding to each target score value, so as to obtain the scoring result of the first response data output by the first language model.

[0228] In some embodiments, the model training apparatus further includes: The fill unit is used to fill in multiple fields to be filled in a predefined prompt template based on the first language model, the second language model, the first response data, and the scoring results, to obtain prompt words; An improvement unit is used to improve the first response data based on prompt words to obtain improved first response data; The fourth determining unit is used to determine the target response data from the improved first response data.

[0229] In some embodiments, the fourth determining unit is further configured to: score the improved first response data based on multiple language models to obtain a scoring result of the improved first response data; and determine target response data in the improved first response data based on the scoring result of the improved first response data.

[0230] In some embodiments, the improvement unit is further configured to, in response to an improvement instruction, improve the first response data based on the prompt word and the improvement strategy carried by the improvement instruction, to obtain improved first response data.

[0231] In some embodiments, the improvement unit is further configured to: obtain a thought process based on the improvement strategy and prompt words; the thought process represents the prompt information when improving the first response data; and improve the first response data based on the thought process to obtain improved first response data.

[0232] In some embodiments, the construction unit is further configured to: determine the original question data as training samples, determine the target response data as sample labels corresponding to the training samples; and construct target training data based on the training samples and sample labels.

[0233] It should be noted that the units included in the model training device provided in this application embodiment can be implemented by a processor in an electronic device; of course, they can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0234] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0235] It should be noted that, in the embodiments of this application, if the above-described model training method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0236] This application provides an electronic device that can implement the model training method described above.

[0237] In one example, reference Figure 5 The electronic device 50, as shown, includes: a processor 501, at least one communication bus 502, a user interface 503, at least one external communication interface 504, and a memory 505. The communication bus 502 is configured to enable communication between these components. The user interface 503 may include a display screen, and the external communication interface 504 may include standard wired and wireless interfaces.

[0238] The memory 505 is configured to store instructions and applications executable by the processor 501, and can also cache data to be processed or already processed by the processor 501 and various modules in the electronic device (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0239] This application provides a storage medium, namely a computer-readable storage medium, on which a computer program or instructions are stored. When the computer program or instructions are executed by a processor, they implement the steps in any of the model training methods provided in the above embodiments.

[0240] This application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, they implement the steps in any of the model training methods provided in the above embodiments.

[0241] It should be noted that the descriptions of the above embodiments of storage media, devices, apparatuses, and program products are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of storage media, devices, apparatuses, and program products of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0242] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0243] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0244] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0245] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0246] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0247] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0248] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: The training model is trained based on the target training data to obtain the target training model; the target training data is constructed based on the original question data and the target response data; the target response data is determined based on the scoring results of multiple first response data corresponding to the original question data, and the scoring results of the multiple first response data are determined based on multiple language models.

2. The method according to claim 1, characterized in that, The method further includes: The original question data is input into the multiple language models to obtain the first response data output by each language model; Based on the multiple language models, each of the first response data is scored to obtain a score result for each of the first response data. Based on the scoring results of each of the first response data, the target response data is determined; Based on the original question data and the target response data, target training data is constructed.

3. The method according to claim 2, characterized in that, Before inputting the original question data into the multiple language models to obtain the first response data output by each language model, the method further includes: Identify at least one second response data point corresponding to each of the multiple training problem data points; For any given training question data, determine the semantic similarity between every two second response data corresponding to the training question data and the perplexity of the training question data; the perplexity characterizes the prediction complexity of the training question data. Based on the semantic similarity and perplexity of multiple training question data, the multiple training question data are filtered to obtain the original question data.

4. The method according to claim 3, characterized in that, The original question data is obtained by filtering the multiple training question data based on their semantic similarity and perplexity, including: The training question data with semantic similarity less than the similarity threshold is identified as the question data to be determined. The question data whose perplexity is greater than the perplexity threshold is identified as the original question data.

5. The method according to claim 3, characterized in that, Determining the perplexity of the problem data to be trained includes: The training question data is input into a benchmark language model to obtain the third response data output by the benchmark language model and the predicted probability corresponding to the data element in the third response data; the benchmark language model is one of the multiple language models, or it is a language model different from the multiple language models. Based on the predicted probabilities corresponding to the data elements, the perplexity of the problem data to be trained is determined.

6. The method according to claim 2, characterized in that, The step of scoring each of the first response data based on the multiple language models to obtain the scoring result corresponding to the first response data includes: The first response data output by the first language model is scored from multiple scoring dimensions based on the second language model, resulting in sub-scoring results corresponding to each scoring dimension; each scoring dimension has multiple scoring values ​​and corresponding scoring descriptions; the first language model is any one of the multiple language models; the second language model is a language model other than the first language model among the multiple language models. Based on the multiple sub-score results, the score result of the first response data output by the first language model is determined.

7. The method according to claim 6, characterized in that, The method of scoring the first response data output by the first language model from multiple scoring dimensions based on the second language model to obtain sub-scoring results corresponding to each scoring dimension includes: For any one of the rating dimensions, based on the second language model, a target rating description that matches the first response data output by the first language model is determined from multiple rating descriptions corresponding to the rating dimension. Based on the pre-defined correspondence between rating descriptions and rating values, a target rating value that matches the target rating description is determined; Based on the target score, the sub-score results of the first response data output by the first language model under the score dimension are determined.

8. The method according to claim 7, characterized in that, The step of determining the rating result of the first response data output by the first language model based on the multiple sub-rating results includes: Determine the weight corresponding to each of the target score values; Based on the weight corresponding to each target score value, a weighted average is performed on the target score values ​​corresponding to multiple score dimensions to obtain the score result of the first response data output by the first language model.

9. The method according to claim 6, characterized in that, The step of determining the target response data based on the scoring results of each of the first response data includes: Based on the first language model, the second language model, the first response data, and the scoring results, multiple fields to be filled in the predefined prompt template are used to obtain prompt words; Based on the prompt words, the first response data is improved to obtain the improved first response data; The target response data is determined from the improved first response data.

10. The method according to claim 9, characterized in that, Determining the target response data from the improved first response data includes: Based on the multiple language models, the improved first response data is scored to obtain the score result of the improved first response data; Based on the scoring results of the improved first response data, the target response data is determined from the improved first response data.

11. The method according to claim 9, characterized in that, The step of improving the first response data based on the prompt words to obtain improved first response data includes: In response to the improvement instruction, the first response data is improved based on the prompt word and the improvement strategy carried by the improvement instruction to obtain the improved first response data.

12. The method according to claim 11, characterized in that, The improvement strategy based on the prompt words and the improvement instructions is used to improve the first response data to obtain the improved first response data, including: Based on the improvement strategy and the prompt words, the thought process is obtained; the thought process represents the prompt information when improving the first response data; Based on the aforementioned thought process, the first response data is improved to obtain the improved first response data.

13. The method according to claim 2 or 9, characterized in that, The construction of target training data based on the original question data and the target response data includes: The original question data is determined as the training sample, and the target response data is determined as the sample label corresponding to the training sample; The target training data is constructed based on the training samples and the sample labels.

14. A model training device, characterized in that, The device includes: The training unit is used to train the model to be trained based on the target training data to obtain the target training model; the target training data is constructed based on the original question data and the target response data; the target response data is determined based on the scoring results of multiple first response data corresponding to the original question data, and the scoring results of the multiple first response data are determined based on multiple language models.

15. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 13.

17. A computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein when read and executed by a computer, the computer program implements the steps of the method of any one of claims 1 to 13.

Images