Model training method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a model training method and device, computer equipment, a readable storage medium and a program product. And respectively outputting the problem information, the thinking chain and the reasoning text corresponding to the task through the plurality of agents in the first model to obtain a target processing result corresponding to the task. And outputting a prediction processing result corresponding to the task through the second model. And based on the matching degree of the prediction processing result and the target processing result, adjusting model parameters of the second model to obtain a trained second model. Compared with a traditional mode of guiding student model training through a teacher model, the method has the advantages that the task is analyzed through multiple agents of the first model to obtain the problem information, the thinking chain and the reasoning text, the information and the prediction processing result of the second model are combined, the second model is trained, and the training efficiency is improved. The second model is trained through the training data provided by the multi-dimensional intelligent agent, and the flexibility of the trained model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, artificial intelligence models have been applied in various fields. In the field of artificial intelligence, model distillation is a key technology for improving the performance of miniaturized and efficient models. Currently, the method of training models through model distillation usually involves a teacher model guiding the training of student models. However, this method of using a teacher model to guide the training of student models lacks flexibility and is difficult to solve complex problems.

[0003] Therefore, current model training methods suffer from the drawback of low flexibility in the trained models. Summary of the Invention

[0004] Therefore, it is necessary to provide a model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the flexibility of the trained model in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a model training method, including:

[0006] The task to be processed is input into the first model; the first model is used to output the problem information, thought chain and reasoning text corresponding to the task through multiple agents respectively; the problem information represents the problem related to the task, the thought chain represents the problem-solving ideas corresponding to the task, and the reasoning text represents the natural language text corresponding to the reasoning process of the task.

[0007] Based on the question information, the thought process, and the reasoning text, the target processing result corresponding to the task is obtained;

[0008] The task is input into the second model to be trained; the second model is used to output the corresponding prediction processing result according to the task.

[0009] Based on the matching degree between the predicted processing result and the target processing result, the model parameters of the second model are adjusted until the preset training termination condition is met, thus obtaining the trained second model.

[0010] Secondly, this application also provides a model training apparatus, comprising:

[0011] The first input module is used to input the task to be processed into the first model; the first model is used to output the problem information, thought chain and reasoning text corresponding to the task through multiple intelligent agents respectively; the problem information represents the problem related to the task, the thought chain represents the problem-solving ideas corresponding to the task, and the reasoning text represents the natural language text corresponding to the reasoning process of the task.

[0012] The acquisition module is used to obtain the target processing result corresponding to the task based on the question information, the thought chain, and the reasoning text;

[0013] The second input module is used to input the task into the second model to be trained; the second model is used to output the corresponding prediction processing result according to the task.

[0014] The training module is used to adjust the model parameters of the second model according to the matching degree between the prediction processing result and the target processing result until the preset training termination condition is met, so as to obtain the trained second model.

[0015] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0016] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0017] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0018] The aforementioned model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product, through multiple agents in the first model, output the problem information, thought chain, and reasoning text corresponding to the task to be processed, to obtain the target processing result corresponding to the task. The second model outputs the corresponding predicted processing result based on the task. Based on the matching degree between the predicted processing result and the target processing result, the model parameters of the second model are adjusted until a preset training termination condition is met, resulting in a trained second model. Compared to the traditional method of using a single teacher model to guide student model training, this application analyzes the task through multiple agents in the first model to obtain problem information, thought chain, and reasoning text. Combining this information with the predicted processing result of the second model, the second model is trained. This achieves training of the second model using training data provided by multi-dimensional agents, improving the flexibility of the trained model. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a model training method in one embodiment;

[0021] Figure 2 This is a flowchart illustrating the model training method in another embodiment;

[0022] Figure 3 This is a structural block diagram of a model training device in one embodiment;

[0023] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0026] In the field of artificial intelligence, model distillation is a key technology for improving the performance of miniaturized and efficient models. Mainstream model distillation methods typically rely on a single, powerful teacher model to guide student models. This single-teacher approach often encounters bottlenecks when dealing with complex tasks involving long logical chains and requiring multi-faceted analysis. Its main problems are: Limited reasoning dimension: When generating guidance information, the single-teacher model can usually only provide a fixed reasoning path or final answer. For complex problems, this single-dimensional guidance cannot fully cover all the logic and knowledge required to solve the problem, resulting in incomplete knowledge learned by the student model and weak ability to apply knowledge to new situations. Lack of process supervision and error correction: The single-teacher approach focuses on the alignment of the final result while neglecting guidance on the reasoning process. How the teacher model arrives at its conclusion is a black box, and the student model cannot learn this structured reasoning ability. If deviations occur in the teacher model's reasoning process, these errors are easily inherited by the student model. Poor task adaptability: The single-teacher approach lacks flexibility when facing different types of tasks. The inability to specifically optimize for different stages of a task (such as problem decomposition, logical reasoning, fact-checking, and language organization) results in inconsistent performance across different tasks. Therefore, designing a distillation mechanism that can simulate expert team collaboration and provide multi-dimensional, end-to-end guidance to systematically improve the reasoning ability of student models on complex tasks and their stability on simple tasks is a pressing issue in the current technological field.

[0027] Based on this, this application analyzes the task through multiple agents of the first model to obtain problem information, thought chain, and reasoning text. Combining this information with the prediction processing results of the second model, the second model is trained. This achieves training of the second model through training data provided by multi-dimensional agents, overcoming the shortcomings of the single teacher model distillation technique in guiding student models to learn structured reasoning ability, as well as the technical difficulty of insufficient robustness on different tasks. It can improve the reasoning ability of student models on complex tasks and the stability on simple tasks, thereby improving the flexibility of the trained model.

[0028] In one embodiment, such as Figure 1 As shown, a model training method is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to a terminal, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server, including the following steps S202 to S208. Wherein:

[0029] Step S202: Input the task to be processed into the first model; the first model is used to output the problem information, chain of thought (CoT) and reasoning text corresponding to the task through multiple agents respectively; the problem information represents the problem related to the task, the chain of thought represents the problem-solving approach corresponding to the task, and the reasoning text represents the natural language text corresponding to the reasoning process of the task.

[0030] The server can be a network device deploying multiple agents and models, and the task to be processed can be a task requiring reasoning and outputting results. The server can input the task to be processed into a first model. This first model can be a teacher model used to generate teacher supervision signals. The first model can be a Large Language Model (LLM), which refers to a deep learning model trained on large-scale text data, adept at generating coherent text, answering questions, translating, and other natural language processing tasks. The first model contains multiple agents. In the field of large language models, an agent refers to a software entity or functional module capable of perceiving the environment, combining the logical reasoning ability of the large model, and making decisions to achieve a specific goal. In large model applications, agents can utilize advanced algorithms (such as Retrieval-Augmented Generation (RAG)) to perform tasks or interact with the environment.

[0031] The first model can output problem information, thought chains, and reasoning text corresponding to the above tasks through multiple agents. Different agents in the first model can output different information. The problem information represents a question related to the task, for example, obtained by the agent asking questions about the input task. The thought chain represents the problem-solving approach for the task. In the field of large language models, a thought chain refers to a method or technique that demonstrates the problem-solving process through step-by-step reasoning. This requires the model to break down complex problems into multiple intermediate reasoning steps, rather than directly outputting the final answer. This improves the model's performance in tasks requiring multi-step thinking, such as mathematical reasoning and logical analysis, and enhances the interpretability and accuracy of the generated results. The reasoning text represents the natural language text corresponding to the reasoning process of the task. For example, the agent obtains the reasoning text by converting and refining the thought chain.

[0032] Step S204: Based on the above problem information, the above thought chain, and the above reasoning text, obtain the target processing result corresponding to the above task.

[0033] The server can output the aforementioned problem information, thought process, and reasoning text based on the various agents in the first model. The server can then determine the target processing result for the task based on this information. The target processing result represents the standard processing result for the task to be processed; that is, when any student model processes the task, the target processing result can be used as the expected output of the student model. In other words, the target processing result can be used to guide the training of the student model.

[0034] Step S206: Input the above task into the second model to be trained; the second model is used to output the corresponding prediction processing result according to the above task.

[0035] The second model to be trained can be the student model corresponding to the first model. The server can input the task into the second model, and the second model can output the corresponding prediction result based on the task. The prediction result can be the output of the second model after the server instructs it to process the task; the server can instruct the second model to output the prediction result according to the pattern of the target processing result output by the first model. That is, the prediction result can also include information such as the question, thought process, and reasoning text.

[0036] Step S208: Based on the matching degree between the prediction processing result and the target processing result, adjust the model parameters of the second model until the preset training termination condition is met, and obtain the trained second model.

[0037] To train the second model, the server can match the predicted processing result with the target processing result to obtain the corresponding matching degree. Based on this matching degree, the server can adjust the model parameters of the second model to improve the matching degree between the predicted processing result and the target processing result. Then, based on the adjusted model parameters, the server returns to step S206 to perform the next round of training. That is, the server uses the matching degree to perform autoregressive training on the second model, adjusting the model parameters in each training iteration until a preset training termination condition is met, at which point the server obtains the trained second model. The training termination condition includes, but is not limited to, a matching degree greater than a preset matching degree threshold within a preset number of training iterations, or the number of training iterations reaching the preset number. The matching degree can be represented by the function value of the cross-entropy loss function.

[0038] In the aforementioned model training method, multiple agents in the first model output the problem information, thought process, and reasoning text corresponding to the task to be processed, respectively, to obtain the target processing result corresponding to the task. The second model outputs the corresponding predicted processing result based on the task. Based on the matching degree between the predicted processing result and the target processing result, the model parameters of the second model are adjusted until the preset training termination condition is met, resulting in the trained second model. Compared with the traditional method of using a teacher model to guide student model training, this application analyzes the task through multiple agents in the first model to obtain problem information, thought process, and reasoning text. Combining this information with the predicted processing result of the second model, the second model is trained. This achieves training of the second model using training data provided by multi-dimensional agents, improving the flexibility of the trained model.

[0039] In one embodiment, the step of outputting the problem information corresponding to the task through the above-mentioned intelligent agent includes: performing semantic analysis on the task through the first intelligent agent in the first model; and outputting the problem information corresponding to the task based on the result of the semantic analysis.

[0040] In this embodiment, the first model can output the question information corresponding to the task through an intelligent agent. For example, the first model includes a first intelligent agent, and the server uses this first intelligent agent to perform semantic analysis on the task and obtain the semantic analysis results. The first model then uses the first intelligent agent, combined with the semantic analysis results, to output the question information corresponding to the task.

[0041] Specifically, the server includes a multi-agent collaboration module, a result fusion module, and a distillation training module. In the multi-agent collaboration module, the first model mentioned above includes four agents, such as a first agent, a second agent, a third agent, and a fourth agent. The task can be a corpus, and the corpus for each task is sequentially processed by the four agents, which operate independently. The first agent can be a questioning agent, whose function is to perform structured decomposition of the input task. For example, the first agent analyzes the semantics of the task through a pre-defined mechanism and generates a set of key questions covering the core elements of the task, forming the aforementioned question information. This allows the macroscopic task to be decomposed into a series of well-defined sub-questions.

[0042] In this embodiment, the server can use the first intelligent agent to perform question analysis on the task, thereby obtaining the corresponding question information. The server uses the question information to train the second model, which serves as the student model, thereby providing training target data to the second model from multiple perspectives and improving the flexibility of the trained second model.

[0043] In one embodiment, the step of outputting the thought chain corresponding to the task through the above-mentioned intelligent agent includes: outputting the thought chain corresponding to the task through the second intelligent agent in the above-mentioned first model based on the above-mentioned problem information and the above-mentioned task.

[0044] In this embodiment, the first model further includes a second agent, which can be used to analyze problem-solving strategies for the task. For example, the server, through the second agent in the first model, outputs a thought chain corresponding to the task based on the aforementioned problem information and the task. Here, the thought chain represents the logical reasoning path corresponding to the task. The second agent can use the aforementioned problem information as guidance to output a problem-solving path for the task, which serves as the thought chain.

[0045] Specifically, the first model in the server can also include a second agent. This second agent can be a thought chain building agent, whose function is to construct a logical reasoning path based on the set of questions provided by the aforementioned question information. Guided by the questions generated by the questioning agent, the second agent, combined with the original task information, generates a logically coherent and clearly defined sequence of reasoning steps (i.e., a thought chain), forming a complete and traceable problem-solving path, including the thought process and the answer.

[0046] In this embodiment, the server can use the second agent in the first model to generate thought chains, thereby using the thought chains to train the second model as a student model, thus providing training target data to the second model from multiple perspectives and improving the flexibility of the trained second model.

[0047] In one embodiment, based on the aforementioned problem information and the aforementioned task, outputting the thought chain corresponding to the aforementioned task includes: outputting the original thought chain corresponding to the aforementioned task based on the aforementioned problem information and the aforementioned task; and modifying the original thought chain through the third intelligent agent in the aforementioned first model to obtain the thought chain corresponding to the aforementioned task; the modification includes one or more of logical consistency modification, factual accuracy modification, and context relevance modification.

[0048] In this embodiment, the first model may further include a third agent, which can be used to verify and repair the thought chain output by the second agent. Specifically, it is used to detect whether the thought chain output by the second agent contains contradictory information or other anomalies. The server can use the second agent in the first model to output the original thought chain corresponding to the task based on the aforementioned problem information and the task. The original thought chain may be an unverified thought chain. The server can use the third agent in the first model to correct the original thought chain to obtain the thought chain corresponding to the task. This correction includes various types of corrections, such as one or more of logical consistency correction, factual accuracy correction, and contextual relevance correction. Logical consistency correction corrects logical inconsistencies in the original thought chain; factual accuracy correction corrects inconsistencies with facts in the original thought chain; and contextual relevance correction corrects errors in contextual association in the original thought chain.

[0049] Specifically, the third agent can be a verification agent, capable of performing correctness checks. The function of the third agent is to verify and correct the generated reasoning sequence. This agent is responsible for evaluating the logical consistency, factual accuracy, and contextual relevance of each step in the thought chain. For identified invalid or suboptimal steps, the third agent can mark and correct them to ensure the overall validity of the reasoning chain. The verified thought chain can then serve as the thought chain corresponding to the aforementioned task.

[0050] Through this embodiment, the server can utilize the third agent in the first model to verify the thought chain, thereby improving the correctness of the thought chain output by the agent.

[0051] In one embodiment, the step of outputting the reasoning text corresponding to the task through the above-mentioned intelligent agent includes: generating natural language text corresponding to the above-mentioned thought chain through the fourth intelligent agent in the above-mentioned first model, thereby obtaining the reasoning text corresponding to the above-mentioned task.

[0052] In this embodiment, the first model further includes a fourth agent. This fourth agent can be used to generate reasoning text. The reasoning text can be natural language text generated by optimizing the thought chain using natural language. The server can use the fourth agent in the first model to perform natural language processing on the thought chain, generating the corresponding natural language text, thereby obtaining the reasoning text for the task. That is, the reasoning text includes text describing the thought chain in natural language.

[0053] Specifically, the fourth agent can be an answer-polishing agent, whose function is to transform validated, structured reasoning chains (thought chains) into high-quality natural language text as reasoning text. For example, the fourth agent can integrate the revised thought chains, generate the final answer, and perform linguistic optimization to obtain reasoning text, thereby improving the fluency and accuracy of the answer.

[0054] Through this embodiment, the server can use the fourth agent in the first model to generate reasoning text, thereby using the reasoning text to train the second model as a student model, thus providing training target data to the second model from multiple perspectives and improving the model flexibility of the trained second model.

[0055] In one embodiment, obtaining the target processing result corresponding to the task based on the aforementioned problem information, the aforementioned thought chain, and the aforementioned reasoning text includes: aggregating the aforementioned problem information, the aforementioned thought chain, and the aforementioned reasoning text to determine a first fusion result; fusing the aforementioned problem information, the aforementioned thought chain, and the aforementioned reasoning text based on the difference information between them to obtain a second fusion result; fusing the aforementioned problem information, the aforementioned thought chain, and the aforementioned reasoning text based on the similarity between them to obtain a third fusion result; and obtaining the target processing result corresponding to the task based on the aforementioned first fusion result, the aforementioned second fusion result, and the aforementioned third fusion result.

[0056] In this embodiment, the server can output the task-related problem information, thought process chain, and reasoning text using the first model described above. The server can integrate these elements to form a teacher supervision signal, i.e., the target processing result. The server can fuse the problem information, thought process chain, and reasoning text using various fusion methods. For example, the server can aggregate the problem information, thought process chain, and reasoning text to determine a first fusion result. The server can also perform difference fusion based on difference information. For example, the server can fuse the problem information, thought process chain, and reasoning text based on the difference information between them to obtain a second fusion result. The server can also perform fusion based on similarity information. For example, the server can fuse the problem information, thought process chain, and reasoning text based on the similarity between them to obtain a third fusion result.

[0057] The server can also obtain the target processing result corresponding to the above task based on the first fusion result, the second fusion result, and the third fusion result.

[0058] Specifically, the server can generate target processing results through the result fusion module. For example, the server integrates the heterogeneous outputs of multiple agents into a unified teacher supervision signal (target processing result) through the result fusion module. The server receives all outputs from the four agents (question set, thought chain, verification information, and reasoning text) through the result fusion module and adopts multi-perspective fusion strategies, including joint fusion, difference fusion, and similarity fusion.

[0059] The three fusion methods are as follows: Joint fusion aggregates the raw information output by each agent to preserve complete data throughout the reasoning process, forming a comprehensive feature representation and yielding the first fusion result. Differential fusion calculates the deviation between the vector representation of each agent's output and the global average representation, aiming to strengthen and highlight the unique features of each reasoning stage, resulting in the second fusion result. Similarity fusion calculates the feature similarity between the outputs of each agent, capturing and aggregating common features to form the third fusion result, extracting high-confidence shared knowledge and core reasoning patterns. The server can then perform a nonlinear transformation on the multidimensional features fused using these strategies (first, second, and third fusion results) to obtain the final teacher supervision signal (target processing result).

[0060] The aforementioned target processing results include high-quality, filtered samples, which are then fused to form a structured training corpus. The training samples corresponding to the target processing results follow a unified format. For example, the format includes a structure containing "instructions," "inputs," and "outputs." Specifically, the server uses the original task text as the "instructions." The server formats the complete problem-solving process, generated collaboratively by multiple agents and filtered and verified, according to a preset template, as the "output." This output is designed as a structured text containing multiple parts, such as a concatenation of "[question]...[thought chain]...[answer]...".

[0061] In this embodiment, the server integrates the originally scattered, multimodal agent outputs into a high-quality, uniformly formatted, and information-complete dataset for supervised fine-tuning (SFT), providing a solid foundation for subsequent distillation training and thereby improving the flexibility of the second model obtained through distillation training.

[0062] In one embodiment, outputting the corresponding prediction processing result based on the above task includes: determining the corresponding prediction question information, prediction thought chain, and prediction inference text based on the above task; and outputting the corresponding prediction processing result based on the above prediction question information, the above prediction thought chain, and the above prediction inference text.

[0063] In this embodiment, the second model described above can output corresponding prediction processing results using the aforementioned task. These prediction processing results include various types of information. For example, the server uses the second model to analyze the question, thought process, and reasoning text based on the task, thereby determining the predicted question information, predicted thought process, and predicted reasoning text corresponding to the task. Therefore, the server can output corresponding prediction processing results based on the predicted question information, the predicted thought process, and the predicted reasoning text.

[0064] Specifically, the server also includes a distillation training module. The server uses the distillation training module to perform distillation training on the second model mentioned above. That is, the server adopts a multi-task supervised fine-tuning training scheme oriented towards structured output, so that the student model (second model) learns to reproduce the complete thinking and expression process of the teacher agent team (first model), rather than just the final answer.

[0065] During training, the server can format the training data. For example, the server can train using a structured corpus generated by the result fusion module. Each data point contains an "instruction" (original task) and a structured "target output" (target processing result), which includes the question information, thought process, and answer text. The server can perform supervised fine-tuning training on the second model. The training process follows a standard autoregressive language model fine-tuning paradigm. For example, the server sets the input information for the second model and its output target: the server uses the aforementioned "instruction" text (task) as input to the student model. The training objective of the student model is to generate a text sequence that perfectly matches the "target output" in an autoregressive manner, given the input instructions. The mathematical objective of the server's training of the second model is to maximize the conditional probability (matching degree) of the model generating the target sequence. The server can achieve this by minimizing the cross-entropy loss function. Minimizing the cross-entropy loss function includes adjusting the model parameters of the second model.

[0066] Through this embodiment, the server can perform supervised fine-tuning training of the second model based on the above-mentioned target processing results and in combination with the cross-entropy loss function, thereby improving the accuracy of the trained second model.

[0067] In one embodiment, adjusting the model parameters of the second model based on the matching degree between the predicted processing result and the target processing result until a preset training termination condition is met, to obtain a trained second model, includes: determining a first loss value based on the similarity between the predicted question information and the question information; determining a second loss value based on the similarity between the predicted thought chain and the thought chain; determining a third loss value based on the similarity between the predicted inference text and the inference text; and adjusting the model parameters of the second model based on the weighted sum of the first loss value, the second loss value, and the third loss value until a preset training termination condition is met, to obtain a trained second model.

[0068] In this embodiment, the prediction processing results include predicted question information, predicted thought chain, and predicted reasoning text. The server's training objective for the second model is to maximize the conditional probability of the second model generating the target sequence (target processing result), which can be achieved by minimizing the cross-entropy loss function. The matching degree mentioned above can be represented by the function value of the cross-entropy loss function. The prediction processing results include information from multiple dimensions, and the server can determine the loss value for each of these dimensions.

[0069] For example, the server can determine a first loss value based on the similarity between the predicted question information and the predicted question information. The first loss value represents the loss corresponding to the matching degree between the question information of the target processing result and the predicted processing result. The server can determine a second loss value based on the similarity between the predicted thought chain and the thought chain. The second loss value represents the loss corresponding to the matching degree between the thought chain of the target processing result and the predicted processing result. The server can determine a third loss value based on the similarity between the predicted inference text and the inference text. The third loss value represents the loss corresponding to the matching degree between the inference text of the target processing result and the predicted processing result.

[0070] Therefore, the server can adjust the model parameters of the second model based on the weighted sum of the first loss value, the second loss value, and the third loss value, until the preset training termination condition is met, thus obtaining the trained second model.

[0071] Specifically, the server trains the second model using structured multi-task loss calculation. The server deconstructs the standard supervised fine-tuning loss function to enable multi-task supervision. While the model macroscopically generates a single, continuous text sequence, when calculating the loss, the server divides the generated sequence into three logical parts based on specific delimiters in the target output text (such as [question], [thought chain], [answer]), and calculates the matching loss for each part against the corresponding portion of the teacher signal (target processing result). This includes: Question Matching Loss (L... Q): This means the server only calculates the loss in the "question information" section, forming the first loss value, which is used to supervise the student model's ability to generate key questions. Inference step matching loss (L... T This means the server only calculates the loss in the "thought chain" part, forming a second loss value, which is used to supervise the student model's ability to construct a logically coherent reasoning process. Answer matching loss (L...) A ): That is, the server only calculates the loss in the "answer" part, forming a third loss value, which is used to supervise the student model's ability to generate high-quality final answers.

[0072] Therefore, the server can combine the above-mentioned total loss into a weighted sum of the losses from these three parts, which can be specifically expressed as: L total =λ Q L Q +λ T L T +λ A L A Among them, L total The weighted sum of the first, second, and third loss values ​​represents the total loss function (cross-entropy loss function). The server updates the parameters of the student model by backpropagating this total loss, and the trained student model (second model) is obtained when the cross-entropy loss function converges.

[0073] In this embodiment, the server uses a deconstructive loss calculation method to enable more precise and targeted optimization and guidance of the student model at different stages of inference during the training process, thereby improving the flexibility of the trained second model.

[0074] In one exemplary embodiment, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating the model training method in another embodiment. In this embodiment, the system logically consists of three core modules: a multi-agent collaboration module, a result fusion module, and a distillation training module. The multi-agent collaboration module receives the input task, and multiple heterogeneous agents within it process and analyze this task in parallel. Subsequently, the result fusion module systematically integrates the outputs of each agent to generate a high-quality, multi-dimensional unified supervision signal. Finally, in the distillation training module, this supervision signal is used as teacher knowledge to guide the parameter optimization of the student model, thereby achieving effective transfer of structured reasoning ability.

[0075] The server includes a multi-agent collaboration module, a result fusion module, and a distillation training module. In the multi-agent collaboration module, the first model mentioned above includes four agents, such as a first agent, a second agent, a third agent, and a fourth agent. The task can be a corpus, and the corpus for each task is processed sequentially by the four agents, which operate independently. The first agent can be a questioning agent, whose function is to perform structured decomposition of the input task. For example, the first agent analyzes the semantics of the task through a pre-defined mechanism and generates a set of key questions covering the core elements of the task, forming the aforementioned question information. This allows the macro-level task to be decomposed into a series of well-defined sub-questions.

[0076] The first model in the server can also include a second agent, which can be a thought chain building agent. Its function is to construct a logical reasoning path based on the set of questions based on the aforementioned problem information. Guided by the questions generated by the questioning agent, the second agent combines the original task information to generate a logically coherent and clearly defined sequence of reasoning steps (i.e., a thought chain), forming a complete and traceable problem-solving path, including the thought process and the answer.

[0077] The third agent can be a verification agent, capable of verifying correctness. Its function is to verify and correct the generated reasoning sequence. This agent is responsible for evaluating the logical consistency, factual accuracy, and contextual relevance of each step in the thought chain. Specifically, for identified invalid or suboptimal steps, the third agent can mark and correct them to ensure the overall validity of the reasoning chain. The verified thought chain can then serve as the thought chain for the aforementioned task.

[0078] The fourth agent can be an answer-polishing agent, whose function is to transform validated, structured reasoning chains (thought chains) into high-quality natural language text as reasoning text. For example, the fourth agent can integrate the revised thought chain, generate the final answer, and perform linguistic optimization to obtain reasoning text, thereby improving the fluency and accuracy of the answer.

[0079] The server can generate target processing results through the result fusion module. For example, the server integrates the heterogeneous outputs of multiple agents into a unified teacher supervision signal (target processing result) through the result fusion module. The server receives all outputs from four agents (question set, thought chain, verification information, and reasoning text) through the result fusion module and adopts multi-perspective fusion strategies, including joint fusion, difference fusion, and similarity fusion.

[0080] The three fusion methods are as follows: Joint fusion aggregates the raw information output by each agent to preserve complete data throughout the reasoning process, forming a comprehensive feature representation and yielding the first fusion result. Differential fusion calculates the deviation between the vector representation of each agent's output and the global average representation, aiming to strengthen and highlight the unique features of each reasoning stage, resulting in the second fusion result. Similarity fusion calculates the feature similarity between the outputs of each agent, capturing and aggregating common features to form the third fusion result, extracting high-confidence shared knowledge and core reasoning patterns. The server can then perform a nonlinear transformation on the multidimensional features fused using these strategies (first, second, and third fusion results) to obtain the final teacher supervision signal (target processing result).

[0081] The aforementioned target processing results include high-quality, filtered samples, which are then fused to form a structured training corpus. The training samples corresponding to the target processing results follow a unified format. For example, the format includes a structure containing "instructions," "inputs," and "outputs." Specifically, the server uses the original task text as the "instructions." The server formats the complete problem-solving process, generated collaboratively by multiple agents and filtered and verified, according to a preset template, as the "output." This output is designed as a structured text containing multiple parts, such as a concatenation of "[question]...[thought chain]...[answer]...".

[0082] The server also includes a distillation training module. The server uses the distillation training module to perform distillation training on the second model mentioned above. That is, the server adopts a multi-task supervised fine-tuning training scheme oriented towards structured output, so that the student model (second model) learns to reproduce the complete thinking and expression process of the teacher agent team (first model), rather than just the final answer.

[0083] During training, the server can format the training data. For example, the server can train using a structured corpus generated by the result fusion module. Each data point contains an "instruction" (original task) and a structured "target output" (target processing result), which includes the question information, thought process, and answer text. The server can perform supervised fine-tuning training on the second model. The training process follows a standard autoregressive language model fine-tuning paradigm. For example, the server sets the input information for the second model and its output target: the server uses the aforementioned "instruction" text (task) as input to the student model. The training objective of the student model is to generate a text sequence that perfectly matches the "target output" in an autoregressive manner, given the input instructions. The mathematical objective of the server's training of the second model is to maximize the conditional probability (matching degree) of the model generating the target sequence. The server can achieve this by minimizing the cross-entropy loss function.

[0084] The server trains the second model using structured multi-task loss calculation. The server deconstructs the standard supervised fine-tuning loss function to enable multi-task supervision. Macroscopically, the model generates a single, continuous text sequence; however, when calculating the loss, the server divides the generated sequence into three logical parts based on specific delimiters in the target output text (such as [question], [thought chain], [answer]), and calculates the matching loss for each part against the corresponding portion of the teacher signal (target processing result). This includes: Question Matching Loss (L... Q ): This means the server only calculates the loss in the "question information" section, forming the first loss value, which is used to supervise the student model's ability to generate key questions. Inference step matching loss (L... T This means the server only calculates the loss in the "thought chain" part, forming a second loss value, which is used to supervise the student model's ability to construct a logically coherent reasoning process. Answer matching loss (L...) A ): That is, the server only calculates the loss in the "answer" part, forming a third loss value, which is used to supervise the student model's ability to generate high-quality final answers.

[0085] Therefore, the server can combine the above-mentioned total loss into a weighted sum of the losses from these three parts, which can be specifically expressed as: L total =λ Q L Q +λ T L T +λ A L A Among them, L totalThe weighted sum of the first, second, and third loss values ​​represents the total loss function (cross-entropy loss function). The server updates the parameters of the student model by backpropagating this total loss, and the trained student model (second model) is obtained when the cross-entropy loss function converges.

[0086] In one embodiment, the above training process can be performed according to the following steps:

[0087] Step 1, Task Reception: The system receives an input task, such as a natural language problem that requires complex data analysis and logical judgment.

[0088] Step 2, Parallel Processing by Multiple Agents: The questioning agent analyzes the question and outputs a set of key sub-questions. The thought chain building agent generates preliminary, sequential problem-solving steps based on the sub-questions. The verification agent checks each step, correcting any logical fallacies or factual errors, and outputs a corrected thought chain. The answer polishing agent transforms the corrected thought chain into a precise and fluent text answer (reasoning text).

[0089] In the agent verification stage, the server can deploy multiple independent judging models, including a logical consistency detection model, a factual accuracy verification model, and a reasoning coherence evaluation model. These models perform multi-dimensional evaluations of the generated thought chains, with each model ultimately outputting a quality score from 0 to 1. The server, trained on a labeled dataset of logical contradictions, can accurately identify errors such as circular reasoning or contradictions between premises and conclusions. By outputting the probability of contradiction, it assesses logical consistency, making the evaluation more quantifiable and providing crucial evidence for selecting high-quality thought chains. The factual accuracy verification model, utilizing knowledge retrieval and embedding similarity combined with vector database queries and semantic similarity calculations, can comprehensively check the factual accuracy within the thought chain, thus more accurately identifying factual errors and improving the factual quality of the thought chain. Furthermore, the reasoning coherence evaluation model, employing a self-attention architecture and mechanism, can calculate the contextual relationships between thought chain steps, quantify the strength of connections between steps, and identify issues such as connection gaps. This allows for a detailed evaluation of reasoning coherence, helping to select thought chains with coherent reasoning.

[0090] The server can integrate scoring results from multiple sources, first standardizing and truncating the scores, and then employing a combination of score consistency screening and high-confidence screening strategies to select high-quality thought chain samples. This ensures that the score distribution of all models conforms to a standard normal distribution, handles extreme values ​​to avoid outlier interference, and guarantees the fairness and accuracy of the screening process. Furthermore, by combining score consistency screening and high-confidence screening to assess sample quality from different perspectives, it can more comprehensively select high-quality samples with fewer logical conflicts, lower factual error rates, and coherent reasoning, making it more reliable than a single screening strategy.

[0091] In the thought chain generation stage, the server selects a generation model based on task characteristics, sets an 'n' parameter to generate multiple candidate samples, controls the consistency of the distribution of inference content, sets a maximum generation length to control the generation length, and employs a hierarchical sampling strategy to ensure that the distribution of generated data matches the actual application scenario, generating a thought chain containing the complete intermediate inference process. This allows for the selection of an appropriate model based on the task type, better meeting the needs of different tasks and improving the quality of thought chain generation. Furthermore, by setting the 'n' parameter, the maximum generation length, and the hierarchical sampling strategy, the generated thought chains are diverse and reasonably distributed, covering more practical application scenarios and providing rich and effective data for subsequent evaluation and training.

[0092] Step 3, Result Fusion: The result fusion module receives all the heterogeneous outputs mentioned above and applies the joint, difference, and similarity fusion strategies to generate a structured, high-dimensional teacher supervision signal (target processing result).

[0093] Step 4, Distillation Training: The original task is input into the student model. The student model outputs its predicted sub-questions, reasoning steps, and final answer. The multi-task loss between each output of the student model and the corresponding part of the teacher supervision signal is calculated. Based on the calculated total loss, the network parameters of the student model are updated using the backpropagation algorithm.

[0094] The server trains student models using selected high-quality samples, employing supervised fine-tuning or reinforcement learning. In reinforcement learning, the reward function is a weighted sum of normalized scores from multiple referees, guiding the student model to generate high-scoring thought processes. The system supports dynamic updates to the referee model library. By training student models with high-quality samples that have fewer logical conflicts, lower factual error rates, and coherent reasoning, the system can more effectively learn correct reasoning logic, improving the model's reasoning reliability and logical consistency. Simultaneously, the server dynamically updates the referee model library based on the error types of the student models on the validation set, continuously optimizing the quality of distilled data and forming a closed-loop optimization mechanism, enabling the model training to constantly adapt to new situations.

[0095] Specifically, when the server detects a new error type in a student model, it can collect samples for manual annotation, fine-tune and train a new referee model based on the Qwen model, add it to the referee database, adjust the evaluation process, and set dynamic update thresholds for the model database. This allows for the rapid training of new referee models for new error types, their integration into the evaluation system, making the evaluation more comprehensive and accurate, continuously optimizing the selection of thought chains and the model training effect, and maintaining the system's adaptability and effectiveness.

[0096] Step 5, Iterative Optimization: Repeat Step 4, iterating through multiple rounds of training until the total loss of the student model converges below a preset threshold, resulting in a trained student model. Thus, the student model not only learns the solution methods for a specific task, but its internal parameters are optimized, enabling it to master the structured abilities of task decomposition, reasoning construction, and answer generation, thereby possessing stronger generalization performance when facing unknown tasks.

[0097] Through the above embodiments, the server analyzes the task using multiple agents of the first model to obtain problem information, thought processes, and reasoning text. Combining this information with the prediction results of the second model, the server trains the second model. This allows for training the second model using training data provided by multi-dimensional agents, improving the flexibility of the trained model. Furthermore, by dividing the work among multiple agents to cover the entire reasoning process and fusing the results to provide multi-perspective supervision signals, it is particularly suitable for complex tasks requiring multi-step reasoning. For complex tasks, it can decompose the reasoning difficulties to ensure accuracy; for simple tasks, it can improve robustness and prevent single-step errors from causing overall failure.

[0098] Furthermore, it can significantly improve the logical reasoning and problem generalization abilities of student models in complex task scenarios, enhancing the performance robustness and reliability of student models across different tasks. This enables student models not only to reproduce the output of teacher models but also to learn their inherent, structured reasoning processes, thereby achieving superior overall performance with lower computational costs.

[0099] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0100] Based on the same inventive concept, this application also provides a model training apparatus for implementing the model training method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more model training apparatus embodiments provided below can be found in the limitations of the model training method described above, and will not be repeated here.

[0101] In one exemplary embodiment, such as Figure 3 As shown, a model training device is provided, including: a first input module 500, an acquisition module 502, a second input module 504, and a training module 506, wherein:

[0102] The first input module 500 is used to input the task to be processed into the first model; the first model is used to output the problem information, thought chain and reasoning text corresponding to the task through multiple intelligent agents respectively; the problem information represents the problem related to the task, the thought chain represents the problem-solving ideas corresponding to the task, and the reasoning text represents the natural language text corresponding to the reasoning process of the task.

[0103] The acquisition module 502 is used to obtain the target processing result corresponding to the above task based on the above problem information, the above thought chain and the above reasoning text.

[0104] The second input module 504 is used to input the above task into the second model to be trained; the second model is used to output the corresponding prediction processing result according to the above task.

[0105] The training module 506 is used to adjust the model parameters of the second model based on the matching degree between the prediction processing result and the target processing result until the preset training termination condition is met, thus obtaining the trained second model.

[0106] In one embodiment, the first input module 500 is used to perform semantic analysis on the task through the first agent in the first model; and output the question information corresponding to the task based on the result of the semantic analysis.

[0107] In one embodiment, the first input module 500 is used to output the thought chain corresponding to the task based on the problem information and the task by the second agent in the first model.

[0108] In one embodiment, the first input module 500 is used to output the original thought chain corresponding to the task based on the problem information and the task; and to modify the original thought chain through the third agent in the first model to obtain the thought chain corresponding to the task; the modification includes one or more of logical consistency modification, factual accuracy modification and context relevance modification.

[0109] In one embodiment, the first input module 500 is used to generate natural language text corresponding to the thought chain through the fourth agent in the first model, thereby obtaining the reasoning text corresponding to the task.

[0110] In one embodiment, the acquisition module 502 is configured to aggregate the problem information, the thought process, and the reasoning text to determine a first fusion result; fuse the problem information, the thought process, and the reasoning text based on the differences between them to obtain a second fusion result; fuse the problem information, the thought process, and the reasoning text based on the similarity between them to obtain a third fusion result; and obtain the target processing result corresponding to the task based on the first fusion result, the second fusion result, and the third fusion result.

[0111] In one embodiment, the second input module 504 is used to determine the corresponding prediction question information, prediction thought chain, and prediction reasoning text according to the above task; and output the corresponding prediction processing result according to the prediction question information, the prediction thought chain, and the prediction reasoning text.

[0112] In one embodiment, the training module 506 is configured to determine a first loss value based on the similarity between the predicted question information and the question information; determine a second loss value based on the similarity between the predicted thought chain and the thought chain; determine a third loss value based on the similarity between the predicted inference text and the inference text; and adjust the model parameters of the second model based on the weighted sum of the first loss value, the second loss value, and the third loss value until a preset training termination condition is met, thereby obtaining the trained second model.

[0113] Each module in the aforementioned model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0114] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores XX data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a model training method.

[0115] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the model training method described above.

[0117] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the model training method described above.

[0118] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the model training method described above.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0120] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0122] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model training method, characterized in that, The method comprises: inputting a task to be processed into a first model; the first model is used to output problem information, a thinking chain and reasoning text corresponding to the task by a plurality of agents respectively; the problem information represents a question related to the task, the thinking chain represents a problem-solving idea corresponding to the task, and the reasoning text represents natural language text corresponding to a reasoning process of the task; obtaining a target processing result corresponding to the task according to the problem information, the thinking chain and the reasoning text; inputting the task into a second model to be trained; the second model is used to output a corresponding predicted processing result according to the task; adjusting model parameters of the second model according to a matching degree of the predicted processing result and the target processing result until a preset training end condition is met, and obtaining a trained second model.

2. The method of claim 1, wherein, The step of outputting the problem information corresponding to the task by the agent comprises: performing semantic analysis on the task by a first agent in the first model; outputting the problem information corresponding to the task according to a result of the semantic analysis.

3. The method of claim 1, wherein, The step of outputting the thinking chain corresponding to the task by the agent comprises: outputting the thinking chain corresponding to the task based on the problem information and the task by a second agent in the first model.

4. The method of claim 3, wherein, The step of outputting the thinking chain corresponding to the task based on the problem information and the task comprises: outputting an original thinking chain corresponding to the task based on the problem information and the task; correcting the original thinking chain by a third agent in the first model to obtain the thinking chain corresponding to the task; the correction comprises one or more of logical consistency correction, fact accuracy correction and context relevance correction.

5. The method of claim 1, wherein, The step of outputting the reasoning text corresponding to the task by the agent comprises: generating natural language text corresponding to the thinking chain by a fourth agent in the first model to obtain the reasoning text corresponding to the task.

6. The method of claim 1, wherein, The step of obtaining the target processing result corresponding to the task according to the problem information, the thinking chain and the reasoning text comprises: aggregating the problem information, the thinking chain and the reasoning text to determine a first fusion result; fusing the problem information, the thinking chain and the reasoning text according to difference information between the problem information, the thinking chain and the reasoning text to obtain a second fusion result; fusing the problem information, the thinking chain and the reasoning text according to similarity between the problem information, the thinking chain and the reasoning text to obtain a third fusion result; obtaining the target processing result corresponding to the task according to the first fusion result, the second fusion result and the third fusion result.

7. The method of claim 1, wherein, The step of outputting the corresponding predicted processing result according to the task comprises: determining corresponding predicted problem information, a predicted thinking chain and a predicted reasoning text according to the task; outputting the corresponding predicted processing result according to the predicted problem information, the predicted thinking chain and the predicted reasoning text.

8. The method according to any one of claims 1 to 7, characterized in that, The prediction processing result includes predicted question information, a predicted thinking chain, and predicted reasoning text; The adjusting of the model parameters of the second model according to the matching degree between the prediction processing result and the target processing result comprises: determining a first loss value according to the similarity between the predicted question information and the question information; determining a second loss value according to the similarity between the predicted thinking chain and the thinking chain; determining a third loss value according to the similarity between the predicted reasoning text and the reasoning text; adjusting the model parameters of the second model according to the weighted sum of the first loss value, the second loss value, and the third loss value until a preset training end condition is met, to obtain a trained second model.

9. A model training apparatus, comprising: The device comprises: a first input module configured to input a task to be processed into a first model; the first model is configured to output question information, a thinking chain, and reasoning text corresponding to the task through a plurality of intelligent agents; the question information represents a question related to the task, the thinking chain represents a problem-solving approach corresponding to the task, and the reasoning text represents a natural language text corresponding to a reasoning process of the task; an obtaining module configured to obtain a target processing result corresponding to the task according to the question information, the thinking chain, and the reasoning text; a second input module configured to input the task into a second model to be trained; the second model is configured to output a corresponding prediction processing result according to the task; a training module configured to adjust model parameters of the second model according to the matching degree between the prediction processing result and the target processing result until a preset training end condition is met, to obtain a trained second model. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

Images