Model optimization method, electronic device, and medium

Abstract

This application provides a model optimization method, electronic device, and medium, relating to the field of model technology. An Agent is built on a first model, which is used to complete tasks by calling tools. The method includes: acquiring negative samples generated by the Agent; the negative samples include data from multiple rounds of the Agent, including erroneous rounds, which are rounds in which the Agent incorrectly calls the tool; extracting negative segments from the negative samples; the negative segments include data from the erroneous rounds but exclude data from rounds after the erroneous rounds in the negative samples; and optimizing the first model based on the negative segments and corresponding positive segments. This application helps improve model optimization performance.

Description

Technical Field

[0001] This application relates to the field of model technology, and in particular to a model optimization method, electronic device, and medium. Background Technology

[0002] An agent is a proxy that can perceive its environment and take actions to achieve a specific goal. An agent can have the ability to perceive user information, autonomously understand, make decisions, execute actions, and return responses.

[0003] Agents can be built on top of models, such as Large Language Models (LLMs). LLMs are deep learning models trained on large amounts of text data. Agents built on top of LLMs can simulate independent thinking processes and flexibly utilize various tools such as knowledge base retrieval, question answering, and database queries. Agents can include planning, memory, tools, and action components, which can be used for task decomposition and strategy evaluation, information storage and recall, environmental perception and decision support, and translating thought into action, respectively.

[0004] Currently, in complex, goal-oriented agent tool invocation scenarios, single-round alignment reinforcement fine-tuning methods can be used to optimize the model in order to better align the model's behavior with the target preferences. For example, single-round alignment reinforcement fine-tuning methods based on Direct Preference Optimization (DPO) can be used to optimize LLM.

[0005] DPO is an alignment technique for LLM that adjusts model weights by directly utilizing human preference data, enabling the model to generate outputs that better align with human expectations. Specifically, DPO uses paired comparative data to directly optimize model behavior based on which responses are more in line with human expectations, ensuring that the model output remains consistent with human preferences, thereby achieving the goal of strengthening and fine-tuning LLM.

[0006] For example, the process of an LLM handling a user's question can include multiple interaction rounds, such as rounds where the agent calls various tools, computation rounds, etc. In the DPO-based single-round alignment reinforcement fine-tuning method, DPO can identify individual interaction rounds and use the "positive-negative" response pairs of those rounds to optimize the model through a preference loss function.

[0007] In practical applications, although the single-round alignment enhancement and fine-tuning method has shown some effectiveness, its focus on individual rounds of optimization can lead to the problem of "local optima and global failure" in model optimization, which is not conducive to improving the model optimization effect. Summary of the Invention

[0008] This application provides a model optimization method, electronic device, and medium, which help improve the model optimization effect.

[0009] In a first aspect, embodiments of this application provide a model optimization method, wherein an Agent is built on a first model, and the Agent is used to complete a task by calling a tool. The method includes: obtaining negative samples generated by the Agent; the negative samples include data from multiple rounds of the Agent, including erroneous rounds, which are rounds in which the Agent incorrectly calls the tool; extracting negative fragments from the negative samples; the negative fragments include data from the erroneous rounds, but do not include data from rounds after the erroneous rounds in the negative samples; and optimizing the first model based on the negative fragments and the corresponding positive fragments.

[0010] The model optimization method in this application is a fragment-level model optimization method, which uses fragment-level preference data pairs (i.e., positive fragments and negative fragments) for model optimization. The fragment-level preference data pairs can involve one or more interaction rounds in a multi-round interaction session as needed, rather than just a single interaction round.

[0011] By expanding the sampling range from a single interaction round to multiple interaction rounds, the "local optima, global failure" model optimization problem inherent in the single-round alignment-based reinforcement fine-tuning method can be avoided. Furthermore, since the immediate fragment-level preference data pairs involve only one interaction round, which is selected on demand, unlike the fixed single-round alignment-based reinforcement fine-tuning method, this problem can also be avoided.

[0012] Furthermore, the interaction rounds corresponding to the fragment-level preference data pairs can exclude the "error-free rounds" in the negative samples, thus avoiding model training when "error-free rounds" are considered bad outputs. This avoids the "error propagation" problem in session-level model optimization methods caused by the inclusion of "error-free rounds" in the negative samples.

[0013] Optionally, the model optimization method further includes: obtaining positive samples corresponding to the negative samples generated by the Agent based on the data in the negative samples, the erroneous rounds and the previous rounds; and extracting positive segments from the positive samples.

[0014] By sampling positive samples based on the error history of negative samples, rather than completely resampling, the positive samples focus on optimizing and improving the error history, strengthening the causal interpretability of the tool invocation strategy. This ensures that positive segments contain correct rounds corresponding to the error rounds, allowing the first model to learn the correct tool invocation strategy reflected in the correct rounds, thus improving model optimization performance. This avoids the "causal ambiguity" problem caused by completely resampling positive samples in session-level model optimization methods.

[0015] Optionally, extracting negative segments from negative samples includes: extracting negative segments from negative samples based on the number of rounds of positive segments, wherein the number of rounds of positive segments and the number of rounds of negative segments are equal.

[0016] In fragment-level preference data pairs, the interaction rounds of positive fragments and negative fragments can be consistent. This allows the normalization factor Z to be eliminated during model optimization, ensuring the theoretical rigor of the preference loss function and resulting in good model training stability. This avoids the "poor training stability" problem caused by inconsistent interaction round lengths, which exists in session-level model optimization methods.

[0017] Optionally, the positive fragment includes data from consecutive rounds of the Agent calling the tool in the positive sample.

[0018] Complex tool calls require multiple rounds of coherent decision-making. By selecting positive segments from the consecutive rounds of positive samples, the model can be optimized to learn coherent tool call strategies, thereby improving the model's optimization performance.

[0019] Optionally, extracting positive segments from positive samples includes: determining whether the positive samples include irrelevant rounds based on negative and positive samples; irrelevant rounds are rounds that precede the correct rounds in the positive samples and do not affect the correct rounds, and correct rounds are rounds that correspond to the incorrect rounds; if the positive samples include irrelevant rounds, extracting positive segments from the positive samples; positive segments include data from the next round after the irrelevant round to the correct round.

[0020] The interaction rounds corresponding to the fragment-level preference data pairs may exclude "rounds unrelated to the Agent's own tool invocation strategy" in the positive samples, in order to support the model in learning the correct behavior patterns from the positive samples. This avoids the "causal ambiguity" model optimization problem caused by the inclusion of irrelevant rounds in the positive samples, which exists in the session-level model optimization method.

[0021] Optionally, before determining whether the positive samples include irrelevant rounds, the model optimization method further includes: obtaining first information based on the samples generated by the Agent; if the first information indicates that the Agent has redundant tool calls, then controlling the selection of irrelevant rounds so that the number of rounds in the positive segment is not higher than a first threshold; if the first information indicates that the Agent has discontinuous rounds, then controlling the selection of irrelevant rounds so that the number of rounds in the positive segment is not lower than the first threshold.

[0022] By controlling the fragment length within an appropriate range as needed, it helps to avoid inappropriate fragment selection from affecting the model optimization effect, and prevents the optimized model from having excessive redundancy in calling tools or the steps of Agent calling tools from being disjointed.

[0023] Optionally, the erroneous tool invocation includes at least one of: incorrectly configured tool parameters, incorrectly selected tool type, and redundant tool invocation. This helps to accurately identify the key rounds that cause tool invocation failure / task obstruction in negative samples, thereby improving the accuracy of model optimization.

[0024] Optionally, obtaining negative samples generated by the Agent includes: obtaining a first sample generated by the Agent; obtaining a score for the first sample according to the scoring rules; the evaluation dimensions of the scoring rules include at least one of the following: tool call success rate, task completion rate, tool call steps, and number of retries to correct tool call errors; if the score of the first sample is less than a second threshold, the first sample is used as a negative sample.

[0025] By evaluating sample scores based on relevant parameters called by the tool, and distinguishing between positive and negative samples based on the scores, it is helpful to accurately select negative samples, thereby improving the accuracy of model optimization.

[0026] Secondly, embodiments of this application provide a model optimization apparatus. An Agent is built on a first model, and the Agent is used to complete tasks by calling tools. The model optimization apparatus includes: an acquisition module for acquiring negative samples generated by the Agent; the negative samples include data from multiple rounds of the Agent, including erroneous rounds, which are rounds in which the Agent incorrectly calls tools; an extraction module for extracting negative fragments from the negative samples; the negative fragments include data from the erroneous rounds but do not include data from rounds following the erroneous rounds in the negative samples; and an optimization module for optimizing the first model based on the negative fragments and corresponding positive fragments.

[0027] Thirdly, embodiments of this application provide a chip, including: a processor, which is configured to execute computer program instructions stored in a memory, wherein when the computer program instructions are executed by the processor, the chip is triggered to execute the method as described in any of the first aspects.

[0028] Fourthly, embodiments of this application provide an electronic device including at least one processor and a memory coupled together. The memory is used to store computer program instructions, and the processor is used to execute the computer program instructions. When the computer program instructions are executed by the processor, the electronic device is triggered to perform a method as described in any of the first aspects.

[0029] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method as described in any of the first aspects.

[0030] In a sixth aspect, embodiments of this application provide a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform the method as described in any of the first aspects.

[0031] The technical effects of the aforementioned aspects can be referenced from each other, and will not be elaborated further here. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below.

[0033] Figure 1 A schematic flowchart illustrating a model optimization method provided in one embodiment of this application; Figure 2 A flowchart illustrating another model optimization method provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0034] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0035] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0036] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0037] It should be understood that the term "at least one" as used in this document refers to one or more, and "more than one" refers to two or more. The term "and / or" as used in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Additionally, the character " / " in this document generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0038] It should be understood that although the terms "first," "second," etc., may be used to describe the set thresholds in the embodiments of this application, these set thresholds should not be limited to these terms. These terms are only used to distinguish the set thresholds from each other. For example, without departing from the scope of the embodiments of this application, the first set threshold may also be referred to as the second set threshold, and similarly, the second set threshold may also be referred to as the first set threshold.

[0039] In complex, goal-oriented agent tool invocation scenarios, a single-round aligned reinforcement fine-tuning method can be used to optimize the model, that is, using positive and negative response pairs to optimize the model. The negative response in the positive-negative response pair can be data from a specific round of negative samples generated by the agent.

[0040] Taking an agent built on an LLM as an example, the sample generated by this agent can be a multi-round interaction session generated by the agent (such as the session process corresponding to the LLM processing a user's question). Negative samples can be samples with scores below a threshold, and negative responses can be data from one round of this multi-round interaction session.

[0041] The aforementioned fine-tuning refers to the agent learning the optimal decision-making strategy by interacting with the environment and receiving rewards. The agent's goal is to maximize cumulative rewards by selecting the best action in different states.

[0042] For example, a DPO-based single-round alignment reinforcement fine-tuning approach can be used to optimize LLM. In the DPO-based single-round alignment reinforcement fine-tuning approach, the DPO (or what may be called the standard DPO) identifies a single interaction round in the agent's multi-round interaction session and uses the "positive-negative" response pairs of that round to optimize the model through a preference loss function.

[0043] In goal-oriented multi-turn interactive sessions, success often depends on high-quality interaction across multiple rounds. In practical applications, single-round alignment enhancement and fine-tuning methods only focus on preference alignment in a single interactive round, failing to cover the continuity requirements of multi-step tool calls. This limits their ability to model the goal in goal-oriented multi-turn interactive sessions, leading to the "local optimum, global failure" model optimization problem, which is not conducive to improving model optimization performance.

[0044] For example, the interaction flow of complex tool calls can be a multi-round closed loop of "user-agent-tool". For instance, a user inputs a question, and the agent processes it through multiple rounds (including calling various tools) to answer the user's question. Complex tool calls, such as multi-step data analysis, application programming interface (API) chaining, and troubleshooting, rely on multiple rounds of coherent decision-making. For example, the agent might first call a data query tool, then call a visualization tool based on the results, and finally generate a report. While preference alignment in a single interaction round might ensure the correct tool selection in a particular round (i.e., local optimum), it may not lay the groundwork for subsequent steps, leading to overall task failure (i.e., global failure).

[0045] Unlike the single-round alignment reinforcement fine-tuning method mentioned above, in order to more effectively adjust the agent behavior in multi-round interactive sessions (such as the agent's behavior of calling knowledge bases to retrieve questions and answers, database queries and other tools), the multi-round alignment reinforcement fine-tuning method can be used to optimize the model.

[0046] The multi-round alignment enhancement fine-tuning method can expand the sampling range from a single interaction round in a multi-round interactive session to multiple interaction rounds. It can focus on the preference alignment of multiple interaction rounds rather than just the preference alignment of a single interaction round. This avoids the "local optimum, global failure" model optimization problem that exists in the single-round alignment enhancement fine-tuning method mentioned above.

[0047] For example, the aforementioned multi-round alignment enhancement and fine-tuning method can be a session-level model optimization method. This method uses positive and negative sample pairs to optimize the model. The negative samples can then be data from all rounds of the aforementioned multi-round interactive session.

[0048] For example, LLM can be optimized using a multi-round alignment enhancement fine-tuning approach based on Episode-level Trajectory Optimization (ETO) and Direct Multi-Turn Preference Optimization (DMPO).

[0049] ETO and DMPO support session-level (i.e., data from all rounds in a multi-round interactive session) preference optimization. By extending preference alignment from a single interaction round to all interaction rounds, they improve model performance in complex, long-term interactions. Specifically, ETO and DMPO apply an improved DPO loss (different from the standard DPO mentioned above) for model training.

[0050] The multi-round alignment enhancement fine-tuning method based on ETO and DMPO is a session-level model optimization method that treats the entire multi-round interactive session as an optimization unit to optimize LLM. Since this method focuses on the preference alignment of all interaction rounds, it has limitations in terms of data granularity, such as the following problems: "error propagation", "causal ambiguity" and "poor training stability", which are not conducive to improving the model optimization effect.

[0051] Regarding the issue of "mispropagation": Since the model is optimized using data from all rounds, "error-free rounds" in negative samples are also considered bad outputs. This introduces a lot of noise during the fine-tuning process, negatively impacting the model training. For example, "error-free rounds" in negative samples could include previous rounds with correct tool call formats but unreasonable parameters, which are misjudged as "bad outputs," thus introducing training noise.

[0052] Regarding the issue of "ambiguity in causality": The sampling method for positive samples is complete resampling (or sampling from zero), which provides a huge action space for the sample output, which is not conducive to strengthening the causal interpretability of the tool invocation strategy.

[0053] The higher scores of positive samples may be due to changes in the behavior of the interacting parties (such as users or other tools) rather than the tool invocation strategy of the agent itself. The presence of "irrelevant rounds" in positive samples makes it difficult for the model to accurately learn the key correct behavioral patterns from them. For example, it is difficult to accurately learn the tool parameter configuration, tool type selection, and tool invocation error retry logic in positive samples.

[0054] Regarding the issue of "poor training stability": The interaction rounds between positive and negative samples may be of different lengths. When optimizing the model, it is impossible to strictly eliminate the normalization factor Z (i.e., the partitioning function), which leads to the lack of theoretical rigor in the preference loss function and poor model training stability (such as the failure of smooth updates when the tool call steps are of different lengths).

[0055] Unlike the session-level model optimization methods described above, the multi-turn alignment enhancement and fine-tuning method can also be considered a fragment-level model optimization method. This method uses positive and negative fragment pairs to optimize the model. The negative fragments can then be data from a portion of the multi-turn interactive session.

[0056] like Figure 1 As shown, one embodiment of this application provides a model optimization method. This method is a fragment-level model optimization method that optimizes the model through positive and negative fragment pairs to achieve the purpose of enhancement and fine-tuning. See also Figure 1 The model optimization method may include the following steps 101 to 103. A feasible implementation of this method can be found by referring to... Figure 2 The embodiments shown are not described in detail here.

[0057] Figure 1 In the model optimization method shown, an Agent is built on the first model (such as LLM), which is used to complete the task by calling tools.

[0058] For example, the tools invoked by the Agent can be tools of the type such as APIs, code execution, and database queries.

[0059] For example, the tasks completed by the Agent calling the tool can be such as "completing data analysis reports through multi-API collaboration" or "using multiple tools in combination during troubleshooting".

[0060] Optionally, the first model can be a model obtained by training a base model based on expert data. Training the base model using high-quality data helps to balance standardization and innovation in complex tool calls.

[0061] Step 101: Obtain negative samples generated by the Agent; negative samples include data from multiple rounds of the Agent, including erroneous rounds, which are the rounds in which the Agent incorrectly invoked the tool.

[0062] Optionally, a negative sample can be a multi-round interaction session generated by the Agent. For example, suppose the negative sample includes rounds 1 to 10, and the erroneous round is round 3.

[0063] For example, the negative sample can be Figure 2 In step 206 of the illustrated embodiment, any low-scoring sample is selected. The erroneous rounds in the negative samples can be the key rounds obtained in step 207. In one example, a feasible implementation for identifying erroneous rounds can be found in the relevant description in step 207, which will not be repeated here.

[0064] In one embodiment, the erroneous tool invocation may include at least one of: incorrectly configured tool parameters, incorrectly selected tool type, and redundant tool invocation. Identifying erroneous rounds in this way helps to accurately find the key rounds that cause tool invocation failure / task obstruction in negative samples, thereby improving the accuracy of model optimization.

[0065] In one embodiment, obtaining the negative sample generated by the Agent in step 101 may include: obtaining the first sample generated by the Agent; obtaining the score of the first sample according to the scoring rules; and using the first sample as the negative sample if the score of the first sample is less than a second threshold.

[0066] Optionally, the first sample can be any sample generated by the Agent. The value of the second threshold can be set as needed, for example, in... Figure 2 In the embodiment shown, the second threshold can be 7 points.

[0067] For example, the evaluation dimensions of the scoring rules include at least one of the following: tool call success rate, task completion rate, number of tool call steps, and number of retries to correct tool call errors. For example, in... Figure 2 The description of step 206 in the illustrated embodiment can be used to calculate the sample score from the target dimension, namely, based on the tool call success rate and task completion rate.

[0068] By evaluating sample scores based on relevant parameters called by the tool, and distinguishing between positive and negative samples based on the scores, it is helpful to accurately select negative samples, thereby improving the accuracy of model optimization.

[0069] Step 102: Extract negative segments from negative samples; negative segments include data from erroneous rounds, but exclude data from rounds following the erroneous rounds in the negative samples.

[0070] For example, suppose the negative samples include rounds 1 through 10, and the erroneous round is round 3. Then the negative samples include the data from round 3, but exclude the data from rounds 4 through 10. Rounds 4 through 10 may or may not include "error-free rounds".

[0071] Fragment-level preference data pairs can include negative fragments and corresponding positive fragments. In negative samples, the rounds following an erroneous round can include "error-free rounds". Thus, the interaction rounds corresponding to the aforementioned negative fragments can exclude "error-free rounds" from the negative samples, avoiding model training when "error-free rounds" are considered bad outputs. This avoids the "error propagation" model optimization problem found in the aforementioned session-level model optimization methods.

[0072] Optionally, corresponding positive samples can be obtained from negative samples, positive segments can be extracted from positive samples, and corresponding negative segments can be extracted from negative samples based on positive segments, thereby obtaining segment-level preference data pairs.

[0073] In one embodiment, Figure 1 The model optimization method shown may also include: obtaining positive samples corresponding to the negative samples generated by the Agent based on data from the negative samples, the erroneous rounds, and the previous rounds; and extracting positive segments from the positive samples.

[0074] For example, assuming the negative samples include rounds 1 to 10, and the erroneous round is round 3, then the error prior history can be the data from rounds 1 to 3 of the negative samples.

[0075] For an example of a feasible implementation of sampling positive samples, please refer to [link / reference]. Figure 2 The description of step 208 in the illustrated embodiment will not be repeated here. For feasible implementations of extracting positive segments, please refer to [reference needed]. Figure 2 The description of step 209 in the illustrated embodiment will not be repeated here.

[0076] By sampling positive samples based on the error history of negative samples, rather than completely resampling, the positive samples focus on optimizing and improving the error history, strengthening the causal interpretability of the tool invocation strategy. This ensures that positive segments contain correct rounds corresponding to the error rounds, allowing the first model to learn the correct tool invocation strategy reflected in the correct rounds, thus improving model optimization performance. This avoids the "causal ambiguity" problem caused by completely resampling positive samples in session-level model optimization methods.

[0077] In one embodiment, step 102 above, which involves extracting negative segments from negative samples, may include: extracting negative segments from negative samples based on the number of rounds of positive segments, wherein the number of rounds of positive segments and the number of rounds of negative segments are equal.

[0078] For example, in the above example, if the positive segment is the data from rounds 2 to 3 in the positive sample, then the negative segment can be the data from rounds 2 to 3 in the corresponding negative sample, while rounds 1 and 4 to 10 in the negative sample are excluded.

[0079] In fragment-level preference data pairs, the interaction rounds of positive fragments and negative fragments can be consistent. Therefore, when optimizing the model, the normalization factor Z can be eliminated, making the preference loss function theoretically rigorous and resulting in good model training stability. This avoids the "poor training stability" problem caused by inconsistent interaction round lengths in the aforementioned session-level model optimization methods.

[0080] In one embodiment, the positive fragment includes data from consecutive rounds of the agent invoking the tool, found in the positive sample. For example, refer to... Figure 2 The description of step 209 in the illustrated embodiment indicates that the requirements for extracting positive samples may include "the fragment contains consecutive tool call key steps".

[0081] Complex tool calls require multiple rounds of coherent decision-making. By selecting positive segments from the consecutive rounds of positive samples, the model can be optimized to learn coherent tool call strategies, thereby improving the model's optimization performance.

[0082] In one embodiment, extracting positive segments from positive samples may include: determining, based on negative and positive samples, whether the positive samples include irrelevant rounds; irrelevant rounds are rounds that precede the correct rounds in the positive samples and do not affect the correct rounds, and correct rounds are rounds that correspond to the incorrect rounds; if the positive samples include irrelevant rounds, extracting positive segments from the positive samples; positive segments include data from the round following the irrelevant rounds to the correct rounds.

[0083] For example, suppose the positive samples include rounds 1 to 10, the key round of the negative samples is round 3, and round 1 in the positive samples is unrelated to the initial Agent's own tool invocation strategy (i.e., round 1 can be identified as an irrelevant round). Based on this, the key segment of data from rounds 2 to 3 can be extracted from the positive samples as the positive segment.

[0084] The interaction rounds corresponding to the fragment-level preference data pairs may exclude "rounds unrelated to the Agent's own tool invocation strategy" in the positive samples, in order to support the model in learning the correct behavior patterns from the positive samples. This avoids the "causal ambiguity" model optimization problem caused by the inclusion of irrelevant rounds in the positive samples, which exists in the above-mentioned session-level model optimization methods.

[0085] Complex tool calls are highly dependent on "critical steps" (such as parameter configuration, tool type selection, and error retry logic). Figure 1 The illustrated embodiment improves model optimization by focusing on core segments rather than generalizing adjustments based on the entire session.

[0086] In one embodiment, before determining whether positive samples include irrelevant rounds, Figure 1 The model optimization method shown may further include: obtaining first information based on the samples generated by the Agent; if the first information indicates that the Agent has redundant tool calls, then controlling the selection of irrelevant rounds so that the number of rounds for positive segments is not higher than a first threshold; if the first information indicates that the Agent has discontinuous rounds, then controlling the selection of irrelevant rounds so that the number of rounds for positive segments is not lower than a first threshold.

[0087] For example, the first threshold can be the length of the positive segment obtained based on the identification of irrelevant rounds, without controlling the selection of irrelevant rounds.

[0088] Optionally, the first information can be obtained after the previous model optimization. This first information can be used to reflect the quality of the Agent's output samples based on the currently optimized model. Depending on different quality issues, the length of the segments used in the current model optimization process can be controlled to be neither too long nor too short.

[0089] By controlling the fragment length within an appropriate range as needed, it helps to avoid inappropriate fragment selection from affecting the model optimization effect, and prevents the optimized model from having excessive redundancy in calling tools or the steps of Agent calling tools from being disjointed.

[0090] Step 103: Optimize the first model based on the negative segments and the corresponding positive segments.

[0091] For example, positive and negative fragment pairs can be used as training data for the model and input into the first model so that the first model can learn the best tool calling strategies in the positive fragments (such as correct parameter selection, reasonable tool switching, and efficient error correction).

[0092] Figure 1 The model optimization method provided in the illustrated embodiment is a fragment-level model optimization method, which uses fragment-level preference data pairs (i.e., positive fragments and negative fragments) for model optimization. Fragment-level preference data pairs can involve one or more interaction rounds in a multi-round interactive session as needed, rather than just a single interaction round.

[0093] By expanding the sampling range from a single interaction round to multiple interaction rounds, the "local optima, global failure" model optimization problem inherent in the single-round alignment-based reinforcement fine-tuning method can be avoided. Furthermore, since the immediate fragment-level preference data pairs involve only one interaction round, which is selected on demand, unlike the fixed single-round alignment-based reinforcement fine-tuning method, this problem can also be avoided.

[0094] like Figure 2 As shown, one embodiment of this application provides another model optimization method, which may include steps 201 to 213.

[0095] Step 201: Set the target of complex tool calls (such as "multi-API collaborative completion of data analysis reports" or "multi-tool combination use in troubleshooting"), tool type (such as API, code execution, database query) and interaction process (such as "multi-round closed loop of user-agent-tool").

[0096] Step 202: Set the evaluation metrics for complex tool calls. The evaluation dimensions of the metrics can include, for example, the objective dimension and the efficiency dimension.

[0097] For example, the target dimension may include: tool call success rate (such as parameter correctness, format compliance, etc.), task completion rate (such as whether the goal has been achieved), etc. The efficiency dimension may include: tool call steps (which can be used to reflect whether the tool is called redundantly), error retries (which can be used to reflect whether tool call errors are corrected quickly), etc.

[0098] For example, in step 201, hard constraints can be set on the tool invocation format, so the tool invocation success rate in step 202 includes format compliance. For instance, a format compliance penalty term can be added to the preference loss function so that format compliance affects the calculation of the tool invocation success rate.

[0099] Step 203, Set up the base model: Choose a suitable LLM. For example, you can choose an open-source LLM that supports long contexts as the base model to adapt to the multi-round historical context requirements of tool calls.

[0100] Step 204, Collect Expert Data: Generate high-quality multi-turn interactive sessions. Expert data can be used as training data for the base model. Training the base model using high-quality data helps balance the standardization and innovation of complex tool calls.

[0101] In one example, a model with strong tool invocation capabilities can be used as an expert model. Based on the settings in step 201, an agent built on the expert model generates high-quality multi-turn interaction sessions involving "user-agent-tool," which serve as expert data. These high-quality multi-turn interaction sessions may include high-quality tool invocation sessions generated by the agent on the expert model. These high-quality tool invocation sessions contain a complete history of multi-turn interactions (such as user needs, the agent's tool invocation instructions, and the tool's return results).

[0102] In another example, historical interaction data with high task completion rates and few call errors can be filtered from the tool usage logs and used as expert data, without the need to generate additional data based on the aforementioned expert model.

[0103] Step 205, Initial Fine-tuning: Optimize the base model using expert data to generate the initial agent. For example, the fine-tuned base model can be used as the initial model, and the agent built on the initial model can be used as the initial agent.

[0104] For example, the base model can learn tool call formats, parameter selection logic, multi-turn interaction response methods, etc. from expert data to achieve the purpose of fine-tuning the base model.

[0105] In one embodiment, the first model described above can be the initial model.

[0106] After completing the initial fine-tuning, refer to steps 206-210 below. Based on the initial Agent, data pairs aligned with positive and negative fragments can be constructed and used as fragment-level preference data pairs. For example, negative samples can be filtered from the interaction data of the initial Agent to generate corresponding positive samples, and then data pairs aligned with positive and negative fragments can be constructed based on the positive and negative samples.

[0107] Step 206, Negative Sample Screening: Collect low-scoring samples generated by the Agent. After initial fine-tuning, low-scoring samples generated by the initial Agent can be collected.

[0108] For example, an initial model can be used, based on the settings in step 201, to generate a multi-round interaction session involving "user-agent-tool" by the initial agent, as a sample. This sample can include tool invocation sessions generated by the initial agent. Unlike the high-quality tool invocation sessions mentioned above, tool invocation sessions generated by the initial agent usually have tool invocation errors (such as incorrect parameters, incorrect tool type selection, or redundant steps), causing task failure (e.g., incomplete or low-quality completion). Among these, the rounds with tool invocation errors are the key rounds in the sample that lead to tool invocation failure / task obstruction.

[0109] Thus, based on the multiple rounds of interaction between the initial Agent and the tool / user, multiple samples generated by the initial Agent can be collected. Then, based on the settings in step 202, negative samples can be collected from these multiple samples to obtain a negative sample set.

[0110] For example, in one instance, a score can be calculated for a sample based on the aforementioned target dimensions. If a sample's score is less than a threshold (e.g., 7 points), then that sample can be used as a negative sample. Negative samples can contain a complete multi-turn interaction history: user requests, agent call records, and tool return results.

[0111] Step 207, Identify key rounds: Identify the key rounds in negative samples that lead to task failure.

[0112] As mentioned above, the key round can be the round in which the tool made an error in the negative sample. In one example, the key round in the negative sample can be automatically identified based on the tool logs (such as API error codes and parameter validation failure messages). After that, the interaction history of the key round and the rounds preceding it in the negative sample can be obtained, i.e., the error prior history can be obtained.

[0113] For example, you can first identify specific error rounds using the automated rules in the tool's logs. For other types of error rounds, such as successful calls but useless returned data or redundant steps, you can use models to assist in localization and identification.

[0114] For example, assuming the negative samples include rounds 1 to 10, and the critical round is round 3, then the error prior history can be the data from rounds 1 to 3 of the negative samples.

[0115] Step 208, Positive Sample Sampling: Based on the error prior history in the negative samples, positive samples are sampled. By sampling based on the error prior history, the causal interpretability of the tool invocation strategy can be enhanced.

[0116] In one embodiment, a positive sample can be a multi-round interaction session generated by the Agent. For example, assuming a negative sample includes rounds 1 to 10, and the key round of the negative sample is round 3, then a positive sample can be a multi-round interaction session generated by the Agent based on data from rounds 1 to 3 of the negative sample. Thus, a positive sample can include rounds 1 to 10.

[0117] In another embodiment, a positive sample may not be a single multi-round interaction session generated by the Agent. For example, the interaction rounds of a positive sample can correspond to the interaction rounds in the aforementioned error prior history. For example, a positive sample can be a multi-round interaction fragment generated by the Agent based on the data from rounds 1 to 3 of the negative sample. Thus, a positive sample can correspondingly include rounds 1 to 3.

[0118] In one example, multiple high-scoring samples can be randomly generated, and then positive samples can be selected from them. Alternatively, if the quality of the positive samples generated by the initial Agent is relatively stable, positive samples can be sampled from this example.

[0119] For example, the initial agent can regenerate multiple (e.g., 5) high-scoring samples based on the error history of the negative samples. For example, the sample score can be calculated from the aforementioned target dimension (or target dimension and efficiency dimension). If the sample score is greater than a threshold (e.g., 7 points), then the sample is considered a high-scoring sample. High-scoring samples can contain a complete multi-round interaction history. Then, the sample with the highest score among these multiple high-scoring samples can be used as a positive sample.

[0120] In another example, positive samples can be sampled using PPO-assisted sampling. Optionally, this example can be used to sample positive samples when the quality of the positive samples generated by the initial Agent is unstable.

[0121] For example, the PPO-assisted sampling method can be: using the target dimension score as a reward, the initial agent dynamically adjusts the tool invocation strategy during the sampling process, giving priority to generating high-scoring sessions as positive samples.

[0122] Compared to complete resampling, sampling positive samples based on the erroneous prior history of negative samples helps ensure that the high performance of positive samples stems from the optimization of the agent's own tool invocation strategy (such as correcting parameters or changing tools), rather than external factors such as changes in user needs or differences in tool return formats. This avoids the problem of causal ambiguity in session-level methods leading to the model learning the wrong effective strategy.

[0123] Step 209, Positive Segment Selection: Select key segments from the positive samples. Key segments can be those from the positive samples that lead to task success / efficiency improvement.

[0124] After obtaining positive and negative sample pairs (i.e., positive and negative samples), you can select the positive segments that meet the requirements based on the positive and negative sample pairs. For example, you can input the positive and negative sample pairs into a model used to select key segments (such as the GPT-4o model) and prompt it to select key segments from the positive samples according to the set requirements.

[0125] For example, the requirements of this setting may include: (1) the segment starts from the above key rounds; (2) the segment contains consecutive key steps of tool calls (such as "error correction → parameter adjustment → tool recall → result verification"); (3) irrelevant rounds are excluded.

[0126] Irrelevant rounds can include rounds following key rounds and rounds preceding key segments (such as rounds preceding key rounds that are unrelated to the agent's own tool invocation strategy). By excluding irrelevant rounds preceding key segments, the initial model can accurately learn its key correct behavioral patterns from positive samples, thus helping to address the aforementioned problem of "causal ambiguity".

[0127] For example, suppose the positive and negative samples include rounds 1-10, the key round for the negative samples is round 3, and round 1 in the positive samples is unrelated to the initial agent's tool invocation strategy. Based on these requirements, the key segment of data from rounds 2-3 can be extracted from the positive samples as the positive segment. The irrelevant rounds 1 and rounds 4-10 in the positive samples have been excluded from this positive segment.

[0128] By identifying key rounds (see step 207) and filtering key segments (see step 209), it is helpful to accurately pinpoint the core decision-making stages in tool calls (such as parameter configuration, error correction, and tool switching), while excluding non-core interaction rounds such as user intent confirmation and format instructions. This allows optimization resources to be focused on the core pain points of the tool call strategy. This addresses the "local optima, global failure" problem in single-round aligned model optimization methods, which cannot cover the continuity requirements of multi-step tool calls, thus improving model optimization performance. It also addresses the "error propagation" problem caused by including "error-free rounds" and the "causal ambiguity" problem caused by including "irrelevant rounds" in session-level model optimization methods, further improving model optimization performance.

[0129] Step 210, Negative Segment Extraction: Based on the positive segments, extract negative segments from the negative samples that have the same length as the positive segments. By extracting pairs of positive and negative segments with the same length (number of rounds of the positive segment), the rounds of the positive and negative segments are aligned.

[0130] Based on fragment-level alignment, it can adapt to the task characteristics of tool calls, such as the structured task characteristics of tool call steps (such as a fixed process of "query-process-output").

[0131] After extracting the negative segments, positive and negative segment pairs can be obtained, which are segment-level preference data pairs used to optimize the initial model. The interaction history of each segment includes the Agent's tool call instructions and tool return results.

[0132] By extracting negative segments from positive segments, the negative segments can exclude rounds in the negative samples that are irrelevant to the aforementioned rounds. For example, in the example above, if the positive segments are the data from rounds 2-3 in the positive samples, then the negative segments can be the data from rounds 2-3 in the corresponding negative samples, while rounds 1 and 4-10 in the negative samples are excluded. Since the "error-free rounds" in these rounds do not participate in model optimization, this helps to address the aforementioned problem of "error propagation."

[0133] By extracting negative segments of the same length as positive samples from negative samples, the round lengths of positive and negative segments can be kept consistent, thus completely eliminating the normalization factor Z. This addresses the problem of poor training stability caused by inconsistent interaction round lengths in session-level model optimization methods, thereby improving model optimization performance.

[0134] Step 211, Reinforce Fine-tuning: Learn high-quality strategies from positive fragments.

[0135] By performing steps 207-210 on each negative sample in the negative sample set, a set of positive and negative fragment pairs can be obtained. This set of positive and negative fragment pairs can be used as training data for the model and input into the initial model so that the initial model can learn high-quality tool invocation strategies (such as correct parameter selection, reasonable tool switching, and efficient error correction) in positive fragments.

[0136] In one example, the reference policy upon which the initial model learns positive fragments can be the policy of the initial agent.

[0137] In another example, instead of using the initial agent's policy as the reference policy, a dynamic reference policy can be adopted. For instance, as the number of training rounds for reinforcement and fine-tuning increases, the reference policy can be gradually switched from the initial agent's policy to the policy of the previous round of the current training model, in order to avoid policy solidification caused by a large difference between the reference policy and the training model.

[0138] Step 212, Fine-tuning ends: End when conditions are met.

[0139] Optionally, after the initial model completes one traversal of the model training data (i.e., one epoch) to update the model parameters, a validation set for the current optimized model can be obtained. This validation set may include several samples output by the Agent based on the current optimized model. After obtaining the validation set, a score for the validation set can be calculated based on the settings in step 202. For example, the score of the validation set on the target dimension can be calculated.

[0140] For example, the conditions for ending the reinforcement fine-tuning could be that the validation set score has not improved for multiple consecutive epochs, or that the validation set score for the current epoch has reached a required threshold.

[0141] Step 213, Iterative optimization: Based on the model performance, iteratively optimize the model.

[0142] After completing one round of enhancement and fine-tuning, the error rate of the agent's calls to various / specific tools can be calculated (e.g., specific API parameter configurations). If the error rate of a certain type of tool is too high, relevant expert data can be added for that type of tool, and enhancement and fine-tuning can be performed again (i.e., targeted enhancement and fine-tuning for calling that type of tool). That is, using the currently optimized model as the new base model, steps 203 to 213 are executed again. By repeating this process, the goal of iterative optimization of the model can be achieved.

[0143] Furthermore, after each round of enhancement and fine-tuning, the quality of the Agent's output samples based on the current optimized model can be assessed. If the Agent exhibits excessive redundant calls, processing operations to appropriately reduce the fragment length can be performed. If the Agent has inconsistent rounds, processing operations to appropriately increase the fragment length can be performed.

[0144] For example, control instructions can be input into a model used to select key segments to influence the model's strategy for selecting key segments, thereby affecting the length of the segments selected by the model.

[0145] For example, if the agent exhibits excessive redundant calls, a control instruction can be input into the GPT-4o model to instruct for appropriately reducing fragment lengths. In the next enhancement and fine-tuning, the GPT-4o model can select positive fragments from the positive samples based on this control instruction. For the same positive sample, the length of the fragment selected by the GPT-4o model according to this control instruction can be less than or equal to the length of the fragment not selected according to this control instruction (equivalent to the aforementioned first threshold).

[0146] One embodiment of this application provides a model optimization apparatus. An Agent is built on a first model, and the Agent is used to complete tasks by calling tools. The model optimization apparatus includes: an acquisition module for acquiring negative samples generated by the Agent; the negative samples include data from multiple rounds of the Agent, including erroneous rounds, which are rounds in which the Agent incorrectly calls the tool; an extraction module for extracting negative fragments from the negative samples; the negative fragments include data from the erroneous rounds but do not include data from rounds after the erroneous rounds in the negative samples; and an optimization module for optimizing the first model based on the negative fragments and corresponding positive fragments.

[0147] One embodiment of this application provides a chip, including: a processor, which is configured to execute computer program instructions stored in a memory, wherein when the computer program instructions are executed by the processor, the chip is triggered to execute the method described in any embodiment of this application.

[0148] One embodiment of this application provides an electronic device including at least one processor and a memory coupled together. The memory is used to store computer program instructions, and the processor is used to execute the computer program instructions. When the computer program instructions are executed by the processor, the electronic device is triggered to execute the method described in any embodiment of this application.

[0149] One embodiment of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in any embodiment of this application.

[0150] One embodiment of this application provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform the methods described in any embodiment of this application.

[0151] Figure 3 This is a schematic diagram of a computer device provided according to one embodiment of this application. Figure 3 As shown, the computer device 20 in this embodiment includes a processor 21 and a memory 22. The memory 22 stores a computer program 23 that can run on the processor 21. When the computer program 23 is executed by the processor 21, it implements the steps in the method embodiments of this application. To avoid repetition, these steps are not described in detail here. Alternatively, when the computer program 23 is executed by the processor 21, it implements the functions of each model / unit in the device embodiments of this application. To avoid repetition, these functions are not described in detail here.

[0152] Computer device 20 includes, but is not limited to, processor 21 and memory 22. Those skilled in the art will understand that... Figure 3 This is merely an example of computer device 20 and does not constitute a limitation on computer device 20. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0153] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or it can be any conventional processor.

[0154] The memory 22 can be an internal storage unit of the computer device 20, such as a hard disk or RAM of the computer device 20. The memory 22 can also be an external storage device of the computer device 20, such as a plug-in hard disk, Smart Media (SM) card, Secure Digital (SD) card, or FlashCard equipped on the computer device 20. Furthermore, the memory 22 can include both internal and external storage units of the computer device 20. The memory 22 is used to store the computer program 23 and other programs and data required by the computer device. The memory 22 can also be used to temporarily store data that has been output or will be output.

[0155] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0156] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the same or similar parts between the various embodiments of this application can be referred to mutually. For example, the specific working processes of the systems, devices, and units described in the embodiments of this application can be referred to the corresponding processes in the method embodiments of this application, and will not be repeated here.

[0157] The above description is merely a specific embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A model optimization method, characterized in that, A first model is built with an agent, which is used to complete tasks by invoking tools. The method includes: Obtain negative samples generated by the Agent; the negative samples include data from multiple rounds of the Agent, including erroneous rounds, which are the rounds in which the Agent incorrectly invoked the tool; Negative segments are extracted from the negative samples; the negative segments include data from the erroneous round, but do not include data from rounds following the erroneous round in the negative samples; The first model is optimized based on the negative segments and the corresponding positive segments.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the positive sample corresponding to the negative sample generated by the Agent based on the data in the negative sample, the error round, and the previous rounds; Extract the positive fragment from the positive sample.

3. The method according to claim 2, characterized in that, The extraction of negative fragments from the negative samples includes: Based on the number of rounds of the positive segment, the negative segment is extracted from the negative sample, wherein the number of rounds of the positive segment is equal to the number of rounds of the negative segment.

4. The method according to claim 2, characterized in that, The positive segments include data from consecutive rounds of the Agent calling the tool in the positive samples.

5. The method according to claim 2, characterized in that, Extracting the positive fragment from the positive sample includes: Based on the negative samples and the positive samples, determine whether the positive samples include irrelevant rounds; the irrelevant rounds are rounds that precede the correct rounds in the positive samples and do not affect the correct rounds, and the correct rounds are rounds that correspond to the incorrect rounds; If the positive sample includes the irrelevant rounds, the positive segment is extracted from the positive sample; the positive segment includes data from the next round after the irrelevant round to the correct round.

6. The method according to claim 5, characterized in that, Before determining whether the positive sample includes irrelevant rounds, the method further includes: Based on the sample generated by the Agent, first information is obtained; If the first information indicates that the Agent has redundant tool calls, then the selection of irrelevant rounds is controlled so that the number of rounds of the positive segment does not exceed the first threshold. If the first information indicates that the Agent has discontinuous rounds, then the selection of the irrelevant rounds is controlled so that the number of rounds of the positive segment is not less than the first threshold.

7. The method according to any one of claims 1-6, characterized in that, The error invocation tool includes at least one of the following: error configuration tool parameters, error selection tool type, and redundant invocation tool.

8. The method according to any one of claims 1-6, characterized in that, The step of obtaining the negative samples generated by the Agent includes: Obtain the first sample generated by the Agent; According to the scoring rules, the score of the first sample is obtained; the evaluation dimensions of the scoring rules include at least one of the following: tool call success rate, task completion rate, tool call steps, and number of retries to correct tool call errors. If the score of the first sample is less than the second threshold, the first sample is used as the negative sample.

9. An electronic device, characterized in that, The electronic device includes at least one processor coupled to a memory for storing computer program instructions and for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-8.

Images