An agent toolkit extension training method based on counterfactual anchor context

Abstract

The application relates to the technical field of artificial agents, and discloses an agent tool set extension training method based on counterfactual anchor context, which comprises the following steps: obtaining a failure trajectory of an agent under an extended tool set; analyzing and identifying a key decision round influencing subsequent reasoning as a counterfactual anchor point; truncating the trajectory to retain the history before the anchor point, generating an alternative decision action, executing the action in a tool environment to obtain feedback, and constructing a candidate counterfactual context; controlling a student agent to continue writing and implementing the task to check the task completion degree and the substantial effectiveness of the anchor point, screening a target counterfactual context; locally supervising and fine-tuning the agent based on the target context; and updating parameters through reinforcement learning on the extended tool set to obtain a target agent. The application can break the behavior inertia of the original tool, improve the adoption rate of the new tool, and realize efficient and low-cost incremental training of the agent by combining local updating and reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence agent technology, and in particular to an extended training method for an intelligent agent toolset based on counterfactual anchor context. Background Technology

[0002] Tool-enhanced agents typically employ a multi-turn interaction approach involving thinking, action, and observation to invoke external tools to complete complex tasks. Existing technologies usually train agents on a fixed set of tools to develop stable inference paths and tool selection strategies. When the toolset expands, the trained strategy often struggles to effectively explore and utilize new tools due to behavioral inertia. For example, the agent might under-invoke or misuse new tools, or abandon the task prematurely after obtaining a valid result, leading to task failure.

[0003] To address the toolkit expansion problem, existing techniques mainly include directly injecting tool descriptions, re-performing reinforcement learning, distilling with complete teacher trajectories, or trajectory correction based on fixed locations. However, these methods have their own drawbacks: injecting tool descriptions alone cannot overcome the behavioral inertia of already trained policies; de novo reinforcement learning faces problems such as reward sparsity, low exploration efficiency, and degradation of original abilities; distilling complete teacher trajectories is costly, and there is a discrepancy between the teacher trajectory distribution and the student's own continuation distribution; trajectory correction at fixed locations cannot determine whether local corrections truly contribute to task success. Existing methods lack a training mechanism that automatically finds high-leverage decision rounds, constructs counterfactual contexts without future information leakage, and allows students to verify and locally internalize these contexts without retraining the entire agent.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention aims to provide a method, apparatus, device and storage medium for expanding the intelligent agent toolset based on counterfactual anchor context. It aims to solve the technical problems of low adoption rate of new tools, poor exploration efficiency and degradation of original capabilities caused by behavioral inertia when expanding the intelligent agent toolset, so as to achieve an efficient, accurate incremental training effect that maintains the original capabilities.

[0006] The first aspect of this invention provides a method for extending the training of an agent's toolset based on counterfactual anchor context, comprising the steps of: acquiring failure trajectories generated by the agent performing tasks under the extended toolset; analyzing the failure trajectories to identify key decision rounds affecting subsequent inference paths, and determining the key decision rounds as counterfactual anchors; truncating the failure trajectories to retain only historical information before the counterfactual anchors, generating alternative decision actions based on the historical information, and executing the alternative decision actions using the tool environment to obtain environmental feedback results; and combining the truncated historical information, alternative decision actions, and corresponding environmental feedback results to construct candidate counterfactual anchors. In fact, the following steps are taken: The student agent continues to execute the task based on the candidate counterfactual context to obtain a continuation trajectory; the continuation trajectory is checked for task completion, and the counterfactual anchors are checked for substantial validity; a target counterfactual context that passes both checks is selected; based on the target counterfactual context, the agent is locally supervised and fine-tuned so that the loss function only applies to the alternative decision action, thereby updating the agent's model parameters; reinforcement learning is performed on the locally supervised and fine-tuned agent on the extended toolset to further update the agent's model parameters, resulting in a target agent that has completed the toolset extended training.

[0007] A second aspect of the present invention provides an extended training apparatus for an agent toolset based on counterfactual anchor context, comprising: a trajectory acquisition module for acquiring failure trajectories generated by an agent performing tasks under an extended toolset; an anchor identification module for analyzing the failure trajectories, identifying key decision rounds that affect subsequent inference paths, and determining the key decision rounds as counterfactual anchors; and a context construction module for truncating the failure trajectories to retain only historical information before the counterfactual anchors, generating alternative decision actions based on the historical information, executing the alternative decision actions using the tool environment to obtain environmental feedback results, and combining the truncated historical information, alternative decision actions, and corresponding environmental feedback results to construct candidate counterfactual anchors. The system comprises: a real context; a verification and filtering module, used to control the student agent to continue executing the task based on the candidate counterfactual context to obtain a continuation trajectory, verify the task completion degree of the continuation trajectory and verify the substantial validity of the counterfactual anchor points, and filter out the target counterfactual context that passes both verifications; a local fine-tuning module, used to perform local supervised fine-tuning of the agent based on the target counterfactual context, so that the loss function only applies to the alternative decision action, thereby updating the model parameters of the agent; and a reinforcement learning module, used to perform reinforcement learning on the agent that has undergone local supervised fine-tuning on the extended toolset, thereby further updating the model parameters of the agent and obtaining the target agent that has completed the toolset extended training.

[0008] A third aspect of the present invention provides an electronic device comprising: a memory and at least one processor, the memory storing computer-readable instructions, the memory and the at least one processor being interconnected via a circuit; the at least one processor invoking the computer-readable instructions in the memory to cause the electronic device to perform various steps of the agent toolset extension training method based on counterfactual anchor context as described above.

[0009] A fourth aspect of the present invention provides a computer-readable storage medium storing computer-readable instructions that, when executed on a computer, cause the computer to perform the steps of the agent toolset extension training method based on counterfactual anchor context as described above.

[0010] Beneficial Effects: This invention first identifies key decision rounds affecting subsequent inference paths by analyzing failure trajectories, using these as counterfactual anchors. Based on historical information prior to these anchors, alternative decision actions are generated to construct candidate counterfactual contexts without future information leakage. Subsequently, a dual verification mechanism—combining task completion verification and anchor effectiveness verification—selects high-quality contexts for local supervised fine-tuning, focusing training on high-leverage behaviors that substantially contribute to task success. Finally, reinforcement learning is performed on an expanded toolset to integrate new behaviors and further explore new approaches. This method, by locating and correcting key erroneous decisions in failure trajectories, effectively breaks the agent's behavioral inertia towards existing tools and inference paths, thereby improving its ability to adopt new tools. Simultaneously, because the training process focuses on key anchors and incorporates dual verification, it significantly improves the relevance and efficiency of training data utilization, avoiding ineffective training. Furthermore, the sequential combination of local supervised fine-tuning and subsequent reinforcement learning can effectively reduce the risk of degradation of existing tool calling capabilities and task processing capabilities while introducing new tool usage capabilities, achieving efficient, accurate incremental training that maintains the original capabilities. In addition, this invention does not require retraining the agent from scratch. The entire training process is based on targeted corrections of local anchor points in the failure trajectory, which significantly reduces the amount of training data, GPU computing resources, and manual annotation costs required for toolset expansion, and has good industrial applicability and economy. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating the extended training method for an agent toolset based on counterfactual anchor context provided in this embodiment of the invention.

[0014] Figure 2 This is a schematic diagram of the structure of the intelligent agent toolset extension training device based on counterfactual anchor context provided by the present invention.

[0015] Figure 3 This is a schematic diagram of the electronic device structure provided by the present invention. Detailed Implementation

[0016] This invention provides a method, apparatus, device, and storage medium for extending the training of an intelligent agent toolset based on counterfactual anchor context. The terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0017] This invention is mainly applied to the field of tool-enhanced agents driven by large language models. In particular, when the set of external tools on which such agents depend is expanded or updated, it aims to provide an efficient incremental training method to overcome the behavioral inertia of trained agents towards existing tools and inference paths, improve the adoption rate of new tools, and avoid the degradation of existing capabilities.

[0018] Before proceeding with a detailed explanation, let's first clarify the key terms used in the claims and specification: Failure trajectory: refers to a complete multi-round interaction sequence record generated by an agent when performing a specific task, from the start of the task to the end of the task failure. This record contains a series of consecutive rounds, each of which usually includes the thought content generated by the model, the selected tool invocation action, and the observation results obtained from the tool environment.

[0019] Counterfactual anchors are decision points identified at a certain stage in a failure trajectory. The original decision at that stage is considered to be the key factor that leads to or exacerbates subsequent failures and therefore needs to be corrected. Counterfactual anchors are the starting point for constructing counterfactual reasoning.

[0020] Candidate counterfactual context: refers to a new starting point for task execution formed by extracting all historical information before the counterfactual anchor point and creating a new decision action and its corresponding environmental feedback at that point, which is different from the original trajectory. This context simulates the assumption of what would have happened if a different choice had been made at that moment.

[0021] Continuation trajectory: refers to a new multi-turn interaction sequence generated when a student agent takes a candidate counterfactual context as a new starting point and continues to execute the subsequent part of the task until the task ends or is interrupted.

[0022] Task completion verification: This is a verification process used to evaluate whether a continuation trajectory has ultimately and correctly completed its original task.

[0023] Substantive validity verification: refers to the verification of the counterfactual anchor itself, aiming to assess whether the specific alternative decision-making actions (thinking and tool usage) introduced at the anchor have made a substantial and necessary contribution to the success of the final task, rather than being merely related or accidental factors.

[0024] Local supervised fine-tuning: In this application, it specifically refers to a machine learning training method in which supervised learning loss is calculated only on the output (inference and action) predicted by the model in a specific round (i.e., the counterfactual anchor round), and the model parameters are updated through gradient backpropagation without affecting the model's ability to predict other rounds in the trajectory.

[0025] Please see Figure 1 , Figure 1 A flowchart of an intelligent agent toolset extension training method based on counterfactual anchor context provided by the present invention is shown in the figure, which includes the following steps: S10. Obtain the failure trajectory generated by the agent when performing tasks under the extended toolset; S20. Analyze the failure trajectory, identify the key decision rounds that affect the subsequent reasoning path, and determine the key decision rounds as counterfactual anchor points; S30. Truncate the failure trajectory to retain only the historical information before the counterfactual anchor point, generate alternative decision actions based on the historical information, and use the tool environment to execute the alternative decision actions to obtain environmental feedback results. Combine the truncated historical information, alternative decision actions and corresponding environmental feedback results to construct a candidate counterfactual context. S40. Control the student agent to continue executing the task based on the candidate counterfactual context to obtain the continuation trajectory, perform task completion verification on the continuation trajectory and substantive validity verification on the counterfactual anchor point, and filter out the target counterfactual context that passes both verifications. S50. Based on the target counterfactual context, perform local supervised fine-tuning on the agent so that the loss function only applies to the alternative decision action, thereby updating the model parameters of the agent; S60. Reinforcement learning is performed on the agent that has undergone local supervised fine-tuning on the extended toolset to further update the model parameters of the agent, thereby obtaining the target agent that has completed the toolset extended training.

[0026] In this embodiment, the above steps work together to form a new method for expanding the training of an agent's toolset. First, the failure trajectories collected in step S10 are the raw materials for identifying problems; step S20, by analyzing these trajectories, accurately locates the counterfactual anchors, avoiding inefficient attempts to correct globally; step S30, by strictly truncating historical and real-world execution, ensures that the generated candidate counterfactual contexts are not contaminated by future information and possess practical possibility, providing a credible starting point for subsequent verification; the task completion verification and substantive validity verification introduced in step S40 constitute a double filter, the former ensuring that training samples can lead to task success, and the latter ensuring that the focus of learning is on truly effective anchor decisions, thereby greatly improving the quality of training data; the local supervised fine-tuning in step S50 is key to efficiently absorbing high-quality samples, it only updates model parameters related to anchor decisions, minimizing interference with existing behaviors; finally, the reinforcement learning in step S60, based on solid local modifications, encourages the agent to more actively explore combinations of new and old tools through trial and error and reward mechanisms, and better integrates local correction behaviors with the overall strategy. These interconnected features collectively address the technical challenges of low adoption rates and poor exploration efficiency of new tools due to agent behavioral inertia during toolset expansion, as well as the degradation of existing capabilities caused by brute-force retraining. This results in high-quality, efficient, and robust incremental training.

[0027] Taking a specific application scenario as an example, suppose an agent that is originally skilled in web search and code interpreting has a new image search tool added to its toolset. When performing the task of finding similar images of a specific item on the expanded toolset, the agent generates a failure trajectory: it only repeatedly uses text search, fails to invoke the new tool, and ultimately fails to find an image that meets the requirements. Analyzing this trajectory, the rounds in which the model saw poor text search results but did not try image search before its last attempt at text search are identified as counterfactual anchor points. The history before this anchor point (including user instructions, obtained search results, etc.) is truncated, and the teacher model generates alternative decisions that include considering using the new tool for image search reasoning and corresponding actions. This is executed through the real image search interface to obtain a set of image links, which are encapsulated as candidate counterfactual contexts. The student agent (i.e., the agent to be trained) starts from this context and successfully continues the trajectory of browsing and filtering images, passing the verification of task correctness and the substantial contribution of the anchor point. Subsequently, local supervised fine-tuning was performed only at this anchor point, considering the learning objective of using the new tool and invoking the image search action, so that the model tends to call the new tool in similar situations. After that, reinforcement learning was performed, and the model continued to train in an environment that mixed new and old tasks. Its attempts to use the new image search tool were rewarded positively, thereby consolidating the learning effect.

[0028] In some implementations, the steps of truncating the failure trajectory to retain only historical information prior to the counterfactual anchor point, generating alternative decision actions based on the historical information, executing the alternative decision actions using a tool environment to obtain environmental feedback results, and combining the truncated historical information, alternative decision actions, and corresponding environmental feedback results to construct a candidate counterfactual context include: S31. Cut off the failure trajectory and retain only the reasoning content, tool call actions and tool return results of all rounds before the round of the counterfactual anchor point; In this step, the sequence number n of the counterfactual anchor point is located from the multi-round interaction sequence of the failure trajectory. Complete records from round 1 to round (n-1) are extracted from the trajectory data structure. Each round record includes: the inference content generated by the model (think), the name of the selected tool and its calling parameters (i.e., the tool calling action), and the tool's returned results (obs). These records are then concatenated in their original chronological order to form a historical information sequence. This sequence does not include content from round n and any subsequent rounds, ensuring that subsequently generated alternative decisions do not access future information.

[0029] As an example, in the multi-round interaction sequence of the failure trajectory (think1, action1, obs1), (think2, action2, obs2), ..., (thinkn, actionn, obsn), if the k-th round is determined to be the counterfactual anchor point, then all triplet information from the 1st to the (k-1)th round is retained to form the historical information sequence H = (think1, action1, obs1), ..., (thinkk... 1, actionk 1, obsk 1) Discard all original data from round k to round n.

[0030] S32. Based on the historical information, generate new reasoning content and new tool invocation actions to replace the original rounds of reasoning and tool invocation; In this step, the historical information sequence, current task instructions, and tool descriptions of the extended toolset are concatenated into an input prompt, which is then input into the teacher model. Based on this input, the teacher model independently generates new thought text (alternative reasoning content) and corresponding new tool invocation actions (alternative tool invocation actions) without accessing the anchor point rounds in the original trajectory or any subsequent information. The teacher model's generation process follows the same thought, action, and observation interaction format as the original agent, ensuring that the generated alternative content is format-compatible with the original trajectory. Specifically, the teacher model first generates a thought text, analyzes the task progress and available tools reflected in the current historical information, and then selects a tool based on the analysis results and constructs its invocation parameters, forming a complete set of alternative decision actions.

[0031] As an example, the historical information sequence H is concatenated with the original task instructions to form the input prompts for the teacher model. The prompt format is: [Historical reasoning and tool call records], [Current round task objective], Please generate the next reasoning content and tool call action based on the existing information. The teacher model then outputs the new reasoning content think'_k and the new tool call action action'_k, where action'_k includes the target tool name and call parameters.

[0032] S33. Call the corresponding tool interface in the extended toolset, execute the new tool call action, and receive the structured observation data returned by the tool as environmental feedback results; In this step, based on the new tool call action generated in step S32, the tool name and call parameters are parsed; the interface address and call protocol of the corresponding tool are found in the extended toolset; the call parameters are formatted into the input format required by the interface (such as a JSON request body) according to the requirements of the tool interface; a call request is sent to the tool environment via API call or function execution; the execution result returned by the tool environment is received, which is structured observation data, including the tool execution status information and return content (such as search results list, code execution output, image matching results, etc.); this structured observation data is used as the environment feedback result for subsequent encapsulation.

[0033] As an example, based on the tool name specified in action'_k, the corresponding external tool service is called through the API gateway or SDK, passing in the parameters carried in action'_k, and receiving the structured observation data obs'_k returned after the tool finishes execution.

[0034] S34. Encapsulate the historical information, new reasoning content, new tool call actions, and environmental feedback results into a new multi-turn interactive data unit as the candidate counterfactual context.

[0035] In this step, the historical information sequence, new inference content, new tool invocation actions, and environmental feedback results are encapsulated in a multi-round interaction sequence format. Specifically, the historical information sequence maintains its original round structure; a new round record is appended to the end of the historical information sequence. This record includes: the "Thinking" field filled with the new inference content generated in S32, the "Action" field filled with the new tool invocation action generated in S32, and the "Observation" field filled with the environmental feedback result obtained in S33. This new round does not contain any subsequent round information; the resulting data unit is the candidate counterfactual context. This context can be directly used as input for the student agent, allowing it to continue generating subsequent interaction content from this anchor round.

[0036] As an example, the historical information sequence H, the new reasoning content think'_k, the new tool call action action'_k, and the structured observation data obs'_k returned by the tool are combined in chronological order into a complete interaction context C' = H∪{(think'_k, action'_k, obs'_k)}, which serves as a candidate counterfactual context for subsequent student agents to continue writing from this anchor point.

[0037] In this embodiment, the core of the candidate counterfactual context construction process lies in information isolation and realistic simulation: only decisions that can be made based on the history before the anchor point are allowed to be modified, and the modified actions are executed in a realistic tool environment to obtain realistic observation results. This mechanism prevents the teacher model from cheating by using future information to generate unrealistic perfect substitutes, ensuring the logical rationality of the constructed context and the realism of the environmental interaction, providing a reliable foundation for subsequent verification and imitation by the student agent. Furthermore, this step can lead to the construction of high-quality training samples with practical feasibility, avoiding the distribution bias caused by directly generating complete and correct trajectories. As a specific implementation method, the subject generating new inferences and tool calls can be a large language model teacher, which is provided with historical information before the anchor point and the task background of the current round, and independently generates substitute content.

[0038] In some implementations, the steps of verifying the task completion rate of the continuation trajectory and verifying the substantive validity of the counterfactual anchor points include: S41. Calculate the matching degree between the final output of the continuation trajectory and the standard answer of the task. If the matching degree exceeds the preset first threshold, it is determined that the task completion degree verification is passed. In this step, the matching degree between the final output of the continuation trajectory and the standard answer of the task can be calculated using a text similarity calculation model. Specifically, the final output result of the continuation trajectory is extracted, which is the final answer generated by the student agent after completing the task based on the candidate counterfactual context; the standard answer or expected output corresponding to the task is obtained; the matching degree calculation method is selected according to the task type: for text generation tasks, a text similarity model (such as BERTScore, ROUGE, etc.) is used to calculate the semantic similarity between the final output and the standard answer; for code execution tasks, the final output is executed as code on test cases, and the pass rate is used as the matching degree; for retrieval tasks, the precision and recall of the final output and the standard answer set are calculated; the calculated matching degree is compared with a preset first threshold (such as 0.8 or 80%). If the matching degree exceeds the preset first threshold, the continuation trajectory is determined to have passed the task completion verification; otherwise, it is determined to have failed.

[0039] As an example, the student agent continues the task from the candidate counterfactual context C' until termination, obtaining the final output result O_final of the continuation trajectory. The matching degree calculation method is selected according to the task type: for tasks with deterministic answers (such as code generation, mathematical calculation, and data query), the match rate between O_final and the standard answer A_std is calculated through program execution or precise string matching; for open-ended tasks (such as text generation and content summarization), a semantic similarity model is used to calculate the semantic similarity score between O_final and A_std, with a value range of [0,1]. If the matching degree score exceeds a preset first threshold, the task completion verification is passed.

[0040] S42. Evaluate the causal contribution of the tool calls or reasoning logic introduced in the counterfactual anchor to the final output. If the contribution exceeds a preset second threshold, the substantive validity check is passed.

[0041] In this step, the causal contribution can be quantified by comparing the success rates of multiple simulation trajectories with and without the modification. Specifically, a control group trajectory is first constructed: Under the same task, based on historical information before the counterfactual anchor point, the alternative decision action generated in step S32 is not introduced. Instead, the original decision action of the anchor point round in the original failed trajectory is retained. The student agent continues the task multiple times from the original decision action to obtain M control group trajectories. Similarly, under the condition of introducing the alternative decision action in step S32 (i.e., using candidate counterfactual context), the student agent continues the task multiple times to obtain N experimental group trajectories. The success rate of the task in the experimental group trajectory, P_exp, and the success rate of the task in the control group trajectory, P_ctrl, are calculated. The causal contribution is calculated as P_exp - P_ctrl (or the ratio P_exp / P_ctrl). The causal contribution is compared with a preset second threshold (e.g., 0.3, i.e., the success rate of the experimental group needs to be 30 percentage points higher than that of the control group). If the contribution exceeds the preset second threshold, the counterfactual anchor point is determined to pass the substantive validity check; otherwise, it is determined to fail. This evaluation method eliminates random factors through controlled experiments, ensuring that the alternative decisions made at the anchor point contribute causally to the success of the task.

[0042] In this embodiment, the introduction of this dual verification mechanism is crucial for selecting high-quality training samples. Task completion verification is fundamental, ensuring that the ultimate goal of the training samples is correct. Substantive validity verification goes a step further, requiring that modifications to anchor points must be causes of success, not merely results occurring simultaneously with success. This avoids mistakenly attributing the task's ultimate success to accidental factors or irrelevant modifications, thereby filtering out truly valuable, high-leverage anchor contexts. Combining these principles, this approach significantly improves the effectiveness and reliability of training data, ensuring that subsequent local supervised fine-tuning truly focuses on behavioral changes crucial to task success.

[0043] In some implementations, the step of performing local supervised fine-tuning of the agent based on the target counterfactual context, so that the loss function applies only to the alternative decision action, includes the following steps: S51. Input the target counterfactual context into the agent for forward reasoning, and calculate the cross-entropy loss between the reasoning content predicted by the model and the tool invocation action and the alternative decision action marked in the target counterfactual context. In this step, the target counterfactual context is input as an input sequence to the agent model for forward propagation computation. Specifically, the historical information sequence in the target counterfactual context serves as the context input, and the alternative decision action portion serves as the target label. The model generates a probability distribution for the next token at the corresponding output position in each round. For the token sequence positions corresponding to the alternative decision action (i.e., the token sequence of the thought text and tool call action in the anchor round), the cross-entropy loss between the model's predicted distribution and the actual token sequence of the alternative decision action labeled in the target counterfactual context is calculated. The loss calculation is limited to the token positions corresponding to the alternative decision action; token positions in other rounds of the historical information sequence are not included in the loss calculation.

[0044] As an example, the target counterfactual context is input into the agent model to be trained in a multi-turn interaction format. The model then performs forward reasoning on the counterfactual anchor points and outputs the probability distribution P of the predicted reasoning content. think = Model (H) and the predicted probability distribution of tool call actions P action = Model(H). Let the alternative reasoning content labeled in the target counterfactual context be think'_k, and the alternative tool invocation action be action'_k. Calculate the cross-entropy loss for each, where the cross-entropy loss for the inference branch is: The cross-entropy loss of the tool invocation action branch is: The total anchor point loss is: , among which, T thinkThe total length of the tokens in the substitution reasoning sequence think'_k; T action The total length of tokens in the alternative tool call sequence action'_k; α and β are weighting coefficients (α=β=1.0 is recommended) used to balance the importance of inference learning and action learning; conditions This represents the entire sequence of tokens generated before this position, i.e., the autoregressive historical context.

[0045] S52. During backpropagation, a masking mechanism is used to limit the gradient of the loss function to update the output or hidden layer parameters of the alternative decision action in the model.

[0046] In this step, before backpropagation, a gradient mask matrix is ​​constructed, which corresponds one-to-one with the token positions in the model's output layer. In the mask matrix, the token positions corresponding to alternative decision actions are set to 1 (allowing gradients to pass through), while all other token positions (including token positions from each round in the historical information sequence) are set to 0 (blocking gradient propagation). During backpropagation, a masking mechanism is used to restrict the gradient of the loss function to backpropagation only from the token position corresponding to the alternative decision action, while the gradient contribution of other token positions is set to zero. Simultaneously, to reduce drastic perturbations to the shared backbone network parameters, the local supervised fine-tuning preferably employs parameter-efficient fine-tuning techniques such as Low-Rank Adaptation (LoRA), updating only the low-rank matrix parameters attached to the backbone network, while keeping the parameters of the original pre-trained backbone network frozen or at a low learning rate during fine-tuning. If full-parameter fine-tuning is used, the learning rate needs to be reduced to less than 1 / 10 of the conventional fine-tuning learning rate to suppress excessive modification of the shared weight matrix. Through this combined strategy, while achieving learning updates for anchor point decisions, the model's ability to generate behaviors for other rounds in the historical trajectory is protected to the greatest extent. This masking mechanism ensures that model parameter updates only affect the network parameters related to the alternative decision action, without affecting the model's ability to generate behaviors for other rounds in the historical trajectory, thus maximizing the protection of existing behavior patterns while updating new behaviors.

[0047] In this embodiment, this approach enables local and targeted updates to model parameters. By constraining gradient calculations through a masking mechanism, it ensures that the backpropagation error signal only affects the neural network parameters related to the decision generated at that specific counterfactual anchor point, without affecting the parameters responsible for generating other parts of the trajectory history. Based on the above principles, the beneficial effects of this approach are: it not only makes the learning objective highly focused (learning the optimal alternative decision), but also maximizes the protection of the model's existing, unmodified behavioral patterns (corresponding to the original tool capabilities) at the update mechanism level. This effectively mitigates the risk of catastrophic forgetting in traditional supervised fine-tuning, providing a solid starting point for the subsequent reinforcement learning stage without disrupting the original knowledge structure.

[0048] In some implementations, the reinforcement learning of the agent, which has undergone local supervised fine-tuning, on the extended toolset includes the steps of: S61. Employ a near-end strategy optimization algorithm to sample new complete task trajectories on the extended toolset; In this step, the Near-End Policy Optimization (PPO) algorithm is applied to the agent that has undergone local supervised fine-tuning. Specifically, using the current agent policy model as the behavioral policy, multiple complete task execution trajectories are sampled on the task distribution defined by the extended toolset. The sampling process for each trajectory is as follows: the agent starts from the initial state of the task, generates a thought text and a tool invocation action based on the current policy distribution, executes it in the tool environment and obtains observation results, and repeats this process until the task is completed or the maximum number of rounds is reached. Simultaneously, an independent commentator model is used to estimate the value function of each state to calculate the advantage function, thereby reducing the variance of the policy gradient.

[0049] As an example, the agent, after local supervision and fine-tuning, is used as the initial policy π_θ for reinforcement learning. The agent then interacts with the environment multiple times using an expanded toolset to sample the complete task trajectory. , where s t (t = 1, 2, ..., T) represents the state in round t (including historical reasoning and tool observation), a t (t = 1, 2, ..., T) represents the action in round t (including reasoning content and tool calls), r t (t = 1, 2, ..., T) represents the instantaneous reward obtained in round t, T is the total number of rounds in the trajectory, and s Ｔ＋１ This is the final state (the state when the task is completed or the maximum number of rounds is reached).

[0050] Policy updates are performed using the PPO-Clip objective function: , in, This represents the empirical expectation for all time steps t in the sampling trajectory. In actual calculation, the arithmetic mean of the loss values ​​of each time step within the batch is taken. The probability ratio between the old and new strategies. For the policy network that needs to be updated, This is the old policy network used when sampling this trajectory; At is the advantage function estimate (calculated using the GAE method), and clip() is the clipping function used to limit the input values ​​to a specified interval, clip(ρt(θ), 1 ε, 1+ε) denotes cropping the probability ratio ρt(θ) of the new and old strategies to the interval [1]. Within ε, 1+ε]; ε is the pruning hyperparameter (ε=0.2 is recommended), and it is used in conjunction with the value network Vφ(s) t State value estimation is performed, and the value network parameters φ are updated using mean squared error loss; multiple environments are sampled in parallel during training to improve data efficiency.

[0051] S62. Design a reward function to give positive rewards for successfully calling the new tool or completing the task, and negative rewards for incorrectly calling the original tool or failing the task. In this step, a multi-dimensional reward function can be designed, including: (1) New tool adoption reward: When the agent successfully calls a new tool in the expanded toolset in a certain round and the calling parameters are correct, a positive reward r_new is given; (2) Task completion reward: When the entire trajectory is finally completed correctly, a large positive reward r_success is given; (3) Existing tool misuse penalty: When the agent calls an existing tool incorrectly (such as calling the wrong parameters or repeatedly calling a tool that has already returned sufficient results), a negative reward r_old_penalty is given; (4) Task failure penalty: When the task is not completed at the end of the trajectory, a negative reward r_fail is given. The total reward R is a weighted combination of the items. This reward function guides the agent to actively try and explore new tools while maintaining the correct use of existing tools.

[0052] S63. During reinforcement learning training, a certain proportion of trajectory data that has been successfully executed only on the original toolset is mixed in to maintain the ability to use the original tools. In each batch of reinforcement learning training in this step, in addition to new trajectories sampled from the expanded toolset, a certain number of trajectory data that were successfully executed only on the original toolset are sampled from a pre-built experience replay pool at a preset ratio (e.g., new trajectory: old trajectory = 7:3). The original toolset trajectory data serves as an auxiliary training signal, and its advantage function and reward are calculated according to the standards of the original toolset task. The gradients of the new and old trajectories are merged in the same optimization step to update the model parameters. This hybrid strategy, by periodically reinforcing the ability to use the original tools, combats the degradation of existing capabilities (catastrophic forgetting) caused by overexploration of new tools, ensuring that the trained agent possesses the ability to call upon both new and old tools.

[0053] In this embodiment, reinforcement learning serves to further consolidate and generalize the new behavioral patterns learned from local supervised fine-tuning (i.e., efficient use of new tools), and encourages the agent to explore better combinations of old and new tools. Mixing and replaying successful trajectory data from old tools acts as a proactive behavioral constraint, periodically reinforcing the model's mastery of existing skills and counteracting the risk of skill degradation due to over-exploration of new tools. Based on these principles, the unique benefit of this approach is that, through strategic data mixing design, it encourages exploration of new toolsets while explicitly maintaining the memory of existing toolset operational capabilities. This ensures that the target agent obtained after incremental training possesses balanced and comprehensive tool-calling capabilities, rather than sacrificing old capabilities for new ones.

[0054] In some implementations, analyzing the failure trajectory to identify key decision rounds affecting subsequent reasoning paths and determining these key decision rounds as counterfactual anchor points includes the following steps: S21. Evaluate the redirection potential and strategic leverage of each round in the failure trajectory for subsequent inference paths; In this step, for each round i (i=1, 2, ..., N, where N is the total number of rounds in the trajectory) in the failed trajectory, at least one of the following metrics is calculated as a strategy leverage evaluation: (a) Downstream impact score, which is based on an estimate of the probability that subsequent paths will ultimately lead to task success if the decision in this round is changed; specifically, in a simplified simulation environment, it is assumed that the decision in the i-th round is replaced by K different alternative decisions (K≥3 and K≤10) generated by a lightweight teacher model or a small-scale student model (with fewer than 1 / 10 of the number of parameters of the agent to be trained); for each alternative decision, the results are executed and observed in the tool environment, and then the lightweight model continues to execute the task for M steps from the alternative context (M is the preset number of short-sighted steps, M≤5), and the proportion of the K short continuation paths that can ultimately lead to task success is counted as the downstream impact score for this round; the lightweight model is only used for rapid estimation in the leverage evaluation stage and does not participate in the subsequent training process; (b) The path redirection probability is based on the degree of deviation of all possible subsequent paths after this round from the original failed path. Specifically, for the i-th round in the failed trajectory, the historical information from rounds 1 to i-1 is retained as a fixed prefix. At the i-th round, K different alternative decision actions (K≥10) are generated using a student agent or a teacher model. For each alternative decision action, it is executed in the tool environment and the observation results are obtained. Then, the student agent continues to write the task from the alternative context until the end, resulting in K subsequent paths. The behavioral sequence difference (e.g., the edit distance of the tool call sequence) between each subsequent path and the original failed path (the subsequent path starting from the original decision in round i) is calculated. If the difference exceeds a preset deviation threshold, the path is considered to have been redirected. The proportion of paths that have been redirected is counted as the path redirection probability. (c) Heterogeneous judgment model scoring: Select a model that is different from the trainable agent in terms of model architecture (e.g., the number of Transformer layers) and parameter scale (e.g., 7B vs. A heterogeneous model is used to determine whether there are significant differences in the distribution of training data (e.g., 70B). The model takes the historical context of round i (including complete interaction records from the previous i-1 rounds), the original decision action of round i, and the task objective description as input. It constructs prompts to ask the model to evaluate the probability of task success if an alternative decision is made in this round, different from the original decision. The model outputs a probability value between 0 and 1 as the heterogeneous model score for that round. Using heterogeneous models can reduce the self-evaluation bias of homogeneous models and provide a more objective anchor value judgment.

[0055] S22. The rounds that meet the preset high leverage conditions are determined as the counterfactual anchor points.

[0056] In this step, high leverage conditions are set, including: the downstream impact score exceeding a preset first leverage threshold (e.g., 0.2, meaning at least 20% of alternative subsequent paths are successful), or the path relocation probability exceeding a preset second leverage threshold (e.g., 0.5, meaning at least 50% of subsequent paths deviate from the original failed path), or the heterogeneous judgment model score exceeding a preset third leverage threshold (e.g., 0.6). When any of the above indicators in a certain round meets the corresponding leverage threshold condition, that round is determined as the counterfactual anchor point. If multiple rounds meet the conditions simultaneously, the round with the highest score or the most met indicators can be selected as the priority anchor point, or multiple rounds can be treated as independent counterfactual anchor points for subsequent processing.

[0057] In this embodiment, the process differs from a fixed rule that simply selects the first or last failed step. Instead, it performs a potential assessment on any given round. This approach can identify rounds that appear to have no direct errors but are actually locked into a low probability of success due to conservatism (failure to try new tools) or path dependence (repeating old patterns), thus finding correction points with higher training value. Based on the above principles, the unique benefit of this method is that it makes the anchor point discovery process intelligent and in-depth, enabling the identification of high-value rounds with greater influence and the ability to drive the overall task towards success. This fundamentally improves the quality ceiling and teaching efficiency of the constructed training samples.

[0058] This embodiment provides multiple perspectives for quantifying the leverage of strategic measures. Downstream impact score indicates the potential impact of a decision on the target outcome; path redirection probability measures the ability of a decision to alter the entire inference flow in multiple directions; and heterogeneous model scoring provides external judgment, reducing the influence of agent bias. Combining these principles and employing some or all of these calculations can lead to a more comprehensive and robust anchor evaluation, avoiding misjudgments that may result from a single indicator and ensuring that the identified anchors have a higher overall leverage value.

[0059] In some implementations, step S20, identifying key decision rounds, specifically includes: Multiple heterogeneous teacher models are invoked to independently analyze the same failure trajectory; each heterogeneous teacher model generates multiple candidate counterfactual contexts with different anchor point rounds or different alternative decision actions.

[0060] This embodiment analyzes the same failure trajectory using multiple heterogeneous teacher models (with different architectures, scales, or domain expertise), enabling the discovery of potential valuable correction points from different perspectives and generating diverse candidate anchors and alternatives. Essentially, this approach involves a more thorough exploration and sampling of the solution space of the failure trajectory, increasing the likelihood of discovering high-quality training samples. Based on the above principles, the beneficial effects of this method are: effectively enriching the set of candidate contexts that can be used for subsequent validation and training; improving the probability and efficiency of discovering and generating high-quality training samples; and providing the agent with more diverse positive learning examples.

[0061] In some implementations, in step S30, to fully explore the solution space of failure trajectories, multiple heterogeneous teacher models with different architectures, parameter scales, or domain expertise can be deployed, allowing each teacher model to independently identify anchor points and generate alternative decisions for the same failure trajectory. Due to differences in inductive bias and capabilities, different teacher models may identify different key decision rounds or generate drastically different alternative reasoning logic and tool invocation actions for the same anchor point. By aggregating the candidate counterfactual contexts produced by multiple heterogeneous teacher models, the diversity of the candidate set can be significantly improved, increasing the probability of discovering high-value training samples. This amplification mechanism works synergistically with the subsequent dual-validation screening: first, the heterogeneous teacher models fully explore the possibility space, and then the student agent continues the process and performs dual validation to ensure quality control, ensuring that the target counterfactual contexts entering the training phase are both diverse and reliable.

[0062] In some implementations, after step S30 and before step S40, a further step is included: filtering multiple candidate counterfactual contexts generated by these heterogeneous teacher models, removing duplicate and redundant contexts, and prioritizing contexts with high diversity of alternative decision actions or high previous validation pass rates for subsequent steps. In this embodiment, deduplication and filtering of the large number of generated candidate contexts is a key step in improving the overall process efficiency. Prioritizing decision actions with high diversity helps the model learn more generalized behavioral patterns; prioritizing contexts from sources with high validation pass rates utilizes historical feedback information to optimize the filtering strategy, which is an adaptive optimization mechanism. Combining the above principles, the beneficial effect of this approach is that, while ensuring sample diversity, it further focuses computational resources on high-quality candidate contexts that are more likely to pass subsequent double validation and are therefore more worthwhile to invest in training costs, thus improving the efficiency and output quality of the overall training process.

[0063] In some implementations, the substantive validity verification in step S40 specifically involves: evaluating the causal contribution of the tool call or reasoning logic introduced by the counterfactual anchor to the final output of the continuation trajectory.

[0064] This embodiment explicitly defines the specific standard for verifying substantive validity as the assessment of causal contribution, pointing out that its core purpose is to distinguish between causal relationships and correlational relationships, and emphasizing that the modification of the anchor point must generate a real and necessary driving force for the final success, rather than merely a temporal accompanying event.

[0065] In this embodiment, the steps for evaluating causal contribution specifically include: constructing a control group trajectory without introducing the alternative decision action under the same task; comparing the success rate of the continuation trajectory generated by the candidate counterfactual context with the success rate of the control group trajectory, and calculating the difference between the two as the causal contribution. A specific and feasible method for measuring causal contribution is provided here, which adopts the idea of ​​a controlled experiment. By constructing a control group (e.g., keeping the anchor point unchanged, or randomly generating an alternative action), the success rate difference of the experimental group (using the selected alternative action) is compared. This difference can be attributed to the specific alternative decision action. This statistically based evaluation method is more objective and persuasive. Combining the above principles, the beneficial effect of this method is that it provides a quantifiable and reproducible objective standard for verifying substantive validity, reducing the interference that subjective judgment or model bias may cause to sample selection, thereby further improving the reliability and educational value of the selected target counterfactual context. As a specific implementation method, the control group trajectory can be simulated by sampling agents randomly continuing the trajectory multiple times based on the original failed trajectory anchor point and previous history.

[0066] In some implementations, the expanded toolset in this training method may include at least one new external tool such as web search, web browsing, code interpreter, file parser, image description, bounding box extraction, and image search. The error types involved in the counterfactual anchor may include at least one of the following: errors in the use of new tools (e.g., not invoked, incorrect selection, incorrect parameters), errors in the use of existing tools (e.g., repeated invocation, insufficient utilization), and errors in reasoning decisions (e.g., incorrect direction, insufficient integration). This clarifies the broad application scenarios of this method and the specific problem types it can correct.

[0067] In some implementations, the failure trajectory, candidate counterfactual context, and continuation trajectory are all recorded using a multi-round interaction sequence. Each round sequence includes the thinking content generated by the model, the tool invocation action, and the corresponding tool return observation results. This defines the basic structure and format of the data processed by this method.

[0068] It should be noted that, in some optional implementations, the core steps described above can be implemented through programming within a model training system, which can be deployed on a cloud server or a local computing cluster. The teacher model, judgment model, etc., used to generate alternative decision actions or perform verification judgments can be selected from large language models with different parameter scales (e.g., 7B, 13B, 70B parameters), or multimodal models with specific modal capabilities. As a specific implementation, the result of anchor-level substantive validity verification (causal contribution) can be compared with a preset second threshold (e.g., a success rate difference exceeding 30%) to determine whether it passes or fails. During the reinforcement learning phase, the reward function design, in addition to the rules mentioned above, can also include rewards for improved inference efficiency and reduced tool call costs. To construct an effective control group trajectory, it may be necessary to sample the agent's continuation behavior under the original history multiple times and perform success rate statistics. The constructed candidate counterfactual contexts and the verified target counterfactual contexts can be stored to form a behavior repair library for easy reuse or analysis later. All trajectory data (failed trajectories and continuation trajectories) strictly follow the multi-round interactive sequence format of thinking, action, and observation, which facilitates programmatic processing and analysis.

[0069] It should also be noted that in some alternative implementations, the teacher model analysis and continuation trajectory generation can use the same language model instance (i.e., self-supervised) or two independent models (i.e., the student model is the agent to be trained). The task completion verification's standard answer matching degree calculation can be achieved through various methods such as program verification, matching with predefined answer templates, or human evaluation. Besides Proximal Policy Optimization (PPO), other policy optimization algorithms, such as TRPO, A2C, and offline reinforcement learning, can also be used in the reinforcement learning phase. Local supervised fine-tuning, in addition to standard cross-entropy loss and masking techniques, can also combine preference optimization (such as DPO) and efficient parameter fine-tuning (such as LoRA). Toolset expansion is not limited to adding specific API tools; it also applies to tool interface version upgrades and changes in tool permission scope. The high-quality context constructed by this method can also be used as training data for other version iterations or transfer learning scenarios of the agent.

[0070] The above describes the agent toolset extension training method based on counterfactual anchor context in the embodiments of the present invention. The following describes the agent toolset extension training device based on counterfactual anchor context in the embodiments of the present invention. Please refer to [link to relevant documentation]. Figure 2 One embodiment of the intelligent agent toolset extension training device based on counterfactual anchor context in this invention includes: The trajectory acquisition module 10 is used to acquire the failed trajectories generated by the agent when performing tasks under the extended toolset; Anchor point identification module 20 is used to analyze the failure trajectory, identify the key decision rounds that affect the subsequent reasoning path, and determine the key decision rounds as counterfactual anchor points; The context construction module 30 is used to truncate the failure trajectory to retain only the historical information before the counterfactual anchor point, generate alternative decision actions based on the historical information, and use the tool environment to execute the alternative decision actions to obtain environmental feedback results. The truncated historical information, alternative decision actions and corresponding environmental feedback results are combined to construct a candidate counterfactual context. The verification and filtering module 40 is used to control the student agent to continue to execute the task based on the candidate counterfactual context to obtain the continuation trajectory, to verify the task completion degree of the continuation trajectory and to verify the substantive validity of the counterfactual anchor point, and to filter out the target counterfactual context that passes both verifications. The local fine-tuning module 50 is used to perform local supervised fine-tuning of the agent based on the target counterfactual context, so that the loss function only applies to the alternative decision action, so as to update the model parameters of the agent; The reinforcement learning module 60 is used to perform reinforcement learning on the agent that has undergone local supervised fine-tuning on the extended toolset, so as to further update the model parameters of the agent and obtain the target agent that has completed the toolset extended training.

[0071] Based on the same ideas as the methods in the above embodiments, the device provided by the present invention can implement the methods in the above embodiments. For ease of explanation, the structural schematic diagram of the device embodiment only shows the parts related to the embodiments of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer modules than illustrated, or combine certain modules, or have different module arrangements.

[0072] Figure 2 The intelligent agent toolset extension training device based on counterfactual anchor context in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The intelligent agent toolset extension training device based on counterfactual anchor context in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0073] Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 100 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 111 (e.g., one or more processors) and a memory 121, and one or more storage media 130 (e.g., one or more mass storage devices) for storing application programs 133 or data 132. The memory 121 and storage media 130 may be temporary or persistent storage. The program stored in the storage media 130 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the electronic device 100. Furthermore, the processor 111 may be configured to communicate with the storage media 130 and execute the series of instruction operations in the storage media 130 on the electronic device 100.

[0074] Electronic device 100 may also include one or more power supplies 141, one or more wired or wireless network interfaces 151, one or more input / output interfaces 161, and / or one or more operating systems 131, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The device structure shown does not constitute a limitation on the electronic device 100, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0075] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the intelligent agent toolset extension training method based on counterfactual anchor context.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0077] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for expanding the training of an agent's toolset based on counterfactual anchor context, characterized in that, Including the following steps: Obtain the failure trajectory generated by the agent when performing tasks under the extended toolset; The failure trajectory is analyzed to identify the key decision rounds that affect the subsequent reasoning path, and these key decision rounds are determined as counterfactual anchor points. The failure trajectory is truncated to retain only the historical information before the counterfactual anchor point. An alternative decision action is generated based on the historical information, and the alternative decision action is executed using the tool environment to obtain environmental feedback results. The truncated historical information, alternative decision actions and corresponding environmental feedback results are combined to construct a candidate counterfactual context. The student agent is controlled to continue executing the task based on the candidate counterfactual context to obtain the continuation trajectory. The task completion degree of the continuation trajectory and the substantive validity of the counterfactual anchor point are verified, and the target counterfactual context that passes both verifications is selected. Based on the target counterfactual context, the agent is locally supervised and fine-tuned so that the loss function only applies to the alternative decision action, thereby updating the model parameters of the agent; Reinforcement learning is performed on the agent that has undergone local supervised fine-tuning on the extended toolset to further update the model parameters of the agent, thereby obtaining the target agent that has completed the extended toolset training.

2. The method for expanding the training of an agent toolset based on counterfactual anchor context according to claim 1, characterized in that, The process of truncating the failure trajectory to retain only historical information prior to the counterfactual anchor point, generating alternative decision actions based on the historical information, executing the alternative decision actions using the tool environment to obtain environmental feedback results, and combining the truncated historical information, alternative decision actions, and corresponding environmental feedback results to construct a candidate counterfactual context includes the following steps: The failure trajectory is truncated, and only the reasoning content, tool call actions, and tool return results of all rounds preceding the counterfactual anchor are retained; Based on the historical information, new reasoning content and new tool invocation actions are generated to replace the original rounds of reasoning and tool invocation. Call the corresponding tool interface in the extended toolset, execute the new tool call action, and receive the structured observation data returned by the tool as environmental feedback results; The historical information, new reasoning content, new tool invocation actions, and environmental feedback results are encapsulated into new multi-turn interactive data units, which serve as the candidate counterfactual context.

3. The method for expanding the training of an agent toolset based on counterfactual anchor context according to claim 1, characterized in that, The steps of verifying the task completion rate of the continuation trajectory and verifying the substantive validity of the counterfactual anchor points include: Calculate the matching degree between the final output of the continuation trajectory and the standard answer of the task. If the matching degree exceeds the preset first threshold, the task completion verification is passed. The causal contribution of the tool calls or reasoning logic introduced in the counterfactual anchor to the final output is evaluated. If the contribution exceeds a preset second threshold, the substantive validity check is passed.

4. The method for expanding the training of an agent toolset based on counterfactual anchor context according to any one of claims 1-3, characterized in that, The step of performing local supervised fine-tuning of the agent based on the target counterfactual context, so that the loss function only applies to the alternative decision action, includes the following steps: The target counterfactual context is input into the agent for forward reasoning, and the cross-entropy loss between the reasoning content predicted by the model and the tool invocation action and the alternative decision action marked in the target counterfactual context is calculated. During backpropagation, a masking mechanism is used to limit the gradient of the loss function to update the output or hidden layer parameters of the alternative decision action in the model.

5. The method for expanding the training of an agent toolset based on counterfactual anchor context according to any one of claims 1-3, characterized in that, The reinforcement learning of the agent that has undergone local supervised fine-tuning on the extended toolset includes the following steps: A near-end strategy optimization algorithm is used to sample new complete task trajectories on the extended toolset; Design a reward function to give positive rewards for successfully calling the new tool and completing the task, and negative rewards for incorrectly calling the original tool or failing the task. During reinforcement learning training, a certain proportion of trajectory data that has been successfully executed only on the original toolset is mixed in in order to maintain the ability to use the original tools.

6. The method for expanding the training of an agent toolset based on counterfactual anchor context according to claim 1, characterized in that, The analysis of the failure trajectory, identification of key decision rounds affecting subsequent reasoning paths, and determination of these key decision rounds as counterfactual anchor points include the following steps: Evaluate the redirection potential and strategic leverage of each round in the failure trajectory for subsequent inference paths; The rounds that meet the preset high leverage conditions are determined as the counterfactual anchor points.

7. The method for expanding the training of an agent toolset based on counterfactual anchor context according to claim 6, characterized in that, The assessment of the redirection potential and policy leverage of each round in the failure trajectory for subsequent inference paths includes performing at least one of the following calculations for each round in the failure trajectory: Calculate a downstream impact score, which is based on an estimate of the probability that subsequent paths will ultimately lead to task success if the decision in this round is changed; Calculate a path redirection probability based on the degree of deviation of all possible subsequent paths after the round from the original failed path; Using a judgment model heterogeneous with the agent, the success potential of the round under counterfactual conditions is directly scored.

8. A training device for extending an agent toolset based on counterfactual anchor context, characterized in that, include: The trajectory acquisition module is used to acquire failed trajectories generated by the agent when performing tasks under the extended toolset; Anchor point identification module is used to analyze the failure trajectory, identify the key decision rounds that affect the subsequent reasoning path, and determine the key decision rounds as counterfactual anchor points; The context building module is used to truncate the failure trajectory to retain only the historical information before the counterfactual anchor point, generate alternative decision actions based on the historical information, and execute the alternative decision actions using the tool environment to obtain environmental feedback results. The truncated historical information, alternative decision actions and corresponding environmental feedback results are combined to construct a candidate counterfactual context. The verification and filtering module is used to control the student agent to continue to execute the task based on the candidate counterfactual context to obtain the continuation trajectory, to verify the task completion degree of the continuation trajectory and to verify the substantive validity of the counterfactual anchor point, and to filter out the target counterfactual context that passes both verifications. The local fine-tuning module is used to perform local supervised fine-tuning of the agent based on the target counterfactual context, so that the loss function only applies to the alternative decision action, so as to update the model parameters of the agent; The reinforcement learning module is used to perform reinforcement learning on the agent that has undergone local supervised fine-tuning on the extended toolset, so as to further update the model parameters of the agent and obtain the target agent that has completed the extended training of the toolset.

9. An electronic device, characterized in that, It includes a memory and at least one processor, wherein the memory stores computer-readable instructions; The at least one processor invokes the computer-readable instructions in the memory to perform the steps of the agent toolset extension training method based on counterfactual anchor context as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by the processor, they implement the steps of the agent toolset extension training method based on counterfactual anchor context as described in any one of claims 1-7.

Images