A tool selection optimization system and method based on error perception reward

Abstract

This invention relates to a tool selection optimization system and method based on error-aware rewards, belonging to the field of artificial intelligence and automated repair. It includes a context representation modeling module, a tool selection decision module, an execution feedback parsing module, and a strategy optimization and update module. The context representation modeling module is used to uniformly encode user queries, candidate tool information, and available historical call features to construct a context state representation. The tool selection decision module is used to score, rank, or select the actions corresponding to candidate tools based on the context state representation. The execution feedback parsing module is used to receive the return results after the selected tool is executed and identify success information, error types, and continuous utility information. The strategy optimization and update module is used to generate reward signals based on execution feedback and update the tool selection strategy. This invention improves the accuracy, error avoidance capability, robustness, and environmental adaptability of tool selection in complex tool call scenarios.

Description

Technical Field

[0001] This invention relates to a tool selection optimization system and method based on error-aware reward, belonging to the field of artificial intelligence and automated repair technology. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), model-driven tool use has gradually become an important paradigm for solving complex tasks. In this paradigm, given a user query, the most suitable tool is selected from a set of candidate tools (such as APIs, functions, or external services) to perform operations such as information retrieval, data processing, or task execution. Therefore, achieving high-precision and generalizable tool selection has become one of the key issues in current intelligent systems.

[0003] The tool selection problem is inherently complex and uncertain, mainly in the following aspects: ① Semantic heterogeneity of tools: different tools differ significantly in their functional descriptions, parameter structures, and return result formats, making it difficult to model them using unified rules; ② Diversity of task contexts: the applicability of the same tool varies significantly in different task scenarios, and relying on static semantic matching is insufficient to accurately characterize its true effect; ③ The actual execution process of tools is dynamic and uncertain: the behavior, stability, and availability of tools fluctuate with changes in the environment, and problems such as parameter errors, timeouts, call failures, and semantic mismatches may occur, thus affecting the selection and invocation effect of tools.

[0004] Existing technologies typically model tool selection as a static relevance ranking problem. For example, supervised learning-to-rank methods rely primarily on offline labeled data to learn the matching relationship between queries and tools. While these methods can improve the semantic matching ability between queries and tools to some extent, they still have the following shortcomings: First, these methods usually only perform a one-time ranking based on static textual information such as tool descriptions and parameter signatures, lacking the ability to model the actual execution effect of tools. Second, existing methods often use coarse-grained success / failure signals as optimization targets, making it difficult to distinguish between different error types such as parameter errors, timeouts, call failures, and semantic mismatches, resulting in insufficient feedback signal information and an unstable strategy learning process. Third, existing methods usually lack online feedback utilization mechanisms, making it difficult to continuously adjust the tool selection strategy based on tool feedback signals during actual interaction, thus making it difficult to adapt to complex and changing task environments and tool operating states. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a tool selection optimization system and method based on error-aware rewards. This method integrates contextual information, static features of candidate tools, historical call performance, and actual execution feedback to optimize tool selection. It elevates the tool selection process from traditional static semantic matching or process-oriented scheduling to a dynamic decision-making optimization process driven by execution feedback. This method should be able to update the tool selection strategy after each tool execution based on the specific error type and actual utility feedback. This allows the model to gradually learn the correlation between different contextual states, candidate tools, and error patterns, thereby reducing the probability of confusion between similar tools, misselection of invalid tools, and repeated invocation of high-risk tools. This improves the accuracy, error avoidance, robustness, and environmental adaptability of tool selection in complex tool call scenarios.

[0006] The present invention adopts the following technical solution:

[0007] A tool selection optimization system based on error-aware reward includes a context representation modeling module, a tool selection decision module, an execution feedback parsing module, and a strategy optimization and update module;

[0008] The context representation modeling module is used to uniformly encode user queries, candidate tool information, and optional historical call features to construct a context state representation for subsequent decision-making.

[0009] The tool selection decision module is used to score, rank, or select the actions corresponding to candidate tools based on contextual state representations.

[0010] The execution feedback parsing module is used to receive the return results after the selected tool is executed, and to identify success information, error type, and continuous utility information;

[0011] The strategy optimization and update module is used to generate reward signals based on execution feedback and update the tool selection strategy.

[0012] A tool selection optimization method based on error-aware reward, implemented through the aforementioned tool selection optimization system based on error-aware reward, includes the following steps:

[0013] S1, given a user query and candidate toolkit ,in This indicates the number of candidate tools corresponding to the current user's query. Indicates the first One candidate tool, using the context representation modeling module to analyze user queries. Encode the candidate tool information to construct a context state representation. , , The context state represents the dimension of the vector;

[0014] S2, in the tool selection decision stage, is determined by the set of candidate tools. Inducing action space Each candidate tool Corresponds to an optional action ,action This indicates that the first option is selected in the current context. Candidate tools Make the call;

[0015] Through policy function Computing the context state representation Select action The probability or preference of selection, and output the optimal tool, Top- The tool or tool sorting results, where Top- This indicates selecting the highest-scoring candidate from the set of candidate tools. A tool, ;

[0016] S3: After the selected tool is executed in the external environment, the corresponding feedback signals are obtained. The feedback signals include, but are not limited to, whether the tool has been executed successfully, whether there are parameter errors, call failures, timeouts, semantic mismatches, empty results or inefficient results, and other error types, as well as continuous utility indicators such as task completion, return result quality, response latency, and execution cost.

[0017] S4. In the policy optimization phase, the system updates the policy function based on feedback signals, which strengthens high-quality decision-making behavior and suppresses low-quality or error-prone behavior.

[0018] This invention realizes the transformation from tool ranking based on static semantic matching to dynamic decision optimization driven by execution results. The entire architecture runs under a unified input-output path and achieves continuous learning and performance improvement through an execution feedback loop.

[0019] Preferably, in step S1, firstly, each candidate tool information includes a tool description. Parameter signature and optional history call features The user query and candidate tool information are encoded separately, as follows:

[0020]

[0021] in, and Encoding functions for user queries and candidate tool information (e.g., implemented based on Transformer or pre-trained language models) respectively map queries and tools to a unified feature space. ; Indicates user query Vectorized representation of; Indicates the first Candidate tools Vectorized representation of;

[0022] Subsequently, the vectorized representation of the user query is combined with the vectorized representation of the candidate tools to construct a context state representation. The format is as follows:

[0023]

[0024] in, The context fusion function can be implemented using vector concatenation, multilayer perceptron, attention mechanism, Transformer interactive coding, weighted pooling, or a combination thereof, and is used to model the interaction relationship between the query and candidate tools.

[0025] After the above processing, the original query and toolset are transformed into a unified context state representation, providing standardized input for subsequent Bandit-based decision-making mechanisms.

[0026] Preferably, in step S2, the action It can be represented using one-hot vectors, index identifiers, or other discrete encoding methods.

[0027] Define the policy function Used to characterize the selection of actions under given context conditions. The probability distribution of actions Corresponding candidate tools ;

[0028] For each candidate tool, the selection probability is expressed as:

[0029]

[0030] in, Represents the context state. With action Corresponding candidate tools The matching score function between them; This represents the learnable parameters in the policy function or matching score function; Represented by natural constant An exponential function with base 0 is used to map the match score to a non-negative value;

[0031] In the decision-making process, the action space is based on the policy function. In this scenario, one or more actions are selected, and the corresponding candidate tools are determined accordingly. For a single-tool selection scenario, the decision-making form is as follows:

[0032]

[0033] in, Indicates the state in the current context. The selected optimal tool selection action; let The corresponding candidate tools are ,but The tool ultimately selected and invoked by the system;

[0034] For scenarios with multiple candidate tools, output the Top-ranked tools in descending order of selection probability or matching score. The sorting result can be represented as follows:

[0035]

[0036] in, Indicates the current context state. Down to action space The sorting results.

[0037] Preferably, the matching scoring function The parameterized model, obtained through the parameterization module, is used to receive context state representations. With action Corresponding candidate tools The vectorized representation is taken as input and the corresponding matching score is output; the parameterization module is implemented using a neural network structure, including but not limited to multilayer perceptron, attention-based encoding network or a combination thereof.

[0038] Preferably, in step S3, the feedback signals contain multiple anomaly patterns with different semantics and consequences, constructing a tool feedback signal type set. :

[0039]

[0040] in, This indicates that the tool was correctly selected and executed successfully, and the returned result is valid for the target task; Indicates the first Error types, , This indicates the number of preset error types, including parameter errors, call failures, timeout errors, semantic mismatches, empty results, or inefficient results; therefore, the set... Includes There are two result types, namely one success type and one... Error types.

[0041] This invention further decomposes the previously uniformly processed failure results into feedback categories with clear semantic boundaries, enabling policy learning not only to make coarse-grained judgments of success / failure, but also to distinguish the specific impact of different error patterns on decision quality.

[0042] For a given context state representation and the selected action The error indication vector corresponding to the feedback signal is defined as follows:

[0043]

[0044] in, Indicates the context state. Next action The resulting type indicator vector is obtained later; Indicates the first One component; when the current feedback signal belongs to type hour, ,otherwise ;

[0045] In single-label error detection scenarios, a single tool call corresponds to only one dominant result type, therefore satisfying:

[0046]

[0047] In complex scenarios, it is extended to multi-label error representation, which allows multiple components to take the value 1 at the same time, so as to characterize multiple abnormal attributes at the same time. For example, a single call may have both timeout and empty result exceptions at the same time.

[0048] To further characterize the actual utility of the tool output for the target task, this invention further introduces a continuous utility index vector:

[0049]

[0050] in, This indicates the number of continuous utility indicators. Indicates the first Each continuous utility information can correspond to continuous quantitative factors such as task completion, return result quality, execution cost, response latency, and result stability.

[0051] To reduce the dimensional differences between different continuous utility indicators, After normalization, we get:

[0052]

[0053] in, Represents the normalized i-th There are several continuous utility indicators. For indicators such as task completion, return result quality, and stability, which are "the bigger the better," they can be directly normalized to a larger value to indicate better performance. For indicators such as response latency and execution cost, which are "the smaller the better," they can be converted into latency scores or cost scores to ensure that all continuous utility indicators are consistent in direction, that is, the larger the value, the higher the utility.

[0054] Define the comprehensive continuous utility score for:

[0055]

[0056] in This represents the continuous utility aggregate weight vector. , Indicates the first The weight of each continuous utility indicator in the overall continuous utility score And satisfy:

[0057]

[0058] Continuous utility aggregate weight vector The weighting can be pre-set by human experience, business priorities, or expert rules, or determined through validation set tuning, grid search, historical call data statistics, or learning algorithms. For example, in scenarios that emphasize result quality, the weighting corresponding to the quality of returned results and task completion can be increased; in scenarios that emphasize real-time performance, the weighting corresponding to response latency score can be increased; and in scenarios that emphasize resource consumption, the weighting corresponding to execution cost score can be increased.

[0059] Construct the error-aware total reward based on the error indication vector and continuous utility index vector. The expression is:

[0060]

[0061] in, Indicates the first The result type indicates the weights corresponding to the vector; It represents the balance coefficient of the continuous utility index vector, used to adjust the relative contributions of discrete error feedback terms and continuous utility feedback terms to the total reward.

[0062] Regarding the specific values, Typically, non-negative values ​​can be taken, i.e. When the system places greater emphasis on error type penalties, it can be set... or When the system needs to consider both error type and continuous utility simultaneously, it can be set as follows: When continuous utility metrics such as return quality, execution cost, or response latency have a significant impact on business objectives, a setting can be made. Or further optimization can be performed based on the validation set. It can also be determined adaptively through manual experience setting, validation set grid search, historical call data statistics, or learning algorithms.

[0063] Indicates whether the execution result belongs to the first... Class result type; basic reward is denoted as :

[0064]

[0065] Under a single-tag setting, it satisfies

[0066]

[0067] Indicate the result type The corresponding reward weight, generally, a successful execution corresponds to a positive reward, that is... Different error types correspond to different levels of negative rewards, i.e. .

[0068] Specifically:

[0069] When a tool is successfully invoked and the returned result is valid, the system provides a positive reward to reinforce the correct tool selection behavior in the current state;

[0070] When the error is a parameter error, it means that the tool may be semantically related to the task, but there is a deviation in the calling parameters or calling method. Therefore, its penalty intensity is usually lower than that of semantic mismatch errors.

[0071] When the error is a semantic mismatch, it indicates that there is a directional bias in the tool selection itself, which is a decision-making error. Therefore, its penalty weight is usually higher than that of parameter errors.

[0072] When the error is a call failure, network error, or timeout error, the penalty intensity can be adaptively set according to the system deployment environment, tool stability, and business tolerance.

[0073] Preferably, the implementation process of step S4 is as follows:

[0074] After obtaining the total reward for error perception, a context bandit mechanism is introduced to guide the update of the policy function. The optimization objective is defined as maximizing the expected reward:

[0075]

[0076] in, Indicates the context state Next action The total reward obtained after error perception, when the action Take as the first Candidate actions At that time, the corresponding total reward for error perception can be recorded as ; Indicates the state representation in a given context. At that time, the policy function is in the action space The complete choice probability distribution on; Representation of different context states and different tool selection actions Calculate the mathematical expectation of the sampling results;

[0077] During policy updates, a parameter optimization method based on policy gradients is adopted, and the update direction is represented as follows:

[0078]

[0079]

[0080] in, Represents the policy objective function; Represents the distribution of context state samples. This indicates that the context state is sampled from historical tasks or online interactions. Indicates the state in a given context Under the given conditions, the tool selects actions based on the action probability distribution output by the policy function;

[0081] In a single interaction sample, the context state is represented Select action Execute the corresponding tool Later received a reward The following parameter update method will be used:

[0082]

[0083] in, Indicates the learning rate. The baseline function can be represented by historical average reward, moving average reward, or a value function estimate, used to reduce the variance of gradient estimation; if no baseline function is set, then it can be set as follows: .

[0084] The above formula shows that when choosing a tool brings a high reward, the probability of its corresponding strategy will be increased; when a certain behavior continuously triggers a high-penalty error type, its selection probability will be decreased.

[0085] Therefore, policy optimization no longer relies solely on static labeled signals but is directly driven by actual execution results. In particular, because the reward signal in this invention contains fine-grained error information, the policy model can explicitly learn which tool selection behaviors are prone to causing parameter errors and which behaviors are more likely to lead to semantic mismatches, thereby achieving proactive avoidance of different error patterns.

[0086] Unlike traditional tool selection methods based on static ranking, this invention models the tool selection process as a dynamic decision-making process based on context state. This allows the policy function to not only utilize the initial selection capabilities obtained during the offline training phase, but also to continuously optimize by incorporating real execution feedback during the interaction process.

[0087] Specifically, during the initial system operation phase, before historical execution feedback is available, an initial state representation is constructed based on user queries, candidate tool descriptions, parameter signatures, and optional prior features. A static semantic matching model is then used to provide an initial preference to complete the first round of tool selection. Subsequently, after each tool execution, the system generates an error-aware reward based on the execution result. The reward is used as a Bandit feedback signal to update the policy parameters. .

[0088] By modeling all candidate tools using a unified strategy function and combining execution feedback to perform closed-loop optimization of decisions, we can avoid relying on fixed rules or static matching and achieve adaptive decision-making for complex and variable tool invocation scenarios.

[0089] Through the above design, this invention transforms the tool selection problem from a ranking task based on relevance scoring into a feedback-driven sequential decision problem, enabling the model to have continuous learning and dynamic optimization capabilities, and providing a unified decision basis for subsequent reward modeling and policy update mechanisms.

[0090] After making a tool selection decision and obtaining execution feedback, the system needs to continuously optimize the strategy using this feedback so that the model can gradually learn the optimal tool selection behavior under different contextual states. To this end, this invention constructs a strategy learning and optimization mechanism based on execution feedback, directly incorporating tool call results into the strategy update process, thereby achieving a shift from offline static learning to online closed-loop optimization.

[0091] The core idea of ​​this mechanism is to treat each tool selection as an interactive process with feedback, given a context state. Below, by the policy function Select tool action And receive feedback signals generated by the error-aware reward mechanism after the tool is executed. Subsequently, this feedback was used to adjust the policy parameters. The update reinforces the selection of high-quality tools while suppressing low-quality or error-prone behaviors.

[0092] Preferably, to improve the robustness of policy updates and sample utilization efficiency, multiple interaction samples are grouped into training batches for joint optimization. Specifically:

[0093] Assuming that within an update cycle, the following data is collected: Interaction Samples ,in, This indicates the number of interaction samples in an update cycle; Indicates the first The context state representation corresponding to each interaction sample; Indicates the first The selected tool selection action in each interactive sample; Indicates the first The total error-perceived reward for each interaction sample. ;

[0094] The batch optimization objective is represented as:

[0095]

[0096] in, Let be the policy loss function. For the first The baseline value corresponding to each sample; during the optimization process, the strategy parameters are optimized by minimizing the batch optimization objective. Iterative updates.

[0097] This batch optimization method can avoid excessive perturbation of parameter updates by a single interactive sample, while improving the comprehensive constraint effect of different types of error samples on policy learning, thereby enhancing the stability of the overall optimization.

[0098] Preferably, to avoid the policy getting stuck in local optima during training, while taking into account both decision stability and exploration ability, the decision-making behavior of the policy function is adjusted:

[0099] Random sampling is performed based on the policy distribution, i.e., from The tool action is selected based on probability, thus preserving the priority of high-probability tools while introducing a certain degree of random exploration capability. The strategy distribution is as follows:

[0100]

[0101] in, For the exploration rate parameter, ; Representing the action space Any candidate action in the list; Indicates from action space The action is randomly selected. The random selection can be a uniform random selection or a random selection based on a preset exploration distribution.

[0102] The system employs the above mechanisms to maintain a high exploration rate in the early stages of training to obtain diverse execution feedback information; as the training progresses, the exploration rate can be gradually reduced. This enhances the utilization of the strategy and achieves a smooth transition from exploration-driven to stable convergence. The exploration rate... It can be set to a fixed value, or dynamically adjusted according to the training rounds, policy convergence degree, or reward distribution.

[0103] In real-world tool usage scenarios, task distribution, candidate tool sets, and tool behavior characteristics can all change continuously over time, exhibiting significant non-stationarity. For example, the same type of user query may correspond to different optimal tools at different stages; the interface definition, return format, response latency, and availability of the same tool may also change dynamically. If a one-time offline training and fixed deployment approach is still adopted, the policy model is prone to performance degradation due to environmental drift, making it difficult to maintain long-term stable tool selection results. To address this, this invention further constructs an online adaptive update mechanism, enabling the system to incrementally adjust and dynamically correct the policy using continuously arriving real-world interactive feedback.

[0104] Preferably, to avoid performance degradation of the policy function due to environmental drift, an online adaptive update mechanism is introduced to incrementally adjust and dynamically correct the policy.

[0105] For each interaction record generated during system operation, construct an online feedback sample:

[0106]

[0107] in, Indicates time The context state representation, Indicates the selected tool action. This represents the total reward value for error perception. Indicates the corresponding feedback signal error type; continuously writes feedback samples into the online experience cache pool. middle:

[0108]

[0109] The experience cache pool is used to store high-value interaction information in the recent operating environment, enabling the model to update its policies based on the latest environmental samples without relying on a static training set. Compared with traditional methods that train based solely on fixed historical data, this design can more timely reflect the changing trends of the tool environment and user needs.

[0110] A sliding window approach is used to select interaction samples from the most recent period as the basis for policy updates; let the window length be... Then at time The local empirical set is defined as:

[0111]

[0112] By optimizing using only recent samples within the window, the system can enhance its sensitivity to the current environmental distribution, avoid over-reliance on outdated feedback, and thus improve the strategy's response speed to changes in tool interfaces, fluctuations in call stability, and drift in user query distribution.

[0113] Based on local experience sets The system periodically or triggered the adjustment of strategy parameters. Incremental updates are performed, with the goal of maximizing the expected reward under the current environmental distribution, i.e.:

[0114]

[0115] The update format is represented as:

[0116]

[0117] in, The formula represents the learning rate; it shows that the system does not need to retrain the entire model from scratch after each environmental change, but can make local corrections based on the existing policy parameters and the newly arrived execution feedback, thereby significantly reducing update costs and improving deployment efficiency.

[0118] While relying solely on the latest samples for updates can improve the speed of environmental adaptation, it may also lead to the model excessively chasing short-term noise or even forgetting its original general decision-making capabilities. Therefore, this invention introduces a stability constraint term during the online update process to limit the magnitude of parameter changes and balance the contradiction between rapid adaptation and knowledge preservation.

[0119] Add parameter regularization terms to the online optimization objective:

[0120]

[0121] in, This represents the parameters of the offline pre-trained or previously stabilized policy. This represents the regularization coefficient; this term is used to constrain the current parameters from deviating too far from the existing stable policy, thereby suppressing drastic policy drift caused by local abnormal samples or stage noise.

[0122] To avoid the overhead of frequent updates, trigger-based update conditions are set to automatically initiate online adaptive updates when any of the following conditions are detected:

[0123] ① When the average reward obtained by the system in the most recent tool calls or the most recent time window exceeds the preset range compared to the historical average reward or the preset benchmark reward;

[0124] ② The frequency of a certain error type exceeds the preset threshold corresponding to the error type;

[0125] ③ The candidate toolset has been added, deleted, or its interface has been changed;

[0126] ④ The tool call success rate, timeout rate, or semantic mismatch rate exceeds the preset threshold compared to the historical reference average.

[0127] This invention constructs an online adaptive update mechanism suitable for complex and dynamic tool environments. Its main technical functions are: first, enabling the strategy model to continuously absorb the latest execution feedback and improve its responsiveness to environmental changes; second, avoiding the need to retrain the entire model for each environmental change, thus reducing deployment and maintenance costs; third, achieving a balance between rapid adaptation and stable maintenance through sliding windows, regularization constraints, and triggered update mechanisms; and fourth, enabling the tool selection system to evolve from a static deployment mode to a sustainable, long-term operating system.

[0128] For any details not covered in this invention, please refer to the prior art.

[0129] The beneficial effects of this invention are as follows:

[0130] (1) It has achieved refined identification and effective avoidance of tool execution failure modes.

[0131] Existing tool selection methods typically rely solely on successful tool invocation for coarse-grained modeling, failing to distinguish between heterogeneous anomalies such as parameter errors, invocation failures, timeout errors, and semantic mismatches, making it difficult for the model to understand the root causes of failures. This invention constructs an error-aware reward mechanism, expanding the execution result from a single binary signal into a structured error feedback representation, and assigning differentiated reward and penalty weights. This enables the model to explicitly learn the deep mapping relationship between "specific context-specific tool-specific error pattern." This mechanism breaks through the limitations of learning only from positive samples, significantly enhancing the ability to identify and automatically suppress high-risk invocation behaviors, thus improving decision-making accuracy from the source.

[0132] (2) Significantly enhanced the model's adaptability to complex task environments and dynamic tool environments.

[0133] Real-world tool usage scenarios exhibit significant non-stationarity: on the one hand, different user queries differ significantly in semantic expression, task objectives, and contextual constraints; on the other hand, the candidate tool set, tool interface behavior, and tool availability may continuously change over time. Traditional methods based on static supervised learning typically rely on a fixed training distribution, making them ill-suited for such dynamic environments. This invention models the tool selection process as a contextualized bandit decision problem and incorporates an online execution feedback continuous update strategy, enabling the model to continuously correct its decision-making behavior based on the latest interaction results during operation. When task distribution, tool behavior, or error patterns change, the system can promptly adjust its selection strategy, thereby maintaining high decision robustness and environmental adaptability in complex and ever-changing real-world environments.

[0134] (3) It provides a unified tool selection and feedback optimization framework, which improves the system's generalization ability.

[0135] Existing methods typically treat tool selection as a static semantic matching or offline ranking problem, making it difficult to simultaneously model task context, tool semantics, execution feedback, and decision optimization processes within a unified framework. This invention constructs an end-to-end closed-loop tool selection optimization framework by organically integrating modules such as unified representation modeling, context state construction, bandit decision-making, error-aware reward, and online policy updates. This framework can not only uniformly handle candidate tools from different sources, with different functions, and different parameter structures, but also dynamically evaluate and select tools for complex task scenarios within a unified decision space. Due to its unified modeling and feedback-driven optimization capabilities, this invention exhibits stronger cross-task generalization and cross-tool environment transfer capabilities compared to traditional static methods.

[0136] (4) It balances decision-making quality and optimization efficiency, and has the potential for practical deployment and business application.

[0137] Traditional tool selection methods often rely on large amounts of statically labeled data for offline training. Once deployed, performance may be insufficient or the environment may change, typically requiring a complete dataset reconstruction and retraining. This results in high update costs and slow response times, failing to meet the continuous optimization needs of real-world systems. This invention utilizes feedback signals naturally generated during tool execution for policy learning, eliminating the need for large-scale manual annotation and enabling continuous optimization of decision-making behavior based on existing models. This design significantly reduces model iteration costs and improves the system's response efficiency to new scenarios. Furthermore, because this invention employs an online incremental policy update mechanism, rather than frequent retraining of the entire model, it controls computational overhead while ensuring optimization effectiveness, providing technical feasibility for deployment in real-world tool invocation platforms, intelligent agent systems, and complex business process automation systems.

[0138] (5) It enhances the practical application value of the method.

[0139] Most existing tool selection techniques remain at the stage of one-time ranking based on semantic relevance. Their optimization objectives often differ from the actual task performance, leading to numerous invalid or erroneous selections even if the ranking results perform well in offline evaluations. This invention elevates the tool selection process from a static ranking task to a dynamic decision-making task based on actual execution performance by introducing error-aware rewards and a contextual bandit mechanism. This allows the system to continuously optimize around the final task benefit. This design means that tool selection no longer relies solely on pre-given supervision labels but can continuously learn "which tools are more effective and which choices are more prone to failure" in real interactions. Therefore, this invention bridges the technical path from offline relevance learning to online execution performance optimization, providing a more long-term evolutionary and practically valuable solution for large language model tool invocation systems, intelligent agent decision-making systems, and complex task automation systems. Attached Figure Description

[0140] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0141] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0142] Figure 2 This is a schematic diagram of the tool selection optimization method based on error-aware reward of the present invention. Detailed Implementation

[0143] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. However, this is not the only description; all aspects not described in detail herein are based on conventional techniques in the art.

[0144] Terminology Explanation:

[0145] Context Bandit mechanism: also known as context multi-armed slot machine decision mechanism, refers to an online decision mechanism in which the system selects one or more actions from multiple candidate actions to execute under the given current context state, and updates the subsequent selection strategy based on the real-time feedback after execution.

[0146] Example 1

[0147] A tool selection optimization system based on error-perceived reward, such as Figure 1 As shown, it includes a context representation modeling module, a tool selection decision module, an execution feedback parsing module, and a strategy optimization and update module;

[0148] The context representation modeling module is used to uniformly encode user queries, candidate tool information, and optional historical call features to construct a context state representation for subsequent decision-making.

[0149] The tool selection decision module is used to score, rank, or select the actions corresponding to candidate tools based on contextual state representations.

[0150] The execution feedback parsing module is used to receive the return results after the selected tool is executed, and to identify success information, error type, and continuous utility information;

[0151] The strategy optimization and update module is used to generate reward signals based on execution feedback and update the tool selection strategy.

[0152] Example 2

[0153] A tool selection optimization method based on error-aware reward is implemented using the tool selection optimization system based on error-aware reward in Example 1, such as... Figure 2 As shown, it includes the following steps:

[0154] S1, given a user query and candidate toolkit ,in This indicates the number of candidate tools corresponding to the current user's query. Indicates the first One candidate tool, using the context representation modeling module to analyze user queries. Encode the candidate tool information to construct a context state representation. , , The context state represents the dimension of the vector;

[0155] S2, in the tool selection decision stage, is determined by the set of candidate tools. Inducing action space Each candidate tool Corresponds to an optional action ,action This indicates that the first option is selected in the current context. Candidate tools Make the call;

[0156] Through policy function Computing the context state representation Select action The probability or preference of selection, and output the optimal tool, Top- The tool or tool sorting results, where Top- This indicates selecting the highest-scoring candidate from the set of candidate tools. A tool, ;

[0157] S3, after the selected tool is executed in the external environment, the corresponding feedback signal is obtained. The feedback signal includes the error type and the continuous utility index.

[0158] S4. In the policy optimization phase, the system updates the policy function based on feedback signals, which strengthens high-quality decision-making behavior and suppresses low-quality or error-prone behavior.

[0159] This invention realizes the transformation from tool ranking based on static semantic matching to dynamic decision optimization driven by execution results. The entire architecture runs under a unified input-output path and achieves continuous learning and performance improvement through an execution feedback loop.

[0160] Example 3

[0161] An error-aware reward-based tool selection optimization method, as described in Example 2, differs in that, in step S1, each candidate tool information first includes a tool description. Parameter signature and optional history call features The user query and candidate tool information are encoded separately, as follows:

[0162]

[0163] in, and Encoding functions for user queries and candidate tool information (e.g., implemented based on Transformer or pre-trained language models) respectively map queries and tools to a unified feature space. ; Indicates user query Vectorized representation of; Indicates the first Candidate tools Vectorized representation of;

[0164] Subsequently, the vectorized representation of the user query is combined with the vectorized representation of the candidate tools to construct a context state representation. The format is as follows:

[0165]

[0166] in, The context fusion function can be implemented using vector concatenation, multilayer perceptron, attention mechanism, Transformer interactive coding, weighted pooling, or a combination thereof, and is used to model the interaction relationship between the query and candidate tools.

[0167] After the above processing, the original query and toolset are transformed into a unified context state representation, providing standardized input for subsequent Bandit-based decision-making mechanisms.

[0168] Example 4

[0169] An error-perceived reward-based tool selection optimization method, as described in Example 3, differs in that, in step S2, the action... It can be represented using one-hot vectors, index identifiers, or other discrete encoding methods.

[0170] Define the policy function Used to characterize the selection of actions under given context conditions. The probability distribution of actions Corresponding candidate tools ;

[0171] For each candidate tool, the selection probability is expressed as:

[0172]

[0173] in, Represents the context state. With action Corresponding candidate tools The matching score function between them; This represents the learnable parameters in the policy function or matching score function; Represented by natural constant An exponential function with base 0 is used to map the match score to a non-negative value;

[0174] In the decision-making process, the action space is based on the policy function. In this scenario, one or more actions are selected, and the corresponding candidate tools are determined accordingly. For a single-tool selection scenario, the decision-making form is as follows:

[0175]

[0176] in, Indicates the state in the current context. The selected optimal tool selection action; let The corresponding candidate tools are ,but The tool ultimately selected and invoked by the system;

[0177] For scenarios with multiple candidate tools, output the Top-ranked tools in descending order of selection probability or matching score. The sorting result can be represented as follows:

[0178]

[0179] in, Indicates the current context state. Down to action space The sorting results.

[0180] Matching score function The parameterized model, obtained through the parameterization module, is used to receive context state representations. With action Corresponding candidate tools The vectorized representation is taken as input and the corresponding matching score is output; the parameterization module is implemented using a neural network structure, including but not limited to multilayer perceptron, attention-based encoding network or a combination thereof.

[0181] Example 5

[0182] An error-aware reward-based tool selection optimization method, as described in Example 4, differs in that, in step S3, the feedback signal contains multiple abnormal patterns with different semantics and consequences, constructing a tool feedback signal type set. :

[0183]

[0184] in, This indicates that the tool was correctly selected and executed successfully, and the returned result is valid for the target task; Indicates the first Error types, , This indicates the number of preset error types, including parameter errors, call failures, timeout errors, semantic mismatches, empty results, or inefficient results; therefore, the set... Includes There are two result types, namely one success type and one... Error types.

[0185] This invention further decomposes the previously uniformly processed failure results into feedback categories with clear semantic boundaries, enabling policy learning not only to make coarse-grained judgments of success / failure, but also to distinguish the specific impact of different error patterns on decision quality.

[0186] For a given context state representation and the selected action The error indication vector corresponding to the feedback signal is defined as follows:

[0187]

[0188] in, Indicates the context state. Next action The resulting type indicator vector is obtained later; Indicates the first One component; when the current feedback signal belongs to type hour, ,otherwise ;

[0189] In single-label error detection scenarios, a single tool call corresponds to only one dominant result type, therefore satisfying:

[0190]

[0191] In complex scenarios, it is extended to multi-label error representation, which allows multiple components to take the value 1 at the same time, so as to characterize multiple abnormal attributes at the same time. For example, a single call may have both timeout and empty result exceptions at the same time.

[0192] To further characterize the actual utility of the tool output for the target task, this invention further introduces a continuous utility index vector:

[0193]

[0194] in, This indicates the number of continuous utility indicators. Indicates the first Each continuous utility information can correspond to continuous quantitative factors such as task completion, return result quality, execution cost, response latency, and result stability.

[0195] To reduce the dimensional differences between different continuous utility indicators, After normalization, we get:

[0196]

[0197] in, Represents the normalized i-th There are several continuous utility indicators. For indicators such as task completion, return result quality, and stability, which are "the bigger the better," they can be directly normalized to a larger value to indicate better performance. For indicators such as response latency and execution cost, which are "the smaller the better," they can be converted into latency scores or cost scores to ensure that all continuous utility indicators are consistent in direction, that is, the larger the value, the higher the utility.

[0198] Define the comprehensive continuous utility score for:

[0199]

[0200] in This represents the continuous utility aggregate weight vector. , Indicates the first The weight of each continuous utility indicator in the overall continuous utility score And satisfy:

[0201]

[0202] Continuous utility aggregate weight vector The weighting can be pre-set by human experience, business priorities, or expert rules, or determined through validation set tuning, grid search, historical call data statistics, or learning algorithms. For example, in scenarios that emphasize result quality, the weighting corresponding to the quality of returned results and task completion can be increased; in scenarios that emphasize real-time performance, the weighting corresponding to response latency score can be increased; and in scenarios that emphasize resource consumption, the weighting corresponding to execution cost score can be increased.

[0203] Construct the error-aware total reward based on the error indication vector and continuous utility index vector. The expression is:

[0204]

[0205] in, Indicates the first The result type indicates the weights corresponding to the vector; It represents the balance coefficient of the continuous utility index vector, used to adjust the relative contributions of discrete error feedback terms and continuous utility feedback terms to the total reward.

[0206] Regarding the specific values, Typically, non-negative values ​​can be taken, i.e. When the system places greater emphasis on error type penalties, it can be set... or When the system needs to consider both error type and continuous utility simultaneously, it can be set as follows: When continuous utility metrics such as return quality, execution cost, or response latency have a significant impact on business objectives, a setting can be made. Or further optimization can be performed based on the validation set. It can also be determined adaptively through manual experience setting, validation set grid search, historical call data statistics, or learning algorithms.

[0207] Indicates whether the execution result belongs to the first... Class result type; basic reward is denoted as :

[0208]

[0209] Under a single-tag setting, it satisfies

[0210]

[0211] Indicate the result type The corresponding reward weight, generally, a successful execution corresponds to a positive reward, that is... Different error types correspond to different levels of negative rewards, i.e. .

[0212] Specifically:

[0213] When a tool is successfully invoked and the returned result is valid, the system provides a positive reward to reinforce the correct tool selection behavior in the current state;

[0214] When the error is a parameter error, it means that the tool may be semantically related to the task, but there is a deviation in the calling parameters or calling method. Therefore, its penalty intensity is usually lower than that of semantic mismatch errors.

[0215] When the error is a semantic mismatch, it indicates that there is a directional bias in the tool selection itself, which is a decision-making error. Therefore, its penalty weight is usually higher than that of parameter errors.

[0216] When the error is a call failure, network error, or timeout error, the penalty intensity can be adaptively set according to the system deployment environment, tool stability, and business tolerance.

[0217] In this embodiment The possible values ​​are as follows:

[0218]

[0219]

[0220] in, The reward weight represents the weight of successful execution and valid results. This indicates the penalty weight for parameter errors. Indicates the weight of the timeout error penalty. Indicates the penalty weight for empty or inefficient results. Indicates the penalty weight for failed invocation. This represents the penalty weight for semantic mismatch. The values ​​mentioned above are for illustrative purposes only; in actual deployment, they can be set or adjusted based on business tolerance, tool importance, error severity, historical call statistics, or validation set performance.

[0221] Example 6

[0222] An error-aware reward-based tool selection optimization method, as described in Example 5, differs in that the implementation process of step S4 is as follows:

[0223] After obtaining the total reward for error perception, a context bandit mechanism is introduced to guide the update of the policy function. The optimization objective is defined as maximizing the expected reward:

[0224]

[0225] in, To indicate the context state Next action The total reward obtained after error perception, when the action Take as the first Candidate actions At that time, the corresponding total reward for error perception can be recorded as ; Indicates the state representation in a given context. At that time, the policy function is in the action space The complete choice probability distribution on; Representation of different context states and different tool selection actions Calculate the mathematical expectation of the sampling results;

[0226] During policy updates, a parameter optimization method based on policy gradients is adopted, and the update direction is represented as follows:

[0227]

[0228]

[0229] in, Represents the policy objective function; Represents the distribution of context state samples. This indicates that the context state is sampled from historical tasks or online interactions. Indicates the state in a given context Under the given conditions, the tool selects actions based on the action probability distribution output by the policy function;

[0230] In a single interaction sample, the context state is represented Select action Execute the corresponding tool Later received a reward The following parameter update method will be used:

[0231]

[0232] in, Indicates the learning rate. The baseline function can be represented by historical average reward, moving average reward, or a value function estimate, used to reduce the variance of gradient estimation; if no baseline function is set, then it can be set as follows: .

[0233] The above formula shows that when choosing a tool brings a high reward, the probability of its corresponding strategy will be increased; when a certain behavior continuously triggers a high-penalty error type, its selection probability will be decreased.

[0234] Therefore, policy optimization no longer relies solely on static labeled signals but is directly driven by actual execution results. In particular, because the reward signal in this invention contains fine-grained error information, the policy model can explicitly learn which tool selection behaviors are prone to causing parameter errors and which behaviors are more likely to lead to semantic mismatches, thereby achieving proactive avoidance of different error patterns.

[0235] Unlike traditional tool selection methods based on static ranking, this invention models the tool selection process as a dynamic decision-making process based on context state. This allows the policy function to not only utilize the initial selection capabilities obtained during the offline training phase, but also to continuously optimize by incorporating real execution feedback during the interaction process.

[0236] Specifically, during the initial system operation phase, before historical execution feedback is available, an initial state representation is constructed based on user queries, candidate tool descriptions, parameter signatures, and optional prior features. A static semantic matching model is then used to provide an initial preference to complete the first round of tool selection. Subsequently, after each tool execution, the system generates an error-aware reward based on the execution result. The reward is used as a Bandit feedback signal to update the policy parameters. .

[0237] By modeling all candidate tools using a unified strategy function and combining execution feedback to perform closed-loop optimization of decisions, we can avoid relying on fixed rules or static matching and achieve adaptive decision-making for complex and variable tool invocation scenarios.

[0238] Through the above design, this invention transforms the tool selection problem from a ranking task based on relevance scoring into a feedback-driven sequential decision problem, enabling the model to have continuous learning and dynamic optimization capabilities, and providing a unified decision basis for subsequent reward modeling and policy update mechanisms.

[0239] After making a tool selection decision and obtaining execution feedback, the system needs to continuously optimize the strategy using this feedback so that the model can gradually learn the optimal tool selection behavior under different contextual states. To this end, this invention constructs a strategy learning and optimization mechanism based on execution feedback, directly incorporating tool call results into the strategy update process, thereby achieving a shift from offline static learning to online closed-loop optimization.

[0240] The core idea of ​​this mechanism is to treat each tool selection as an interactive process with feedback, given a context state. Below, by the policy function Select tool action And receive feedback signals generated by the error-aware reward mechanism after the tool is executed. Subsequently, this feedback was used to adjust the policy parameters. The update reinforces the selection of high-quality tools while suppressing low-quality or error-prone behaviors.

[0241] Example 7

[0242] An error-aware reward-based tool selection optimization method, as described in Example 6, differs in that, to improve the robustness of policy updates and sample utilization efficiency, multiple interaction samples are grouped into training batches for joint optimization. Specifically:

[0243] Assuming that within an update cycle, the following data is collected: Interaction Samples ,in, This indicates the number of interaction samples in an update cycle; Indicates the first The context state representation corresponding to each interaction sample; Indicates the first The selected tool selection action in each interactive sample; Indicates the first The total error-perceived reward for each interaction sample. ;

[0244] The batch optimization objective is represented as:

[0245]

[0246] in, Let be the policy loss function. For the first The baseline value corresponding to each sample; during the optimization process, the strategy parameters are optimized by minimizing the batch optimization objective. Iterative updates.

[0247] This batch optimization method can avoid excessive perturbation of parameter updates by a single interactive sample, while improving the comprehensive constraint effect of different types of error samples on policy learning, thereby enhancing the stability of the overall optimization.

[0248] Example 8

[0249] An error-aware reward-based tool selection optimization method, as described in Example 7, differs in that, to avoid the policy getting trapped in local optima during training, while simultaneously considering decision stability and exploratory ability, the decision-making behavior of the policy function is controlled:

[0250] Random sampling is performed based on the policy distribution, i.e., from The tool action is selected based on probability, thus preserving the priority of high-probability tools while introducing a certain degree of random exploration capability. The strategy distribution is as follows:

[0251]

[0252] in, For the exploration rate parameter, ; Representing the action space Any candidate action in the list; Indicates from action space The action is randomly selected. The random selection can be a uniform random selection or a random selection based on a preset exploration distribution.

[0253] The system employs the above mechanisms to maintain a high exploration rate in the early stages of training to obtain diverse execution feedback information; as the training progresses, the exploration rate can be gradually reduced. This enhances the utilization of the strategy and achieves a smooth transition from exploration-driven to stable convergence. The exploration rate... It can be set to a fixed value, or dynamically adjusted according to the training rounds, policy convergence degree, or reward distribution.

[0254] In real-world tool usage scenarios, task distribution, candidate tool sets, and tool behavior characteristics can all change continuously over time, exhibiting significant non-stationarity. For example, the same type of user query may correspond to different optimal tools at different stages; the interface definition, return format, response latency, and availability of the same tool may also change dynamically. If a one-time offline training and fixed deployment approach is still adopted, the policy model is prone to performance degradation due to environmental drift, making it difficult to maintain long-term stable tool selection results. To address this, this invention further constructs an online adaptive update mechanism, enabling the system to incrementally adjust and dynamically correct the policy using continuously arriving real-world interactive feedback.

[0255] Example 9

[0256] An error-aware reward-based tool selection optimization method, as described in Example 8, differs in that, to avoid performance degradation of the policy function due to environmental drift, an online adaptive update mechanism is introduced to incrementally adjust and dynamically correct the policy.

[0257] For each interaction record generated during system operation, construct an online feedback sample:

[0258]

[0259] in, Indicates time The context state representation, Indicates the selected tool action. This represents the total reward value for error perception. Indicates the corresponding feedback signal error type; continuously writes feedback samples into the online experience cache pool. middle:

[0260]

[0261] The experience cache pool is used to store high-value interaction information in the recent operating environment, enabling the model to update its policies based on the latest environmental samples without relying on a static training set. Compared with traditional methods that train based solely on fixed historical data, this design can more timely reflect the changing trends of the tool environment and user needs.

[0262] A sliding window approach is used to select interaction samples from the most recent period as the basis for policy updates; let the window length be... Then at time The local empirical set is defined as:

[0263]

[0264] By optimizing using only recent samples within the window, the system can enhance its sensitivity to the current environmental distribution, avoid over-reliance on outdated feedback, and thus improve the strategy's response speed to changes in tool interfaces, fluctuations in call stability, and drift in user query distribution.

[0265] Based on local experience sets The system periodically or triggered the adjustment of strategy parameters. Incremental updates are performed, with the goal of maximizing the expected reward under the current environmental distribution, i.e.:

[0266]

[0267] The update format is represented as:

[0268]

[0269] in, The formula represents the learning rate; it shows that the system does not need to retrain the entire model from scratch after each environmental change, but can make local corrections based on the existing policy parameters and the newly arrived execution feedback, thereby significantly reducing update costs and improving deployment efficiency.

[0270] While relying solely on the latest samples for updates can improve the speed of environmental adaptation, it may also lead to the model excessively chasing short-term noise or even forgetting its original general decision-making capabilities. Therefore, this invention introduces a stability constraint term during the online update process to limit the magnitude of parameter changes and balance the contradiction between rapid adaptation and knowledge preservation.

[0271] Add parameter regularization terms to the online optimization objective:

[0272]

[0273] in, This represents the parameters of the offline pre-trained or previously stabilized policy. This represents the regularization coefficient; this term is used to constrain the current parameters from deviating too far from the existing stable policy, thereby suppressing drastic policy drift caused by local abnormal samples or stage noise.

[0274] Example 10

[0275] An error-aware reward-based tool selection optimization method, as described in Example 9, differs in that, to avoid the additional overhead caused by frequent updates, a trigger-based update condition is set, automatically initiating online adaptive updates when any of the following conditions are detected:

[0276] ① When the average reward obtained by the system in the most recent tool calls or the most recent time window exceeds the preset range compared to the historical average reward or the preset benchmark reward;

[0277] ② The frequency of a certain error type exceeds the preset threshold corresponding to the error type;

[0278] ③ The candidate toolset has been added, deleted, or its interface has been changed;

[0279] ④ The tool call success rate, timeout rate, or semantic mismatch rate exceeds the preset threshold compared to the historical reference average.

[0280] This invention constructs an online adaptive update mechanism suitable for complex and dynamic tool environments. Its main technical functions are: first, enabling the strategy model to continuously absorb the latest execution feedback and improve its responsiveness to environmental changes; second, avoiding the need to retrain the entire model for each environmental change, thus reducing deployment and maintenance costs; third, achieving a balance between rapid adaptation and stable maintenance through sliding windows, regularization constraints, and triggered update mechanisms; and fourth, enabling the tool selection system to evolve from a static deployment mode to a sustainable, long-term operating system.

[0281] To ensure that the method proposed in this invention can learn representative tool invocation decision patterns and obtain objective and accurate evaluations, this embodiment constructs a dedicated tool selection protocol dataset (baseline_v3_rerank) covering diverse task queries and a large candidate tool space. The basic protocol portion of this dataset contains 1000 training set queries (a total of 13326 rows of positive and negative sample pairs and 7710 supervised fine-tuning data, covering 5457 candidate tools), and is equipped with development and test sets, each containing 200 queries. The test set has an average of 7.35 candidate tools, used to calibrate the model's basic ranking ability. Simultaneously, to verify the model's robustness in extreme scenarios, this invention selectively samples from baseline training error samples to construct a hard sample set (Harder Split) containing 200 queries and 2669 rows of sample pairs. The average first-hit rank (avg_first_hit_rank) of this set is as high as 11.44, and the recall metric of traditional baseline models on this set suffers severe decay, thus forming the core benchmark for verifying the effectiveness of the "error perception" and "online feedback" strategies of this invention.

[0282] The comparative experiments were conducted in a uniformly configured hardware and software environment. The base model for the comparative experiments was the lightweight pre-trained language model DistilGPT2, with full parameter fine-tuning implemented.

[0283] To ensure fairness in the experimental comparison, the hyperparameters of the model training were strictly controlled: the input truncation length was set to 512, the batch size per card was 2, combined with gradient accumulation of 4 steps (equivalent to a global batch size of 8), and the learning rate was set to 5×10⁻. 5 It is supplemented by a cosine learning rate scheduling strategy and a warmup ratio of 0.1. Under the uniform conditions of a fixed random seed of 42 and a training epoch count of 1, the method in this embodiment is compared with two control techniques. The control techniques include a comparison model and an ablation method. The comparison model is a baseline model for tool selection reranking based on static supervised learning (baseline_v3_rerank_e1), and the ablation method is used to verify the independent role of the error-aware reward mechanism without introducing online policy updates. The experimental evaluation system consists of main metrics ( , and , The values ​​are 1, 3, and 5 respectively, and together with the output health auxiliary indicators (empty prediction rate, format error rate, and candidate violation rate) constitute the total.

[0284] The empty prediction rate is the proportion of times when the statistical model outputs no candidate tools or the Top-k results are empty; the format error rate is the proportion of times when the statistical model output cannot be parsed into the specified tool identifier, tool list, or sorting result format; and the candidate out-of-bounds rate is the proportion of times when the tool output by the statistical model does not belong to the current candidate tool set. The proportion. In this invention, subsequent tool execution, feedback parsing, error type identification, and reward calculation are only meaningful when the model output is non-empty, in a valid format, and does not exceed the bounds. Therefore, the output health auxiliary index is used to verify the executability, stability, and deployment reliability of the tool selection strategy, and together with the ranking performance index, constitutes the comprehensive evaluation system of the method of this invention.

[0285] Under the above unified settings, the following comparison methods are constructed respectively:

[0286] (1) Baseline method: Tool selection model based on supervised learning (training only based on static annotations);

[0287] (2) Improved sorting method: Transform the tool selection task into a candidate tool scoring and sorting problem;

[0288] (3) The method of the present invention: Based on the ranking modeling, an error-aware reward mechanism and an online strategy optimization method are introduced.

[0289] Key indicators include , and This is used to measure the model's performance in terms of tool selection accuracy and ranking quality. Among them:

[0290]

[0291]

[0292]

[0293] in, This represents the total number of test samples. Indicates the first A set of tools that provide true relevance to a single sample. The model predicts the previous A collection of tools, and These represent the cumulative gain due to loss and the ideal cumulative gain due to loss, respectively, and are calculated as follows:

[0294]

[0295] in, Indicates the first The sample at the th The relevance score at each predicted location. If the tool corresponding to that location belongs to the set of truly relevant tools. Then it can be made Otherwise .

[0296] Indicates the first The maximum cumulative gain that a sample can achieve under ideal ranking conditions, which is the value obtained after arranging the real-relevant tools in the optimal order. The calculation method is as follows:

[0297]

[0298] Experiments show that while Model 1 performs reasonably well on the standard test set, its performance drops drastically when dealing with the aforementioned "difficult sample set," with a significant decrease in ranking metrics such as nDCG@5. This phenomenon fully exposes the severe limitations of static models relying solely on offline annotations when dealing with complex, ambiguous, or rare tool invocation scenarios. To overcome this bottleneck, this invention introduces an error-aware reward and Bandit weighting mechanism into the model training process. The system not only absorbs regular label feedback but also sensitively captures abnormal signals returned from the tool execution end (such as missing parameters, incorrect formatting, etc.), and dynamically weights the reward function accordingly.

[0299] Table 1 below shows the comparative results on the difficult sample set in a set of experimental results:

[0300] Table 1 Comparative Experiment Results

[0301]

[0302] As shown in Table 1 above, the nDCG@5 score of Comparison Model 1 on the difficult sample set is 0.197. This result indicates that static supervised learning models trained solely on offline labeled data have significantly insufficient ranking quality when faced with complex, ambiguous, or rare tool invocation scenarios. This is because such models primarily learn the static relevance between queries and tools, making it difficult to utilize error types, invocation states, and feedback results during the actual tool execution process. Therefore, they are prone to incorrect ranking when tools have similar semantics, complex task contexts, or unstable tool execution results.

[0303] Based on comprehensive experimental data, the proposed solution achieves a dual leap in tool selection accuracy and system robustness while fully ensuring output health. In terms of core ranking performance, the method consistently maintains a high level comparable to strong baselines on the standard test set (nDCG@5 approximately 0.464), while demonstrating outstanding technical advantages on the most challenging Harder Split set. By introducing error-aware rewards and a Bandit mechanism, the nDCG@5 index on the Harder Split set jumped from 0.197 to 0.224; with the further addition of an online incremental update strategy, this index continued to climb to 0.235, achieving an average absolute improvement of +0.038 compared to the global baseline. Simultaneously, the trajectory of online incremental validation clearly shows that the model can trigger continuous performance improvements with minimal real-time feedback signals, confirming its excellent adaptive capability of "learning while running" in real-world deployment environments. More importantly, in all core comparative experimental groups, the proposed method consistently maintained an ideal state of zero for health indicators such as null prediction rate, format error rate, and candidate out-of-bounds rate. Compared with traditional generative schemes, this invention fundamentally eliminates the occurrence of illegal tool calls and invalid predictions. While effectively solving the problem of difficult sample identification and the bottleneck of static strategy generalization, it provides a technical solution with both extremely high accuracy and absolute execution reliability for complex industrial decision-making scenarios.

[0304] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A tool selection optimization system based on error-aware reward, characterized by, It includes a context representation modeling module, a tool selection decision module, an execution feedback parsing module, and a strategy optimization and update module; The context representation modeling module is used to uniformly encode user queries, candidate tool information, and optional historical call features to construct a context state representation for subsequent decision-making. The tool selection decision module is used to score, rank, or select the actions corresponding to candidate tools based on contextual state representations. The execution feedback parsing module is used to receive the return results after the selected tool is executed, and to identify success information, error type, and continuous utility information; The strategy optimization and update module is used to generate reward signals based on execution feedback and update the tool selection strategy.

2. A method for tool selection optimization based on error-aware reward, the method comprising: The tool selection optimization system based on error-aware reward as described in claim 1 is implemented by the following steps: S1, given user query and candidate tool set wherein denotes the number of candidate tools corresponding to the current user query, denotes the th candidate tool, the context representation modeling module encodes the user query and candidate tool information to construct the context state representation ; S2, in the tool selection decision stage, is determined by the set of candidate tools. Inducing action space Each candidate tool Corresponding to one action ,action This indicates that the first option is selected in the current context. Candidate tools Make the call; Through policy function Computing the context state representation Select action The probability or preference of selection, and output the optimal tool, Top- The tool or tool sorting results, where Top- This indicates selecting the highest-scoring candidate from the set of candidate tools. A tool, ; S3, after the selected tool is executed in the external environment, the corresponding feedback signal is obtained. The feedback signal includes the error type and the continuous utility index. S4. In the policy optimization phase, the system updates the policy function based on feedback signals, which strengthens high-quality decision-making behavior and suppresses low-quality or error-prone behavior.

3. The tool selection optimization method based on error-aware reward according to claim 2, characterized in that, In step S1, firstly, the information for each candidate tool includes a tool description. Parameter signature and optional history call features The user query and candidate tool information are encoded separately, as follows: in, and These are the encoding functions for user queries and candidate tool information, respectively. Indicates user query Vectorized representation of; Indicates the first Candidate tools Vectorized representation of; Subsequently, the vectorized representation of the user query is combined with the vectorized representation of the candidate tools to construct a context state representation. The format is as follows: in, This is the context fusion function.

4. The tool selection optimization method based on error-aware reward according to claim 3, characterized in that, In step S2, the action Represented using one-hot vectors or indexed identifiers; Define the policy function Used to characterize the selection of actions under given context conditions. The probability distribution of actions Corresponding candidate tools ; For each candidate tool, the selection probability is expressed as: in, Represents the context state. With action Corresponding candidate tools The matching score function between them; This represents the learnable parameters in the policy function or matching score function; Represented by natural constant An exponential function with base 0 is used to map the match score to a non-negative value; In the decision-making process, the action space is based on the policy function. In this scenario, one or more actions are selected, and the corresponding candidate tools are determined accordingly. For a single-tool selection scenario, the decision-making form is as follows: in, Indicates the state in the current context. The selected optimal tool selection action; let The corresponding candidate tools are ,but The tool ultimately selected and invoked by the system; For scenarios with multiple candidate tools, output the Top-ranked tools in descending order of selection probability or matching score. The sorting results are presented as follows: in, Indicates the current context state. Downward movement space The sorting results.

5. The tool selection optimization method based on error-aware reward according to claim 4, characterized in that, Matching scoring function The parameterized model, obtained through the parameterization module, is used to receive context state representations. With action Corresponding candidate tools The vectorized representation is taken as input, and the corresponding matching score is output; the parameterization module is implemented using a neural network structure.

6. The tool selection optimization method based on error-aware reward according to claim 5, characterized in that, In step S3, the feedback signals contain various anomaly patterns with different semantics and consequences, and a set of tool feedback signal types is constructed. : in, This indicates that the tool was correctly selected and executed successfully, and the returned result is valid for the target task; Indicates the first Error types, , This indicates the number of preset error types, which include parameter errors, call failures, timeout errors, semantic mismatches, empty results, or inefficient results. For a given context state representation and the selected action The error indication vector corresponding to the feedback signal is defined as follows: in, Indicates the context state. Next action The resulting type indicator vector is obtained later; Indicates the first One component; when the current feedback signal belongs to type hour, ,otherwise ; Introducing a continuous utility index vector: in, This indicates the number of continuous utility indicators. Indicates the first A continuous utility information; To reduce the dimensional differences between different continuous utility indicators, After normalization, we get: in, Represents the normalized i-th A continuous utility index; Define the comprehensive continuous utility score for: in This represents the continuous utility aggregate weight vector. , Indicates the first The weight of each continuous utility indicator in the overall continuous utility score; Construct the error-aware total reward based on the error indication vector and continuous utility index vector. The expression is: in, Indicates the first The result type indicates the weights corresponding to the vector; This represents the balance coefficient of the continuation utility index vector.

7. The tool selection optimization method based on error-aware reward according to claim 6, characterized in that, The implementation process of step S4 is as follows: After obtaining the total reward for error perception, a context bandit mechanism is introduced to guide the update of the policy function. The optimization objective is defined as maximizing the expected reward: in, Indicates the context state Next action The total reward obtained after error perception; Indicates the state representation in a given context. At that time, the policy function is in the action space The complete choice probability distribution on; Representation of different context states and different tool selection actions Calculate the mathematical expectation of the sampling results; During policy updates, a parameter optimization method based on policy gradients is adopted, and the update direction is represented as follows: in, This represents the policy objective function. Represents the distribution of context state samples. This indicates that the context state is sampled from historical tasks or online interactions. Indicates the state in a given context Under the condition that the action probability distribution output by the policy function is used to sample and select actions; In a single interaction sample, the context state is represented Select action Execute the corresponding tool Later received a reward The following parameter update method will be used: in, Indicates the learning rate. This represents the baseline function.

8. The tool selection optimization method based on error-aware reward according to claim 7, characterized in that, To improve the robustness of policy updates and sample utilization efficiency, multiple interaction samples are grouped into training batches for joint optimization. Specifically: Assuming that within an update cycle, the following data is collected: Interaction Samples ,in, This indicates the number of interaction samples in an update cycle; Indicates the first The context state representation corresponding to each interaction sample; Indicates the first The selected tool selection action in each interactive sample; Indicates the first The total error-perceived reward for each interaction sample. ; The batch optimization objective is represented as: in, Let be the policy loss function. For the first The baseline value corresponding to each sample; during the optimization process, the strategy parameters are optimized by minimizing the batch optimization objective. Iterative updates.

9. The tool selection optimization method based on error-aware reward according to claim 8, characterized in that, To prevent the policy from getting stuck in local optima during training, while balancing decision stability and exploration ability, the decision-making behavior of the policy function is adjusted: Random sampling is performed based on the policy distribution, i.e., from The tool action is selected based on probability, thus preserving the priority of high-probability tools while introducing a certain degree of random exploration capability. The strategy distribution is as follows: in, For the exploration rate parameter, ; Representing the action space Any candidate action in the list; Indicates from action space Randomly select an action.

10. The tool selection optimization method based on error-aware reward according to claim 9, characterized in that, To avoid performance degradation of policy functions due to environmental drift, an online adaptive update mechanism is introduced to incrementally adjust and dynamically correct the policy. For each interaction record generated during system operation, construct an online feedback sample: in, Indicates time The context state representation, Indicates the selected tool action. This represents the total reward value for error perception. Indicates the corresponding feedback signal error type; continuously writes feedback samples into the online experience cache pool. middle: A sliding window approach is used to select interaction samples from the most recent period as the basis for policy updates; let the window length be... Then at time The local empirical set is defined as: Based on local experience sets The system periodically or triggered the adjustment of strategy parameters. Incremental updates are performed, with the goal of maximizing the expected reward under the current environmental distribution, i.e.: The update format is represented as: in, Indicates the learning rate; Add parameter regularization terms to the online optimization objective: in, This represents the parameters of the offline pre-trained or previously stabilized policy. Represents the regularization coefficient; To avoid the overhead of frequent updates, trigger-based update conditions are set to automatically initiate online adaptive updates when any of the following conditions are detected: ① When the average reward obtained by the system in the most recent tool calls or the most recent time window exceeds the preset range compared to the historical average reward or the preset benchmark reward; ② The frequency of a certain error type exceeds the preset threshold corresponding to the error type; ③ The candidate toolset has been added, deleted, or its interface has been changed; ④ The tool call success rate, timeout rate, or semantic mismatch rate exceeds the preset threshold compared to the historical reference average.

Images