Similarity-sensitive reward-based large language model tool selection method and system

Abstract

The application discloses a large language model tool selection method and system based on similarity sensitive reward, and the method comprises the following steps: obtaining a training sample containing a query text, a candidate tool set and a real tool set; inputting the query and the candidate into a strategy model to generate a predicted tool set; calculating a comprehensive similarity matrix among the candidate tools; regarding the non-real tools as interference, determining the difficulty weight of the real tools and the interference according to the maximum similarity between the real tools and the interference; calculating the similarity sensitive reward value based on the weighted precision and recall of the correct predicted tool; and updating the strategy model parameters by using the reward value. The application makes the reward signal reflect the similarity between the real tools and the interference tools, guides the model to focus on the subtle differences among the high similarity tools, and effectively improves the selection accuracy of the large language model in the similar tool scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for selecting large language model tools based on similarity-sensitive rewards. Background Technology

[0002] With the development of large language model technology, large language models have demonstrated strong capabilities in natural language understanding, text generation, and reasoning analysis. To further expand the application scope of large language models in complex real-world tasks, tool learning technology has gradually gained attention. Tool learning refers to enabling large language models to proactively select and invoke external tools based on user queries, thereby leveraging these tools to complete tasks such as web searches, code execution, mathematical calculations, weather inquiries, and information retrieval, thus compensating for the model's own shortcomings in knowledge timeliness, computational accuracy, or execution capabilities.

[0003] In existing large language model tool usage workflows, tool selection typically involves two stages: candidate tool retrieval and candidate tool selection. The candidate tool retrieval stage primarily involves filtering out several candidate tools relevant to the user's query from the tool library. The candidate tool selection stage, based on the user query and candidate tool information, further selects one or more tools truly suitable for performing the current task from the candidate tool set. As the tool library grows, tools may exhibit high similarity in name, functional description, and parameter settings, resulting in a candidate tool set often containing multiple tools with similar functions but not entirely identical application scenarios. Therefore, in the candidate tool selection stage, the large language model not only needs to determine whether a tool should be used but also needs to finely differentiate between multiple similar tools.

[0004] In existing technologies, some studies have improved the tool selection problem from two aspects: evaluation benchmarks and model training. For example, some evaluation benchmarks construct similar tool selection tasks to test the ability of large language models to distinguish between functionally similar tools. These benchmarks typically calculate the similarity between tools based on the semantic vectors of tool descriptions and select tools that are semantically similar to the real tools as distractors, thus forming a set of candidate tools. This approach can, to some extent, evaluate the model's selection performance when faced with similar tools.

[0005] In addition, existing technologies employ reinforcement learning methods to enhance the tool-use capabilities of large language models. These methods typically design reward functions to score the tool-calling results generated by the model and update the model parameters based on the scores. Compared to binary reward methods that only judge whether a tool call is completely correct, fine-grained reward functions can evaluate the model output from dimensions such as tool name, parameter name, and parameter content, thus providing richer feedback signals for model training.

[0006] However, existing similarity tool selection tasks typically only select a few tools most similar to the real tool as distractors, without further distinguishing the similarity levels between candidate tools. This makes it difficult to comprehensively evaluate the model's tool selection capabilities in different scenarios such as low, medium, and high similarity. Furthermore, existing similarity tool selection tasks are mostly focused on single-tool selection scenarios, while real-world user tasks often require a combination of multiple tools. For example, first calling a weather query tool to get weather information, and then calling a recommendation tool to generate travel or clothing suggestions. Therefore, existing evaluation methods cannot fully reflect the problem of similar tool confusion in multi-tool selection scenarios.

[0007] Furthermore, while existing reinforcement learning reward functions can provide fine-grained evaluation of tool invocation results, they primarily focus on overall invocation accuracy and do not fully consider the similarity between candidate tools. When the model correctly selects the real tool from among highly similar candidate tools, the difficulty of differentiation is significantly higher than when selecting the real tool from among significantly different candidate tools. However, existing reward mechanisms typically award the same or similar rewards to both types of correct selections, failing to effectively highlight the training value of correct selection in scenarios with highly similar tools. This leads to the model still being prone to misselection when faced with tools that have similar functional descriptions, parameter structures, or applicability ranges. Summary of the Invention

[0008] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the training reward signal of the tool selection model in the prior art fails to reflect the confusion difficulty between similar tools, which makes it difficult for large language models to accurately distinguish between real tools and interference tools in a set of highly similar candidate tools.

[0009] To address the aforementioned technical problems, this invention provides a method for selecting large language model tools based on similarity-sensitive rewards, comprising the following steps: S1: Obtain training samples, including query text, candidate tool set, and real tool set, wherein the real tool set is a subset of the candidate tool set; S2: Input the query text and the candidate tool set into the strategy model to generate a prediction tool set; S3: Calculate the comprehensive similarity between every two tools in the candidate tool set to obtain the comprehensive similarity matrix; S4: Tools in the candidate tool set that do not belong to the real tool set are identified as interference tools; based on the comprehensive similarity matrix, the maximum value of the comprehensive similarity between the real tool and all interference tools is determined as the difficulty weight of the real tool. S5: Determine the intersection of the predicted tool set and the real tool set as the real tool that was correctly predicted; calculate the similarity-sensitive reward value based on the difficulty weight of the real tool that was correctly predicted; S6: Update the parameters of the strategy model using the similarity-sensitive reward value to improve the strategy model's ability to select among similar tools.

[0010] In one embodiment of the present invention, in step S4, the method for determining the maximum value of the comprehensive similarity between the real tool and all interfering tools, based on the comprehensive similarity matrix, as the difficulty weight of the real tool is as follows: For each real tool in the set of real tools, the comprehensive similarity value between the real tool and each of the interference tools is obtained from the comprehensive similarity matrix, and the maximum value among the obtained comprehensive similarity values ​​is used as the difficulty weight of the real tool.

[0011] In one embodiment of the present invention, the method for calculating the similarity-sensitive reward value based on the difficulty weight of the correctly predicted real tool in step S5 is as follows: The weighted accuracy is calculated based on the sum of the difficulty weights of the correctly predicted real tools and the total number of tools in the predicted tool set. The weighted recall rate is calculated based on the sum of the difficulty weights of the correctly predicted real tools and the total number of tools in the set of real tools. The similarity-sensitive reward value is calculated based on the harmonic mean of the weighted precision and the weighted recall.

[0012] In one embodiment of the present invention, the method for calculating the similarity-sensitive reward value is as follows: Calculate weighted precision and weighted recall : , , in, The set of prediction tools, For the aforementioned set of real tools, For real tools Difficulty weighting Indicates the number of elements in the set; Calculate the weighted accuracy With the weighted recall rate The harmonic mean is used as the similarity-sensitive reward value. : .

[0013] In one embodiment of the present invention, the method for calculating the comprehensive similarity between every two tools in the candidate tool set and obtaining the comprehensive similarity matrix in step S3 is as follows: The name, function description, and parameter list of each tool are converted into vector representations, and the semantic similarity between the two vectors is calculated using the cosine similarity formula. For any two tools in the candidate tool set, calculate the semantic similarity between the two tools in the three parts of the text: name, function description, and parameter list. The average of the three semantic similarities is taken as the comprehensive similarity between the two tools. The comprehensive similarity matrix is ​​constructed based on the comprehensive similarity between all pairs of tools in the candidate tool set.

[0014] In one embodiment of the present invention, the method for obtaining training samples in step S1 is as follows: Obtain the query text and the actual set of tools corresponding to the query text; Use the real toolset as the initial element of the candidate toolset; Calculate the matching similarity between the query text and each other tool in the tool library. The matching similarity is obtained by averaging the cosine similarity between the embedding vector of the query text and the embedding vector of the name, function description, and parameter list of the other tool, respectively. Sort the tools by matching similarity from high to low, and select the top-ranked tools to add to the candidate tool set until the number of tools in the candidate tool set reaches a preset number. The training sample is composed of the query text, the candidate tool set, and the real tool set.

[0015] In one embodiment of the present invention, the method for generating a prediction tool set by inputting the query text and the candidate tool set into the strategy model in step S2 is as follows: The query text, along with the name, function description, and tool parameters of each tool in the candidate tool set, are concatenated according to a preset prompt word template to construct input prompt information; the preset prompt word template contains instructions for instructing the strategy model to select a suitable tool from the candidate tool set; The input prompt information is input into the strategy model, and the model outputs the corresponding tool name and corresponding filling parameters. The set of tools corresponding to all the output tool names is taken as the prediction tool set.

[0016] In one embodiment of the present invention, the method for updating the parameters of the policy model using the similarity-sensitive reward value in step S6 is as follows: A group-relative strategy optimization algorithm is used to generate multiple sets of prediction tools for the same query text, forming a response group; Calculate the similarity-sensitive reward value for each set of prediction tools in the response group; Based on the similarity-sensitive reward value of each prediction tool set within the response group, calculate the advantage function value corresponding to each prediction tool set; The parameters of the policy model are updated using gradient descent, with the objective of minimizing the loss function that includes the advantage function value and the probability ratio pruning mechanism.

[0017] In one embodiment of the present invention, the policy model employs a pre-trained large language model based on the Transformer architecture.

[0018] Based on the same inventive concept, this invention also provides a large language model tool selection system based on similarity-sensitive rewards, comprising: The training sample acquisition module is used to acquire training samples, including query text, candidate tool set and real tool set, wherein the real tool set is a subset of the candidate tool set; The prediction generation module is used to input the query text and the candidate tool set into the strategy model to generate a prediction tool set; The similarity calculation module is used to calculate the comprehensive similarity between every two tools in the candidate tool set, and obtain a comprehensive similarity matrix; The difficulty weight determination module is used to identify tools in the candidate tool set that do not belong to the real tool set as interference tools; and to determine the maximum value of the comprehensive similarity between the real tool and all interference tools according to the comprehensive similarity matrix, which is used as the difficulty weight of the real tool. The reward calculation module is used to determine the intersection of the predicted tool set and the real tool set as the real tool that was correctly predicted; and to calculate the similarity-sensitive reward value based on the difficulty weight of the real tool that was correctly predicted. The strategy update module is used to update the parameters of the strategy model using the similarity-sensitive reward value, thereby improving the strategy model's ability to select among similar tools.

[0019] The technical solution of the present invention has the following advantages over the prior art: The large language model tool selection method and system based on similarity-sensitive rewards described in this invention calculates the comprehensive similarity between tools in a candidate tool set, determines the difficulty weight based on the similarity between real and interfering tools, and then incorporates this difficulty weight into the reward value calculation process. This ensures that the reward signal during model training reflects the confusion difficulty between similar tools. Compared to existing technologies that award the same reward for correct selections of different difficulties, this invention guides the policy model to focus on learning the differences between highly similar tools, improving the model's ability to distinguish between candidate tools with similar functions, descriptions, or parameter structures, thereby reducing the probability of misselection and improving the accuracy and stability of tool selection in large language models. Attached Figure Description

[0020] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0021] Figure 1 This is a flowchart illustrating the tool selection method for large language models based on similarity-sensitive rewards in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0023] Example 1: like Figure 1 As shown, this invention provides a method for selecting tools for large language models based on similarity-sensitive rewards, comprising the following steps: S1: Obtain training samples, including query text, candidate tool set, and real tool set, wherein the real tool set is a subset of the candidate tool set; S2: Input the query text and the candidate tool set into the strategy model to generate a prediction tool set; S3: Calculate the comprehensive similarity between every two tools in the candidate tool set to obtain the comprehensive similarity matrix; S4: Tools in the candidate tool set that do not belong to the real tool set are identified as interference tools; based on the comprehensive similarity matrix, the maximum value of the comprehensive similarity between the real tool and all interference tools is determined as the difficulty weight of the real tool. S5: Determine the intersection of the predicted tool set and the real tool set as the real tool that was correctly predicted; calculate the similarity-sensitive reward value based on the difficulty weight of the real tool that was correctly predicted; S6: Update the parameters of the strategy model using the similarity-sensitive reward value to improve the strategy model's ability to select among similar tools.

[0024] This invention acquires training samples including query text, a candidate tool set, and a real tool set, where the real tool set is a subset of the candidate tool set. The query text and candidate tool set are input into a policy model, which generates a predicted tool set. Further, the comprehensive similarity between each pair of tools in the candidate tool set is calculated to obtain a comprehensive similarity matrix, and tools in the candidate tool set that do not belong to the real tool set are identified as interference tools. For each real tool, the maximum comprehensive similarity between the real tool and each interference tool is determined based on the comprehensive similarity matrix, and this maximum value is used as the difficulty weight of the real tool. The intersection of the predicted tool set and the real tool set is determined as the correctly predicted real tool, and a similarity-sensitive reward value is calculated based on the difficulty weight of the correctly predicted real tool. Finally, the parameters of the policy model are updated using this similarity-sensitive reward value. Through this method, this invention enables the reward signal during model training to reflect the similarity and confusion difficulty between real tools and interference tools, guiding the policy model to focus on learning the differences between highly similar tools, thereby improving the discrimination ability and tool selection accuracy of large language models in similar tool scenarios.

[0025] Preferably, in step S1, the training samples include query text, candidate tool set, and real tool set. The real tool set is a subset of the candidate tool set, and the interfering tools in the candidate tool set are all selected by similarity matching with the query text, ensuring that the training samples can truly reflect the complexity and deceptiveness of tool selection in the actual application environment.

[0026] Specifically, a large number of query texts are obtained from the sample library, along with the real toolkit corresponding to each query text.

[0027] The actual toolset refers to the set of one or more correct tools that are actually needed to complete the query task. This set also serves as the standard answer for judging the accuracy of the model's predictions in subsequent steps. Simultaneously, a tool library containing a vast number of available tools is required. Each tool in this library is described with structured information, including at least three fields: tool name, function description, and parameter list. The tool name concisely and clearly summarizes the tool's purpose, such as "get hourly weather trends" or "delete files." The function description details the specific operations the tool can perform, its applicable scenarios, and limitations. The parameter list records the names, types, mandatory / unmandable parameters, and meanings of each parameter required to call the tool. This raw, structured tool information forms the foundation for constructing the training samples. To ensure the diversity of training samples, the query text should cover both single-tool call scenarios and multi-tool combination call scenarios. The former, such as "check today's weather", only requires calling a weather query tool; the latter, such as "check today's weather and recommend suitable outfits", requires the model to call the weather query tool and the outfit recommendation tool in sequence. These scenarios better test the model's tool selection ability in complex multi-step tasks.

[0028] When constructing a candidate tool set for a specific query text, all tools in the real tool set corresponding to the query text are used as the initial elements of the candidate tool set. This operation ensures that the correct tools must exist in the candidate tool set, avoiding training signal distortion caused by the absence of correct tools.

[0029] Furthermore, it is necessary to introduce several other tools from the tool library that are highly relevant to the query text but are not the correct answer as distractors. The existence of these distractors is crucial for training the model to distinguish similar tools. To this end, a pre-trained text embedding model is used to transform the query text and tool information into dense vector representations, respectively, so as to perform similarity measurement calculations in a unified semantic space. The aforementioned text embedding model can be a lightweight model based on the Sentence-BERT architecture, which can map input text of arbitrary length to a dense vector of fixed dimensions while maintaining semantic similarity.

[0030] Specifically, for the current query text The text is input into the text embedding model, and after word segmentation, Transformer encoding, and average pooling, the corresponding query embedding vector is obtained. For every other tool in the tool library besides the actual set of tools. This requires extracting textual information from three dimensions: tool name, function description, and parameter string. The parameter string is constructed by iterating through each parameter in the tool's parameter list, concatenating the parameter's name, type, and textual description into a complete string, with parameters separated by delimiters. This creates a structured text that comprehensively reflects the tool's parameter characteristics.

[0031] Furthermore, this tool The tool name, function description, and parameter string are each input into the same text embedding model, which sequentially produces the tool's name embedding vector. Functional description embedding vector and parameter list embedding vector These three vectors characterize the tool's position in the semantic space from different perspectives: the name vector emphasizes concise identification, the function description vector emphasizes application scenarios and operational semantics, and the parameter vector emphasizes the structured features of the API call.

[0032] After obtaining the above vector representations, the semantic similarity between the query text and each other tool in three dimensions is calculated using the cosine similarity formula. For any two vectors... and The cosine similarity between the two is defined as: , in, Representing vectors with vector dot product, and These represent the magnitudes of the two vectors, respectively.

[0033] Based on this formula, calculate the query embedding vector respectively. With tools Name embedding vector Name similarity and functional description embedding vectors Functional similarity between them, and embedding vectors of parameter lists The similarity of parameters between them.

[0034] Furthermore, the arithmetic mean of these three similarity values ​​is taken, and the result is the comparison between the query text and the other tools. The similarity between the three aspects is calculated by taking the average of the three aspects. This method ensures that the similarity can comprehensively and evenly reflect the overall degree of matching between the tool and the user query in terms of name identification, function purpose and parameter structure, which are the three key semantic levels. This avoids the one-sidedness that may result from relying on only a single dimension.

[0035] After calculating the matching similarity between the current query text and all other tools in the tool library using the method described above, these similarity values ​​are sorted in descending order. The sorting results reflect the semantic relevance of each tool to the query. The higher the matching similarity, the easier it is for the model to mistakenly identify the tool's functional description or parameter configuration as relevant to the current query, and therefore, the higher its training value as a distractor. Other tools with higher rankings are added to the candidate tool set that was previously initialized with the real tool set, until the total number of tools in the candidate tool set reaches a preset number, thus forming a final candidate tool set of fixed size. The preset number can be flexibly adjusted according to the overall size of the tool library, the convergence efficiency of model training, and the limitations of training computing power. The principle is to ensure that the candidate tool set contains enough distractors to create sufficient challenge, but not so large that the model training difficulty or computational cost becomes too high.

[0036] The candidate tool set constructed in this way contains both the truly correct tools to be selected and distractors that are highly semantically related to the query, have similar functional descriptions and parameter structures, but do not truly meet the user's needs. Furthermore, these distractors are distributed in a tiered manner according to their matching degree with the query text, effectively simulating the confusion that can easily occur when large language models face a vast tool library in real-world environments. Finally, the query text, the candidate tool set constructed in the above steps, and the corresponding real tool set are combined to form a complete training sample. By iterating through all original query texts and repeating the above process one by one, a complete training sample set for subsequent policy model training can be constructed in batches.

[0037] In a preferred embodiment, two public datasets, UltraTool and Seal-Tools, are involved. Training samples are constructed based on the training set of the UltraTool dataset, while test samples are subsequently constructed using test sets from both UltraTool and Seal-Tools. UltraTool provides a toolset covering 22 domains, containing 2032 tools, with task samples derived from complex real-world queries across multiple domains. Seal-Tools, on the other hand, is constructed using a self-guided method, generating 4076 tools and corresponding task instances. Both datasets offer a large number of tools and diverse task scenarios ranging from single-tool calls to multi-tool combinations. To evaluate the impact of similar tools on model capabilities, complex samples involving nested tool calls can be removed during the construction process. The text embedding model can be "all-MiniLM-L6-v2," a lightweight model in the Sentence-BERT series with approximately 22M parameters. It maps input text to a dense 384-dimensional vector, maintaining semantic discriminability while achieving high computational efficiency. The training set built based on this embodiment contains thousands of training instances. Each instance consists of three parts: query text, a set of candidate tools containing a preset number of tools, and a set of real tools. This provides a rich and targeted data foundation, which helps to improve the model's ability to distinguish between similar tools and the accuracy of tool selection.

[0038] Further, in step S2, the query text and candidate tool set are input into the strategy model to generate a prediction tool set.

[0039] Preferably, the strategy model adopts a pre-trained large language model based on the Transformer architecture. This architecture can effectively capture long-distance dependencies between any positions in the input sequence through multi-head self-attention mechanism and stacking of feedforward neural network layers, so as to fully understand the deep semantic association between query intent and the functions of each candidate tool after given query text and candidate tool information.

[0040] Specifically, the query text, along with the name, function description, and parameters of each tool in the candidate tool set, are concatenated according to a preset prompt word template to construct input prompt information. The preset prompt word template contains instruction text that instructs the strategy model to select a suitable tool from the candidate tool set; for example, it may instruct the model to provide the selected tool name and corresponding fill parameters based on the user's query. After the constructed input prompt information is input into the strategy model, the strategy model outputs the corresponding tool name and corresponding fill parameters. The set of tools corresponding to all output tool names is taken as the prediction tool set.

[0041] In a preferred embodiment, the policy model can be based on the open-source "Qwen-2.5-7B-Instruct", which has strong instruction compliance and context understanding capabilities and can effectively perform the task of selecting multiple tools from a set of candidate tools.

[0042] Further, in step S3, the comprehensive similarity between every two tools in the candidate tool set is calculated, and a comprehensive similarity matrix is ​​constructed. The calculation method in step S3 needs to be consistent with the calculation method of the matching similarity between the query and the tool in step S1 at the basic level of vectorization and similarity measurement, but the calculation objects and application purposes are different.

[0043] Specifically, for each tool in the candidate tool set, its tool name, function description, and parameter list are input into a text embedding model and converted into corresponding vector representations. The parameter list is processed in the same way as in step S1, that is, the name, parameter type, and description of each parameter in the tool's parameter list are concatenated to form a parameter string before being input into the text embedding model. For any two different tools in the candidate tool set, the semantic similarity between their name vectors, between their function description vectors, and between their parameter list vectors are calculated. The semantic similarity in each dimension is calculated using the cosine similarity formula, which is also... .

[0044] Furthermore, the arithmetic mean of the semantic similarity values ​​across these three dimensions is taken to obtain the overall similarity between the two tools. This overall similarity reflects the degree of semantic closeness between the two tools across multiple dimensions such as name identification, function usage, and API. A higher value indicates a greater likelihood that the two tools will be confused by the model. After calculating the overall similarity between all pairs of tools in the candidate tool set, these similarity values ​​are organized according to tool pairs to construct a complete overall similarity matrix. This matrix is ​​a symmetric matrix, where each element corresponds to the overall similarity value of a pair of tools. The diagonal elements represent the similarity between a tool and itself, which can be defined as 1 or not used.

[0045] The comprehensive similarity matrix constructed in the above manner provides a direct query basis for determining the maximum comprehensive similarity between each real tool and all interfering tools.

[0046] Further, in step S4, tools in the candidate tool set that do not belong to the real tool set are identified as interference tools; based on the comprehensive similarity matrix, the maximum value of the comprehensive similarity between the real tool and all interference tools is determined as the difficulty weight of the real tool.

[0047] Specifically, all tools in the candidate tool set that do not belong to the real tool set are identified as interference tools, that is, for the candidate tool set... and real tool collection The set of interference tools is the difference between the two. Distractor tools are semantically related to the query text, and their functional descriptions or parameter structures may be similar to the real tools to varying degrees. However, they are not the tools required to complete the current query task. Due to the existence of such distractor tools, the strategy model will face varying degrees of difficulty in selecting the real tools from the candidate tool set.

[0048] After determining the set of interference tools, the real set of tools is established. Each real tool in the set has a difficulty weight calculated to quantify the difficulty of correctly identifying it. The core idea is that the more easily a real tool is confused by a distractor tool in the candidate tool set, the greater the difficulty of correctly selecting that real tool, and correspondingly, the higher its weight should be in the reward calculation. Therefore, for the set of real tools... Every real tool in From the comprehensive similarity matrix obtained in step S3, the real tool is obtained. With each interference tool (in The overall similarity value between ) This comprehensive similarity score reflects the actual tool. With this jamming tool The overall semantic similarity across multiple dimensions, including name, function description, and parameter list, is considered. A higher numerical value indicates a greater likelihood that the two models will be confused.

[0049] Furthermore, from the real tools acquired The maximum value among all the corresponding comprehensive similarity scores is selected as the real tool. The difficulty weight is expressed by the formula: , in, Representation of real tools Difficulty weighting Represents the set of candidate tools. Represents a real set of tools. This represents a set of interference tools. Representation of real tools Interference tools The overall similarity between them. The maximum value is used here instead of the average or other statistics because the risk of the real tool being confused depends primarily on the difficulty of distinguishing it from the most similar interfering tool, rather than the average similarity with all interfering tools. Since the presence of even one interfering tool highly similar to the real tool in the candidate tool set can lead to the model misselecting or omitting that real tool, taking the maximum value accurately reflects the degree to which the real tool is most easily confused in the current candidate tool set.

[0050] Furthermore, in step S5, a similarity-sensitive reward value is calculated based on the difficulty weights of each real tool obtained in step S4. Integrating the difficulty weights into the F1 score calculation framework ensures that the reward signal not only reflects the accuracy of the model's prediction results but also the confusion difficulty of the correctly predicted tool in the current candidate environment, thus guiding the model to focus on the differences between highly similar tools.

[0051] Specifically, identify the true tools correctly predicted by the policy model. This involves the set of predicted tools generated by the policy model in step S2. With real toolset intersection The set of true tools that are correctly predicted is the intersection of tools that belong to the true tool set and also appear in the model's prediction output, indicating that the model has successfully identified and selected them from the current set of candidate tools.

[0052] Furthermore, based on the set of real tools that were correctly predicted, the sum of their difficulty weights is incorporated into the calculation of precision and recall, respectively, to obtain similarity-weighted precision and similarity-weighted recall.

[0053] Similarity-weighted precision The calculation method is as follows: the sum of the difficulty weights of all correctly predicted real tools is used as the numerator, and the set of predicted tools is used as the numerator. Total number of tools As the denominator, its formula is expressed as: , in, A collection of prediction tools, A collection of real tools. For real tools Difficulty weighting Represents a set The number of elements in the model's predictions. This weighted precision measures the proportion of correctly selected tools out of all tools predicted by the model, with each correctly selected tool weighted according to its difficulty of differentiation. If the model successfully selects a true tool with a higher difficulty weight in the presence of highly similar interfering tools, the contribution of that true tool to the weighted precision will increase accordingly; conversely, if the selected true tool is significantly different from the interfering tools and easily distinguishable, its difficulty weight will be lower, and its contribution to the weighted precision will be smaller.

[0054] Similarity-weighted recall The formula for calculating it is expressed as follows: , in, Represents the real toolset The number of elements in the model. This weighted recall rate measures the proportion of tools that the model successfully selects out of all real tools that need to be selected, and is also weighted according to the difficulty of distinguishing each real tool. If a real tool with a higher difficulty weight is successfully recalled by the model, it contributes more to the weighted recall rate; if a real tool is missed, it will not contribute to the recall rate regardless of its difficulty weight.

[0055] To obtain similarity-weighted accuracy Similarity-weighted recall Next, the harmonic mean of the two is calculated to obtain the final similarity-sensitive reward value. Its formula is expressed as: , The reason for using the harmonic mean instead of the arithmetic mean is that the harmonic mean is more sensitive to the balance between precision and recall. The reward value will only reach a high level when the model performs well in both precision and recall; if the model tends to output too many tools (leading to low precision) or too few tools (leading to low recall), the reward value will decrease accordingly. This design better meets the need for a comprehensive evaluation of the model in scenarios involving multiple tool selections.

[0056] The similarity-sensitive reward value calculated using the above method has a significant advantage over traditional binary rewards or ordinary F1 rewards. This reward value not only reflects the degree of matching between the predicted tool set and the real tool set, but also, by incorporating difficulty weights, comprehensively reflects the confusion difficulty of the correctly predicted tool in the current candidate environment. When the model correctly selects a real tool in the presence of highly similar interfering tools, the contribution of that real tool to the reward value is correspondingly increased due to its higher difficulty weight; conversely, when the model only correctly selects low-difficulty real tools that are significantly different from the interfering tools, its contribution to the reward value is relatively limited. This differentiated reward mechanism helps guide the policy model to focus on learning the subtle differences between similar tools during training, gradually improving its tool differentiation ability in high-similarity scenarios, and thus enhancing the overall accuracy and robustness of tool selection in large language models.

[0057] Furthermore, in step S6, the parameters of the strategy model are updated using the similarity-sensitive reward value calculated in step S5, so as to improve the strategy model's ability to select among similar tools.

[0058] In this embodiment of the invention, the Group Relative Policy Optimization (GRPO) algorithm framework is preferably adopted. The core idea is that for each query text, multiple responses are sampled from the current policy model to form a response group, and the relative advantage value is calculated within the group. This replaces the Critic network in the traditional Proximal Policy Optimization (PPO) algorithm, effectively reducing the computational complexity and memory overhead during training. Within this framework, the invention introduces the aforementioned similarity-sensitive reward function to guide the model to focus on the differences between highly similar tools during training.

[0059] Specifically, for the same query text, the current strategy model generates multiple independent sets of prediction tools, which together constitute a response group. Let the number of prediction tool sets contained in the response group be denoted as . Each set of prediction tools corresponds to a complete autoregressive generation result of the strategy model for that query text, i.e., a response. ( Each response Each response includes several tool names and corresponding filler parameters, representing one or more tools selected by the policy model from the candidate tool set. By generating multiple responses for the same query text, the relative merits of different responses can be compared in the subsequent policy optimization phase, thus providing more comparative training signals for updating model parameters.

[0060] After obtaining the response group, for each response in the response group Following the methods described in steps S4 and S5, calculate the corresponding similarity-sensitive reward values ​​respectively. .

[0061] Specifically, regarding the response The corresponding set of prediction tools Identify candidate tools that do not belong to the real tool set. For the interference tools, the comprehensive similarity value between each real tool and each interference tool is read from the comprehensive similarity matrix obtained in step S3, and the maximum value is taken as the difficulty weight of each real tool. Furthermore, the set of real tools that will be correctly predicted. The sum of the difficulty weights is incorporated into the calculation of precision and recall, respectively, to obtain the similarity-weighted precision. Similarity-weighted recall Furthermore, the harmonic mean of the two is calculated to obtain the response. Similarity-sensitive reward value Therefore, each response in the response group receives a reward value that comprehensively reflects its predictive accuracy and the difficulty of obfuscation by the selected real instrument.

[0062] After obtaining the similarity-sensitive reward values ​​for each response within a response group, the dominance function value for each response is further calculated based on its reward value. The dominance function measures the relative quality of a response within a group compared to the group's average level. Responses with rewards higher than the group's average receive a positive advantage, while responses with rewards lower than the group's average receive a negative advantage. In the presence of highly similarity-based interference tools, responses that correctly select the real tool will receive higher rewards than incorrect responses, thus making it easier to obtain a positive advantage during group standardization. Selecting an incorrect response, on the other hand, results in a negative advantage. In a preferred embodiment, for a response... Its standardization advantages Based on the reward value of the response Average reward within the response group Standard deviation The calculation yields the following formula: , in, , , The size of the corresponding group is represented by the standardization process. By eliminating the influence of the absolute value of the reward through standardization, the advantage value of each response can more stably reflect the relative performance ranking of each response within the group, thus providing a standardized learning signal for subsequent policy gradient updates.

[0063] Based on this, the parameters of the policy model are updated by minimizing the loss function that includes the advantage function value and the probability ratio pruning mechanism. This process is equivalent to maximizing the proxy objective function that includes the advantage function value and the probability ratio pruning mechanism.

[0064] Specifically, let the parameters of the current strategy model to be optimized be... The old strategy parameters used during sampling are: Regarding the response The first in Each token defines an importance weight ratio. This is the ratio of the probability of generating this token under the current strategy to the probability of generating this token under the old strategy, i.e.: , in, This indicates the input suggestion information consisting of the query text and the set of candidate tools. Indicates a response Center front A sequence of tokens This represents the conditional probability distribution of the policy model in generating the next token given the context. To prevent excessive policy shifts during updates, a probability ratio pruning mechanism is introduced, with a pruning hyperparameter set. (For example ), limiting the importance weight ratio to Within the interval. The loss function is defined as: , in, Indicates a response The length of the token sequence, This indicates the weighting ratio of importance. Cut to Within the interval, This indicates the operation of taking the smaller of the two values. The design of this loss function ensures that when a response has a positive advantage (…), When the probability ratio is within a certain range, the model tends to increase the probability of token generation in that response, but the increase is limited to a certain limit. Within; when a response has a negative advantage ( When the probability ratio decreases, the model tends to reduce the probability of token generation in that response, but the decrease is limited to a certain extent. Within this range. In this way, the policy model can gradually increase the probability of generating high-reward responses and suppress the probability of generating low-reward responses with each update, while maintaining training stability, thereby enhancing the model's overall ability to make correct selections in scenarios with high similarity tools.

[0065] In practice, the parameters of the strategy model are updated using gradient descent. In each iteration, a batch of query texts and their corresponding candidate tool sets and real tool sets are sampled from the training sample set. For each query text, a set containing... For each response group, the similarity-sensitive reward value and standardized advantage of each response are calculated according to the steps described above, and then the loss function is calculated. Regarding model parameters The gradient is calculated, and the parameters are updated along the negative gradient direction. ,in The learning rate is used. Through multiple rounds of iterative training, the policy model gradually learns to accurately identify and select the real tool even in the presence of highly similar interfering tools, thus significantly improving its ability to select among similar tools.

[0066] Example 2: To verify the effectiveness of the large language model tool selection method based on similarity-sensitive rewards proposed in this invention, the model was trained based on the training samples constructed in Example 1, and corresponding system evaluation experiments were carried out.

[0067] In this embodiment of the invention, the open-source large language model "Qwen-2.5-7B-Instruct" is used as the basic policy model. This model possesses strong instruction compliance and contextual understanding capabilities, enabling it to effectively perform the task of selecting multiple tools from a candidate tool set. The training process employs the Group Relative Policy Optimization (GRPO) algorithm framework, replacing the Critic network in the traditional PPO algorithm, effectively reducing computational complexity and memory overhead during training. This invention introduces a similarity-sensitive reward function within this framework to guide the model to focus on the differences between highly similar tools during training.

[0068] In the specific implementation of training, for each training sample, sample data containing query text, candidate tool set, and real tool set is constructed as prompt words and input into the current policy model. The model generates a response group containing multiple independent responses for each query text. In this embodiment, the number of responses generated for each query text is set to 8, and each response is a tool selection sequence predicted by the model. Subsequently, the similarity-sensitive reward value corresponding to each response in the response group is calculated. The calculation process strictly follows the method described in Embodiment 1: tools in the candidate tool set that do not belong to the real tool set are identified as interference tools. The comprehensive similarity value between each real tool and each interference tool is read from the pre-calculated comprehensive similarity matrix, and the maximum value is taken as the difficulty weight of the real tool. This difficulty weight is integrated into the F1 score calculation framework. The sum of the difficulty weights of the correctly predicted real tools is included in the calculation of weighted precision and weighted recall, and the harmonic mean of the two is used as the similarity-sensitive reward value of the response.

[0069] To enhance understanding of the invention, a specific example of reward calculation is provided here: In a typical file operation scenario, a user query is "I have a file, I want to modify its content, and then I want to delete the file." The candidate tool set includes real tool A "delete file," real tool B "file modification," distractor tool C "delete specific content in file," and distractor tool D "add content to the beginning of file." Based on the calculation of the comprehensive similarity between tools, the similarity between tool A and tool C is 0.60, and the similarity between tool A and tool D is 0.40. The similarity between tool B and tool C is 0.36, and the similarity between tool B and tool D is 0.56. Therefore, the difficulty weight of real tool A is 0.60, and the difficulty weight of real tool B is 0.56. Suppose the model outputs both tool A and tool D in a prediction, meaning it correctly selected only tool A. In this case, the sum of the difficulty weights for the correctly selected tool is 0.60, the number of true tools and predicted tools are both 2, the weighted precision and weighted recall are both 0.30, and the final similarity-sensitive reward is 0.30. In contrast, if the model correctly outputs both tool A and tool B, the sum of the difficulty weights for this response is 1.16, the weighted precision and weighted recall are both 0.58, and the similarity-sensitive reward is 0.58, significantly higher than a partially correct response. This design allows the reward value to more precisely reflect the actual performance differences of the model in scenarios with similar tools.

[0070] After obtaining the similarity-sensitive reward values ​​for each response within a response group, a standardized dominance function is calculated for each response based on the group reward values. For the response... Its standardization advantages Based on the reward value of the response Average reward within the response group Standard deviation Calculated, i.e. ,in , Through standardization, the advantage value of each response can more stably reflect its relative performance ranking within the group. Based on this, updates are performed by minimizing a loss function that incorporates the advantage function value and a probability ratio pruning mechanism. This process is equivalent to maximizing a surrogate objective function that incorporates both the advantage function value and the probability ratio pruning mechanism. The loss function is defined as follows: ,in This indicates that the current strategy and the old strategy are compared at the 1st and 2nd time. The first response The probability ratio on each token To prune hyperparameters, the model parameters are updated along the negative gradient of the loss function using gradient descent. The policy model gradually learns to accurately identify and select the true tool even in the presence of highly similar interfering tools.

[0071] In terms of training settings, this embodiment of the invention employs the TRL (Transformer Reinforcement Learning) framework for training and uses the LoRA (Low-Rank Adaptation) parameter fine-tuning method to reduce computational resource consumption. The training experiments were completed on a single A100 GPU (40GB VRAM). The key hyperparameter settings during training are as follows: learning rate 5e-5, total batch size 8 (achieved through gradient accumulation), maximum input prompt length 4096 tokens, maximum length of model-generated responses 512 tokens, 2 training epochs, 8 responses generated per query text, and a generation temperature of 1.0.

[0072] After training, a test set is first constructed. The training set from Example 1 and the test set from this example together constitute the SimTools dataset. Test samples are taken from the UltraTool and Seal-Tools test sets, respectively. For each test sample, a corresponding candidate tool set is constructed according to three similarity levels: low, medium, and high, to comprehensively evaluate the model's tool selection performance under different obfuscation difficulties. Specifically, for a single query text, by calculating the comprehensive similarity between its corresponding real tool and the other tools in the tool library, obfuscated tools belonging to the low, medium, and high similarity intervals with the real tool are selected, and these obfuscated tools, together with the real tool, form the candidate tool set corresponding to that test sample. Combining the query text, the candidate tool set constructed in the above steps, and the real tool set corresponding to the query text forms a complete test sample.

[0073] During the evaluation phase, a greedy decoding strategy was consistently used to generate results, selecting the output with the highest probability each time. To eliminate interference from other factors, a zero-shot learning approach was consistently adopted, without providing any example guidance. The primary evaluation metric was the F1 score, along with definitions for precision and recall. The definitions and calculation methods for these metrics are consistent with existing work. Precision is the proportion of correctly predicted tools, recall is the proportion of all real tools successfully selected by the model, and the F1 score is the harmonic mean of precision and recall.

[0074] The experimental results are shown in Table 1. The results show that the similarity between tools is significantly negatively correlated with the model selection performance, and all methods show a significant performance decline on the high similarity test subset.

[0075] Table 1:

[0076] Regarding the base model, the Qwen2.5-7B-Instruct model achieved F1 scores of 87.2%, 83.3%, and 57.3% on the low, medium, and high similarity subsets of the UltraTool dataset, respectively, and 98.6%, 97.9%, and 91.2% on the three subsets of the Seal-Tools dataset. The performance on the high similarity subset was significantly lower than that on the low similarity subset, with a drop of nearly 30 percentage points in the UltraTool portion, demonstrating the severe impact of similar tool confusion on the model's selection ability. After training using the Supervised Fine-Tuning (SFT) method, the model's F1 score improved to 79.8% on the high similarity subset of UltraTools and 87.4% on the high similarity subset of Seal-Tools. While this showed improvement, the generalization ability remained insufficient, especially on the high similarity subset of Seal-Tools, where its performance was even worse than the base model. While the GRPO method, which employs both traditional binary rewards (GRPO-Bin) and ordinary F1 rewards (GRPO-F1), can further improve performance, its F1 scores on the most challenging UltraTool high similarity subset are 78.7% and 79.3%, respectively, neither of which exceeds the supervised fine-tuning method. This indicates that the traditional reward design has limited improvement in similar tool confusion scenarios.

[0077] In comparison, the similarity-sensitive reward method (SimS) proposed in this invention achieves state-of-the-art performance across all similarity subsets. On the UltraTool dataset, SimS achieves F1 scores of 92.0%, 90.5%, and 82.1% for low, medium, and high similarity subsets, respectively; on the Seal-Tools dataset, the F1 scores for the three subsets are 98.8%, 98.2%, and 93.3%, respectively. Regarding the average F1 score on the high similarity test subset, SimS achieves 87.7%, a 4.1 percentage point improvement over the traditional supervised fine-tuning method's 83.6%, and a 1.4 percentage point improvement over the ordinary F1 reward method's 86.3%, validating the effectiveness of incorporating tool similarity information into the reward design. This result demonstrates that by assigning higher reward weights to the correct selection when distinguishing highly similar tools, this invention effectively guides the policy model to focus on learning subtle differences between similar tools during training, thereby significantly improving the discrimination ability and accuracy and robustness of tool selection in high-similarity tool scenarios for large language models.

[0078] Example 3: Based on the same inventive concept as Embodiment 1, the present invention also provides a large language model tool selection system based on similarity-sensitive rewards, comprising: The training sample acquisition module is used to acquire training samples, including query text, candidate tool set and real tool set, wherein the real tool set is a subset of the candidate tool set; The prediction generation module is used to input the query text and the candidate tool set into the strategy model to generate a prediction tool set; The similarity calculation module is used to calculate the comprehensive similarity between every two tools in the candidate tool set, and obtain a comprehensive similarity matrix; The difficulty weight determination module is used to identify tools in the candidate tool set that do not belong to the real tool set as interference tools; and to determine the maximum value of the comprehensive similarity between the real tool and all interference tools according to the comprehensive similarity matrix, which is used as the difficulty weight of the real tool. The reward calculation module is used to determine the intersection of the predicted tool set and the real tool set as the real tool that was correctly predicted; and to calculate the similarity-sensitive reward value based on the difficulty weight of the real tool that was correctly predicted. The strategy update module is used to update the parameters of the strategy model using the similarity-sensitive reward value, thereby improving the strategy model's ability to select among similar tools.

[0079] The training sample acquisition module, prediction generation module, similarity calculation module, difficulty weight determination module, reward calculation module, and strategy update module of the large language model tool selection system based on similarity-sensitive rewards proposed in this embodiment are respectively used to implement steps S1, S2, S3, S4, S5, and S6 in the large language model tool selection method based on similarity-sensitive rewards in Embodiment 1. To avoid redundancy, they will not be described in detail here.

[0080] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for selecting tools for large language models based on similarity-sensitive rewards, characterized in that, Includes the following steps: S1: Obtain training samples, including query text, candidate tool set, and real tool set, wherein the real tool set is a subset of the candidate tool set; S2: Input the query text and the candidate tool set into the strategy model to generate a prediction tool set; S3: Calculate the comprehensive similarity between every two tools in the candidate tool set to obtain the comprehensive similarity matrix; S4: Tools in the candidate tool set that do not belong to the real tool set are identified as interference tools; based on the comprehensive similarity matrix, the maximum value of the comprehensive similarity between the real tool and all interference tools is determined as the difficulty weight of the real tool. S5: Determine the intersection of the predicted tool set and the real tool set as the real tool that was correctly predicted; calculate the similarity-sensitive reward value based on the difficulty weight of the real tool that was correctly predicted; S6: Update the parameters of the strategy model using the similarity-sensitive reward value to improve the strategy model's ability to select among similar tools.

2. The method for selecting large language model tools based on similarity-sensitive rewards according to claim 1, characterized in that: In step S4, based on the comprehensive similarity matrix, the maximum value of the comprehensive similarity between the real tool and all interfering tools is determined, and this value is used as the difficulty weight of the real tool. For each real tool in the set of real tools, the comprehensive similarity value between the real tool and each of the interference tools is obtained from the comprehensive similarity matrix, and the maximum value among the obtained comprehensive similarity values ​​is used as the difficulty weight of the real tool.

3. The method for selecting large language model tools based on similarity-sensitive rewards according to claim 1, characterized in that: In step S5, the method for calculating the similarity-sensitive reward value based on the difficulty weight of the correctly predicted real tool is as follows: The weighted accuracy is calculated based on the sum of the difficulty weights of the correctly predicted real tools and the total number of tools in the predicted tool set. The weighted recall rate is calculated based on the sum of the difficulty weights of the correctly predicted real tools and the total number of tools in the set of real tools. The similarity-sensitive reward value is calculated based on the harmonic mean of the weighted precision and the weighted recall.

4. The method for selecting large language model tools based on similarity-sensitive rewards according to claim 3, characterized in that: The method for calculating the similarity-sensitive reward value is as follows: Calculate weighted precision and weighted recall : ， ， in, The set of prediction tools, For the aforementioned set of real tools, For real tools Difficulty weighting Indicates the number of elements in the set; Calculate the weighted accuracy With the weighted recall rate The harmonic mean is used as the similarity-sensitive reward value. : 。 5. The method for selecting large language model tools based on similarity-sensitive rewards according to claim 1, characterized in that: In step S3, the method for calculating the comprehensive similarity between every two tools in the candidate tool set and obtaining the comprehensive similarity matrix is ​​as follows: The name, function description, and parameter list of each tool are converted into vector representations, and the semantic similarity between the two vectors is calculated using the cosine similarity formula. For any two tools in the candidate tool set, calculate the semantic similarity between the two tools in the three parts of the text: name, function description, and parameter list. The average of the three semantic similarities is taken as the comprehensive similarity between the two tools. The comprehensive similarity matrix is ​​constructed based on the comprehensive similarity between all pairs of tools in the candidate tool set.

6. The method for selecting large language model tools based on similarity-sensitive rewards according to claim 1, characterized in that: In step S1, the method for obtaining training samples is as follows: Obtain the query text and the actual set of tools corresponding to the query text; Use the real toolset as the initial element of the candidate toolset; Calculate the matching similarity between the query text and each other tool in the tool library. The matching similarity is obtained by averaging the cosine similarity between the embedding vector of the query text and the embedding vector of the name, function description, and parameter list of the other tool, respectively. Sort the tools by matching similarity from high to low, and select the top-ranked tools to add to the candidate tool set until the number of tools in the candidate tool set reaches a preset number. The training sample is composed of the query text, the candidate tool set, and the real tool set.

7. The method for selecting large language model tools based on similarity-sensitive rewards according to claim 1, characterized in that: In step S2, the method for inputting the query text and the candidate tool set into the strategy model to generate the prediction tool set is as follows: The query text, along with the name, function description, and tool parameters of each tool in the candidate tool set, are concatenated according to a preset prompt word template to construct input prompt information; The preset prompt word template contains instructions for instructing the strategy model to select a suitable tool from the candidate tool set; The input prompt information is input into the strategy model, and the model outputs the corresponding tool name and corresponding filling parameters. The set of tools corresponding to all the output tool names is taken as the prediction tool set.

8. The method for selecting large language model tools based on similarity-sensitive rewards according to claim 1, characterized in that: In step S6, the method for updating the parameters of the policy model using the similarity-sensitive reward value is as follows: A group-relative strategy optimization algorithm is used to generate multiple sets of prediction tools for the same query text, forming a response group; Calculate the similarity-sensitive reward value for each set of prediction tools in the response group; Based on the similarity-sensitive reward value of each prediction tool set within the response group, calculate the advantage function value corresponding to each prediction tool set; The parameters of the policy model are updated by gradient descent with the goal of minimizing the loss function that includes the advantage function value and the probability ratio pruning mechanism.

9. The method for selecting large language model tools based on similarity-sensitive rewards according to claim 1, characterized in that: The strategy model adopts a pre-trained large language model based on the Transformer architecture.

10. A tool selection system for large language models based on similarity-sensitive rewards, characterized in that, include: The training sample acquisition module is used to acquire training samples, including query text, candidate tool set and real tool set, wherein the real tool set is a subset of the candidate tool set; The prediction generation module is used to input the query text and the candidate tool set into the strategy model to generate a prediction tool set; The similarity calculation module is used to calculate the comprehensive similarity between every two tools in the candidate tool set, and obtain a comprehensive similarity matrix; The difficulty weight determination module is used to identify tools in the candidate tool set that do not belong to the real tool set as interference tools; and to determine the maximum value of the comprehensive similarity between the real tool and all interference tools according to the comprehensive similarity matrix, which is used as the difficulty weight of the real tool. The reward calculation module is used to determine the intersection of the predicted tool set and the real tool set as the real tool that was correctly predicted; and to calculate the similarity-sensitive reward value based on the difficulty weight of the real tool that was correctly predicted. The strategy update module is used to update the parameters of the strategy model using the similarity-sensitive reward value, thereby improving the strategy model's ability to select among similar tools.

Images