Tool determination method, apparatus, and electronic device

Abstract

This application discloses a tool determination method, apparatus, and electronic device, relating to the field of artificial intelligence technology. The method includes: determining the recall quantity based on N request intentions corresponding to a user request; after vectorizing the user request to obtain a user request vector, filtering in a preset database based on the user request vector to obtain K tools; obtaining the recall similarity, semantic score, and historical success rate for each of the K tools; after weighted summing of the recall similarity, semantic score, and historical success rate for each tool to obtain a decision score for each tool, selecting the tool with the highest decision score among the K tools as the target tool corresponding to the user request. This application solves the technical problems of low efficiency and low accuracy in the prior art when selecting and calling tools integrated into a large language model backend.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a tool determination method, apparatus, and electronic device. Background Technology

[0002] In recent years, with the improvement of LLM (Large Language Model) in understanding and generating natural language, LLM-based intelligent agent systems have been widely used in the field of enterprise business process automation. Intelligent agent systems have achieved a breakthrough in the traditional question-and-answer mode by recognizing user intent, extracting key parameters, and converting them into call instructions for backend business interfaces. However, in actual enterprise applications, the backend toolsets integrated by LLM are large in number and highly complex, often containing dozens or even hundreds of business interfaces, which brings severe technical challenges to the invocation of existing intelligent agent systems.

[0003] Existing technologies, when calling tools integrated into the LLM backend, mainly rely on loading the tool description text corresponding to all available tools into the LLM's prompts. The LLM then performs a one-time global inference and tool decision based on the prompts. As the toolset size increases, the input context window corresponding to the prompts expands dramatically, greatly extending the LLM's inference time. This not only leads to high response latency and a degraded user experience in the intelligent agent system, but also significantly increases enterprise operating costs due to the increased token consumption. This method is difficult to meet the tool selection needs in high-concurrency scenarios.

[0004] Furthermore, this single-stage decision-making model also faces challenges in the accuracy and reliability of tool selection. LLM tool selection is mainly based on semantic matching of language, lacking consideration of actual business execution results. When faced with a large number of functionally similar or semantically overlapping interfaces, the model is prone to confusion and misselection. Moreover, its decision-making process fails to incorporate business data such as the historical success rate or failure rate of tool calls for calibration, so even if the selected tool matches in language, it may still cause transaction execution failure due to low business reliability. In addition, the rigidity of the existing screening mechanism also limits the robustness of the system, and it cannot adaptively adjust the screening strategy according to the ambiguity or explicitness of user input information.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a tool determination method, apparatus, and electronic device to at least solve the technical problems of low efficiency and low accuracy in the prior art when selecting and calling tools integrated into the background of a large language model.

[0007] According to one aspect of this application, a tool determination method is provided, comprising: determining a recall quantity based on N request intents corresponding to a user request, wherein N is a positive integer, each request intent is used to characterize a business operation that the user request needs to invoke, and the recall quantity is linearly related to the total number of N request intents; after vectorizing the user request to obtain a user request vector, filtering based on the user request vector in a preset database to obtain K tools, wherein K is equal to the recall quantity, and the preset database includes tool information of all tools integrated into the backend of a large language model; obtaining the recall similarity, semantic score, and historical success rate of each of the K tools, wherein the recall similarity is the similarity between the user request vector and the tool vector of the tool, the semantic score is used to characterize the confidence of the large language model in the tool, and the historical success rate is the success rate of the tool being invoked in the scenario type corresponding to the user request; after weighted summing of the recall similarity, semantic score, and historical success rate of each tool to obtain the decision score of each tool, the tool with the highest decision score among the K tools is taken as the target tool corresponding to the user request.

[0008] Optionally, before determining the recall quantity based on the N request intents corresponding to the user request, the tool's determination method further includes: performing semantic analysis on the user request based on preset analysis instructions using a large language model to obtain N analysis results, wherein each analysis result is used to store a business operation that the user request needs to invoke in a preset format; and determining N request intents based on the N analysis results.

[0009] Optionally, the recall quantity is determined based on the N request intents corresponding to the user request, including: obtaining the product of the total number of N request intents and a preset correction coefficient, wherein the preset correction coefficient is determined by backtracking the historical success rate of each tool in the preset database; and summing the product, the preset basic recall quantity, and the preset redundant recall quantity to obtain the recall quantity.

[0010] Optionally, K tools are obtained by filtering in a preset database based on user request vectors, including: detecting the similarity between the tool vector of each tool in the preset database and the user request vector to obtain L recall similarities for L tools, where L is a positive integer greater than or equal to K, and the tool vector is used to characterize the functional description and parameter requirements of the tool; after sorting the L tools based on the L recall similarities, the tools ranked in the top K positions in the sorting results are selected to obtain K tools.

[0011] Optionally, after filtering K tools from a pre-defined database based on user request vectors, the tool determination method further includes: obtaining metadata corresponding to each of the K tools, wherein the metadata includes at least the tool identifier, function description, and recall similarity; generating target prompt words based on the K metadata corresponding to the K tools and the user request, and inputting the target prompt words into a large language model; determining the confidence level of each tool and the business parameters corresponding to the user request based on the target prompt words through the large language model; normalizing the confidence level of each tool to obtain the semantic score of each tool, and binding the business parameters corresponding to the user request to each tool.

[0012] Optionally, after selecting the tool with the highest decision score among the K tools as the target tool corresponding to the user request, the tool determination method further includes: detecting the intent type corresponding to each of the N request intents; generating a target instruction sequence through the target tool based on the intent type corresponding to each request intent and the business parameters bound to the target tool, wherein the target instruction sequence includes at least N tool call instructions, each tool call instruction being used to implement a business operation corresponding to a request intent; and executing the N tool call instructions in the order of the instructions corresponding to the target instruction sequence.

[0013] Optionally, N tool invocation instructions are executed sequentially based on the instruction sequence corresponding to the target instruction sequence, including: after the i-th tool invocation instruction among the N tool invocation instructions is executed, the business result of the i-th tool invocation instruction is detected, wherein the business result is one of the following: a first result, used to indicate that the business operation is successful; a second result, used to indicate that the business operation fails; a third result, used to indicate that the system facility or interface has failed; if the business result of the i-th tool invocation instruction is the first result / second result, the historical success rate of the target tool under the scenario type corresponding to the user request is updated based on the business result of the i-th tool invocation instruction.

[0014] Optionally, after updating the historical success rate of the target tool in the scenario type corresponding to the user request based on the business result of the i-th tool invocation instruction, the tool determination method further includes: if the business result of the i-th tool invocation instruction is the second result, updating the decision score of the target tool based on the new historical success rate of the target tool; updating the decision score sequence corresponding to K tools based on the new decision score of the target tool; after determining a new target tool based on the new decision score sequence, re-executing the tool invocation instruction corresponding to each remaining request intent based on the new target tool, wherein each remaining request intent is a request intent among N request intents where the business operation was not executed / the business operation failed.

[0015] According to another aspect of this application, a tool determination apparatus is also provided, comprising: a recall quantity determination unit, configured to determine the recall quantity based on N request intents corresponding to a user request, wherein N is a positive integer, each request intent represents a business operation that the user request needs to invoke, and the recall quantity is linearly related to the total number of N request intents; and a tool filtering unit, configured to, after performing vector transformation on the user request to obtain a user request vector, filter the user request vector in a preset database to obtain K tools, wherein K is equal to the recall quantity, and the preset database includes at least all tools integrated into the large language model backend. Information; a similarity determination unit, used to obtain the recall similarity, semantic score, and historical success rate for each of the K tools, where the recall similarity is the similarity between the user request vector and the tool vector, the semantic score is used to characterize the confidence of the large language model in the tool, and the historical success rate is the success rate of the tool being invoked in the scenario type corresponding to the user request; a target tool determination unit, used to obtain the decision score for each tool by weighted summing of the recall similarity, semantic score, and historical success rate, and then select the tool with the highest decision score among the K tools as the target tool corresponding to the user request.

[0016] According to another aspect of this application, a computer program product is also provided, which stores a computer program, wherein a tool determination method is provided to control the computer program product to perform any of the above-mentioned tasks during the execution of the computer program.

[0017] According to another aspect of this application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the tool determination method of any one of the above.

[0018] In this application, the recall quantity is first determined based on N request intents corresponding to the user request, where N is a positive integer. Each request intent represents a business operation that the user request needs to invoke. The recall quantity is linearly related to the total number of N request intents. Then, after vectorizing the user request to obtain the user request vector, the user request vector is filtered in a preset database to obtain K tools, where K is equal to the recall quantity. The preset database includes tool information for all tools integrated into the large language model backend. The recall similarity, semantic score, and historical success rate of each of the K tools are obtained. The recall similarity is the similarity between the user request vector and the tool vector, the semantic score represents the confidence of the large language model in the tool, and the historical success rate is the success rate of the tool invoking in the scenario type corresponding to the user request. Subsequently, the recall similarity, semantic score, and historical success rate of each tool are weighted and summed to obtain the decision score of each tool. The tool with the highest decision score among the K tools is selected as the target tool corresponding to the user request.

[0019] As can be seen from the above, this application adopts a two-stage cascaded tool selection method. By dynamically calculating the recall quantity K based on the number of user request intents, and combining the recall similarity matching results, K tools in the preset database are initially screened. Then, this application makes a comprehensive decision based on the three-dimensional weighted results corresponding to recall similarity, semantic score and historical success rate. This achieves the goal of accurately locking the target tool while greatly compressing the prompt word input context window of the large language model.

[0020] Therefore, this application avoids the high latency and high token overhead caused by loading the full tool description text in traditional technologies by separating the rapid recall and accurate decision-making stages of tools. At the same time, this application introduces the historical success rate of tools as one of the reliability dimensions of target tool decision-making, overcoming the low accuracy of target tool selection caused by relying solely on semantic matching in traditional technologies. This improves the response speed and execution success rate of intelligent agent systems in large-scale toolset environments, thereby achieving a technical effect that balances tool invocation efficiency and business reliability. In turn, it solves the technical problems of low efficiency and low accuracy in the selection and invocation of tools integrated into the backend of large language models in existing technologies. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1This is a flowchart of an optional tool determination method according to an embodiment of this application;

[0023] Figure 2 This is an optional tool for determining the system architecture diagram according to an embodiment of this application;

[0024] Figure 3 This is an optional tool for determining the module interaction flowchart of the system according to an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of an optional tool determining device according to an embodiment of this application;

[0026] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It should also be noted that all information and data (including but not limited to information used for display and analysis) involved in this application are authorized by the user or fully authorized by all parties. For example, if there is an interface between this system and the relevant user or organization, before obtaining the relevant information, it is necessary to send a request to the aforementioned user or organization through the interface, and obtain the relevant information only after receiving consent from the aforementioned user or organization.

[0030] Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of relevant information and data involved in this application all comply with the relevant laws, regulations, and standards of the relevant regions, and necessary confidentiality measures have been taken. This application does not violate public order and good morals. In addition, this application provides a corresponding operation entry point for users to choose to agree to or refuse authorization. If the user chooses to refuse authorization, the corresponding expert decision-making process will be initiated.

[0031] To overcome the shortcomings of traditional technologies, this application proposes a tool determination method. This mechanism aims to decouple the tool scheduling process and replace the full inference of LLM with a tool fast recall mechanism. It solves the problems of high latency and high cost under large-scale toolsets with an efficient vector retrieval mechanism. On this basis, when making the final tool selection, the method introduces a weighted correction mechanism based on historical success rate and a dynamic recall mechanism, thereby solving the problems of insufficient accuracy and reliability of tool selection. This enables the intelligent agent system to maintain efficient, accurate and stable operation in a large-scale toolset environment.

[0032] The present application will now be described in detail with reference to various embodiments.

[0033] Example 1

[0034] According to an embodiment of this application, an embodiment of a tool determination method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] This application provides a tool determination system (hereinafter referred to as the determination system) for performing the tool determination method of this application. Figure 1 This is a flowchart of an optional tool determination method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0036] Step S101: Determine the recall quantity based on the N request intents corresponding to the user request, where N is a positive integer, each request intent is used to represent a business operation that the user request needs to invoke, and there is a linear relationship between the recall quantity and the total number of N request intents.

[0037] For example, after the system receives the user request "I want to change my flight from Shanghai to Beijing tomorrow to the day after tomorrow, and check if there are any business class seats available," it performs semantic analysis using a lightweight large language model to identify two request intentions: 1) changing flight dates; 2) checking for business class seats. =2, then, determine the system based on Determine the number of items to be recalled.

[0038] Optionally, request intent refers to the business operation that can be executed independently in a user request, such as "rebooking a flight" or "checking business class seats". Request intent is the smallest semantic unit that the system uses to identify the complexity of the task.

[0039] Optionally, a linear relationship means that the number of recalls increases proportionally to the number of graphs. For each additional independent intent, the number of recall tools increases accordingly, thus avoiding omissions in tool selection.

[0040] Optionally, the system can dynamically adjust the recall scale based on the number of intents. This avoids over-recalling tools for user requests with simple intents, thus saving system resources. It also ensures sufficient coverage of the request intents corresponding to relatively complex user requests, which helps improve the efficiency and accuracy of tool recall in the initial tool screening stage.

[0041] Step S102: After performing vector transformation on the user request to obtain the user request vector, filter the user request vector in the preset database to obtain K tools, where K is equal to the recall number. The preset database includes tool information of all tools integrated into the large language model backend.

[0042] For example, the system determines that the user request "I want to change my flight from Shanghai to Beijing tomorrow to the day after tomorrow and see if there are any business class seats available" is transformed into a 128-dimensional user request vector. Then, for each tool vector corresponding to 150 tools in the preset database, the system calculates the cosine similarity between the user request vector and each tool vector based on the approximate nearest neighbor algorithm. After that, the tools are sorted from high to low according to the cosine similarity, and the top K tools are selected as the initial recall results.

[0043] Optionally, the system determines the selection of K tools based on the cosine similarity between the user request vector and the tool vector of each tool. Compared with the LLM model inference method in related technologies, the vector similarity comparison method improves the selection speed of K tools.

[0044] Step S103: Obtain the recall similarity, semantic score, and historical success rate for each of the K tools. The recall similarity is the similarity between the user request vector and the tool vector. The semantic score is used to characterize the confidence of the large language model in the tool. The historical success rate is the success rate of the tool being invoked in the scenario type corresponding to the user request.

[0045] Optionally, if a tool has no historical call records for the business scenario type corresponding to the user request, then the global average success rate of the tool across all business scenario types is used as the initial value of the historical success rate.

[0046] Optionally, after acquiring K tools, the system determines the recall similarity (derived from the cosine similarity calculated during similarity matching in step S102), semantic score (derived from the normalized result of LLM scoring the confidence of each tool in the restricted context window corresponding to the K tools), and historical success rate (derived from the historical success rate of each tool in the business scenario type corresponding to the user request, for example, 0.93) of each tool among the K candidate tools. Then, the system determines the decision score of each tool based on the recall similarity, semantic score, and historical success rate of each tool, and uses the decision score of each tool as the decision basis for each tool. By introducing the historical success rate of each user in the corresponding business scenario type as one of the dimensions of the decision basis, the system no longer blindly trusts the semantic inference results of the LLM model when calling multiple tools integrated in the backend of the large language model, which helps to improve the accuracy of the final selection of the target tool and the success rate of the tool call.

[0047] Step S104: After weighted summation of the recall similarity, semantic score and historical success rate for each tool to obtain the decision score for each tool, the tool with the highest decision score among the K tools is selected as the target tool corresponding to the user's request.

[0048] Optionally, the system determines the decision score for each tool based on the following three-dimensional weighted correction formula. :

[0049] ;

[0050] In the above formula, To recall similarity, Score the semantics of LLM. Based on historical success rate, The preset weighting coefficients are used, and they satisfy the following conditions: .

[0051] Optionally, the system determines that by integrating the evaluation metrics (recall similarity, semantic score, and historical success rate) of each tool across three independent dimensions into a single decision metric, information synergy among multi-source evaluation metrics is achieved. This ensures that the decision score used for the final tool decision integrates the inference results of semantic analysis from a large language model and the dual experience of historical success rates in the corresponding business scenario type. This breaks through the limitations of traditional technologies that "only rely on semantic matching of large models," thereby helping to improve the decision certainty of the target tool.

[0052] Optionally, when the system determines the semantic score of each tool in the control of the large model, it only determines the semantic score of each of the K tools initially selected, instead of performing semantic score inference on all tools integrated in the background of the large model. This avoids the problem of the input context window of the large language model expanding rapidly in traditional techniques and helps to improve the speed of subsequent calls to the target tool by the large language model.

[0053] As can be seen from the above, this application adopts a two-stage cascaded tool selection method. By dynamically calculating the recall quantity K based on the number of user request intents, and combining the recall similarity matching results, K tools in the preset database are initially screened. Then, this application makes a comprehensive decision based on the three-dimensional weighted results corresponding to recall similarity, semantic score and historical success rate. This achieves the goal of accurately locking the target tool while greatly compressing the prompt word input context window of the large language model.

[0054] Therefore, this application avoids the high latency and high token overhead caused by loading the full tool description text in traditional technologies by separating the rapid recall and accurate decision-making stages of tools. At the same time, this application introduces the historical success rate of tools as one of the reliability dimensions of target tool decision-making, overcoming the low accuracy of target tool selection caused by relying solely on semantic matching in traditional technologies. This improves the response speed and execution success rate of intelligent agent systems in large-scale toolset environments, thereby achieving a technical effect that balances tool invocation efficiency and business reliability. In turn, it solves the technical problems of low efficiency and low accuracy in the selection and invocation of tools integrated into the backend of large language models in existing technologies.

[0055] In one alternative embodiment, Figure 2 This is an optional tool used to determine the system architecture diagram according to an embodiment of this application, such as... Figure 2 As shown, the system comprises a user interaction and input module (M1), a tool rapid retrieval module (M2), a precise reasoning and decision-making module (M3), and a tool execution and fault tolerance module (M4). A knowledge and data management layer (DB) is located in the background to provide data support. The knowledge and data management layer (DB) maintains a historical success rate record table for each tool under a specific business scenario type. The data structure of this table includes:

[0056] (1) Tool ID (identifier);

[0057] (2) Scenario types of business scenarios: such as "flight inquiry", "flight rescheduling", "seat inquiry", etc.;

[0058] (3) Historical success rate: The historical success rate of this tool in a specific business scenario;

[0059] (4) Number of tool calls: used to determine the confidence level of selecting and calling the tool.

[0060] Optionally, Figure 2 The module interaction process includes: the user interaction and input module (M1) is responsible for receiving the user's natural language request and passing it to the subsequent tool quick recall module (M2) and accurate reasoning and decision-making module (M3). Through the cascading collaboration of the tool quick recall module (M2) and accurate reasoning and decision-making module (M3), the target tool that the large language model should actually call is determined. Then, the tool execution and fault tolerance module (M4) executes the business operations that the user request needs to achieve based on the target tool.

[0061] In one alternative embodiment, Figure 3 This is an optional tool for determining the module interaction flowchart of the system according to an embodiment of this application, such as... Figure 3 As shown, the interaction flow between the user interaction and input module (M1), the tool quick recall module (M2), the accurate reasoning and decision-making module (M3), and the tool execution and fault tolerance module (M4) is as follows:

[0062] First, upon receiving a user request from the user interaction and input module (M1), the tool rapid recall module (M2) is immediately activated. This is the first stage of tool selection. Instead of directly requesting the LLM (Learning Management Module), the tool rapid recall module (M2) utilizes pre-stored tool vectors in the Knowledge and Data Management layer (DB) to perform high-speed vector similarity retrieval of the user request. Simultaneously, the tool rapid recall module (M2) incorporates a dynamic tool recall mechanism, adaptively determining the optimal recall quantity based on the number of independent intents corresponding to the user request, according to the ambiguity of the user's input request. and output The most relevant tool names and concise tool descriptions are used to enable a quick initial screening of a large set of tools.

[0063] Subsequently, the screening results are transmitted to the Precise Reasoning and Decision-Making module (M3) to enter the second stage of tool selection. The M3 module receives the user's original request and a concise list of candidate tools provided by the M2 module. By limiting the inference scope of the LLM, the M3 module can perform efficient and accurate intent recognition and parameter extraction within a very small context window. In the final tool selection, the M3 module does not rely entirely on the semantic judgment of the LLM, but instead queries historical success rate data stored in the database layer and incorporates a weighted correction mechanism to take into account the tool's historical business performance in the decision-making process. Ultimately, it generates standardized and highly reliable tool invocation instructions.

[0064] Finally, the tool execution and fault tolerance module (M4) receives standardized instructions output by the M3 module and is responsible for calling the actual backend API (Application Programming Interface). The M4 module has internal fault tolerance and rollback logic, which can handle abnormal situations such as interface call failure, timeout, or return error information, ensuring high reliability of transaction execution. The results returned by the API are encapsulated by the M4 module and fed back to the user in a natural and user-friendly manner through the M1 module. Through the cascading and data feedback mechanism of the M2 and M3 modules, this application successfully delegates the speed problem in the large-scale tool selection process to M2, while delegating the accuracy and reliability problems to M3, thereby achieving high-performance intelligent agent tool scheduling.

[0065] Furthermore, the tool execution and fault tolerance module (M4), upon receiving the data, categorizes the success rate of the invoked tool in the current scenario based on the business results. (This will be corrected in real time, and then the updated version will be...) The value is written to the corresponding scenario-based record table in the DB layer. Upon the next user request, the M3 module will query and retrieve the new tool's corresponding information based on the identified scenario type. The value is used for three-dimensional weighted correction, thereby prioritizing the tool with the highest business success rate in the current scenario.

[0066] In one optional embodiment, before determining the recall quantity based on the N request intents corresponding to the user request, the system first performs semantic analysis on the user request based on preset analysis instructions using a large language model to obtain N analysis results. Each analysis result is used to store a business operation that the user request needs to invoke in a preset format. Then, the system determines N request intents based on the N analysis results.

[0067] Optionally, the large language model in the above embodiments can be set as an independent lightweight large language model (SLLM) to distinguish it from the large language model used for reasoning by integrating multiple tools, so as to facilitate the rapid analysis of the user's request intent.

[0068] Optionally, the tool's quick recall module (M2) can initiate a lightweight large language model to perform preliminary analysis of the user request. The system submits a structured prompt to the large language model, explicitly requesting the model to decompose the user request and output the identified independent intents in a standardized JSON (JavaScript Object Notation) format. The preset analysis instructions in the prompt can be set to "Analyze the following user request, identify all executable independent operations, name each operation, and calculate the final number of intents".

[0069] For example, a lightweight large language model performs semantic analysis on a user request ("I want to change my flight from Shanghai to Beijing tomorrow to the day after tomorrow, and check if there are any business class seats available."). The N analysis results are: "flights from Shanghai to Beijing the day after tomorrow" and "business class seat search". Then, the system determines N independent intents based on the N analysis results: "rebook flight" and "search for business class seats".

[0070] Optionally, when determining the independent intent corresponding to a user request, the system adopts a lightweight large language model + structured instruction mechanism to avoid using a resource-intensive large language model for intent splitting, thereby reducing the processing response latency of user requests. Furthermore, by outputting the analysis results in a structured manner, the system enables subsequent independent intents to be automatically parsed without the need for manual rules or regular expression matching, avoiding the limitations of manual rules or keyword matching. This allows the system to automatically adapt to new business operations without modifying the code, improving the recognition accuracy and generalization ability of independent intents.

[0071] In one optional embodiment, the determining system first obtains the product of the total number of N request intents and a preset correction coefficient, wherein the preset correction coefficient is determined by backtesting the historical success rate of each tool in a preset database. Then, the determining system sums the product, the preset basic recall, and the preset redundant recall to obtain the recall number.

[0072] Optionally, a preset basic recall quantity is set to ensure that the recall tools can cover the business operations corresponding to the N independent intents that fulfill the user's request.

[0073] Optionally, a preset redundant recall setting is provided to address scenarios where intent recognition boundaries are ambiguous or semantic overlaps, thus providing a buffer.

[0074] Optionally, the system determines the recall quantity based on the following formula:

[0075] ;

[0076] In the above formula, K represents the number of recalls. Baseline recall (i.e., the preset baseline recall, for example) ), Minimum reserved redundancy (i.e., preset redundancy recall, for example) ), The intended correction factor (i.e., the preset correction factor, for example) ), The value is a constant that has been verified through offline business operations. Its value is the minimum compensation required to maintain the business success rate under the corresponding business scenario type, determined by the system engineer through backtesting of the historical success rate data of each tool in the preset database.

[0077] For example, in At that time, the number of tools recalled based on the above formula. ;exist At that time, the number of tools recalled based on the above formula. .

[0078] Optionally, the correction coefficient in the above formula is not set based on experience, but is driven by the actual business call success rate (i.e., historical success rate) corresponding to each tool, ensuring that its value reflects the actual tool call demand. The above formula establishes a quantitative and verifiable linear relationship between the number of recalls and the task complexity (i.e. the number of independent intents), avoiding the problem of redundancy in simple requests and insufficiency in complex requests when the number of recalls is fixed (e.g., fixed at 5 or 10). Furthermore, the method of determining the intent correction coefficient through backtracking tests ensures the objectivity and stability of the number of recalls. Even if the business call success rate of the toolset or tool changes over time, the value of the intent correction coefficient can be automatically updated simply by re-performing the backtracking test, without the need for manual parameter tuning.

[0079] In summary, the system determines the recall quantity by summing the product, the preset basic recall quantity, and the preset redundant recall quantity, achieving the following technical effects:

[0080] (1) By using the preset correction coefficient determined by backtesting based on historical success rate, combined with the basic recall and redundant recall, a quantifiable recall calculation formula is formed, which realizes the accurate matching between the recall scale and the complexity of user requests.

[0081] (2) This method avoids the subjectivity and lag of manually setting parameters, enabling the system to have data-driven adaptive capabilities. Without increasing the complexity of the model, it helps to improve the coverage and selection reliability of candidate tools.

[0082] In one optional embodiment, the determining system first detects the similarity between the tool vector and the user request vector of each tool in the preset database to obtain L recall similarities for L tools, where L is a positive integer greater than or equal to K, and the tool vector is used to characterize the functional description and parameter requirements of the tool. Then, after the determining system sorts the L tools based on the L recall similarities, it filters the tools ranked in the top K positions in the sorting results to obtain K tools.

[0083] Optionally, the user request vector, which is a vector obtained by transforming the current user's natural language request through the same semantic model, is used to represent the semantic intent of the request.

[0084] Optionally, a tool vector refers to a numerical vector generated after the functional description and parameter requirements of each tool are encoded by a unified semantic model, which is used to represent the semantic content of the tool.

[0085] Optionally, recall similarity refers to the cosine similarity between the user request vector and the tool vector.

[0086] Optionally, after the system receives a user request, "I want to change my flight from Shanghai to Beijing tomorrow to the day after tomorrow, and see if there are any business class seats available," it transforms this request into a 128-dimensional user request vector using a unified semantic model, thus determining the... Value (10) and user request vector The data is sent to the database layer, which then performs a retrieval using an approximate nearest neighbor algorithm engine, and calculates... With all tool vectors Cosine similarity between After obtaining 150 recall similarities, the DB layer sorts these 150 tools from highest to lowest recall similarity and selects the top 10 tools (K is determined by the previous steps to be 10) as the candidate tool set. The remaining 140 tools are eliminated and will not enter the subsequent inference stage of the large language model. Then, the DB layer extracts the ID, simplified description and similarity score of the top 10 tools. This structured metadata is then sent back to the M2 module, which in turn outputs it to the M3 module for further precise inference and weighted decision-making.

[0087] Optionally, the system determines that by detecting the similarity between the tool vector of each tool in the preset database and the user request vector, the semantic-level fast matching of tools can be achieved through vector similarity calculation. This method does not rely on keywords or rules and can handle complex situations such as synonyms and semantic generalization. Furthermore, this process is executed in the vector database using an approximate nearest neighbor algorithm, which has low time consumption (milliseconds) and can efficiently handle tool sets of hundreds or thousands of levels. This is superior to the inference method of reading each item in LLM. In addition, the tool vectors are generated from standardized text to ensure semantic consistency and avoid matching deviations caused by differences in description formats.

[0088] Optionally, after the system ranks L tools based on L recall similarities, it selects the top K tools from the ranking results to obtain K tools. The system reduces the number of tools that need to be calculated for decision scoring through a ranking + truncation mechanism. For example, it can compress 150 tools into 10, reducing the input size by 93%, reducing the input burden of the context window of the subsequent LLM, and thus helping to improve the efficiency of subsequent tool decision scoring and the efficiency of target tool selection.

[0089] In summary, this step of the system establishes an efficient first-stage tool recall mechanism through vector similarity matching and ranking. This mechanism enables the rapid identification of the most relevant candidate subsets from a massive tool set. It does not rely on language model inference, is fast, low-cost, and scalable, and provides a reliable and concise input foundation for subsequent LLM to perform high-precision semantic parsing and weighted decision-making with small-scale inputs.

[0090] In one optional embodiment, after filtering K tools from a preset database based on user request vectors, the system first obtains the metadata corresponding to each of the K tools. The metadata includes at least the tool identifier, function description, and recall similarity. Then, after generating target prompt words based on the K metadata corresponding to the K tools and the user request, the target prompt words are input into a large language model. The system then uses the large language model to determine the confidence level of each tool and the business parameters corresponding to the user request based on the target prompt words. Subsequently, the system normalizes the confidence level of each tool to obtain a semantic score for each tool and binds the business parameters corresponding to the user request to each tool.

[0091] Optionally, the M3 module receives the Top 10 candidate tool list (including ID, concise description, and recall similarity score S) output by the M2 module. The M3 module integrates the original user request ("I want to change my flight from Shanghai to Beijing tomorrow to the day after tomorrow, and see if there are any business class seats available") with the metadata of the K tools to construct a context-window-compact target prompt word to guide the LLM in confidence analysis. Then, the LLM in the M3 module evaluates each tool based on semantic reasoning and outputs a semantic score for each tool. At the same time, the LLM parses out key business parameters (e.g., the day after tomorrow, Shanghai to Beijing, business class) and binds them to each of the K tools in the initial decision.

[0092] Optionally, compared with the traditional approach of constructing target prompts based on the metadata of all tools integrated in the backend, this system generates target prompts based on the K metadata corresponding to K tools and user requests. This avoids passing redundant information (such as parameter structures, error codes, etc.) to the LLM, thereby compressing the length of the target prompts and reducing inference latency and token consumption. At the same time, since the input range is strictly limited, the LLM's inference attention is more focused, and it can more accurately identify the matching relationship between intent and tools, thereby improving the accuracy of the semantic score of each tool obtained by the LLM inference.

[0093] In summary, the system normalizes the confidence level of each tool by scaling the confidence level output by LLM to a fixed range to obtain a semantic score for each tool, thus achieving standardization of semantic scores. Subsequently, the system binds the business parameters corresponding to user requests to each tool. The parameter binding mechanism ensures that the parameters in subsequent call instructions correspond one-to-one with the tools, avoiding call failures caused by parameter mismatches.

[0094] In one optional embodiment, after selecting the tool with the highest decision score among K tools as the target tool corresponding to the user request, the system first detects the intent type corresponding to each of the N request intents. Then, the system generates a target instruction sequence based on the intent type corresponding to each request intent and the business parameters bound to the target tool. The target instruction sequence includes at least N tool call instructions, each of which is used to implement a business operation corresponding to a request intent. Subsequently, the system executes the N tool call instructions in the order of the instructions corresponding to the target instruction sequence.

[0095] Optionally, the system selects via the M3 module. The highest target tool generates a tool sequence: [Query Flight API] → [Business Class Filter Tool]. Then, the M3 module converts the selected tool sequence and its bound business parameters into a structured sequence of N tool call instructions, resulting in the target instruction sequence.

[0096] Optionally, the M4 module receives the structured tool call sequence output by the M3 module. The M4 module then starts a state machine and calls the tools sequentially. For example, the M4 module calls the "Flight Query API" and automatically uses the query result as input for the next step, the "Business Class Filter Tool." Afterward, the M4 module receives the execution sequence results and directly returns the final filtering result (e.g., "Among the flights from Shanghai to Beijing the day after tomorrow, there are 3 with business class seats") to the user. If the current execution step returns a business failure status code, the M4 module does not call the next tool in the sequence but immediately performs the following operations:

[0097] Immediate fix: The M4 module first executes closed-loop feedback to reduce the failure rate of this tool. The value is then written to the DB layer. This triggers a re-decision; the M4 module then sends an instruction to the system, requesting to re-enter the M3 module with the original request and remaining intent, and utilize the updated... Re-select the target tools and re-plan the sequence.

[0098] Optionally, the system generates a target instruction sequence based on the intent type corresponding to each request intent and the business parameters bound to the target tool. This ensures that the instruction sequence is generated based on the intent type and the bound parameters, guaranteeing the accuracy of the instruction content, avoiding errors caused by manual splicing, and ensuring that the instruction format of the N tool call instructions in the target instruction sequence is uniform. No additional parsing logic is required during subsequent execution, thus improving the robustness of the system.

[0099] Optionally, the system determines that the system executes N tool call instructions in sequence based on the instruction sequence corresponding to the target instruction sequence through the M4 module, supporting chained calls of the target tool. This enables the large language model to handle complex user requests that include multiple independent intents (such as "change ticket + search seat + change seat"), rather than just single-point queries, thereby improving the service capabilities of the large language model agent.

[0100] In summary, the above steps collectively achieve automated, orderly, and closed-loop execution of multi-intent requests. By identifying intent types to determine business logic, constructing precise instructions through parameter binding, and completing the task chain through sequential execution, the technical shortcomings of single-point response and multiple interactions required for multiple independent intents in related technologies are resolved. Without increasing the user's burden, it can complete multi-step business operations corresponding to complex user requests, thereby improving the processing efficiency of user requests and the user experience.

[0101] In one optional embodiment, after the system determines that the i-th tool call instruction out of N tool call instructions has been executed, it detects the business result of the i-th tool call instruction. Then, if the business result of the i-th tool call instruction is a first result / a second result, it updates the historical success rate of the target tool in the scenario type corresponding to the user request based on the business result of the i-th tool call instruction.

[0102] Alternatively, the business outcome may be one of the following:

[0103] The first result is used to characterize the success of the business operation, that is, the tool completed all the business operations required by the user as expected;

[0104] The second result is used to characterize business operation failures, that is, the tool returns a clear business error code, indicating that the tool is unreliable under the current parameters or business scenario type;

[0105] The third result is used to characterize system facilities or interfaces that have failed.

[0106] Optionally, the system can accurately identify the reasons for tool failures by clearly classifying three types of business results. The first and second results reflect the reliability of the target tool in performing business operations in the business scenario corresponding to the user request. In this case, the historical success rate of the target tool needs to be updated. The third result is a system-level anomaly and is not included in the tool evaluation to avoid misjudging the tool as unreliable due to occasional failures. This classification mechanism ensures that the update of the tool's historical success rate is based only on business performance attributable to the tool's capabilities, eliminating interference from non-tool factors and improving the authenticity and reliability of the historical success rate evaluation. Furthermore, this classification mechanism can autonomously update the tool's business success rate (i.e., historical success rate) based on the tool's actual business performance, achieving self-evolution in tool selection.

[0107] In one optional embodiment, after updating the historical success rate of the target tool under the scenario type corresponding to the user request based on the business result of the i-th tool call instruction, if it is determined that the business result of the i-th tool call instruction is the second result, the decision score of the target tool is updated based on the new historical success rate of the target tool. Then, it is determined that the system updates the decision score sequence corresponding to K tools based on the new decision score of the target tool. After determining the new target tool based on the new decision score sequence, the tool call instruction corresponding to each remaining request intent is re-executed based on the new target tool, wherein each remaining request intent is a request intent among N request intents where the business operation was not executed or the business operation failed.

[0108] Optionally, the system updates the historical success rate of the target tool for the scenario type corresponding to the user request based on the following formula:

[0109] ;

[0110] In the above formula, This represents the tool's historical success rate before the M3 call. The result for this execution is set to 1 if the business result is the first result, and set to 0 if the business result is the second result. It is the forgetting factor, and the closer its value is to 1, the higher the weight of historical data.

[0111] For example: If the "Query Flight API" fails to execute in the "Flight Rescheduling" business scenario corresponding to the user request, only the corresponding business scenario of {Query API, Flight Rescheduling} in the tool will be downgraded. Values ​​and tools in other business scenarios (e.g., {query API, flight query}) The value remains unchanged.

[0112] Optionally, this application adopts a scenario-aware success rate update mechanism to avoid mutual interference between execution results in different business scenarios.

[0113] Optionally, after updating the historical success rate of the target tool in the scenario type corresponding to the user request based on the business result of the i-th tool call instruction, if it is determined that the business result of the i-th tool call instruction is the second result, the decision score of the target tool is updated based on the new historical success rate corresponding to the target tool. At this time, the update only affects the score of the replaced tool, while the decision scores of the other tools remain unchanged. The computational overhead is small, the system response is fast, and after the score sequence is reordered, the original low-scoring but highly reliable tool may jump to the new target tool due to this update. The selection process of the new target tool does not require manual intervention or re-recall, and does not require re-search, thus ensuring the efficiency and low latency of the heavy decision-making process.

[0114] Optionally, after determining the new target tool based on the new decision scoring sequence, the system re-executes the tool invocation instructions corresponding to each remaining request intent based on the new target tool. The system avoids invalid retries and reduces resource waste by only retrying failed or unexecuted intents and not repeating successful steps (such as "rescheduling" has been successful and will not be retried). At the same time, the re-decision and re-execution form a closed-loop fault tolerance mechanism. Even if a tool temporarily fails or data is abnormal, the system can still automatically switch to a reliable alternative tool based on historical performance, ensuring the continuity of business operation execution and improving the robustness of the business system and the reliability of user request processing.

[0115] As can be seen from the above, the two-stage cascading tool selection mechanism adopted in this application has the following technical advantages when dealing with large-scale, highly complex business scenarios:

[0116] 1. The three-dimensional weighted correction algorithm of the M3 module ensures high reliability and security of business decisions:

[0117] (1) Overcoming the subjective bias of LLM: by using traditional semantic ranking scores ( By integrating with objective data, the system avoids blindly relying on subjective reasoning from large language models.

[0118] (2) Ensure business reliability: Decision score It also depends on the accuracy of the recall. semantic accuracy ) and business reliability ( This ensures the reliability of the target tool ultimately selected by the system, and improves the user experience and the stability of the business system.

[0119] (3) Risk resistance: The system passes The mechanism has the ability to quickly identify and avoid tools that become unreliable due to unstable interfaces, frequent failures, or returning incorrect data.

[0120] 2. Closed-Loop Feedback and System Adaptability: The M4 module closed-loop feedback mechanism introduced in this solution enables the system to possess continuous, data-driven self-optimization capabilities.

[0121] (1) Real-time learning and adaptation: The M4 module corrects and updates the historical success rate in the DB in real time based on the actual execution results of the tool. .

[0122] (2) Optimization without human intervention: The dynamic updates enable the entire tool selection system to autonomously adapt to changes in the backend toolset without relying on manual retraining or adjustment of model parameters, thus achieving "tool adaptive optimization".

[0123] 3. Streamlined Processing Capabilities for Complex Business Scenarios: This solution, through the collaborative work of M3 and M4, can efficiently and accurately handle complex multi-step query tasks.

[0124] (1) Efficient sequence planning: The M3 module can generate a logically clear automatic execution sequence (e.g., [query]) based on user intent. [Filtering] ensures that multi-step query tasks can be executed correctly and in an orderly manner.

[0125] (2) Seamless automatic execution: The M4 module can automatically execute all query and filtering steps in the sequence until the final result is obtained, reducing unnecessary user interaction waiting time and improving process efficiency in pure query scenarios.

[0126] Example 2

[0127] This application embodiment can also provide a tool determining device. It should be noted that the tool determining device of this application embodiment can be used to execute the tool determining method provided in this application embodiment. The tool determining device provided in this application embodiment is described below.

[0128] According to an embodiment of this application, an apparatus for implementing the above-described tool determination method is also provided. Figure 4 This is a schematic diagram of an optional tool determining device according to an embodiment of this application, such as... Figure 4 As shown, the device includes: a recall quantity determination unit 401, a tool screening unit 402, a similarity determination unit 403, and a target tool determination unit 404.

[0129] Optionally, the recall quantity determination unit 401 is used to determine the recall quantity based on N request intents corresponding to the user request, where N is a positive integer, each request intent is used to represent a business operation that the user request needs to invoke, and there is a linear relationship between the recall quantity and the total number of N request intents; the tool filtering unit 402 is used to perform vector transformation on the user request to obtain a user request vector, and then filter the user request vector in a preset database to obtain K tools, where K is equal to the recall quantity, and the preset database includes tool information of all tools integrated into the large language model backend; similarity determination unit 40... 3. Used to obtain the recall similarity, semantic score, and historical success rate of each of the K tools. The recall similarity is the similarity between the user request vector and the tool vector of the tool. The semantic score is used to represent the confidence of the large language model in the tool. The historical success rate is the success rate of the tool being called in the scenario type corresponding to the user request. The target tool determination unit 404 is used to calculate the weighted sum of the recall similarity, semantic score, and historical success rate of each tool to obtain the decision score of each tool. Then, the tool with the highest decision score among the K tools is selected as the target tool corresponding to the user request.

[0130] As can be seen from the above, this device adopts a two-stage cascaded tool selection method. It dynamically calculates the recall quantity K based on the number of user request intents, and initially selects K tools from the preset database by combining the recall similarity matching results. Then, this device makes a comprehensive decision based on the three-dimensional weighted results corresponding to recall similarity, semantic score and historical success rate. This achieves the goal of accurately locking the target tool while greatly compressing the prompt word input context window of the large language model.

[0131] Therefore, this device avoids the high latency and high token overhead caused by loading the full tool description text in traditional technologies by separating the rapid recall and accurate decision-making stages of tools. At the same time, this device introduces the historical success rate of tools as one of the reliability dimensions of target tool decision-making, overcoming the low accuracy of target tool selection caused by relying solely on semantic matching in traditional technologies. This improves the response speed and execution success rate of intelligent agent systems in large-scale tool set environments, thereby achieving a technical effect that balances tool invocation efficiency and business reliability. In turn, it solves the technical problems of low efficiency and low accuracy in the selection and invocation of tools integrated into the large language model backend in existing technologies.

[0132] In an optional embodiment, the tool determination device further includes a semantic analysis unit and a request intent determination unit.

[0133] Optionally, the semantic analysis unit is used to perform semantic analysis on the user request based on preset analysis instructions using a large language model before determining the recall quantity based on the N request intents corresponding to the user request, and obtain N analysis results. Each analysis result is used to store a business operation that the user request needs to call in a preset format. The request intent determination unit is used to determine N request intents based on the N analysis results.

[0134] In one optional embodiment, the recall quantity determination unit 401 includes a product determination subunit and a quantity summation subunit.

[0135] Optionally, the product determination subunit is used to obtain the product of the total number of N request intents and the preset correction coefficient, wherein the preset correction coefficient is determined by backtracking the historical success rate of each tool in the preset database; the quantity summation subunit is used to sum the product, the preset basic recall quantity, and the preset redundant recall quantity to obtain the recall quantity.

[0136] In one alternative embodiment, the tool filtering unit 402 includes a similarity detection subunit and a tool filtering subunit.

[0137] Optionally, the similarity detection subunit is used to detect the similarity between the tool vector and the user request vector of each tool in the preset database, and obtain L recall similarities for L tools, where L is a positive integer greater than or equal to K, and the tool vector is used to characterize the functional description and parameter requirements of the tool; the tool filtering subunit is used to filter the tools ranked in the top K positions in the ranking results after sorting the L tools based on the L recall similarities, and obtain K tools.

[0138] In one optional embodiment, the tool determination device further includes: a metadata acquisition unit, a prompt word input unit, a confidence determination unit, and a normalization processing unit.

[0139] Optionally, the metadata acquisition unit is used to obtain the metadata corresponding to each of the K tools after filtering from a preset database based on the user request vector. The metadata includes at least the tool identifier, function description, and recall similarity of the tool. The prompt word input unit is used to generate target prompt words based on the K metadata corresponding to the K tools and the user request, and then input the target prompt words into the large language model. The confidence determination unit is used to determine the confidence of each tool and the business parameters corresponding to the user request based on the target prompt words through the large language model. The normalization processing unit is used to normalize the confidence of each tool to obtain the semantic score of each tool, and bind the business parameters corresponding to the user request to each tool.

[0140] In an optional embodiment, the tool determination device further includes: an intent type detection unit, an instruction sequence generation unit, and an instruction sequence invocation unit.

[0141] Optionally, the intent type detection device is used to detect the intent type corresponding to each of the N request intents after selecting the tool with the highest decision score among the K tools as the target tool corresponding to the user request; the instruction sequence generation unit is used to generate a target instruction sequence based on the intent type corresponding to each request intent and the business parameters bound to the target tool through the target tool, wherein the target instruction sequence includes at least N tool call instructions, each tool call instruction is used to implement a business operation corresponding to a request intent; and the instruction sequence call unit is used to execute the N tool call instructions in the order of the instructions corresponding to the target instruction sequence.

[0142] In one alternative embodiment, the instruction sequence invocation unit includes a detection subunit and a historical success rate update subunit.

[0143] Optionally, the detection subunit is used to detect the business result of the i-th tool call instruction after the i-th tool call instruction among N tool call instructions has been executed, wherein the business result is one of the following: a first result, used to indicate that the business operation is successful; a second result, used to indicate that the business operation is unsuccessful; a third result, used to indicate that the system facility or interface has failed; and the historical success rate update subunit is used to update the historical success rate of the target tool under the scenario type corresponding to the user request based on the business result of the i-th tool call instruction if the business result of the i-th tool call instruction is the first result / second result.

[0144] In an optional embodiment, the tool determination device further includes: a decision scoring update unit, a scoring sequence update unit, and a re-execution unit.

[0145] Optionally, the decision scoring update unit is used to update the historical success rate of the target tool under the scenario type corresponding to the user request based on the business result of the i-th tool call instruction, and if the business result of the i-th tool call instruction is the second result, update the decision score of the target tool based on the new historical success rate of the target tool; the scoring sequence update unit is used to update the decision scoring sequence corresponding to K tools based on the new decision score of the target tool; the re-execution unit is used to re-execute the tool call instruction corresponding to each remaining request intent based on the new target tool after determining the new target tool based on the new decision scoring sequence, wherein each remaining request intent is a request intent among N request intents where the business operation was not executed / the business operation failed.

[0146] It should be noted that the recall quantity determination unit 401, tool screening unit 402, similarity determination unit 403 and target tool determination unit 404 mentioned above correspond to steps S101 to S104 in the method embodiment. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in the above embodiment.

[0147] Example 3

[0148] Embodiments of this application can also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application, such as... Figure 5 As shown, the electronic device includes: one or more ( Figure 5 (Only one is shown) processor 502, memory 504, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0149] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, to implement the above-mentioned tool determination method.

[0150] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, which can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0151] The processor can access information and applications stored in memory via a transmission device to execute the following steps: First, determine the recall quantity based on N request intents corresponding to the user request, where N is a positive integer, each request intent represents a business operation that the user request needs to invoke, and the recall quantity is linearly related to the total number of N request intents. Second, after vectorizing the user request to obtain a user request vector, filter the user request vector in a pre-defined database to obtain K tools, where K equals the recall quantity, and the pre-defined database includes tool information for all tools integrated into the large language model backend. Third, obtain the recall similarity, semantic score, and historical success rate for each of the K tools, where the recall similarity is the similarity between the user request vector and the tool vector, the semantic score represents the large language model's confidence in the tool, and the historical success rate is the success rate of the tool being invoked in the scenario type corresponding to the user request. Fourth, after weighted summing of the recall similarity, semantic score, and historical success rate for each tool to obtain a decision score for each tool, select the tool with the highest decision score among the K tools as the target tool corresponding to the user request.

[0152] The processor can access the information and application stored in the memory via the transmission device to perform the following steps: Before determining the recall quantity based on N request intents corresponding to the user request, semantic analysis of the user request is performed using a large language model based on preset analysis instructions to obtain N analysis results, wherein each analysis result is used to store a business operation that the user request needs to invoke in a preset format; N request intents are determined based on the N analysis results.

[0153] The processor can access information and applications stored in memory via a transmission device to perform the following steps: obtain the product of the total number of N request intents and a preset correction coefficient, wherein the preset correction coefficient is determined by backtesting the historical success rate of each tool in a preset database; sum the product, the preset base recall, and the preset redundant recall to obtain the recall number.

[0154] The processor can access the information and application stored in the memory via the transmission device to perform the following steps: detect the similarity between the tool vector and the user request vector of each tool in the preset database to obtain L recall similarities for L tools, where L is a positive integer greater than or equal to K, and the tool vector is used to characterize the tool's functional description and parameter requirements; after sorting the L tools based on the L recall similarities, filter the tools ranked in the top K positions in the sorting results to obtain K tools.

[0155] The processor can access information and applications stored in memory via a transmission device to perform the following steps: After filtering K tools from a pre-defined database based on user request vectors, the processor obtains metadata for each of the K tools, where the metadata includes at least the tool identifier, functional description, and recall similarity; After generating target prompt words based on the K metadata corresponding to the K tools and the user request, the target prompt words are input into a large language model; The large language model determines the confidence level of each tool and the business parameters corresponding to the user request based on the target prompt words; The confidence level of each tool is normalized to obtain a semantic score for each tool, and the business parameters corresponding to the user request are bound to each tool.

[0156] The processor can access information and applications stored in memory via a transmission device to perform the following steps: After selecting the tool with the highest decision score among K tools as the target tool corresponding to the user request, detect the intent type corresponding to each of the N request intents; generate a target instruction sequence through the target tool based on the intent type corresponding to each request intent and the business parameters bound to the target tool, wherein the target instruction sequence includes at least N tool invocation instructions, each tool invocation instruction being used to implement a business operation corresponding to a request intent; and execute the N tool invocation instructions sequentially based on the instructions corresponding to the target instruction sequence.

[0157] The processor can access information and applications stored in memory via a transmission device to perform the following steps: After the i-th tool call instruction out of N tool call instructions is executed, the business result of the i-th tool call instruction is detected, wherein the business result is one of the following: a first result, indicating that the business operation was successful; a second result, indicating that the business operation failed; a third result, indicating that the system facility or interface failed; if the business result of the i-th tool call instruction is the first result / second result, the historical success rate of the target tool in the scenario type corresponding to the user request is updated based on the business result of the i-th tool call instruction.

[0158] The processor can invoke information and applications stored in memory via a transmission device to perform the following steps: After updating the historical success rate of the target tool under the scenario type corresponding to the user request based on the business result of the i-th tool invocation instruction, if the business result of the i-th tool invocation instruction is the second result, update the decision score of the target tool based on the new historical success rate corresponding to the target tool; update the decision score sequence corresponding to K tools based on the new decision score corresponding to the target tool; after determining a new target tool based on the new decision score sequence, re-execute the tool invocation instruction corresponding to each remaining request intent based on the new target tool, wherein each remaining request intent is a request intent among N request intents where the business operation was not executed / the business operation failed.

[0159] This application provides a tool selection scheme. It employs a two-stage cascaded tool selection approach. First, it dynamically calculates the recall quantity K based on the number of user request intents. Then, it initially filters out K tools from a pre-set database based on recall similarity matching results. Finally, it makes a comprehensive decision based on a three-dimensional weighted result corresponding to recall similarity, semantic score, and historical success rate. This approach achieves the goal of accurately identifying the target tool while significantly compressing the prompt word input context window of a large language model.

[0160] Therefore, this application avoids the high latency and high token overhead caused by loading the full tool description text in traditional technologies by separating the rapid recall and accurate decision-making stages of tools. At the same time, this application introduces the historical success rate of tools as one of the reliability dimensions of target tool decision-making, overcoming the low accuracy of target tool selection caused by relying solely on semantic matching in traditional technologies. This improves the response speed and execution success rate of intelligent agent systems in large-scale toolset environments, thereby achieving a technical effect that balances tool invocation efficiency and business reliability. In turn, it solves the technical problems of low efficiency and low accuracy in the selection and invocation of tools integrated into the backend of large language models in existing technologies.

[0161] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, PDAs, mobile internet devices, PADs, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0162] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0163] Example 4

[0164] Embodiments of this application may also provide a storage medium.

[0165] Optionally, in this embodiment of the application, the storage medium can be used to store the program code executed by the tool determined by the method provided in the above method embodiment.

[0166] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0167] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform a tool to determine method steps.

[0168] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0169] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0174] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for determining a tool, characterized in that, include: The recall quantity is determined based on N request intents corresponding to a user request, where N is a positive integer, each request intent is used to characterize a business operation that needs to be invoked by the user request, and there is a linear relationship between the recall quantity and the total number of the N request intents; After performing vector transformation on the user request to obtain the user request vector, the user request vector is filtered in a preset database to obtain K tools, where K is equal to the recall number, and the preset database includes tool information of all tools integrated into the large language model backend. Obtain the recall similarity, semantic score, and historical success rate for each of the K tools. The recall similarity is the similarity between the user request vector and the tool vector. The semantic score is used to characterize the confidence of the large language model in the tool. The historical success rate is the success rate of the tool being invoked in the scenario type corresponding to the user request. After weighted summing of the recall similarity, semantic score, and historical success rate for each tool to obtain the decision score for each tool, the tool with the highest decision score among the K tools is selected as the target tool corresponding to the user request.

2. The tool determination method according to claim 1, characterized in that, Before determining the recall quantity based on the N request intents corresponding to the user request, the tool's determination method further includes: The user request is semantically analyzed using a large language model based on preset analysis instructions to obtain N analysis results. Each analysis result is used to store a business operation that the user request needs to invoke in a preset format. The N request intentions are determined based on the N analysis results.

3. The tool determination method according to claim 1, characterized in that, The recall quantity is determined based on N request intents corresponding to the user's request, including: Obtain the product of the total number of the N request intents and a preset correction coefficient, wherein the preset correction coefficient is determined by backtesting the historical success rate of each tool in the preset database; The product, the preset basic recall quantity, and the preset redundant recall quantity are summed to obtain the recall quantity.

4. The tool determination method according to claim 1, characterized in that, Based on the user request vector, K tools are obtained by filtering in a preset database, including: The similarity between the tool vector of each tool in the preset database and the user request vector is detected to obtain L recall similarities for L tools, where L is a positive integer greater than or equal to K, and the tool vector is used to characterize the functional description and parameter requirements of the tool; After sorting the L tools based on the L recall similarities, the tools ranked in the top K positions in the sorting results are selected to obtain the K tools.

5. The tool determination method according to claim 1, characterized in that, After filtering through a preset database based on the user request vector to obtain K tools, the tool determination method further includes: Obtain the metadata corresponding to each of the K tools, wherein the metadata includes at least the tool identifier, function description and recall similarity of the tool; After generating target prompt words based on the K metadata corresponding to the K tools and the user request, the target prompt words are input into the large language model; The confidence level of each tool and the business parameters corresponding to the user request are determined by the large language model based on the target prompt words. The confidence level of each tool is normalized to obtain a semantic score for each tool, and the business parameters corresponding to the user request are bound to each tool.

6. The tool determination method according to claim 1, characterized in that, After selecting the tool with the highest decision score among the K tools as the target tool corresponding to the user request, the tool determination method further includes: Detect the intent type corresponding to each of the N request intents; The target tool generates a target instruction sequence based on the intent type corresponding to each request intent and the business parameters bound to the target tool. The target instruction sequence includes at least N tool invocation instructions, each of which is used to implement a business operation corresponding to a request intent. The N tool call instructions are executed sequentially based on the instruction sequence corresponding to the target instruction sequence.

7. The tool determination method according to claim 6, characterized in that, The N tool call instructions are executed sequentially based on the instruction sequence corresponding to the target instruction sequence, including: After the i-th tool invocation instruction among the N tool invocation instructions is executed, the business result of the i-th tool invocation instruction is detected, wherein the business result is one of the following: The first result is used to indicate that the business operation was successful. The second result is used to characterize business operation failures; The third result is used to characterize system facilities or interface failures. If the business result of the i-th tool invocation instruction is the first result / the second result, the historical success rate of the target tool under the scenario type corresponding to the user request is updated based on the business result of the i-th tool invocation instruction.

8. The tool determination method according to claim 7, characterized in that, After updating the historical success rate of the target tool under the scenario type corresponding to the user request based on the business result of the i-th tool invocation instruction, the tool determination method further includes: If the business result of the i-th tool invocation instruction is the second result, the decision score of the target tool is updated based on the new historical success rate corresponding to the target tool; Update the decision score sequence corresponding to the K tools based on the new decision score corresponding to the target tool; After determining a new target tool based on the new decision scoring sequence, the tool invocation instruction corresponding to each remaining request intent is re-executed based on the new target tool, wherein each remaining request intent is a request intent among the N request intents where the business operation was not executed or the business operation failed.

9. A tool determining device, characterized in that, include: The recall quantity determination unit is used to determine the recall quantity based on N request intents corresponding to the user request, where N is a positive integer, each request intent is used to characterize a business operation that needs to be invoked by the user request, and the recall quantity is linearly related to the total number of the N request intents; The tool filtering unit is used to perform vector transformation on the user request to obtain the user request vector, and then filter the user request vector in a preset database to obtain K tools, where K is equal to the recall number, and the preset database includes tool information of all tools integrated into the large language model backend. The similarity determination unit is used to obtain the recall similarity, semantic score and historical success rate of each of the K tools. The recall similarity is the similarity between the user request vector and the tool vector of the tool. The semantic score is used to characterize the confidence of the large language model in the tool. The historical success rate is the success rate of the tool being called in the scenario type corresponding to the user request. The target tool determination unit is used to obtain the decision score for each tool by weighted summing of the recall similarity, semantic score and historical success rate corresponding to each tool, and then select the tool with the highest decision score among the K tools as the target tool corresponding to the user request.

10. An electronic device, characterized in that, It includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the tool determination method according to any one of claims 1 to 8.

Images