A method for evaluating the accuracy of large model tool calls

Abstract

The present invention discloses a method for evaluating the accuracy of calling a large model tool, which relates to the field of artificial intelligence technology. Step S1: obtaining a large model output result obtained after a user requests a task to be input into a large model, performing tool evaluation based on the task called by the tool in the large model output result, and obtaining a tool label corresponding to the large model tool; Step S2: designing corresponding calling indicators for the tool calling task based on different application scenarios, and obtaining tool calling evaluation information based on the calling indicators; calling labels, filtering user calling tools based on similarity matching of tool labels corresponding to user request tasks; by evaluating the large model tool calling class process and outputting corresponding indicators, not only can the performance of the model be improved, but also the user experience can be optimized, the stability and robustness of the system can be enhanced, and the further optimization of the model and the improvement of generalization ability can be guided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large model tool calling accuracy assessment method. Background Art

[0002] The large language model is trained with a large amount of text data, showing strong language comprehension and reasoning capabilities, as well as a wide range of world knowledge. However, there are still many problems in putting it into practical use. One of the most important problems is how to reduce the errors in the content it generates. There are two scenarios where errors are most likely to occur:

[0003] For knowledge that is not included in the training data set of the large language model (such as knowledge after the training data set deadline or private knowledge of customers in actual business), directly calling the large language model cannot generate relevant content; for many professional problems (such as mathematical calculations, etc.), directly calling the content generated by the large language model often results in errors;

[0004] For the above problems, due to the strong reasoning ability and extensive knowledge demonstrated by the large language model, it is possible to use various tools to solve these problems. However, due to the current lack of large language model capabilities and the imperfect environment system of related calling tools, the accuracy of large language model calling tools is currently low and needs to be optimized and improved. However, most of the indicators currently used to improve accuracy are end-to-end indicators, which are usually QA (question and answer) tasks for some data sets. These indicators cannot help us accurately analyze and locate the internal problems and defects when the current large language model calls tools.

[0005] In view of this, the present invention proposes a large model tool calling accuracy assessment method. Summary of the invention

[0006] The purpose of the present invention is to provide a large model tool call accuracy assessment method that can accurately analyze and locate the internal problems and defects when the current large language model call tool is used.

[0007] In a first aspect, the present invention provides a method for evaluating the accuracy of calling a large model tool, comprising the following steps:

[0008] Step S1: Obtain the output result of the big model obtained after the user requests a task to be input into the big model, perform tool evaluation based on the task called by the tool in the output result of the big model, and obtain the tool label corresponding to the big model tool;

[0009] Step S2: design corresponding calling indicators for tool calling tasks based on different application scenarios, and obtain tool calling evaluation information based on the calling indicators;

[0010] Step S3: based on the tool call evaluation information, the tool call tags in the user request database are updated, and the user call tools are filtered based on the similarity matching of the tool tags corresponding to the user request tasks.

[0011] As a preferred solution of the present invention, the tool tag includes a tool call tag, a negative tool call tag and an exit tag;

[0012] Wherein: the tool call tag is the tool tag that needs to be called;

[0013] A negative tool call label is a tool label when no tool call is made;

[0014] The exit tag is a tool tag that exits directly without any tool call in the tool call process.

[0015] As a preferred solution of the present invention, the evaluation logic of the tool evaluation is:

[0016] Analyze user request tasks and corresponding prediction tool labels based on historical data sets, and use an evaluator to evaluate the prediction similarity between the prediction tool labels and the real tool labels;

[0017] The historical data set includes user request tasks and corresponding tool tags.

[0018] As a preferred solution of the present invention, the logic for obtaining the tool tag is:

[0019] Directly obtain tasks based on user requests;

[0020] Obtained from the call result module based on historical user request tasks.

[0021] As a preferred solution of the present invention, the application scenarios include tool calling scenarios, negative sample exit scenarios and direct exit scenarios;

[0022] Among them: tool call scenarios include but are not limited to SRSC single-round single tool call, SRMC single-round multiple tool calls, MRSC multiple-round single tool call and MRMC multiple-round multiple tool calls;

[0023] The call indicators include the total number of samples in the application scenario, the number of correct tool selection predictions, the number of incorrect tool selection predictions, the number of exits, the number of tool call format errors, and the similarity;

[0024] The evaluator evaluates the similarity between the predicted tool label and the true tool label and outputs the call metric.

[0025] In a second aspect, the present invention provides an electronic device, the electronic device comprising:

[0026] at least one processor;

[0027] a storage device communicatively connected to the at least one processor, for storing a program,

[0028] When the program is executed by the at least one processor, the at least one processor implements the large model tool call accuracy assessment method described in the first aspect of the present invention.

[0029] In a third aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor in an electronic device, implements a large model tool call accuracy assessment method as described in the first aspect of the present invention.

[0030] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0031] The present invention can not only improve the performance of the model, but also optimize the user experience, enhance the stability and robustness of the system, and guide the further optimization of the model and the improvement of its generalization ability by evaluating the large model tool call class process and outputting corresponding indicators;

[0032] Accurate tool calling and parameter generation can reduce the number of times users manually adjust parameters or select tools, thereby improving the efficiency of user interaction with the system and enhancing the user experience; by improving the negative sample recognition rate, the system can more accurately determine when it is not necessary to call a tool, thereby avoiding unnecessary calculations and waste of resources, enhancing the stability and robustness of the system, and optimizing the control of the entire process and improving the response speed of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0034] Figure 1 It is a flow chart of a large model tool calling accuracy assessment method of the present invention;

[0035] Figure 2 It is the tool of the present invention calling the evaluation information graph;

[0036] Figure 3 It is a schematic diagram of the structure of the electronic equipment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The current tool calling process includes the large language model analysis module LLM, the tool calling module Tool and the calling result module Observation:

[0039] The large language model analysis module LLM is the output result of the large language model excluding the content of the tool call module;

[0040] The tool calling module Tool is the content of the tool calling module in the output result of the large language model, which includes the selection of the tool and the generation of parameters;

[0041] The call result module Observation is the result generated after calling the tool;

[0042] During the tool evaluation process, the large language model mainly acts on the tool call module Tool for user request tasks of user tool calls, and obtains the corresponding tool call label in the tool call module Tool; the large language model analysis module LLM is directly generated by the large language model and does not include tool call information, while the call result module Observation is generated by the tool call module Tool and has nothing to do with the capabilities of the large language model, so we ignore these two parts in the evaluation.

[0043] In addition to the tool call module Tool, since the original dataset only contains tool call labels, we also add negative tool call labels when no tool call is made; we insert exit labels after each tool call process and evaluate the exit labels as part of the tool. When there is no tool call at the beginning of the tool call process but it exits directly, we will perform additional statistics.

[0044] Example 1

[0045] See also Figure 1 As shown, a large model tool calling accuracy assessment method described in this embodiment includes the following steps:

[0046] Step S1: Obtain the output result of the big model obtained after the user requests a task to be input into the big model, perform tool evaluation based on the task called by the tool in the output result of the big model, and obtain the tool label corresponding to the big model tool;

[0047] It should be noted that: the user request task represents the purpose of the user input, and the calling of the large model tool is determined based on the user input purpose. If the calling tool required by the user can be obtained accurately and quickly, it will not only save the user's time, but also improve the user experience;

[0048] The tool tags include a tool call tag, a negative tool call tag, and an exit tag;

[0049] Wherein: the tool call tag is the tool tag that needs to be called;

[0050] A negative tool call label is a tool label when no tool call is made;

[0051] The exit tag is a tool tag that exits directly without any tool call in the tool call process.

[0052] The evaluation logic of the tool evaluation is:

[0053] Based on the historical data set, the user request tasks and the corresponding predicted tool labels are analyzed, and the similarity between the predicted tool labels and the real tool labels is evaluated by the evaluator.

[0054] The historical data set includes user request tasks and corresponding tool tags.

[0055] Specifically, it should be noted that: the tool tag corresponding to the large model tool obtained in step S1 is directly obtained based on the user request task on the one hand, and is obtained from the call result module Observation obtained based on the historical user request task on the other hand;

[0056] After obtaining the corresponding tool label based on the user request task, the user request task and the corresponding tool label are input into the tool calling module, and then passed into the evaluator based on the tool calling module. The evaluator evaluates the similarity between the predicted tool label and the real tool label, and outputs the calling index.

[0057] Step S2: design corresponding calling indicators for tool calling tasks based on different application scenarios, and obtain tool calling evaluation information based on the calling indicators;

[0058] Specific as Figure 2 As shown, the labels in the first row are the indicators of tool calls, and the first column is the classification of different application scenarios.

[0059] The application scenarios include tool calling scenarios, negative sample exit scenarios and direct exit scenarios;

[0060] Specifically, the tool call scenario is marked as TOTAL_TOOL; the negative sample exit scenario is marked as EXIT, which is the application scenario of the negative tool call label; the direct exit scenario is marked as SIMPLE_EXIT, which is the scenario of direct exit without triggering tool call in a round of dialogue, that is, the scenario with only EXIT in this tool call chain;

[0061] Among them: tool call scenarios include but are not limited to SRSC single-round single tool call, SRMC single-round multiple tool calls, MRSC multiple-round single tool call and MRMC multiple-round multiple tool calls;

[0062] Specifically, SRSC (Single Round Single Call) refers to a single tool call in a single round, which is a scenario where the first round of dialogue in a multi-round dialogue sample triggers the first tool call;

[0063] SRMC (Single Round Multi Call) refers to multiple tool calls in a single round, which is a scenario where tool calls are continuously triggered after the first tool call in the first round of dialogue in a multi-round dialogue sample;

[0064] MRSC (Multi Round Single Call) refers to multi-round single tool calls, which is a scenario in which the first tool call in a conversation round is triggered in a subsequent conversation round in a multi-round conversation sample;

[0065] MRMC (Multi Round Multi Call) refers to multi-round tool calls, which is a scenario in which a tool call other than the first one in a multi-round conversation is triggered in a subsequent conversation round in a multi-round conversation sample;

[0066] It should be noted that the four scene faces corresponding to the tool call scene are positive samples for tool call, not negative sample exit scenes or direct exit scenes.

[0067] The call indicators include the total number of samples in the application scenario, the number of correct tool selection predictions, the number of incorrect tool selection predictions, the number of exits, the number of tool call format errors, and the similarity;

[0068] Specifically, the total number of samples in the corresponding application scenario is marked as NUMBER;

[0069] The correct number of tool selection predictions is marked as RIGHT;

[0070] The number of tool selection prediction errors is marked as WRONG. It should be noted here that it is the number of tool selection prediction errors but not direct exits;

[0071] The exit number is labeled as EXIT, where the exit number is the number of exits predicted by the tool selection;

[0072] The tool call format error number is marked as FORMAT_ERROR;

[0073] The similarity is marked as BLEU(RIGHT), which is specifically expressed as the bleu (Bilingual Evaluation Understudy) similarity of the input parameters when the tool selection prediction is successful.

[0074] Step S3: based on the tool call evaluation information, the tool call tags in the user request database are updated, and the user call tools are filtered based on the similarity matching of the tool tags corresponding to the user request tasks.

[0075] Example 2

[0076] See also Figure 3 As shown, the part not described in detail in this embodiment can be found in the description of Embodiment 1. This embodiment describes an electronic device, which includes:

[0077] at least one processor;

[0078] The storage device is communicatively connected to the at least one processor and is used to store programs.

[0079] When the program is executed by the at least one processor, the at least one processor implements the large model tool call accuracy assessment method described in the first aspect of the present invention.

[0080] The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) and one or more memories, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement a large model tool call accuracy assessment method provided by the above-mentioned various method embodiments. The electronic device may also include other components for implementing the functions of the device, for example, the electronic device may also have components such as a wired or wireless network interface and an input and output interface for input and output. The embodiments of this application are not described in detail here.

[0081] In an exemplary embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the program is executed by a processor in an electronic device, a large model tool call accuracy assessment method described in the first aspect of the present invention is implemented.

[0082] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0083] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0084] A person of ordinary skill in the art will understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0085] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0086] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0087] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0088] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0089] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A large model tool call accuracy assessment method, characterized in that: The following steps are involved: Step S1: Obtain the output result of the big model obtained after the user requests a task to be input into the big model, perform tool evaluation based on the task called by the tool in the output result of the big model, and obtain the tool label corresponding to the big model tool; The tool tags include a tool call tag, a negative tool call tag, and an exit tag; Wherein: the tool call tag is the tool tag that needs to be called; A negative tool call label is a tool label when no tool call is made; The exit tag is the tool tag of a tool that exits directly without any tool call in the tool call process; Step S2: design corresponding calling indicators for tool calling tasks based on different application scenarios, and obtain tool calling evaluation information based on the calling indicators; Step S3: updating the tool call tags in the user request database based on the tool call evaluation information, and filtering the user call tools based on the similarity matching of the tool tags corresponding to the user request tasks.

2. A large model tool calling accuracy assessment method according to claim 1, characterized in that: The evaluation logic of the tool evaluation is: Analyze user request tasks and corresponding prediction tool labels based on historical data sets, and use an evaluator to evaluate the prediction similarity between the prediction tool labels and the real tool labels; The historical data set includes user request tasks and corresponding tool tags.

3. A large model tool calling accuracy assessment method according to claim 1, characterized in that: The logic for obtaining the tool tag is as follows: Directly obtain tasks based on user requests; Obtained from the call result module based on historical user request tasks.

4. A large model tool calling accuracy assessment method according to claim 1, characterized in that: The application scenarios include tool calling scenarios, negative sample exit scenarios and direct exit scenarios; Among them: tool call scenarios include but are not limited to SRSC single-round single tool call, SRMC single-round multiple tool calls, MRSC multiple-round single tool call and MRMC multiple-round multiple tool calls; The call indicators include the total number of samples in the application scenario, the number of correct tool selection predictions, the number of incorrect tool selection predictions, the number of exits, the number of tool call format errors, and the similarity; The evaluator evaluates the similarity between the predicted tool labels and the true tool labels and outputs the call metrics.

5. An electronic device, characterized in that: include: at least one processor; A storage device communicatively connected to the at least one processor is used to store a program, and when the program is executed by the at least one processor, the at least one processor implements a large model tool call accuracy assessment method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor in an electronic device, a large model tool calling accuracy assessment method as described in any one of claims 1-4 is implemented.

Images