Large model tool calling multi-dimensional performance evaluation method and system

Abstract

The invention discloses a large model tool calling multi-dimensional performance evaluation method and system, and relates to the technical field of artificial intelligence and natural language processing, and the method achieves the comprehensive evaluation of the calling capability of a large model tool through the four core steps of data preparation, data standardization processing, multi-dimensional quantitative evaluation and result output. The method specifically comprises the steps of constructing a standard test data set and obtaining model output, analyzing and normalizing tool calling information, carrying out quantitative evaluation from multiple dimensions of function name matching, parameter name matching, parameter value semantic matching and unauthorized tool calling detection, and finally generating an automatic comparison report. According to the method, the problems of single dimension, semantic understanding deficiency, dependence on manual annotation and insufficient security management and control of an existing assessment method are solved, the method has the advantages of multi-dimensional interpretability, semantic level assessment accuracy, security enhancement and full-automatic efficient evaluation, and the method can be widely applied to transverse comparison and optimization iteration of large model tool calling capability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, and in particular to a method and system for multi-dimensional performance evaluation of large model tools. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, expanding the functional boundaries of models through toolcall capabilities has become a mainstream trend in the industry. The toolcall capability of large models directly determines their effectiveness in practical application scenarios. However, the current field of performance evaluation for large model toolcalls lacks systematic and standardized solutions, and existing evaluation methods have many significant shortcomings.

[0003] The core problems with existing evaluation methods include: First, the lack of a unified standardized evaluation framework makes it difficult to conduct fair and effective horizontal comparisons of the performance of different models. Second, the evaluation dimensions are too narrow, mostly focusing only on the accuracy of tool selection, ignoring key evaluation dimensions such as parameter name matching accuracy, parameter value accuracy, and unauthorized tool calls, thus failing to comprehensively reflect the model's tool call capabilities. Third, the semantic understanding capability is lacking; parameter values ​​rely heavily on simple literal matching and cannot handle scenarios such as Boolean equivalence (e.g., "true" and True), differences in number format (e.g., "100" and 100.0), and semantic equivalence in natural language descriptions (e.g., "takes 30 minutes" and "requires half an hour"), resulting in insufficient accuracy of evaluation results. Fourth, the security control mechanism is imperfect, lacking quantitative monitoring methods for unauthorized tool calls, and the model may have security risks such as calling disabled APIs. Fifth, reliance on manual annotation makes it difficult to achieve large-scale automated evaluation, resulting in low evaluation efficiency and high costs.

[0004] Therefore, there is an urgent need for a large-scale model tool to call upon multi-dimensional performance evaluation methods and systems to solve existing technical problems. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art, and proposes a method and system for multi-dimensional performance evaluation of large model tools.

[0006] In a first aspect, embodiments of the present invention provide a method for calling multi-dimensional performance evaluation of large model tools, including:

[0007] Construct a standard test dataset and collect tool call outputs from at least two large language models to be evaluated on the standard test dataset.

[0008] The tool call output of each large language model to be evaluated is analyzed to extract function names, parameter names, and parameter values; the extracted function names, parameter names, and parameter values ​​are then standardized.

[0009] Based on the standardized function names, parameter names, and parameter values, a multi-dimensional evaluation is performed, specifically including unauthorized tool call detection evaluation, function name matching evaluation, parameter name matching evaluation, and numerical semantic matching evaluation.

[0010] Based on the multi-dimensional evaluation results, a model comparison report is generated, providing detailed error analysis and visualizing the evaluation results.

[0011] Furthermore, the standard test dataset contains multiple test samples, each of which includes input data, a list of authorized tools, and the corresponding standard tool call output;

[0012] Furthermore, the extracted function names, parameter names, and parameter values ​​are standardized. Specific methods include: sorting the extracted parameter names in alphabetical order to eliminate order differences, and recursively normalizing the parameter values.

[0013] Furthermore, the detection and evaluation of unauthorized tool calls includes the following specific methods: comparing the function names output by the large language model to be evaluated with the list of authorized tools in the standard test dataset, counting the number of unauthorized tool calls and the total number of calls, and calculating the unauthorized tool call rate, wherein the unauthorized tool call rate = number of unauthorized tool calls / total number of calls.

[0014] Furthermore, the function name matching evaluation method includes: calculating the function name recall and function name precision respectively, wherein the function name recall = number of correctly predicted functions / total number of standard data functions, and the function name precision = number of correctly predicted functions / total number of model output functions.

[0015] Furthermore, the parameter name matching evaluation method includes: grouping by function name, statistically analyzing the intersection and matching of parameter names output by the large language model under evaluation and parameter names output by standard tool calls, and quantifying the completeness and accuracy of parameter names.

[0016] Furthermore, the specific methods for evaluating numerical semantic matching include: using a hierarchical matching strategy to determine the semantic equivalence of parameter values, including: performing type adaptive conversion for Boolean values ​​and numbers to achieve equivalent matching; ignoring element order differences for list and dictionary types to achieve equivalent matching; and using a pre-trained BERT model to calculate the semantic similarity of parameter values ​​in natural language description classes to determine semantic equivalence.

[0017] Furthermore, based on the multi-dimensional evaluation results, a model comparison report is generated, outputting detailed error analysis and visualizing the evaluation results. Specific methods include: generating a multi-model comparison report, which includes indicators such as function name accuracy, function name recall, parameter name accuracy, parameter name recall, parameter value accuracy, parameter value recall, and unauthorized tool misuse rate, and outputting detailed error analysis and visualizing the evaluation results.

[0018] Secondly, this invention also discloses a large model tool invocation multi-dimensional performance evaluation system, including a data preparation module, a data standardization processing module, a multi-dimensional evaluation module, and a result output module; wherein:

[0019] The data preparation module is used to construct a standard test dataset and collect tool call output results from at least two large language models to be evaluated on the standard test dataset.

[0020] The data standardization module is used to parse the tool call output results of each large language model to be evaluated, extract function names, parameter names, and parameter values; and standardize the extracted function names, parameter names, and parameter values.

[0021] The multi-dimensional evaluation module is used to perform multi-dimensional evaluation based on the standardized function name, parameter name, and parameter value, specifically including unauthorized tool call detection evaluation, function name matching evaluation, parameter name matching evaluation, and numerical semantic matching evaluation.

[0022] The results output module is used to generate model comparison reports based on multi-dimensional evaluation results, output detailed error analysis, and visualize the evaluation results.

[0023] Thirdly, an electronic device, characterized in that it comprises:

[0024] One or more processors;

[0025] Memory, used to store one or more programs;

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 8.

[0027] This invention discloses a method and system for multi-dimensional performance evaluation of large model tool invocation. The method achieves a comprehensive evaluation of the tool invocation capability of large models through four core steps: data preparation, data standardization, multi-dimensional quantitative evaluation, and result output. Specifically, it includes constructing a standard test dataset and obtaining model output, parsing and normalizing tool invocation information, conducting quantitative evaluation from multiple dimensions such as function name matching, parameter name matching, parameter value semantic matching, and unauthorized tool invocation detection, and finally generating an automated comparison report. This invention solves the problems of existing evaluation methods, such as single-dimensionality, lack of semantic understanding, reliance on manual annotation, and insufficient security control. It has the advantages of multi-dimensional interpretability, semantic-level evaluation accuracy, enhanced security, and fully automated and efficient evaluation, and can be widely applied to the horizontal comparison and optimization iteration of large model tool invocation capabilities.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. Strong multi-dimensional interpretability: Through separate evaluation of multiple independent dimensions such as function name, parameter name, parameter value, and unauthorized tool calls, it is possible to accurately locate the specific defects of the model in the tool call process (such as distinguishing between "function name prediction error", "parameter name omission" and "parameter value semantic deviation"), providing clear directional guidance for model optimization.

[0030] 2. Improved accuracy through semantic-level evaluation: The innovative hierarchical matching strategy achieves semantic equivalence determination of parameter values. Combined with type adaptive conversion and semantic similarity calculation of the BERT model, it solves the semantic equivalence scenarios that traditional literal matching cannot handle. Test data shows that the false alarm rate is reduced by 40%, and the evaluation results are more in line with actual application scenarios.

[0031] 3. Outstanding security control capabilities: The addition of an unauthorized tool call detection dimension quantifies the security compliance of the model, effectively blocking high-risk operations such as unauthorized access to the database and calls to disabled APIs, and reducing the security risks of large model tool calls.

[0032] 4. Fully automated and efficient evaluation: Adopting the JSONL standardized data format, an end-to-end automated evaluation pipeline is built, eliminating the need for manual annotation, supporting large-scale testing at the level of thousands of samples, and improving evaluation efficiency by 10 times compared to traditional manual methods, significantly reducing evaluation costs. Attached Figure Description

[0033] Figure 1 A flowchart illustrating the process of a large model tool calling a multi-dimensional performance evaluation method, provided in an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the tool call performance evaluation process provided in an embodiment of the present invention;

[0035] Figure 3 A structural block diagram of a large model tool calling a multi-dimensional performance evaluation system provided in an embodiment of the present invention;

[0036] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0038] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0039] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0040] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0041] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0042] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0043] In related technologies, a method for evaluating the accuracy of large-scale tool calls has been disclosed. This method classifies user requests by tool tags (call / negative call / exit) and statistically analyzes metrics such as the number of samples and accuracy in different scenarios (single-round / multi-round calls). However, this method only evaluates the tool selection stage and does not cover parameter generation accuracy (such as missing parameter names or incorrect values); it relies on manually labeled "real tool tags" and cannot automate the processing of large-scale test data; and it lacks the quantification of security risks associated with unauthorized calls (such as calling disabled APIs).

[0044] Among related technologies, a GTA (General Tool Agents) benchmark has also been disclosed, which evaluates tool invocation capabilities end-to-end through real-world tasks (such as multimodal inputs and complex toolchains). However, it only outputs the final task completion rate and cannot pinpoint specific error points (such as parameter passing errors), and it also relies on manual annotation of the toolchain, resulting in high evaluation costs.

[0045] To address at least one of the technical problems existing in the aforementioned related technologies, this invention provides a method and system for calling multi-dimensional performance evaluation using large model tools.

[0046] This invention provides a method for calling multi-dimensional performance evaluation in large model tools, such as... Figure 1 ,include:

[0047] S100. Construct a standard test dataset and collect tool call output results of at least two large language models to be evaluated on the standard test dataset; in this embodiment, the standard test dataset contains multiple test samples, each of which includes input data, a list of authorized tools and corresponding standard tool call output.

[0048] Specifically, a standard test dataset is constructed using the JSONL standardized format. This dataset contains multiple test samples, each including input data (such as a user query prompt), a list of authorized tools (clearly defining the scope of tools the model can call), and corresponding standard tool call outputs (containing standard information such as function names, parameter names, and parameter values). Tool call outputs from at least two large language models to be evaluated are collected on this standard test dataset to ensure that all models are evaluated based on the same test data, guaranteeing fairness and comparability in the evaluation.

[0049] S200. Parse the tool call output of each large language model to be evaluated, and extract the function name, parameter name, and parameter value; standardize the extracted function name, parameter name, and parameter value; in this embodiment, the standardization of the extracted function name, parameter name, and parameter value includes: sorting the extracted parameter name in alphabetical order to eliminate order differences, and recursively normalizing the parameter value.

[0050] Specifically, the tool call output of each large language model to be evaluated is parsed to extract core elements: function name, parameter name, and parameter value. To eliminate the interference of non-core differences on the evaluation results, the extracted parameter names are sorted in alphabetical order; the parameter values ​​are recursively normalized, specifically by converting string-format Boolean values ​​(such as "true", "True", "false", "False") into standard Boolean types (True / False), and string-format numbers (such as "100", "25.5") into their corresponding numeric types (100, 25.5), thus unifying the data format and ensuring the accuracy of subsequent evaluations.

[0051] S300. Based on the standardized function names, parameter names, and parameter values, perform a multi-dimensional evaluation, specifically including unauthorized tool call detection evaluation, function name matching evaluation, parameter name matching evaluation, and numerical semantic matching evaluation; this step is the core of the present invention, achieving a comprehensive quantitative evaluation of the performance of large model tool calls through multiple independent dimensions.

[0052] Specifically, such as Figure 2 In this embodiment, the method for detecting and evaluating unauthorized tool calls includes: comparing the function names output by the large language model to be evaluated with the list of authorized tools in the standard test dataset, counting the number of unauthorized tool calls and the total number of calls, and calculating the unauthorized tool call rate, whereby the unauthorized tool call rate = number of unauthorized tool calls / total number of calls. This formula quantifies the security and compliance of model tool calls.

[0053] In this embodiment, the evaluation of function name matching specifically includes: calculating the function name recall and function name precision, where the function name recall = number of correctly predicted functions / total number of standard data functions, and the function name precision = number of correctly predicted functions / total number of model output functions. This formula reflects the accuracy of the model's predicted functions, avoiding redundant or erroneous function calls.

[0054] This involves matching the function names used during model tool calls with the function names called by standard tools to calculate the number of correctly predicted functions. The methods for calculating the number of correctly predicted functions include:

[0055] Assume the standard tool call is set to [f1, f1, f2, f3, f3, f3].

[0056] The model tool call is set to `model_tool_call=[f1,f2,f2,f2,f3,f4,f4]`.

[0057] For example, the number of matches for the standard tool call and the model tool call is 3.

[0058] The correct number is the sum of the minimum number of times each function appears in both lists.

[0059] match_count=i= ;

[0060] Where fi_std represents the number of times function fi appears in the standard_tool_call list. fi_model represents the number of times function fi appears in the model_tool_call list. n represents the total number of functions.

[0061] Using the above method to calculate the number of matches, the complexity can be reduced from O(mn) to O(max(m,n)), where m and n represent the lengths of the standard_tool_call and model_tool_call lists, respectively.

[0062] In this embodiment, the parameter name matching evaluation method includes: grouping by function name, statistically analyzing the intersection and matching of parameter names output by the large language model to be evaluated and parameter names output by standard tool calls, and quantifying the completeness and accuracy of parameter names.

[0063] For example, in {"city": "Beijing"}, city is the parameter name. Construct a tuple of the form (function name, parameter name). First compare the function names; if they are the same, then compare the parameter names. Count the number of matching parameter names.

[0064] In this embodiment, the evaluation of numerical semantic matching specifically includes: employing a hierarchical matching strategy to determine the semantic equivalence of parameter values, including: adaptively converting Boolean values ​​and numbers to achieve equivalence matching; ignoring element order differences for list and dictionary types to achieve equivalence matching; and using a pre-trained BERT model to calculate the semantic similarity of natural language description parameter values ​​to determine semantic equivalence. Specifically, the parameter value semantic matching unit in the evaluation calculation module integrates a pre-trained BERT model for calculating the semantic similarity of natural language description parameter values.

[0065] Specifically, for example, equivalence matching of basic data types: through type adaptive conversion rules, equivalence matching of boolean values ​​("True" is equivalent to 1, "False" is equivalent to 0) and numbers ("100" is equivalent to 100.0) is achieved.

[0066] For complex data types, equivalence matching is performed: for list and dictionary types, the difference in element order is ignored, and ["A","B"] is determined to be equivalent to ["B","A"], and {"a":1,"b":2} is equivalent to {"b":2,"a":1} respectively.

[0067] Semantic matching of natural language description parameter values: Using a pre-trained BERT model, the text-class parameter values ​​output by the model to be evaluated are compared with the corresponding text input model of the standard output to obtain the semantic vectors of the two, and the cosine similarity is calculated. When the similarity is greater than a preset threshold (such as 0.85), it is determined to be semantically equivalent, so as to achieve accurate matching of scenarios such as "from A to B, the estimated driving distance is 25 kilometers and the time is 30 minutes" and "the total distance from A to B is 25 kilometers and it takes half an hour".

[0068] For example, in {"city": "Beijing"}, "Beijing" is the parameter value. Construct triplet tuples of the form (function name, parameter name, parameter value). First, compare the function name and parameter name. If both are the same, then compare the parameter value and count the matches. The methods for counting matches include exact matching and semantic matching. If both the function name and parameter name match, check if the parameter name is in the semantic matching parameter name list. If it is, perform semantic matching on the parameter value. Otherwise, perform exact matching.

[0069] Specifically, the semantics of parameter values ​​are compared using a deep recursive approach. First, lists are processed: each element is recursively compared. True is returned only if all elements in both lists match, but the order of elements in the two lists is not required. For example, [f1,f2,f2,f3] and [f2,f3,f1,f2] are the same, so True is returned. Then dictionaries are processed: key-value pairs are recursively compared. True is returned only if both dictionaries are identical, but the order of keys is not required. For example, {"k1":a,"k2":b,"k3":c} and {"k3":c,"k2":b,"k1":a} are the same, so True is returned.

[0070] In this embodiment, for specific parameter names such as "text", "steps", "title", and "message", the corresponding parameter values ​​are descriptive text generated by the large model. Therefore, semantic matching is required. First, it checks if the model parameter name is in the list ["text", "steps", "title", "message"]. If it is, semantic matching is required. Otherwise, a full match is performed. Then, the function `semantic_compare(value1, value2, flag, para_name=None)` is called. When `flag` is True and `value1` and `value2` are strings, the BERT algorithm is used for semantic comparison. Otherwise, a full match is performed. The BERT algorithm is only used to compare parameter values ​​when the parameter name is "text", "title", or "message" and the parameter value is a string; or when the parameter name is "steps" and the parameter value is a list of strings. The BERT algorithm is not used in other cases.

[0071] In this embodiment, when performing semantic comparison of parameter values, the BERT algorithm is used to compare whether the parameter values ​​in the tool call output by the model are semantically equal to the parameter values ​​in the tool call output by the standard model. For example, "Your commute route has been planned: from A to B, estimated distance 25 kilometers, time 30 minutes." is equivalent to "Planned commute route: from A to B, total distance 25 kilometers, time half an hour." When the parameter name is "steps" and the parameter value is a list of strings, all elements in the list of strings are concatenated with ";" to form a single string. Then, the BERT semantic comparison is performed.

[0072] In some embodiments, sample-level accuracy is also calculated by comparing the data called by the model tool with the data called by the standard tool one by one, and only when they are completely identical is it considered a match. The number of perfect matches is counted. The requirements for a perfect match are as follows: (1) the number of tool calls is the same; (2) the function names called by each tool are the same; (3) the number of parameters is the same; (4) the parameter names and parameter values ​​of each parameter are completely identical; Sample-level accuracy = number of perfectly matched samples / total number of samples.

[0073] In some embodiments, the format conformance rate is also calculated. The tool call output data format conformance rate is calculated by comparing the actual tool call result of the model with the provided tools and checking whether the following conditions exist: (1) whether the function name in the actual tool call result of the model is in the tools list; (2) whether the parameter name is in the parameters under the corresponding function in the tools list; (3) whether the required parameters are present; (4) whether the parameter value type is correct (for example, if the parameter value type is required to be string, it is wrong if the actual model output parameter value type is list); (5) whether the parameter value is in the "enum" under the corresponding parameter name. Tool call output data format conformance rate = number of actual tool calls that conform to the format conformance / total number of tool calls (excluding cases where tool calls are empty).

[0074] S400. Based on the multi-dimensional evaluation results, generate a model comparison report, output detailed error analysis, and visualize the evaluation results. In this embodiment, based on the multi-dimensional evaluation results, generate a model comparison report, output detailed error analysis, and visualize the evaluation results. The specific method includes: generating a multi-model comparison report, which includes indicators such as function name accuracy, function name recall, parameter name accuracy, parameter name recall, parameter value accuracy, parameter value recall, and unauthorized tool misuse rate, and outputting detailed error analysis and visualized evaluation results.

[0075] Specifically, the system integrates multi-dimensional evaluation data to generate a multi-model comparison report in Markdown format. The report includes core metrics such as function name accuracy, function name recall, parameter name accuracy, parameter name recall, parameter value accuracy, parameter value recall, and unauthorized tool misuse rate. It also outputs detailed error analysis, identifying the deficiencies of each model in different dimensions (such as missing parameter names, semantic bias in parameter values, and unauthorized tool calls). The visualization module displays the evaluation results in bar charts, line charts, or radar charts, intuitively presenting the performance differences between models. Batch report export is supported, providing data support for model optimization.

[0076] This invention discloses a multi-dimensional performance evaluation method for large model tool invocation. This method achieves a comprehensive evaluation of the tool invocation capability of large models through four core steps: data preparation, data standardization, multi-dimensional quantitative evaluation, and result output. Specifically, it includes constructing a standard test dataset and obtaining model output, parsing and normalizing tool invocation information, conducting quantitative evaluation from multiple dimensions such as function name matching, parameter name matching, parameter value semantic matching, and unauthorized tool invocation detection, and finally generating an automated comparison report. This invention solves the problems of existing evaluation methods, such as single-dimensionality, lack of semantic understanding, reliance on manual annotation, and insufficient security control. It has the advantages of multi-dimensional interpretability, semantic-level evaluation accuracy, enhanced security, and fully automated and efficient evaluation, and can be widely applied to the horizontal comparison and optimization iteration of large model tool invocation capabilities.

[0077] Based on the same inventive concept, embodiments of the present invention also provide a large model tool that invokes a multi-dimensional performance evaluation system, such as... Figure 3 It includes a data preparation module, a data standardization module, a multi-dimensional evaluation module, and a results output module; among which:

[0078] The data preparation module is used to construct a standard test dataset and collect tool call output results from at least two large language models to be evaluated on the standard test dataset.

[0079] The data standardization module is used to parse the tool call output results of each large language model to be evaluated, extract function names, parameter names, and parameter values; and standardize the extracted function names, parameter names, and parameter values.

[0080] The multi-dimensional evaluation module is used to perform multi-dimensional evaluation based on the standardized function name, parameter name, and parameter value, specifically including unauthorized tool call detection evaluation, function name matching evaluation, parameter name matching evaluation, and numerical semantic matching evaluation.

[0081] The results output module is used to generate model comparison reports based on multi-dimensional evaluation results, output detailed error analysis, and visualize the evaluation results.

[0082] The specific working methods of the data preparation module, data standardization processing module, multi-dimensional evaluation module, and result output module have been described in detail in the above-mentioned large model tool calling multi-dimensional performance evaluation method, and will not be repeated here in this embodiment.

[0083] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 4As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the performance evaluation methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0084] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0085] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0086] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0087] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the performance evaluation methods described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0088] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described performance evaluation method.

[0089] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0090] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0091] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0092] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0093] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0094] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0095] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0096] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0098] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A method for calling multi-dimensional performance evaluation in large model tools, characterized in that, include: Construct a standard test dataset and collect tool call outputs from at least two large language models to be evaluated on the standard test dataset. The tool call output of each large language model to be evaluated is analyzed to extract function names, parameter names, and parameter values; the extracted function names, parameter names, and parameter values ​​are then standardized. Based on the standardized function names, parameter names, and parameter values, a multi-dimensional evaluation is performed, specifically including unauthorized tool call detection evaluation, function name matching evaluation, parameter name matching evaluation, and numerical semantic matching evaluation. Based on the multi-dimensional evaluation results, a model comparison report is generated, providing detailed error analysis and visualizing the evaluation results.

2. The method according to claim 1, characterized in that, The standard test dataset contains multiple test samples, each of which includes input data, a list of authorized tools, and the corresponding standard tool call output.

3. The method according to claim 1, characterized in that, The extracted function names, parameter names, and parameter values ​​are standardized. The specific methods include: sorting the extracted parameter names in alphabetical order to eliminate order differences, and recursively normalizing the parameter values.

4. The method according to claim 1, characterized in that, The specific method for detecting and evaluating unauthorized tool calls includes: comparing the function names output by the large language model to be evaluated with the list of authorized tools in the standard test dataset, counting the number of unauthorized tool calls and the total number of calls, and calculating the unauthorized tool call rate, wherein the unauthorized tool call rate = number of unauthorized tool calls / total number of calls.

5. The method according to claim 1, characterized in that, The evaluation of function name matching includes the following methods: calculating function name recall and function name precision respectively. The function name recall = number of correctly predicted functions / total number of standard data functions, and the function name precision = number of correctly predicted functions / total number of model output functions.

6. The method according to claim 1, characterized in that, The parameter name matching evaluation method includes: grouping by function name, statistically analyzing the intersection and matching of parameter names output by the large language model under evaluation and parameter names output by standard tool calls, and quantifying the completeness and accuracy of parameter names.

7. The method according to claim 1, characterized in that, For numerical semantic matching evaluation, specific methods include: using a hierarchical matching strategy to determine the semantic equivalence of parameter values, including: performing type adaptive conversion for Boolean values ​​and numbers to achieve equivalent matching; ignoring element order differences for list and dictionary types to achieve equivalent matching; and using a pre-trained BERT model to calculate the semantic similarity of parameter values ​​in natural language description classes to determine semantic equivalence.

8. The method according to claim 1, characterized in that, Based on the multi-dimensional evaluation results, a model comparison report is generated, which outputs detailed error analysis and visualizes the evaluation results. The specific method includes: generating a multi-model comparison report, which includes indicators such as function name accuracy, function name recall, parameter name accuracy, parameter name recall, parameter value accuracy, parameter value recall, and unauthorized tool misuse rate, and outputs detailed error analysis and visualized evaluation results.

9. A large model tool calling a multi-dimensional performance evaluation system, characterized in that, It includes a data preparation module, a data standardization module, a multi-dimensional evaluation module, and a results output module; among which: The data preparation module is used to construct a standard test dataset and collect tool call output results from at least two large language models to be evaluated on the standard test dataset. The data standardization module is used to parse the tool call output results of each large language model to be evaluated, extract function names, parameter names, and parameter values; and standardize the extracted function names, parameter names, and parameter values. The multi-dimensional evaluation module is used to perform multi-dimensional evaluation based on the standardized function name, parameter name, and parameter value, specifically including unauthorized tool call detection evaluation, function name matching evaluation, parameter name matching evaluation, and numerical semantic matching evaluation. The results output module is used to generate model comparison reports based on multi-dimensional evaluation results, output detailed error analysis, and visualize the evaluation results.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 8.

Images