Method and device for improving large language model agent evaluation accuracy through fine-grained classification modeling
Through fine-grained classification modeling, the problems of inaccurate and inefficient evaluation results in large language model agent evaluation are solved, and higher evaluation accuracy and consistency are achieved, and it is suitable for different types of instruction evaluation.
Patent Information
- Application Number
- CN202411054825.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-26
AI Technical Summary
The existing large language model proxy evaluation methods have the problem of inaccurate and inefficient evaluation results, especially in real tool API interactions, which are difficult for traditional evaluation methods to accurately evaluate the tool calling capabilities and real-time knowledge of large language models.
Through fine-grained classification modeling, determine the instruction type of the target instruction and determine the model reference results and tool reference results based on its type. Based on the model execution results and tool call results, perform task completion evaluation and tool call evaluation to ensure the accuracy of the evaluation and consistency.
It improves the accuracy and efficiency of the proxy evaluation of large language model, reduces the evaluation cost, ensures the consistency of evaluation results, avoids the limitations of hallucination phenomena and knowledge timeliness, and is suitable for different types of instruction evaluations.
Smart Images

Figure CN119025416A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a method and device for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling. Background Art
[0002] The large language model (LLM) has greatly advanced the boundaries of natural language processing, enabling it to analyze complex problems and call various tools to complete instructions, thus forming a new paradigm of large language model agent (LLM Agent).
[0003] Existing large language model proxy evaluation methods involving real tool API interactions usually include advanced large language model proxy evaluation and manual evaluation.
[0004] In the process of implementing the inventive concept of the present disclosure, the inventors discovered that the application of related technologies has at least the following problems: the evaluation results are inaccurate and the efficiency is low. Summary of the invention
[0005] In view of the above problems, the present disclosure provides a method and device for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling.
[0006] According to a first aspect of the present disclosure, a method for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling is provided, including: inputting a target instruction into a large language model to obtain a model execution result tool call result, wherein the tool call result represents the ability to call a tool during the operation of the large language model; determining a model reference result and a tool reference result, wherein the model reference result and the tool reference result are both determined in a manner that matches the target instruction; determining a task completion evaluation result based on the model execution result and the model reference result; and determining a tool call evaluation result based on the tool call result and the tool reference result.
[0007] According to an embodiment of the present disclosure, determining a model reference result includes: determining an instruction type of a target instruction; and determining the model reference result in a manner that matches the instruction type.
[0008] According to an embodiment of the present disclosure, the instruction type includes a fixed instruction type, and the fixed instruction type represents a unique type of model reference result. The model reference result is determined in a manner that matches the instruction type, including: determining a fixed result that has a mapping relationship with the target instruction as the model reference result. Correspondingly, based on the model execution result and the model reference result, the task completion evaluation result is determined, including: determining the result matching degree between the model execution result and the model reference result; based on the result matching degree, determining the task completion evaluation result.
[0009] According to an embodiment of the present disclosure, the instruction type includes an open instruction type, and the open instruction type represents a type in which the model reference result is not unique. The model reference result is determined in a manner that matches the instruction type, including: determining one of the results that has a mapping relationship with the target instruction as the model reference result. Correspondingly, based on the model execution result and the model reference result, the task completion evaluation result is determined, including: inputting the model execution result, the model reference result and the target instruction into a predetermined evaluation model to obtain the task completion evaluation result.
[0010] According to an embodiment of the present disclosure, the instruction type includes a real-time instruction type, and the real-time instruction type characterizes the type of the model output result that changes over time. Determining the model reference result in a manner that matches the instruction type includes: combining database data that matches the target instruction with a predetermined template to obtain a model reference result. Correspondingly, determining the task completion evaluation result based on the model execution result and the model reference result includes: determining the result matching degree between the model execution result and the model reference result; and determining the task completion evaluation result based on the result matching degree.
[0011] According to an embodiment of the present disclosure, the instruction type includes an operational instruction type, and the operational instruction type represents a type used to call database data. Determining the model reference result in a manner that matches the instruction type includes: determining the update data that matches the target instruction as the model reference result. Correspondingly, determining the task completion evaluation result based on the model execution result and the model reference result includes: determining the result matching degree between the model execution result and the model reference result; and determining the task completion evaluation result based on the result matching degree.
[0012] According to an embodiment of the present disclosure, determining a tool reference result includes: determining a tool reference result matching a target instruction according to a predetermined execution mode. Correspondingly, obtaining a tool call result includes: collecting call data of a target tool matching the target instruction; and obtaining a tool call result based on the call data.
[0013] According to an embodiment of the present disclosure, determining a tool call evaluation result based on a tool call result and a tool reference result includes: determining a tool matching degree between the tool call result and the tool reference result; and determining a tool call evaluation result based on the tool matching degree.
[0014] According to an embodiment of the present disclosure, the above-mentioned method for improving the evaluation accuracy of large language model agents through fine-grained classification modeling also includes: in response to the instruction type of the target instruction including an operational instruction type, determining a rollback instruction based on the target instruction; in response to the obtained model execution result, executing an operation matching the rollback instruction.
[0015] A second aspect of the present disclosure provides an apparatus for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling, comprising: a model module, used to input a target instruction into a large language model, obtain a model execution result and a tool call result, wherein the tool call result represents the ability to call a tool during the operation of the large language model; a reference module, used to determine a model reference result and a tool reference result, wherein the model reference result and the tool reference result are both determined in a manner that matches the target instruction; a task module, used to determine a task completion evaluation result based on the model execution result and the model reference result; and a tool module, used to determine a tool call evaluation result based on the tool call result and the tool reference result.
[0016] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.
[0017] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.
[0018] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.
[0019] By using the method for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling provided by the embodiments of the present disclosure, the model reference results and tool reference results are determined in a manner that matches the target instructions, highlighting the division of the target instructions in a predetermined fine-grained manner, so that the evaluation accuracy is high. In addition, the task completion apple results and tool call evaluation results are determined, and the evaluation is performed from two aspects: model execution capability and tool call capability, with a wide evaluation range. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0021] Figure 1 A diagram schematically illustrates an application scenario of a method for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling according to an embodiment of the present disclosure;
[0022] Figure 2 A flowchart of a method for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling according to an embodiment of the present disclosure is schematically shown;
[0023] Figure 3 The basic architecture diagram of a large language model agent according to an embodiment of the present disclosure is schematically shown;
[0024] Figure 4 A schematic diagram schematically illustrates a method for determining a task completion evaluation result of a fixed instruction according to some embodiments of the present disclosure;
[0025] Figure 5 A schematic diagram schematically illustrates a method for determining a task completion evaluation result of an open-ended instruction according to other embodiments of the present disclosure;
[0026] Figure 6 A schematic diagram schematically illustrates a method for determining a task completion evaluation result of a real-time instruction according to other embodiments of the present disclosure;
[0027] Figure 7 A schematic diagram schematically illustrates a method for determining a task completion evaluation result of an operational instruction according to other embodiments of the present disclosure;
[0028] FIG8 schematically shows a schematic diagram of a tool call evaluation result according to an embodiment of the present disclosure;
[0029] Fig. 9 A schematic diagram schematically shows a method for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling according to a specific embodiment of the present disclosure;
[0030] Fig.10 The structure block diagram of the device for improving the evaluation accuracy of a large language model proxy through fine-grained classification modeling according to an embodiment of the present disclosure is schematically shown;
[0031] Fig.11 A block diagram of an electronic device suitable for implementing a method for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0032] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0033] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0034] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.
[0035] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0036] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0037] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.
[0038] The powerful capabilities of Large Language Models (LLMs) have greatly advanced the boundaries of natural language processing. Large models are no longer limited to understanding and generating natural language texts. They are now able to analyze complex problems and call various tools to complete instructions, thus forming a new paradigm called Large Language Model Agents.
[0039] The LLM agent is composed of a LLM, memory, task planning, and tool use. The LLM is the core brain, while memory, task planning, and tool use are the three key components of the agent system. With LLM and the three key components, the LLM can have a complex workflow where the model basically talks to itself without human involvement in every part of the interaction.
[0040] The process of the large language model agent completing instructions includes three parts: task decomposition, tool calling, and reasoning analysis. First, the large language model agent breaks down the large task into smaller manageable subtasks and defines the task content, input, output, and required tools of the subtasks. Subsequently, the large language model agent extracts structured parameters from the task instructions and inputs them into the available tools to call the tools correctly. If called correctly, the tool will return the corresponding information; if the input parameters are incorrect or an incorrect tool is used, the corresponding error message and irrelevant information will also be returned; the large language model agent will use reasoning capabilities to analyze the information returned by the tool, repeat the process of tool calling and reasoning analysis until the task instructions are completed and the correct response is output.
[0041] In the process of a large language model acting as an agent to use tools to solve a problem, the model first extracts the query and the list of available tools and their descriptions from the prompt. It then decomposes the query into smaller subtasks and calls the corresponding tools in sequence, using the knowledge returned by each tool to gradually infer the correct answer. However, previous studies have revealed the shortcomings of most large language model agents in tool use, highlighting their failures in task decomposition, tool calling, and knowledge application, with frequent errors at various stages.
[0042] Existing large language model proxy evaluation methods involving real tool API interactions are usually advanced large language model evaluation and manual evaluation. Since large language models usually interact with tools to complete tasks in many rounds, the output of a task is usually long, and traditional open-ended automatic evaluation indicators such as BLEU and ROUGE are often unable to accurately evaluate such long answers. Therefore, advanced large language models are increasingly used in open-ended evaluation.
[0043] When evaluating the capabilities of large language model agents, there are usually two evaluation methods based on large language models. The first method takes the output of the large language model agent being evaluated as input, and requires the large language model evaluator to score the tool call process and final response of the large language model agent in terms of information richness, factuality, logic, etc.; the second method takes the output of the language model agent being evaluated and the manually annotated reference solution as input, and requires the large language model evaluator to score the evaluated solution with reference to the reference solution. Manual evaluation is also a common practice for large language model agent evaluation. Usually, evaluators will check the correctness of the large language model agent tool selection, parameter input, solution planning, and final response, and give scores for the corresponding stages.
[0044] However, the existing evaluation methods have the problem of inaccurate evaluation results. The knowledge of the large language model comes from the corpus during pre-training. However, most of the existing instructions that need to be called by the tool require real-time knowledge from the tool. In principle, the advanced large language model can only analyze whether the large language model proxy solution being evaluated is correct from the perspective of rationality. For some factual errors that may occur in it, the large language model evaluator cannot actually detect them, which leads to the problem of low accuracy. In fact, even the advanced large language model cannot completely and accurately complete all instructions in the existing large language model proxy evaluation benchmark. Using such a large language model to evaluate other large language models with weaker capabilities will result in limited accuracy of the evaluation results. Similarly, although the use of manual evaluation can avoid some evaluation errors caused by real-time knowledge, it cannot completely avoid related problems because the knowledge of the evaluator, especially the real-time knowledge, is also limited. In addition, the output of the large language model proxy may contain hallucinations. If the tool is not actually called and only the output of the large language model proxy is analyzed, neither manual evaluation nor advanced large language model evaluation can completely distinguish the hallucinations in the output.
[0045] Existing evaluation methods also have the problem of inconsistent evaluation standards. Advanced large language model evaluators often have inconsistent evaluation standards when evaluating answers to open questions. Even if reference answers are given, this phenomenon can be alleviated, but it cannot be completely eliminated. Similarly, multiple evaluators will have different evaluation standards when evaluating different test examples. Even if a sample is evaluated by multiple people, such differences are difficult to completely eliminate.
[0046] In addition, whether it is large language model evaluation or manual evaluation, the evaluation cost is very high and the evaluation speed is slow. Advanced large language models are generally closed-source models, which are expensive to use and slow to infer. Although manual evaluation is more accurate than large language model evaluation, the cost is generally higher than that of large language model evaluation and the evaluation speed is also slower.
[0047] In response to the above technical problems, the present disclosure provides a method for improving the evaluation accuracy of a large language model through fine-grained classification modeling, including: inputting a target instruction into the large language model to obtain a model execution result and a tool call result, wherein the tool call result represents the ability to call the tool during the operation of the large language model; determining a model reference result and a tool reference result, wherein the model reference result and the tool reference result are both determined in a manner that matches the target instruction; determining a task completion evaluation result based on the model execution result and the model reference result; and determining a tool call evaluation result based on the tool call result and the tool reference result.
[0048] According to the embodiments of the present disclosure, the present disclosure has a higher evaluation accuracy when the instruction type of the target instruction is a fixed instruction type, an operational instruction type, and a real-time instruction type. The target instructions of the fixed instruction type, the operational instruction type, and the real-time instruction type all require the large language model agent to provide an accurate response before they can be considered as correctly completed instructions. However, traditional advanced large language model evaluators and manual evaluations are limited by knowledge timeliness and illusions, and thus cannot provide accurate evaluation results. The large language model may generate wrong replies to provide deceptive responses, which is a common illusion phenomenon in large language models. The knowledge of the large language model comes from the pre-training stage, and the knowledge of the evaluator also has a certain range. Therefore, for the real-time updated answers to questions and the illusions that may be contained, neither of the two evaluation methods can produce accurate results. The present disclosure evaluates the synchronization of the large language model agent calling the tool, calls the same API to obtain real-time answers, checks the authenticity of network requests and responses, and fundamentally guarantees the accuracy of the evaluation.
[0049] The present disclosure can ensure the consistency of evaluation between different evaluation objects. Since the output of the large language model is random, using the large language model as an evaluator cannot ensure the consistency of evaluation between different evaluation objects. Manual evaluation can enhance consistency through multiple rounds of evaluation, but it cannot eliminate it. The present disclosure uses automatic evaluation methods when the instruction type of the target instruction is a fixed instruction type, an operational instruction type, and a real-time instruction type, thereby ensuring consistency between multiple evaluations.
[0050] The present disclosure is low-cost and highly efficient. The large language model evaluator has high requirements on the capabilities of the base large model, but the advanced large language model is generally a closed-source model, which is expensive to use and has a slow reasoning speed. Manual evaluation is more expensive and slower. The present disclosure is an automatic evaluation, and the large language model is used only when the instruction type of the target instruction is an open instruction type, thereby effectively reducing costs and improving efficiency.
[0051] It should be noted that the method for improving the accuracy of large language model evaluation through fine-grained classification modeling provided in the embodiments of the present disclosure can be used in the field of artificial intelligence. The method and device for improving the accuracy of large language model evaluation through fine-grained classification modeling provided in the embodiments of the present disclosure can also be used in any field other than the field of artificial intelligence, such as the field of the Internet of Things. The application field of the software testing method and device provided in the embodiments of the present disclosure is not limited.
[0052] Figure 1 The following schematically shows an application scenario diagram of a method for improving the evaluation accuracy of a large language model through fine-grained classification modeling according to an embodiment of the present disclosure.
[0053] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a terminal device 101, a terminal device 102, a terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0054] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).
[0055] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
[0056] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.
[0057] It should be noted that the method for improving the evaluation accuracy of a large language model through fine-grained classification modeling provided in the embodiment of the present disclosure can generally be performed by the server 105. Accordingly, the device for improving the evaluation accuracy of a large language model through fine-grained classification modeling provided in the embodiment of the present disclosure can generally be set in the server 105. The method for improving the evaluation accuracy of a large language model through fine-grained classification modeling provided in the embodiment of the present disclosure can also be performed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the device for improving the evaluation accuracy of a large language model through fine-grained classification modeling provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.
[0058] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0059] The following will be based on Figure 1 The scene described by Figure 2~Figure 6 A method for improving the evaluation accuracy of a large language model through fine-grained classification modeling in a disclosed embodiment is described in detail.
[0060] Figure 2 The flowchart of the method for improving the evaluation accuracy of a large language model through fine-grained classification modeling according to an embodiment of the present disclosure is schematically shown.
[0061] like Figure 2 As shown, the method of improving the evaluation accuracy of a large language model through fine-grained classification modeling in this embodiment includes operations S210 to S240, and the transaction processing method can be executed by the server 105.
[0062] In operation S210, the target instruction is input into the large language model to obtain a model execution result and a tool calling result, wherein the tool calling result represents the ability to call the tool during the operation of the large language model.
[0063] According to an embodiment of the present disclosure, the target instruction refers to a task instruction received by the large language model, for example, a question raised by a user to the large language model through an input device.
[0064] According to an embodiment of the present disclosure, a large language model refers to a model that is trained using a large-scale data set and is capable of understanding and generating natural language text.
[0065] Large language model agents are no longer limited to understanding and generating natural language text, but are also able to analyze complex problems and call various tools to complete instructions.
[0066] The Big Language Model Agent is composed of a Big Language Model, memory, task planning, and tool use. The Big Language Model is the core brain, while memory, task planning, and tool use are the three key components of the agent system. With the Big Language Model and the three key components, the Big Language Model Agent can have a complex workflow where the model basically talks to itself without human involvement in every part of the interaction.
[0067] Figure 3 The basic architecture diagram of a large language model agent according to an embodiment of the present disclosure is schematically shown.
[0068] like Figure 3As shown, the large language model agent includes modules such as tools, memory, planning, and action. The tool modules that the large language model agent can call may include calculators, databases, weather forecasts, etc. The memory module can be divided into short-term memory submodules and long-term memory submodules. The planning module may include submodules such as reflection, thought chain, and subtask decomposition. The actions performed by the action module may be implemented by calling the tool module together.
[0069] According to an embodiment of the present disclosure, the model execution result refers to a processing result obtained by the large language model after receiving a target instruction through non-tool-invoked processing and tool-invoked processing.
[0070] According to an embodiment of the present disclosure, a tool call result refers to a characterization result of a tool call process in which a large language model obtains a processing result through a tool call after receiving a task instruction. The tool call result can be used to characterize the ability to call a tool during the operation of a large language model, and may include a tool name, parameters, and status information returned by the tool.
[0071] According to an embodiment of the present disclosure, the target instruction is input into a large language model to obtain a model execution result and a tool calling result. The execution result of the large language model can be used as the model execution result, and the name, parameters, returned status code, etc. of the calling tool can be used as the tool calling result.
[0072] In operation S220 , a model reference result and a tool reference result are determined, wherein both the model reference result and the tool reference result are determined in a manner of matching the target instruction.
[0073] According to the embodiments of the present disclosure, the model reference result refers to the correct answer that matches the target instruction, which can be a fixed answer or an open answer, which can be preset or generated by calling a real tool API. The real tool can be a computer program used in the real world for real life needs.
[0074] According to an embodiment of the present disclosure, a tool reference result refers to a correct tool call result that matches a target instruction, and may include a tool name, parameters, and status information returned by the tool.
[0075] In operation S230 , a task completion evaluation result is determined based on the model execution result and the model reference result.
[0076] According to an embodiment of the present disclosure, the task completion evaluation result is used to characterize the task completion capability of the large language model, which may be an accuracy rate or a score.
[0077] According to an embodiment of the present disclosure, determining a task completion evaluation result based on a model execution result and a model reference result may include determining the task completion evaluation result based on a matching degree between the model execution result and the model reference result, and may also include obtaining the task completion evaluation result using a predetermined evaluation model. The matching degree may be one or more of accuracy, rationality, completeness, and richness. The predetermined evaluation model may be an advanced large language model, such as a large language model based on a Transformer model.
[0078] In operation S240 , a tool call evaluation result is determined based on the tool calling result and the tool reference result.
[0079] According to an embodiment of the present disclosure, the tool call evaluation result is used to characterize the tool selection capability of the large language model, and may be a ratio.
[0080] According to an embodiment of the present disclosure, determining a tool call evaluation result based on a tool call result and a tool reference result may include determining a tool matching degree between the tool call result and the tool reference result, and determining the tool call evaluation result based on the tool matching degree.
[0081] According to an embodiment of the present disclosure, the target instruction is input into a large language model to obtain the model execution result and the tool call result; based on the model execution result and the model reference result, the task completion evaluation result is determined, and based on the tool call result and the tool reference result, the tool call evaluation result is determined. Both the model reference result and the tool reference result are determined in a manner that matches the target instruction, highlighting the division of the target instruction according to a predetermined fine-grained division method, so that the evaluation accuracy is high. In addition, the task completion apple result and the tool call evaluation result are determined, and the evaluation is performed from two aspects: model execution capability and tool call capability, with a wide evaluation range. This evaluation method avoids possible hallucinations and improves the accuracy of the evaluation by synchronously calling the API with the large language model to be evaluated; based on the automated evaluation process, the consistency of the evaluation is improved.
[0082] According to an embodiment of the present disclosure, determining a model reference result includes: determining an instruction type of a target instruction; and determining the model reference result in a manner that matches the instruction type.
[0083] According to an embodiment of the present disclosure, the instruction type refers to the type of task instruction received by the large language model, which may include a fixed instruction type, an open instruction type, a real-time instruction type, and an operational instruction type.
[0084] According to an embodiment of the present disclosure, determining the instruction type of the target instruction may include classifying the target instruction according to whether the model reference result is unique, whether the model reference result changes over time, whether database data needs to be called when executing the target instruction, etc.
[0085] According to the embodiments of the present disclosure, by classifying target instructions and determining model reference results for different types of target instructions using different methods, the granularity of the evaluation can be improved, laying the foundation for improving the accuracy, consistency and efficiency of the evaluation.
[0086] According to an embodiment of the present disclosure, the instruction type includes a fixed instruction type, and the fixed instruction type represents a unique type of a model reference result.
[0087] The model reference result is determined in a manner that matches the instruction type, including: determining a fixed result that has a mapping relationship with the target instruction as the model reference result.
[0088] Correspondingly, based on the model execution result and the model reference result, determining the task completion evaluation result includes: determining the result matching degree between the model execution result and the model reference result; and determining the task completion evaluation result based on the result matching degree.
[0089] Figure 4 A schematic diagram of determining a task completion evaluation result of a fixed instruction according to some embodiments of the present disclosure is schematically shown.
[0090] like Figure 4 As shown, based on the fixed instructions, the model execution results and the model reference results are obtained; based on the model execution results and the model reference results, the result matching degree is obtained; based on the result matching degree, the task completion evaluation result is obtained.
[0091] According to an embodiment of the present disclosure, fixed instructions may include requests such as inquiring about common knowledge, querying real public historical data, etc.
[0092] According to an embodiment of the present disclosure, determining a fixed result having a mapping relationship with a target instruction as a model reference result may include: using an output matched to the target instruction in an authoritative public database as the model reference result.
[0093] According to an embodiment of the present disclosure, determining the result matching degree between the model execution result and the model reference result, and determining the task completion evaluation result based on the result matching degree may include: comparing the output of the model execution result and the model reference result to obtain the consistency between the two as the accuracy score, using the above score as the result matching degree, and using the result matching degree as the task completion evaluation result.
[0094] According to the embodiments of the present disclosure, the task completion evaluation results of fixed instructions are obtained through the above steps, thereby improving the evaluation granularity; by introducing real tool calls, the limitations of the knowledge scope and timeliness of manual evaluation are overcome, the illusion phenomenon caused by bypassing tool calls is avoided, and the accuracy of the evaluation is improved; the evaluation process is automated and standardized, and the consistency between multiple evaluations is improved; unnecessary advanced large language model calls in related technologies can be avoided, thereby improving the evaluation efficiency.
[0095] According to an embodiment of the present disclosure, the instruction type includes an open instruction type, and the open instruction type represents a type in which the model reference result is not unique.
[0096] The model reference result is determined in a manner that matches the instruction type, including: determining one of the results that has a mapping relationship with the target instruction as the model reference result.
[0097] Correspondingly, based on the model execution result and the model reference result, the task completion evaluation result is determined, including: inputting the model execution result, the model reference result and the target instruction into a predetermined evaluation model to obtain the task completion evaluation result.
[0098] Figure 5 A schematic diagram of determining a task completion evaluation result of an open instruction according to some other embodiments of the present disclosure is schematically shown.
[0099] like Figure 5 As shown, based on the open instructions, the model execution results and the model reference results are obtained; the model execution results and the model reference results are input into a predetermined evaluation model to obtain the task completion evaluation results.
[0100] According to an embodiment of the present disclosure, an open instruction may include a request without clear directionality and scope.
[0101] According to an embodiment of the present disclosure, determining one of the results that has a mapping relationship with the target instruction as a model reference result may include: scoring multiple results that have a mapping relationship with the open instruction from three aspects: rationality, completeness, and richness, and weighting the scores of these three aspects to obtain a comprehensive score, and selecting the result with the highest comprehensive score as the model reference result.
[0102] According to an embodiment of the present disclosure, the model execution result, the model reference result and the target instruction are input into a predetermined evaluation model to obtain a task completion evaluation result, which may include: the model execution result, the model reference result and the target instruction are input into a predetermined evaluation model, and the predetermined evaluation model is required to score the rationality, completeness and richness of the output, and the above score is used as the task completion evaluation result. In addition, the scores of these three aspects can also be weighted to obtain a comprehensive score as the task completion evaluation result. The predetermined evaluation model can be an advanced large language model, such as a Transformer-based model.
[0103] According to the embodiments of the present disclosure, the task completion evaluation results of open instructions are obtained through the above steps, thereby improving the evaluation granularity; the call of the predetermined evaluation model is limited to the minimum range, and the evaluation efficiency is improved as much as possible on the basis of ensuring the comprehensiveness of the instructions that can be covered by the evaluation.
[0104] According to an embodiment of the present disclosure, the instruction type includes a real-time instruction type, and the real-time instruction type represents the type of model output result changing over time.
[0105] The model reference result is determined in a manner matching the instruction type, including: combining database data matching the target instruction with a predetermined template to obtain the model reference result.
[0106] Correspondingly, based on the model execution result and the model reference result, the task completion evaluation result is determined, including: determining the result matching degree between the model execution result and the model reference result; and determining the task completion evaluation result based on the result matching degree.
[0107] Figure 6 A schematic diagram of determining a task completion evaluation result of a real-time instruction according to some other embodiments of the present disclosure is schematically shown.
[0108] like Figure 6 As shown, based on real-time instructions, the model execution results and database data are obtained; the predetermined template is combined with the database data to obtain the model reference result; based on the model execution result and the model reference result, the result matching degree is obtained; based on the result matching degree, the task completion evaluation result is obtained.
[0109] According to an embodiment of the present disclosure, real-time instructions may include real-time situation query requests in the real world, and results may be obtained based on the content returned by real-time tools, such as real-time traffic query, real-time ranking query, real-time news broadcast, etc.
[0110] According to an embodiment of the present disclosure, database data matching the target instruction is combined with a predetermined template to obtain a model reference result, which may include: matching a corresponding tool based on a real-time instruction, querying data in a database through an API interface of the tool based on a predetermined template, and outputting results based on the predetermined template.
[0111] The predefined template may include an input template and an output template. The input template is used to provide the tool name, API interface, parameters, parameter usage, and operation steps corresponding to the real-time instructions of different requirements before the tool is run. The output template is used to extract or assemble the format or operation of the tool return information after the tool is run to output the result. The database data matching the target instruction may be a database with an API interface provided by a third party.
[0112] According to the embodiments of the present disclosure, the task completion evaluation results of real-time instructions are obtained through the above steps, thereby improving the evaluation granularity; by introducing the real-time call of real tools, the limitations of the knowledge scope and timeliness of manual evaluation are overcome, and real-time dynamic evaluation of accuracy is achieved, avoiding the illusion caused by bypassing tool calls, thereby improving the accuracy of the evaluation; the evaluation process is automated and standardized, thereby improving the consistency between multiple evaluations; unnecessary calls to advanced large language models in related technologies can be avoided, thereby improving the evaluation efficiency.
[0113] According to an embodiment of the present disclosure, the instruction type includes an operational instruction type, and the operational instruction type represents a type used to call database data.
[0114] The model reference result is determined in a manner that matches the instruction type, including: determining update data that matches the target instruction as the model reference result.
[0115] Correspondingly, based on the model execution result and the model reference result, the task completion evaluation result is determined, including: determining the result matching degree between the model execution result and the model reference result; and determining the task completion evaluation result based on the result matching degree.
[0116] Figure 7 A schematic diagram schematically illustrates a method for determining a task completion evaluation result of an operational instruction according to some other embodiments of the present disclosure.
[0117] like Figure 7 As shown, based on the operational instructions, the model execution results and updated data are obtained; based on the updated data, the model reference results are obtained; based on the model execution results and the model reference results, the result matching degree is obtained; based on the result matching degree, the task completion evaluation result is obtained.
[0118] According to an embodiment of the present disclosure, the operational instruction may require the large language model to call a tool API that is capable of changing database information.
[0119] According to an embodiment of the present disclosure, determining the updated data matching the target instruction as the model reference result may include: based on the operational instruction, by calling the tool API, updating the data in the database to which the API leads, and using the updated data as the model reference result. For example, if the operational instruction requires the execution of adding a contact in the office chat software, the model reference result may be the newly added contact in the contact list after the contact is successfully added.
[0120] According to the embodiments of the present disclosure, the task completion evaluation results of the operational instructions are obtained through the above steps, thereby improving the evaluation granularity; by introducing the real-time call of the real tool, the illusion phenomenon caused by bypassing the tool call is avoided, thereby improving the accuracy of the evaluation; the evaluation process is automated and standardized, thereby improving the consistency between multiple evaluations; and unnecessary calls to advanced large language models in related technologies can be avoided, thereby improving the evaluation efficiency.
[0121] According to an embodiment of the present disclosure, determining a tool reference result includes: determining a tool reference result that matches a target instruction according to a predetermined execution mode.
[0122] Correspondingly, obtaining the tool calling result includes: collecting calling data of the target tool that matches the target instruction; and obtaining the tool calling result based on the calling data.
[0123] According to an embodiment of the present disclosure, the predetermined execution method may be to check the status code of the tool response of each subtask call. The above-mentioned subtasks may be smaller manageable subtasks obtained after the large language model agent to be evaluated performs task splitting in response to the target instruction during task splitting. The call data of the target tool may be the status code representing the request result contained in the network response returned by the API. Based on the call data, the tool call result is obtained, which may include: after the operation is successful, a status code of a specific value returned by the tool is obtained; after the parameter input is incorrect or the incorrect tool is used, the corresponding error message or irrelevant information returned by the tool is obtained.
[0124] According to the embodiments of the present disclosure, the tool call results are obtained through the above steps, and the authenticity of the network request and response can be checked, thereby realizing the evaluation of the authenticity of the large language model tool call and avoiding the illusion phenomenon caused by bypassing the tool call.
[0125] According to an embodiment of the present disclosure, determining a tool call evaluation result based on a tool call result and a tool reference result includes: determining a tool matching degree between the tool call result and the tool reference result; and determining a tool call evaluation result based on the tool matching degree. The tool matching degree may be a ratio, specifically: the number of tools that respond correctly divided by the number of tools preset as needed.
[0126] Figure 8 A schematic diagram schematically shows a tool call evaluation result according to an embodiment of the present disclosure.
[0127] like Figure 8 As shown, based on the target instruction, the calling data and tool reference result of the target tool are obtained; based on the calling data of the target tool, the tool calling result is obtained; based on the tool calling result and the tool reference result, the tool matching degree is obtained; based on the tool matching degree, the tool calling evaluation result is obtained.
[0128] According to the embodiments of the present disclosure, the tool call evaluation results are obtained through the above steps, and the accuracy of the tool call can be quantified, thereby improving the accuracy and consistency of the overall evaluation.
[0129] According to an embodiment of the present disclosure, the method further includes: in response to the target instruction including an operational instruction type, determining a rollback instruction based on the target instruction; and in response to the obtained model execution result, executing an operation matching the rollback instruction. For example, based on the target instruction, a rollback function is set as a rollback instruction, and after each successful call to the tool, the call result is obtained as the model execution result, and the "reverse operation" of the call is executed. Specifically, after adding a friend using an instant messaging tool, the system will delete the friend.
[0130] According to an embodiment of the present disclosure, by executing the above-mentioned operations matching the rollback instruction, the data changes caused by the backend database corresponding to the calling tool during the evaluation process can be erased, ensuring the consistency of the backend database used in multiple evaluation processes, so as to ensure the consistency and accuracy of subsequent evaluations.
[0131] Fig. 9 A schematic diagram of a method for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling according to a specific embodiment of the present disclosure is schematically shown.
[0132] like Fig. 9 As shown, the large language model agent can handle fixed, real-time, operational, and open instructions.
[0133] A fixed instruction, such as "Please provide me with the closing price of SMGR.JK on the Jakarta Stock Exchange, Indonesia on October 13, 2022", is input into the large language model agent. The large language model agent can call an external database to analyze and answer the instruction to obtain answer 1; the same fixed instruction is input into the evaluation system, and the evaluation system obtains answer 2 by searching the above database; compare answer 1 and answer 2 to obtain the task completion score.
[0134] Real-time instructions, such as "I plan to go to No. 789 Dayan Pagoda. What is the congestion information within 600 meters around here?" are input into the large language model agent. The large language model agent can call external tools to analyze and answer the instructions to obtain answer 1; the evaluation system calls the tool to organize the answers and obtains answer 2; compares answer 1 and answer 2 to obtain the task completion score.
[0135] Input operational instructions, such as "Please create a student score table, which should have three fields: student number, name and score", into the large language model agent. The large language model agent can call tools and analyze multiple times, integrate the results of multiple calls, and output result 1; the evaluation system gradually checks the status codes when the agent calls tools and analyzes, obtains the tool call score, and outputs result 2; compares result 1 and result 2 to obtain the task completion score.
[0136] An open instruction, such as "search for popular e-book readers on Taobao, recommend one, and provide information about its screen size and storage capacity", is input into the large language model agent. The large language model agent can call the tool and analyze multiple times, integrate the results of multiple calls, and output result 1; the evaluation system evaluates the rationality of result 1 and obtains the task completion score.
[0137] According to an embodiment of the present disclosure, Table 1 schematically shows the evaluation results of a method for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling according to a specific embodiment of the present disclosure. Specifically, the result is the evaluation result of the evaluation device CToolEval under fixed instructions (Fixed answer, abbreviated as FA), open instructions (Open Ending, abbreviated as OE), operational instructions (Operation, abbreviated as OP), and real-time instructions (Realtime, abbreviated as RT).
[0138] Table 1
[0139]
[0140] According to the embodiments of the present disclosure, the present disclosure has a higher evaluation accuracy when the instruction type of the target instruction is a fixed instruction type, an operational instruction type, and a real-time instruction type. The target instructions of the fixed instruction type, the operational instruction type, and the real-time instruction type all require the large language model agent to provide an accurate response before they can be considered as correctly completed instructions. However, traditional advanced large language model evaluators and manual evaluations are limited by knowledge timeliness and illusions, and thus cannot provide accurate evaluation results. The large language model may generate wrong replies to provide deceptive responses, which is a common illusion phenomenon in large language models. The knowledge of the large language model comes from the pre-training stage, and the knowledge of the evaluator also has a certain range. Therefore, for the real-time updated answers to questions and the illusions that may be contained, neither of the two evaluation methods can produce accurate results. The present disclosure evaluates the synchronization of the large language model agent calling the tool, calls the same API to obtain real-time answers, checks the authenticity of network requests and responses, and fundamentally guarantees the accuracy of the evaluation.
[0141] The present disclosure can ensure the consistency of evaluation between different evaluation objects. Since the output of the large language model is random, using the large language model as an evaluator cannot ensure the consistency of evaluation between different evaluation objects. Manual evaluation can enhance consistency through multiple rounds of evaluation, but it cannot eliminate it. The present disclosure uses automatic evaluation methods when the instruction type of the target instruction is a fixed instruction type, an operational instruction type, and a real-time instruction type, thereby ensuring consistency between multiple evaluations.
[0142] The present disclosure is low-cost and highly efficient. The large language model evaluator has high requirements on the capabilities of the base large model, but the advanced large language model is generally a closed-source model, which is expensive to use and has a slow reasoning speed. Manual evaluation is more expensive and slower. The present disclosure is an automatic evaluation, and the large language model is used only when the instruction type of the target instruction is an open instruction type, thereby effectively reducing costs and improving efficiency.
[0143] Based on the above method for improving the evaluation accuracy of a large language model through fine-grained classification modeling, the present disclosure also provides a device for improving the evaluation accuracy of a large language model through fine-grained classification modeling.
[0144] The following will be combined Fig.10 The device is described in detail.
[0145] Fig.10 The structural block diagram of the device for improving the evaluation accuracy of a large language model through fine-grained classification modeling according to an embodiment of the present disclosure is schematically shown.
[0146] like Fig.10As shown, the device 1000 of this embodiment for improving the evaluation accuracy of a large language model through fine-grained classification modeling includes a model module 1010 , a reference module 1020 , a task module 1030 , and a tool module 1040 .
[0147] The model module 1010 is used to input the target instruction into the large language model to obtain the model execution result and the tool call result, wherein the tool call result represents the ability to call the tool during the operation of the large language model. In one embodiment, the model module 1010 can be used to perform the operation S210 described above, which will not be repeated here.
[0148] The reference module 1020 is used to determine the model reference result and the tool reference result, wherein the model reference result and the tool reference result are determined in a manner that matches the target instruction. In one embodiment, the reference module 1020 can be used to perform the operation S220 described above, which will not be repeated here.
[0149] The task module 1030 is used to determine the task completion evaluation result based on the model execution result and the model reference result. In one embodiment, the task module 1030 can be used to perform the operation S230 described above, which will not be described in detail here.
[0150] The tool module 1040 is used to determine the tool call evaluation result based on the tool call result and the tool reference result. In one embodiment, the tool module 1040 can be used to perform the operation S240 described above, which will not be described in detail here.
[0151] According to an embodiment of the present disclosure, any multiple modules of the model module 1010, the reference module 1020, the task module 1030, and the tool module 1040 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the model module 1010, the reference module 1020, the task module 1030, and the tool module 1040 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of them. Alternatively, at least one of the model module 1010 , the reference module 1020 , the task module 1030 , and the tool module 1040 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.
[0152] According to an embodiment of the present disclosure, the reference module 1020 includes: an instruction type submodule for determining the instruction type of the target instruction; and a model reference result submodule for determining the model reference result in a manner that matches the instruction type.
[0153] According to an embodiment of the present disclosure, the model reference result submodule includes: a fixed model reference result unit, which is used to determine a fixed result that has a mapping relationship with the target instruction as the model reference result. Correspondingly, the task module 1030 includes: a fixed matching degree submodule, which is used to determine the result matching degree between the model execution result and the model reference result; and a fixed task evaluation submodule, which is used to determine the task completion evaluation result based on the result matching degree.
[0154] According to an embodiment of the present disclosure, the model reference result submodule includes: an open model reference result unit, which is used to determine one of the results that has a mapping relationship with the target instruction as the model reference result. Correspondingly, the task module 1030 includes: an open task evaluation submodule, which is used to input the model execution result, the model reference result and the target instruction into a predetermined evaluation model to obtain a task completion evaluation result.
[0155] According to an embodiment of the present disclosure, the model reference result submodule includes: a real-time model reference result unit, which is used to combine the database data matching the target instruction with the predetermined template to obtain the model reference result. Correspondingly, the task module 1030 includes: a real-time matching degree submodule, which is used to determine the result matching degree between the model execution result and the model reference result; and a real-time task evaluation submodule, which is used to determine the task completion evaluation result based on the result matching degree.
[0156] According to an embodiment of the present disclosure, the model reference result submodule includes: operating the model reference result unit to determine the update data matching the target instruction as the model reference result. Correspondingly, the task module 1030 includes: operating the matching degree submodule to determine the result matching degree between the model execution result and the model reference result; operating the task evaluation submodule to determine the task completion evaluation result based on the result matching degree.
[0157] According to an embodiment of the present disclosure, the reference module 1020 includes: a tool reference result submodule, which is used to determine a tool reference result matching the target instruction according to a predetermined execution mode. Correspondingly, the model module 1010 includes: a call data submodule, which is used to collect the call data of the target tool matching the target instruction; and a tool call result submodule, which is used to obtain the tool call result based on the call data.
[0158] According to an embodiment of the present disclosure, the tool module 1040 includes: a tool matching degree submodule for determining the tool matching degree between the tool calling result and the tool reference result; and a tool evaluation submodule for determining the tool calling evaluation result based on the tool matching degree.
[0159] According to an embodiment of the present disclosure, the model reference result submodule also includes: a rollback instruction unit, which is used to determine a rollback instruction in response to the instruction type of the target instruction including an operational instruction type, based on the target instruction; and a rollback operation unit, which is used to execute an operation matching the rollback instruction in response to the acquired model execution result.
[0160] Fig.11 A block diagram of an electronic device suitable for implementing a method for improving the evaluation accuracy of a large language model through fine-grained classification modeling according to an embodiment of the present disclosure is schematically shown.
[0161] like Fig.11As shown, the electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 to a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include an onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0162] In RAM 1103, various programs and data required for the operation of electronic device 1100 are stored. Processor 1101, ROM 1102 and RAM 1103 are connected to each other through bus 1104. Processor 1101 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1102 and / or RAM 1103. It should be noted that the program can also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.
[0163] According to an embodiment of the present disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to the bus 1104. The electronic device 1100 may further include one or more of the following components connected to the I / O interface 1105: an input portion 1106 including a keyboard, a mouse, etc.; an output portion 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1108 including a hard disk, etc.; and a communication portion 1109 including a network interface card such as a LAN card, a modem, etc. The communication portion 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage portion 1108 as needed.
[0164] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0165] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above and / or one or more memories other than ROM 1102 and RAM 1103.
[0166] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method for improving the evaluation accuracy of a large language model through fine-grained classification modeling provided by the embodiment of the present disclosure.
[0167] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1101. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0168] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1109, and / or installed from a removable medium 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0169] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.
[0170] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0171] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0172] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0173] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. A method for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling, characterized in that: The method comprises: Inputting the target instruction into the large language model to obtain the model execution result and the tool calling result, wherein the tool calling result represents the ability to call the tool during the operation of the large language model; Determining a model reference result and a tool reference result, wherein the model reference result and the tool reference result are both determined in a manner matching the target instruction; Determining a task completion evaluation result based on the model execution result and the model reference result; and A tool call evaluation result is determined based on the tool call result and the tool reference result.
2. The method according to claim 1, characterized in that The determining of the model reference result includes: Determining the instruction type of the target instruction; The model reference result is determined in a manner matching the instruction type.
3. The method according to claim 2, characterized in that The instruction type includes a fixed instruction type, and the fixed instruction type represents a unique type of the model reference result; The determining the model reference result in a manner matching the instruction type includes: Determine a fixed result that has a mapping relationship with the target instruction as the model reference result; Correspondingly, determining the task completion evaluation result based on the model execution result and the model reference result includes: Determining a result match between the model execution result and the model reference result; and Based on the result matching degree, the task completion evaluation result is determined.
4. The method according to claim 2, characterized in that: The instruction type includes an open instruction type, and the open instruction type represents a type in which the model reference result is not unique; The determining the model reference result in a manner matching the instruction type includes: Determine one of the results that has a mapping relationship with the target instruction as the model reference result; Correspondingly, determining the task completion evaluation result based on the model execution result and the model reference result includes: The model execution result, the model reference result and the target instruction are input into a predetermined evaluation model to obtain the task completion evaluation result.
5. The method according to claim 2, characterized in that: The instruction type includes a real-time instruction type, and the real-time instruction type represents the type of change of the model output result over time; The determining the model reference result in a manner matching the instruction type includes: Combining the database data matching the target instruction with a predetermined template to obtain the model reference result; Correspondingly, determining the task completion evaluation result based on the model execution result and the model reference result includes: Determining a result matching degree between the model execution result and the model reference result; Based on the result matching degree, the task completion evaluation result is determined.
6. The method according to claim 2, characterized in that The instruction type includes an operational instruction type, and the operational instruction type represents a type used to call database data; The determining the model reference result in a manner matching the instruction type includes: Determining update data matching the target instruction as the model reference result; Correspondingly, determining the task completion evaluation result based on the model execution result and the model reference result includes: Determining a result matching degree between the model execution result and the model reference result; Based on the result matching degree, the task completion evaluation result is determined.
7. The method according to any one of claims 1 to 6, characterized in that The determination tool reference result includes: Determining the tool reference result matching the target instruction according to a predetermined execution mode; Correspondingly, obtaining the tool call result includes: collecting call data of a target tool matching the target instruction; and Based on the calling data, the tool calling result is obtained.
8. The method according to claim 1, characterized in that The determining of the tool call evaluation result based on the tool call result and the tool reference result includes: Determining a tool matching degree between the tool call result and the tool reference result; and Based on the tool matching degree, the tool call evaluation result is determined.
9. The method according to claim 6, characterized in that The method further comprises: In response to the instruction type of the target instruction including the operational instruction type, determining a rollback instruction based on the target instruction; and In response to obtaining the model execution result, an operation matching the rollback instruction is executed.
10. A device for improving the accuracy of large language model proxy evaluation through fine-grained classification modeling, characterized in that: The device comprises: A model module, used to input the target instruction into the large language model, obtain the model execution result and the tool calling result, wherein the tool calling result represents the ability to call the tool during the operation of the large language model; A reference module, used to determine a model reference result and a tool reference result, wherein the model reference result and the tool reference result are both determined in a manner matching the target instruction; A task module, configured to determine a task completion evaluation result based on the model execution result and the model reference result; and The tool module is used to determine the tool call evaluation result based on the tool call result and the tool reference result.
Citation Information
Cited By
Large language model evaluation method and device, equipment, storage medium and program product
CN119721246A
Large model task detection method and device, medium and equipment
CN122153374A
