Model evaluation methods, apparatus, electronic devices and computer-readable storage media

Abstract

This application discloses a model evaluation method, apparatus, electronic device, and computer-readable storage medium. Embodiments of this application obtain an evaluation set including at least one evaluation sample. The evaluation sample includes question-and-answer data related to function tools in a function toolset, and the function toolset itself. The question-and-answer data includes a question and an instruction to invoke the function tool based on the question. The question and the function toolset are input into a trained target model for processing to generate a prediction instruction to invoke the function tools in the function toolset. Based on the function tools invoked by the prediction instruction and the function tools invoked by the instruction, the ability of the trained target model to invoke the function tools in the function toolset to handle question-and-answer tasks is evaluated, thereby improving the accuracy of model evaluation.

Description

Technical Field

[0001] This application relates to the field of neural network model technology, specifically to a model evaluation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Large Language Model (LLM) refers to a deep learning model trained on a large amount of text data that can handle question-answering tasks.

[0003] Large language models can invoke various functional tools when handling question-answering tasks. Currently, the ability of large language models to invoke functional tools for question-answering tasks is evaluated based on the answers output by the large language models. However, the accuracy of methods based on the output answers is relatively low. Summary of the Invention

[0004] This application provides a model evaluation method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of model evaluation.

[0005] In a first aspect, embodiments of this application provide a model evaluation method, the method comprising:

[0006] Obtain an evaluation set including at least one evaluation sample, wherein the evaluation sample includes question and answer data related to function tools in the function tool set and the function tool set, wherein the question and answer data includes questions and instructions to call the function tools based on the questions;

[0007] The above problems and the above function toolset are input into the trained target model for processing, generating prediction instructions that call the above function toolset;

[0008] Based on the function tools invoked by the aforementioned prediction instructions and the function tools invoked by the aforementioned instructions, evaluate the ability of the trained target model to invoke the aforementioned function tools in the aforementioned function toolset to process question-answering tasks.

[0009] Secondly, embodiments of this application also provide a model evaluation apparatus, the apparatus comprising:

[0010] The acquisition module is used to acquire an evaluation set including at least one evaluation sample, wherein the evaluation sample includes question and answer data related to function tools in the function tool set and the function tool set, wherein the question and answer data includes questions and instructions to call the function tools based on the questions;

[0011] The processing module is used to input the above problems and the above function toolset into the trained target model for processing, and generate prediction instructions that call the above function toolset.

[0012] The evaluation module is used to evaluate the ability of the trained target model to call the function tools in the above-mentioned function toolset to process question-answering tasks, based on the function tools called by the above-mentioned prediction instructions and the function tools called by the above-mentioned instructions.

[0013] Thirdly, embodiments of this application also provide an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute any of the model evaluation methods provided in embodiments of this application.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the model evaluation methods provided in embodiments of this application.

[0015] In this embodiment, an evaluation set including at least one evaluation sample is obtained. The evaluation sample includes question-and-answer data related to function tools in a function tool set, as well as the function tool set itself. The question-and-answer data includes questions and instructions to call function tools based on the questions. The questions and function tool sets are input into the trained target model for processing to generate predicted instructions to call function tools in the function tool set. Based on the function tools called by the predicted instructions and the function tools called by the instructions, the ability of the trained target model to call function tools in the function tool set to handle question-and-answer tasks is evaluated. This simplifies the evaluation task of the trained target model's ability to handle question-and-answer tasks into a function tool classification task, allowing the performance of the trained target model on each function tool to be obtained, refining the granularity of evaluating the trained target model's ability to handle question-and-answer tasks, and improving the accuracy of evaluating the trained target model's ability to handle question-and-answer tasks. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic flowchart of one embodiment of the model evaluation method provided in this application;

[0018] Figure 2 This is a schematic diagram of the model training process provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the confusion matrix provided in the embodiments of this application;

[0020] Figure 4This is a schematic diagram of another embodiment of the model evaluation method provided in this application;

[0021] Figure 5 This is a schematic diagram of the structure of the model evaluation device provided in the embodiments of this application;

[0022] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Before providing a detailed explanation of the embodiments of this application, some terms involved in the embodiments of this application will be explained.

[0025] In the description of the embodiments of this application, the terms "first," "second," etc., may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. "A plurality of" refers to two or more.

[0026] This application provides a model evaluation method, apparatus, electronic device, and computer-readable storage medium. Specifically, the model evaluation method of this application can be executed by an electronic device, which can be a terminal or a server, etc.

[0027] The terminal can be a smartphone, tablet, laptop, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. The terminal can also include a client, which can be a game application client, a browser client with a game program, or an instant messaging client, etc.

[0028] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0029] The following is a detailed description in conjunction with the accompanying drawings. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments. Although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the drawings.

[0030] In this embodiment, a terminal is used as an example for illustration. This embodiment provides a model evaluation method, such as... Figure 1 As shown, the specific process of this model evaluation method can be as follows:

[0031] 101. Obtain an evaluation set that includes at least one evaluation sample, the evaluation sample including question and answer data related to the function tools in the function toolset and the function toolset, the question and answer data including questions and instructions to call the function tools based on the questions.

[0032] In this context, "function tool" refers to a tool that encapsulates code that performs certain functionalities, and may include at least one such tool. The type of code implementing these functionalities can be set according to actual needs; for example, the code implementing these functionalities may be a function, interface, model, or workflow, etc., and this implementation does not impose any limitations on this. The implemented functionalities can be set according to actual needs; for example, the implemented functionalities may be a weather query function or a product query function, and this embodiment does not impose any limitations on this.

[0033] Optionally, the function tool may include function tools in a general domain and / or function tools in a target professional domain, which is not limited in this embodiment. When the function tool is a function tool in a target professional domain, the trained target model can be a model that is good at handling question-answering tasks in the target professional domain. The evaluation set is used to evaluate the ability of the trained target model to call the function tool to handle question-answering tasks in the target professional domain. When the function tool is a function tool in a general domain, the evaluation set is used to evaluate the ability of the trained target model to call the function tool to handle question-answering tasks in a general domain. The target professional domain refers to a specific knowledge category, which can be set according to the actual situation. For example, the target professional domain can be the medical field or the automotive field, which is not limited in this embodiment.

[0034] Optionally, the question-and-answer data related to the function tool may include at least one of real question-and-answer data and simulated question-and-answer data. Real question-and-answer data refers to manually generated question-and-answer data, that is, questions and corresponding instructions manually generated by users. Simulated question-and-answer data refers to question-and-answer data automatically generated based on the function tool's functional description and parameter description through the first target model.

[0035] In some embodiments, when the question-and-answer data associated with the function tool includes simulated question-and-answer data, it can eliminate the need to manually generate question-and-answer data, thereby improving the efficiency of question-and-answer data generation and thus improving the efficiency of model evaluation.

[0036] In some embodiments, when the question-and-answer data associated with the function tool includes both real question-and-answer data and simulated question-and-answer data, not only can the efficiency of model evaluation be improved, but the accuracy of model evaluation can also be further enhanced.

[0037] In some embodiments, when the question-answering data related to the function tool includes simulated question-answering data, obtaining an evaluation set including at least one evaluation sample includes:

[0038] Obtain the function toolkit, which includes the function tools, their functional descriptions, and parameter descriptions of the input parameters required to call the function tools;

[0039] Based on the first target model, and according to the functional and parameter descriptions, generate questions related to the function tools and instructions for calling the function tools.

[0040] Based on the problem, instruction, and function toolset, test samples are generated, and a test set is generated based on the test samples.

[0041] The first target model can refer to a trained neural network model, such as a Large Language Model (LLM), which can handle natural language tasks, including question answering. The specific type of the large language model can be determined according to the actual situation. For example, the first target model can be Azure-OpenAI-GPT-3.5 or Microsoft Azure-GPT-4.0, but this embodiment of the application does not limit it.

[0042] The function description of a function tool refers to information that describes the function tool's functionality. For example, if the function tool's function is to query the weather, the function description could be: "This function is used to query the weather in a specific location."

[0043] When calling a function tool, you need to input some parameters. The description of these parameters is called the parameter description. When the function tool performs different functions, the parameters required to call it will also be different, and so will the parameter description. For example, if the function tool's function is to query the weather, the required parameters are location and time, and the parameter description would be: "Required parameters representing location and time." Another example is a function tool that queries salaries, where the required parameter is name, and the parameter description would be: "Required parameters representing name."

[0044] In this embodiment, a function toolset is obtained, which includes function tools, function descriptions of the function tools, and parameter descriptions of the input parameters required to call the function tools. Through a first target model, based on the function descriptions and parameter descriptions, questions related to the function tools and instructions for calling the function tools are generated. Based on the questions, instructions, and function toolset, evaluation samples are generated, and based on the evaluation samples, an evaluation set is generated, thereby achieving automatic generation of the evaluation set and improving the generation efficiency of the evaluation set.

[0045] In some embodiments, using a first target model, based on the functional description and parameter description, a problem can be generated first, and then instructions can be generated based on the problem; alternatively, instructions can be generated first, and then a problem can be generated. When instructions are generated first, and then a problem is generated, using the first target model, based on the functional description and parameter description, the problem related to the function tool and the instructions for calling the function tool are generated, including:

[0046] Based on the first target model, and according to the functional and parameter descriptions, instructions for calling the function tool are generated.

[0047] The first target model predicts the input question needed to trigger the generation instruction.

[0048] In this embodiment, the first target model generates instructions to call the function tool based on the function description and parameter description. The first target model also predicts the questions that need to be input to trigger the generation of instructions, thus generating instructions first and then questions. This not only improves the efficiency of question and instruction generation but also ensures that the quality of questions and instructions is not heavily dependent on the capabilities of the first target model. Furthermore, it allows the generated questions and instructions to possess diversity, structural completeness, and stability, further enhancing the accuracy of evaluating the ability of the trained target model to handle question-answering tasks.

[0049] For example, if the function tool implements a salary query function, the generated command could be "salary_search(name='Zhang San')", and the question that triggers the generation of this command could be "How much is Zhang San's salary?".

[0050] In some embodiments, the terminal can generate instructions based on the function information of the function tool. The function information includes the tool name, function description, parameter name, parameter description, parameter type, and parameter requirement of the function tool.

[0051] In some embodiments, based on the functional description and parameter description, instructions for calling the function tool are generated using the first target model, including:

[0052] Based on the first target model, the target parameters required for the function calling tool are generated according to the function description and parameter description.

[0053] By assembling the target parameters and the tool name of the function tool, the instruction to call the function tool is obtained.

[0054] The target parameter required to call the function tool refers to the specific value of the parameter that needs to be entered. For example, if the function tool's function is a weather query function, the parameters required to call the function tool are parameters representing the location and time, and the target parameters required to call the function tool are Wuhan and tomorrow.

[0055] After obtaining the target parameters, the terminal can assemble the target parameters and the tool name of the function tool to obtain the instruction to call the function tool.

[0056] For example, if the function tool is for salary lookup, the target parameter is "Zhang San", the tool name is salary_search, and the command to call the function tool can be "salary_search(name='Zhang San')".

[0057] Understandably, when calling a function tool requires multiple input parameters, and each required input parameter corresponds to multiple target parameters, the terminal can first generate each target parameter, then combine the target parameters to obtain a combined parameter, and then assemble the combined parameter with the tool name of the function tool to obtain the instruction to call the function tool.

[0058] For example, the parameters required to call a function tool include parameters representing time and parameters representing location. Target parameters generated for the time parameters include tomorrow and the day after tomorrow, and target parameters generated for the location parameters include Wuhan and Shenzhen. Combining tomorrow and Wuhan yields a combined parameter, combining tomorrow and Shenzhen yields a combined parameter, combining the day after tomorrow and Shenzhen yields a combined parameter, and combining the day after tomorrow and Wuhan yields a combined parameter, resulting in a total of four combined parameters. These four combined parameters are then combined with the tool name to obtain four commands for calling the function tool.

[0059] Alternatively, when calling a function tool requires multiple input parameters, and each required input parameter has multiple target parameters, the terminal can directly generate combined parameters, which include at least two target parameters. For example, if the input parameters for calling a function tool include parameters representing time and parameters representing location, the terminal can directly generate the target parameter "Wuhan" and the target parameter "tomorrow" to obtain the combined parameters.

[0060] Optionally, when one of the target parameters affects the generation of other target parameters, the combined parameters can be generated directly through the first target model.

[0061] Optionally, when generating the target parameters required for the function calling tool based on the function description and parameter description using the first target model, a parameter generation prompt template can also be input into the first target model so that the target parameters required for the function calling tool can be generated using the first target model based on the function description, parameter description, and parameter generation prompt template.

[0062] Optionally, if generating target parameters requires samples or generation rules, the samples or generation rules can be stored in the parameter description. In this case, the parameter description includes not only a description of the required input parameters, but also samples or generation rules, so that the first target model can refer to the samples or generation rules and generate target parameters according to the functional description and parameter description.

[0063] In some embodiments, before predicting the required input question for triggering the generation instruction using a first target model, the method further includes:

[0064] Perform quality checks on the instructions to obtain the quality check results;

[0065] Determine the instructions corresponding to the quality inspection results that meet the preset quality conditions.

[0066] The first target model predicts the input questions needed to trigger the generation instructions, including:

[0067] Based on the target instruction, predict the information that needs to be input to trigger the generation of the target instruction. The target instruction refers to the instruction corresponding to the quality inspection result that meets the preset quality conditions.

[0068] The process of performing quality checks on instructions and obtaining the quality check results can be as follows:

[0069] The quality of the instruction is checked based on at least one of the following: the instruction's structure, the tool name in the instruction, the parameters in the instruction, and the type of the parameters in the instruction, and the quality check result of the instruction is obtained.

[0070] For example, when performing quality checks on instructions based on their structure, we can analyze whether the instruction includes a tool name and parameters. If the instruction includes both, it indicates that the quality check result is the first quality check result. The first quality check result indicates a quality check result that meets preset quality conditions.

[0071] For example, when performing quality checks on an instruction based on the parameters in the instruction, it is possible to analyze whether the parameters in the instruction are mandatory and whether they are required input parameters. If the parameters in the instruction are required input parameters, the quality check result of the instruction is the first quality check result. If the parameters in the instruction are not required input parameters, the quality check result of the instruction is the second quality check result, which is a quality check result that does not meet the preset quality conditions.

[0072] Understandably, when performing quality checks on an instruction based on its composition structure, the tool name within the instruction, the parameters within the instruction, and the types of the parameters within the instruction, the quality check result of the instruction can be determined based on the results of various sub-quality checks.

[0073] For example, based on the composition structure of the instruction, a quality check is performed on the instruction, and the resulting sub-quality check result is the first quality check result. Based on the parameters in the instruction, a quality check is performed on the instruction, and the resulting sub-quality check result is the second quality check result. Based on the first quality check result and the second quality check result, the quality check result of the instruction is determined.

[0074] It is understandable that when the sub-quality inspection result includes the second quality inspection result, the quality inspection result of the instruction can be determined as the second quality inspection result.

[0075] In some embodiments, the question can be a short sentence or a long sentence. For example, when the function tool is a weather query function, the question can be "Wuhan, weather tomorrow" (short sentence), or "I want to ask about the weather in Wuhan tomorrow" (long sentence).

[0076] In some embodiments, the problem includes target parameters. When the problem includes all target parameters, it can be called a complete problem; when the problem includes some target parameters, it can be called a non-complete problem. The problem in this embodiment can be a complete problem and / or a non-complete problem.

[0077] For example, the parameters required to call a function tool include a parameter representing the location and a parameter representing the time. A complete question could be "What will the weather be like in Wuhan tomorrow?" In this case, the complete question includes the target parameter "Wuhan" representing the location and the target parameter "tomorrow" representing the time. An incomplete question could be "What will the weather be like tomorrow?" In this case, the incomplete question only includes the target parameter "tomorrow" representing the time and lacks the target parameter "Wuhan" representing the location.

[0078] In the embodiments of this application, when the question includes a complete question and an incomplete question, the complete question includes all the target parameters required to call the function tool, and the incomplete question includes some of the target parameters required to call the function tool. This can make the generated questions more diverse and further improve the accuracy of evaluating the ability of the trained target model to handle question-answering tasks.

[0079] In some embodiments, this embodiment further includes:

[0080] Based on the incomplete problem, a supplementary problem is generated using the first target model. The supplementary problem is used to prompt the input of the target parameters that are missing in the incomplete problem.

[0081] Based on the first objective model and the supplementary question, supplementary information is generated, which includes the missing objective parameters.

[0082] Evaluation samples are generated based on instructions, questions, supplementary questions, and supplementary information.

[0083] This can involve generating complete and incomplete problems based on instructions. Alternatively, a complete problem can be generated based on instructions, and then an incomplete problem can be generated based on the target parameters.

[0084] After obtaining the incomplete question, a supplementary question is generated based on the incomplete question, and supplementary information is generated based on the supplementary question.

[0085] For example, an incomplete question might be "I want to check my salary for January." Since the incomplete question lacks the target parameter representing the name, the generated supplementary question could be "Are you checking for January of this year? Also, could you provide the name you are checking?" Based on the supplementary question, the generated supplementary information could be something like "It's for January 2023, and the name is Zhang San."

[0086] Specifically, the process of generating evaluation samples based on instructions, questions, supplementary questions, and supplementary information can be as follows:

[0087] The instructions, complete questions, and function toolsets are assembled to obtain the evaluation samples;

[0088] Based on incomplete and supplementary questions, generate evaluation samples;

[0089] Evaluation samples are generated based on incomplete questions, supplementary questions, supplementary information, instructions, and function toolsets.

[0090] In the evaluation sample obtained by assembling instructions, complete questions, and function toolsets, complete questions serve as data in the evaluation sample, and instructions serve as labels for the evaluation sample. This evaluation sample is used to evaluate the ability of the target model to call function tools after training.

[0091] In the evaluation samples generated from incomplete and supplementary questions, the incomplete questions serve as the data in the evaluation samples, and the supplementary questions serve as the labels for the evaluation samples. This evaluation sample is used to evaluate the probing ability of the target model after training.

[0092] In the evaluation samples generated from incomplete questions, supplementary questions, supplementary information, instructions, and function toolsets, incomplete questions, supplementary questions, and supplementary information are used as data, and instructions are used as labels. This data sample is used to evaluate the ability of the target model to call function tools after training.

[0093] It is understandable that the evaluation samples generated based on incomplete questions and supplementary questions, as well as the evaluation samples generated based on incomplete questions, supplementary questions, supplementary information, instructions, and function tools, belong to the multi-turn dialogue type.

[0094] In this embodiment, a supplementary question is generated based on the incomplete question using the first target model. The supplementary question is used to prompt the input of the target parameters missing in the incomplete question. Supplementary information is generated based on the supplementary question using the first target model. The supplementary information includes the missing target parameters. Evaluation samples are generated based on the instructions, questions, supplementary questions, and supplementary information, making the evaluation of the trained target model more comprehensive and further improving the accuracy of the evaluation of the trained target model.

[0095] In some embodiments, this embodiment further includes:

[0096] According to the instructions, the function tool is invoked to obtain the parameter values ​​corresponding to the problem;

[0097] Generate the answer to the question based on the parameter values;

[0098] Based on the parameter values ​​and answers, evaluation samples are generated.

[0099] For example, if the question is "What is Zhang San's salary in January 2023?", the command is "salary_search(name='Zhang San')". Based on the command, the function tool is called, and the corresponding parameter values ​​are "{'emp_no':'123456','name':'Zhang San','time':'2023 / 01,'salary':'xxxx'}". The answer to the question is "Hello, we found that Zhang San's salary in January 2023 is xxxx".

[0100] In this embodiment, according to the instructions, a function tool is called to obtain the parameter value corresponding to the question; according to the parameter value, the answer corresponding to the question is generated; based on the parameter value and the answer, an evaluation sample is generated. The evaluation sample generated based on the parameter value and the answer can be used to evaluate the ability of the target model to summarize the answer after training, making the evaluation of the target model after training more comprehensive and further improving the accuracy of the evaluation of the target model after training.

[0101] Understandably, when generating evaluation samples based on supplementary questions and information, it's also possible to generate evaluation samples based on incomplete questions, supplementary questions, supplementary information, instructions, function toolsets, parameter values, and answers. In this evaluation sample, the incomplete questions, supplementary questions, supplementary information, instructions, and parameter values ​​serve as data, and the answers serve as labels. This evaluation sample is used to assess the ability of the trained target model to summarize the parameter values ​​returned by the function tools into answers. The type of evaluation sample generated based on incomplete questions, supplementary questions, supplementary information, instructions, parameter values, and answers belongs to the multi-turn dialogue type.

[0102] In some embodiments, the first target model predicts the required input question to trigger the generation instruction, including:

[0103] Retrieve the set character information;

[0104] Using the first target model, based on the instructions and role information, predict the questions that need to be input to trigger the generation of instructions.

[0105] Role information refers to information describing a user's identity. Different role information indicates different users. For example, role information could indicate the identity of a human resources person or the identity of a regular employee.

[0106] For example, if the function tool is for salary inquiry, and the role information indicates the identity of the HR person, then the predicted input question to trigger the generation command could be "I need to inquire about Zhang San's salary". If the role information indicates the identity of a regular employee, then the predicted input question to trigger the generation command could be "I want to know how much Zhang San earns each month".

[0107] Since different users may input different questions for the same instruction, in this embodiment, the first target model predicts the questions that need to be input to trigger the generation instruction based on the instruction and role information, thereby diversifying the generated questions and further improving the accuracy of the trained target model.

[0108] In some embodiments, after predicting the required input question for triggering the generation instruction using a first target model, the method further includes:

[0109] The problem is augmented to obtain the augmented problem.

[0110] In this process, the enhanced issues and instructions are used together to generate evaluation samples. The type of enhancement processing can be set according to the actual situation. For example, enhancement processing includes at least one of replacement processing, rewriting processing, simplification processing, and error correction processing. This embodiment does not limit the specific type of enhancement processing.

[0111] Replacement processing refers to replacing non-critical content in the question; the replaced content does not affect the generation of instructions. Rewriting processing refers to changing the syntax and question format without altering the key content. Simplification processing refers to reducing the question to a shorter sentence. Error correction processing refers to adding typos to the question.

[0112] In this embodiment of the application, after obtaining the problem, the problem is enhanced to diversify the problems used to generate the final evaluation samples.

[0113] In some embodiments, evaluation samples are generated based on a set of questions, instructions, and functions, including:

[0114] A test sample is obtained by assembling a single instruction, the corresponding problem, and the set of function tools.

[0115] The evaluation sample is obtained by combining the two instructions, the corresponding problems and function toolsets for the two instructions respectively.

[0116] The two instructions can be any two instructions generated. Optionally, the order of the questions corresponding to the two instructions in the evaluation samples is determined based on the order of the two instructions in the evaluation samples. For example, the instruction to call function tool c1 is instruction d1, and the question that triggers the generation of instruction d1 is q1. The instruction to call function tool c2 is instruction d2, and the question that triggers the generation of instruction d2 is q2. In the evaluation samples generated based on instruction d1, question q1, instruction d2, and question q2, the order of instruction d1 precedes the order of instruction d2, and the order of question q1 precedes the order of question q2.

[0117] It is understandable that the evaluation samples obtained by combining two instructions and the corresponding problem and function toolkits are of the parallel type.

[0118] In this application, an evaluation sample is obtained by assembling a single instruction, the corresponding problem, and the function tool set. Similarly, an evaluation sample is obtained by combining two instructions, the corresponding problems, and the function tool sets. This results in an evaluation sample containing two instructions, which enables the evaluation of the ability of the trained target model to call two function tools in parallel.

[0119] In some embodiments, the evaluation sample type includes at least two of the following: single-tool type, multi-tool type, parallel type, and multi-turn dialogue type. The single-tool type refers to a type in which the function tool set includes only one function tool. The multi-tool type refers to a type in which the function tool set includes at least two function tools. The parallel type refers to a type in which the evaluation sample includes two instructions and two questions that trigger the generation of the two instructions. The multi-turn dialogue type refers to a type in which the evaluation sample contains multi-turn dialogue information, wherein the multi-turn dialogue information is used to ask follow-up questions based on the parameters required to call the function tool to obtain all the parameters required for the input.

[0120] In this embodiment of the application, the evaluation samples include at least two types, which increases the diversity of evaluation samples and further improves the accuracy of the evaluation of the trained target model.

[0121] In some embodiments, the order of the various types of evaluation samples in the evaluation set is shuffled.

[0122] In some embodiments, the trained target model can also be evaluated based on evaluation samples of irrelevant types, which are evaluation samples generated based on question-and-answer data unrelated to function tool calls.

[0123] In some embodiments, evaluation samples are generated based on a set of questions, instructions, and functions, including:

[0124] Based on some questions, some instructions, and function toolsets, evaluation samples are generated;

[0125] The order of the function tools in the function toolset is adjusted to obtain the adjusted function toolset;

[0126] Based on the additional questions, additional instructions, and the adjusted function toolset, an evaluation sample is generated.

[0127] For example, a function tool set includes function tool c1, function tool c2, and function tool c3. Function tool c1 is the first function tool in the function tool set, function tool c2 is the second function tool in the function tool set, and function tool c3 is the third function tool in the function tool set. After adjusting the order of the function tools in the function tool set, function tool c1 becomes the second function tool in the adjusted function tool set, function tool c3 becomes the first function tool in the adjusted function tool set, and function tool c2 becomes the third function tool in the adjusted function tool set.

[0128] In this embodiment, evaluation samples are generated based on a portion of the questions, a portion of the instructions, and a set of function tools; the order of the function tools in the set of function tools is adjusted to obtain an adjusted set of function tools; and evaluation samples are generated based on another portion of the questions, another portion of the instructions, and the adjusted set of function tools. This allows for the evaluation of whether the trained target model overfits to the order of the function tools in the set of function tools.

[0129] In some embodiments, the evaluation set may also include evaluation samples from a general domain to assess the ability of the trained target model to handle question-answering tasks in a general domain based on the evaluation samples from the general domain.

[0130] In some embodiments, before obtaining the evaluation set including at least one evaluation sample, the method further includes:

[0131] Obtain training samples, which include question-and-answer data and function tool sets related to function tools. The question-and-answer data related to function tools includes questions and instructions for calling function tools.

[0132] The second target model is trained based on the training samples to obtain the trained target model.

[0133] The second target model can be either an untrained neural network model or a trained neural network model. When the second target model is a trained neural network model, it can refer to a Large Language Model (LLM), which can handle natural language tasks, including question-answering tasks. In this case, the second target model and the first target model can be the same model or different models.

[0134] The process of generating training samples can refer to the process of generating evaluation samples, and the process of generating question-and-answer data in training samples can refer to the process of generating question-and-answer data in evaluation samples. The embodiments of this application will not be described in detail here.

[0135] Optionally, the training samples may include training samples generated from real question-and-answer data and / or training samples generated from simulated question-and-answer data. When the training samples include training samples generated from real question-and-answer data and training samples generated from simulated question-and-answer data, the second target model can be trained based on simulated question-and-answer data and real question-and-answer data, further improving the ability of the trained target model to handle question-and-answer tasks.

[0136] Optionally, the training samples may include training samples generated based on question-answering data from the target domain and / or training samples generated based on question-answering data from a general domain. When the training samples include training samples generated based on question-answering data from the target domain and training samples generated based on question-answering data from a general domain, the trained target model not only excels at handling question-answering tasks in the target domain but also retains its ability to handle question-answering tasks in the general domain.

[0137] For ease of description, the problem in the training sample is called the first problem, the instruction in the training sample is called the first instruction, the problem in the evaluation sample is called the second problem, and the instruction in the evaluation sample is called the second instruction.

[0138] Specifically, the terminal can input the first problem and function toolset from the training samples into the second target model for prediction processing to obtain a first prediction instruction. Based on the first prediction instruction and the instructions in the training samples, a loss function value is determined. If the loss function value meets a preset loss condition, the second target model is determined as the trained target model. If the loss function value does not meet the preset loss condition, the weights of the second target model are updated according to the loss function value, and the process returns to the step of inputting the first problem and function toolset from the training samples into the second target model for prediction processing. The preset loss condition is used to indicate that the loss function value has converged.

[0139] Understandably, the loss function value determined based on the first prediction instruction and the instructions in the training samples is called the first loss function value. When the training samples include incomplete questions and supplementary questions, the incomplete questions are input into the second target model for prediction processing to obtain the predicted supplementary questions. The second loss function value is determined based on the predicted supplementary questions and the supplementary questions. When the training samples include parameter values ​​and answers, the parameter values ​​are input into the second target model for prediction processing to obtain the predicted answers. The third loss function value is determined based on the answers and the predicted answers. At this point, the loss function value of the second target model can be determined based on the first, second, and third loss function values. If the loss function value meets the preset loss condition, the second target model is determined as the post-trained target model. If the loss function value does not meet the preset loss condition, the weights of the second target model are updated according to the loss function value, and the process returns to the step of inputting the training samples into the second target model for prediction processing.

[0140] In some embodiments, before training the second target model based on training samples to obtain the trained target model, the method further includes:

[0141] Obtain model description information for each candidate large language model;

[0142] Based on the target professional field and model introduction information, the second target model is determined from various candidate large language models.

[0143] The model description information includes at least one of the following: basic information about the model, the characteristics of the model, and the model's logs.

[0144] The terminal can determine the matching degree between each candidate large language model and the target professional field based on the model introduction information and the requirements of the target professional field. Then, the candidate large language model with the highest matching degree is determined as the second target model. Alternatively, the terminal can predict the matching degree between the candidate large language model and the target professional field based on the model introduction information and the requirements of the target professional field, recommend each candidate large language model to the user according to the matching degree, and then, in response to the user's selection operation of the candidate large language model, determine the candidate large language model corresponding to the selection operation as the second target model.

[0145] In this regard, the matching degree between each candidate large language model and the target professional field can be determined according to the characteristics of the model and the requirements of the target professional field. For example, if the target professional field is hotel customer service, the questions in this field are usually in the form of text. In this case, the matching degree of the candidate large language model that is good at handling text-based questions is greater than that of the candidate large language model that is good at handling image-based questions. As another example, if the target professional field is law, the questions in this field are usually longer. In this case, the matching degree of the candidate large language model that is good at handling long questions is greater than that of the candidate large language model that is good at handling short questions.

[0146] In this embodiment of the application, model description information of each candidate large language model is obtained. Based on the target professional field and model description information, a second target model is determined from each candidate large language model. This allows the training of the second target model to be completed more quickly, further improving the efficiency of obtaining the trained target model.

[0147] In some embodiments, before training the second target model based on training samples to obtain the trained target model, the method further includes:

[0148] Display the various candidate large language models;

[0149] In response to the selection operation of each candidate large language model, the candidate large language model corresponding to the selection operation is determined as the second target model.

[0150] In this embodiment, various candidate large language models are displayed. In response to a selection operation on each candidate large language model, the candidate large language model corresponding to the selection operation is determined as the second target model, allowing the user to select the second target model themselves, thus improving the user experience.

[0151] In some embodiments, obtaining training samples includes:

[0152] Display the data generation interface;

[0153] In response to input operations on the data generation interface, obtain the name of the target professional field;

[0154] Training samples are generated based on the name of the target professional field using the first target model.

[0155] In this embodiment, a data generation interface is displayed. In response to input operations on the data generation interface, the name of the target professional field is obtained. Based on the name of the target professional field, a training sample is generated through the first target model, so that the user can select the target professional field according to their own needs. This makes the trained target model good at handling question-and-answer tasks in the professional field required by the user, and realizes the user's customization of the model.

[0156] In some embodiments, users may also provide function tools in the target professional field to make the trained target model more in line with user needs.

[0157] In some embodiments, the second target model is trained based on training samples to obtain the trained target model, including:

[0158] Set up an adapter for the second target model;

[0159] The training samples are input into the second target model for prediction processing to obtain the loss function value;

[0160] If the loss function value does not meet the preset loss condition, the weights of the adapter are updated based on the loss function value, and the process returns to the step of inputting the training samples into the second target model for prediction processing to obtain the loss function value.

[0161] If the loss function value meets the preset loss condition, the weights of the adapter and the weights of the network layers in the second target model are fused to obtain the trained target model for the target professional domain.

[0162] The adapter for the second target model can be defined as adding a new network layer in a position parallel to at least one network layer in the second target model. The weights of the new network layer are obtained by multiplying two low-rank matrices. The size of the weights of the new network layer is the same as the size of the weights of the at least one network layer. If the size of the weights of the at least one network layer is m*n, splitting the m*n matrix yields two matrices, m*r and r*n. Therefore, the sizes of the two matrices for the newly added network layer are m*r and r*n, respectively. If the weights of the at least one network layer are updated directly, the number of parameters to be updated is m*n. However, if the weights of the additional network layer are updated, the number of parameters to be updated is m*r + r*n, where m*r + r*n is less than m*n.

[0163] Therefore, in this embodiment, an adapter for the second target model is set up, and training samples are input into the second target model for prediction processing to obtain a loss function value. If the loss function value does not meet the preset loss condition, the weights of the adapter are updated based on the loss function value, and the process returns to the step of inputting training samples into the second target model for prediction processing to obtain a loss function value. If the loss function value meets the preset loss condition, the weights of the adapter and the weights of at least one network layer in the second target model are fused to obtain a post-trained target model for the target professional domain, thereby reducing the number of parameters that need to be updated and further improving the training efficiency of the second target model.

[0164] For example, when the training samples include training samples generated from real question-and-answer data and training samples generated from simulated question-and-answer data, the training process of the second target model in this embodiment can be as follows: Figure 2 As shown.

[0165] In some embodiments, this embodiment further includes:

[0166] Get the preset number of updates;

[0167] Get the adapter's weight after the preset number of updates;

[0168] The weights of the adapter and the weights of the network layers in the second target model are fused to obtain the trained target model for the target professional domain, including:

[0169] The weights after being updated a preset number of times will be merged to obtain the adapter weights;

[0170] The weights of the adapter and the weights of the network layers in the second target model are fused to obtain the trained target model for the target professional domain.

[0171] The preset update count, also known as breakpoints, refers to certain numbers of updates made by the adapter. Since the training count of the model can be set before training, the preset update count can be determined based on the training count. For example, the preset update count could be the 1st / 3rd, 2nd / 3rd, and last training count, but this embodiment does not impose any limitations on this.

[0172] For example, if the training iterations are 30, and the adapter weights are updated 30 times when the loss function value meets the preset loss condition (the preset update times are the 10th, 20th, and 30th), the adapter weights after the 10th, 20th, and 30th updates are fused to obtain the adapter weights. Then, the adapter weights are fused with the weights of at least one network layer in the second target model to obtain the trained target model. At this point, the adapter weights fused with the weights of at least one network layer in the second target model are not the last updated adapter weights, but rather the weights fused with the adapter weights after the preset update times.

[0173] It is understandable that the terminal can record the weight of the adapter after a preset number of updates, and then obtain the weight of the adapter after a preset number of updates when the loss function value meets the preset loss condition.

[0174] As training time increases, the ability of the second target model to handle general domain question-answering tasks decreases. In this embodiment, a preset number of updates is obtained, and the weights of the adapter after the preset number of updates are obtained. The weights after the preset number of updates are fused to obtain the adapter weights. The adapter weights are then fused with the weights of the network layers in the second target model to obtain the trained target model for the target professional domain. This not only improves the ability of the trained target model to handle question-answering tasks in the target professional domain, but also maintains its ability to handle general domain question-answering tasks, preventing the trained target model from overfitting to the training samples and improving its generalization ability.

[0175] 102. Input the problem and function toolset into the trained target model for processing, and generate prediction instructions that call the function tools in the function toolset.

[0176] In this process, after inputting the question and the set of function tools into the trained target model, the trained target model can predict which function tool in the set of function tools will provide the answer to the question, and then generate a prediction instruction to call that function tool.

[0177] For example, the function toolkit includes function tool c1, function tool c2, and function tool c3. The prediction is to obtain the answer to the question through function tool c2 and generate the prediction instruction to call function tool c2.

[0178] 103. Based on the function tools called by the prediction instructions and the function tools called by the instructions, evaluate the ability of the trained target model to call the function tools in the function tool set to handle question-answering tasks.

[0179] The prediction instruction includes a tool name, and the instruction also includes a tool name. The function tool called by the prediction instruction refers to the function tool corresponding to the tool name in the prediction instruction, and the function tool called by the instruction refers to the function tool corresponding to the tool name in the instruction. The function tool called by the instruction is like a label, and based on the label, it is determined whether the function tool called by the prediction instruction is correct.

[0180] For the same problem, if the function tool called by the predicted instruction is the same as the function tool called by the instruction, it indicates that the target training model's call to that function tool is accurate after training. If the function tool called by the predicted instruction is different from the function tool called by the instruction, it indicates that the target training model's call to that function tool is incorrect after training, and other function tools in the function tool set are interfering with the call to that function tool.

[0181] For example, the evaluation sample includes a question q1 and an instruction d1. Instruction d1 is an instruction to call function tool c3. The function tool set includes function tool c1, function tool c2, and function tool c3. The question q1 and the function tool set are input into the trained target model for processing to obtain the prediction instruction d11. If the prediction instruction d11 is a prediction instruction to call function tool c3, it indicates that the call to function tool c3 is accurate. If the prediction instruction d11 is a prediction instruction to call function tool c2, it indicates that the call to function tool c3 is incorrect, and function tool c2 interferes with the call to function tool c3.

[0182] After receiving the prediction instruction, the terminal can count the number of correct and incorrect calls to each function tool according to the function tools called by the prediction instruction and the instruction itself. Finally, based on the statistical results of each function tool, the terminal can evaluate the ability of the trained target model to call the function tools in the function tool set to handle question answering tasks.

[0183] In this embodiment, based on the function tools called by the prediction instructions and the function tools called by the instructions, the ability of the trained target model to call the function tools in the function tool set to handle question-answering tasks is evaluated. This simplifies the evaluation task of the trained target model's ability to handle question-answering tasks into a function tool classification task, so as to obtain the calling ability of the trained target model on each function tool, refine the granularity of the evaluation of the trained target model's ability to handle question-answering tasks, and improve the accuracy of the evaluation of the trained target model's ability to handle question-answering tasks.

[0184] In some embodiments, the ability of the trained target model to call function tools in the function tool set to handle question-answering tasks is evaluated based on the function tools invoked by the prediction instruction and the function tools invoked by the instruction, including:

[0185] Based on the function tools called by the prediction command and the function tools called by the command, the influence coefficient between the function tools is determined. The influence coefficient is used to indicate the degree of mutual influence between the function tools.

[0186] Based on the impact coefficient, we evaluate the ability of the target model after training to call functions in the function toolkit to handle question-answering tasks.

[0187] In this embodiment, when the function tool called by the prediction instruction is different from the function tool called by the instruction, it can be determined that the function tool called by the prediction instruction is interfering with the function tool called by the instruction. This allows us to obtain the degree of mutual influence between the various function tools, so that we can adjust the training samples according to the degree of mutual influence between the various function tools. Based on the training samples, we can continue to fine-tune the target model after training. Therefore, the evaluation method of this embodiment can provide guidance for the fine-tuning of the target model after training, and improve the efficiency of obtaining the final target model.

[0188] In some embodiments, the ability of the trained target model to call function tools in the function tool set to handle question-answering tasks is evaluated based on the function tools invoked by the prediction instruction and the function tools invoked by the instruction, including:

[0189] Based on the function tools called by the prediction command and the function tools called by the command, a confusion matrix is ​​constructed. The column data in the confusion matrix indicates the function tools called by the command, and the row data in the confusion matrix indicates the function tools called by the prediction command.

[0190] Based on the confusion matrix, we evaluate the ability of the trained target model to call functions from the function toolkit to handle question-answering tasks.

[0191] The confusion matrix, also known as the error matrix, is used to show the classification results of the target model on the function tool after training.

[0192] For example, a confusion matrix can be like this: Figure 3As shown, from Figure 3 As can be seen, there are 11 evaluation samples for function tool c1. Among the predicted instructions generated based on the problems in the 11 evaluation samples, the function tool in 8 of the predicted instructions is function tool c1, indicating that the 8 predicted instructions are accurate and the call to function tool c1 is accurate. The function tool in 3 of the predicted instructions is function tool c3, indicating that the call to function tool c1 was misidentified as a call to function tool c3, indicating that there was an error in the call to function tool c3.

[0193] In this embodiment, a confusion matrix is ​​constructed based on the function tools called by the prediction instruction and the function tools called by the instruction. The column data in the confusion matrix indicates the function tools called by the instruction, and the row data in the confusion matrix indicates the function tools called by the prediction instruction. Based on the confusion matrix, the ability of the target model after training to call the function tools in the function tool set to handle question-answering tasks is evaluated. This allows for not only an intuitive representation of the classification results of the target model after training on the function tools, but also a comprehensive evaluation of the target model after training.

[0194] In some embodiments, based on the confusion matrix, the ability of the trained target model to invoke functions from a set of functions to handle question-answering tasks is evaluated, including:

[0195] Based on the row data in the confusion matrix, the influence coefficients between the function tools are determined. The influence coefficients are used to indicate the degree of mutual influence between the function tools.

[0196] Based on the impact coefficient, we evaluate the ability of the target model after training to call functions in the function toolkit to handle question-answering tasks.

[0197] The type of influence coefficient can be set according to the actual situation. For example, the influence coefficient can be at least one of True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). This application embodiment does not limit this.

[0198] True positives represent the number of evaluation samples that predict a function tool as that function tool. True negatives represent the number of evaluation samples that predict other function tools as other function tools. False positives represent the number of evaluation samples that predict other function tools as that function tool, which can represent the interference that function tool causes to other function tools. False negatives represent the number of evaluation samples that predict a function tool as other function tools, which can represent the interference that other function tools cause to that function tool.

[0199] In this embodiment, the influence coefficients between functional tools are determined based on the row data in the confusion matrix. The influence coefficients are used to indicate the degree of mutual influence between functional tools. Based on the influence coefficients, the ability of the trained target model to call functional tools in the functional tool set to handle question-answering tasks is evaluated. This allows the training samples to be adjusted according to the degree of mutual influence between the various functional tools. Based on the training samples, the trained target model can be further fine-tuned. Therefore, the evaluation method of this embodiment can provide guidance for the fine-tuning of the trained target model and improve the efficiency of obtaining the final target model.

[0200] In some embodiments, the ability of the trained target model to invoke functions from a set of functions to handle question-answering tasks is evaluated based on an impact coefficient, including:

[0201] Based on the influence coefficient, call capability indicator parameters are determined for each function tool. The call capability indicator parameters include at least one of accuracy, precision, recall, F1 score, specificity, and area under the ROC curve.

[0202] Based on the call capability indicator parameter for each function tool, the ability of the trained target model to call each function tool is determined.

[0203] For example, based on the influence coefficient of function tool c1, determine the calling capability indicator parameter of function tool c1; based on the calling capability indicator parameter of function tool c1, determine the ability of the target model to call function tool c1 after training; based on the influence coefficient of function tool c2, determine the calling capability indicator parameter of function tool c2; based on the calling capability indicator parameter of function tool c2, determine the ability of the target model to call function tool c2 after training.

[0204] In the embodiments of this application, based on the influence coefficient, the calling capability indicator parameter for each function tool is determined. The calling capability indicator parameter includes at least one of accuracy, precision, recall, F1 score, specificity, and area under the ROC curve. Based on the calling capability indicator parameter for each function tool, the ability of the trained target model to call the function tool is determined, thereby obtaining the performance capability of the trained target model on each function tool.

[0205] In some embodiments, the ability of the trained target model to invoke functions from a set of functions to handle question-answering tasks is evaluated based on an impact coefficient, including:

[0206] Based on the influence coefficient, determine the overall evaluation coefficient of the target model after training for each function tool;

[0207] Based on the overall evaluation coefficient, the ability of the target model after training to call functions in the function toolkit to handle question-answering tasks is evaluated.

[0208] The process of determining the overall evaluation coefficients of the trained target model for each function tool based on the influence coefficient can be described as follows:

[0209] Based on the impact coefficient, the invocation capability indicator parameters for each function tool are determined. The invocation capability indicator parameters include at least one of precision and recall.

[0210] The call capability indicator parameters are weighted to obtain the overall evaluation coefficient of the target model for each function tool after training; or the call capability indicator parameters are averaged to obtain the overall evaluation coefficient of the target model for each function tool after training.

[0211] For example, when the invocation capability indicator parameter is precision, the overall evaluation coefficient can be weighted precision or average precision; when the invocation capability indicator parameter is recall, the overall evaluation coefficient can be weighted recall.

[0212] The weighted precision rate can be calculated using the following formula:

[0213]

[0214] p_i represents the accuracy of the i-th function tool, Number_i represents the number of evaluation samples associated with the i-th function tool, and Total represents the total number of evaluation samples in the evaluation set.

[0215] Average precision can be calculated using the following formula:

[0216]

[0217] p_i represents the accuracy of the i-th function tool, and N represents the number of function tools in the function tool set.

[0218] In this embodiment, the overall evaluation coefficient of the trained target model for each function tool is determined based on the influence coefficient. Based on the overall evaluation coefficient, the ability of the trained target model to call the function tools in the function tool set to handle question-answering tasks is evaluated. This realizes the evaluation of the trained target model based on the overall evaluation coefficient, thereby further improving the accuracy of the evaluation of the trained target model.

[0219] In some embodiments, after receiving the prediction instruction, the method further includes:

[0220] Based on the target dimension, the correctness of the prediction instruction is analyzed to obtain the analysis results. The target dimension includes at least one of the following: the structure of the prediction instruction, the parameters in the prediction instruction, and the applicable scenario of the prediction instruction.

[0221] Based on the function tools invoked by the prediction instructions and the function tools invoked by the instructions, evaluate the ability of the trained target model to invoke the function tools set to handle question-answering tasks, including:

[0222] Based on the function tools invoked by the prediction instructions, the function tools invoked by the instructions, and the analysis results, evaluate the ability of the trained target model to invoke the function tools in the function toolset to handle question-answering tasks.

[0223] When the target dimension is the structure of the prediction instruction, the process of analyzing the correctness of the prediction instruction based on its structure and obtaining the analysis results can be as follows:

[0224] Determine whether the prediction instruction includes the tool name and parameters;

[0225] If the prediction instruction includes a tool name and parameters, it indicates that the structure of the prediction instruction is correct, and the analysis result will be determined as the correct result.

[0226] When the target dimension is a parameter in the prediction instruction, the process of analyzing the correctness of the prediction instruction based on the parameters of the prediction instruction and obtaining the analysis results can be as follows:

[0227] The analysis results are obtained by examining whether the parameters in the prediction instruction are missing, whether there are unnecessary parameters, whether the parameter types are incorrect, and whether the parameters match.

[0228] When the target dimension is the applicable scenario of the prediction instruction, the process of analyzing the correctness of the prediction instruction based on the applicable scenario and obtaining the analysis results can be as follows:

[0229] Determine the applicable scenarios for prediction instructions, which indicate whether or not function tools need to be invoked;

[0230] Based on the applicable scenarios of the prediction instructions, the correctness of the prediction instructions is analyzed, and the analysis results are obtained.

[0231] In particular, when the application scenario of the prediction instruction is that it requires calling a function tool, it may be that the function tool should not be called when it is not required.

[0232] In this embodiment, the correctness of the predicted instructions is analyzed based on the target dimension to obtain the analysis results. The target dimension includes at least one of the following: the structure of the predicted instructions, the parameters in the predicted instructions, and the applicable scenarios of the predicted instructions. Based on the function tools called according to the predicted instructions, the function tools called by the instructions, and the analysis results, the ability of the trained target model to call function tools in the function tool set to handle question-answering tasks is evaluated. This achieves evaluation of the trained target model from multiple dimensions, further improving the accuracy of the evaluation of the trained target model.

[0233] In some embodiments, when the question-answering data includes real question-answering data and simulated question-answering data, the terminal can generate evaluation samples based on the real and simulated question-answering data. Based on the evaluation samples, the number of correct prediction instructions obtained through the trained target model and the number of correct prediction instructions obtained through the second target model are determined. Based on the number of correct prediction instructions, the accuracy of the trained target model and the accuracy of the second target model are obtained. The accuracy of the trained target model indicates its ability to handle question-answering tasks, and the accuracy of the second target model indicates its ability to handle question-answering tasks. This allows for a comparison of the ability of the trained target model and the second target model to handle question-answering tasks. For example, ... Figure 4 As shown.

[0234] As can be seen from the above, in this embodiment, an evaluation set including at least one evaluation sample is obtained. The evaluation sample includes question-and-answer data related to function tools in the function tool set, as well as the function tool set itself. The question-and-answer data includes questions and instructions to call function tools based on the questions. The questions and function tool sets are input into the trained target model for processing to generate predicted instructions to call function tools in the function tool set. Based on the function tools called by the predicted instructions and the function tools called by the instructions, the ability of the trained target model to call function tools in the function tool set to handle question-and-answer tasks is evaluated. This simplifies the evaluation task of the trained target model's ability to handle question-and-answer tasks into a function tool classification task, enabling the evaluation of the trained target model's performance on various function tools, refining the granularity of evaluating the trained target model's ability to handle question-and-answer tasks, and improving the accuracy of evaluating the trained target model's ability to handle question-and-answer tasks.

[0235] To better implement the above methods, this application also provides a model evaluation device, which can be integrated into an electronic device, such as a computer device, which can be a terminal, server or other device.

[0236] The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0237] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the model evaluation device specifically integrated into the terminal as an example. This embodiment provides a model evaluation device, such as... Figure 5 As shown, the model evaluation device may include:

[0238] The acquisition module 501 is used to acquire an evaluation set including at least one evaluation sample. The evaluation sample includes question and answer data related to the function tools in the function toolset and the function toolset. The question and answer data includes questions and instructions to call the function tools based on the questions.

[0239] The processing module 502 is used to input the problem and the function tool set into the trained target model for processing, and generate prediction instructions that call the function tools in the function tool set.

[0240] Evaluation module 503 is used to evaluate the ability of the trained target model to call the function tools in the function toolset set to handle question-answering tasks based on the function tools called by the prediction instructions and the function tools called by the instructions.

[0241] In some embodiments, the evaluation module 503 is specifically used to perform:

[0242] Based on the function tools called by the prediction command and the function tools called by the command, a confusion matrix is ​​constructed. The column data in the confusion matrix indicates the function tools called by the command, and the row data in the confusion matrix indicates the function tools called by the prediction command.

[0243] Based on the confusion matrix, we evaluate the ability of the trained target model to call functions from the function toolkit to handle question-answering tasks.

[0244] In some embodiments, the evaluation module 503 is specifically used to perform:

[0245] Based on the row data in the confusion matrix, the influence coefficients between the function tools are determined. The influence coefficients are used to indicate the degree of mutual influence between the function tools.

[0246] Based on the impact coefficient, we evaluate the ability of the target model after training to call functions in the function toolkit to handle question-answering tasks.

[0247] In some embodiments, the evaluation module 503 is specifically used to perform:

[0248] Based on the influence coefficient, determine the overall evaluation coefficient of the target model after training for each function tool;

[0249] Based on the overall evaluation coefficient, the ability of the target model after training to call functions in the function toolkit to handle question-answering tasks is evaluated.

[0250] In some embodiments, the evaluation module 503 is further configured to perform:

[0251] Based on the target dimension, the correctness of the prediction instruction is analyzed to obtain the analysis results. The target dimension includes at least one of the following: the structure of the prediction instruction, the parameters in the prediction instruction, and the applicable scenario of the prediction instruction.

[0252] Based on the function tools invoked by the prediction instructions, the function tools invoked by the instructions, and the analysis results, evaluate the ability of the trained target model to invoke the function tools in the function toolset to handle question-answering tasks.

[0253] In some embodiments, the acquisition module 501 is specifically used to perform:

[0254] Obtain the function toolkit, which includes the function tools, their functional descriptions, and parameter descriptions of the input parameters required to call the function tools;

[0255] Based on the first target model, and according to the functional and parameter descriptions, generate questions related to the function tools and instructions for calling the function tools.

[0256] Based on the problem, instruction, and function toolset, test samples are generated, and a test set is generated based on the test samples.

[0257] In some embodiments, the acquisition module 501 is specifically used to perform:

[0258] Based on the first target model, and according to the functional and parameter descriptions, instructions for calling the function tool are generated.

[0259] The first target model predicts the input question needed to trigger the generation instruction.

[0260] In practice, each of the above modules can be implemented as an independent entity or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation methods and corresponding beneficial effects of each of the above modules, please refer to the previous method embodiments, which will not be repeated here.

[0261] Accordingly, embodiments of this application also provide an electronic device, which can be a terminal, such as a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. Figure 6 As shown, Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 600 includes a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, and a computer program stored on the memory 602 and executable on the processor. The processor 601 and the memory 602 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0262] The processor 601 is the control center of the electronic device 600. It connects various parts of the electronic device 600 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it performs various functions of the electronic device 600 and processes data, thereby monitoring the electronic device 600 as a whole.

[0263] In this embodiment, the processor 601 in the electronic device 600 loads the instructions corresponding to the processes of one or more applications into the memory 602 according to the following steps, and the processor 601 runs the applications stored in the memory 602 to achieve various functions, such as:

[0264] Obtain an evaluation set including at least one evaluation sample, the evaluation sample including question and answer data related to function tools in the function tool set and the function tool set, the question and answer data including questions and instructions to call function tools based on questions;

[0265] The problem and function toolset are input into the trained target model for processing, generating prediction instructions that call the function tools in the function toolset;

[0266] Based on the function tools invoked by the prediction instructions and the function tools invoked by the instructions, evaluate the ability of the trained target model to invoke the function tools in the function toolset to handle question-answering tasks.

[0267] For details on the specific implementation methods and corresponding beneficial effects of each of the above operations, please refer to the detailed description of the model evaluation method above, which will not be repeated here.

[0268] Optional, such as Figure 6 As shown, the electronic device 600 also includes: a touch display screen 603, a radio frequency circuit 604, an audio circuit 605, an input unit 606, and a power supply 607. The processor 601 is electrically connected to the touch display screen 603, the radio frequency circuit 604, the audio circuit 605, the input unit 606, and the power supply 607. Those skilled in the art will understand that... Figure 6The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0269] The touch display screen 603 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 603 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), or other similar technology. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 601. It can also receive and execute commands from the processor 601. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 601 to determine the type of touch event. Subsequently, the processor 601 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 603 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 603 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 603 can also be used as part of the input unit 606 to achieve input functions.

[0270] The radio frequency circuit 604 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0271] Audio circuit 605 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuit 605 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 605, converted back into audio data, and then processed by processor 601 before being transmitted via radio frequency circuit 604 to, for example, another electronic device, or output to memory 602 for further processing. Audio circuit 605 may also include an earphone jack to provide communication between peripheral headphones and electronic devices.

[0272] The input unit 606 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0273] Power supply 607 is used to supply power to various components of electronic device 600. Optionally, power supply 607 can be logically connected to processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 607 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0274] although Figure 6 As not shown in the diagram, the electronic device 600 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0275] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0276] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0277] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute any of the model evaluation methods provided in embodiments of this application. For example, the computer program can perform the following steps:

[0278] Obtain an evaluation set including at least one evaluation sample, the evaluation sample including question and answer data related to function tools in the function tool set and the function tool set, the question and answer data including questions and instructions to call function tools based on questions;

[0279] The problem and function toolset are input into the trained target model for processing, generating prediction instructions that call the function tools in the function toolset;

[0280] Based on the function tools invoked by the prediction instructions and the function tools invoked by the instructions, evaluate the ability of the trained target model to invoke the function tools in the function toolset to handle question-answering tasks.

[0281] For details on the specific implementation methods and corresponding beneficial effects of each of the above operations, please refer to the detailed description of the model evaluation method above, which will not be repeated here.

[0282] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0283] Since the computer program stored in the computer-readable storage medium can execute any of the model evaluation methods provided in the embodiments of this application, it can achieve the beneficial effects that any of the model evaluation methods provided in the embodiments of this application can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0284] The foregoing has provided a detailed description of a model evaluation method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model evaluation method, characterized in that, The method includes: Obtain an evaluation set including at least one evaluation sample, the evaluation sample including question and answer data related to function tools in a function tool set and the function tool set, the question and answer data including questions and instructions to call the function tools based on the questions; The problem and the function tool set are input into the trained target model for processing, generating prediction instructions that call the function tools in the function tool set; Based on the function tool invoked by the prediction instruction and the function tool invoked by the instruction, a confusion matrix is ​​constructed, wherein the column data in the confusion matrix indicates the function tool invoked by the instruction, and the row data in the confusion matrix indicates the function tool invoked by the prediction instruction; Based on the confusion matrix, evaluate the ability of the trained target model to generate an answer to the input question by invoking the function tools in the function toolkit.

2. The model evaluation method as described in claim 1, characterized in that, The evaluation of the ability of the trained target model to call the function tools in the function toolkit to process question-answering tasks based on the confusion matrix includes: Based on the row data in the confusion matrix, the influence coefficients between the function tools are determined, and the influence coefficients are used to indicate the degree of mutual influence between the function tools. Based on the influence coefficient, the ability of the trained target model to call the function tools in the function toolkit to process question-answering tasks is evaluated.

3. The model evaluation method as described in claim 2, characterized in that, The evaluation of the ability of the trained target model to call the function tools in the function toolkit to handle question-answering tasks based on the influence coefficient includes: Based on the influence coefficient, the overall evaluation coefficient of the trained target model for each of the function tools is determined; Based on the overall evaluation coefficient, the ability of the trained target model to call the function tools in the function toolkit to process question-answering tasks is evaluated.

4. The model evaluation method as described in claim 1, characterized in that, After receiving the prediction instruction, the process also includes: The correctness of the prediction instruction is analyzed based on the target dimension to obtain the analysis results. The target dimension includes at least one of the following: the structure of the prediction instruction, the parameters in the prediction instruction, and the applicable scenarios of the prediction instruction. The evaluation of the ability of the trained target model to call the function tools in the function tool set to handle question-answering tasks, based on the function tools invoked according to the prediction instruction and the function tools invoked according to the instruction, includes: Based on the function tools invoked by the prediction instruction, the function tools invoked by the instruction, and the analysis results, evaluate the ability of the trained target model to invoke the function tools in the function toolset to process question-answering tasks.

5. The model evaluation method according to any one of claims 1-4, characterized in that, The acquisition of the evaluation set, which includes at least one evaluation sample, includes: Obtain a set of function tools, which includes function tools, function descriptions of the function tools, and parameter descriptions of the parameters required to call the function tools; Based on the first target model, the function description and parameter description are used to generate questions related to the function tool and instructions for calling the function tool. Based on the problem, the instructions, and the function toolkit, the evaluation samples are generated, and an evaluation set is generated based on the evaluation samples.

6. The model evaluation method as described in claim 5, characterized in that, The step of generating questions related to the function tool and instructions for calling the function tool based on the first target model, according to the function description and the parameter description, includes: Based on the first target model, and according to the function description and the parameter description, an instruction to call the function tool is generated; The first target model is used to predict the input required to trigger the generation of the instruction.

7. A model evaluation device, characterized in that, The device includes: An acquisition module is used to acquire an evaluation set including at least one evaluation sample, wherein the evaluation sample includes question and answer data related to function tools in a function tool set and the function tool set, wherein the question and answer data includes questions and instructions to call the function tools based on the questions; The processing module is used to input the problem and the function tool set into the trained target model for processing, and generate prediction instructions to call the function tools in the function tool set; An evaluation module is used to construct a confusion matrix based on the function tools invoked by the prediction instruction and the function tools invoked by the instruction. The column data in the confusion matrix indicates the function tools invoked by the instruction, and the row data in the confusion matrix indicates the function tools invoked by the prediction instruction. Based on the confusion matrix, the module evaluates the ability of the trained target model to invoke the function tools in the function toolset to generate an answer to the input question based on the input question.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to execute the model evaluation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the model evaluation method according to any one of claims 1 to 6.

Images