Model evaluation method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a model evaluation method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining an evaluation set comprising at least one evaluation sample, the evaluation sample comprising question and answer data related to a function tool in a function tool set and the function tool set; the question and answer data comprises a question and an instruction for calling the function tool based on the question; inputting the problem and the function tool set into a trained target model for processing, and generating a prediction instruction for calling the function tools in the function tool set; according to the function tool called by the prediction instruction and the function tool called by the instruction, the ability of the trained target model for calling the function tool in the function tool set to process the question and answer task is evaluated, and the accuracy of model evaluation can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of neural network models, and in particular to a model evaluation method, device, electronic device and computer-readable storage medium. Background Art

[0002] Large Language Model (LLM) refers to a deep learning model trained with large amounts of text data that can handle question-answering tasks.

[0003] When processing question-answering tasks, the large language model can call various function tools. Currently, the ability of the large language model to call function tools to process question-answering tasks is evaluated based on the answers output by the large language model. However, the accuracy of the method based on the output answers is low. Summary of the invention

[0004] The embodiments of the present application provide a model evaluation method, device, electronic device and computer-readable storage medium, which can improve the accuracy of model evaluation.

[0005] In a first aspect, an embodiment of the present application provides a model evaluation method, the method comprising:

[0006] Obtaining an evaluation set including at least one evaluation sample, wherein the evaluation sample includes question-and-answer data related to a function tool in a function tool set and the function tool set, wherein the question-and-answer data includes a question and an instruction for calling the function tool based on the question;

[0007] Input the above problem and the above function tool set into the trained target model for processing, and generate prediction instructions that call the above function tools in the above function tool set;

[0008] Based on the function tools called by the above prediction instructions and the function tools called by the above instructions, the ability of the above trained target model to call the above function tools in the above function tool set to process question and answer tasks is evaluated.

[0009] In a second aspect, an embodiment of the present application further provides a model evaluation device, the device comprising:

[0010] An acquisition module, configured to acquire an evaluation set including at least one evaluation sample, wherein the evaluation sample includes question-and-answer data related to a function tool in a function tool set and the function tool set, wherein the question-and-answer data includes a question and an instruction for calling the function tool based on the question;

[0011] A processing module, used for inputting the above problem and the above function tool set into the trained target model for processing, and generating a prediction instruction for calling the above function tools in the above function tool set;

[0012] An evaluation module is used to evaluate the ability of the trained target model to process question-answering tasks by calling the function tools in the function tool set based on the function tools called by the prediction instructions and the function tools called by the instructions.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory storing a plurality of instructions; the above-mentioned processor loads instructions from the above-mentioned memory to execute any one of the model evaluation methods provided in the embodiments of the present application.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute any model evaluation method provided in an embodiment of the present application.

[0015] In an embodiment of the present application, an evaluation set including at least one evaluation sample is obtained, the evaluation sample including question and answer data related to a function tool in a function tool set and a function tool set, the question and answer data including questions and instructions for calling the function tool based on the questions; the questions and the function tool set are input into a trained target model for processing to generate prediction instructions for calling the function tool in the function tool set; based on the function tool called by the prediction instruction and the function tool called by the instruction, the ability of the trained target model to call the function tool in the function tool set to process question and answer tasks is evaluated, thereby simplifying the evaluation task of the ability of the trained target model to process question and answer tasks into a classification task of function tools, so that the performance ability of the trained target model on each function tool can be obtained, the granularity of evaluating the ability of the trained target model to process question and answer tasks is refined, and the accuracy of evaluating the ability of the trained target model to process question and answer tasks is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a schematic diagram of an embodiment of the model evaluation method provided in the embodiment of the present application;

[0018] Figure 2 is a schematic diagram of the model training process provided in the embodiments of the present application;

[0019] Figure 3 is a schematic diagram of a confusion matrix provided in an embodiment of the present application;

[0020] Figure 4is a schematic diagram of another embodiment of the model evaluation method provided in the embodiments of the present application;

[0021] Figure 5 is a schematic diagram of the structure of a model evaluation device provided in an embodiment of the present application;

[0022] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0024] Before explaining the embodiments of the present application in detail, some terms involved in the embodiments of the present application are first explained.

[0025] Among them, in the description of the embodiment of the present application, the terms "first", "second", etc. can be used in this article to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. "Multiple" refers to two or more.

[0026] The embodiments of the present application provide a model evaluation method, device, electronic device and computer-readable storage medium. Specifically, the model evaluation method of the embodiments of the present application can be executed by an electronic device, wherein the electronic device can be a terminal or a server.

[0027] The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), etc. The terminal can also include a client, which can be a game application client, a browser client carrying a game program, or an instant messaging client, etc.

[0028] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0029] The following is a detailed description of the embodiments in conjunction with the accompanying drawings. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0030] In this embodiment, a terminal is used as an example for explanation. This embodiment provides a model evaluation method, such as Figure 1 As shown, the specific process of the model evaluation method can be as follows:

[0031] 101. Obtain an evaluation set including at least one evaluation sample, where the evaluation sample includes question and answer data related to a function tool in a function tool set and the function tool set, where the question and answer data includes questions and instructions for calling the function tool based on the questions.

[0032] Among them, the function tool refers to a tool obtained by encapsulating the code that implements certain functions, which may include at least one. The type of code that implements certain functions can be set according to actual conditions, for example, the code that implements certain functions can be a function, an interface, a model, or a workflow, etc., which is not limited in this embodiment. Certain functions that are implemented can be set according to actual conditions, for example, the functions that are implemented can be a weather query function or a product query function, which is not limited in this embodiment.

[0033] Optionally, the function tool may include a function tool in a general field and / or a function tool in a target professional field, which is not limited in this embodiment of the present application. When the function tool is a function tool in a target professional field, the target model after training can be a model that is good at processing question-and-answer tasks in the target professional field. The evaluation set is used to evaluate the ability of the target model after training to call the function tool to process question-and-answer tasks in the target professional field. When the function tool is a function tool in a general field, the evaluation set is used to evaluate the ability of the target model after training to call the function tool to process question-and-answer tasks in the general field. The target professional field refers to a specific knowledge category, which can be set according to actual conditions. For example, the target professional field can be the medical field or the automotive field, which is not limited in this embodiment.

[0034] Optionally, the question-and-answer data related to the function tool may include at least one of real question-and-answer data and simulated question-and-answer data. Real question-and-answer data refers to manually generated question-and-answer data, that is, questions and instructions corresponding to the questions are manually generated by the user. Simulated question-and-answer data refers to question-and-answer data automatically generated by the first target model based on the function description and parameter description of the function tool.

[0035] In some embodiments, when the question and answer data related to the function tool includes simulated question and answer data, it is possible to eliminate the need to manually generate the question and answer data, thereby improving the efficiency of generating the question and answer data and thus improving the efficiency of model evaluation.

[0036] In some embodiments, when the question and answer data related to the function tool includes real question and answer data and simulated question and answer data, not only the efficiency of model evaluation can be improved, but also the accuracy of model evaluation can be further improved.

[0037] In some embodiments, when the question-and-answer data associated with the function tool includes simulated question-and-answer data, obtaining a test set including at least one test sample includes:

[0038] Obtain a function tool set, which includes a function tool, a functional description of the function tool, and a parameter description of the parameters required to be input when calling the function tool;

[0039] Generate questions related to the function tool and instructions for calling the function tool according to the function description and the parameter description through the first target model;

[0040] Generate evaluation samples based on questions, instructions and function tool sets, and generate evaluation sets based on the evaluation samples.

[0041] Among them, the first target model can refer to a trained neural network model, for example, the first target model can refer to a large language model (LLM), which can process natural language tasks, including question-answering tasks. The specific type of the large language model can be based on actual conditions, for example, the first target model can be Azure-OpenAI-GPT-3.5 or Microsoft Azure-GPT-4.0, which is not limited in this embodiment of the present application.

[0042] The functional description of a function tool refers to information describing the function of the function tool. For example, if the function of the function tool is a weather query function, the functional description of the function tool may be that the function is used to query the weather of a certain place.

[0043] When calling a function tool, some parameters need to be input, and the description of these parameters is the parameter description. When the functions implemented by function tools are different, the parameters required to call the function tools are different, and the parameter descriptions are also different. For example, if the function of the function tool is a weather query function, the parameters required to be input are location and time, and the parameter description can be that parameters representing location and time need to be input. Another example is that if the function of the function tool is a salary query function, the parameter required to be input is name, and the parameter description is that a parameter representing name needs to be input.

[0044] In an embodiment of the present application, a function tool set is obtained. The function tool set includes function tools, function descriptions of the function tools, and parameter descriptions of the parameters required to call the function tools. Through a first target model, based on the function description and the parameter description, problems related to the function tools and instructions for calling the function tools are generated. Based on the problems, instructions, and the function tool set, an evaluation sample is generated, and an evaluation set is generated according to the evaluation sample, realizing the automatic generation of the evaluation set and improving the generation efficiency of the evaluation set.

[0045] In some embodiments, through the first target model, based on the function description and the parameter description, problems can be generated first, and then instructions can be generated according to the problems. Or, instructions can be generated first, and then problems can be generated. When generating instructions first and then problems, through the first target model, based on the function description and the parameter description, problems related to the function tools and instructions for calling the function tools are generated, including:

[0046] Through the first target model, based on the function description and the parameter description, an instruction for calling the function tool is generated;

[0047] Through the first target model, predict the problem that needs to be input to trigger the generation of the instruction.

[0048] In an embodiment of the present application, through the first target model, based on the function description and the parameter description, an instruction for calling the function tool is generated. Through the first target model, predict the problem that needs to be input to trigger the generation of the instruction, realizing generating instructions first and then problems, which can not only improve the generation efficiency of problems and instructions, but also make the quality of problems and instructions not seriously dependent on the ability of the first target model, and can also make the generated problems and instructions have diversity, structural completeness, and stability, further improving the accuracy of evaluating the ability of the target model to handle question-and-answer tasks after training.

[0049] For example, if the function implemented by the function tool is a salary query function, the generated instruction can be "salary_search(name = 'Zhang San')", and the problem that triggers the generation of this instruction can be "What is Zhang San's salary?".

[0050] In some embodiments, the terminal may generate instructions according to the function information of the function tool, wherein the function information includes the tool name, function description, parameter name, parameter description, parameter type, and parameter necessity of the function tool.

[0051] In some embodiments, generating instructions for calling a function tool according to a function description and a parameter description through a first target model includes:

[0052] Generate target parameters required to be input for calling the function tool according to the function description and the parameter description through the first target model;

[0053] Assemble the target parameter and the tool name of the function tool to obtain an instruction for calling the function tool.

[0054] The target parameter required to be input when calling the function tool refers to the specific value of the parameter required to be input when calling the function tool. For example, the function of the function tool is the weather query function, the parameters required to be input when calling the function tool are the parameters representing the location and time, and the target parameters required to be input when calling the function tool are Wuhan and tomorrow.

[0055] After obtaining the target parameter, the terminal may assemble the target parameter and the tool name of the function tool to obtain an instruction for calling the function tool.

[0056] For example, the function of the function tool is the salary query function, the target parameter is "Zhang San", the tool name is salary_search, and the instruction for calling the function tool can be "salary_search(name='Zhang San')".

[0057] It can be understood that when there are multiple parameters required to be input for calling a function tool, and each of the required input parameters corresponds to multiple target parameters, the terminal can first generate each target parameter, then combine the various target parameters to obtain a combined parameter, and then assemble the combined parameter and the tool name of the function tool to obtain an instruction to call the function tool.

[0058] For example, the parameters required to call a function tool include parameters representing time and parameters representing location. The target parameters generated for the parameters representing time include tomorrow and the day after tomorrow. The target parameters generated for the parameters representing location include Wuhan and Shenzhen. Tomorrow and Wuhan are combined to obtain a combined parameter. Tomorrow and Shenzhen are combined to obtain a combined parameter. The day after tomorrow and Shenzhen are combined to obtain a combined parameter. The day after tomorrow and Wuhan are combined to obtain a combined parameter. A total of 4 combined parameters are obtained. The 4 combined parameters are then combined with the tool name respectively to obtain 4 instructions for calling the function tool.

[0059] Alternatively, when the parameters required to be input for calling a function tool include multiple parameters, and each required input parameter corresponds to multiple target parameters, the terminal can directly generate a combined parameter, and the combined parameter includes at least two target parameters. For example, when the parameters required to be input for calling a function tool include a parameter indicating time and a parameter indicating location, the terminal can directly generate a target parameter "Wuhan" and a target parameter "tomorrow" to obtain a combined parameter.

[0060] Optionally, when one of the target parameters affects the generation of other target parameters, a method of directly generating the combined parameter through the first target model may be adopted.

[0061] Optionally, when generating the target parameters required for inputting a function tool through the first target model according to the function description and parameter description, the parameter generation prompt template can also be input into the first target model, so as to generate the target parameters required for inputting a function tool through the first target model according to the function description, parameter description and parameter generation prompt template.

[0062] Optionally, if samples or generation rules are required to generate target parameters, the samples or generation rules can be stored in the parameter description. In this case, the parameter description not only includes a description of the parameters required to be input, but also includes samples or generation rules, so that the first target model can refer to the samples or generation rules and generate the target parameters according to the function description and parameter description.

[0063] In some embodiments, before predicting, by the first target model, the question that needs to be input for triggering the generation of the instruction, the method further includes:

[0064] Perform quality inspection on the instruction and obtain the quality inspection result of the instruction;

[0065] Determine the instructions corresponding to the quality test results that meet the preset quality conditions,

[0066] Through the first target model, predict the questions that need to be input to trigger the generation instruction, including:

[0067] According to the target instruction, the information required to be input to trigger the generation of the target instruction is predicted. The target instruction refers to the instruction corresponding to the quality inspection result that meets the preset quality conditions.

[0068] The process of performing quality inspection on the instruction and obtaining the quality inspection result of the instruction may be:

[0069] The quality inspection of the instruction is performed according to at least one of the composition structure of the instruction, the tool name in the instruction, the parameter in the instruction and the type of the parameter in the instruction to obtain a quality inspection result of the instruction.

[0070] For example, when quality inspection is performed on the instruction according to the composition structure of the instruction, it can be analyzed whether the instruction includes the tool name and parameters. If the instruction includes the tool name and parameters, the quality inspection result of the instruction is represented as the first quality inspection result. The first quality inspection result represents the quality inspection result that meets the preset quality conditions.

[0071] For another example, when quality inspection is performed on an instruction based on the parameters in the instruction, it can be analyzed whether the parameters in the instruction are parameters that must appear and whether they are parameters that need to be input. If the parameters in the instruction are parameters that need to be input, it indicates that the quality inspection result of the instruction is a first quality inspection result. If the parameters in the instruction are parameters that do not need to be input, it indicates that the quality inspection result of the instruction is a second quality inspection result. The second quality inspection result is a quality inspection result that does not meet the preset quality conditions.

[0072] It can be understood that when quality inspection is performed on instructions based on multiple types of instruction composition structure, tool name in the instruction, parameters in the instruction, and parameter types in the instruction, the quality inspection result of the instruction can be determined based on various sub-quality inspection results.

[0073] For example, the instruction is quality checked based on its composition structure, and the obtained sub-quality check result is the first quality check result. The instruction is quality checked based on the parameters in the instruction, and the obtained sub-quality check result is the second quality check result. The quality check result of the instruction is determined based on the first quality check result and the second quality check result.

[0074] It can be understood that when the sub-quality detection result includes the second quality detection result, the quality detection result of the instruction can be determined as the second quality detection result.

[0075] In some embodiments, the question can be a short sentence or a long sentence. For example, when the function tool is a weather query function, the question can be "Wuhan, tomorrow's weather", "Wuhan, tomorrow's weather" is a short sentence, and for another example, the question can be "I want to ask about Wuhan's weather tomorrow", "I want to ask about Wuhan's weather tomorrow" is a long sentence.

[0076] In some embodiments, the question includes a target parameter. When the question includes all target parameters, the question can be called a complete question, and when the question includes some target parameters, the question can be called an incomplete question. The question of this embodiment can be a complete question and / or an incomplete question.

[0077] For example, the parameters required to call a function tool include parameters representing location and parameters representing time. A complete question may be "What will the weather be like in Wuhan tomorrow?" In this case, the complete question includes the target parameter "Wuhan" representing the location and the target parameter "tomorrow" representing the time. An incomplete question may be "What will the weather be like tomorrow?" In this case, the incomplete question only includes the target parameter "tomorrow" representing the time, and lacks the target parameter "Wuhan" for the location.

[0078] In an embodiment of the present application, when the question includes a complete question and an incomplete question, the complete question includes all the target parameters required to be input for calling the function tool, and the incomplete question includes some of the target parameters required to be input for calling the function tool. This can diversify the generated questions and further improve the accuracy of evaluating the ability of the trained target model to handle question and answer tasks.

[0079] In some embodiments, the present embodiment further includes:

[0080] Generate a supplementary question based on the incomplete question through the first target model, where the supplementary question is used to prompt the input of target parameters missing in the incomplete question;

[0081] Generate supplementary information based on the supplementary question through the first target model, the supplementary information including the missing target parameters;

[0082] Generate assessment samples based on instructions, questions, supplementary questions, and supplementary information.

[0083] Among them, a complete question and an incomplete question can be generated based on the instruction. Alternatively, a complete question can be generated based on the instruction, and then an incomplete question can be generated according to the target parameter.

[0084] After obtaining the incomplete question, a supplementary question is generated based on the incomplete question, and supplementary information is generated based on the supplementary question.

[0085] For example, the incomplete question is "I want to check the salary for January". The incomplete question lacks the target parameter representing the name. The generated supplementary question can be "Are you querying for January of this year? In addition, can you provide the name of the query?" Based on the supplementary question, the generated supplementary information can be, for example, "It is January 2023, and the name is Zhang San."

[0086] Specifically, the process of generating evaluation samples based on instructions, questions, supplementary questions and supplementary information can be:

[0087] Assemble the instructions, complete questions, and function tool sets to obtain evaluation samples;

[0088] Generate evaluation samples based on incomplete questions and supplementary questions;

[0089] Generates benchmark samples based on incomplete questions, supplementary questions, supplementary information, instructions, and function toolkits.

[0090] Among them, in the evaluation sample obtained by assembling the instructions, complete questions and function tool sets, the complete questions are used as data in the evaluation sample, and the instructions are used as labels of the evaluation sample. The evaluation sample is used to evaluate the ability of the target model to call the function tool after training.

[0091] In the evaluation samples generated based on incomplete questions and supplementary questions, the incomplete questions serve as the data in the evaluation samples, and the supplementary questions serve as the labels of the evaluation samples. The evaluation samples are used to evaluate the question-asking ability of the target model after training.

[0092] In the evaluation samples generated according to the incomplete questions, supplementary questions, supplementary information, instructions and function tool sets, the incomplete questions, supplementary questions and supplementary information are used as data, and the instructions are used as labels. The data samples are used to evaluate the ability of the target model to call the function tools after training.

[0093] It can be understood that the types of evaluation samples generated based on incomplete questions and supplementary questions and the types of evaluation samples generated based on incomplete questions, supplementary questions, supplementary information, instructions and function tool sets belong to multi-round dialogue types.

[0094] In an embodiment of the present application, a supplementary question is generated based on an incomplete question through the first target model, and the supplementary question is used to prompt for input of target parameters missing from the incomplete question; supplementary information is generated based on the supplementary question through the first target model, and the supplementary information includes the missing target parameters; evaluation samples are generated based on instructions, questions, supplementary questions and supplementary information, so that the evaluation of the trained target model is more comprehensive, thereby further improving the accuracy of the evaluation of the trained target model.

[0095] In some embodiments, the present embodiment further includes:

[0096] According to the instructions, call the function tool to get the parameter value corresponding to the problem;

[0097] Generate the answer to the question based on the parameter value;

[0098] Generate evaluation samples based on parameter values ​​and answers.

[0099] For example, the question is "What is Zhang San's salary in January 2023", and the instruction is "salary_search(name='Zhang San')". According to the instruction, a function tool is called to obtain the parameter values corresponding to the question as "{ 'emp_no': '123456', 'name': 'Zhang San', 'time': '2023 / 01','salary': 'xxxx'}", and the answer corresponding to the question is "Hello, it is found that Zhang San's salary in January 2023 is xxxx".

[0100] In the embodiment of the present application, according to the instruction, a function tool is called to obtain the parameter values corresponding to the question; according to the parameter values, the answer corresponding to the question is generated; based on the parameter values and the answer, an evaluation sample is generated. The evaluation sample generated based on the parameter values and the answer can be used to evaluate the ability of the trained target model to summarize the answer, making the evaluation of the trained target model more comprehensive and further improving the accuracy of the evaluation of the trained target model.

[0101] It can be understood that when generating an evaluation sample according to the supplementary question and supplementary information, an evaluation sample can also be generated according to the incomplete question, supplementary question, supplementary information, instruction, function tool set, parameter values and answer. In this evaluation sample, the incomplete question, supplementary question, supplementary information, instruction, and parameter values are used as data, and the answer is used as a label. This evaluation sample is used to evaluate the ability of the trained target model to summarize the parameter values returned by the function tool into an answer. The type of evaluation sample generated according to the incomplete question, supplementary question, supplementary information, instruction, parameter values and answer belongs to the multi-round dialogue type.

[0102] In some embodiments, the first target model is used to predict the question that needs to be input to trigger the generation of an instruction, including:

[0103] Obtain the set role information;

[0104] Through the first target model, according to the instruction and the role information, predict the question that needs to be input to trigger the generation of an instruction.

[0105] Among them, the role information refers to the information describing the user's identity. Different role information indicates different users. For example, the role information can be the information indicating the identity of the personnel or the information indicating the identity of an ordinary employee.

[0106] For example, if the function of the function tool is the salary query function, if the role information is the information indicating the identity of the personnel, the question that needs to be input to trigger the generation of an instruction can be "I need to query Zhang San's salary", and if the role information is the information indicating the identity of an ordinary employee, the question that needs to be input to trigger the generation of an instruction can be "I want to know how much Zhang San earns every month".

[0107] Since different users may input different questions for the same instruction, in this embodiment, the first target model is used to predict the questions that need to be input to trigger the generation of instructions based on the instructions and role information, so that the generated questions are diversified and the accuracy of the trained target model is further evaluated.

[0108] In some embodiments, after predicting the question that needs to be input to trigger the generation of the instruction through the first target model, the method further includes:

[0109] The problem is enhanced to obtain the enhanced problem.

[0110] The enhanced questions and instructions are used together to generate evaluation samples. The type of enhanced processing can be set according to actual conditions, for example, the enhanced processing includes at least one of replacement processing, rewriting processing, simplification processing and error addition processing, which is not limited in this embodiment.

[0111] Replacement processing refers to replacing non-critical content in the question, and the replaced content does not affect the generation of instructions. Rewriting processing refers to changing the grammar and questioning method without changing the critical content. Simplification processing refers to simplifying the question into a short sentence. Error addition processing refers to adding typos to the question.

[0112] In an embodiment of the present application, after the questions are obtained, the questions are enhanced so that the questions ultimately used to generate evaluation samples are diversified.

[0113] In some embodiments, based on the questions, instructions and function tool set, an evaluation sample is generated, including:

[0114] Assemble an instruction, the problem corresponding to the instruction, and the function tool set to obtain an evaluation sample;

[0115] The two instructions, the problems corresponding to the two instructions, and the function tool set are composed to obtain an evaluation sample.

[0116] The two instructions may be any two instructions in the generated instructions. Optionally, the order of the questions corresponding to the two instructions in the evaluation sample is determined based on the order of the two instructions in the evaluation sample. For example, the instruction for calling the function tool c1 is instruction d1, and the question that triggers the generation of instruction d1 is q1. The instruction for calling the function tool c2 is instruction d2, and the question that triggers the generation of instruction d2 is q2. In the evaluation sample generated according to instruction d1, question q1, instruction d2, and question q2, the order of instruction d1 precedes the order of instruction d2, and the order of question q1 precedes the order of question q2.

[0117] It can be understood that by composing two instructions, problems corresponding to the two instructions, and a function tool set, the type of evaluation samples obtained belongs to the parallel type.

[0118] In the present application, an instruction, the problem corresponding to the instruction, and the function tool set are assembled to obtain an evaluation sample, and two instructions, the problems corresponding to the two instructions, and the function tool sets are assembled to obtain an evaluation sample, so that one evaluation sample includes two instructions, thereby making it possible to evaluate the ability of the trained target model to call two function tools in parallel.

[0119] In some embodiments, the type of evaluation sample includes at least two of a single-tool type, a multi-tool type, a parallel type, and a multi-round dialogue type. The single-tool type refers to a type in which a function tool set includes only one function tool. The multi-tool type refers to a type in which a function tool set includes at least two function tools. The parallel type refers to a type in which the evaluation sample includes two instructions and two questions that trigger the generation of the two instructions. The multi-round dialogue type refers to an evaluation sample that contains multi-round dialogue information, wherein the multi-round dialogue information is used to inquire based on the parameters required to input the function tool to obtain all the required input parameters.

[0120] In an embodiment of the present application, the types of evaluation samples include at least two, which improves the diversity of evaluation samples and further improves the accuracy of the evaluation of the trained target model.

[0121] In some embodiments, the order of each type of evaluation samples in the evaluation set is disrupted.

[0122] In some embodiments, the trained target model may also be evaluated based on irrelevant evaluation samples, where irrelevant evaluation samples are evaluation samples generated based on question-and-answer data that are irrelevant to the function tool call.

[0123] In some embodiments, based on the questions, instructions and function tool set, an evaluation sample is generated, including:

[0124] Generate evaluation samples based on some questions, some instructions and function tool sets;

[0125] The order of the function tools in the function tool set is adjusted to obtain an adjusted function tool set;

[0126] Generate evaluation samples based on additional questions, additional instructions, and the adjusted function tool set.

[0127] For example, a function tool set includes function tool c1, function tool c2, and function tool c3. Function tool c1 is the first function tool in the function tool set, function tool c2 is the second function tool in the function tool set, and function tool c3 is the third function tool in the function tool set. After adjusting the order of the function tools in the function tool set, function tool c1 is the second function tool in the adjusted function tool set, function tool c3 is the first function tool in the adjusted function tool set, and function tool c2 is the third function tool in the adjusted function tool set.

[0128] In this embodiment, evaluation samples are generated based on some questions, some instructions and a function toolset; the order of the function tools in the function toolset is adjusted to obtain an adjusted function toolset; and evaluation samples are generated based on another part of the questions, another part of the instructions and the adjusted function toolset, so as to evaluate whether the trained target model has overfitting phenomenon to the order of the function tools in the function toolset.

[0129] In some embodiments, the evaluation set may also include evaluation samples in a general domain, so as to evaluate the ability of the trained target model to handle question-answering tasks in a general domain based on the evaluation samples in the general domain.

[0130] In some embodiments, before obtaining a test set including at least one test sample, the method further includes:

[0131] Obtain training samples, where the training samples include question-and-answer data and a function tool set related to the function tool, and the question-and-answer data related to the function tool includes questions and instructions for calling the function tool;

[0132] The second target model is trained based on the training samples to obtain a trained target model.

[0133] The second target model may be an untrained neural network model or a trained neural network model. When the second target model is a trained neural network model, the second target model may refer to a large language model (LLM), which can process natural language tasks, including question-answering tasks. In this case, the second target model and the first target model may be the same model or different models.

[0134] The generation process of training samples can refer to the generation process of evaluation samples, and the generation process of question and answer data in training samples can refer to the generation process of question and answer data in evaluation samples, which will not be repeated in the embodiments of the present application.

[0135] Optionally, the training samples may include training samples generated from real question-and-answer data and / or training samples generated from simulated question-and-answer data. When the training samples include training samples generated from real question-and-answer data and training samples generated from simulated question-and-answer data, the second target model can be trained based on the simulated question-and-answer data and the real question-and-answer data, further improving the ability of the trained target model to handle question-and-answer tasks.

[0136] Optionally, the training samples may include training samples generated from question-answering data in the target professional field and / or training samples generated from question-answering data in a general field. When the training samples include training samples generated from question-answering data in the target professional field and training samples generated from question-answering data in a general field, the trained target model is not only good at processing question-answering tasks in the target professional field, but also can maintain the ability to process question-answering tasks in the general field.

[0137] For the convenience of description, the problem in the training sample is called the first problem, the instruction in the training sample is called the first instruction, the problem in the evaluation sample is called the second problem, and the instruction in the evaluation sample is called the second instruction.

[0138] Specifically, the terminal can input the first problem and function tool set in the training sample into the second target model for prediction processing, obtain the first prediction instruction, determine the loss function value based on the first prediction instruction and the instruction in the training sample, and if the loss function value meets the preset loss condition, the second target model is determined as the trained target model. If the loss function value does not meet the preset loss condition, the weight of the second target model is updated according to the loss function value, and return to execute the step of inputting the first problem and function tool set in the training sample into the second target model for prediction processing. The preset loss condition is used to indicate that the loss function value converges.

[0139] It can be understood that the loss function value determined based on the first prediction instruction and the instruction in the training sample is referred to as the first loss function value. When the training sample includes an incomplete question and a supplementary question, the incomplete question is input into the second target model for prediction processing to obtain a predicted supplementary question, and the second loss function value is determined based on the predicted supplementary question and the supplementary question. When the training sample includes a parameter value and an answer, the parameter value is input into the second target model for prediction processing to obtain a predicted answer, and the third loss function value is determined based on the answer and the predicted answer. At this time, the loss function value of the second target model can be determined based on the first loss function value, the second loss function value, and the third loss function value. If the loss function value meets the preset loss condition, the second target model is determined as the trained target model. If the loss function value does not meet the preset loss condition, the weight of the second target model is updated according to the loss function value, and the step of inputting the training sample into the second target model for prediction processing is returned.

[0140] In some embodiments, before training the second target model based on the training sample to obtain the trained target model, the method further includes:

[0141] Obtain model introduction information of each candidate large language model;

[0142] According to the target professional field and model introduction information, a second target model is determined from each candidate large language model.

[0143] The model introduction information includes at least one of basic information about the model, features of the model, and a log of the model.

[0144] The terminal can determine the degree of match between each candidate large language model and the target professional field based on the model introduction information and the requirements of the target professional field, and then determine the candidate large language model with the highest match as the second target model. Alternatively, the terminal can predict the degree of match between the candidate large language model and the target professional field based on the model introduction information and the requirements of the target professional field, recommend each candidate large language model to the user according to the degree of match, and then, in response to the user's selection operation on the candidate large language model, determine the candidate large language model corresponding to the selection operation as the second target model.

[0145] Among them, the matching degree between each candidate large language model and the target professional field can be determined according to the characteristics of the model and the requirements of the target professional field. For example, the target professional field is the hotel customer service field, and the problems in this field usually exist in the form of text. At this time, the matching degree of the candidate large language model that is good at processing text-based problems is greater than the matching degree of the candidate large language model that is good at processing image-based problems. For another example, the target professional field is the legal field, and the problems in this field are usually long. At this time, the matching degree of the candidate large language model that is good at processing long problems is greater than the matching degree of the candidate large language model that is good at processing short problems.

[0146] In an embodiment of the present application, model introduction information of each candidate large language model is obtained, and a second target model is determined from each candidate large language model based on the target professional field and the model introduction information, so that when the second target model is trained, the training of the second target model can be completed more quickly, further improving the efficiency of obtaining the trained target model.

[0147] In some embodiments, before training the second target model based on the training sample to obtain the trained target model, the method further includes:

[0148] Display each candidate large language model;

[0149] In response to the selection operation on each candidate large language model, the candidate large language model corresponding to the selection operation is determined as the second target model.

[0150] In the embodiment of the present application, each candidate large language model is displayed. In response to a selection operation on each candidate large language model, the candidate large language model corresponding to the selection operation is determined as the second target model, so that the user can select the second target model by himself, thereby improving the user experience.

[0151] In some embodiments, obtaining a training sample includes:

[0152] Display the data generation interface;

[0153] In response to an input operation on the data generation interface, obtaining a name of a target professional field;

[0154] Through the first target model, training samples are generated based on the name of the target professional field.

[0155] In an embodiment of the present application, a data generation interface is displayed, and in response to an input operation on the data generation interface, the name of the target professional field is obtained. Through a first target model, training samples are generated based on the name of the target professional field, so that the user can select the target professional field according to his or her needs, thereby making the trained target model good at handling question-and-answer tasks in the professional field required by the user, thereby enabling the user to customize the model.

[0156] In some embodiments, the user can also provide function tools in the target professional field so that the trained target model can better meet the user's needs.

[0157] In some embodiments, based on the training sample, the second target model is trained to obtain a trained target model, including:

[0158] Set the adapter of the second target model;

[0159] Input the training samples into the second target model for prediction processing to obtain the loss function value;

[0160] If the loss function value does not meet the preset loss condition, the weight of the adapter is updated based on the loss function value, and the step of inputting the training sample into the second target model for prediction processing to obtain the loss function value is returned;

[0161] If the loss function value meets the preset loss condition, the weights of the adapter and the weights of the network layer in the second target model are fused to obtain a trained target model for the target professional field.

[0162] Among them, setting the adapter of the second target model may refer to adding a new network layer in parallel with at least one network layer in the second target model, and the weight of the new network layer is obtained by multiplying two low-rank matrices. The size of the weight of the new network layer is the same as the size of the weight of at least one network layer. If the size of the weight of at least one network layer is m*n, the two matrices obtained by splitting the m*n matrix are m*r and r*n, that is, the sizes of the two matrices of the new added network layer are m*r and r*n respectively. If the weight of at least one network layer is updated directly, the amount of parameters to be updated is m*n, and if the weight of the additional network layer is updated, the amount of parameters to be updated is m*r+r*n, and m*r+r*n is less than m*n.

[0163] Therefore, in an embodiment of the present application, an adapter of the second target model is set, and the training sample is input into the second target model for prediction processing to obtain a loss function value. If the loss function value does not meet the preset loss condition, the weight of the adapter is updated based on the loss function value, and the step of inputting the training sample into the second target model for prediction processing to obtain the loss function value is returned to execute. If the loss function value meets the preset loss condition, the weight of the adapter and the weight of at least one network layer in the second target model are fused to obtain a trained target model for the target professional field, thereby reducing the amount of parameters that need to be updated and further improving the training efficiency of the second target model.

[0164] For example, when the training samples include training samples generated based on real question-answering data and training samples generated based on simulated question-answering data, the training process of the second target model in this embodiment can be as follows: Figure 2 shown.

[0165] In some embodiments, the present embodiment further includes:

[0166] Get the preset update times;

[0167] Get the weight of the adapter after the preset update times;

[0168] The adapter weights are fused with the weights of the network layers in the second target model to obtain a trained target model for the target professional domain, including:

[0169] The weights updated at the preset update times are merged to obtain the weight of the adapter;

[0170] The weights of the adapter are fused with the weights of the network layers in the second target model to obtain a trained target model for the target professional field.

[0171] The preset update times may also be referred to as breakpoints, which refer to some of the update times of the adapter. Since the model training times can be set before training, the preset update times can be determined according to the training times. For example, the preset update times may be the 1 / 3, 2 / 3 and last of the training times, which is not limited in this embodiment.

[0172] For example, the number of training times is 30 times. When the loss function value meets the preset loss condition, the weight of the adapter is updated 30 times. The preset update times are the 10th, 20th and 30th times. The weight of the adapter after the 10th update, the weight after the 20th update and the weight after the 30th update are fused to obtain the weight of the adapter. Then, the weight of the adapter and the weight of at least one network layer in the second target model are fused to obtain the trained target model. At this time, the weight of the adapter fused with the weight of at least one network layer in the second target model is not the weight of the adapter after the last update, but the weight after the weight of the adapter updated at the preset number of times is fused.

[0173] It is understandable that the terminal can record the weight of the adapter after being updated for a preset number of times, and then obtain the weight of the adapter after being updated for a preset number of times when the loss function value meets the preset loss condition.

[0174] As the training time increases, the ability of the second target model to handle question-answering tasks in general fields will decrease. In this embodiment, a preset number of updates is obtained, the weight of the adapter after the preset number of updates is obtained, the weight after the preset number of updates is fused to obtain the weight of the adapter, the weight of the adapter is fused with the weight of the network layer in the second target model, and the trained target model for the target professional field is obtained, which can not only improve the ability of the trained target model to handle question-answering tasks in the target professional field, but also maintain the ability of the trained target model to handle question-answering tasks in general fields, so that the trained target model will not be overfitted to the training samples, and the generalization ability of the trained target model is improved.

[0175] 102. Input the problem and the function tool set into the trained target model for processing, and generate prediction instructions for calling the function tools in the function tool set.

[0176] Among them, after the question and the function tool set are input into the trained target model, the trained target model can predict which function tool in the function tool can get the answer to the question, and then generate a prediction instruction to call the function tool.

[0177] For example, the function tool set includes function tool c1, function tool c2, and function tool c3. The prediction is to obtain the answer to the question through function tool c2, and a prediction instruction for calling function tool c2 is generated.

[0178] 103. Based on the function tools called by the prediction instructions and the function tools called by the instructions, evaluate the ability of the trained target model to call the function tools in the function tool set to handle question-answering tasks.

[0179] The predicted instruction includes the tool name, and the instruction also includes the tool name. The function tool called by the predicted instruction refers to the function tool corresponding to the tool name in the predicted instruction, and the function tool called by the instruction refers to the function tool corresponding to the tool name in the instruction. The function tool called by the instruction is equivalent to a label, and based on the label, it is judged whether the function tool called by the predicted instruction is correct.

[0180] For the same problem, if the function tool called by the predicted instruction is the same as the function tool called by the instruction, it indicates that the target training model after training calls the function tool accurately. If the function tool called by the predicted instruction is different from the function tool called by the instruction, it indicates that the target training model after training calls the function tool incorrectly, and other function tools in the function tool set interfere with the call of this function tool.

[0181] For example, the evaluation samples include question q1 and instruction d1. Instruction d1 is an instruction for calling function tool c3. The function tool set includes function tool c1, function tool c2 and function tool c3. Question q1 and the function tool set are input into the trained target model for processing to obtain predicted instruction d11. If the predicted instruction d11 is a predicted instruction for calling function tool c3, it indicates that the call to function tool c3 is accurate. If the predicted instruction d11 is a predicted instruction for calling function tool c2, it indicates that an error occurs in the call to function tool c3 and function tool c2 interferes with the call to function tool c3.

[0182] After receiving the prediction instruction, the terminal can count the number of correct calls and the number of incorrect calls of each function tool based on the function tools called by the prediction instruction and the function tools called by the instruction. Finally, based on the statistical results of each function tool, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated.

[0183] In this embodiment, based on the function tools called by the predicted instructions and the function tools called by the instructions, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated, thereby simplifying the evaluation task of the ability of the trained target model to process question and answer tasks into a classification task of function tools. This makes it possible to obtain the calling ability of the trained target model on each function tool, refine the granularity of evaluating the ability of the trained target model to process question and answer tasks, and improve the accuracy of evaluating the ability of the trained target model to process question and answer tasks.

[0184] In some embodiments, based on the function tools called by the predicted instructions and the function tools called by the instructions, the ability of the trained target model to call the function tools in the function tool set to process the question-answering task is evaluated, including:

[0185] Determine the influence coefficient between the function tools according to the function tools called by the predicted instruction and the function tools called by the instruction, where the influence coefficient is used to indicate the degree of mutual influence between the function tools;

[0186] Based on the influence coefficient, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0187] Among them, when the function tool predicted to be called by an instruction is different from the function tool called by the instruction, it can be determined that the function tool predicted to be called by the instruction interferes with the function tool called by the instruction, thereby obtaining the degree of mutual influence between the various function tools, so that the training samples can be adjusted according to the degree of mutual influence between the various function tools, and the trained target model can continue to be fine-tuned based on the training samples. Therefore, the evaluation method of this embodiment can provide guidance for the fine-tuning of the trained target model and improve the efficiency of obtaining the target model for final use.

[0188] In some embodiments, based on the function tools called by the predicted instructions and the function tools called by the instructions, the ability of the trained target model to call the function tools in the function tool set to process the question-answering task is evaluated, including:

[0189] Constructing a confusion matrix according to the function tool predicted to be called by the instruction and the function tool called by the instruction, wherein the column data in the confusion matrix indicates the function tool called by the instruction, and the row data in the confusion matrix indicates the function tool predicted to be called by the instruction;

[0190] Based on the confusion matrix, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0191] The confusion matrix can also be called the error matrix, which is used to display the classification results of the target model for the function tool after training.

[0192] For example, the confusion matrix can be written as Figure 3As shown, from Figure 3 It can be seen that there are 11 evaluation samples for function tool c1. Among the prediction instructions generated based on the problems in the 11 evaluation samples, the function tool in 8 prediction instructions is function tool c1, indicating that 8 prediction instructions are accurate and the call to function tool c1 is accurate. The function tool in 3 prediction instructions is function tool c3, indicating that the call to function tool c1 is mistakenly identified as a call to function tool c3, indicating that an error occurs in the call to function tool c3.

[0193] In this embodiment, a confusion matrix is ​​constructed based on the function tools called by the predicted instructions and the function tools called by the instructions. The column data in the confusion matrix indicates the function tools called by the instructions, and the row data in the confusion matrix indicates the function tools called by the predicted instructions. Based on the confusion matrix, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated, so that not only can the classification results of the function tools by the trained target model be intuitively displayed, but also the trained target model can be comprehensively evaluated.

[0194] In some embodiments, based on the confusion matrix, evaluating the ability of the trained target model to call a function tool in a function tool set to process a question-answering task includes:

[0195] Based on the row data in the confusion matrix, determine the influence coefficient between the function tools, and the influence coefficient is used to indicate the degree of mutual influence between the function tools;

[0196] Based on the influence coefficient, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0197] Among them, the type of influence coefficient can be set according to actual conditions. For example, the influence coefficient can be at least one of a true positive example (True Positive, TP), a true negative example (True Negative, TN), a false positive example (False Positive, FP) and a false negative example (False Negative, FN), which is not limited in the embodiments of the present application.

[0198] True positives indicate the number of evaluation samples that predict a function tool as the function tool. True negatives indicate the number of evaluation samples that predict other function tools as other function tools. False positives indicate the number of evaluation samples that predict other function tools as the function tool, which can indicate the interference caused by the function tool to other function tools. False negatives indicate the number of evaluation samples that predict a function tool as other function tools, which can indicate the interference caused by other function tools to the function tool.

[0199] In an embodiment of the present application, based on the row data in the confusion matrix, the influence coefficient between the function tools is determined, and the influence coefficient is used to indicate the degree of mutual influence between the function tools. Based on the influence coefficient, the ability of the trained target model to call the function tools in the function tool set to process the question and answer task is evaluated, so that the training samples can be adjusted according to the degree of mutual influence between the various function tools, and the trained target model can continue to be fine-tuned based on the training samples. Therefore, the evaluation method of this embodiment can provide guidance for the fine-tuning of the trained target model and improve the efficiency of obtaining the target model for final use.

[0200] In some embodiments, based on the influence coefficient, evaluating the ability of the trained target model to call a function tool in a function tool set to process a question-answering task includes:

[0201] Based on the influence coefficient, a call capability indication parameter for each function tool is determined, and the call capability indication parameter includes at least one of accuracy, precision, recall, F1 score, specificity, and area under the ROC curve.

[0202] Based on the calling capability indication parameter for each function tool, the ability of the trained target model to call each function tool is determined.

[0203] For example, based on the influence coefficient of function tool c1, the calling capability indication parameter of function tool c1 is determined, and based on the calling capability indication parameter of function tool c1, the ability of the target model to call function tool c1 after training is determined. Based on the influence coefficient of function tool c2, the calling capability indication parameter of function tool c2 is determined, and based on the calling capability indication parameter of function tool c2, the ability of the target model to call function tool c2 after training is determined.

[0204] In an embodiment of the present application, based on the influence coefficient, a calling capability indication parameter for each function tool is determined, and the calling capability indication parameter includes at least one of accuracy, precision, recall, F1 score, specificity, and the area under the ROC curve. Based on the calling capability indication parameter for each function tool, the ability of the trained target model to call the function tool is determined, thereby obtaining the performance ability of the trained target model on each function tool.

[0205] In some embodiments, based on the influence coefficient, evaluating the ability of the trained target model to call a function tool in a function tool set to process a question-answering task includes:

[0206] Based on the influence coefficient, determine the overall evaluation coefficient of the trained target model for each function tool;

[0207] Based on the overall evaluation coefficient, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0208] Among them, based on the influence coefficient, the process of determining the overall evaluation coefficient of the trained target model for each function tool can be:

[0209] Based on the influence coefficient, determining a call capability indication parameter for each function tool, where the call capability indication parameter includes at least one of a precision rate and a recall rate;

[0210] The call capability indication parameters are weighted to obtain the overall evaluation coefficient of the trained target model for each function tool, or the call capability indication parameters are averaged to obtain the overall evaluation coefficient of the trained target model for each function tool.

[0211] For example, when the call capability indication parameter is precision, the overall evaluation coefficient may be weighted precision or average precision; when the call capability indication parameter is recall, the overall evaluation coefficient may be weighted recall.

[0212] Among them, the weighted precision can be calculated by the following formula:

[0213]

[0214] p_i represents the precision of the i-th function tool, Number_i represents the number of evaluation samples related to the i-th function tool, and Total represents the number of all evaluation samples in the evaluation set.

[0215] The average precision can be calculated by the following formula:

[0216]

[0217] p_i represents the precision of the i-th function tool, and N represents the number of function tools in the function tool set.

[0218] In an embodiment of the present application, based on the influence coefficient, the overall evaluation coefficient of the trained target model for each function tool is determined, and based on the overall evaluation coefficient, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated, thereby evaluating the trained target model based on the overall evaluation coefficient, thereby further improving the accuracy of the evaluation of the trained target model.

[0219] In some embodiments, after obtaining the prediction instruction, the method further includes:

[0220] Based on the target dimension, the correctness of the prediction instruction is analyzed to obtain an analysis result, where the target dimension includes at least one of a structure of the prediction instruction, a parameter in the prediction instruction, and an applicable scenario of the prediction instruction;

[0221] Based on the function tools called by the predicted instructions and the function tools called by the instructions, evaluate the ability of the trained target model to call the function tools in the function toolset to handle question-answering tasks, including:

[0222] Based on the function tools called by the predicted instructions, the function tools called by the instructions, and the analysis results, the ability of the trained target model to call the function tools in the function toolset to handle question-answering tasks is evaluated.

[0223] When the target dimension is the structure of the prediction instruction, the correctness of the prediction instruction is analyzed according to the structure of the prediction instruction, and the process of obtaining the analysis result can be:

[0224] Determine whether the forecast instruction includes the tool name and parameters;

[0225] If the prediction instruction includes the tool name and parameters, it means that the structure of the prediction instruction is correct and the analysis result is determined to be the correct result.

[0226] When the target dimension is a parameter in the prediction instruction, the correctness of the prediction instruction is analyzed according to the parameter of the prediction instruction, and the process of obtaining the analysis result can be:

[0227] Whether the parameters in the predicted instruction are missing, whether there are unnecessary parameters, whether the parameter types are wrong, and whether the parameters match are analyzed to obtain analysis results.

[0228] When the target dimension is the applicable scenario of the prediction instruction, the correctness of the prediction instruction is analyzed according to the applicable scenario of the prediction instruction, and the process of obtaining the analysis result can be:

[0229] Determine applicable scenarios of the predicted instructions, where the applicable scenarios indicate whether a function tool needs to be called or not;

[0230] According to the applicable scenarios of the predicted instructions, the correctness of the predicted instructions is analyzed to obtain analysis results.

[0231] Among them, when the applicable scenario of the predicted instruction is that a function tool needs to be called, the function tool that does not need to be called may be mistakenly identified as a function tool that needs to be called. When the applicable scenario of the predicted instruction is that the function tool does not need to be called, the function tool that needs to be called may be mistakenly identified as a function tool that does not need to be called.

[0232] In the embodiment of the present application, based on the target dimension, the correctness of the prediction instruction is analyzed to obtain the analysis result. The target dimension includes the structure of the prediction instruction, the parameters in the prediction instruction, and at least one of the applicable scenarios of the prediction instruction. Based on the function tools called according to the prediction instruction, the function tools called by the instruction, and the analysis results, the ability of the trained target model to call the function tools in the function tool set to process the question-answering task is evaluated. The trained target model is evaluated from multiple dimensions, and the accuracy of the evaluation of the trained target model is further evaluated.

[0233] In some embodiments, when the question-and-answer data includes real question-and-answer data and simulated question-and-answer data, the terminal can generate evaluation samples based on the real question-and-answer data and the simulated question-and-answer data, determine the number of correct prediction instructions obtained by the trained target model and the number of correct prediction instructions obtained by the second target model based on the evaluation samples, and obtain the accuracy of the trained target model and the accuracy of the second target model based on the number of correct prediction instructions. The accuracy of the trained target model is used to indicate the ability of the trained target model to process question-and-answer tasks, and the accuracy of the second target model is used to indicate the ability of the second target model to process question-and-answer tasks, thereby comparing the ability of the trained target model to process question-and-answer tasks with the ability of the second target model to process question-and-answer tasks, for example. Figure 4 shown.

[0234] From the above content, it can be seen that in an embodiment of the present application, an evaluation set including at least one evaluation sample is obtained, the evaluation sample includes question and answer data related to the function tools in the function toolset and the function toolset, the question and answer data including questions and instructions for calling the function tools based on the questions; the questions and the function toolset are input into the trained target model for processing, and a prediction instruction for calling the function tools in the function toolset is generated; based on the function tools called by the prediction instructions and the function tools called by the instructions, the ability of the trained target model to call the function tools in the function toolset to process question and answer tasks is evaluated, so as to simplify the evaluation task of the ability of the trained target model to process question and answer tasks into the classification task of function tools, so that the performance ability of the trained target model on each function tool can be obtained, the granularity of evaluating the ability of the trained target model to process question and answer tasks is refined, and the accuracy of evaluating the ability of the trained target model to process question and answer tasks is improved.

[0235] In order to better implement the above method, an embodiment of the present application also provides a model evaluation device, which can be specifically integrated into an electronic device, such as a computer device, which can be a terminal, a server, or other device.

[0236] Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop computer, a personal computer and other devices; the server can be a single server or a server cluster composed of multiple servers.

[0237] For example, in this embodiment, the method of the embodiment of the present application is described in detail by taking the model evaluation device specifically integrated in the terminal as an example. This embodiment provides a model evaluation device, such as Figure 5 As shown, the model evaluation device may include:

[0238] The acquisition module 501 is used to acquire an evaluation set including at least one evaluation sample, wherein the evaluation sample includes question and answer data related to a function tool in a function tool set and the function tool set, wherein the question and answer data includes questions and instructions for calling the function tool based on the questions.

[0239] The processing module 502 is used to input the problem and the function tool set into the trained target model for processing, and generate a prediction instruction for calling the function tools in the function tool set.

[0240] The evaluation module 503 is used to evaluate the ability of the trained target model to process question-answering tasks by calling the function tools in the function tool set based on the function tools called by the predicted instructions and the function tools called by the instructions.

[0241] In some embodiments, the evaluation module 503 is specifically configured to perform:

[0242] Constructing a confusion matrix according to the function tool predicted to be called by the instruction and the function tool called by the instruction, wherein the column data in the confusion matrix indicates the function tool called by the instruction, and the row data in the confusion matrix indicates the function tool predicted to be called by the instruction;

[0243] Based on the confusion matrix, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0244] In some embodiments, the evaluation module 503 is specifically configured to perform:

[0245] Based on the row data in the confusion matrix, determine the influence coefficient between the function tools, and the influence coefficient is used to indicate the degree of mutual influence between the function tools;

[0246] Based on the influence coefficient, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0247] In some embodiments, the evaluation module 503 is specifically configured to perform:

[0248] Based on the influence coefficient, determine the overall evaluation coefficient of the trained target model for each function tool;

[0249] Based on the overall evaluation coefficient, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0250] In some embodiments, the evaluation module 503 is further configured to perform:

[0251] Based on the target dimension, the correctness of the prediction instruction is analyzed to obtain an analysis result, where the target dimension includes at least one of a structure of the prediction instruction, a parameter in the prediction instruction, and an applicable scenario of the prediction instruction;

[0252] Based on the function tools called by the predicted instructions, the function tools called by the instructions, and the analysis results, the ability of the trained target model to call the function tools in the function toolset to handle question-answering tasks is evaluated.

[0253] In some embodiments, the acquisition module 501 is specifically used to execute:

[0254] Obtain a function tool set, which includes a function tool, a functional description of the function tool, and a parameter description of the parameters required to be input when calling the function tool;

[0255] Generate questions related to the function tool and instructions for calling the function tool according to the function description and the parameter description through the first target model;

[0256] Generate evaluation samples based on questions, instructions and function tool sets, and generate evaluation sets based on the evaluation samples.

[0257] In some embodiments, the acquisition module 501 is specifically used to execute:

[0258] Generate instructions for calling a function tool according to the function description and the parameter description through the first target model;

[0259] Through the first target model, the questions that need to be input to trigger the generation of instructions are predicted.

[0260] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation methods and corresponding beneficial effects of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0261] Accordingly, an embodiment of the present application further provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), or other terminal devices. Figure 6 As shown, Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 600 includes a processor 601 having one or more processing cores, a memory 602 having one or more computer-readable storage media, and a computer program stored in the memory 602 and executable on the processor. The processor 601 is electrically connected to the memory 602. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0262] The processor 601 is the control center of the electronic device 600. It uses various interfaces and lines to connect various parts of the entire electronic device 600, executes various functions of the electronic device 600 and processes data by running or loading software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, thereby monitoring the electronic device 600 as a whole.

[0263] In the embodiment of the present application, the processor 601 in the electronic device 600 will load instructions corresponding to the processes of one or more application programs into the memory 602 according to the following steps, and the processor 601 will run the application programs stored in the memory 602 to implement various functions, such as:

[0264] Obtaining an evaluation set including at least one evaluation sample, the evaluation sample including question-and-answer data related to a function tool in a function tool set and the function tool set, the question-and-answer data including questions and instructions for calling the function tool based on the questions;

[0265] Input the problem and the function tool set into the trained target model for processing, and generate prediction instructions that call the function tools in the function tool set;

[0266] Based on the function tools called by the prediction instructions and the function tools called by the instructions, the ability of the trained target model to call the function tools in the function toolset to handle question-answering tasks is evaluated.

[0267] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the model evaluation method above, which will not be elaborated here.

[0268] Optional, such as Figure 6 As shown, the electronic device 600 further includes: a touch screen 603, a radio frequency circuit 604, an audio circuit 605, an input unit 606, and a power supply 607. The processor 601 is electrically connected to the touch screen 603, the radio frequency circuit 604, the audio circuit 605, the input unit 606, and the power supply 607, respectively. Those skilled in the art can understand that Figure 6The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0269] The touch display screen 603 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 603 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 601, and can receive the command sent by the processor 601 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 601 to determine the type of touch event, and then the processor 601 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 603 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 603 can also be used as a part of the input unit 606 to realize the input function.

[0270] The radio frequency circuit 604 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals between the network device or other electronic devices.

[0271] The audio circuit 605 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 605 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 605 and converted into audio data, and then the audio data is output to the processor 601 for processing, and then sent to another electronic device through the radio frequency circuit 604, or the audio data is output to the memory 602 for further processing. The audio circuit 605 may also include an earplug jack to provide communication between an external headset and an electronic device.

[0272] The input unit 606 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0273] The power supply 607 is used to supply power to various components of the electronic device 600. Optionally, the power supply 607 can be logically connected to the processor 601 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 607 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0274] although Figure 6 Not shown, the electronic device 600 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0275] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0276] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0277] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored, and the computer program can be loaded by a processor to execute any model evaluation method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0278] Obtaining an evaluation set including at least one evaluation sample, the evaluation sample including question-and-answer data related to a function tool in a function tool set and the function tool set, the question-and-answer data including questions and instructions for calling the function tool based on the questions;

[0279] Input the problem and the function tool set into the trained target model for processing, and generate prediction instructions that call the function tools in the function tool set;

[0280] Based on the function tools called by the prediction instructions and the function tools called by the instructions, the ability of the trained target model to call the function tools in the function toolset to handle question-answering tasks is evaluated.

[0281] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the model evaluation method above, which will not be elaborated here.

[0282] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0283] Since the computer program stored in the computer-readable storage medium can execute any one of the model evaluation methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the model evaluation methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0284] The above is a detailed introduction to a model evaluation method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model evaluation method, characterized in that: The method comprises: Acquire an evaluation set including at least one evaluation sample, wherein the evaluation sample includes question-and-answer data related to a function tool in a function tool set and the function tool set, wherein the question-and-answer data includes a question and an instruction for calling the function tool based on the question; Inputting the problem and the function tool set into the trained target model for processing, and generating a prediction instruction for calling the function tool in the function tool set; Based on the function tools called by the prediction instructions and the function tools called by the instructions, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated.

2. The model evaluation method according to claim 1, characterized in that: The step of evaluating the ability of the trained target model to process the question-answering task by calling the function tool in the function tool set according to the function tool called by the prediction instruction and the function tool called by the instruction includes: Constructing a confusion matrix according to the function tool called by the predicted instruction and the function tool called by the instruction, wherein the column data in the confusion matrix indicates the function tool called by the instruction, and the row data in the confusion matrix indicates the function tool called by the predicted instruction; Based on the confusion matrix, the ability of the trained target model to call the function tools in the function tool set to process the question-answering task is evaluated.

3. The model evaluation method according to claim 2, characterized in that: The step of evaluating the ability of the trained target model to call the function tool in the function tool set to process the question-answering task based on the confusion matrix includes: Determine the influence coefficients between the function tools based on the row data in the confusion matrix, wherein the influence coefficients are used to indicate the degree of mutual influence between the function tools; Based on the influence coefficient, the ability of the trained target model to call the function tool in the function tool set to process the question-answering task is evaluated.

4. The model evaluation method according to claim 3, characterized in that: The step of evaluating, based on the influence coefficient, the ability of the trained target model to call the function tool in the function tool set to process the question-answering task includes: Based on the influence coefficient, determining the overall evaluation coefficient of the trained target model for each of the function tools; Based on the overall evaluation coefficient, the ability of the trained target model to call the function tools in the function tool set to process the question-answering task is evaluated.

5. The model evaluation method according to claim 1, characterized in that: After obtaining the prediction instruction, the method further includes: Analyzing the correctness of the prediction instruction based on a target dimension to obtain an analysis result, wherein the target dimension includes at least one of a structure of the prediction instruction, a parameter in the prediction instruction, and an applicable scenario of the prediction instruction; The step of evaluating the ability of the trained target model to process the question-answering task by calling the function tool in the function tool set according to the function tool called by the prediction instruction and the function tool called by the instruction includes: Based on the function tools called by the prediction instructions, the function tools called by the instructions, and the analysis results, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated.

6. The model evaluation method according to any one of claims 1 to 5, characterized in that: The obtaining of the evaluation set including at least one evaluation sample comprises: Acquire a function tool set, the function tool set including a function tool, a functional description of the function tool, and a parameter description of a parameter required to be input when calling the function tool; Generate, through the first target model, questions related to the function tool and instructions for calling the function tool according to the function description and the parameter description; Based on the problem, the instruction and the function tool set, the evaluation sample is generated, and based on the evaluation sample, an evaluation set is generated.

7. The model evaluation method according to claim 6, characterized in that: The step of generating, through the first target model, questions related to the function tool and instructions for calling the function tool according to the function description and the parameter description includes: Generate, through the first target model, an instruction for calling the function tool according to the function description and the parameter description; Through the first target model, the questions that need to be input to trigger the generation of the instructions are predicted.

8. A model evaluation device, characterized in that: The device comprises: An acquisition module, configured to acquire an evaluation set including at least one evaluation sample, wherein the evaluation sample includes question-and-answer data related to a function tool in a function tool set and the function tool set, wherein the question-and-answer data includes a question and an instruction for calling the function tool based on the question; A processing module, used for inputting the problem and the function tool set into the trained target model for processing, and generating a prediction instruction for calling the function tool in the function tool set; An evaluation module is used to evaluate the ability of the trained target model to call the function tools in the function tool set to process question-answering tasks based on the function tools called by the prediction instructions and the function tools called by the instructions.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the model evaluation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the model evaluation method according to any one of claims 1 to 7.

Images