Artificial intelligence model evaluation method and device and storage medium

Abstract

The invention discloses an artificial intelligence model evaluation method and device and a storage medium, and the method comprises the steps: obtaining to-be-evaluated questions, the to-be-evaluated questions are selected from a target question bank, and the target question bank is formed by a plurality of questions passing the question quality inspection based on evaluation framework information and standard information; inputting a to-be-evaluated question into the evaluation model to obtain a target SQL and a target DSL; performing database query on the target SQL and a standard SQL associated with the to-be-evaluated question to obtain a target answer and a standard answer; and determining an evaluation result and a question positioning result based on the target answer, the standard answer, the target DSL and the standard DSL. According to the method, the questions are generated and subjected to quality inspection based on the evaluation framework information and the standard information, the high-quality evaluation questions are automatically constructed, the evaluation efficiency is improved, the evaluation result and the problem positioning result are determined based on the target answer, the standard answer, the target DSL and the standard DSL associated with the to-be-evaluated question, and reasons of wrong answers can be automatically recognized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence model evaluation method, device and storage medium. Background Art

[0002] NL2SQL (Natural Language to Structured Query Language) is a technology that converts natural language (NL) into structured query language (SQL). This technology can convert natural language query statements entered by users into SQL statements that can be executed in the database, thereby retrieving the required data from the database. Currently, the evaluation of the accuracy of the NL2SQL system mainly includes the following two aspects:

[0003] 1: Execution accuracy: For each natural language query question, the NL2SQL system is used to generate the corresponding SQL statement; the generated SQL statement is executed and the query results are recorded. The query results are compared with the manual standard results. If the two are the same, the generated SQL is considered correct; the proportion of correct SQL generated in all questions is calculated, which is the execution accuracy, and the performance of the system is evaluated by the execution accuracy. 2: Matching accuracy: For each natural language query question, the NL2SQL system is used to generate the corresponding SQL statement; the generated SQL statement is compared with the manually written standard SQL statement. If the two are the same in syntax and semantics, the generated SQL is considered to be matched; the proportion of matching SQL generated for all questions is calculated, which is the matching accuracy, and the performance of the system is evaluated by the matching accuracy. However, the comparison process of these two aspects requires high labor costs. It is also impossible to analyze the cause of the execution error and provide closed-loop feedback for the optimization of the NL2SQL algorithm. Summary of the invention

[0004] Based on this, it is necessary to provide an artificial intelligence model evaluation method, device and storage medium to address the above-mentioned technical problems, so as to solve at least one of the above-mentioned technical problems.

[0005] The present invention provides an artificial intelligence model evaluation method, comprising:

[0006] Obtaining the test questions for the current business scenario, wherein the test questions are selected from a target question bank, and the target question bank is formed by designing questions based on the evaluation framework information and specification information preset for the business scenario and by a number of questions that have passed quality inspection;

[0007] Input the to-be-evaluated topic into the evaluation model, and obtain the target SQL and target DSL output by the evaluation model;

[0008] Performing database queries on the target SQL and the standard SQL associated with the test question respectively to obtain a target answer corresponding to the target SQL and a standard answer to the standard SQL;

[0009] Based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated, an evaluation result and a problem location result are determined.

[0010] Optionally, according to an artificial intelligence model evaluation method provided by the present invention, the target question bank is formed based on the following steps:

[0011] Obtain a set of questions generated by a preset question-generating service software for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the set of questions;

[0012] According to the preset evaluation framework information and specification information, use the preset quality inspection service software to perform quality inspection on each of the above mentioned topics and its standard DSL and standard SQL;

[0013] If the topic and its corresponding standard DSL or standard SQL fail the quality inspection, then delete the topic that fails the quality inspection from the topic set;

[0014] Determine the similarity between the questions that have passed the quality inspection, so as to delete the questions whose similarity exceeds the preset similarity threshold;

[0015] Return to the step of obtaining a set of questions generated by the preset question-setting service software for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the set of questions, until the number of all questions that pass the quality inspection reaches a preset number threshold, and the target question bank is obtained.

[0016] Optionally, according to an artificial intelligence model evaluation method provided by the present invention, the step of determining the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated includes:

[0017] Compare the target answer with the standard answer to obtain a first result;

[0018] If the first result is that the answer is wrong, the dimensions, indicators and operators in the target DSL are compared with the dimensions, indicators and operators in the standard DSL respectively, so as to determine the problem location result according to the second result, wherein the dimension refers to a feature or attribute of the data, the indicator is used to identify the quantity of measuring or evaluating a specific dimension or data set, and the operator refers to a function or method for performing data operations or calculations;

[0019] The number of first questions whose first result is a correct answer and the number of second questions whose first result is an incorrect answer are counted, so as to determine the evaluation result based on the first number of questions and the second number of questions.

[0020] Optionally, according to an artificial intelligence model evaluation method provided by the present invention, determining the problem locating result according to the second result includes:

[0021] If the second result is that the comparison of the dimension, the index and the operator is successful, it is determined that the problem location result is a backend logic error;

[0022] If the second result is that the dimension comparison fails, the corresponding dimension error information is used as the problem location result;

[0023] If the second result is that the indicator comparison fails, the corresponding indicator error information is used as the problem location result;

[0024] If the second result is that there is an operator comparison failure, the corresponding operator error information is used as the problem locating result.

[0025] Optionally, according to an artificial intelligence model evaluation method provided by the present invention, after determining the evaluation result based on the first number of questions and the second number of questions, the method further includes:

[0026] Based on the evaluation results of the evaluation model in all business scenarios and the weight associated with each business scenario, the total score of the evaluation model is determined.

[0027] Optionally, according to an artificial intelligence model evaluation method provided by the present invention, after determining the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated, the method further includes:

[0028] For any business scenario:

[0029] Classify the test questions corresponding to the wrong answers according to the backend logic errors, the dimension error information, the indicator error information, and the operator error information;

[0030] Counting the number of questions of each type of test questions identified by the evaluation model in two evaluations;

[0031] Based on the number of questions, the proportion of each type of questions to be evaluated is determined.

[0032] Based on the proportion of each type of questions to be evaluated in the two evaluations, determine the changes in the proportion of each type of questions to be evaluated.

[0033] Optionally, according to an artificial intelligence model evaluation method provided by the present invention, the evaluation framework information includes business problems, problem dimensions and problem indicators of different business scenarios and question template information; the specification information includes DSL standard specifications corresponding to business problems of different business scenarios and SQL query metadata specifications for different business scenarios.

[0034] Optionally, according to an artificial intelligence model evaluation method provided by the present invention, after quality inspection of each of the questions and its standard DSL and standard SQL according to the preset evaluation framework information and specification information, the method further includes:

[0035] The questions that fail the quality inspection are fed back to the question setting service software to perform a small number of sample training on the question setting service software.

[0036] The present invention also provides an artificial intelligence model evaluation device, comprising:

[0037] An acquisition module is used to acquire the test questions for the current business scenario, wherein the test questions are selected from a target question bank, and the target question bank is formed by designing questions based on the evaluation framework information and specification information preset for the business scenario and passing a number of questions that have passed quality inspection;

[0038] An input module, used to input the to-be-evaluated topic into the evaluation model, and obtain the target SQL and target DSL output by the evaluation model;

[0039] A query module, used to perform database query on the target SQL and the standard SQL associated with the test question, respectively, to obtain a target answer corresponding to the target SQL and a standard answer to the standard SQL;

[0040] The determination module is used to determine the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated.

[0041] The present invention also provides a computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned artificial intelligence model evaluation method when executing the computer-readable instructions.

[0042] The present invention also provides one or more readable storage media storing computer-readable instructions, which implement the above-mentioned artificial intelligence model evaluation method when executed by a processor.

[0043] The above-mentioned artificial intelligence model evaluation method, device and storage medium include: obtaining the to-be-evaluated questions of the current business scenario, wherein the to-be-evaluated questions are selected from the target question bank, and the target question bank is formed by designing questions based on the evaluation framework information and specification information preset in the business scenario and passing the quality inspection of several questions; inputting the to-be-evaluated questions into the evaluation model to obtain the target SQL and target DSL output by the evaluation model; respectively querying the target SQL and the standard SQL associated with the to-be-evaluated questions in the database to obtain the target answer corresponding to the target SQL and the standard answer of the standard SQL; determining the evaluation results and the problem location results based on the target answer, the standard answer, the target DSL and the standard DSL associated with the to-be-evaluated questions. The present invention generates questions and performs quality inspection based on the evaluation framework information and specification information, realizes the automatic construction of high-quality evaluation questions, improves the efficiency of the evaluation, and then determines the evaluation results and the problem location results based on the target answer, the standard answer, the target DSL and the standard DSL associated with the to-be-evaluated questions, and can automatically identify the evaluation results of the model and locate the reasons for the wrong answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0045] Figure 1 It is a flow chart of an artificial intelligence model evaluation method in one embodiment of the present invention;

[0046] Figure 2 is a structural schematic diagram of an artificial intelligence model evaluation device in one embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The terms used in one or more embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present invention. The singular forms of "a", "said" and "the" used in one or more embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and includes any or all possible combinations of one or more associated listed items.

[0050] In one embodiment, specifically, Figure 1 As shown, Figure 1 1 is a flow chart of an artificial intelligence model evaluation method in an embodiment of the present invention. The present invention provides an artificial intelligence model evaluation method, comprising the following steps:

[0051] Step S11, obtaining the topic to be evaluated for the current business scenario;

[0052] It should be noted that the questions to be evaluated are selected from a target question bank, which is formed by designing questions based on evaluation framework information and specification information preset in business scenarios and by a number of questions that have passed quality inspection.

[0053] Optionally, the evaluation framework information includes information such as business problems, problem dimensions and problem indicators of different business scenarios, and question asking template information; business problems of different business scenarios refer to the specific types of business problems that users will ask. The proportion of the number of questions asked by users in each business scenario will be different, and there will be a roughly fixed distribution over a period of time. Problem dimensions and problem indicators refer to the detailed dimensions and indicators of the questions asked by users, which are summarized based on real business problems, and all enumerated values ​​of dimensions and indicators of different business scenarios need to be given. Question asking template information is a template that specifies how users will ask questions. The question design should take into account the actual spoken expression of users, provide different questioning templates for different business scenarios, and provide question examples.

[0054] The specification information includes DSL standard specifications corresponding to business problems in different business scenarios and SQL query metadata specifications for different business scenarios. DSL is a "Domain-Specific Language", which is a programming language or scripting language designed for a specific domain or task. SQL (Structured Query Language) is a standard programming language for managing and operating relational database management systems. SQL is used to perform various operations, such as querying, updating, inserting and deleting data in a database.

[0055] It should be noted that the DSL modes corresponding to the problem types in different business scenarios will be different, and different DSL standard specifications should be formulated for different modes; at the same time, SQL query metadata specifications such as the fields to be queried by SQL and the corresponding database tables are generated. For example, analyze the database architecture, including tables, views, indexes and constraints; determine the data model and data flow; determine the fields to be queried for each business scenario; map the fields to specific database tables; define SQL query specifications, including selection, projection, connection and conditions; ensure that the query complies with data security and privacy policies; define SQL specifications such as query permissions and access levels.

[0056] Specifically, for the current business scenario, all or part of the questions are extracted from the target question bank as the test questions for the current business scenario. Optionally, the question volume data for each business scenario should be determined according to the actual distribution of questions asked by users to ensure that the test questions are aligned with the actual user scenarios.

[0057] Step S12, inputting the to-be-assessed topic into the evaluation model, and obtaining the target SQL and target DSL output by the evaluation model;

[0058] Specifically, the evaluation model of the current question is first determined, and then the question to be evaluated is input into the evaluation model to obtain the target SQL and target DSL output by the evaluation model. The evaluation model is the NL2SQL (Natural Language to SQL) model, which can convert natural language queries into structured query language (SQL). Non-technical users can describe the information they want to obtain from the database in natural language without writing complex SQL statements.

[0059] Step S13, respectively querying the target SQL and the standard SQL associated with the test question in the database to obtain a target answer corresponding to the target SQL and a standard answer to the standard SQL;

[0060] Specifically, based on the target SQL, data query is performed in the database to obtain the target answer corresponding to the target SQL, and based on the standard SQL associated with the question to be evaluated, data query is performed in the database to obtain the standard answer of the standard SQL.

[0061] Step S14, determining an evaluation result and a problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated.

[0062] Specifically, the target answer is compared with the standard answer, and whether the returned target answer is correct is determined based on the comparison result. The returned result includes a correct answer and an incorrect answer. In other embodiments, the returned result may also include an irrelevant answer. A correct answer means that the answer given by the model has no factual errors; an incorrect answer means that the answer given by the model has factual errors; an irrelevant answer means that the answer given by the model has no factual errors but the answer does not match the question. An irrelevant answer can be classified as an incorrect answer. If the returned result is an incorrect answer, the number of questions with incorrect answers can be counted, and the number of questions with correct answers can be counted, so as to determine the evaluation result of the evaluation model based on the number of questions with incorrect answers and the number of questions with correct answers. The specific evaluation process is specifically described in the following embodiments and will not be repeated here.

[0063] In addition, for questions with incorrect answers, the next step of locating the cause can be performed, and the dimensions, indicators and operators in the target DSL are respectively compared with the dimensions, indicators and operators in the standard DSL to determine the problem locating result according to the second result. The specific locating process is described in detail in the following embodiment and will not be repeated here. Among them, dimension refers to a feature or attribute of data, indicator is used to identify the quantity of measuring or evaluating a specific dimension or data set, and operator refers to a function or method for performing data operation or calculation.

[0064] The embodiment of the present invention, through the above scheme, includes: obtaining the test questions of the current business scenario, wherein the test questions are selected from the target question bank, and the target question bank is formed by designing questions based on the test framework information and specification information preset in the business scenario and passing the quality inspection; inputting the test questions into the evaluation model to obtain the target SQL and target DSL output by the evaluation model; performing database query on the target SQL and the standard SQL associated with the test questions respectively to obtain the target answer corresponding to the target SQL and the standard answer of the standard SQL; determining the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the test questions. The embodiment of the present invention generates questions and performs quality inspection based on the evaluation framework information and specification information, realizes the automatic construction of high-quality test questions, improves the efficiency of the evaluation, and then determines the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the test questions to be evaluated, and can automatically identify the evaluation result of the model and locate the cause of the wrong answer.

[0065] In one embodiment of the present invention, the target question bank is formed based on the following steps:

[0066] Obtain a set of questions generated by a preset question-generating service software for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the set of questions;

[0067] According to the preset evaluation framework information and specification information, use the preset quality inspection service software to perform quality inspection on each of the above mentioned topics and its standard DSL and standard SQL;

[0068] If the topic and its corresponding standard DSL or standard SQL fail the quality inspection, then delete the topic that fails the quality inspection from the topic set;

[0069] Determine the similarity between the questions that have passed the quality inspection, so as to delete the questions whose similarity exceeds the preset similarity threshold;

[0070] Return to the step of obtaining a set of questions generated by the preset question-setting service software for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the set of questions, until the number of all questions that pass the quality inspection reaches a preset number threshold, and the target question bank is obtained.

[0071] It should be noted that in this embodiment, a question-setting service software Agent and a quality inspection service software Agent are provided, which can generate a specified number of questions for different business scenarios and perform quality inspection on the generated questions. Optionally, the question-setting service software Agent and the quality inspection service software Agent are developed based on a large language model, wherein Agent generally refers to an entity that performs a specific task in a computer system or network. It can be a software program, a hardware device, or a person. The role of the Agent is to complete tasks on behalf of a user or other system, such as data collection, data processing, decision support, etc. The Agent can be autonomous and have a certain degree of intelligence and adaptability to perform tasks in different scenarios. In one embodiment, for different business scenarios, a number of instruction data are manually designed, and these instructions are added to the Agent's Prompt as a few-shot sample for training and learning. Each instruction data includes three parts: instruction, question, and output, wherein the output includes the standard DSL and standard SQL of each question. For example: Instruction: The following question belongs to business scenario X, and the standard DSL and standard SQL of the question are given. Question: Question Q. Standard DSL: DSL corresponding to question Q. Standard SQL: SQL corresponding to question Q.

[0072] Specifically, obtain a set of questions generated by the preset question-generating service software for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the question set, and use the preset quality inspection service software to perform quality inspection on each of the questions and their standard DSL and standard SQL to determine whether each of the questions and their standard DSL and standard SQL meets the requirements of the evaluation framework information and specification information. If the question and its corresponding standard DSL or standard SQL fail the quality inspection, that is, one of the question, DSL or SQL is incorrect, then the question is determined to fail the quality inspection, and then the questions in the question set that fail the quality inspection are fed back to the question-generating service software as Badcases, where Badcases refer to cases that perform poorly during training or evaluation, so as to perform a few-sample training on the question-generating service software. And delete the questions in the question set that fail the quality inspection. In addition, the similarity between each question that has passed the quality inspection is calculated, and then the questions in the question set whose similarity exceeds the preset similarity threshold are deleted; return to execute the step of obtaining the question set generated by the preset question-setting service software for any business scenario, and the standard DSL and standard SQL corresponding to each question in the question set, until the number of all questions that have passed the quality inspection reaches the preset number threshold, and the target question bank is obtained, wherein the preset similarity threshold and the preset number threshold are set according to actual conditions and are not specifically limited here.

[0073] Through the above scheme, the embodiment of the present invention realizes the use of question service software to generate a question set for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the question set, realizes the automatic construction of evaluation questions, and uses preset quality inspection service software to perform quality inspection on each of the questions and their standard DSL and standard SQL, thereby improving the quality of the evaluation questions and effectively improving the efficiency of model evaluation.

[0074] In one embodiment of the present invention, based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated, determining the evaluation result and the problem location result includes:

[0075] Compare the target answer with the standard answer to obtain a first result;

[0076] If the first result is that the answer is wrong, then the dimensions, indicators and operators in the target DSL are compared with the dimensions, indicators and operators in the standard DSL respectively, so as to determine the problem location result according to the second result;

[0077] The number of first questions whose first result is a correct answer and the number of second questions whose first result is an incorrect answer are counted, so as to determine the evaluation result based on the first number of questions and the second number of questions.

[0078] It should be noted that dimension refers to a feature or attribute of the data, an indicator is used to identify the quantity of measuring or evaluating a specific dimension or data set, and an operator refers to a function or method for performing data operations or calculations.

[0079] Specifically, the target answer is compared with the standard answer to obtain a first result. If the first result is that the answer is wrong, it is necessary to further locate the cause of the problem, and then the dimensions, indicators and operators in the target DSL are respectively compared with the dimensions, indicators and operators in the standard DSL to obtain a second result. If the second result is that the dimension comparison fails, the corresponding dimension error information is used as the problem location result; if the second result is that there is an indicator comparison failure, the corresponding indicator error information is used as the problem location result; if the second result is that there is an operator comparison failure, the corresponding operator error information is used as the problem location result, and the operator error information includes information such as the number of operators, operator names, operator parameters and operator dependencies. If the second result is that the comparison of dimensions, indicators and operators is successful, it is determined that the problem location result is a backend logic error.

[0080] In a specific example: if all dimensions in the target DSL and the standard DSL are consistent, the dimension comparison result is marked as "correct"; otherwise, the dimension comparison result is marked as "wrong" and the specific dimension name of the error is output.

[0081] If all indicators in the target DSL and the standard DSL are consistent, the indicator comparison result is marked as "correct"; otherwise, the indicator comparison result is marked as "wrong" and the specific indicator name of the error is output.

[0082] If all operators in the target DSL and the standard DSL are consistent (the comparison content includes the number of operators, operator names, operator parameters, and operator dependencies), the operator comparison result is marked as "correct"; otherwise, the operator comparison result is marked as "wrong" and the specific comparison content of the error is output.

[0083] If the target DSL and the standard DSL are exactly the same, it will be marked as "backend logic error". The backend logic includes the implementation logic of converting DSL to SQL, data processing after querying data using SQL, and display processing.

[0084] In addition, for the evaluation and scoring of the evaluation model: the first result is the number of first questions with correct answers, the first result is the number of second questions with incorrect answers, and the total number of questions is determined, and then based on the first number of questions, the second number of questions and the total number of questions, the evaluation result of the evaluation model in the current business scenario is calculated. The calculation formula is as follows:

[0085]

[0086] When the answer is correct, S k =1, when the answer is wrong, S k =0, q represents the qth evaluation model, j represents the jth business scenario, K j Indicates the total number of questions for the jth business scenario.

[0087] In addition, the total score of the evaluation model can be determined based on the evaluation results of the evaluation model in all business scenarios and the weight associated with each business scenario. The calculation formula is as follows:

[0088]

[0089] Where J represents the number of business scenarios, w j Represents the weight of the jth business scenario, and the sum of the weights of all business scenarios is equal to 1.

[0090] The embodiment of the present invention uses the above solution to compare the target answer with the standard answer, and determine the evaluation result of the evaluation model according to the number of wrong answers or correct answers returned. In addition, by comparing the dimensions, indicators and operators in the target DSL with the dimensions, indicators and operators in the standard DSL one by one, the reasons for the wrong answers can be accurately located. Thus, the performance and accuracy of the model are comprehensively evaluated.

[0091] In one embodiment of the present invention, after determining the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated, the method further includes:

[0092] For any business scenario:

[0093] Classify the test questions corresponding to the wrong answers according to the backend logic errors, the dimension error information, the indicator error information, and the operator error information;

[0094] Counting the number of questions of each type of test questions identified by the evaluation model in two evaluations;

[0095] Based on the number of questions, the proportion of each type of questions to be evaluated is determined.

[0096] Based on the proportion of each type of questions to be evaluated in the two evaluations, determine the changes in the proportion of each type of questions to be evaluated.

[0097] Specifically, for each business scenario, the test questions corresponding to the wrong answers can be classified according to the backend logic errors, dimension error information, indicator error information and operator error information; then the number of questions of each type of test questions identified by the evaluation model in the two evaluations is counted; further, based on the number of questions and the total number of questions, the proportion of each type of test questions is determined. For example, the proportion of each type of test questions can be as shown in Table 1. Based on the proportion of each type of test questions in the two evaluations, the change in the proportion of each type of test questions is determined. The embodiment of the present invention prevents the model's ability from regressing in a certain scenario by calculating the changes in the proportion of four types of reasons, namely, dimension identification problems, indicator identification problems, operator problems and backend problems. If the overall effect of the model regresses, the contribution of different causes in different scenarios can also be calculated to determine the importance of different scenarios and different causes.

[0098] Result classification Number of questions The proportion of questions correct N1 R1 Dimension identification error N2 R2 Indicator identification error N3 R3 Wrong number of operators N4 R4 The operator name is incorrect. N5 R5 Operator parameter error N6 R6 Operator dependency error N7 R7 Backend logic N8 R8

[0099] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0100] In one embodiment, an artificial intelligence model evaluation device is provided, and the artificial intelligence model evaluation device corresponds one-to-one to the artificial intelligence model evaluation method in the above embodiment. Figure 2 As shown, Figure 2 : is a schematic diagram of a structure of an artificial intelligence model evaluation device in one embodiment of the present invention, the artificial intelligence model evaluation device comprises:

[0101] The acquisition module 21 is used to acquire the test questions for the current business scenario, wherein the test questions are selected from a target question bank, and the target question bank is formed by designing questions based on the evaluation framework information and specification information preset in the business scenario and passing the quality inspection;

[0102] An input module 22, used to input the to-be-evaluated topic into the evaluation model, and obtain the target SQL and target DSL output by the evaluation model;

[0103] A query module 23 is used to perform database query on the target SQL and the standard SQL associated with the test question to obtain a target answer corresponding to the target SQL and a standard answer to the standard SQL;

[0104] The determination module 24 is used to determine the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated.

[0105] The artificial intelligence model evaluation device also includes:

[0106] The target question bank is formed based on the following steps:

[0107] Obtain a set of questions generated by a preset question-generating service software for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the set of questions;

[0108] According to the preset evaluation framework information and specification information, use the preset quality inspection service software to perform quality inspection on each of the above mentioned topics and its standard DSL and standard SQL;

[0109] If the topic and its corresponding standard DSL or standard SQL fail the quality inspection, then delete the topic that fails the quality inspection from the topic set;

[0110] Determine the similarity between the questions that have passed the quality inspection, so as to delete the questions whose similarity exceeds the preset similarity threshold;

[0111] Return to the step of obtaining a set of questions generated by the preset question-setting service software for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the set of questions, until the number of all questions that pass the quality inspection reaches a preset number threshold, and the target question bank is obtained.

[0112] The determination module 24 is further used for:

[0113] Compare the target answer with the standard answer to obtain a first result;

[0114] If the first result is that the answer is wrong, the dimensions, indicators and operators in the target DSL are compared with the dimensions, indicators and operators in the standard DSL respectively, so as to determine the problem location result according to the second result, wherein the dimension refers to a feature or attribute of the data, the indicator is used to identify the quantity of measuring or evaluating a specific dimension or data set, and the operator refers to a function or method for performing data operations or calculations;

[0115] The number of first questions whose first result is a correct answer and the number of second questions whose first result is an incorrect answer are counted, so as to determine the evaluation result based on the first number of questions and the second number of questions.

[0116] The determination module 24 is further used for:

[0117] If the second result is that the comparison of the dimension, the index and the operator is successful, it is determined that the problem location result is a backend logic error;

[0118] If the second result is that the dimension comparison fails, the corresponding dimension error information is used as the problem location result;

[0119] If the second result is that the indicator comparison fails, the corresponding indicator error information is used as the problem location result;

[0120] If the second result is that there is an operator comparison failure, the corresponding operator error information is used as the problem locating result.

[0121] The artificial intelligence model evaluation device also includes:

[0122] The total score determination module is used to determine the total score of the evaluation model based on the evaluation results of the evaluation model in all business scenarios and the weight associated with each business scenario.

[0123] The artificial intelligence model evaluation device also includes:

[0124] For any business scenario:

[0125] A classification module, used to classify the test questions corresponding to the wrong answers according to the backend logic errors, the dimension error information, the indicator error information and the operator error information;

[0126] A statistical module, used to count the number of questions of each type of test questions identified by the evaluation model in two evaluations;

[0127] The proportion determination module is used to determine the proportion of each type of questions to be evaluated based on the number of questions.

[0128] The change determination module is used to determine the change in the proportion of each type of questions to be evaluated based on the proportion of each type of questions to be evaluated in two evaluations.

[0129] The artificial intelligence model evaluation device also includes:

[0130] The evaluation framework information includes business problems, problem dimensions and problem indicators of different business scenarios, and question template information; the specification information includes DSL standard specifications corresponding to business problems of different business scenarios and SQL query metadata specifications of different business scenarios.

[0131] The artificial intelligence model evaluation device also includes:

[0132] The training module is used to feed back the questions that fail the quality inspection to the question setting service software so as to perform a small number of sample training on the question setting service software.

[0133] For the specific definition of the artificial intelligence model evaluation device, please refer to the definition of the artificial intelligence model evaluation method above, which will not be repeated here. Each module in the above-mentioned artificial intelligence model evaluation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0134] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, Figure 3 : is a schematic diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface and a database connected by a device bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating device, a computer-readable instruction and a database. The internal memory provides an environment for the operation of the operating device and the computer-readable instructions in the readable storage medium. The database of the computer device is used to store data involved in the artificial intelligence model evaluation method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instruction is executed by the processor, an artificial intelligence model evaluation method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0135] In one embodiment, a computer device is provided. The computer device may be a terminal device, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, and a network interface connected by a device bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, an artificial intelligence model evaluation method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0136] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, and when the processor executes the computer-readable instructions, the steps of the above-mentioned artificial intelligence model evaluation method are implemented.

[0137] In one embodiment, a readable storage medium is provided, and the readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the artificial intelligence model evaluation method as described above are implemented. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0138] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0139] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An artificial intelligence model evaluation method, characterized in that: include: Obtaining the test questions for the current business scenario, wherein the test questions are selected from a target question bank, and the target question bank is formed by designing questions based on the evaluation framework information and specification information preset for the business scenario and by a number of questions that have passed quality inspection; Input the to-be-evaluated topic into the evaluation model, and obtain the target SQL and target DSL output by the evaluation model; Performing database queries on the target SQL and the standard SQL associated with the test question respectively to obtain a target answer corresponding to the target SQL and a standard answer to the standard SQL; Based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated, an evaluation result and a problem location result are determined.

2. The artificial intelligence model evaluation method according to claim 1, characterized in that: The target question bank is formed based on the following steps: Obtain a set of questions generated by a preset question-generating service software for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the set of questions; According to the preset evaluation framework information and specification information, use the preset quality inspection service software to perform quality inspection on each of the above mentioned topics and its standard DSL and standard SQL; If the topic and its corresponding standard DSL or standard SQL fail the quality inspection, then delete the topic that fails the quality inspection from the topic set; Determine the similarity between the questions that have passed the quality inspection, so as to delete the questions whose similarity exceeds the preset similarity threshold; Return to the step of obtaining a set of questions generated by the preset question-setting service software for any business scenario, as well as the standard DSL and standard SQL corresponding to each question in the set of questions, until the number of all questions that pass the quality inspection reaches a preset number threshold, and the target question bank is obtained.

3. The artificial intelligence model evaluation method according to claim 1, characterized in that: The step of determining the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated includes: Compare the target answer with the standard answer to obtain a first result; If the first result is that the answer is wrong, then the dimensions, indicators and operators in the target DSL are compared with the dimensions, indicators and operators in the standard DSL respectively, so as to determine the problem location result according to the second result, wherein the dimension refers to the feature or attribute of the data, the indicator is used to identify the quantity of measuring or evaluating a specific dimension or data set, and the operator refers to a function or method for performing data operations or calculations; The number of first questions whose first result is a correct answer and the number of second questions whose first result is an incorrect answer are counted, so as to determine the evaluation result based on the first number of questions and the second number of questions.

4. The artificial intelligence model evaluation method according to claim 3, characterized in that: Determining the problem locating result according to the second result includes: If the second result is that the comparison of the dimension, the index and the operator is successful, it is determined that the problem location result is a backend logic error; If the second result is that the dimension comparison fails, the corresponding dimension error information is used as the problem location result; If the second result is that the indicator comparison fails, the corresponding indicator error information is used as the problem location result; If the second result is that there is an operator comparison failure, the corresponding operator error information is used as the problem locating result.

5. The artificial intelligence model evaluation method according to claim 3, characterized in that: After determining the evaluation result based on the first number of questions and the second number of questions, the method further includes: Based on the evaluation results of the evaluation model in all business scenarios and the weight associated with each business scenario, the total score of the evaluation model is determined.

6. The artificial intelligence model evaluation method according to claim 3, characterized in that: After determining the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated, the method further includes: For any business scenario: Classify the test questions corresponding to the wrong answers according to the backend logic errors, the dimension error information, the indicator error information, and the operator error information; Counting the number of questions of each type of test questions identified by the evaluation model in two evaluations; Based on the number of questions, the proportion of each type of questions to be evaluated is determined. Based on the proportion of each type of questions to be evaluated in the two evaluations, determine the changes in the proportion of each type of questions to be evaluated.

7. The artificial intelligence model evaluation method according to claim 1, characterized in that: The evaluation framework information includes business problems, problem dimensions and problem indicators of different business scenarios, and question template information; the specification information includes DSL standard specifications corresponding to business problems of different business scenarios and SQL query metadata specifications of different business scenarios.

8. The artificial intelligence model evaluation method according to claim 2, characterized in that: After quality checking each of the questions and their standard DSL and standard SQL according to the preset evaluation framework information and specification information, it also includes: The questions that fail the quality inspection are fed back to the question setting service software to perform a small number of sample training on the question setting service software.

9. An artificial intelligence model evaluation device, characterized in that: include: An acquisition module is used to acquire the test questions for the current business scenario, wherein the test questions are selected from a target question bank, and the target question bank is formed by designing questions based on the evaluation framework information and specification information preset for the business scenario and passing a number of questions that have passed quality inspection; An input module, used to input the to-be-evaluated topic into the evaluation model, and obtain the target SQL and target DSL output by the evaluation model; A query module, used to perform database query on the target SQL and the standard SQL associated with the test question, respectively, to obtain a target answer corresponding to the target SQL and a standard answer to the standard SQL; The determination module is used to determine the evaluation result and the problem location result based on the target answer, the standard answer, the target DSL and the standard DSL associated with the question to be evaluated.

10. A readable storage medium having computer readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the artificial intelligence model evaluation method as described in any one of claims 1 to 8 is implemented.

Images