Model evaluation method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a model evaluation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a prediction problem set, wherein the prediction problem set comprises a plurality of prediction problems; each prediction problem corresponds to reference information; inputting each prediction problem into the to-be-tested large model, and obtaining a structured query statement corresponding to the prediction problem output by the to-be-tested large model; and extracting element information of each structured query statement, and determining an evaluation result of the to-be-tested large model based on the element information and the reference information. By adopting the method, the large model can be automatically evaluated, and the efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a model evaluation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of natural language processing (NLP) and machine learning technologies, NL2SQL (Natural Language to SQL) technology has gradually matured. NL2SQL is a natural language processing technology that aims to convert natural language queries into structured query language (SQL). By leveraging the power of large models, it can accurately convert users' natural language queries into corresponding SQL queries, allowing non-database professionals to query database information smoothly.

[0003] In traditional technology, by using open source big models to build their own NL2SQL capabilities, not only the efficiency of NL2SQL generation is accelerated, but also data security is effectively guaranteed. In this process, the selection of big models is particularly important, so it is necessary to self-test the selected big models to determine whether the test results meet the expected results.

[0004] However, the current large model evaluation method requires professional evaluators to participate in the evaluation. Therefore, model evaluation is time-consuming, resulting in inefficient evaluation process. Summary of the invention

[0005] Based on this, it is necessary to provide a model evaluation method, device, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of model evaluation in response to the above technical problems.

[0006] In a first aspect, the present application provides a model evaluation method, the method comprising:

[0007] Obtaining a prediction problem set, wherein the prediction problem set includes a plurality of prediction problems; each of the prediction problems corresponds to reference information;

[0008] Input each of the prediction problems into the large model to be tested, and obtain a structured query statement corresponding to the prediction problem output by the large model to be tested;

[0009] Element information of each of the structured query statements is extracted, and an evaluation result of the large model to be tested is determined based on the element information and the reference information.

[0010] In one embodiment, determining the evaluation result of the large model to be tested based on the element information and the reference information includes:

[0011] Determine, based on the element information and the reference information of the structured query statement, an evaluation result of each element information corresponding to the prediction question set;

[0012] For each of the structured query statements, determining a first dimension evaluation result according to the evaluation results of the element information corresponding to the same structured query statement obtained;

[0013] Determining a second dimension evaluation result according to an evaluation result of each element information in the prediction problem set;

[0014] Based on the first dimension evaluation result and the second dimension evaluation result, the evaluation result of the large model to be tested is determined.

[0015] In one embodiment, the element information includes prompt information and metadata information corresponding to the prompt information; the reference information includes a reference prompt information set and a constraint condition set corresponding to the reference prompt information set; and determining the evaluation result of each element information corresponding to the prediction question set based on the element information and the reference information of the structured query statement includes:

[0016] Matching and querying the prompt information of each structured query statement with the reference prompt information set to determine the prompt information result of each structured query statement;

[0017] vectorizing the metadata information and the constraint condition set of each structured query statement to obtain a first vector corresponding to the metadata information and a second vector set corresponding to the constraint condition set, respectively; the second vector set includes a plurality of second vectors, the constraint condition set includes a plurality of constraint conditions, and the second vectors correspond to the constraint conditions one by one;

[0018] Performing cosine similarity calculation on the first vector and each of the second vectors to obtain multiple vector similarities, and determining a meta-information similarity result of each of the structured query statements according to the multiple vector similarities;

[0019] According to the prompt information results of each structured query statement and the meta information similarity results, an evaluation result of each element information corresponding to the prediction question set is determined.

[0020] In one embodiment, the element information includes prompt information and metadata information corresponding to the prompt information; and for each structured query statement, determining the first dimension evaluation result according to the evaluation result of the element information corresponding to the same structured query statement obtained includes:

[0021] For each of the structured query statements, when the prompt information corresponding to the structured query statement indicates that it is correct, and the metadata information indicates that it is correct, then the structured query statement indicates that it is correct;

[0022] When at least one of the prompt information corresponding to the structured query statement or the metadata information indicates an error, the structured query statement indicates an error;

[0023] Acquire the number of the structured query statements that are correct and the number of the structured query statements that are incorrect;

[0024] The first dimension evaluation result is determined according to the number of the structured query statements that are correct and the number of the structured query statements that are incorrect.

[0025] In one embodiment, determining the second dimension evaluation result according to the evaluation result of each element information in the prediction problem set includes:

[0026] For each meta-information similarity result of the structured query statement, when at least one of the vector similarities is greater than a preset similarity threshold, it indicates that the element information meets the constraint condition;

[0027] When there is no vector whose similarity is greater than a preset similarity threshold, it indicates that the element information does not meet the constraint condition;

[0028] Obtaining the number of element information corresponding to the prediction problem set that meets the constraint condition and the number of element information corresponding to the prediction problem set that does not meet the constraint condition;

[0029] The second dimension evaluation result is determined according to the number of the element information that indicates compliance with the constraint condition and the number of the element information that indicates non-compliance with the constraint condition.

[0030] In one embodiment, determining the evaluation result of the large model to be tested based on the first dimension evaluation result and the second dimension evaluation result includes:

[0031] Obtain a first weight of the evaluation result of the first dimension and a second weight of the evaluation result of the second dimension;

[0032] Determining a target evaluation value according to the first dimension evaluation result, the first weight, the second dimension evaluation result, and the second weight;

[0033] When the target evaluation value is greater than a preset evaluation threshold, the evaluation result of the large model to be tested is passed.

[0034] In a second aspect, the present application also provides a model evaluation device, the device comprising:

[0035] An acquisition module, used for acquiring a prediction problem set, wherein the prediction problem set includes a plurality of prediction problems; each of the prediction problems corresponds to reference information;

[0036] An input-output module, used to input each prediction problem into the large model to be tested, and obtain a structured query statement corresponding to the prediction problem output by the large model to be tested;

[0037] The determination module is used to extract element information of each structured query statement, and determine the evaluation result of the large model to be tested based on the element information and the reference information.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0040] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0041] The above-mentioned model evaluation method, device, computer equipment, computer-readable storage medium and computer program product use the matching degree between the element information of the structured query statement and the reference information as the evaluation index, and then evaluate the matching degree between the structured query statement output by the large model according to the prediction problem and the reference information of the prediction problem, thereby avoiding the manual judgment of each structured query statement output by the large model by professional testers, and realizing automatic extraction of the element information of each structured query statement output by the large model; matching the element information and the reference information, and realizing fully automatic determination of the evaluation result of the large model to be tested based on the matching result. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] FIG1 is an application environment diagram of a model evaluation method in one embodiment;

[0044] FIG2 is a schematic diagram of a flow chart of a model evaluation method in one embodiment;

[0045] FIG3 is a schematic diagram of a process for determining an evaluation result of a large model to be tested in one embodiment;

[0046] FIG4 is a schematic diagram of a process for determining an evaluation result of each element information corresponding to a prediction problem set in one embodiment;

[0047] FIG5 is a schematic diagram of a process for determining a first dimension evaluation result in one embodiment;

[0048] FIG6 is a schematic diagram of a process for determining a second dimension evaluation result in one embodiment;

[0049] FIG7 is a schematic diagram of a process for determining an evaluation result of a large model to be tested in another embodiment;

[0050] FIG8 is a structural block diagram of a model evaluation device in one embodiment;

[0051] FIG. 9 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] The model evaluation method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains a set of prediction problems, which includes multiple prediction problems; each prediction problem corresponds to reference information; each prediction problem is input into the large model to be tested, and the structured query statement corresponding to the prediction problem output by the large model to be tested is obtained; the element information of each structured query statement is extracted, and the evaluation result of the large model to be tested is determined based on the element information and reference information. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0054] In an exemplary embodiment, as shown in FIG2 , a model evaluation method is provided, which is described by taking the method applied to the server in FIG1 as an example, and includes the following steps S202 to S206 . Among them:

[0055] Step S202, obtaining a prediction problem set, the prediction problem set includes a plurality of prediction problems; each prediction problem corresponds to reference information.

[0056] The prediction problem set includes multiple prediction problems, and each prediction problem corresponds to reference information.

[0057] Optionally, the prediction question set may be generated by a server that executes the model evaluation method, or may be generated by other servers that do not execute the model evaluation method, and then acquired by the server that executes the model evaluation method.

[0058] Optionally, before obtaining the predicted problem set, the server builds a system database, such as a library management system, and then inputs the blueprint Schema information in the database into the general large model deployed in the server, and inputs a constraint set, which includes at least one constraint, such as requiring the generation of n problems with index constraints and Join algorithm constraints, and the general large model generates a set of test problems that meet the constraints. Each test problem in the test problem set corresponds to reference information, and the reference information includes a constraint set. After the server generates the test problem set through the deployed general large model, the server obtains the test problem set.

[0059] Step S204: input each prediction question into the large model to be tested, and obtain a structured query statement corresponding to the prediction question output by the large model to be tested.

[0060] Optionally, the server inputs each prediction question in the test question set into the large model to be tested, and the large model to be tested parses and processes each prediction question and inputs a structured query statement corresponding to each prediction question, that is, an SQL statement.

[0061] Optionally, the server inputs the prediction questions into the large model to be tested one by one, and obtains the SQL statements corresponding to the test questions output by the large model to be tested. The server inputs multiple prediction questions into the large model to be tested at one time, and the large model to be tested analyzes the multiple prediction questions. The large model to be tested generates SQL statements corresponding to each prediction question in the multiple prediction questions at one time, wherein the large model to be tested generates at least one SQL statement corresponding to one prediction question. For example, the large model to be tested generates SQL1 for test question 1, and for another example, the large model to be tested generates SQL21 and SQL22 for test question 2. It can be understood that the number of structured statements generated for test questions in this embodiment can be one, two or more, which is not limited here.

[0062] Step S206, extracting element information of each structured query statement, and determining the evaluation result of the large model to be tested based on the element information and reference information.

[0063] The element information is the information carried by the structured query statement SQL. A structured query statement SQL includes at least one element information.

[0064] Optionally, the server extracts element information of each structured query statement SQL generated for each test question. For example, for SQL1, one element information is extracted; for SQL2, n element information is extracted; n is a positive integer. The server determines the evaluation result of the large model to be tested based on the matching of the extracted element information of each SQL with the reference information of the prediction question that generated the SQL.

[0065] For example, the test problem set includes 10 test problems, and each test problem corresponds to reference information, such as constraints. The server inputs the 10 test problems into the large model to be tested, and receives the SQL output by the large model to be tested for each test problem. If the test problems correspond to the SQLs one by one, the large model to be tested outputs 10 SQLs. The server extracts the element information of each of the 10 SQLs. The element information corresponding to each SQL is matched and analyzed with the reference information corresponding to the test problem that generated the SQL. The evaluation result of the large model to be tested is determined based on the evaluation of the 10 SQLs and the evaluation of the element information of each of the 10 SQLs corresponding to the test problem set.

[0066] In the above model evaluation method, the matching degree between the element information of the structured query statement and the reference information is used as an evaluation index, and then the matching degree between the structured query statement output by the large model according to the prediction problem and the reference information of the prediction problem is evaluated, thereby avoiding the manual judgment of each structured query statement output by the large model by professional testers, and realizing automatic extraction of the element information of each structured query statement output by the large model; matching the element information and the reference information, and realizing fully automatic determination of the evaluation result of the large model to be tested based on the matching result.

[0067] In an exemplary embodiment, as shown in FIG3 , based on the element information and the reference information, determining the evaluation result of the large model to be tested includes steps S302 to S308 . In which:

[0068] Step S302: determining an evaluation result of each element information corresponding to the prediction question set based on the element information and reference information of the structured query statement.

[0069] Optionally, the server determines the evaluation result of each element information corresponding to the prediction problem set based on the element information of each structured query statement generated according to the test problem set and the reference information corresponding to each test problem in the test problem set. For example, the test problem set includes 10 test questions; 10 SQLs are generated according to the test questions, SQL1 includes two element information; SQL2 includes n element information; ...; SQL10 includes m element information; where nm is a random positive integer, and the test problem set corresponds to 100 element information. The server matches each element information corresponding to each SQL judgment with the reference information of the test problem that generated the SQL, and obtains the evaluation result of each element information corresponding to the 10 SQLs, that is, the evaluation result of each element information corresponding to the prediction problem set, and the evaluation result of 100 element information.

[0070] Step S304: for each structured query statement, determine the first dimension evaluation result according to the evaluation result of the element information corresponding to the same structured query statement obtained.

[0071] Optionally, for each SQL, the server obtains the evaluation results of the element information corresponding to the same structured query statement, such as the evaluation results of two element information corresponding to SQL1, the evaluation results of n element information corresponding to SQL2; ...; the evaluation results of m element information corresponding to SQL10. The server determines the evaluation result of SQL1 based on the evaluation results of the two element information corresponding to SQL1; determines the evaluation result of SQL2 based on the evaluation results of n element information corresponding to SQL2, and so on until SQL10. Based on the evaluation results of these 10 SQLs, the first dimension evaluation result is determined, that is, the first dimension evaluation result is determined based on the number of SQLs.

[0072] Step S306, determining the second dimension evaluation result according to the evaluation result of each element information in the prediction problem set.

[0073] Optionally, the server determines the second dimension evaluation result based on the evaluation result of each element information in the prediction problem set, such as the evaluation result of 100 element information, that is, the second dimension evaluation result is determined based on the number of element information.

[0074] Step S308, determining the evaluation result of the large model to be tested based on the first dimension evaluation result and the second dimension evaluation result.

[0075] Optionally, the server determines the evaluation result of the large model to be tested based on the evaluation results of the first dimension and the evaluation results of the second dimension, for example, by calculating the evaluation results of the first dimension and the evaluation results of the second dimension using a weighted average, arithmetic mean, harmonic mean, or the like.

[0076] It should be noted that the number of test questions in the test question set, the number of SQLs generated according to the test questions, and the number of element information included in each SQL can be set according to actual conditions. The above numbers are only for ease of understanding and are not specific limitations.

[0077] In this embodiment, the large model is evaluated from two aspects: the first dimension evaluation results and the second dimension evaluation results. The multi-faceted evaluation ensures the accuracy of the evaluation of the SQL generation ability of the model to be tested. By decomposing each SQL into elements for judgment, the problem of inconsistent element order but correct SQL semantics can be effectively avoided.

[0078] In an exemplary embodiment, as shown in FIG4 , the element information includes prompt information and metadata information corresponding to the prompt information; the reference information includes a reference prompt information set and a constraint condition set corresponding to the reference prompt information set; based on the element information and reference information of the structured query statement, determining the evaluation result of each element information corresponding to the prediction question set includes steps S402 to S408. Among them:

[0079] Step S402: Match the prompt information of each structured query statement with the reference prompt information set to determine the prompt information result of each structured query statement.

[0080] The element information includes the hint information and the metadata information corresponding to the hint information, which can be recorded as [Hint, corresponding metadata information]. Hints can also be called SQL Hints (SQL hints or optimization hints), which are a special type of comments added to SQL statements to provide additional information or instructions to the database query optimizer to guide it on how to execute queries. These hints cover different aspects, such as join strategies, index usage, query execution paths, etc., with the aim of helping the optimizer make more expected decisions.

[0081] The reference information includes a reference hint information set and a constraint condition set corresponding to the reference hint information set. Multiple constraint conditions of each problem correspond to reference hints. It can be recorded as [[reference hint1, constraint condition 1 corresponding to the reference hint information set]...[reference hintn, constraint condition n corresponding to the reference hint information set]].

[0082] If SQLn corresponds to m element information, the server extracts the element information of SQL, and the element information can be recorded according to the element list. The element list format is as follows: [[Hint1, corresponding to metadata information 1] ... [Hintm, corresponding to metadata information m]], where m is a positive integer.

[0083] Optionally, the server performs a matching query on the hint information Hint of each structured query statement from the reference hint information set [reference hint1...reference hintm] by hashing, such as calculating the hash value of the hint information Hint and the hash value of each reference hint information in the reference hint information set. The hash value of the hint information Hint is traversed through the hash values ​​of each reference hint information in the reference hint information set. If a match is found, the hint information result is a successful match, which can be recorded as a correct hint information result. If a match is not found, the hint information result is a failed match, which can be recorded as a failed hint information result.

[0084] Step S404, vectorize the metadata information and constraint set of each structured query statement to obtain a first vector corresponding to the metadata information and a second vector set corresponding to the constraint set; the second vector set includes multiple second vectors, and the constraint set includes multiple constraints.

[0085] Among them, the second vector corresponds to the constraint condition one by one.

[0086] Optionally, the server inputs the metadata information of each SQL into the Embedding model for vectorization to obtain an initial first vector; at the same time, the constraint conditions of the test problem corresponding to the SQL are input into the Embedding model for vectorization to obtain an initial second vector set, the initial second vector set includes multiple initial second vectors, and the constraint condition set includes multiple constraint conditions.

[0087] Optionally, the server normalizes the initial first vector and all the initial second vectors to obtain the first vector and each second vector.

[0088] In practical applications, the reason why the test problem and constraints are not directly vectorized is that in addition to the constraints, the test problem also contains other irrelevant information, and the constraints will also affect the vectorization results. Therefore, we choose to obtain a set of constraints from a test problem and then vectorize each constraint, thereby ensuring the accuracy of the final verification result.

[0089] Step S406 , performing cosine similarity calculation on the first vector and each second vector respectively to obtain multiple vector similarities, and determining the meta information similarity result of each structured query statement according to the multiple vector similarities.

[0090] Optionally, the server performs Cosine Similarity calculation on the normalized first vector corresponding to the metadata information and the normalized second vector corresponding to each constraint condition in the test question to obtain multiple vector similarities, and the server determines the metadata similarity result of each structured query statement based on the multiple vector similarities. The closer the cosine value is to 1, the higher the vector similarity. A threshold is set here, and as long as it exceeds the threshold range, it is determined that the metadata similarity result of the element meets the constraint condition. For example, if the similarity between the first vector and the second vector of a constraint condition reaches a threshold, it is determined that the metadata similarity result of the element meets the constraint condition, and it is recorded as the metadata similarity result indicating that it is correct.

[0091] Step S408: Determine the evaluation result of each element information corresponding to the prediction question set according to the prompt information result and the meta information similarity result of each structured query statement.

[0092] Optionally, the server determines the evaluation results of the 100 element information based on the prompt information results and the meta information similarity results of each structured query statement, such as when the prompt information result indicates success and the meta information similarity result indicates that the constraint condition is satisfied. If the prediction question set corresponds to 100 element information, the server determines the evaluation results of the 100 element information based on the prompt information results and the meta information similarity results of each structured query statement.

[0093] In this embodiment, a SQL statement is divided into more fine-grained elements through prompt information results and metadata information results, thereby improving accuracy; the accuracy of the SQL generation capability evaluation of the model to be tested can also be ensured based on multiple results. The test problem is divided into multiple constraints, and the similarity between the metadata information in the SQL and each constraint of the corresponding test problem is calculated. This method not only eliminates irrelevant information related to the constraints, but also makes the resolution between the constraints higher, thereby ensuring the accuracy of the vector similarity calculation.

[0094] In an exemplary embodiment, as shown in FIG5 , the element information includes prompt information and metadata information corresponding to the prompt information; for each structured query statement, the first dimension evaluation result is determined according to the evaluation result of the element information corresponding to the same structured query statement obtained, including steps S502 to S506. Among them:

[0095] Step S502: for each structured query statement, when the prompt information corresponding to the structured query statement indicates correctness and the metadata information indicates correctness, the structured query statement indicates correctness. When at least one of the prompt information or the metadata information corresponding to the structured query statement indicates an error, the structured query statement indicates an error.

[0096] Optionally, for each structured query statement, obtain the element information corresponding to the SQL, as well as the prompt information and metadata information corresponding to the element information. When the prompt information corresponding to the element information indicates success and the metadata information indicates success, it indicates that the element information is correct. When the element information corresponding to the SQL indicates that all are correct, the SQL indicates that it is correct, which is recorded as .

[0097] When at least one of the prompt information or metadata information corresponding to the structured query statement indicates an error, the structured query statement indicates an error, which is recorded as .

[0098] For example, in a SQL statement, SELETE divides a SQL statement into multiple elements to calculate the accuracy separately. If each element is equal, the overall SQL statement is equivalent. That is, SELECT * FROM t1 USE INDEX (i1) IGNORE INDEX FOR ORDER BY (i2) ORDER BY a is equivalent to SELECT * FROM t1 IGNORE INDEX FOR ORDER BY (i2) ORDER BY a USE INDEX (i1). Therefore, decomposing the statement into elements for judgment can effectively avoid the problem of inconsistent element order but correct SQL semantics.

[0099] Step S504: obtaining the number of correct structured query statements and the number of incorrect structured query statements.

[0100] Optionally, the server obtains a representation The number of For example, 100 SQL statements are generated for the test problem set. The number of is x; it means The number of is y. Among them, x+y=100.

[0101] Step S506: determining the first dimension evaluation result according to the number of correct structured query statements and the number of incorrect structured query statements.

[0102] Optionally, the server obtains a representation The number of The number of queries is calculated based on the SQL correctness according to formula (1) to determine the evaluation result of the first dimension. For example, the evaluation result of the first dimension = x / (x+y).

[0103] Formula (1)

[0104] In the formula, represents the evaluation result of the first dimension; represents the number of correct SQL statements ; indicates the number of incorrect SQL, recorded as .

[0105] That is, there is a constraint that matches multiple SQL metadata information, or a metadata information matches multiple constraints. However, in most cases, when the correct SQL is generated, the metadata information corresponds to the user constraint one by one; when the wrong SQL is generated, the constraint is likely to fail to match, and only a very small probability will occur. In the evaluation of a large number of data sets, this special case will be washed away by a large number of test samples and will not affect the final result.

[0106] In this embodiment, by evaluating the evaluation results of all corresponding SQLs in the test problem set, the model to be tested can be evaluated from the dimension of SQL.

[0107] In an exemplary embodiment, as shown in FIG6 , determining the second dimension evaluation result according to the evaluation result of each element information in the prediction problem set includes steps S602 to S606 . Among them:

[0108] Step S402, for each structured query sentence's meta-information similarity result, when at least one vector similarity exists that is greater than a preset similarity threshold, it indicates that the element information meets the constraint condition; when no vector similarity exists that is greater than the preset similarity threshold, it indicates that the element information does not meet the constraint condition.

[0109] Optionally, if the 1000 elements corresponding to the test question set are tested, and the element information is judged whether it meets the constraint condition based on whether there is at least one vector similarity greater than the preset similarity threshold. The server calculates the meta-information similarity result for each structured query statement. If there is at least one vector similarity greater than the preset similarity threshold, it indicates that the element information meets the constraint condition, which is recorded as ; When there is no vector similarity greater than the preset similarity threshold, it means that the element information does not meet the constraint conditions, which is recorded as .

[0110] Step S404, obtaining the number of element information corresponding to the prediction problem set that meets the constraint conditions and the number of element information corresponding to the prediction problem set that does not meet the constraint conditions.

[0111] Optionally, the server obtains The number and The number of. The number of is a; The number of is b. Among them, a+b=1000.

[0112] Step S406: determining the second dimension evaluation result according to the number of element information indicating that the constraint condition is met and the number of element information indicating that the constraint condition is not met.

[0113] Optionally, the server calculates the accuracy of the element dimension according to the number of element information that meets the constraint condition and the number of element information that does not meet the constraint condition, and determines the evaluation result of the second dimension according to formula (2).

[0114] Formula (2)

[0115] In the formula, represents the evaluation result of the second dimension; the number of element information that meets the constraint conditions is recorded as ; The number of element information that does not meet the constraint conditions is recorded as .

[0116] In this embodiment, a complex SQL may include multiple elements, and a simple SQL may include only one element. The difficulty of generating correct complex SQL for a large model is much greater than the difficulty of generating simple SQL. If it is judged from one dimension of SQL, it still cannot fully reflect the SQL generation capability of the large model. Therefore, a new element dimension is added for evaluation, which improves the accuracy of the evaluation of the large model to be tested.

[0117] In an exemplary embodiment, as shown in FIG7 , based on the first dimension evaluation result and the second dimension evaluation result, determining the evaluation result of the large model to be tested includes steps S702 to S706 . Among them:

[0118] Step S702, obtaining a first weight of the first dimension evaluation result and a second weight of the second dimension evaluation result.

[0119] Optionally, the server obtains a first weight a of the evaluation result of the first dimension and a second weight b of the evaluation result of the second dimension, which can be freely given by the evaluator according to actual conditions.

[0120] Step S704, determining a target evaluation value according to the first dimension evaluation result, the first weight, the second dimension evaluation result and the second weight.

[0121] Optionally, the server evaluates the result according to the first dimension , first weight a, second dimension evaluation result and the second weight b, and the target evaluation value is determined by formula (3).

[0122] Formula (3)

[0123] In the formula, a represents the first weight; b represents the second weight; Indicates the evaluation result of the first dimension; Represents the evaluation result of the second dimension.

[0124] Step S706: When the target evaluation value is greater than the preset evaluation threshold, the evaluation result of the large model to be tested is passed.

[0125] Optionally, when the target evaluation value is greater than a preset evaluation threshold, the evaluation result of the large model to be tested is passed.

[0126] In this embodiment, by weighting the evaluation results of the first dimension and the evaluation results of the second dimension, the evaluation results can be made to be in line with reality and the accuracy can be improved.

[0127] In an exemplary embodiment, before obtaining the set of predicted questions, the server builds a database of a system, such as a library management system, and then inputs the blueprint Schema information in the database into the general large model deployed in the server, and inputs a set of constraints, the set of constraints includes at least one constraint, such as requiring the generation of n questions with index constraints and Join algorithm constraints, and the general large model generates a set of test questions that meet the constraints. Among them, each test question in the test question set corresponds to reference information, and the reference information includes a set of constraints and reference hint information corresponding to the set of constraints. For example, it is required to generate 100 questions with Join order constraints and subquery alias constraints, and 100 questions with JOIN connection order constraints and index constraints. Use a set to store the output Hints of the large model, such as {USEINDEX, STRAIGHT_JOIN}; use a list to store the output of the constraint set of the large model, such as [using the primary key index of Table 1, first JOIN Table 1 with Table 2, and finally JOIN Table 3].

[0128] The server inputs the problem data set into the large model to be tested, and the large model to be tested outputs SQL according to the test problem.

[0129] The server extracts element information of each structured query statement, where the element information includes hint information and metadata information corresponding to the hint information. Here, taking an SQL statement as an example, the following is the example SQL and the corresponding element information.

[0130] SQL:

[0131] SELECT / *+ JOIN_PREFIX(t2, t5@subq2) JOIN_ORDER(t4@subq1, t3) * / COUNT(*)FROM t2 JOIN t3

[0132] WHERE t2.f1 IN (SELECT / *+ QB_NAME(subq1) * / f1 FROM t4)

[0133] AND t2.f1 IN (SELECT / *+ QB_NAME(subq2) * / f1 FROM t5);

[0134] Element information:

[0135] [[JOIN_PREFIX, [t2, t5@subq2]],

[0136] [JOIN_ORDER, [t4@subq1, t3]],

[0137] [QB_NAME, [subq1]],

[0138] [QB_NAME, [subq2]]].

[0139] The server matches the hint information Hint of each structured query statement from the reference hint information set [reference hint1...reference hintm] by hashing, such as calculating the hash value of the hint information Hint and the hash value of each reference hint information in the reference hint information set. The hash value of the hint information Hint is traversed through the hash values ​​of each reference hint information in the reference hint information set. If a match is found, the hint information result is a successful match, which can be recorded as the hint information result indicating correctness; if a match is not found, the hint information result is a failed match, which can be recorded as the hint information result indicating failure, thereby determining the hint information result.

[0140] The server inputs the metadata information of each SQL into the Embedding model for vectorization to obtain the initial first vector; at the same time, the constraints of the test problem corresponding to the SQL are input into the Embedding model for vectorization to obtain the initial second vector set, which includes multiple initial second vectors, and the constraint set includes multiple constraints. Cosine Similarity calculation is performed on the normalized first vector corresponding to the metadata information and the normalized second vector of each constraint in the corresponding test problem to obtain multiple vector similarities. The server determines the meta information similarity result of each structured query statement based on the multiple vector similarities. The closer the cosine value is to 1, the higher the vector similarity. A threshold is set here. As long as it exceeds the threshold range, it is determined that the meta information similarity result of the element meets the constraint condition. For example, if the similarity between the first vector and the second vector of a constraint condition reaches a threshold, it is determined that the meta information similarity result of the element meets the constraint condition, and it is recorded as the meta information similarity result indicating that it is correct.

[0141] When the meta information similarity result indicates that it is correct and the prompt information result indicates that it is correct, the element information indicates that it is correct. If all elements of an SQL are judged to be correct, the SQL indicates that it is correct.

[0142] For each structured query statement, obtain the element information corresponding to the SQL, as well as the prompt information and metadata information corresponding to the element information. When the prompt information corresponding to the element information indicates success and the metadata information indicates success, it indicates that the element information is correct. When the element information corresponding to the SQL indicates correctness, the SQL is correct, which is recorded as When at least one of the prompt information or metadata information corresponding to the structured query statement indicates an error, the structured query statement indicates an error, which is recorded as The server obtains the representation The number of The number of queries is calculated based on the SQL correctness according to formula (1) to determine the evaluation result of the first dimension. For example, the evaluation result of the first dimension = x / (x+y).

[0143] Formula (1)

[0144] In the formula, represents the evaluation result of the first dimension; represents the number of correct SQL statements ; indicates the number of incorrect SQL, recorded as .

[0145] If the 100 SQL statements corresponding to the test problem set have 1000 elements, and whether the element information meets the constraint conditions is determined based on whether there is at least one vector similarity greater than the preset similarity threshold. The server generates the meta-information similarity results for each structured query statement. If there is at least one vector similarity greater than the preset similarity threshold, it indicates that the element information meets the constraint conditions, which is recorded as ; When there is no vector similarity greater than the preset similarity threshold, it means that the element information does not meet the constraint conditions, which is recorded as . Server acquisition The number and The number of. The number of is a; The number is b. Where a+b=1000. According to the number of element information that meets the constraint conditions and the number of element information that does not meet the constraint conditions, the correctness of the element dimension is calculated, and the evaluation result of the second dimension is determined according to formula (2).

[0146] Formula (2)

[0147] In the formula, represents the evaluation result of the second dimension; the number of element information that meets the constraint conditions is recorded as ; The number of element information that does not meet the constraint conditions is recorded as .

[0148] The server evaluates the results based on the first dimension , first weight a, second dimension evaluation result and the second weight b, and the target evaluation value is determined by formula (3).

[0149] Formula (3) In the formula, a represents the first weight; b represents the second weight; Indicates the evaluation result of the first dimension; Represents the evaluation result of the second dimension.

[0150] When the target evaluation value is greater than the preset evaluation threshold, the evaluation result of the large model to be tested is passed. Or when multiple large models to be tested are evaluated, the evaluation result of the large model to be tested with the highest target evaluation value is passed, and the evaluation results of other large models to be tested are failed.

[0151] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0152] Based on the same inventive concept, the embodiment of the present application also provides a model evaluation device for implementing the model evaluation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more model evaluation device embodiments provided below can refer to the limitations on the model evaluation method above, and will not be repeated here.

[0153] In an exemplary embodiment, as shown in FIG8 , a model evaluation device is provided, including: an acquisition module 801, an input-output module 802, and a determination module 803, wherein:

[0154] The acquisition module 801 is used to acquire a prediction problem set, where the prediction problem set includes multiple prediction problems; each prediction problem corresponds to reference information.

[0155] The input-output module 802 is used to input each prediction problem into the large model to be tested, and obtain the structured query statement corresponding to the prediction problem output by the large model to be tested.

[0156] The determination module 803 is used to extract element information of each structured query statement, and determine the evaluation result of the large model to be tested based on the element information and reference information.

[0157] In an exemplary embodiment, the determination module 803 includes:

[0158] The element evaluation determination unit is used to determine the evaluation result of each element information corresponding to the prediction question set based on the element information and reference information of the structured query statement.

[0159] The first dimension evaluation determination unit is used to determine the first dimension evaluation result for each structured query statement according to the evaluation result of the element information corresponding to the same structured query statement obtained.

[0160] The second dimension evaluation determination unit is used to determine the second dimension evaluation result according to the evaluation result of each element information in the prediction problem set.

[0161] The model evaluation determination unit is used to determine the evaluation result of the large model to be tested based on the first dimension evaluation result and the second dimension evaluation result.

[0162] In an exemplary embodiment, the element information includes prompt information and metadata information corresponding to the prompt information; the reference information includes a reference prompt information set and a constraint condition set corresponding to the reference prompt information set; and the element evaluation determination unit includes:

[0163] The prompt information result generating subunit is used to match the prompt information of each structured query statement with the reference prompt information set to determine the prompt information result of each structured query statement.

[0164] The vectorization subunit is used to vectorize the metadata information and constraint set of each structured query statement, and obtain a first vector corresponding to the metadata information and a second vector set corresponding to the constraint set respectively; the second vector set includes multiple second vectors, the constraint set includes multiple constraints, and the second vectors correspond to the constraints one by one.

[0165] The similarity result generating subunit is used to perform cosine similarity calculation on the first vector and each second vector respectively to obtain multiple vector similarities, and determine the meta information similarity result of each structured query statement according to the multiple vector similarities.

[0166] The element evaluation determination subunit is used to determine the evaluation result of each element information corresponding to the prediction question set according to the prompt information result and the meta information similarity result of each structured query statement.

[0167] In an exemplary embodiment, the element information includes prompt information and metadata information corresponding to the prompt information; the first dimension evaluation determination unit includes:

[0168] The prompt information representation subunit is used for each structured query statement. When the prompt information corresponding to the structured query statement is correct and the metadata information is correct, the structured query statement is correct; when at least one of the prompt information or the metadata information corresponding to the structured query statement is wrong, the structured query statement is wrong.

[0169] The first acquisition subunit is used to acquire the number of correct structured query statements and the number of incorrect structured query statements.

[0170] The first dimension evaluation determination subunit is used to determine the first dimension evaluation result according to the number of correct structured query statements and the number of incorrect structured query statements.

[0171] In an exemplary embodiment, the second dimension evaluation determination unit includes:

[0172] The element information representation subunit is used for the meta-information similarity results of each structured query statement. When at least one vector similarity is greater than the preset similarity threshold, it indicates that the element information meets the constraint conditions; when no vector similarity is greater than the preset similarity threshold, it indicates that the element information does not meet the constraint conditions.

[0173] The element information evaluation subunit is used to obtain the number of element information that meets the constraint conditions and the number of element information that does not meet the constraint conditions corresponding to the prediction problem set.

[0174] The second dimension evaluation determination subunit is used to determine the second dimension evaluation result according to the number of element information that meets the constraint conditions and the number of element information that does not meet the constraint conditions.

[0175] In an exemplary embodiment, the model evaluation determination unit includes:

[0176] The second acquisition subunit is used to acquire a first weight of the first dimension evaluation result and a second weight of the second dimension evaluation result.

[0177] The target evaluation value determination subunit is used to determine the target evaluation value according to the first dimension evaluation result, the first weight, the second dimension evaluation result and the second weight.

[0178] The determination subunit is used to determine that when the target evaluation value is greater than a preset evaluation threshold, the evaluation result of the large model to be tested is passed.

[0179] Each module in the above-mentioned model evaluation device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0180] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store structured query statement data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model evaluation method is implemented.

[0181] Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0182] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0183] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0184] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0185] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A model evaluation method, characterized in that: The method comprises: Obtaining a prediction problem set, wherein the prediction problem set includes a plurality of prediction problems; each of the prediction problems corresponds to reference information; Input each of the prediction problems into the large model to be tested, and obtain a structured query statement corresponding to the prediction problem output by the large model to be tested; Element information of each of the structured query statements is extracted, and an evaluation result of the large model to be tested is determined based on the element information and the reference information.

2. The method according to claim 1, characterized in that The step of determining the evaluation result of the large model to be tested based on the element information and the reference information includes: Determine, based on the element information and the reference information of the structured query statement, an evaluation result of each element information corresponding to the prediction question set; For each of the structured query statements, determining a first dimension evaluation result according to the evaluation results of the element information corresponding to the same structured query statement obtained; Determining a second dimension evaluation result according to an evaluation result of each element information in the prediction problem set; Based on the first dimension evaluation result and the second dimension evaluation result, the evaluation result of the large model to be tested is determined.

3. The method according to claim 2, characterized in that The element information includes prompt information and metadata information corresponding to the prompt information; The reference information includes a reference prompt information set and a constraint condition set corresponding to the reference prompt information set; The step of determining the evaluation result of each element information corresponding to the prediction question set based on the element information and the reference information of the structured query statement includes: Matching and querying the prompt information of each structured query statement with the reference prompt information set to determine the prompt information result of each structured query statement; vectorizing the metadata information and the constraint condition set of each structured query statement to obtain a first vector corresponding to the metadata information and a second vector set corresponding to the constraint condition set, respectively; the second vector set includes a plurality of second vectors, the constraint condition set includes a plurality of constraint conditions, and the second vectors correspond to the constraint conditions one by one; Performing cosine similarity calculation on the first vector and each of the second vectors to obtain multiple vector similarities, and determining a meta-information similarity result of each of the structured query statements according to the multiple vector similarities; According to the prompt information results of each structured query statement and the meta information similarity results, an evaluation result of each element information corresponding to the prediction question set is determined.

4. The method according to claim 2, characterized in that: The element information includes prompt information and metadata information corresponding to the prompt information; The step of determining the first dimension evaluation result for each structured query statement according to the evaluation result of the element information corresponding to the same structured query statement obtained includes: For each of the structured query statements, when the prompt information corresponding to the structured query statement indicates that it is correct, and the metadata information indicates that it is correct, then the structured query statement indicates that it is correct; When at least one of the prompt information corresponding to the structured query statement or the metadata information indicates an error, the structured query statement indicates an error; Acquire the number of the structured query statements that are correct and the number of the structured query statements that are incorrect; The first dimension evaluation result is determined according to the number of the structured query statements that are correct and the number of the structured query statements that are incorrect.

5. The method according to claim 2, characterized in that: Determining the second dimension evaluation result according to the evaluation result of each element information in the prediction problem set includes: For each meta-information similarity result of the structured query statement, when at least one vector similarity exists that is greater than a preset similarity threshold, it indicates that the element information meets the constraint condition; When there is no vector whose similarity is greater than a preset similarity threshold, it indicates that the element information does not meet the constraint condition; Obtaining the number of element information corresponding to the prediction problem set that meets the constraint condition and the number of element information corresponding to the prediction problem set that does not meet the constraint condition; The second dimension evaluation result is determined according to the number of the element information that indicates compliance with the constraint condition and the number of the element information that indicates non-compliance with the constraint condition.

6. The method according to claim 2, characterized in that The step of determining the evaluation result of the large model to be tested based on the evaluation result of the first dimension and the evaluation result of the second dimension includes: Obtain a first weight of the evaluation result of the first dimension and a second weight of the evaluation result of the second dimension; Determining a target evaluation value according to the first dimension evaluation result, the first weight, the second dimension evaluation result, and the second weight; When the target evaluation value is greater than a preset evaluation threshold, the evaluation result of the large model to be tested is passed.

7. A model evaluation device, characterized in that: The device comprises: An acquisition module, used for acquiring a prediction problem set, wherein the prediction problem set includes a plurality of prediction problems; each of the prediction problems corresponds to reference information; An input-output module, used to input each prediction problem into the large model to be tested, and obtain a structured query statement corresponding to the prediction problem output by the large model to be tested; The determination module is used to extract element information of each structured query statement, and determine the evaluation result of the large model to be tested based on the element information and the reference information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images