Model evaluation method and device, computer equipment, storage medium and program product

Abstract

The invention relates to the field of data processing, text generation technologies, large model technologies and large language model technologies, and discloses a model evaluation method and device, computer equipment, a storage medium and a program product.The model evaluation method comprises the steps that a first query statement and a second query statement corresponding to a language model are obtained; comparing the first query statement with the second query statement based on preset evaluation dimensions to obtain a matching result corresponding to each preset evaluation dimension; and based on the matching result, determining an evaluation result corresponding to the language model. According to the method, an execution environment for the first query language does not need to be constructed, the evaluation result of the language model is determined according to the matching result of the first query language and the second query language of the language model, the evaluation result can be quantified and is more visual, the accuracy of the evaluation result and the evaluation efficiency are improved, and the evaluation cost is reduced.

Description

Technical Field

[0001] The present disclosure relates to the fields of data processing, text generation technology, large model technology, and large language model technology, and specifically to model evaluation methods, devices, computer equipment, storage media, and program products. Background Art

[0002] In the field of data processing, query processing for databases is usually involved. Among them, database queries are generally based on database structured language (Structured Query Language, hereinafter referred to as SQL), but not all users have the ability to write SQL. Therefore, a language model that can convert natural language into SQL has emerged. As a result, the conversion accuracy of the model affects the accuracy of data processing. Therefore, the evaluation of the model has become an urgent problem to be solved. Summary of the invention

[0003] In view of this, the present disclosure provides a model evaluation method, apparatus, computer device, storage medium and program product to solve the problem of how to improve the accuracy of data processing of a language model.

[0004] In a first aspect, the present disclosure provides a model evaluation method, the method comprising:

[0005] Obtaining a first query statement and a second query statement corresponding to the language model, wherein the first query statement is used to represent a structured statement generated by the language model based on an input natural language, and the second query statement is used to represent a structured statement matching the natural language;

[0006] Compare the first query statement with the second query statement based on the preset evaluation dimensions to obtain matching results corresponding to each preset evaluation dimension;

[0007] Based on the matching results, an evaluation result corresponding to the language model is determined.

[0008] In the disclosed embodiment, the first query statement and the second query statement corresponding to the language model can be obtained first, wherein the first query statement is used to characterize the structured statement generated by the language model based on the input natural language, and the second query statement is used to characterize the structured statement matching the natural language. Then, the first query statement and the second query statement can be compared based on the preset evaluation dimension to obtain the matching results corresponding to each preset evaluation dimension, and based on the matching results, the evaluation result corresponding to the language model is determined, so that there is no need to construct an execution environment for the first query language, but directly match the first query language and the second query language based on the language model to determine the evaluation result of the language model according to the matching result, so that the evaluation result can be quantified and more intuitive, which improves the accuracy of the evaluation result and the evaluation efficiency, and reduces the evaluation cost.

[0009] In a second aspect, the present disclosure provides a model evaluation device, the device comprising:

[0010] An acquisition module, used to acquire a first query statement and a second query statement corresponding to the language model, wherein the first query statement is a statement for operating data determined based on a natural language input into the language model, and the second query statement is a target statement matching the natural language;

[0011] A comparison module, used to compare the first query statement with the second query statement based on preset evaluation dimensions to obtain matching results corresponding to each preset evaluation dimension;

[0012] The determination module is used to determine the evaluation result corresponding to the language model based on the matching result.

[0013] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the model evaluation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0014] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the model evaluation method of the first aspect or any corresponding embodiment thereof.

[0015] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the model evaluation method of the first aspect or any corresponding embodiment thereof.

[0016] In the disclosed embodiment, the first query statement and the second query statement corresponding to the language model can be obtained first, wherein the first query statement is used to characterize the structured statement generated by the language model based on the input natural language, and the second query statement is used to characterize the structured statement matching the natural language. Then, the first query statement and the second query statement can be compared based on the preset evaluation dimension to obtain the matching results corresponding to each preset evaluation dimension, and based on the matching results, the evaluation result corresponding to the language model is determined, so that there is no need to construct an execution environment for the first query language, but directly match the first query language and the second query language based on the language model to determine the evaluation result of the language model according to the matching result, so that the evaluation result can be quantified and more intuitive, which improves the accuracy of the evaluation result and the evaluation efficiency, and reduces the evaluation cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a flow chart of a model evaluation method according to an embodiment of the present disclosure;

[0019] Figure 2 is a flowchart of another model evaluation method according to an embodiment of the present disclosure;

[0020] Figure 3 is an architecture diagram of a language model evaluation system according to an embodiment of the present disclosure;

[0021] Figure 4 is a structural block diagram of a model evaluation device according to an embodiment of the present disclosure;

[0022] Figure 5 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0024] In conjunction with the application scenarios on which the execution of the model evaluation method depends, the application scenarios are described here.

[0025] In the field of data processing, query processing for databases is usually involved. Among them, database queries are generally based on database structured language (Structured Query Language, hereinafter referred to as SQL), but not all users have the ability to write SQL. Therefore, a model that can convert natural language into SQL has emerged. As a result, the conversion accuracy of the model affects the accuracy of data processing. Therefore, the evaluation of the model has become an urgent problem to be solved.

[0026] In the relevant language model evaluation scheme, it is usually necessary to create a specific, simulated or real execution environment for the language model, so as to execute the SQL statement of the language model based on the environment, obtain the execution result, and evaluate the accuracy of the language model according to the execution result. However, in actual usage scenarios, due to the uncertainty of data distribution, the diversity of tasks and fields, and the diversity of evaluation indicator selection, the construction difficulty and construction cost of the execution environment are relatively high. At the same time, when the execution environment of the language model is more complex, the evaluation accuracy of the constructed execution environment in each single environment is also poor, which cannot meet the user's requirements for accuracy, efficiency and low cost of the language model evaluation scheme.

[0027] Based on this, the present disclosure provides a model evaluation method, which can first obtain a first query statement and a second query statement corresponding to a language model, wherein the first query statement is used to characterize a structured statement generated by the language model based on an input natural language, and the second query statement is used to characterize a structured statement that matches the natural language. Then, the first query statement and the second query statement can be compared based on a preset evaluation dimension to obtain matching results corresponding to each preset evaluation dimension, and based on the matching results, the evaluation result corresponding to the language model can be determined, thereby eliminating the need to construct an execution environment for the first query language, and directly matching the first query language and the second query language based on the language model to determine the evaluation result of the language model based on the matching results, so that the evaluation result can be quantified and more intuitive, thereby improving the accuracy of the evaluation result and the evaluation efficiency, and reducing the evaluation cost.

[0028] According to an embodiment of the present disclosure, a model evaluation method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0030] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0031] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0032] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0033] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0034] According to an embodiment of the present disclosure, a model evaluation method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] In this embodiment, a model evaluation method is provided, which can be used in a computer system. Figure 1 is a flow chart of a model evaluation method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0036] Step S101, obtaining a first query statement and a second query statement corresponding to a language model, wherein the first query statement is used to represent a structured statement generated by the language model based on an input natural language, and the second query statement is used to represent a structured statement matching the natural language.

[0037] In the disclosed embodiment, the language model may be the above-mentioned large language model, which can learn the grammar, semantics and contextual relationship of the language, so as to be able to perform various complex natural language processing tasks. Here, the large language model can convert the natural language input by the user into a query statement, such as the above-mentioned SQL statement, to simplify the database query process, improve data analysis efficiency and enhance user experience.

[0038] Based on this, when evaluating a large language model, a first query statement predictionSQL can be generated based on the input natural language, and a correct second query statement ground truthSQL can be set for the natural language, so as to compare the first query statement with the second query statement to evaluate the accuracy of the first query statement output by the language model.

[0039] Step S102: compare the first query statement with the second query statement based on preset evaluation dimensions to obtain matching results corresponding to each preset evaluation dimension.

[0040] In the embodiments of the present disclosure, the preset evaluation dimensions may be pre-set test points, i.e., the content in the SQL statement that needs to be examined, such as whether the statement type is correct, whether the execution action corresponding to the statement is correct, etc. It should be understood that when setting the test points, they may be set according to the dimensions of the SQL statement, wherein the dimensions of the SQL statement may be used to group, filter, and sort the data, so as to better understand and analyze the data.

[0041] Here, the key contents corresponding to the test points in the first query statement and the second query statement can be marked, so as to compare the two according to the key contents, and determine the matching result according to the matching degree of the two under each preset evaluation dimension. For example, if the preset evaluation dimension includes the statement type, according to the marking result, it can be known that the statement type corresponding to the first query statement is select, and the statement type corresponding to the second query statement is create, then, under the preset evaluation dimension, the matching result of the two is mismatch.

[0042] Step S103: determining the evaluation result corresponding to the language model based on the matching result.

[0043] In the embodiment of the present disclosure, a comprehensive evaluation may be performed based on the matching results corresponding to each preset evaluation dimension to determine the evaluation result corresponding to the language model.

[0044] For example, the matching results may be weighted according to the weights corresponding to the preset evaluation dimensions to obtain the evaluation results. For example, the weight corresponding to the preset evaluation dimension 1 is 0.4, the weight corresponding to the preset evaluation dimension 2 is 0.3, and the weight corresponding to the preset evaluation dimension 3 is 0.3. Then, the matching degree corresponding to each preset evaluation dimension may be weighted and summed according to the weight value to obtain the evaluation result.

[0045] It can be known from the above description that in the embodiment of the present disclosure, the first query statement and the second query statement corresponding to the language model can be obtained first, wherein the first query statement is used to characterize the structured statement generated by the language model based on the input natural language, and the second query statement is used to characterize the structured statement matching the natural language. Then, the first query statement and the second query statement can be compared based on the preset evaluation dimension to obtain the matching results corresponding to each preset evaluation dimension, and based on the matching results, the evaluation result corresponding to the language model is determined, so that there is no need to construct an execution environment for the first query language, but directly match the first query language and the second query language based on the language model to determine the evaluation result of the language model according to the matching result, so that the evaluation result can be quantified and more intuitive, which improves the accuracy of the evaluation result and the evaluation efficiency, and reduces the evaluation cost.

[0046] In this embodiment, another model evaluation method is provided, which can be used in a computer system. Figure 2 is a flow chart of a model evaluation method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:

[0047] Step S201, obtaining a first query statement and a second query statement corresponding to a language model, wherein the first query statement is used to represent a structured statement generated by the language model based on an input natural language, and the second query statement is used to represent a structured statement matching the natural language. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0048] Step S202: compare the first query statement with the second query statement based on preset evaluation dimensions to obtain matching results corresponding to each preset evaluation dimension.

[0049] Specifically, the above step S202 includes:

[0050] Step S2021: Generate a first abstract syntax tree based on the sentence structure of the first query sentence.

[0051] Step S2022: Obtain a second abstract syntax tree corresponding to the second query statement.

[0052] Step S2023: compare the first abstract syntax tree with the second abstract syntax tree based on preset evaluation dimensions to obtain matching results corresponding to each preset evaluation dimension.

[0053] In the disclosed embodiments, an abstract syntax tree is a tree-like data structure that can be used to identify the syntax structure of an SQL statement. For example, the abstract syntax tree can be an AST (Abstract Syntax Tree), which can present various parts of an SQL statement in a logically clear and easy-to-process manner through a structured and hierarchical representation method, thereby facilitating searching and identifying key content corresponding to a preset evaluation dimension based on the abstract syntax tree.

[0054] Specifically, the above step S2023 compares the first abstract syntax tree with the second abstract syntax tree based on preset evaluation dimensions to obtain matching results corresponding to each preset evaluation dimension, including:

[0055] Step a1: Identify target content matching a preset evaluation dimension in the first abstract syntax tree and the second abstract syntax tree.

[0056] Step a2: Identify the sub-sentences corresponding to the preset evaluation dimensions in the first query statement and the second query statement based on the target content to obtain identification results.

[0057] Step a3: perform comparison based on the identification results to obtain matching results corresponding to each preset evaluation dimension.

[0058] In the embodiments of the present disclosure, the above-mentioned preset evaluation dimensions may include statement types, execution actions, execution purposes, and grammatical structures. Among them, statement types may include select, create, alter, drop, delete, insert, truncate, show, etc. Execution actions may be used to indicate key actions performed by SQL statements, such as conditional judgments, aggregate functions, column-to-row conversions, row-to-column conversions, window functions, string functions, time functions, regular expressions, etc. The execution purpose may be used to indicate the core intent of SQL statements, such as incremental comparison, conversion, column aggregation, row aggregation, year-on-year comparison, retention analysis, deduplication processing, calculation of average values, counting statistics, sum operations, and determination of maximum and minimum values. The grammatical structure may include: simple queries, complex structure construction, subquery use, grouping by group by, partitioning by partition by, data sorting (order), result restrictions (limit), and conditional filtering, etc.

[0059] Based on this, identifying the target content in the first abstract syntax tree and the second abstract syntax tree that matches the preset evaluation dimension can specifically include: respectively identifying the statement type, execution action, execution purpose and grammatical structure in the first abstract syntax tree and the second abstract syntax tree, and obtaining the target content containing at least one preset evaluation dimension.

[0060] Here, first, the target content in the first abstract syntax tree can be marked. Specifically, the keywords that match each preset dimension in the AST structure can be identified. For example, the keywords corresponding to the statement type can be select, create and other statement types. The keywords corresponding to the execution action can be specific action names, such as SUM for summation actions. The keywords for the execution purpose can be clauses related to the core intent, such as where clauses. The keywords corresponding to the grammatical structure can be clauses related to the structure, such as group by clauses.

[0061] After identifying the structure corresponding to the preset evaluation dimension in the first abstract syntax tree, the target content in the second abstract syntax tree can be identified, or a pre-identified second abstract syntax tree can be obtained, and the sub-statements corresponding to each preset evaluation dimension in the SQL statement can be identified according to the target contents of the two to obtain an identification result.

[0062] For example, if the natural language input into the language model is: What is the ratio of the total gross profit of the authorized stores in the East District in 2023 to the total gross profit of all authorized stores in the country, the first query statement corresponding to the natural language is: SELECT SUM(`gross profit`) FROM dataset WHERE `year` = '2023' AND `region` = 'East District', the identification result corresponding to the first query statement can be: {'statement type': ['select'], 'syntactic structure': ['group by', 'conditional filtering'], 'key action': ['aggregate function', 'simple function'], 'core intent': ['sum', 'data screening']}, where the execution action is the key action and the execution purpose is the core intent.

[0063] Specifically, we can identify the SELECT node in the first query statement and get the statement type test point as select; we can identify the SUM node and get the key action test points as aggregate function and simple function, and the core intention test point as summation; we can identify the WHERE node and get the core intention test point as data screening; we can identify the GROUP node and get the grammatical structure test point as group by.

[0064] Step S203: Determine the evaluation result corresponding to the language model based on the matching result.

[0065] In the disclosed embodiment, a first abstract syntax tree corresponding to the first query statement can be compared with a second abstract syntax tree corresponding to the second query statement to obtain matching results corresponding to the first query statement and the second query statement in each preset evaluation dimension, thereby eliminating the need to identify the query statement through a language model or manually. Since the abstract syntax tree can present the various parts of the SQL statement in a logically clear and easy-to-process manner through a structured and hierarchical representation method, it is relatively easy to identify the abstract syntax tree with high accuracy, while saving labor costs.

[0066] In some optional implementations, the above step S202, comparing the first query statement with the second query statement based on preset evaluation dimensions to obtain matching results corresponding to each preset evaluation dimension, further includes:

[0067] Step S11, obtaining the target recognition result corresponding to the second query statement in the preset evaluation dimension.

[0068] Step S12, searching for target recognition results based on the first query statement, and determining matching results according to the search results.

[0069] In the embodiments of the present disclosure, considering that in a SQL statement, the same preset evaluation dimension may correspond to multiple identification results, and users pay different attention to different identification results, the user can customize the weight corresponding to each identification result so that the evaluation result for the language model meets the user's model iteration requirements.

[0070] For example, for the purpose of executing the preset evaluation dimension, the identification result corresponding to the second query statement may include "sum", "data screening" and "ratio conversion". The user can set the most concerned target recognition result in the identification result, such as "ratio conversion", and set a weight for the target recognition result. For example, if the weight is set to 1, then if the first query statement does not contain "ratio conversion" or the result of "ratio conversion" does not match the second query statement, the matching degree can be directly determined to be 0.

[0071] In the embodiments of the present disclosure, considering that in an SQL statement, the same preset evaluation dimension may correspond to multiple identification results, and users pay different attention to different identification results, therefore, a target recognition result with higher user attention can be set in the second query statement to increase the weight corresponding to the target recognition result, thereby determining the matching result based on the search result for the target recognition result in the first query statement, so that the final evaluation result is more in line with the user's model iteration requirements.

[0072] In some optional implementations, the above step S103, based on the matching result, determines the evaluation result corresponding to the language model, including:

[0073] Based on the weights corresponding to each preset evaluation dimension, the matching results are fused to obtain the evaluation result corresponding to the language model.

[0074] In the embodiment of the present disclosure, the matching results can be weighted according to the weights corresponding to the preset evaluation dimensions to obtain the evaluation results. For example, the weight corresponding to the preset evaluation dimension 1 is 0.4, the weight corresponding to the preset evaluation dimension 2 is 0.3, and the weight corresponding to the preset evaluation dimension 3 is 0.3. Then, the matching degree corresponding to each preset evaluation dimension can be weighted and summed according to the weight value to obtain the evaluation result.

[0075] Specifically, when setting the weights corresponding to the preset evaluation dimensions, the corresponding weights can be set according to the user's attention to the different preset evaluation dimensions, and it should be understood that the sum of the weights corresponding to the preset evaluation dimensions is 1. For example, if the user pays more attention to the execution purpose, and the attention is relatively low because the statement type in the first query statement will hardly have errors, therefore, the weight of the statement type can be set to 0, the weight of the execution action can be set to 0.3, the weight of the execution purpose can be set to 0.4, and the weight of the grammatical structure can be set to 0.3.

[0076] Here, if Figure 3 The figure shows the architecture diagram of the language model evaluation system, wherein, firstly, a preset evaluation dimension can be defined in the test point classification module, and the preset evaluation dimension can include the above-mentioned statement type, execution action, execution purpose and grammatical structure. Then, in the marking module, based on the abstract syntax tree corresponding to the first query statement and the second query statement, the target content in the abstract syntax tree is identified according to the keywords in the natural language that match the preset evaluation dimension, so that the evaluation module can identify the first query statement according to the target content, and evaluate the language model according to the identification results in the first query statement and the second query statement, and obtain a multi-dimensional evaluation result based on taxonomy.

[0077] In the disclosed embodiment, the matching results can be weighted according to the weights corresponding to the preset evaluation dimensions to obtain the evaluation results, wherein the user can customize the weight of the matching results corresponding to each preset evaluation dimension according to the attention paid to each node in the SQL statement. For example, the weight of the matching results with higher attention is set higher, so that the evaluation results are more accurate to meet the user's model iteration requirements.

[0078] In some optional embodiments, the above Figure 1 The corresponding implementation method also includes:

[0079] Step S21: Based on the evaluation result, the error type corresponding to each preset evaluation dimension of the first query statement is analyzed.

[0080] Step S22: generating evaluation information based on the analysis result corresponding to the error type, wherein the evaluation information is used to guide model optimization of the language model.

[0081] In the embodiment of the present disclosure, the evaluation information may include an analysis of the error type appearing in the first query statement and the language model optimization direction corresponding to each error type. Here, the optimization direction corresponding to each error type may be pre-set, and the evaluation information may be generated by citing the optimization direction.

[0082] For example, if the preset evaluation dimensions include the above-mentioned statement type, execution action, execution purpose and grammatical structure. Here, when the statement type corresponding to the first query statement does not match the second query statement, the error type may be a statement type identification error. When the identification results of the statement type corresponding to the first query statement and the execution purpose corresponding to the second query statement are inconsistent, the error type may be a key intent understanding error. When the identification results of the execution actions corresponding to the first query statement and the second query statement are inconsistent, the error type may be a key action understanding error.

[0083] When the error type corresponding to the first query statement is determined, each error type may be analyzed to obtain a corresponding optimization direction. For example, when the error type is a grammatical structure recognition error, the optimization direction for the language model may include: improving semantic understanding ability, enhancing grammatical error correction ability, optimizing prompt information, etc.

[0084] In an embodiment of the present disclosure, the error types corresponding to the first query statement in each preset evaluation dimension can be analyzed based on the evaluation result to generate evaluation information based on the analysis results corresponding to the error types, thereby refining the evaluation results and providing the user with optimization directions until the iteration of the language model.

[0085] In the present embodiment, a model evaluation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0086] This embodiment provides a model evaluation device, such as Figure 4 As shown, including:

[0087] An acquisition module 401 is used to acquire a first query statement and a second query statement corresponding to the language model, wherein the first query statement is a statement for operating data determined based on a natural language input into the language model, and the second query statement is a target statement matching the natural language;

[0088] A comparison module 402 is used to compare the first query statement with the second query statement based on preset evaluation dimensions to obtain matching results corresponding to each preset evaluation dimension;

[0089] The determination module 403 is used to determine the evaluation result corresponding to the language model based on the matching result.

[0090] In some optional implementations, the comparison module 402 is further configured to:

[0091] Based on the statement structure of the first query statement, generate a first abstract syntax tree;

[0092] Obtain a second abstract syntax tree corresponding to the second query statement;

[0093] The first abstract syntax tree is compared with the second abstract syntax tree based on preset evaluation dimensions to obtain matching results corresponding to each preset evaluation dimension.

[0094] In some optional implementations, the comparison module 402 is further configured to:

[0095] Identify target content matching a preset evaluation dimension in the first abstract syntax tree and the second abstract syntax tree;

[0096] Based on the target content, the sub-sentences corresponding to the preset evaluation dimensions in the first query statement and the second query statement are identified to obtain an identification result;

[0097] Based on the identification results, comparison is performed to obtain matching results corresponding to each preset evaluation dimension.

[0098] In some optional implementations, the preset evaluation dimensions include: statement type, execution action, execution purpose, and grammatical structure; the comparison module 402 is further used to:

[0099] The statement types, execution actions, execution purposes, and grammatical structures in the first abstract syntax tree and the second abstract syntax tree are respectively identified to obtain target content including at least one preset evaluation dimension.

[0100] In some optional implementations, the comparison module 402 is further configured to:

[0101] Obtaining a target recognition result corresponding to the second query statement in a preset evaluation dimension;

[0102] The target recognition result is searched based on the first query statement, and a matching result is determined according to the search result.

[0103] In some optional implementations, the determination module 403 is further configured to:

[0104] Based on the weights corresponding to each preset evaluation dimension, the matching results are fused to obtain the evaluation result corresponding to the language model.

[0105] In some optional embodiments, the device is also used for:

[0106] Based on the evaluation result, analyzing the error types corresponding to the first query statement in each preset evaluation dimension;

[0107] Evaluation information is generated based on the analysis results corresponding to the error types, wherein the evaluation information is used to guide model optimization of the language model.

[0108] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0109] The model evaluation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0110] The present disclosure also provides a computer device having the above Figure 4 Model evaluation apparatus shown.

[0111] See also Figure 5 , Figure 5 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5A processor 10 is taken as an example.

[0112] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0113] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0114] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0116] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0117] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0118] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0119] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0120] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0121] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0122] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0123] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A model evaluation method, characterized in that: The method comprises: Obtaining a first query statement and a second query statement corresponding to the language model, wherein the first query statement is used to represent a structured statement generated by the language model based on an input natural language, and the second query statement is used to represent a structured statement matching the natural language; Comparing the first query statement with the second query statement based on preset evaluation dimensions to obtain matching results corresponding to each of the preset evaluation dimensions; Based on the matching result, an evaluation result corresponding to the language model is determined.

2. The method according to claim 1, characterized in that The comparing the first query statement with the second query statement based on the preset evaluation dimensions to obtain matching results corresponding to each of the preset evaluation dimensions includes: Based on the statement structure of the first query statement, generate a first abstract syntax tree; Obtaining a second abstract syntax tree corresponding to the second query statement; The first abstract syntax tree is compared with the second abstract syntax tree based on the preset evaluation dimensions to obtain matching results corresponding to the preset evaluation dimensions.

3. The method according to claim 2, characterized in that The comparing the first abstract syntax tree with the second abstract syntax tree based on the preset evaluation dimensions to obtain matching results corresponding to the preset evaluation dimensions includes: Identifying target content in the first abstract syntax tree and the second abstract syntax tree that matches the preset evaluation dimension; Based on the target content, sub-sentences corresponding to the preset evaluation dimensions in the first query statement and the second query statement are identified to obtain an identification result; A comparison is performed based on the identification results to obtain matching results corresponding to each of the preset evaluation dimensions.

4. The method according to claim 3, characterized in that: The preset evaluation dimensions include: statement type, execution action, execution purpose and grammatical structure; The identifying target content in the first abstract syntax tree and the second abstract syntax tree that matches the preset evaluation dimension includes: The statement types, execution actions, execution purposes, and grammatical structures in the first abstract syntax tree and the second abstract syntax tree are respectively identified to obtain target content including at least one of the preset evaluation dimensions.

5. The method according to claim 1, characterized in that The comparing the first query statement with the second query statement based on the preset evaluation dimension to obtain the matching results corresponding to each of the preset evaluation dimensions further includes: Obtaining a target recognition result corresponding to the second query statement in the preset evaluation dimension; The target recognition result is searched based on the first query statement, and the matching result is determined according to the search result.

6. The method according to claim 1, characterized in that The determining, based on the matching result, an evaluation result corresponding to the language model includes: Based on the weights corresponding to the preset evaluation dimensions, the matching results are fused to obtain the evaluation result corresponding to the language model.

7. The method according to claim 1, characterized in that The method further comprises: Based on the evaluation result, analyzing the error type corresponding to each of the preset evaluation dimensions of the first query statement; Evaluation information is generated based on the analysis result corresponding to the error type, wherein the evaluation information is used to guide model optimization of the language model.

8. A model evaluation device, characterized in that: The device comprises: An acquisition module, configured to acquire a first query statement and a second query statement corresponding to the language model, wherein the first query statement is a statement for operating data determined based on a natural language input into the language model, and the second query statement is a target statement matching the natural language; A comparison module, configured to compare the first query statement with the second query statement based on preset evaluation dimensions to obtain matching results corresponding to each of the preset evaluation dimensions; A determination module is used to determine an evaluation result corresponding to the language model based on the matching result.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model evaluation method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the model evaluation method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the model evaluation method according to any one of claims 1 to 7.

Images