Model evaluation method and device, computer equipment, storage medium and program product

Abstract

The invention relates to the field of data processing, text generation technologies, large model technologies and large language model technologies, and discloses a model evaluation method and device, computer equipment, a storage medium and a program product.The model evaluation method comprises the steps that a first query statement generated by a to-be-evaluated model language model is obtained; performing exception analysis on a target field matched with a preset evaluation index in the first query statement to obtain exception information difference information; and performing clustering analysis on the abnormal information difference information based on the preset abnormal type and the preset difference type to obtain an evaluation result of the language model of the to-be-evaluated model. According to the method and the device, the difference information of the first query statement generated by the language model can be subjected to clustering analysis based on the preset difference type to obtain the evaluation result corresponding to the language model, so that the carrying of the language model can be known through the evaluation result, and the iteration efficiency and accuracy of the language model are improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of data processing, text generation technology, large model technology, and large language model technology, and specifically to model evaluation methods, devices, computer equipment, storage media, and program products. Background Art

[0002] With the development of big language models, Text2SQL has become one of the most popular application areas. Specifically, Text2SQL can convert natural language text into structured query language (SQL). Therefore, how to analyze and summarize when the big language model generates incorrect content is crucial for the rapid iteration of the big language model.

[0003] In relevant large language model evaluation schemes, a score or an AUC (Area Under Curve) curve is usually given to the output result of the large language model to evaluate the confidence of the large language model. However, due to the lack of error evaluation of the large language model, it is impossible to provide good guidance for the iteration of the large language model, which affects the iteration efficiency and accuracy of the large language model. Summary of the invention

[0004] In view of this, the present disclosure provides a model evaluation method, apparatus, computer device, storage medium and program product to solve the problem of lack of erroneous evaluation of large language models when giving confidence in large model output results.

[0005] In a first aspect, the present disclosure provides a model evaluation method, the method comprising:

[0006] Acquire a first query sentence generated by the language model, wherein the first query sentence is used to represent a structured language for operating data determined by the language model based on an input natural language;

[0007] Analyze the target field in the first query statement that matches the preset evaluation index to obtain difference information;

[0008] The difference information is clustered and analyzed based on preset difference types to obtain an evaluation result of the language model.

[0009] In a second aspect, the present disclosure provides a model evaluation device, the device comprising:

[0010] An acquisition module, configured to acquire a first query statement generated by a language model, wherein the query statement is a statement for operating data determined based on a natural language input into the language model;

[0011] An analysis module, used to perform an anomaly analysis on a target field in the first query statement that matches a preset evaluation index to obtain difference information;

[0012] An evaluation module is used to perform cluster analysis on the difference information based on preset difference types to obtain an evaluation result of the language model.

[0013] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the model evaluation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0014] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the model evaluation method of the first aspect or any corresponding embodiment thereof.

[0015] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the model evaluation method of the first aspect or any corresponding embodiment thereof.

[0016] In an embodiment of the present disclosure, first, a first query statement generated by a language model can be obtained, wherein the first query statement is used to characterize a structured language determined by the language model to operate on data based on an input natural language. Next, the target field in the first query statement that matches a preset evaluation index can be analyzed to obtain difference information. Then, the difference information can be clustered based on a preset difference type to obtain an evaluation result of the language model, so that the difference information of the first query statement generated by the language model can be clustered based on a preset difference type to obtain an evaluation result corresponding to the language model, and then the language model can be known through the evaluation result to improve the iteration efficiency and accuracy of the language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a flowchart of a model evaluation method according to an embodiment of the present disclosure;

[0019] Figure 2is a flowchart of another model evaluation method according to an embodiment of the present disclosure;

[0020] Figure 3 is a flowchart of another model evaluation method according to an embodiment of the present disclosure;

[0021] Figure 4 is a visual graph of evaluation results according to an embodiment of the present disclosure;

[0022] Figure 5 is an architectural diagram of a model evaluation system according to an embodiment of the present disclosure;

[0023] Figure 6 is a structural block diagram of a model evaluation device according to an embodiment of the present disclosure;

[0024] Figure 7 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0026] In conjunction with the application scenarios on which the execution of the model evaluation method depends, the application scenarios are described here.

[0027] With the development of big language models, Text2SQL has become one of the most popular application areas. Specifically, Text2SQL can convert natural language text into structured query language (SQL). Therefore, how to analyze and summarize when the big language model generates incorrect content is crucial for the rapid iteration of the big language model.

[0028] In relevant large language model evaluation schemes, a score or an AUC (Area Under Curve) curve is usually given to the output result of the large language model to evaluate the confidence of the large language model. However, due to the lack of error evaluation of the large language model, it is impossible to provide good guidance for the iteration of the large language model, which affects the iteration efficiency and accuracy of the large language model.

[0029] For example, the language model can be evaluated using evaluation methods based on content matching (ExactSetMatch) and based on execution result matching (ExecuteMatch).

[0030] Specifically, Exact Set Match is an exact matching algorithm used to evaluate the similarity between two texts to output a score or AUC curve used to characterize the similarity. In natural language processing, it is often used to evaluate the degree of match between model-generated text and reference text to judge the quality of the generated text. However, it is too strict and requires the generated text to be exactly the same as the reference text, which may ignore semantic similarities or partially correct situations. In addition, for long texts or complex tasks, exact matching may be difficult to achieve, and there is a lack of error evaluation of large language models.

[0031] In addition, Execute Match allows the language model to generate code snippets, then execute the generated code in the actual environment to verify its correctness, and output a score or AUC curve to characterize the similarity. Executing the code may require additional computing resources and time, especially for complex code snippets. In addition, some programming tasks may be difficult to automate or verify, such as code involving user interaction or specific hardware environments, and lack of error evaluation of large language models.

[0032] Based on this, the embodiment of the present disclosure provides a model evaluation method, which first obtains a first query statement generated by a language model, wherein the first query statement is used to characterize a structured language for operating data determined by the language model based on the input natural language. Next, the target field in the first query statement that matches the preset evaluation index can be analyzed to obtain difference information. Then, the difference information can be clustered based on the preset difference type to obtain the evaluation result of the language model, so that the difference information of the first query statement generated by the language model can be clustered based on the preset difference type to obtain the evaluation result corresponding to the language model, and then the language model can be known through the evaluation result to improve the iteration efficiency and accuracy of the language model.

[0033] According to an embodiment of the present disclosure, a model evaluation method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0035] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0036] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0037] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0038] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0039] According to an embodiment of the present disclosure, a model evaluation method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] In this embodiment, a model evaluation method is provided, which can be used in a computer system. Figure 1 is a flow chart of a model evaluation method according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:

[0041] Step S101 : obtaining a first query sentence generated by a language model, wherein the first query sentence is used to represent a structured language for operating data determined by the language model based on an input natural language.

[0042] In the disclosed embodiment, the language model may be the above-mentioned large language model, which can learn the grammar, semantics and contextual relationship of the language, so as to be able to perform various complex natural language processing tasks. Here, the large language model can convert the natural language input by the user into a query statement, such as the above-mentioned SQL statement, to simplify the database query process, improve data analysis efficiency and enhance user experience.

[0043] Taking into account that when evaluating a large language model, it is more complicated and costly to evaluate the execution results of the SQL query statement, the present application can directly generate a first query statement predictionSQL based on the input natural language for error analysis to reduce the cost and complexity of model evaluation.

[0044] Step S102: Analyze the target field in the first query statement that matches the preset evaluation index to obtain difference information.

[0045] In the disclosed embodiment, the preset evaluation index may be a pre-set test point, i.e., a node in the SQL statement that needs to be examined. Specifically, the user may set the corresponding test point according to the business scenario of the language model. For example, the test point may include statistical functions, conditional filtering, sorting, etc. The target field may be a node in the first query statement that matches the preset evaluation index, for example, DESC (descending) and ASC (ascending) corresponding to the conditional filtering. The difference information may be used to indicate the error field content in the first query statement.

[0046] When analyzing the target field, preset reference data, ie, correct field content matching the target field, may be obtained, and a difference analysis may be performed between the correct field and the target field to obtain difference information between the two.

[0047] Step S103: performing cluster analysis on the difference information based on preset difference types to obtain an evaluation result of the language model.

[0048] In the embodiment of the present disclosure, the preset difference type may be an error type set by the user according to the business scenario of the language model, for example, a statement content error and a table selection exception, etc. The statement content error may be used to indicate that there are errors or omissions in the first query statement generated by the language model, and the table selection exception may be used to indicate that there are errors in the data table searched by the first query statement.

[0049] When clustering analysis is performed on the difference information based on the preset difference types, the difference information may be clustered into corresponding preset difference types, and analysis is performed based on the difference information under each preset difference type to obtain an evaluation result of the language model.

[0050] Specifically, statistics can be performed based on the number of difference information items under each preset difference type to determine the factor that has a greater impact on the language model. For example, if the number of difference information items corresponding to the sorting in the preset difference type is the largest, the sorting is determined to be a factor that has a greater impact on the language model, and an evaluation result is generated based on the factor to guide the iteration of the language model based on the evaluation result.

[0051] It can be seen from the above description that in the embodiment of the present disclosure, first, the first query statement generated by the language model can be obtained, wherein the first query statement is used to characterize the structured language determined by the language model based on the input natural language to operate on the data. Next, the target field in the first query statement that matches the preset evaluation index can be analyzed to obtain difference information. Then, the difference information can be clustered based on the preset difference type to obtain the evaluation result of the language model, so that the difference information of the first query statement generated by the language model can be clustered based on the preset difference type to obtain the evaluation result corresponding to the language model, and then the language model can be known through the evaluation result to improve the iteration efficiency and accuracy of the language model.

[0052] In this embodiment, another model evaluation method is provided, which can be used in a computer system. Figure 2 is a flow chart of another model evaluation method according to an embodiment of the present disclosure. Figure 2 As shown, the process includes the following steps:

[0053] Step S201, obtaining a first query sentence generated by a language model, wherein the first query sentence is used to represent a structured language determined by the language model based on an input natural language to operate on data. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0054] Step S202: Analyze the target field in the first query statement that matches the preset evaluation index to obtain difference information.

[0055] Specifically, the above step S202 includes:

[0056] Step S2021, obtaining a second query sentence corresponding to the language model, wherein the second query sentence is used to represent a structured language that matches the natural language.

[0057] Step S2022: Mark the second query statement and the first query statement based on a preset evaluation index to obtain target fields corresponding to the second query statement and the first query statement.

[0058] Step S2023, compare the target fields to obtain difference information.

[0059] In the disclosed embodiment, the second query statement ground truthSQL may be a preset reference statement that matches the natural language in the input language model. Here, the target fields in the second query statement and the first query statement may be marked based on the test points corresponding to the preset evaluation indicators to compare the target fields between the two to obtain difference information. Among them, the first query statement and the second query statement may contain at least one target field corresponding to the preset evaluation indicator. It should be understood that the second query statement may also be marked in advance to facilitate direct acquisition of the target field in the second query statement.

[0060] Specifically, the above step S2022, marking the second query statement and the first query statement based on the preset evaluation index to obtain the target fields corresponding to the second query statement and the first query statement, includes:

[0061] Step a1: Generate a corresponding first abstract syntax tree based on the statement structure of the first query statement, and obtain a second abstract syntax tree corresponding to the second query statement.

[0062] Step a2: marking a first target structure corresponding to a preset evaluation indicator in the first abstract syntax tree, and marking a first target field in the first query statement based on the first target structure.

[0063] Step a3, marking a second target structure corresponding to a preset evaluation indicator in the second abstract syntax tree, and marking a second target field in the second query statement based on the second target structure, wherein the target field includes a first target field and a second target field.

[0064] In the embodiments of the present disclosure, an abstract syntax tree is a tree-like data structure that can be used to identify the grammatical structure of a query statement. For example, the abstract syntax tree can be an AST (Abstract Syntax Tree), which can present the various parts of the query statement in a logically clear and easy-to-process manner through a structured and hierarchical representation method, thereby facilitating the search and identification of key content corresponding to preset evaluation indicators based on the abstract syntax tree. By marking the target field through the abstract syntax tree, when marking the target structure corresponding to the preset evaluation indicator in the abstract syntax tree, it can be matched layer by layer based on the structure of the abstract syntax tree to obtain a target structure that matches the preset evaluation indicator. The specific abstract syntax tree structure matching method is subject to what can be implemented, and is not specifically limited in the present disclosure.

[0065] Here, the above step a2, marking the first target structure corresponding to the preset evaluation indicator in the first abstract syntax tree, includes:

[0066] (1) Determine a target evaluation indicator that matches the statement type of the first query statement from among the preset evaluation indicators.

[0067] (2) Marking a first target structure that matches the target evaluation indicator in the first abstract syntax tree.

[0068] In an embodiment of the present disclosure, the statement type may be used to indicate the node type corresponding to each node in the first query statement, wherein during the SQL parsing and execution process, the query statement will be split into multiple nodes, each node representing a specific part of the query.

[0069] For example, the above statement types may include SelectNode, FromNode, WhereNode, GroupByNode, OrderByNode, LimitNode, JoinNode, SubqueryNode, etc. Here, SelectNode represents the SELECT clause, which defines the columns or expressions to be retrieved. FromNode represents the FROM clause, which specifies the source of the data, that is, the table or view. WhereNode represents the WHERE clause, which contains the filter conditions and is used to filter data. GroupByNode represents the GROUP BY clause, which is used to group the results. OrderByNode represents the ORDER BY clause, which is used to sort the results. LimitNode represents the LIMIT clause (called TOP in some databases), which is used to limit the number of results returned. JoinNode represents the JOIN clause, including different types of connections such as INNER JOIN, LEFT JOIN, and RIGHT JOIN. SubqueryNode represents a subquery, which can appear in clauses such as SELECT, FROM, WHERE, and HAVING.

[0070] Considering the large number of statement types, the user can pre-set the statement type that he wants to evaluate, so that during the model evaluation process, the first target structure in the first abstract syntax tree can be marked based on the target evaluation indicator that matches the statement type to achieve the evaluation of the statement type set by the user.

[0071] Specifically, the preset evaluation indicators can be divided into two dimensions: primary indicators and secondary indicators. Among them, the primary dimension can include: 'Visualization': ['Line chart'], 'Statistical function': ['Extreme value', 'Count', 'Average', 'Count difference', 'Sum', 'Aggregation indicator'], 'Sorting': ['Multi-dimensional sorting', 'ASC', 'DESC'], 'Time processing': ['Dynamic date', 'Clear time period', 'Clear date', 'Dynamic time period'], 'Condition screening': ['Comparison', 'Equal', 'Range'], 'Dimension grouping': [], 'TOPN': [], 'Subquery': [], 'Result calculation': ['Sum', 'YoY and YoY'], 'Knowledge': ["], 'Time field': ['Business time'], 'High cardinality dimension': [], 'Time aggregation': [].

[0072] In addition, the secondary indicators corresponding to the above-mentioned primary indicators may include: TOPN, sort-DESC, sort-ASC, sort-multi-dimensional sorting, time aggregation, dimension grouping, condition filtering-equal, condition filtering-comparison, condition filtering-range, high-base dimension, subquery, statistical function-mean, statistical function-count, statistical function-sum, statistical function-extreme value, statistical function-count difference, statistical function-aggregation indicator, knowledge-professional terms or jargon, visualization-line chart, result calculation-same-year and year-on-year, result calculation-sum, time processing-clear time period, time field-business time, time processing-clear date, time processing-dynamic time period, time processing-dynamic date.

[0073] After marking the target structure in the abstract syntax tree, corresponding target fields may be marked in the first query statement and the second query statement according to the target structure.

[0074] For example, the labeling result corresponding to the second query statement can be: SELECT`gender`,COUNT(`user_id`)as user_count FROM dataset WHERE`gender`IS NOT NULL GROUP BY`gender`. The second target field in the second query statement can be the gender corresponding to the preset evaluation indicator SELECT, the dataset corresponding to the preset evaluation indicator FROM, the `gender`IS NOT NULL GROUP BY`gender` corresponding to the preset evaluation indicator WHERE clause, and the preset evaluation indicator GROUP BY corresponding to gender.

[0075] Step S203: cluster the difference information based on the preset difference type to obtain the evaluation result of the language model. Figure 1Step S103 of the illustrated embodiment will not be described in detail here.

[0076] In the disclosed embodiment, the first abstract syntax tree corresponding to the first query statement and the second abstract syntax tree corresponding to the second query statement can be marked to obtain a target field, and the first query statement and the second query statement can be marked according to the target field, so that the query statement does not need to be identified by a language model or manually. Since the abstract syntax tree can present the various parts of the query statement in a logically clear and easy-to-process manner through a structured and hierarchical representation method, it is relatively easy to mark the abstract syntax tree with a high accuracy, while saving labor costs.

[0077] In some optional implementations, the above step S2023, comparing the target fields to obtain difference information, includes:

[0078] A difference comparison operation is performed on the first target field and the second target field to obtain difference information, wherein the difference comparison operation is used to find field missing, field mismatch, and field addition of the first query statement relative to the second query statement.

[0079] In the embodiment of the present disclosure, the above-mentioned difference comparison operation is a SQLDiff operation, which can be used to compare the differences of field missing, field mismatch and field addition between the first target field and the second target field to obtain difference information.

[0080] For example, if the tagging result of the second query statement is: SELECT `gender`, COUNT (`user_id`) as user_count FROM dataset WHERE `gender` IS NOT NULL GROUP BY `gender`, the tagging result of the first query statement corresponding to the second query statement is: SELECT gender, COUNT (user_id) AS user_count FROM your_table_name GROUP BY gender.

[0081] According to the SQLDiff operation, there is a field mismatch between the WHERE clauses in the first query statement and the second query statement. Therefore, the difference information SQL-Diff can be expressed as: {'where_info':{'is_match':False,'same':”,NULL','p_diff':”}}.

[0082] For another example, if the tagging result of the first query statement is: SELECT `organization id`, `organization name`, SUM (`store bound quantity`) as `store bound quantity sum` FROM dataset GROUP BY `organization id`, `organization name`. The tagging result of the second query statement is: SELECT `organization id`, `organization name`, SUM (`store bound quantity`) AS `store bound quantity sum` FROM data table GROUP BY `organization name`.

[0083] According to the SQLDiff operation, there is a missing field in the GROUP BY field in the first query statement and the second query statement, that is, the "organization id" is missing in the first query statement. Therefore, the difference information SQL-Diff can be expressed as: {'group_info':{'is_match':False,'same':'`Organization name`','p_diff':”}}.

[0084] In the disclosed embodiment, a difference comparison operation may be performed on the first target field and the second target field to obtain difference information between the first query statement and the second query statement to visualize the difference between the two, thereby improving the accuracy of the comparison between the two and the interpretability of the difference information.

[0085] In this embodiment, another model evaluation method is provided, which can be used in a computer system. Figure 3 is a flowchart of another model evaluation method according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:

[0086] Step S301, obtaining a first query sentence generated by a language model, wherein the first query sentence is used to represent a structured language determined by the language model based on an input natural language to operate on data. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0087] Step S302: Analyze the target fields in the first query statement that match the preset evaluation index to obtain difference information. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0088] Step S303: performing cluster analysis on the difference information based on the preset difference type to obtain an evaluation result of the language model.

[0089] Specifically, the above step S303 includes:

[0090] Step S3031: cluster the difference information based on preset difference types to obtain the current number of anomalies corresponding to each preset difference type.

[0091] Step S3032, based on the comparison between the current number of exceptions and the number of historical exceptions and the analysis of the comparison results, obtain the evaluation result of the language model.

[0092] In the embodiment of the present disclosure, the number of historical anomalies may be the number of anomalies contained in the evaluation results of the last one or several iterations of the language model. The preset difference types include: sentence content anomalies and table selection anomalies. The above step S3031 includes:

[0093] Step b1, based on the sentence content anomalies, respectively determine the number of sentence anomalies corresponding to sentence comprehension anomalies, sentence misselection, sentence multiple selection and sentence missing in the difference information.

[0094] Step b2: based on the table selection anomalies, determining the table selection anomaly numbers of the table field anomalies and the table selection anomalies in the difference information respectively, wherein the anomaly numbers include the statement anomaly numbers and the table selection anomaly numbers.

[0095] In the disclosed embodiments, statement understanding anomalies can be expressed as insufficient LLM understanding, statement misselection can be expressed as LLM misselection, statement multiple selection can be expressed as LLM multiple selection, and statement missing can be expressed as LLM missed selection. Table field anomalies can be expressed as Hallucination (hallucination error), and the number of table selection anomalies can be expressed as Table Selection Problem. Among them, Hallucination can be used to indicate that the data table selected by the first query statement is correct, but the queried field is wrong or the field does not exist in the data table, and Table Selection Problem can be used to indicate that the data table selected by the first query statement is incorrect.

[0096] It should be understood that the present disclosure can pre-set the association between the statement type corresponding to various difference information, the preset evaluation index and the preset difference type, so that the difference information can be classified based on the association. Here, the statement type can include the above-mentioned SelectNode, FromNode, WhereNode, GroupByNode, OrderByNode, LimitNode, JoinNode.

[0097] For example, the preset evaluation index covered by LimitNode can be the above-mentioned 'TOPN', and the corresponding error types can include the above-mentioned LLM insufficient understanding, LLM missed selection, and LLM multiple selection. Among them, when LLM is insufficiently understood, the second query statement is limit10 and the first query statement is limit1000. When LLM is missed, the second query statement is limit10 and the first query statement is pred. When LLM is multiple selected, the second query statement is gt: and the first query statement is pred:limit 100.

[0098] For another example, the preset evaluation indicators covered by OrderByNode may be the above-mentioned 'Sort-DESC', 'Sort-ASC', and 'Sort-Multi-dimensional Sort', and the error types may include the above-mentioned LLM insufficient understanding, LLM wrong selection, LLM missed selection, and LLM multiple selection. Among them, when LLM is insufficiently understood, the second query statement is order by id desc, and the first query statement is orderbyidasc. When LLM is wrongly selected, the second query statement is order by id desc, and the first query statement is orderbynameasc. When LLM is missed, the second query statement is order by id,name desc, and the first query statement is orderbyuserid. When LLM is multiple selected, the second query statement is order by id, and the first query statement is order byuseridasc,name.

[0099] For another example, the preset evaluation indicators covered by GroupByNode may be the above-mentioned 'time aggregation', 'dimensional grouping' + time category, and the error types may include the above-mentioned LLM insufficient understanding, LLM wrong selection, LLM missed selection and LLM multiple selection. Among them, when LLM is insufficiently understood, the second query statement is group by to_date(p_date), and the first query statement is group by p_date. When LLM is wrongly selected, the second query statement is group by p_date, and the first query statement is group byorder_id. When LLM is missed, the second query statement is group by date,order_id, and the first query statement is group by p_date. When LLM is multiple selected, the second query statement is group by date, and the first query statement is group byp_date,order_id.

[0100] For another example, the preset evaluation indicators covered by WhereNode may be the above-mentioned 'conditional screening-equal', 'conditional screening-comparison', 'conditional screening-range', high base dimension value + time category, and the error types may include the above-mentioned LLM insufficient understanding, LLM wrong selection, LLM missed selection and LLM multiple selection. Among them, when LLM is insufficiently understood, the second query statement is where order_id>100, and the first query statement is whereorder_id=100. When LLM is wrongly selected, the second query statement is where order_id>100, and the first query statement is whereuser_id=100. When LLM is missed, the second query statement is where order_id>100, and the first query statement is empty. When LLM is multiple selected, the second query statement is empty, and the first query statement is where order_id>100.

[0101] For another example, the preset evaluation indicators covered by FromNode and JoinNode may be undetermined, and the error types may include the above-mentioned Table Selection Problem, insufficient LLM understanding, wrong LLM selection, missed LLM selection, and multiple LLM selections. Among them, in the case of Table Selection Problem, the second query statement is from dataset, and the first query statement is fromtable. When LLM is insufficiently understood, the second query statement is JOIN users ONid=owneruserid, and the first query statement is JOINusersONid=userid. When LLM is wrongly selected, the second query statement is JOIN people ONid=owneruserid, and the first query statement is JOINusersONid=owneruserid. When LLM is missed, the second query statement is JOINbudgetONbudget_id=link_to_budgetJOINmemberONlink_to_member=member_id, and the first query statement is JOIN expenseONlink_to_member=member_id. When LLM is selected multiple times, the second query statement is JOINexpense ONlink_to_member=member_id, and the first query statement is JOIN budgetONbudget_id=link_to_budgetJOINmemberONlink_to_member=member_id.

[0102] For another example, the preset evaluation indicators covered by SelectNode may be subquery, statistical function-mean, statistical function-count, statistical function-sum, statistical function-extreme value, statistical function-count difference, statistical function-aggregation indicator, Hallucination, and the error types may include the above-mentioned LLM insufficient understanding, LLM wrong selection, LLM missed selection, and LLM multiple selection. Among them, when LLM is insufficiently understood, the second query statement is select sum(order_id), and the first query statement is selectcount(order_id). When LLM is wrongly selected, the second query statement is select sum(order_id), and the first query statement is select count(user_id). When LLM is missed, the second query statement is select sum(order_id), and the first query statement is empty. When LLM is multiple selected, the second query statement is empty, and the first query statement is select sum(order_id).

[0103] On this basis, the difference information corresponding to the preset difference types can be aggregated based on the above association relationship to obtain the number of difference information under each preset difference type. For example, if the above preset difference types include the above LLM insufficient understanding, LLM wrong selection, LLM multiple selection, LLM missing selection, Hallucination and Table Selection Problem.

[0104] For example, the aggregation result of aggregating the difference information corresponding to the preset difference type based on the above association relationship can be expressed as: 'Hallucination': 1, 'LLM multiple selection': 63, 'LLM missed selection': 143, 'LLM lack of understanding': 123, 'LLM wrong selection': 124, 'Table Selection Problem': 17.

[0105] In addition, on the basis of clustering the difference information based on the preset information type, the aggregation results can be further aggregated based on the above statement type dimension, and the aggregation results can be expressed as: 'FromNode': 2, 'GroupByNode': 17, 'Hallucination': 1, 'LimitNode': 32, 'OrderByNode': 36, 'SelectNode': 176, 'Table Selection Problem': 17, 'WhereNode': 190.

[0106] In addition, the above statement types can be further refined to obtain SQLThought dimensions, where the SQLThought dimensions can include Dimensions (dimension values), Metrics (metric values), and Filters (filter conditions). Here, Dimensions is the SelectNode in the above statement type, Metrics is the GroupByNode in the above statement type, and Filters is the WhereNode in the above statement type.

[0107] Based on the above SQLThought dimensions, the aggregation results are further aggregated, and the obtained aggregation results can be expressed as: 'Chronological Understanding': 28, 'Hallucination': 1, 'Metric Understanding Problem': 102, 'TOPN': 32, 'Table Selection Problem': 17, 'Sorting-ASC': 11, 'Sorting-DESC': 25, 'Time Aggregation': 1, 'Condition Filtering-Not Equal': 3, 'Condition Filtering-Comparison': 7, 'Condition Filtering-Equal': 51, 'Condition Filtering-Range': 31, 'Filtering Condition-Different': 64, 'Statistical Function-Other': 2, 'Statistical Function-Mean': 6, 'Statistical Function-Extreme Value': 6, 'Statistical Function-Sum': 20, 'Statistical Function-Aggregation Indicator': 17, 'Statistical Function-Count': 15, 'Statistical Function-Count Different': 13, 'Dimension Item Aggregation': 16, 'Join Table Error': 2, 'High Base Dimension Value': 1.

[0108] In the disclosed embodiment, the difference information can be clustered based on preset difference types, so as to perform attribution analysis on the error categories of the language model, so as to realize the visualization of the error evaluation of the language model, and help users to understand and analyze the evaluation results more intuitively.

[0109] In some optional implementations, the above step S3032, based on comparing the current number of anomalies with the historical number of anomalies and analyzing the comparison result, obtains the evaluation result of the language model, including:

[0110] Step c1, determining the number of historical anomalies based on the historical evaluation results corresponding to the language model.

[0111] Step c2, based on the comparison between the historical abnormal number and the current abnormal number, obtain the abnormal number change data corresponding to each preset difference type.

[0112] Step c3, determining the evaluation result of the language model based on the change data.

[0113] In the disclosed embodiment, the change data may be used to indicate the difference between the current number of anomalies and the historical number of anomalies corresponding to each preset difference type. For example, if the historical number of anomalies corresponding to the preset evaluation indicator LLM omission is 67 and the current number of anomalies is 45, then the change data may be -22, i.e., LLM omissions are reduced by 22.

[0114] On this basis, the evaluation result of the language model may include change data of each preset difference type. For example, the evaluation result may include: {'Hallucination':'0', 'LLM multiple selection':'-2 / 9', 'LLM missed selection':'-22 / 45', 'LLM lack of understanding':'-8 / 45', 'LLM wrong selection':'1 / 45', 'Table Selection Problem':'-2 / 15'}.

[0115] Next, the number of anomalies corresponding to each statement type in the evaluation results can be visualized, for example, the number of anomalies corresponding to each statement type can be converted into a columnar visualization. Figure 4 What is shown is a visual graph of the evaluation results, where the visual graph may include the number of exceptions corresponding to the above statement types SelectNode, FromNode, WhereNode, GroupByNode, OrderByNode, LimitNode, JoinNode, and SubqueryNode.

[0116] In an embodiment of the present disclosure, the evaluation result of the language model can be determined based on the change data of the number of anomalies corresponding to each preset difference type, so that the evaluation result can more intuitively observe the iterative effect of the language model based on the change in the number of anomalies in the first query statement output by the language model, so as to better guide the iteration of the language model based on the evaluation result.

[0117] In some optional implementations, the above step c3, determining the evaluation result of the language model based on the change data, includes:

[0118] (1) Based on the change data, a positive factor and a negative factor are determined in the preset difference type, wherein the number of abnormalities of the preset difference type corresponding to the positive factor increases positively, and the number of abnormalities of the preset difference type corresponding to the negative factor increases negatively.

[0119] (2) Analyze the abnormal parameters of the language model based on the forward factor and the reverse factor to obtain the evaluation results.

[0120] In the embodiment of the present disclosure, the maximum contributing forward factor may be determined in the forward factors, and the maximum contributing reverse factor may be determined in the reverse factors, wherein the maximum contributing factor is used to indicate the preset difference type with the largest change data.

[0121] For example, if the contribution of each factor corresponding to each preset difference type is {'Hallucination':'0', 'LLM multiple selection':'-2 / 9', 'LLM missed selection':'-22 / 45', 'LLM lack of understanding':'-8 / 45', 'LLM wrong selection':'1 / 45', 'Table Selection Problem':'-2 / 15'}, then the maximum contributing positive factor is 'LLM missed selection':'-22 / 45', and the maximum contributing negative factor is 'LLM wrong selection':'1 / 45'.

[0122] In addition, the absolute decrease value and relative decrease value of the number of anomalies of the current language model compared with the historical language model may also be determined. For example, the absolute decrease value may be 45, and the relative decrease value may be 9.55%.

[0123] On this basis, the evaluation result can be determined based on the clustering method of difference information, the absolute and relative decrease in the number of anomalies, the maximum contributing positive factor, the maximum contributing reverse factor and the contribution of each factor. The evaluation result can be expressed as: {'Aggregation method':'xxx','The number of anomalies in the current language model compared with the historical language model':'Absolute decrease of 45, relative decrease of 9.55%','Maximum contributing positive factor':{'LLM missed selection':'-22 / 45'},'Maximum contributing reverse factor':{'LLM wrong selection':'1 / 45'},'Contribution of each factor':{'Hallucination':'0','LLM multiple selection':'-2 / 9','LLM missed selection':'-22 / 45','LLM lack of understanding':'-8 / 45','LLM wrong selection':'1 / 45','Table SelectionProblem':'-2 / 15'}}.

[0124] In the embodiments of the present disclosure, Figure 5 The figure shows the architecture diagram of the model evaluation system, in which, firstly, the above statement types, SQLThought and preset evaluation indicators can be defined through the error type definition module. In addition, the above preset difference types can be defined, including sentence understanding anomalies, wrong statement selection, multiple statement selection, missing statement, table field anomalies and table selection anomalies.

[0125] Next, the SQLDiff acquisition module can be used to search for differences in the target structure of the abstract syntax tree SQLAST corresponding to the first query statement and the second query statement, and the error type marking module can be used to mark the difference information in the first query statement according to the search results. Then, the attribution analysis module can be used to perform cluster analysis on the difference information based on the preset difference type to obtain the evaluation results of the language model, and to perform contribution index analysis, while visually displaying the evaluation results.

[0126] In the disclosed embodiment, positive factors and reverse factors can be determined in preset difference types based on change data, and abnormal parameters of the language model can be analyzed based on the positive factors and reverse factors to obtain evaluation results, thereby providing more reasonable suggestions for the training of the language model.

[0127] In the present embodiment, a model evaluation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0128] This embodiment provides a model evaluation device, such as Figure 6 As shown, including:

[0129] An acquisition module 601 is used to acquire a first query statement generated by a language model, wherein the query statement is a statement for operating data determined based on a natural language input into the language model;

[0130] An analysis module 602 is used to perform an abnormality analysis on the target field in the first query statement that matches the preset evaluation index to obtain difference information;

[0131] The evaluation module 603 is used to perform cluster analysis on the difference information based on the preset difference type to obtain the evaluation result of the language model.

[0132] In some optional implementations, the analysis module 602 is further configured to:

[0133] Obtaining a second query sentence corresponding to the language model, wherein the second query sentence is used to represent a structured language that matches the natural language;

[0134] Marking the second query statement and the first query statement based on a preset evaluation index to obtain target fields corresponding to the second query statement and the first query statement;

[0135] Compare the target fields to get the difference information.

[0136] In some optional implementations, the analysis module 602 is further configured to:

[0137] Generate a corresponding first abstract syntax tree based on the statement structure of the first query statement, and obtain a second abstract syntax tree corresponding to the second query statement;

[0138] Marking a first target structure corresponding to a preset evaluation indicator in a first abstract syntax tree, and marking a first target field in a first query statement based on the first target structure;

[0139] A second target structure corresponding to a preset evaluation indicator is marked in the second abstract syntax tree, and a second target field is marked in the second query statement based on the second target structure, wherein the target field includes a first target field and a second target field.

[0140] In some optional implementations, the analysis module 602 is further configured to:

[0141] Determine a target evaluation indicator that matches the statement type of the first query statement from the preset evaluation indicators;

[0142] A first target structure matching the target evaluation indicator is marked in the first abstract syntax tree.

[0143] In some optional implementations, the analysis module 602 is further configured to:

[0144] A difference comparison operation is performed on the first target field and the second target field to obtain difference information, wherein the difference comparison operation is used to find field missing, field mismatch, and field addition of the first query statement relative to the second query statement.

[0145] In some optional implementations, the evaluation module 603 is further configured to:

[0146] Cluster the difference information based on the preset difference types to obtain the current number of anomalies corresponding to each preset difference type;

[0147] The current number of anomalies is compared with the historical number of anomalies and the comparison results are analyzed to obtain the evaluation results of the language model.

[0148] In some optional implementations, the preset difference types include: abnormal statement content and abnormal table selection, and the evaluation module 603 is further used to:

[0149] Based on the abnormalities in the sentence content, the number of sentence abnormalities corresponding to the sentence comprehension abnormalities, the sentence misselections, the sentence multiple selections, and the sentence missing in the difference information are determined respectively;

[0150] Based on the table selection exceptions, the table selection exception numbers of the table field exceptions and the table selection exceptions in the difference information are determined respectively, wherein the exception numbers include the statement exception number and the table selection exception number.

[0151] In some optional implementations, the evaluation module 603 is further configured to:

[0152] Determine the number of historical anomalies based on the historical evaluation results corresponding to the language model;

[0153] Based on the comparison between the historical abnormal number and the current abnormal number, the change data of the abnormal number corresponding to each preset difference type is obtained;

[0154] Based on the change data, an evaluation result of the language model is determined.

[0155] In some optional implementations, the evaluation module 603 is further configured to:

[0156] Determine a positive factor and a negative factor in the preset difference type based on the change data, wherein the number of abnormalities of the preset difference type corresponding to the positive factor increases positively, and the number of abnormalities of the preset difference type corresponding to the negative factor increases negatively;

[0157] The abnormal parameters of the language model are analyzed based on the forward and reverse factors to obtain the evaluation results.

[0158] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0159] The model evaluation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0160] The present disclosure also provides a computer device having the above Figure 6 Model evaluation apparatus shown.

[0161] See also Figure 7 , Figure 7 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0162] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0163] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0164] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0166] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0167] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0168] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0169] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0170] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0171] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0172] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0173] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A model evaluation method, characterized in that: The method comprises: Acquire a first query sentence generated by the language model, wherein the first query sentence is used to represent a structured language for operating data determined by the language model based on an input natural language; Analyze the target field in the first query statement that matches the preset evaluation index to obtain difference information; The difference information is clustered and analyzed based on preset difference types to obtain an evaluation result of the language model.

2. The method according to claim 1, characterized in that The abnormality analysis of the target field in the query statement that matches the preset evaluation index to obtain difference information includes: Acquire a second query statement corresponding to the language model, wherein the second query statement is used to represent a structured language matching the natural language; Marking the second query statement and the first query statement based on the preset evaluation index to obtain target fields corresponding to the second query statement and the first query statement; The target fields are compared to obtain the difference information.

3. The method according to claim 2, characterized in that The marking the second query statement and the first query statement based on the preset evaluation index to obtain the target fields corresponding to the second query statement and the first query statement includes: Generate a corresponding first abstract syntax tree based on the statement structure of the first query statement, and obtain a second abstract syntax tree corresponding to the second query statement; Marking a first target structure corresponding to the preset evaluation indicator in the first abstract syntax tree, and marking the first target field in the first query statement based on the first target structure; A second target structure corresponding to the preset evaluation indicator is marked in the second abstract syntax tree, and the second target field is marked in the second query statement based on the second target structure, wherein the target field includes the first target field and the second target field.

4. The method according to claim 3, characterized in that The marking of the first target structure corresponding to the preset evaluation indicator in the first abstract syntax tree includes: Determining a target evaluation indicator that matches the statement type of the first query statement from among the preset evaluation indicators; The first target structure matching the target evaluation indicator is marked in the first abstract syntax tree.

5. The method according to claim 3, characterized in that: The comparing the target fields to obtain the difference information includes: A difference comparison operation is performed on the first target field and the second target field to obtain the difference information, wherein the difference comparison operation is used to find field missing, field mismatch, and field addition of the first query statement relative to the second query statement.

6. The method according to claim 1, characterized in that The performing cluster analysis on the difference information based on the preset difference type to obtain the evaluation result of the language model includes: Clustering the difference information based on the preset difference types to obtain the current number of anomalies corresponding to each preset difference type; Based on comparing the current number of exceptions with the historical number of exceptions and analyzing the comparison results, an evaluation result of the language model is obtained.

7. The method according to claim 6, characterized in that The preset difference types include: abnormal statement content and abnormal table selection; The clustering of the difference information based on the preset difference types to obtain the number of exceptions corresponding to each preset difference type includes: Based on the abnormal content of the sentences, respectively determine the number of sentence abnormalities corresponding to the sentence comprehension abnormalities, sentence misselections, sentence multiple selections, and sentence missing in the difference information; Based on the table selection exceptions, the table selection exception numbers of the table field exceptions and the table selection exceptions in the difference information are determined respectively, wherein the exception numbers include the statement exception number and the table selection exception number.

8. The method according to claim 6, characterized in that The step of comparing the number of anomalies with the number of historical anomalies and analyzing the comparison result to obtain the evaluation result of the language model includes: Determining the number of historical anomalies based on historical evaluation results corresponding to the language model; Based on the comparison between the historical abnormal number and the current abnormal number, the change data of the abnormal number corresponding to each preset difference type is obtained; Based on the change data, an evaluation result of the language model is determined.

9. The method according to claim 8, characterized in that The determining, based on the change data, an evaluation result of the language model includes: Determine a positive factor and a negative factor in the preset difference type based on the change data, wherein the number of abnormalities of the preset difference type corresponding to the positive factor increases positively, and the number of abnormalities of the preset difference type corresponding to the negative factor increases negatively; The abnormal parameters of the language model are analyzed based on the forward factors and the reverse factors to obtain the evaluation result.

10. A model evaluation device, characterized in that: The device comprises: An acquisition module, configured to acquire a first query statement generated by a language model, wherein the query statement is a statement for operating data determined based on a natural language input into the language model; An analysis module, used to perform an anomaly analysis on a target field in the first query statement that matches a preset evaluation index to obtain difference information; An evaluation module is used to perform cluster analysis on the difference information based on preset difference types to obtain an evaluation result of the language model.

11. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model evaluation method according to any one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the model evaluation method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the model evaluation method according to any one of claims 1 to 9.

Images