Model performance analysis method, device and equipment

Abstract

The invention provides a model performance analysis method, device and equipment, relates to the field of artificial intelligence, in particular to the technical field of large model and natural language processing, and can be applied to the fields of intelligent transportation, meteorology and generative business intelligence. Comprising the steps of obtaining a to-be-analyzed data set; the to-be-analyzed data set comprises question information and actual query statements, and the actual query statements represent correct SQL statements corresponding to the question information; determining a predicted query statement and reply information corresponding to the question information through the to-be-analyzed model; predicting an SQL (Structured Query Language) statement obtained by converting the question information by the query statement characterization to-be-analyzed model, and replying the reply of the information characterization model to the question information; according to the actual query statement, the prediction query statement and the reply information, obtaining an analysis result of the to-be-analyzed model; the analysis result represents the data query effect of the to-be-analyzed model. The analysis precision of the model is improved by evaluating two aspects of SQL statements and reply information output by the model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and is applied to the fields of large models and natural language processing technology. It can be used in the fields of intelligent transportation, meteorology, and generative business intelligence (Business Intelligence), and in particular, relates to a method, device, and equipment for analyzing model performance. Background Art

[0002] With the development of artificial intelligence technology, big models are gradually being applied in various fields. Users can send questions to big models, and big models process the user's questions, get answers to the questions and reply to the users. For example, users can query data and ask about road operation status, vehicle traffic volume and other data.

[0003] In order to enable users to receive correct response information, it is necessary to evaluate the query performance of the big model to ensure that the big model can correctly understand the user's intention and feed back the correct data to the user. Summary of the invention

[0004] The present invention provides a method, device and equipment for analyzing model performance.

[0005] According to a first aspect of the present disclosure, a method for analyzing model performance is provided, comprising:

[0006] Acquire a data set to be analyzed; wherein the data set to be analyzed includes question information and actual query statements, the question information represents information used to instruct the model to be analyzed to perform data query, and the actual query statement represents a correct structured query statement corresponding to the question information;

[0007] Determine the predicted query statement and reply information corresponding to the question information through the model to be analyzed; wherein the predicted query statement represents the structured query statement obtained by converting the question information by the model to be analyzed, and the reply information represents the answer of the model to be analyzed to the question information;

[0008] According to the actual query statement, predicted query statement, and reply information corresponding to the question information, the performance analysis of the model to be analyzed is performed to obtain an analysis result; wherein the analysis result represents the effect of the data query of the model to be analyzed.

[0009] According to a second aspect of the present disclosure, there is provided a device for analyzing model performance, comprising:

[0010] An acquisition unit is used to acquire a data set to be analyzed; wherein the data set to be analyzed includes question information and actual query statements, the question information represents information used to instruct the model to be analyzed to perform data query, and the actual query statement represents a correct structured query statement corresponding to the question information;

[0011] A determination unit, configured to determine, by means of the model to be analyzed, a predicted query statement and reply information corresponding to the question information; wherein the predicted query statement represents a structured query statement obtained by converting the question information by the model to be analyzed, and the reply information represents a reply of the model to be analyzed to the question information;

[0012] The analysis unit is used to perform performance analysis on the model to be analyzed according to the actual query statement, predicted query statement, and reply information corresponding to the question information to obtain an analysis result; wherein the analysis result represents the effect of the data query of the model to be analyzed.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively coupled to the at least one processor;

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the present disclosure when executed by a processor.

[0019] According to the technology disclosed in the present invention, the analysis accuracy of the model query performance is improved, ensuring that the model can output accurate answers.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flow chart of a method for analyzing model performance provided according to an embodiment of the present disclosure;

[0023] Figure 2is a flow chart of a method for analyzing model performance provided according to an embodiment of the present disclosure;

[0024] Figure 3 is an analysis flow chart of a model to be analyzed provided according to an embodiment of the present disclosure;

[0025] Figure 4 is a structural block diagram of a model performance analysis device provided according to an embodiment of the present disclosure;

[0026] Figure 5 is a structural block diagram of a model performance analysis device provided according to an embodiment of the present disclosure;

[0027] Figure 6 is a block diagram of an electronic device for implementing the method for analyzing model performance according to an embodiment of the present disclosure;

[0028] Figure 7 The block diagram of an electronic device used to implement the method for analyzing model performance according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] GBI (Generative Business Intelligence) can be an intelligent agent deployed with a large query model. The query model is a large model for data query through human-computer interaction, that is, query can be performed through GBI. For example, a user can input query information through the intelligent agent, and the query model converts the query information in text form into a statement in the form of SQL (Structured Query Language). Then, through the SQL statement, data is queried from the preset database, and the query results are converted into text data as reply information, which is fed back to the user, thereby realizing the query process of natural language interaction.

[0031] GBI is widely used in natural language interactive query in intelligent transportation and other fields. For example, GBI can be used to query the operating status of toll stations and service areas, vehicle traffic, details and number of traffic accidents, etc.

[0032] In order to ensure that users can obtain accurate data, it is necessary to evaluate the GBI's large query model and analyze the model's data query performance. The current model evaluation method is to determine the response information output by the model for the preset question information, and then determine whether the data in the response information is correct. This evaluation method cannot determine whether there is a problem when the model converts the question information into SQL statements, and does not evaluate the response information in multiple dimensions. For example, there is no evaluation of whether the language organization of the response information is clear, and there is no evaluation of whether the semantics of the response information is related to the question information, etc., which affects the analysis accuracy of the model's query performance, and thus affects the user's query experience.

[0033] The present disclosure provides a method, device and equipment for analyzing model performance, which relate to the fields of large models and natural language processing technology in the field of artificial intelligence, and can be applied to the fields of intelligent transportation, meteorology, and generative business intelligence to improve the evaluation accuracy of the model.

[0034] It should be noted that the model in this embodiment is not a model for a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0035] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0036] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 1-Figure 7 The embodiment is further refined.

[0037] Figure 1 FIG. 1 is a flow chart of a method for analyzing model performance according to an embodiment of the present disclosure, which method can be performed by a device for analyzing model performance. Figure 1 As shown, the method comprises the following steps:

[0038] S101, obtaining a data set to be analyzed; wherein the data set to be analyzed includes question information and actual query statements, the question information represents information used to instruct the model to be analyzed to perform data query, and the actual query statement represents a correct structured query statement corresponding to the question information.

[0039] Exemplarily, a data set to be analyzed is pre-built, and the performance analysis of the model to be analyzed is performed through the data set to be analyzed. The model to be analyzed is a preset large model, that is, the model to be analyzed can query the required data for the user through human-computer interaction with the user. In this embodiment, the model structure of the model to be analyzed is not specifically limited.

[0040] The data set to be analyzed may include multiple question information, which can be collected or written in advance, and each question information corresponds to an actual query statement. The question information is data in text form, indicating what data the user wants to query, that is, the question information can instruct the model to be analyzed to perform data query. The actual query statement represents the correct structured query statement corresponding to the question information, that is, the correct SQL statement corresponding to each question information is determined in advance as the actual query statement.

[0041] Multiple question information can be manually pre-written, and the actual query statements corresponding to the question information can be manually determined. The question information needs to meet multiple conditions. For example, the question information needs to cover different query conditions within the scope of business requirements and simulate the query operations performed by real users as much as possible. The query method in the question information can include daily language and professional terms. For example, the question information in daily language can be "We want to know how many devices the monitoring system includes in total", and the question information in professional terms can be "Query the total number of device units deployed in the monitoring system". The query complexity of the sentence in the question information can include simple sentence patterns and complex sentence patterns, for example, the question information in a simple sentence pattern can be "How many devices are there in the monitoring system", and the question information in a complex sentence pattern can be "On the section from Shanhaiguan to the end boundary, how long has the PTZ camera manufactured by brand A and marked as in normal operation been running since it was put into use?"

[0042] Manually writing question information requires a lot of manpower and time. The parameter information that needs to be included in the question information can be pre-set. The parameter information can include parameter type and parameter value. The parameter information represents the information that needs to be included in the question information. The parameter type can include parameters such as time type, equipment type, location type, project type, system type, equipment status, etc. The parameter value is the specific content of the parameter in the question information. For example, the parameter value of the time type can include today, yesterday, weekend, weekday, last week, last quarter, last month, first half of the year, second half of the year, last year, and holidays; the parameter value of the equipment type can include gun-type cameras, speed dome cameras, etc.; the parameter value of the equipment status can include normal, faulty, etc.

[0043] A sentence generation model is pre-built, and the sentence generation model is a large model. The model structures of the sentence generation model and the model to be analyzed can be different. Parameter information such as pre-configured parameter types and parameter values ​​are input into the sentence generation model, and question information in natural language form is output based on a preset prompt word template. The question information output by the model can be used as candidate information, and the candidate information can be manually screened to select representative question information as the question information in the data set to be analyzed. In other words, in this embodiment, question information can be automatically generated through a large model, which reduces manpower and time, improves the comprehensiveness of the data set to be analyzed, and thereby improves the accuracy and efficiency of model analysis.

[0044] The preset prompt word template can be: "You are an expert in natural language question generation. Please generate query natural language descriptions for the following database tables as required. Please help me summarize 50 questions that meet the requirements. The questions need to include the following query requirements. No introduction or ending is required. Only the summary content is output:

[0045] Schema information description:

[0046] Weather station basic information table schema information: see the file weather_schema.txt, which is used to generate natural language questions to query weather or visibility data

[0047] Alarm table schema information: see the file visibility_schema, which needs to be combined with weather_schema to generate natural language questions for weather alarm data

[0048] Parameter information description:

[0049] Time usage: needs to cover today, yesterday, weekends, weekdays, last week, last quarter, last month, first half of the year, second half of the year, last year, and a specific time, such as 2023-06-2000:00:00. The time period is expressed as yyyy-MM-dd HH:mm:ss

[0050] Road section: see road_section.txt

[0051] Stake information: stake.txt

[0052] Query requirements: Please describe completely as follows. Do not use the description "Same as above, with changes..."

[0053] Query weather data based on road section, pile number or time. The road section pile number is randomly covered. For details, see the parameter information description section. The schema referenced in this section is weather_schema.txt

[0054] Take the road section as an example, how to query the weather data of section M to section N on XX Expressway?

[0055] Taking the pile number as an example, how to query the meteorological data of pile number K84+300 on XX Expressway?

[0056] Other meteorological data or visibility query natural language questions, combined with the table weather_schema.txt file random combination, parameter information can be randomly overwritten (including query parameters, condition parameters), if you need to use time, road section, pile number, see parameter information description for details

[0057] Query weather warning data, warning times, warning data distribution, etc. according to road section, pile number, time, and warning type. The warning type can be randomly selected: icy road surface, water on the road surface, snow on the road surface, slippery road surface, low visibility. For other queries, please use the normal human query dimensions, such as detailed data, distribution, and trend. The reference schemas for this part are weather_schema.txt and visibility_schema.txt

[0058] You can use groupby, orderby, distinct, having, limit, COUNT, SUM, AVG, etc.

[0059] You can use daily expressions, professional expressions, simple sentences, and complex sentences randomly.

[0060] The time filter is randomly increased, accounting for a maximum of 10%, and the content with time query is guaranteed to be different. You can also add queries for specific time periods, such as "2023-06-20 00:00:00".

[0061] S102. Determine the predicted query statement and reply information corresponding to the question information through the model to be analyzed; wherein the predicted query statement represents the structured query statement obtained by converting the question information by the model to be analyzed, and the reply information represents the reply of the model to be analyzed to the question information.

[0062] Exemplarily, the model to be analyzed can process the question information, determine the answer corresponding to the question information, and feed back the answer to the user as reply information. After obtaining the data set to be analyzed, the question information in the data set to be analyzed can be input into the model to be analyzed, and the model to be analyzed can query the data corresponding to the question information, that is, the query result, as reply information. In the process of querying data, the model to be analyzed can convert the question information into an SQL statement, and use the SQL statement converted by the model to be analyzed as a predicted query statement. The predicted query statement and the actual query statement may be the same or different.

[0063] The data set to be analyzed may include multiple question information, each of which may correspond to its own predicted query statement and reply information. In the data set to be analyzed, each question information also corresponds to its own actual query statement.

[0064] In this embodiment, the predicted query statement and reply information corresponding to the question information are determined through the model to be analyzed, including: inputting the question information into the model to be analyzed to obtain the predicted query statement corresponding to the question information; and obtaining the reply information output by the model to be analyzed based on the predicted query statement.

[0065] Specifically, the question information is input into the model to be analyzed. Each question information can be input into the model to be analyzed in sequence, or multiple question information can be input into the model to be analyzed in batches. For each question information, the model to be analyzed can convert the question information into an SQL statement as a prediction query statement corresponding to the question information. In this embodiment, the process of converting to obtain the SQL statement is not specifically limited. For example, the question information is "Please provide the road surface water thickness and snow thickness at the K35+911 pile number on XX Expressway, sorted by reporting time", and the prediction query statement output by the model to be analyzed can be "SELECT ntrc_check_water, ntrc_check_snow, alert_source, db_create_time, position, road_name FROM tpi_im_road_weather WHERE position = 'K35+911' and delete_flag = 0 ORDER BY db_creat_time".

[0066] The model to be analyzed can perform data query from a preset database according to the predicted query statement, and use the queried data as the data query result. According to the data query result, a reply message is generated. For example, the data query result can be organized into a natural language form, that is, a natural language is generated, the data query result exists in this natural language, and this natural language is used as the reply message.

[0067] The beneficial effect of such a setting is that by obtaining the SQL statements and response information output by the model to be analyzed, the model performance can be analyzed from both the SQL statements and the response information, thereby improving the accuracy of the model analysis.

[0068] S103. Perform performance analysis on the model to be analyzed according to the actual query statement, predicted query statement, and reply information corresponding to the question information to obtain analysis results; wherein the analysis results represent the effect of data query of the model to be analyzed.

[0069] Exemplarily, the data set to be analyzed includes the actual query statement corresponding to the question information, and the actual query statement is a correct SQL statement. That is, if the SQL conversion process of the model to be analyzed is correct, the predicted query statement output by the model to be analyzed should be consistent with the actual query statement. Therefore, the query performance of the model to be analyzed can be analyzed based on the actual query statement and the predicted query statement corresponding to the question information. If the two are consistent, the analysis result of the model to be analyzed is considered to be passed; if the two are inconsistent, the analysis result of the model to be analyzed is considered to be failed.

[0070] The model to be analyzed can also be analyzed based on the reply information. For example, semantic recognition can be performed on the reply information. If the semantics of the reply information are clear, the segmentation is clear, and the key points are highlighted, the analysis result of the model to be analyzed is considered to be passed; if the language organization is not clear enough and the language is ambiguous, the analysis result of the model to be analyzed is considered to be failed. In this embodiment, the model to be analyzed can be evaluated by a preset referee model. The preset referee model can also be a large model, and the model structure can be different from the model to be analyzed. The reply information can be semantically recognized and scored by the referee model. The clearer the semantics, the higher the score. If the score exceeds the preset score range, the semantics of the reply information is considered to be clear; if the score does not exceed the preset score range, the semantics of the reply information is considered to be unclear.

[0071] If the analysis is successful, it means that the data query effect of the model to be analyzed is good, and human-computer interaction can be performed with the user to query data for the user; if the analysis is unsuccessful, it means that the data query effect of the model to be analyzed is poor, and the model to be analyzed needs to be optimized. The analysis of the model to be analyzed can be considered successful only when the actual query statement and the predicted query statement are consistent, and the language organization of the reply information is clear. If the actual query statement and the predicted query statement are inconsistent, or the language organization of the reply information is unclear, the analysis of the model to be analyzed is considered unsuccessful.

[0072] In the disclosed embodiment, the preset question information and the corresponding actual query statement are obtained, the question information is input into the model to be analyzed, and the predicted query statement and reply information corresponding to the question information are obtained. The query performance of the model to be analyzed is analyzed by combining the actual query statement, the predicted query statement, and the reply information to obtain the analysis result. The model is evaluated in detail from the two aspects of SQL statement and reply information, which saves manpower and time, improves the analysis accuracy and efficiency of the model, and thus improves the user's data query experience.

[0073] Figure 2 A flowchart of a method for analyzing model performance provided in an embodiment of the present disclosure.

[0074] In this embodiment, a performance analysis is performed on the model to be analyzed based on the actual query statement, predicted query statement, and reply information corresponding to the question information to obtain analysis results, including: obtaining a first score of the model to be analyzed based on the actual query statement and predicted query statement corresponding to the question information; wherein the first score represents the accuracy of the model to be analyzed in converting the question information into a structured query statement; determining a second score of the model to be analyzed based on the reply information; wherein the second score represents the reply effect of the model to be analyzed on the question information; and obtaining an analysis result of the performance of the model to be analyzed based on the first score and the second score.

[0075] This embodiment is based on the above embodiment. Figure 2 As shown, the method comprises the following steps:

[0076] S201, obtaining a data set to be analyzed; wherein the data set to be analyzed includes question information and actual query statements, the question information represents information used to instruct the model to be analyzed to perform data query, and the actual query statement represents a correct structured query statement corresponding to the question information.

[0077] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0078] S202. Determine the predicted query statement and reply information corresponding to the question information through the model to be analyzed; wherein the predicted query statement represents the structured query statement obtained by converting the question information by the model to be analyzed, and the reply information represents the reply of the model to be analyzed to the question information.

[0079] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0080] S203. Obtain a first score of the model to be analyzed according to the actual query statement and the predicted query statement corresponding to the question information; wherein the first score represents the accuracy of the model to be analyzed in converting the question information into a structured query statement.

[0081] Exemplarily, after obtaining the predicted query statement corresponding to the question information through the model to be analyzed, the actual query statement corresponding to the question information is obtained. The predicted query statement and the actual query statement are compared, and a first score of the model to be analyzed is obtained according to the comparison result. The first score can represent the accuracy of the model to be analyzed in converting the question information into a structured query statement. The closer the predicted query statement is to the actual query statement, the more accurate the predicted query statement is.

[0082] The consistency between the predicted query and the actual query can be calculated. The higher the consistency between the predicted query and the actual query, the higher the first score. In this embodiment, the calculation method of the consistency is not specifically limited. For example, the characters at the same position in the predicted query and the actual query can be determined, and the number of identical characters at the same position divided by the total number of characters in the predicted query is used as the consistency between the predicted query and the actual query.

[0083] In this embodiment, a first score of the model to be analyzed is obtained based on the actual query statement and the predicted query statement corresponding to the question information, including: determining the field information in the predicted query statement, and determining the field information in the actual query statement; wherein the field information includes components and corresponding component values; matching the field information in the predicted query statement with the field information in the actual query statement to obtain the first score of the model to be analyzed.

[0084] Specifically, the predicted query statement and the actual query statement are parsed respectively, and multiple field information is parsed from the predicted query statement, and multiple field information is parsed from the actual query statement. Each field information may include a component and a corresponding component value. A component is a component in an SQL statement. For example, if the SQL statement is "where system_type=4", the component may be "where" and the corresponding component value may be 4. In an SQL statement, there are multiple preset components. For example, the components may include select column, select column without agg, where column, wherecolumn without op, group by, group by having, order by, and / or, iuen, sql key words, etc., among which agg may include none, max, min, count, sum, avg and other components; where ops may include not, between, =, >, <, >=, <=,! Components include =, in', like, is, exists, etc.; and / or can include and, or; iuen includes intersect, union, except; SQL keywords include where, group by, having, desc, asc, order by, limit, except, union, intersect, not in, in, or, like, etc.

[0085] During parsing, each time a preset component is identified, parsing is performed until the next component is identified. The content between two components can be used as the component value of the previous component of the two components.

[0086] Match the field information in the predicted query statement with the field information in the actual query statement, determine whether the field information in the predicted query statement is consistent with the field information in the actual query statement, and determine the first score of the model to be analyzed based on the consistency of the field information. The higher the consistency, the higher the first score. When comparing, the field information can be compared in sequence according to the order in which the field information appears in the SQL statement. For example, determine whether the components and component values ​​of the first field information in the predicted query statement are consistent with the components and component values ​​of the first field information in the actual query statement, until determining whether the components and component values ​​of the last field information in the predicted query statement are consistent with the components and component values ​​of the last field information in the actual query statement. The number of consistent field information can be divided by the total number of field information in the predicted query statement to serve as the first score.

[0087] The beneficial effect of this setting is that it matches the components in the actual SQL and the predicted SQL to determine whether the model to be analyzed can generate correct SQL statements, thereby analyzing the SQL generation capability of the model to be analyzed, ensuring that the model can query the data required by the user and improving the user's query experience.

[0088] In this embodiment, the field information in the predicted query statement is matched with the field information in the actual query statement to obtain the first score of the model to be analyzed, including: if the field information in the predicted query statement is the same as the field information in the actual query statement, then the first score of the model to be analyzed is determined to be a first preset score; if the field information in the predicted query statement is different from the field information in the actual query statement, then the first score of the model to be analyzed is determined to be a second preset score; wherein the first preset score is greater than the second preset score.

[0089] Specifically, it is determined whether each field information in the predicted query statement is exactly the same as the corresponding field information in the actual query statement. If so, the first score of the model to be analyzed is determined to be the first preset score; if not, the first score of the model to be analyzed is determined to be the second preset score. That is, as long as there is field information in the predicted query statement that is different from the corresponding field information in the actual query statement, no matter how many field information is different, and no matter whether the components are different or the component values ​​are different, the first score is the second preset score. The first preset score is greater than the second preset score. For example, the first preset score is 2 points and the second preset score is 0 points.

[0090] The beneficial effect of such a setting is that the predicted SQL is matched with all components and component values ​​in the actual SQL, thereby achieving a comprehensive evaluation of the SQL statement and improving the evaluation accuracy of the model's SQL generation capability.

[0091] In this embodiment, the field information in the predicted query statement is matched with the field information in the actual query statement to obtain the first score of the model to be analyzed, including: if the field information in the predicted query statement is different from the field information in the actual query statement, then the difference field in the predicted query statement and the difference field in the actual query statement are determined; wherein the difference field represents the different field information between the predicted query statement and the actual query statement; the type of the component in the difference field of the predicted query statement is determined to be a first type, and the type of the component in the difference field of the actual query statement is determined to be a second type; if the first type and the second type are both preset types, then the first score of the model to be analyzed is determined according to the component value in the difference field of the predicted query statement and the component value in the difference field of the actual query statement.

[0092] Specifically, if the field information in the predicted query statement is different from the corresponding field information in the actual query statement, the first score of the model to be analyzed may be further confirmed. That is, the first score of the model to be analyzed may not be the second preset score.

[0093] The field information in the predicted query statement is compared with the corresponding field information in the actual query statement one by one to determine the difference fields in the predicted query statement and the difference fields in the actual query statement. The difference fields refer to the different field information between the predicted query statement and the actual query statement. For example, after comparing the first field information in the predicted query statement with the first field information in the actual query statement, if a difference is found, the first field information in the predicted query statement is determined to be the difference field in the predicted query statement, and the first field information in the actual query statement is determined to be the difference field in the actual query statement.

[0094] For each difference field, determine the type of the component in the difference field. Determine the type of the component in the difference field of the predicted query statement as the first type, and determine the type of the component in the corresponding difference field of the actual query statement as the second type.

[0095] If the first type and the second type are inconsistent, the first score of the model to be analyzed is determined to be the second preset score; if the first type and the second type are consistent, but not the preset type, the first score of the model to be analyzed is determined to be the second preset score; if the first type and the second type are consistent and are the preset type, it is necessary to further determine the first score of the model to be analyzed based on the component value in the difference field of the predicted query statement and the component value in the difference field of the actual query statement. The preset type is one or more pre-set component types, for example, the preset type can be a select type or an order type.

[0096] If the first type and the second type are the same preset type, the component value is obtained from the difference field of the predicted query statement as the first component value, and the component value is obtained from the difference field of the actual query statement as the second component value. The first component value and the second component value are compared to determine the first score of the model to be analyzed. For example, the degree of consistency between the first component value and the second component value can be determined, and if the degree of consistency is equal to or greater than the preset value, the first score is determined to be the first preset score value; if the degree of consistency is less than the preset value, the first score is determined to be the second preset score value.

[0097] The beneficial effect of such a setting is that the difference fields between the actual SQL and the predicted SQL are determined. If the components in the difference fields are the same and of the preset type, the difference will be judged separately as a special case to avoid misjudgment of the predicted SQL statement and improve the analysis accuracy of the analysis model.

[0098] In this embodiment, the preset type is a select type; the first score of the model to be analyzed is determined based on the component value in the difference field of the predicted query statement and the component value in the difference field of the actual query statement, including: if the component value in the difference field of the predicted query statement exists in the component value in the difference field of the actual query statement, then the first score of the model to be analyzed is determined to be a first preset score.

[0099] Specifically, the preset type may be a select type. If both the first type and the second type are select types, it is determined whether the first component value exists in the second component value. If the first component value exists in the second component value, the first score of the model to be analyzed is determined to be the first preset score; if the first component value does not exist in the second component value, the first score of the model to be analyzed is determined to be the second preset score.

[0100] For example, if the question is "I need to know the wind direction information of the EF section on the XX highway last week, sorted by reporting time", the corresponding actual query statement is "SELECT wind_direction,db_create_time FROM tpi_im_road_weather WHERE DATE_SUB(CURDATE(),INTERVAL

[0101] WEEKDAY(CURDATE())+7DAY)<=DATE(db_create_time)AND DATE_SUB(CURDATE(),INTERVAL

[0102] WEEKDAY(CURDATE())+1DAY)>=DATE(db_create_time)AND road_name='E-F'AND delete_flag=0ORDER BY db_create_time", the prediction query statement is "SELECT alert_source,db_create_time,position,road_name,wind_directionFROM tpi_im_road_weatherWHEREroad_name='E-F'and DATE_SUB(CURDATE(),INTERVAL

[0103] WEEKDAY(CURDATE())+7DAY)<=db_create_time AND DATE_SUB(CURDATE(),INTERVALWEEKDAY(CURDATE())+1DAY)>=db_create_time and delete_flag=0order bydb_create_time". Among them, the component values ​​corresponding to the components of the select type are different. The component value corresponding to the select in the actual query statement is to query the wind direction information, while the component value corresponding to the select in the prediction query statement is to query the alarm source, creation time, location, road name, and wind direction information. That is, the first component value of the prediction query statement includes the second component value of the actual query statement. Although the first component value and the second component value are not exactly the same, it does not affect the query of the wind direction information. Therefore, it is considered that the prediction query statement is correct, and the first score of the model to be analyzed is determined as the first preset score.

[0104] The beneficial effect of such a setting is that the select type is judged as a special case, which avoids misjudgment of the first score, improves the diversity of SQL statements generated by the model, and effectively improves the analysis accuracy of the model.

[0105] In this embodiment, the preset type is the order type; the first score of the model to be analyzed is determined according to the component values ​​in the difference fields of the predicted query statement and the component values ​​in the difference fields of the actual query statement, including: if the difference between the component values ​​in the difference fields of the predicted query statement and the component values ​​in the difference fields of the actual query statement represents a sorting direction, then the first score of the model to be analyzed is determined to be the first preset score.

[0106] Specifically, the preset type may be an order type. If both the first type and the second type are order types, the difference between the first component value and the second component value is determined, that is, the difference between the first component value and the second component value is determined. If the content of the difference represents the sorting direction, the first score of the model to be analyzed is determined to be the first preset score, and the sorting direction may be ascending or descending; if the content of the difference includes other content except the sorting direction, the first score of the model to be analyzed is determined to be the second preset score.

[0107] For example, if the question is "Please provide the thickness of water and snow on the road at K35+911 on XX Expressway, sorted by reporting time", the corresponding actual query statement is "SELECT alert_source,db_create_time,position,road_name,ntrc_check_water,ntrc_check_snow FROM tpi_im_road_weatherWHERE position='K35+911'AND delete_flag=0ORDER BY db_creat_time ASC", and the predicted query statement is "SELECT ntrc_check_water,ntrc_check_snow,alert_source,db_create_time,position,road_name FROM tpi_im_road_weather WHERE position='K35+911'anddelete_flag=0ORDER BY db_creat_time". Among them, the component values ​​corresponding to the components of the order type are different. The component value corresponding to order in the actual query statement is "db_creat_time ASC", while the component value corresponding to order in the predicted query statement is "db_creat_time". That is, the difference between the first component value and the second component value is "ASC", ASC means ascending order, and no ASC means descending order. Sorting is performed in both ascending and descending order, so the predicted query statement is considered correct, and the first score of the model to be analyzed is determined as the first preset score. It is worth noting that if the sorting direction is not specified in the question information, the predicted query statement is considered correct when the difference represents the sorting direction. If the sorting direction is clearly indicated in the question information, the predicted query statement is considered to have a problem when the difference represents the sorting direction, and the first score is the second preset score.

[0108] The beneficial effect of this setting is that the order type is judged as a special case, which avoids misjudgment of the first score, improves the diversity of SQL statements generated by the model, and effectively improves the analysis accuracy of the model.

[0109] In this embodiment, the predicted query statement and the actual query statement can be input into the preset referee model, and the score of the predicted query statement can be output based on the preset prompt word, that is, the first score can be obtained. The preset prompt word can be "You are a professional SQL evaluator, score the accuracy of [model output SQL] based on [question] and [standard SQL]:

[0110] The grades are 0 and 2. Please refer to the grade description for the specific scoring standards. In addition, you also need to provide the score and the corresponding scoring basis. The reference answer is in JOSN format, KEY is the indicator, and the value is the value corresponding to the indicator.

[0111]

Tier Description

[0112] 0: The model output SQL does not contain the key components, values, tables, or aggregate functions in standard SQL, or is inconsistent with the intent of the question.

[0113] 2: The model output SQL and standard SQL are logically matched to ensure that the number of components, components, values, functions, connectors, tables, aggregate functions, and data are consistent and fully consistent with the intent of the problem, and the model output SQL is executable.

[0114] [Special note]: First, you need to structure the SQL according to [Split SQL to output structured information], and then please strictly score according to [partial_scores components rec, f1, acc logic]. The components include [select], [select(no AGG)], [where], [where(no OP)], [group(no Having)], [group], [order], [and / or], and [keywords]. The scoring content includes acc, f1, and rec, and their values ​​are either 1 or 0 according to the logic, and cannot be other values. Finally, determine whether it matches according to the content of [Matching logic]. If all components match, 2 points are given, otherwise 0 points are given. Of course, according to the query, the intention of the model output SQL is completely inconsistent, so 0 points are given directly.

[0115] Example of inconsistent components: SELECT sum(rain_fall_size), position FROM tpl_im_road_weather GROUP BY position; SELECT sum(rain_fall_size), position FROM tpl_im_road_weather GROUP BY position order by position; the two components are different, and the gap is in order by

[0116] It is important to note that different expressions with the same meaning are also considered a match and get 2 points, such as <> and ! =, multiple or and in, distinct colume and group by colume.

[0117] If the following situations are written in the same way, please give 2 points directly:

[0118] Example 1: If you use sum in the inner word query and sum in the outer layer, it is equivalent to directly querying the inner layer and summing the outer layer. Pay special attention to the fact that if the inner layer selects a column that does not affect the result, ignore it and consider it the same, and only focus on the processing of the outermost layer;

[0119] Standard SQL:

[0120] select sum(as) / 100from(select rwsj,sum(ysje)s from parking_orderunion all select rwsj,sum(ysje)s from parking_order_history_69)a

[0121] Model output SQL:

[0122] SELECT SUM(ysje) / 100AS total_receivable FROM(SELECT ysje FROMparking_order UNION ALL SELECT ysje FROM parking_order_history_69)AS t". The standard SQL is the actual query statement, and the model output SQL is the predicted query statement. The prompt word is set with a matching rule between the actual query statement and the predicted query statement, that is, a scoring rule for the first score is set.

[0123] S204. Determine a second score of the model to be analyzed based on the reply information; wherein the second score represents the reply effect of the model to be analyzed on the question information.

[0124] For example, based on the reply information, the text summarization ability of the model to be analyzed can be scored to obtain a second score of the model to be analyzed. The second score can represent the effect of the model to be analyzed in replying to the question information, that is, it can evaluate whether the sentence of the reply information is fluent and whether it is relevant to the question information. If the sentence of the reply information is fluent and relevant to the question information, a higher second score can be obtained.

[0125] The response information may be scored using a preset referee model. For example, the referee model may perform semantic recognition on the response information to determine whether the sentence in the response information is fluent; it may also perform semantic recognition on the question information to determine whether the semantics of the response information and the question information are related. In this embodiment, the model structure of the preset referee model is not specifically limited.

[0126] In this embodiment, the second score of the model to be analyzed is determined based on the reply information, including: determining the similarity between the reply information and the question information; wherein the similarity represents the degree of semantic similarity between the reply information and the question information; if the similarity is greater than a preset similarity threshold, then determining that the second score of the model to be analyzed is the first preset score.

[0127] Specifically, semantic recognition is performed on the reply information and the question information respectively to obtain semantic information of the reply information and semantic information of the question information. The similarity between the semantic information of the reply information and the semantic information of the question information is calculated. The similarity can represent the degree of semantic similarity between the reply information and the question information. For example, the cosine similarity can be used to calculate the degree of semantic similarity.

[0128] A similarity threshold is set in advance. If the calculated similarity is greater than the preset similarity threshold, the second score of the model to be analyzed is determined to be the first preset score, that is, 2 points are obtained; if the calculated similarity is less than or equal to the preset similarity threshold, the second score of the model to be analyzed is determined to be the second preset score, that is, 0 points are obtained.

[0129] It is also possible to perform semantic recognition on the reply information and the question information respectively, extract keywords that can represent semantics from the reply information, and extract keywords that can represent semantics from the question information. If the keywords in the reply information are consistent with the keywords in the question information, the second score of the model to be analyzed is determined to be the first preset score; if they are inconsistent, the second score of the model to be analyzed is determined to be the second preset score.

[0130] The beneficial effect of such a setting is that by comparing the question information with the reply information, the correlation between the question information and the reply information can be determined, thereby determining whether the model to be analyzed can accurately respond to the question information, thereby improving the analysis accuracy of the model to be analyzed.

[0131] In this embodiment, the second score of the model to be analyzed is determined based on the reply information, including: obtaining the data query result found by the model to be analyzed from a preset database; wherein the data query result represents the recall data determined by the model to be analyzed from the preset database according to the predictive query statement, and the reply information is the reply data in text form generated by the model to be analyzed; if the data query result exists in the reply information, the second score of the model to be analyzed is determined to be the first preset score.

[0132] Specifically, a database is pre-set to store a large amount of data. The model to be analyzed can query data from the preset database according to the predicted query statement, and determine the queried data as the data query result, which is the recalled data. The model to be analyzed constructs sentences according to the data query result, and the generated text is the reply information, that is, the reply information should contain the data query result.

[0133] Perform a comprehensive analysis of the data query results and the reply information to determine whether the data in the reply information is completely consistent with the data query results, that is, to determine whether the data query results are all present in the reply information. If so, determine the second score of the model to be analyzed to be the first preset score; if there is any inconsistency, determine the second score of the model to be analyzed to be the second preset score.

[0134] The beneficial effect of such a setting is that by comparing the data query results with the reply information, the comprehensiveness of the data in the reply information can be determined, avoiding the model to be analyzed from missing data when generating the reply information, and improving the analysis accuracy of the model to be analyzed.

[0135] In this embodiment, the second score of the model to be analyzed is determined based on the reply information, including: performing semantic recognition processing on the reply information to obtain a semantic score of the reply information; wherein the semantic score represents the clarity of the language organization of the reply information; if the semantic score is greater than a preset score threshold, then the second score of the model to be analyzed is determined to be the first preset score.

[0136] Specifically, the language organization ability of the model to be analyzed can also be analyzed. The reply information is a text generated by the language organization of the model to be analyzed. The reply information can be semantically recognized by a preset referee model to output a semantic score of the reply information. The semantic score represents the clarity of the language organization of the reply information. For example, it can represent whether the logic of the reply information is clear, whether the segmentation is clear, whether the key points are prominent, whether it is easy for readers to understand, etc. In this embodiment, the calculation process of the semantic score is not specifically limited.

[0137] A score threshold is set in advance, and the semantic score is compared with the score threshold. If the semantic score is greater than the preset score threshold, the second score of the model to be analyzed is determined to be the first preset score; if the semantic score is less than or equal to the preset score threshold, the second score of the model to be analyzed is determined to be the second preset score.

[0138] The beneficial effect of this setting is that the language organization ability of the model to be analyzed is analyzed, ensuring that the model to be analyzed can output logically clear response information, thereby improving the user's number-checking experience.

[0139] After receiving the reply information, the reply information can be analyzed in terms of relevance, comprehensiveness, and language organization ability. Relevance refers to the relevance between the reply information and the question information, comprehensiveness refers to the comprehensiveness of the data query results in the reply information, and language organization ability refers to the clarity of the language organization of the reply information. If the scores of these three aspects are all the first preset scores, the second score of the model to be analyzed is determined to be the first preset score; if the scores of these three aspects have the second preset score, the second score of the model to be analyzed is determined to be the second preset score, thereby achieving a comprehensive evaluation of the model to be analyzed.

[0140] S205. Obtain an analysis result of the performance of the model to be analyzed according to the first score and the second score.

[0141] Exemplarily, after obtaining the first score and the second score, the analysis result of the model to be analyzed is determined by combining the first score and the second score. For example, the weight of the first score and the weight of the second score can be preset, and the first score and the second score are weighted and summed according to the preset weights to obtain the weighted sum score. A score threshold is preset. If the score is greater than the preset score threshold, the analysis result of the model to be analyzed is considered to have passed the analysis, and the number can be checked for the user; if the score is less than or equal to the preset score threshold, the analysis result of the model to be analyzed is considered to have failed the analysis, and further optimization is required.

[0142] The data set to be analyzed includes multiple question information, and each question information corresponds to an analysis result. The analysis result is determined as the number of question information that passed, and the number of question information in the data set to be analyzed is determined. The analysis result is determined as the ratio between the number of question information that passed and the number of question information in the data set to be analyzed. A ratio threshold is pre-set. If the ratio is greater than the preset ratio threshold, it is finally determined that there is no problem with the model to be analyzed, and the number can be checked for the user; if the ratio is less than or equal to the preset ratio threshold, it is determined that the model to be analyzed needs to be optimized.

[0143] In this embodiment, the first aspect of the evaluation is first performed based on the actual SQL and the predicted SQL, and then the second aspect of the evaluation is performed based on the reply information, thereby achieving multi-dimensional analysis of the model to be analyzed, improving the analysis accuracy of the model, and thus enhancing the user's experience of looking up numbers using the model.

[0144] In this embodiment, the analysis result of the performance of the model to be analyzed is obtained according to the first score and the second score, including: if the first score and the second score are both first preset scores, determining that the analysis result of the performance of the model to be analyzed is analysis passed.

[0145] Specifically, the first preset score may be 2 points, and the second preset score may be 0 points. The scores of the first score and the second score are determined. For each question information, if the first score and the second score are both the first preset score, the analysis result of the performance of the model to be analyzed is determined to be passed; if the first score and the second score have the second preset score, the analysis result of the performance of the model to be analyzed is determined to be failed. That is, only when the analysis of the SQL statement and the analysis of the reply information are both 2 points, the model to be analyzed is considered to be passed.

[0146] The beneficial effect of such a setting is that the analysis results of the model are determined by combining the scores of the two aspects, thereby achieving a comprehensive evaluation of the model to be analyzed, thereby improving the analysis accuracy of the model.

[0147] In the disclosed embodiment, the preset question information and the corresponding actual query statement are obtained, the question information is input into the model to be analyzed, and the predicted query statement and reply information corresponding to the question information are obtained. The query performance of the model to be analyzed is analyzed by combining the actual query statement, the predicted query statement, and the reply information to obtain the analysis result. The model is evaluated in detail from the two aspects of SQL statement and reply information, which saves manpower and time, improves the analysis accuracy and efficiency of the model, and thus improves the user's data query experience.

[0148] Figure 3 The analysis flow chart of the model to be analyzed provided by the embodiment of the present disclosure. Based on the above embodiment, as Figure 3 As shown, the analysis process is:

[0149] The parameter types and parameter values ​​required for the question information are pre-set, and multiple question information is generated based on the pre-built statement generation model and the preset prompt (prompt word), thereby constructing the data set to be analyzed. Each question information in the data set to be analyzed corresponds to a correct SQL statement as the actual query statement.

[0150] The question information in the data set to be analyzed is input into the model to be analyzed, and the model to be analyzed converts the question information in text form into SQL statements as prediction query statements. According to the prediction query statements, data is queried in the preset database to obtain response information.

[0151] The predicted query statement and the reply information are input to the referee model, and the referee model analyzes the predicted query statement and the reply information respectively. The predicted query statement is analyzed to obtain a first score, and the reply information is analyzed to obtain a second score. The first score and the second score are combined to obtain the final analysis result.

[0152] When analyzing the predicted query statement, the predicted query statement and the corresponding actual query statement can be matched. If the components and component values ​​in the two are consistent, the first score is determined to be 2; if there is an inconsistency, the first score is determined to be 0. When determining that there is an inconsistency, it is also possible to determine whether the component of the inconsistent part is a preset special component. If so, the first score may be 2. The preset special component can be a select component or an order component. For the select component, if the component value of the component in the predicted query statement includes the component value of the corresponding component in the actual query statement, the predicted query statement is considered to be correct and the first score is 2; for the ordering component of order, if the component value part after order in the predicted query statement and the component value part after order in the actual query statement are only different in sorting direction, the predicted query statement is also considered to be correct and the first score is 2.

[0153] When analyzing the reply information, it can be analyzed from three dimensions: relevance, comprehensiveness, and language organization ability. Relevance refers to the semantic relevance between the reply information and the question information, comprehensiveness refers to whether the data in the reply information is all the recalled data queried by the model to be analyzed, and language organization ability refers to whether the reply information generated by the model to be analyzed is a logically clear text. Each of these three aspects can get a score. If all three scores are 2, the second score is determined to be 2.

[0154] When the final analysis result is obtained, if the first score and the second score are both 2 points, it is determined that the analysis of the model to be analyzed has passed; if one or both of the first score and the second score are 0 points, it is determined that the analysis of the model to be analyzed has failed.

[0155] Figure 4 A structural block diagram of a model performance analysis device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 The model performance analysis device 400 includes: an acquisition unit 401, a determination unit 402 and an analysis unit 403.

[0156] The acquisition unit 401 is used to acquire a data set to be analyzed; wherein the data set to be analyzed includes question information and actual query statements, wherein the question information represents information used to instruct the model to be analyzed to perform data query, and the actual query statement represents a correct structured query statement corresponding to the question information;

[0157] The determination unit 402 is used to determine the predicted query statement and reply information corresponding to the question information through the model to be analyzed; wherein the predicted query statement represents the structured query statement obtained by converting the question information by the model to be analyzed, and the reply information represents the answer of the model to be analyzed to the question information;

[0158] The analysis unit 403 is used to perform performance analysis on the model to be analyzed according to the actual query statement, predicted query statement, and reply information corresponding to the question information to obtain an analysis result; wherein the analysis result represents the effect of the data query of the model to be analyzed.

[0159] Figure 5 A structural block diagram of a model performance analysis device provided in an embodiment of the present disclosure, such as Figure 5 As shown, the device 500 for analyzing model performance includes an acquisition unit 501 , a determination unit 502 and an analysis unit 503 , wherein the analysis unit 503 includes a first acquisition module 5031 , a second acquisition module 5032 and a result analysis module 5033 .

[0160] The first obtaining module 5031 is used to obtain a first score of the model to be analyzed according to the actual query statement and the predicted query statement corresponding to the question information; wherein the first score represents the accuracy of the model to be analyzed in converting the question information into a structured query statement;

[0161] The second obtaining module 5032 is used to determine a second score of the model to be analyzed according to the reply information; wherein the second score represents the reply effect of the model to be analyzed on the question information;

[0162] The result analysis module 5033 is used to obtain an analysis result of the performance of the model to be analyzed according to the first score and the second score.

[0163] In one example, the first obtaining module 5031 includes:

[0164] A field determination submodule, used to determine field information in the predicted query statement and field information in the actual query statement; wherein the field information includes components and corresponding component values;

[0165] The field matching submodule is used to match the field information in the predicted query statement with the field information in the actual query statement to obtain a first score of the model to be analyzed.

[0166] In one example, the field matching submodule is used specifically for:

[0167] If the field information in the predicted query statement is the same as the field information in the actual query statement, determining that the first score of the model to be analyzed is a first preset score;

[0168] If the field information in the predicted query statement is different from the field information in the actual query statement, the first score of the model to be analyzed is determined to be a second preset score; wherein the first preset score is greater than the second preset score.

[0169] In one example, the field matching submodule is used specifically for:

[0170] If the field information in the predicted query statement is different from the field information in the actual query statement, determining the difference field in the predicted query statement and the difference field in the actual query statement; wherein the difference field represents the different field information between the predicted query statement and the actual query statement;

[0171] Determine that the type of the component in the difference field of the predicted query statement is a first type, and determine that the type of the component in the difference field of the actual query statement is a second type;

[0172] If both the first type and the second type are preset types, a first score of the model to be analyzed is determined according to component values ​​in the difference fields of the predicted query statement and component values ​​in the difference fields of the actual query statement.

[0173] In one example, the preset type is a select type; the field matching submodule is specifically used for:

[0174] If the component value in the difference field of the predicted query statement exists in the component value in the difference field of the actual query statement, then the first score of the model to be analyzed is determined to be the first preset score.

[0175] In one example, the preset type is the order type; the field matching submodule is specifically used for:

[0176] If the difference between the component values ​​in the difference fields of the predicted query statement and the component values ​​in the difference fields of the actual query statement represents a sorting direction, then the first score of the model to be analyzed is determined to be the first preset score.

[0177] In one example, the second obtaining module 5032 includes:

[0178] A similarity determination submodule, used to determine the similarity between the reply information and the question information; wherein the similarity represents the degree of semantic similarity between the reply information and the question information;

[0179] The similarity judgment submodule is used to determine that the second score of the model to be analyzed is a first preset score if the similarity is greater than a preset similarity threshold.

[0180] In one example, the second obtaining module 5032 includes:

[0181] A result acquisition submodule is used to obtain the data query results found by the model to be analyzed from the preset database; wherein the data query results represent the recall data determined by the model to be analyzed from the preset database according to the prediction query statement, and the reply information is the reply data in text form generated by the model to be analyzed;

[0182] A result query submodule is used to determine that the second score of the model to be analyzed is the first preset score if the data query result exists in the reply information.

[0183] In one example, the second obtaining module 5032 includes:

[0184] A semantic recognition submodule, used to perform semantic recognition processing on the reply information to obtain a semantic score of the reply information; wherein the semantic score represents the clarity of the language organization of the reply information;

[0185] The semantic judgment submodule is used to determine that the second score of the model to be analyzed is the first preset score if the semantic score is greater than a preset score threshold.

[0186] In one example, the result analysis module 5033 includes:

[0187] The analysis passed submodule is used to determine that the analysis result of the performance of the model to be analyzed is analysis passed if the first score and the second score are both first preset scores.

[0188] In one example, the determining unit 502 includes:

[0189] A sentence prediction module, used for inputting the question information into the model to be analyzed to obtain a predicted query sentence corresponding to the question information;

[0190] The information reply module is used to obtain the reply information output by the model to be analyzed according to the prediction query statement.

[0191] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0192] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present disclosure. The electronic device may be a terminal device, a processor or a server. Figure 6As shown, the electronic device 60 includes: at least one processor 602; and a memory 601 communicatively connected to the at least one processor 602; wherein the memory stores instructions executable by the at least one processor 602, and the instructions are executed by the at least one processor 602 so that the at least one processor 602 can execute the model performance analysis method disclosed in the present invention.

[0193] The electronic device 60 further includes a receiver 603 and a transmitter 604. The receiver 603 is used to receive instructions and data sent by other devices, and the transmitter 604 is used to send instructions and data to external devices.

[0194] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0195] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0196] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0197] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0198] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0199] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the analysis method of model performance. For example, in some embodiments, the analysis method of model performance may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the analysis method of model performance described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the analysis method of model performance in any other appropriate manner (e.g., by means of firmware).

[0200] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0201] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0202] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0204] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0205] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0206] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0207] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for analyzing model performance, comprising: Acquire a data set to be analyzed; wherein the data set to be analyzed includes question information and actual query statements, the question information represents information used to instruct the model to be analyzed to perform data query, and the actual query statement represents a correct structured query statement corresponding to the question information; Determine the predicted query statement and reply information corresponding to the question information through the model to be analyzed; wherein the predicted query statement represents the structured query statement obtained by converting the question information by the model to be analyzed, and the reply information represents the answer of the model to be analyzed to the question information; According to the actual query statement, predicted query statement, and reply information corresponding to the question information, the performance analysis of the model to be analyzed is performed to obtain an analysis result; wherein the analysis result represents the effect of the data query of the model to be analyzed.

2. The method according to claim 1, wherein: The step of performing performance analysis on the model to be analyzed according to the actual query statement, the predicted query statement, and the reply information corresponding to the question information to obtain analysis results includes: Obtaining a first score of the model to be analyzed according to the actual query statement and the predicted query statement corresponding to the question information; wherein the first score represents the accuracy of the model to be analyzed in converting the question information into a structured query statement; Determine a second score of the model to be analyzed according to the reply information; wherein the second score represents the reply effect of the model to be analyzed on the question information; An analysis result of the performance of the model to be analyzed is obtained according to the first score and the second score.

3. The method according to claim 2, wherein: The step of obtaining a first score of the model to be analyzed according to the actual query statement and the predicted query statement corresponding to the question information includes: Determine field information in the predicted query statement, and determine field information in the actual query statement; wherein the field information includes components and corresponding component values; The field information in the predicted query statement is matched with the field information in the actual query statement to obtain a first score of the model to be analyzed.

4. The method according to claim 3, wherein: The matching of the field information in the predicted query statement with the field information in the actual query statement to obtain a first score of the model to be analyzed includes: If the field information in the predicted query statement is the same as the field information in the actual query statement, determining that the first score of the model to be analyzed is a first preset score; If the field information in the predicted query statement is different from the field information in the actual query statement, the first score of the model to be analyzed is determined to be a second preset score; wherein the first preset score is greater than the second preset score.

5. The method according to claim 3 or 4, wherein: The matching of the field information in the predicted query statement with the field information in the actual query statement to obtain a first score of the model to be analyzed includes: If the field information in the predicted query statement is different from the field information in the actual query statement, determining the difference field in the predicted query statement and the difference field in the actual query statement; wherein the difference field represents the different field information between the predicted query statement and the actual query statement; Determine that the type of the component in the difference field of the predicted query statement is a first type, and determine that the type of the component in the difference field of the actual query statement is a second type; If both the first type and the second type are preset types, a first score of the model to be analyzed is determined according to component values ​​in the difference fields of the predicted query statement and component values ​​in the difference fields of the actual query statement.

6. The method according to claim 5, wherein: The preset type is a select type; and determining the first score of the model to be analyzed according to the component value in the difference field of the predicted query statement and the component value in the difference field of the actual query statement comprises: If the component value in the difference field of the predicted query statement exists in the component value in the difference field of the actual query statement, then the first score of the model to be analyzed is determined to be the first preset score.

7. The method according to claim 5 or 6, wherein: The preset type is an order type; and determining the first score of the model to be analyzed according to the component value in the difference field of the predicted query statement and the component value in the difference field of the actual query statement includes: If the difference between the component values ​​in the difference fields of the predicted query statement and the component values ​​in the difference fields of the actual query statement represents a sorting direction, then the first score of the model to be analyzed is determined to be the first preset score.

8. The method according to any one of claims 2 to 7, wherein: The step of determining a second score of the model to be analyzed according to the reply information includes: Determine the similarity between the reply information and the question information; wherein the similarity represents the degree of semantic similarity between the reply information and the question information; If the similarity is greater than a preset similarity threshold, the second score of the model to be analyzed is determined to be a first preset score.

9. The method according to claim 8, wherein: The step of determining a second score of the model to be analyzed according to the reply information includes: Obtaining data query results found by the model to be analyzed from a preset database; wherein the data query results represent the recall data determined by the model to be analyzed from the preset database according to the prediction query statement, and the reply information is the reply data in text form generated by the model to be analyzed; If the data query result exists in the reply information, the second score of the model to be analyzed is determined to be the first preset score.

10. The method according to claim 8 or 9, wherein: The step of determining a second score of the model to be analyzed according to the reply information includes: Performing semantic recognition processing on the reply information to obtain a semantic score of the reply information; wherein the semantic score represents the clarity of the language organization of the reply information; If the semantic score is greater than a preset score threshold, the second score of the model to be analyzed is determined to be the first preset score.

11. The method according to any one of claims 2 to 10, wherein: The obtaining, according to the first score and the second score, an analysis result of the performance of the model to be analyzed includes: If the first score and the second score are both first preset scores, it is determined that the analysis result of the performance of the model to be analyzed is analysis passed.

12. The method according to any one of claims 1 to 11, wherein: The step of determining the predicted query statement and reply information corresponding to the question information by using the model to be analyzed includes: Inputting the question information into the model to be analyzed to obtain a predicted query statement corresponding to the question information; According to the prediction query statement, the response information output by the model to be analyzed is obtained.

13. A device for analyzing model performance, comprising: An acquisition unit is used to acquire a data set to be analyzed; wherein the data set to be analyzed includes question information and actual query statements, the question information represents information used to instruct the model to be analyzed to perform data query, and the actual query statement represents a correct structured query statement corresponding to the question information; A determination unit, configured to determine, by means of the model to be analyzed, a predicted query statement and reply information corresponding to the question information; wherein the predicted query statement represents a structured query statement obtained by converting the question information by the model to be analyzed, and the reply information represents a reply of the model to be analyzed to the question information; The analysis unit is used to perform performance analysis on the model to be analyzed according to the actual query statement, predicted query statement, and reply information corresponding to the question information to obtain an analysis result; wherein the analysis result represents the effect of the data query of the model to be analyzed.

14. The device according to claim 13, wherein: The analysis unit comprises: A first obtaining module, configured to obtain a first score of the model to be analyzed according to the actual query statement and the predicted query statement corresponding to the question information; wherein the first score represents the accuracy of the model to be analyzed in converting the question information into a structured query statement; A second obtaining module is used to determine a second score of the model to be analyzed according to the reply information; wherein the second score represents the reply effect of the model to be analyzed on the question information; A result analysis module is used to obtain an analysis result of the performance of the model to be analyzed according to the first score and the second score.

15. The device according to claim 14, wherein: The first obtaining module comprises: A field determination submodule, used to determine field information in the predicted query statement and field information in the actual query statement; wherein the field information includes components and corresponding component values; The field matching submodule is used to match the field information in the predicted query statement with the field information in the actual query statement to obtain a first score of the model to be analyzed.

16. The device according to claim 15, wherein: The field matching submodule is specifically used for: If the field information in the predicted query statement is the same as the field information in the actual query statement, determining that the first score of the model to be analyzed is a first preset score; If the field information in the predicted query statement is different from the field information in the actual query statement, the first score of the model to be analyzed is determined to be a second preset score; wherein the first preset score is greater than the second preset score.

17. The device according to claim 15 or 16, wherein: The field matching submodule is specifically used for: If the field information in the predicted query statement is different from the field information in the actual query statement, determining the difference field in the predicted query statement and the difference field in the actual query statement; wherein the difference field represents the different field information between the predicted query statement and the actual query statement; Determine that the type of the component in the difference field of the predicted query statement is a first type, and determine that the type of the component in the difference field of the actual query statement is a second type; If both the first type and the second type are preset types, a first score of the model to be analyzed is determined according to component values ​​in the difference fields of the predicted query statement and component values ​​in the difference fields of the actual query statement.

18. The device according to claim 17, wherein: The preset type is a select type; the field matching submodule is specifically used for: If the component value in the difference field of the predicted query statement exists in the component value in the difference field of the actual query statement, then the first score of the model to be analyzed is determined to be the first preset score.

19. The device according to claim 17 or 18, wherein: The preset type is the order type; the field matching submodule is specifically used for: If the difference between the component values ​​in the difference fields of the predicted query statement and the component values ​​in the difference fields of the actual query statement represents a sorting direction, then the first score of the model to be analyzed is determined to be the first preset score.

20. The device according to any one of claims 14 to 19, wherein: The second acquisition module comprises: A similarity determination submodule, used to determine the similarity between the reply information and the question information; wherein the similarity represents the degree of semantic similarity between the reply information and the question information; The similarity judgment submodule is used to determine that the second score of the model to be analyzed is a first preset score if the similarity is greater than a preset similarity threshold.

21. The device according to claim 20, wherein: The second acquisition module comprises: A result acquisition submodule is used to obtain the data query results found by the model to be analyzed from the preset database; wherein the data query results represent the recall data determined by the model to be analyzed from the preset database according to the prediction query statement, and the reply information is the reply data in text form generated by the model to be analyzed; A result query submodule is used to determine that the second score of the model to be analyzed is the first preset score if the data query result exists in the reply information.

22. The device according to claim 20 or 21, wherein: The second acquisition module comprises: A semantic recognition submodule, used to perform semantic recognition processing on the reply information to obtain a semantic score of the reply information; wherein the semantic score represents the clarity of the language organization of the reply information; The semantic judgment submodule is used to determine that the second score of the model to be analyzed is the first preset score if the semantic score is greater than a preset score threshold.

23. The device according to any one of claims 14 to 22, wherein: The result analysis module comprises: The analysis passed submodule is used to determine that the analysis result of the performance of the model to be analyzed is analysis passed if the first score and the second score are both first preset scores.

24. The device according to any one of claims 13 to 23, wherein: The determining unit comprises: A sentence prediction module, used for inputting the question information into the model to be analyzed to obtain a predicted query sentence corresponding to the question information; The information reply module is used to obtain the reply information output by the model to be analyzed according to the prediction query statement.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product, wherein: The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 12 when being executed by a processor.

Images