Test method and device of SQL statement generation model, equipment and storage medium

Abstract

The invention discloses a test method and device for an SQL statement generation model, equipment and a storage medium, and belongs to the technical field of computers. And in response to a model test request for the SQL statement generation model, converting the plurality of target test statements into a plurality of test SQL statements corresponding to the target test statements by using the SQL statement generation model. And executing the test SQL statement and the annotation SQL statement of each target test statement to obtain a first execution result of the test SQL statement of each target test statement and a second execution result of the annotation SQL statement, thereby increasing the test on the execution level of the SQL statement. The model test result is determined by combining the structural difference information between the test SQL statement and the annotation SQL statement of each target test statement and the execution result difference information between the first execution result and the second execution result, so that the structural difference of the SQL statements is considered, and the difference of the SQL statements in the execution result is also considered; and the accuracy of the obtained model test result is relatively high.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a testing method, apparatus, device, and storage medium for an SQL statement generation model. Background Technology

[0002] With the development of computer technology, more and more fields are using large models to improve efficiency and accuracy. For example, due to the powerful natural language understanding capabilities of large models, the use of SQL statement generation models to convert natural language into SQL statements has been adopted in multiple fields. These SQL statement generation models are large models that have undergone pre-training and instruction fine-tuning. To ensure the production performance of the SQL statements, the SQL statement generation model needs to be tested before deployment to the production environment to determine the accuracy of the generated SQL statements. Only after the SQL statement generation model passes testing will it be formally deployed to the production environment.

[0003] In related technologies, testing the SQL statement generation model relies on manual methods, that is, testers compare the SQL statements generated by the SQL statement generation model with the annotated SQL statements. This results in a lengthy testing process and a high error rate.

[0004] Therefore, improving the efficiency of testing SQL statement generation models and reducing the error rate of testing is a hot research topic. Summary of the Invention

[0005] This application provides a testing method, apparatus, device, and storage medium for SQL statement generation models, which can improve the efficiency of testing SQL statement generation models and reduce the error rate of testing. The technical solution is as follows:

[0006] On the one hand, a testing method for an SQL statement generation model is provided, the method comprising:

[0007] In response to a model test request for the SQL statement generation model, multiple target test statements are input into the SQL statement generation model, and the multiple target test statements are processed by the SQL statement generation model to obtain test SQL statements for each target test statement.

[0008] Execute the test SQL statement and annotation SQL statement for each of the target test statements to obtain the first execution result and the second execution result corresponding to each of the target test statements. The first execution result is the execution result of the test SQL statement, and the second execution result is the execution result of the annotation SQL statement.

[0009] Based on the structural differences between the test SQL statements and the labeled SQL statements of each target test statement, as well as the differences between the execution results of the first execution result and the second execution result, the model test result of the SQL statement generation model is determined. The model test result is used to represent the accuracy of the SQL statements generated by the SQL statement generation model.

[0010] In one possible implementation, the step of processing the plurality of target test statements through the SQL statement generation model to obtain test SQL statements for each of the target test statements includes:

[0011] For any target test statement among the plurality of target test statements, the target test statement is semantically encoded using the SQL statement generation model to obtain the test statement features of the target test statement;

[0012] The test SQL statement of the target test statement is obtained by performing multiple rounds of iterative decoding on the test statement features of the target test statement through the SQL statement generation model.

[0013] In one possible implementation, executing the test SQL statement and the annotation SQL statement for each of the target test statements to obtain the first execution result and the second execution result corresponding to each of the target test statements includes:

[0014] For any one of the plurality of target test statements, the test SQL statement and the annotation SQL statement of the target test statement are executed in the target database respectively to obtain the first execution result and the second execution result corresponding to the target test statement. Both the first execution result and the second execution result include the field values ​​recalled from the target database.

[0015] In one possible implementation, determining the model test result of the SQL statement generation model based on the structural difference information between the test SQL statements and the labeled SQL statements of each of the target test statements, and the execution result difference information between the first execution result and the second execution result, includes:

[0016] Based on the structural differences between the test SQL statements and the labeled SQL statements of each target test statement, the statement structure test results of the SQL statement generation model are determined.

[0017] Based on the difference information between the first execution result and the second execution result, the statement execution test result of the SQL statement generation model is determined;

[0018] The test results of the model are determined based on the test results of the statement structure and the test results of the statement execution.

[0019] In one possible implementation, the structural difference information includes table difference information, field difference information, and aggregation method difference information. The step of determining the statement structure test result of the SQL statement generation model based on the structural difference information between the test SQL statements and the labeled SQL statements for each of the target test statements includes:

[0020] For any target test statement among the plurality of target test statements, the structure test score corresponding to the target test statement is determined based on the table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of the target test statement;

[0021] The average of the structure test scores corresponding to the multiple target test statements is determined as the statement structure test result of the SQL statement generation model.

[0022] In one possible implementation, the method for determining the structural difference information includes:

[0023] For any one of the multiple target test statements, compare the table name, field name, and aggregation method of the data table indicated in the test SQL statement and the annotation SQL statement of the target test statement, respectively, to obtain the table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of the target test statement;

[0024] By concatenating the table differences, field differences, and aggregation method differences between the test SQL statement and the annotation SQL statement of the target test statement, the structural differences between the test SQL statement and the annotation SQL statement of the target test statement are obtained.

[0025] In one possible implementation, the execution result difference information includes null value percentage difference information and field value difference information. Determining the statement execution test result of the SQL statement generation model based on the execution result difference information between the first execution result and the second execution result includes:

[0026] For any target test statement among the plurality of target test statements, the statement execution test score corresponding to the target test statement is determined based on the difference information of the proportion of null values ​​and the difference information of field values ​​between the first execution result and the second execution result;

[0027] The average of the statement execution test scores corresponding to the multiple target test statements is determined as the statement execution test result of the SQL statement generation model.

[0028] In one possible implementation, the method for determining the execution result difference information includes:

[0029] For any one of the plurality of target test statements, determine the target field name in the second execution result corresponding to the target test statement, wherein both the first execution result and the second execution result include the field value recalled from the target database;

[0030] The target field name is used to identify the field in the first execution result corresponding to the target test statement, and the reference field name in the first execution result is obtained. The reference field name is the field that matches the target field name.

[0031] Based on the field value of the reference field name in the first execution result corresponding to the target test statement and the field value of the target field name in the second execution result corresponding to the target test statement, the execution result difference information corresponding to the target test statement is determined.

[0032] In one possible implementation, the step of using the target field name to perform field identification in the first execution result corresponding to the target test statement to obtain the reference field name in the first execution result includes:

[0033] Semantic encoding is performed on the target field name to obtain the field name feature of the target field name;

[0034] Semantic encoding is performed on at least one candidate field name in the first execution result corresponding to the target test statement to obtain the field name features of the at least one candidate field name;

[0035] The reference field name is determined from the at least one candidate field name based on the field name characteristics of the target field name and the field name characteristics of the at least one candidate field name.

[0036] In one possible implementation, the execution result difference information includes null value percentage difference information and field value difference information. Determining the execution result difference information corresponding to the target test statement based on the field value of the reference field name in the first execution result corresponding to the target test statement and the field value of the target field name in the second execution result corresponding to the target test statement includes:

[0037] Determine the percentage of first null values ​​in the first execution result corresponding to the target test statement and the percentage of second null values ​​in the second execution result corresponding to the target test statement;

[0038] The difference between the first null value percentage and the second null value percentage is determined as the null value percentage difference information between the first execution result and the second execution result corresponding to the target test statement;

[0039] Determine the result deviation value and result deviation ratio between the first execution result corresponding to the target test statement and the second execution result corresponding to the target test statement. The field value difference information includes the result deviation value and the result deviation ratio.

[0040] The null value percentage difference information, the result deviation value, and the result deviation ratio are determined as the field value difference information between the first execution result and the second execution result corresponding to the target test statement.

[0041] In one possible implementation, the method further includes:

[0042] Obtain the model performance parameters of the SQL statement generation model when generating test SQL statements for each of the target test statements. The model performance parameters include the model response time and the number of exception retries.

[0043] Based on the model performance parameters, the structural differences between the test SQL statements and the labeled SQL statements of each target test statement, and the differences between the execution results of the first execution result and the second execution result, the model test result of the SQL statement generation model is determined.

[0044] On the one hand, a testing device for an SQL statement generation model is provided, the device comprising:

[0045] The processing module is used to respond to a model test request for the SQL statement generation model, input multiple target test statements into the SQL statement generation model, process the multiple target test statements through the SQL statement generation model, and obtain test SQL statements for each target test statement.

[0046] The execution module is used to execute the test SQL statements and annotation SQL statements of each of the target test statements, and obtain the first execution result and the second execution result corresponding to each of the target test statements. The first execution result is the execution result of the test SQL statement, and the second execution result is the execution result of the annotation SQL statement.

[0047] The testing module is used to determine the model test result of the SQL statement generation model based on the structural difference information between the test SQL statement and the labeled SQL statement of each target test statement, as well as the difference information between the execution result of the first execution result and the second execution result. The model test result is used to represent the accuracy of the SQL statement generated by the SQL statement generation model.

[0048] In one possible implementation, the processing module is configured to, for any one of the plurality of target test statements, perform semantic encoding on the target test statement using the SQL statement generation model to obtain the test statement features of the target test statement; and perform multiple rounds of iterative decoding on the test statement features of the target test statement using the SQL statement generation model to obtain the test SQL statement of the target test statement.

[0049] In one possible implementation, the execution module is configured to execute the test SQL statement and the annotation SQL statement of any one of the plurality of target test statements in the target database respectively, to obtain a first execution result and a second execution result corresponding to the target test statement, wherein both the first execution result and the second execution result include field values ​​recalled from the target database.

[0050] In one possible implementation, the testing module is configured to: determine the statement structure test result of the SQL statement generation model based on the structural difference information between the test SQL statements and the labeled SQL statements of each of the target test statements; determine the statement execution test result of the SQL statement generation model based on the execution result difference information between the first execution result and the second execution result; and determine the model test result based on the statement structure test result and the statement execution test result.

[0051] In one possible implementation, the structural difference information includes table difference information, field difference information, and aggregation method difference information. The testing module is used to determine the structural test score corresponding to any target test statement among the plurality of target test statements based on the table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of the target test statement; and to determine the average of the structural test scores corresponding to the plurality of target test statements as the statement structure test result of the SQL statement generation model.

[0052] In one possible implementation, the apparatus further includes a structural difference information determination module, configured to, for any target test statement among the plurality of target test statements, compare the table name, field name, and aggregation method of the data table indicated in the test SQL statement and the annotation SQL statement of the target test statement, respectively, to obtain table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of the target test statement; and concatenate the table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of the target test statement to obtain structural difference information between the test SQL statement and the annotation SQL statement of the target test statement.

[0053] In one possible implementation, the execution result difference information includes null value ratio difference information and field value difference information. The testing module is used to determine the statement execution test score corresponding to any target test statement among the plurality of target test statements based on the null value ratio difference information and field value difference information between the first execution result and the second execution result; and to determine the average of the statement execution test scores corresponding to the plurality of target test statements as the statement execution test result of the SQL statement generation model.

[0054] In one possible implementation, the apparatus further includes an execution result difference information determination module, configured to, for any one of the plurality of target test statements, determine a target field name in a second execution result corresponding to the target test statement, wherein both the first execution result and the second execution result include field values ​​retrieved from a target database; use the target field name to perform field identification in the first execution result corresponding to the target test statement to obtain a reference field name in the first execution result, wherein the reference field name is a field that matches the target field name; and determine execution result difference information corresponding to the target test statement based on the field value of the reference field name in the first execution result corresponding to the target test statement and the field value of the target field name in the second execution result corresponding to the target test statement.

[0055] In one possible implementation, the execution result difference information determination module is used to perform semantic encoding on the target field name to obtain the field name feature of the target field name; perform semantic encoding on at least one candidate field name in the first execution result corresponding to the target test statement to obtain the field name feature of the at least one candidate field name; and determine the reference field name from the at least one candidate field name based on the field name feature of the target field name and the field name feature of the at least one candidate field name.

[0056] In one possible implementation, the execution result difference information includes null value percentage difference information and field value difference information. The execution result difference information determination module is used to determine the first null value percentage of the first execution result corresponding to the target test statement and the second null value percentage of the second execution result corresponding to the target test statement; determine the difference between the first null value percentage and the second null value percentage as the null value percentage difference information between the first execution result and the second execution result corresponding to the target test statement; determine the result deviation value and the result deviation ratio between the first execution result and the second execution result corresponding to the target test statement, wherein the field value difference information includes the result deviation value and the result deviation ratio; and determine the null value percentage difference information, the result deviation value, and the result deviation ratio as the field value difference information between the first execution result and the second execution result corresponding to the target test statement.

[0057] In one possible implementation, the apparatus further includes: a performance parameter acquisition module, configured to acquire model performance parameters of the SQL statement generation model when generating test SQL statements for each of the target test statements, the model performance parameters including model response time and number of abnormal retries;

[0058] The testing module is further configured to determine the model test result of the SQL statement generating model based on the model performance parameters, the structural difference information between the test SQL statements and the labeled SQL statements of each of the target test statements, and the execution result difference information between the first execution result and the second execution result.

[0059] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement a test method for the SQL statement generation model.

[0060] On one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement a test method for the SQL statement generation model.

[0061] On the one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to execute the test method of the above-mentioned SQL statement generation model.

[0062] The technical solution provided in this application, in response to a model testing request for an SQL statement generation model, converts multiple target test statements into multiple corresponding test SQL statements using the SQL statement generation model. The test SQL statements and annotated SQL statements of each target test statement are executed to obtain a first execution result of the test SQL statements and a second execution result of the annotated SQL statements, thereby increasing the testing at the SQL statement execution level. The model test result is determined by combining the structural differences between the test SQL statements and annotated SQL statements of each target test statement, as well as the differences in execution results between the first and second execution results. This approach considers not only the structural differences in SQL statements but also the differences in their execution results, resulting in highly accurate model test results. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of the implementation environment of a test method for an SQL statement generation model provided in an embodiment of this application;

[0065] Figure 2 This is a flowchart of a test method for an SQL statement generation model provided in an embodiment of this application;

[0066] Figure 3 This is a flowchart of a testing method for another SQL statement generation model provided in an embodiment of this application;

[0067] Figure 4 This is a flowchart of a test method for another SQL statement generation model provided in an embodiment of this application;

[0068] Figure 5 This is a schematic diagram of the structure of a test device for an SQL statement generation model provided in an embodiment of this application;

[0069] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0070] Figure 7 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0072] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0073] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain better results.

[0074] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence.

[0075] Semantic features: Features used to represent the semantics expressed by text. Different texts can correspond to the same semantic features; for example, the text "How's the weather today?" and the text "How's the weather today?" can correspond to the same semantic feature. Computer devices can map characters in text into character vectors, and combine and operate on these character vectors according to the relationships between characters to obtain the semantic features of the text. For example, computer devices can use a bidirectional encoder representation from transformers (BERT).

[0076] Normalization: Mapping sequences of values ​​with different ranges to the interval (0, 1) to facilitate data processing. In some cases, normalized values ​​can be directly expressed as probabilities.

[0077] The Gaussian distribution, also known as the normal distribution, has a bell-shaped curve, high in the middle and low at both ends. The expected value μ determines the position of the Gaussian distribution curve, while the standard deviation σ determines its range. The Gaussian distribution with μ = 0 and σ = 1 is the standard Gaussian distribution.

[0078] Embedded coding, mathematically speaking, represents a correspondence, that is, mapping data in space X to space Y using a function F. This function F is injective, and the mapping result preserves the structure. An injective function means that the mapped data uniquely corresponds to the original data, and preserving the structure means that the order of the original data remains the same. For example, if there are data X1 and X2 before mapping, after mapping we get Y1 corresponding to X1 and Y2 corresponding to X2. If the original data X1 > X2, then correspondingly, the mapped data Y1 > Y2. For words, this means mapping words to another space to facilitate subsequent machine learning and processing.

[0079] Attention weights represent the importance of a piece of data during training or prediction. Importance indicates the magnitude of the influence of input data on output data. Data with high importance corresponds to higher attention weights, while data with low importance corresponds to lower attention weights. The importance of data varies in different scenarios, and training the model to assign attention weights is essentially the process of determining data importance.

[0080] SQL (Structured Query Language) statements are a programming language used to store, process, and retrieve information in relational databases.

[0081] In related technologies, many scenarios utilize SQL statement generation models to generate SQL statements, thereby efficiently obtaining and applying them in downstream data acquisition. The accuracy of the SQL statements directly impacts the accuracy of downstream data acquisition. Therefore, before officially deploying an SQL statement generation model to a production environment, it is necessary to test the model to determine the accuracy of the generated SQL statements, thus determining whether the model can be deployed to production.

[0082] In related technologies, the text similarity between the SQL statements generated by the SQL statement generation model and the labeled SQL statements is usually directly compared, and the text similarity is used to evaluate the SQL statement generation model.

[0083] However, since the SQL statements generated by the model and the labeled SQL statements have similar structures, the text similarity is already relatively high. But due to the characteristics of SQL statements, a difference of even one character in the field name or table name of the two SQL statements can lead to completely different execution results. This can result in two SQL statements having high text similarity but vastly different execution results, making it impossible to accurately determine the accuracy of the SQL statements generated by the SQL statement generation model.

[0084] The technical solution provided in this application can improve the accuracy of model test results obtained by testing SQL statement generation models.

[0085] The implementation environment of the technical solutions provided in the embodiments of this application will be described below. Figure 1 This is a schematic diagram illustrating the implementation environment of a test method for an SQL statement generation model provided in this application embodiment. See also... Figure 1 The implementation environment may include terminal 110 and server 140.

[0086] Terminal 110 is connected to server 140 via a wireless or wired network. Optionally, terminal 110 may be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. Terminal 110 has an application installed and running that supports SQL statement generation.

[0087] Server 140 is a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. Server 140 has an SQL statement generation model deployed on it. Server 140 provides an application programming interface (API) through this SQL statement generation model to provide backend services for the applications running on terminal 110.

[0088] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more, in which case other terminals may also be included in the above implementation environment. This application does not limit the number of terminals or the type of device in its embodiments.

[0089] After introducing the implementation environment of the embodiments of this application, the application scenarios of the embodiments of this application will be introduced below.

[0090] The technical solution provided in this application can be applied to scenarios where various SQL statement generation models are tested. By adopting the technical solution provided in this application, the SQL statement generation model can be tested more comprehensively, thereby obtaining more accurate model test results.

[0091] After introducing the application scenarios of the embodiments of this application, the following describes a testing method for an SQL statement generation model provided by the embodiments of this application. See [link to documentation]. Figure 2 Taking the server as the executing entity as an example, the method includes the following steps.

[0092] 201. In response to a model test request for the SQL statement generation model, the server inputs multiple target test statements into the SQL statement generation model, processes the multiple target test statements through the SQL statement generation model, and obtains the test SQL statements for each target test statement.

[0093] The SQL statement generation model is a text-to-sql model capable of converting natural language (text) into structured query language (SQL). Its base model is the Large Language Model, which leverages its natural language understanding and language generation capabilities to achieve this conversion. The model test request is used to request testing of the SQL statement generation model to determine the accuracy of the generated SQL statements. The target test statements are natural language statements used to test the SQL statement generation model. Multiple target test statements originate from a test dataset, which can be either a dataset designed by technical personnel or a publicly available dataset in the relevant field; this application embodiment does not impose such limitations. The test SQL statement is the SQL statement generated by the SQL statement generation model from the input target test statement. In this application embodiment, testing the SQL statement generation model is performed in conjunction with the test SQL statements output by the model.

[0094] 202. The server executes the test SQL statement and the annotation SQL statement for each target test statement, and obtains the first execution result and the second execution result corresponding to each target test statement. The first execution result is the execution result of the test SQL statement, and the second execution result is the execution result of the annotation SQL statement.

[0095] The test dataset includes multiple target test statements and corresponding labeled SQL statements, which are the actual SQL statements corresponding to each target test statement. These labeled SQL statements serve as the benchmark for evaluating the SQL statement generation model. Executing an SQL statement refers to querying the database using the SQL statement to obtain the corresponding query data, which is the execution result of the SQL statement. Accordingly, the first execution result is the first query data obtained by the test SQL statement in the database, and the second execution result is the second query data obtained by the labeled SQL statement in the database. In step 202 above, for any target test statement among the multiple target test statements, executing the test SQL statement and the labeled SQL statement involves querying the same database using the test SQL statement and the labeled SQL statement.

[0096] 203. Based on the structural differences between the test SQL statements and the labeled SQL statements of each target test statement, as well as the differences between the execution results of the first execution result and the second execution result, the server determines the model test result of the SQL statement generation model. The model test result is used to represent the accuracy of the SQL statement generated by the SQL statement generation model.

[0097] The structural difference information refers to the differences in the statement structure of the SQL statement, and the execution result difference information refers to the differences in the execution results of the SQL statement, that is, the differences between the query data corresponding to the SQL statement. In this embodiment, the smaller the structural difference information and execution result difference information between the test SQL statement and the labeled SQL statement, the closer the test SQL statement is to the labeled SQL statement, and the higher the accuracy of the SQL statement generated by the SQL statement generation model. Conversely, the larger the structural difference information or execution result difference information between the test SQL statement and the labeled SQL statement, the greater the difference between the test SQL statement and the labeled SQL statement, and the lower the accuracy of the SQL statement generated by the SQL statement generation model.

[0098] The technical solution provided in this application, in response to a model testing request for an SQL statement generation model, converts multiple target test statements into multiple corresponding test SQL statements using the SQL statement generation model. The test SQL statements and annotated SQL statements of each target test statement are executed to obtain a first execution result of the test SQL statements and a second execution result of the annotated SQL statements, thereby increasing the testing at the SQL statement execution level. The model test result is determined by combining the structural differences between the test SQL statements and annotated SQL statements of each target test statement, as well as the differences in execution results between the first and second execution results. This approach considers not only the structural differences in SQL statements but also the differences in their execution results, resulting in highly accurate model test results.

[0099] The above steps 201-203 are a brief description of the technical solutions provided in the embodiments of this application. The technical solutions provided in the embodiments of this application will be described more clearly below with reference to some examples. See [link to relevant documentation]. Figure 3 Taking the server as the executing entity as an example, the method includes the following steps.

[0100] 301. In response to a model test request for generating a model from SQL statements, the server retrieves multiple target test statements.

[0101] The SQL statement generation model is a text-to-SQL model capable of converting natural language (text) into structured query language (SQL). The base model for the SQL statement generation model is the Large Language Model, which leverages its natural language understanding and language generation capabilities to achieve this conversion. The model test request is used to request testing of the SQL statement generation model, with the purpose of determining the accuracy of the generated SQL statements. The target test statements are natural language statements used to test the SQL statement generation model. Multiple target test statements originate from a test dataset, which can be a dataset designed by technical personnel or a publicly available dataset in the relevant field; this embodiment does not limit the scope of the dataset. In some embodiments, the target test statements are also referred to as target test questions or target test text, etc.

[0102] In one possible implementation, in response to a model test request for generating a model of SQL statements, the server determines the test dataset corresponding to the model test request. The server then retrieves multiple target test statements from this test dataset.

[0103] The server stores multiple candidate test datasets, each containing different test statements. These multiple candidate test datasets are set up by technicians according to their needs, and this application embodiment does not limit this.

[0104] In this implementation, in response to a model test request for the SQL statement generation model, the test dataset corresponding to the model test request is first determined, and then multiple target test statements are obtained from the test dataset so that the multiple target test statements match the model test request, thereby improving the accuracy of the test results for testing the SQL statement generation model.

[0105] To illustrate the above implementation methods, several examples will be provided below.

[0106] Example 1: In response to a model test request that generates a model from an SQL statement, the server obtains a dataset identifier from the model test request. Based on this dataset identifier, the server queries multiple candidate test datasets to obtain the test dataset corresponding to the model test request. The server then obtains multiple target test statements from this test dataset.

[0107] The model test request is sent from the terminal to the server. When the user uses the terminal to test the SQL statement generation model, they can specify the test dataset by selecting a dataset identifier. The server can then use the user-specified test dataset to test the SQL statement generation model.

[0108] In this approach, the model test request carries a dataset identifier, which can be used to find the test dataset corresponding to the model test request, making the determination of the test dataset more efficient.

[0109] Example 2: In response to a model testing request for generating a model from SQL statements, the server determines the model application domain indicated by the test request. Based on this model application domain, the server determines the test dataset corresponding to the model testing request from multiple candidate test datasets. This test dataset belongs to the model application domain. The server then obtains multiple target test statements from this test dataset.

[0110] Here, the model application domain refers to the domain in which the SQL statement generation model will be deployed. The test statements in the candidate test datasets differ across domains. Using the test dataset corresponding to a specific model application domain to test the SQL statement generation model will yield model test results that are more closely matched to that specific domain.

[0111] For example, in response to a model test request that generates a model from SQL statements, the server retrieves a domain identifier from the request. This domain identifier indicates the domain in which the model applies. The server compares this domain identifier with candidate domain identifiers corresponding to multiple candidate test datasets, and obtains a candidate test dataset whose candidate domain identifier matches the requested domain identifier. This candidate test dataset is the test dataset corresponding to the model test request. The server then retrieves multiple target test statements from this test dataset.

[0112] The correspondence between the candidate test dataset and the candidate domain identifier is set by the technicians according to the actual situation, and this application embodiment does not limit this.

[0113] The following explains how the candidate test dataset is constructed.

[0114] In one possible implementation, the server inputs commonly used business queries into an SQL statement generation model to obtain predicted SQL statements. Technical personnel then refine these predicted SQL statements to obtain labeled SQL statements. Alternatively, the server inputs commonly used business metrics into a problem expansion model to generate multiple target test statements. The server then inputs these multiple target test statements into the SQL statement generation model to obtain predicted SQL statements. Technical personnel then refine these predicted SQL statements to obtain labeled SQL statements. Through these methods, a candidate test dataset can be obtained.

[0115] In some embodiments, the server can also expand the candidate test dataset to enrich the content of the target test statements in the candidate test dataset.

[0116] The dataset expansion can be performed using methods from related technologies, such as using a large language model to expand the dataset. This application does not limit this method.

[0117] 302. The server inputs multiple target test statements into the SQL statement generation model, processes the multiple target test statements through the SQL statement generation model, and obtains the test SQL statements for each target test statement.

[0118] The test SQL statement is the SQL statement that the SQL statement generation model transforms into the input target test statement. In this embodiment, the test of the SQL statement generation model is performed in conjunction with the test SQL statement output by the SQL statement generation model.

[0119] In one possible implementation, for any one of the plurality of target test statements, the server performs semantic encoding on the target test statement using the SQL statement generation model to obtain the test statement features of the target test statement. The server then performs multiple rounds of iterative decoding on the test statement features of the target test statement using the SQL statement generation model to obtain the test SQL statement for the target test statement.

[0120] Semantic encoding is the process of extracting semantic features. The features of this test statement are an abstract expression of the target test statement, which facilitates subsequent processing by the server. Multi-round iterative decoding is used to obtain the complete test SQL statement. Each round of iterative decoding yields one character, and the multiple characters obtained from multiple rounds of iterative decoding constitute the test SQL statement.

[0121] In this implementation, the target test statement is semantically encoded using an SQL statement generation model to obtain its test statement features. These features are then iteratively decoded multiple times by the SQL statement generation model to obtain the test SQL statement. This achieves the goal of using the natural language understanding capabilities of the SQL statement generation model to convert the target test statement into a test SQL statement.

[0122] For example, for any one of the multiple target test statements, the server splits the target test statement into multiple word groups. The server performs embedding encoding on these multiple word groups to obtain the word group embedding encoding for each word group. The server uses this SQL statement generation model, based on an attention mechanism, to encode the multiple word group embedding encodings of the target test statement, obtaining the test statement features of the target test statement. The server uses this SQL statement generation model, based on an attention mechanism, to perform multiple rounds of iterative decoding on the test statement features of the target test statement, obtaining the test SQL statement of the target test statement.

[0123] Another implementation of step 302 described above will be described below.

[0124] In one possible implementation, for any one of the plurality of target test statements, the server semantically encodes the target test statement using the SQL statement generation model to obtain the test statement features of the target test statement. The server then iteratively decodes the test statement features of the target test statement using the SQL statement generation model to obtain the SQL framework and SQL content. Finally, the server fills the SQL content into the SQL framework to obtain the test SQL statement for the target test statement.

[0125] The SQL frame is a SQL statement to be populated. It includes the table name, field names, and aggregation method to be populated. The SQL content includes the table name, field names, and aggregation method. Populating the SQL content into the SQL frame yields the target test statement.

[0126] In this implementation, the target test statement is semantically encoded using an SQL statement generation model to obtain its test statement features. These features are then iteratively decoded multiple times using the SQL statement generation model to obtain the SQL framework and SQL content. This two-part approach generates the target test statement, resulting in a highly interpretable output.

[0127] For example, for any one of the multiple target test statements, the server splits the target test statement into multiple phrases. The server performs embedding encoding on these phrases to obtain the phrase embedding encoding for each phrase. The server uses the SQL statement generation model, based on an attention mechanism, to encode the multiple phrase embedding encodings of the target test statement, obtaining the test statement features of the target test statement. The server uses the first decoder of the SQL statement generation model, based on an attention mechanism, to perform multiple rounds of iterative decoding on the test statement features of the target test statement, obtaining the SQL framework of the target test statement. The server uses the first decoder of the SQL statement generation model, based on an attention mechanism, to perform multiple rounds of iterative decoding on the test statement features of the target test statement, obtaining the SQL content of the target test statement. The server fills the SQL content into the SQL framework to obtain the test SQL statement of the target test statement.

[0128] 303. The server executes the test SQL statements and annotation SQL statements for each target test statement, and obtains the first execution result and the second execution result corresponding to each target test statement. The first execution result is the execution result of the test SQL statement, and the second execution result is the execution result of the annotation SQL statement.

[0129] The test dataset includes multiple target test statements and corresponding labeled SQL statements, which are the actual SQL statements corresponding to each target test statement. These labeled SQL statements serve as the benchmark for evaluating the SQL statement generation model. Executing an SQL statement refers to querying the database using the SQL statement to obtain the corresponding query data, which is the execution result of the SQL statement. Accordingly, the first execution result is the first query data obtained by the test SQL statement in the database, and the second execution result is the second query data obtained by the labeled SQL statement in the database. In step 302 above, for any target test statement among the multiple target test statements, executing the test SQL statement and the labeled SQL statement involves querying the same database using the test SQL statement and the labeled SQL statement.

[0130] In one possible implementation, for any one of the plurality of target test statements, the server executes the test SQL statement and the annotation SQL statement of the target test statement in the target database respectively, and obtains the first execution result and the second execution result corresponding to the target test statement. The first execution result and the second execution result both include the field values ​​recalled from the target database.

[0131] The target database corresponds to multiple test statements, and the query data for these statements is stored in this database, which is maintained by the server. This field value belongs to the aforementioned query data. Obtaining the first and second execution results is significant because it reveals the actual execution results of the test SQL statements and the labeled SQL statements in a real business scenario, helping to evaluate the consistency between them. Generally, if the consistency between the test SQL statements and the labeled SQL statements is high, their actual execution results are the same or very close; if the consistency is low, their actual execution results differ significantly.

[0132] In this implementation, by executing the test SQL statement and the annotation SQL statement of the target test statement in the target database, the first execution result and the second execution result corresponding to the target test statement can be obtained, thereby obtaining the actual execution results of the test SQL statement and the annotation SQL statement, which is helpful for subsequent testing of the SQL statement generation model.

[0133] For example, for any one of the multiple target test statements, the server calls the target database's database query interface to send a first data query request and a second data query request to the target database. The first data query request requests a query based on the test SQL statement, and the second data query request requests a query based on the labeled SQL statement. The server uses the database query interface to obtain the first execution result corresponding to the first data query request and the second execution result corresponding to the second data query request.

[0134] The method of querying the target database using SQL statements can be any of the methods described in related technologies, and this application does not limit this method.

[0135] 304. The server determines the structural differences between the test SQL statements and the annotation SQL statements of each target test statement, as well as the differences between the execution results of the first execution result and the second execution result.

[0136] Among them, the structural difference information refers to the difference information in the statement structure of the SQL statement, and the execution result difference information refers to the difference information in the execution result of the SQL statement, that is, the difference information between the query data corresponding to the SQL statement.

[0137] In one possible implementation, the server compares the test SQL statements and annotation SQL statements of each target test statement to obtain structural difference information between the test SQL statements and annotation SQL statements of each target test statement. The server compares the first execution result and the second execution result of each target test statement to obtain execution result difference information between the first execution result and the second execution result of each target test statement.

[0138] To provide a clearer explanation of the above embodiments, the following description is divided into two parts.

[0139] Part 1 explains the methods for determining structural difference information.

[0140] In one possible implementation, for any one of the plurality of target test statements, the server compares the table names, field names, and aggregation methods of the data tables indicated in the test SQL statement and the annotation SQL statement, respectively, to obtain table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement. The server then concatenates the table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement to obtain structural difference information between the test SQL statement and the annotation SQL statement.

[0141] The table differences between the test SQL statement and the annotation SQL statement refer to the differences in table names. If the table names in the test SQL statement and the annotation SQL statement are either the same or different, then the table difference information is either "there is a difference" or "there is no difference." The field differences between the test SQL statement and the annotation SQL statement refer to the differences in field names. If the field names in the test SQL statement and the annotation SQL statement are either the same or different, then the field difference information is either "there is a difference" or "there is no difference." Aggregation methods include summation, averaging, and counting. If the aggregation methods in the test SQL statement and the annotation SQL statement are either the same or different, then the aggregation method difference information is either "there is a difference" or "there is no difference." In some embodiments, the above process can be implemented using an SQL structure analysis model.

[0142] In some embodiments, if the table names in the test SQL statement and the annotation SQL statement are the same, 5 points are awarded; otherwise, 0 points are awarded. If the field names in the test SQL statement and the annotation SQL statement are the same, 2 points are awarded; otherwise, 0 points are awarded. If the aggregation methods in the test SQL statement and the annotation SQL statement are the same, 1 point is awarded; otherwise, 0 points are awarded. The specific scores mentioned above are set by the technicians according to the actual situation, and this application embodiment does not limit them.

[0143] Part Two explains the methods for determining differences in execution results.

[0144] In one possible implementation, for any one of the plurality of target test statements, the server determines the target field name in the second execution result corresponding to that target test statement. The server uses this target field name to perform field identification in the first execution result corresponding to the target test statement, obtaining a reference field name in the first execution result. This reference field name is the field that matches the target field name. Based on the field value of the reference field name in the first execution result corresponding to the target test statement and the field value of the target field name in the second execution result corresponding to the target test statement, the server determines the execution result difference information corresponding to the target test statement.

[0145] The second execution result includes field names and field values. The field names indicate the meaning of the field values; for example, the field name is "order acceptance rate" and the field value is "30%". The target field name is the field name in the second execution result. There may be one or more such field names. For ease of understanding, the following explanation will use the example of the second execution result including one field name.

[0146] To provide a clearer explanation of the above implementation methods, the following sections will further explain the methods for determining the reference field name and the difference information in the execution results in the above implementation methods.

[0147] A. The server uses the target field name to identify the field in the first execution result corresponding to the target test statement, and obtains the reference field name in the first execution result.

[0148] In one possible implementation, the server performs semantic encoding on the target field name to obtain the field name characteristics of the target field name. The server performs semantic encoding on at least one candidate field name in the first execution result corresponding to the target test statement to obtain the field name characteristics of the at least one candidate field name. Based on the field name characteristics of the target field name and the field name characteristics of the at least one candidate field name, the server determines the reference field name from the at least one candidate field name.

[0149] Semantic encoding of field names aims to obtain their semantic features; correspondingly, these features are the semantic characteristics of the field names. Using these features to match target and candidate field names allows us to identify field names that differ in form but are essentially the same, thus eliminating test result errors caused by formal discrepancies. For example, if the target field name in the second execution result is "Beijing's order acceptance rate yesterday," while the reference field name in the first execution result is also "Beijing's order acceptance rate yesterday," the target and reference field names actually express the same meaning and should be considered the same field name.

[0150] For example, the server encodes the target field name using an attention mechanism to obtain the field name feature of the target field name. The server then encodes at least one candidate field name in the first execution result corresponding to the target test statement using the attention mechanism to obtain the field name feature of the at least one candidate field name. The server determines the feature similarity between the field name feature of the target field name and the field name features of the at least one candidate field name. The server then determines the candidate field name with the highest feature similarity among the at least one candidate field names as the reference field name.

[0151] Since the first execution result is obtained by executing the test SQL statement, which is generated by the SQL statement generation model, the first execution result may contain many candidate field names. This means that other field names associated with the target field name are also recalled. To evaluate the test SQL statement, it is necessary to filter out reference field names that match the target field name from these candidate field names. For example, the second execution result may only include the field name "order acceptance rate," while the first execution result includes "order volume (numerator of order acceptance rate)," "total order volume (denominator of order acceptance rate)," and "order acceptance rate." Although this redundant information is not completely consistent with the second execution result, the first execution result is actually correct. Therefore, by using the above method, "order acceptance rate" can be filtered out from the first execution result as a reference field name for subsequent comparison.

[0152] B. The server determines the difference information of the execution results corresponding to the target test statement based on the field value of the reference field name in the first execution result corresponding to the target test statement and the field value of the target field name in the second execution result corresponding to the target test statement.

[0153] In one possible implementation, the execution result difference information includes null value percentage difference information and field value difference information. The server determines the first null value percentage of the first execution result corresponding to the target test statement and the second null value percentage of the second execution result corresponding to the target test statement. The server determines the difference between the first null value percentage and the second null value percentage as the null value percentage difference information between the first and second execution results corresponding to the target test statement. The server determines the result deviation value and the result deviation ratio between the first and second execution results corresponding to the target test statement. The server determines the null value percentage difference information, the result deviation value, and the result deviation ratio as the field value difference information between the first and second execution results corresponding to the target test statement.

[0154] The percentage of null values ​​reflects the amount of invalid information. For example, if the first execution result is "Order Quantity: 00030", then the percentage of null values ​​is 3 (the number of zeros before the 3) / 5 (the total number of digits) = 0.6. The result deviation is the difference between field values. For example, if one execution result has a field value of 3 and another has a field value of 5, then the result deviation is 5 - 3 = 2. The result deviation ratio is the percentage of deviation between field values. For example, if one execution result has a field value of 3 and another has a field value of 5, then the result deviation ratio is (5 - 3) / 5 * 100% = 40%. In some embodiments, the above process can be implemented using an SQL result analysis model.

[0155] Optionally, after step 304, steps 305 or 306 can also be performed, but this embodiment does not limit the scope of the application.

[0156] 305. Based on the structural differences between the test SQL statements and the labeled SQL statements of each target test statement, as well as the differences between the execution results of the first execution result and the second execution result, the server determines the model test result of the SQL statement generation model. The model test result is used to represent the accuracy of the SQL statement generated by the SQL statement generation model.

[0157] The smaller the structural and execution result differences between the test SQL statement and the labeled SQL statement, the closer the test SQL statement is to the labeled SQL statement, and the higher the accuracy of the SQL statement generated by the SQL statement generation model. Conversely, the larger the structural or execution result differences between the test SQL statement and the labeled SQL statement, the greater the difference between the test SQL statement and the labeled SQL statement, and the lower the accuracy of the SQL statement generated by the SQL statement generation model.

[0158] In one possible implementation, the server determines the statement structure test result of the SQL statement generation model based on the structural differences between the test SQL statements and the labeled SQL statements for each target test statement. The server determines the statement execution test result of the SQL statement generation model based on the execution result differences between the first execution result and the second execution result. The server determines the model test result based on the statement structure test result and the statement execution test result.

[0159] The results of the statement structure test and the statement execution test can both be represented by scores. Similarly, the model test results can also be represented by scores. The higher the score, the higher the accuracy of the SQL statement generated by the SQL statement generation model; the lower the score, the lower the accuracy of the SQL statement generated by the SQL statement generation model.

[0160] To provide a clearer explanation of the above embodiments, the following description is divided into several parts.

[0161] Part 1: Based on the structural differences between the test SQL statements and the labeled SQL statements of each target test statement, the server determines the statement structure test results of the SQL statement generation model.

[0162] In one possible implementation, for any one of the multiple target test statements, the server determines the structural test score corresponding to that target test statement based on the table differences, field differences, and aggregation method differences between the test SQL statement and the annotation SQL statement. The server then determines the average of the structural test scores corresponding to the multiple target test statements as the statement structure test result of the SQL statement generation model.

[0163] For example, for any one of the multiple target test statements, the server determines a table difference score based on the table difference information between the test SQL statement and the annotation SQL statement. The server determines a field difference score based on the field difference information between the test SQL statement and the annotation SQL statement. The server determines an aggregation method difference score based on the aggregation method difference information between the test SQL statement and the annotation SQL statement. The server then performs a weighted fusion of the table difference score, the field difference score, and the aggregation method difference score to obtain the structural test score corresponding to the target test statement. The server then uses the average of the structural test scores corresponding to the multiple target test statements as the statement structure test result of the SQL statement generation model.

[0164] The correspondence between the difference information and the difference score, as well as the weight of the weighted fusion, shall be set by the technicians according to the actual situation, and this application embodiment does not limit this.

[0165] Part Two: Based on the difference information between the first execution result and the second execution result, the server determines the statement execution test result of the SQL statement generation model.

[0166] In one possible implementation, the execution result difference information includes null value percentage difference information and field value difference information. For any target test statement among the plurality of target test statements, the server determines the statement execution test score corresponding to the target test statement based on the null value percentage difference information and field value difference information between the first execution result and the second execution result. The server determines the average of the statement execution test scores corresponding to the plurality of target test statements as the statement execution test result of the SQL statement generation model.

[0167] For example, for any one of the multiple target test statements, the server determines a duty cycle difference score based on the difference in the proportion of null values ​​between the first execution result and the second execution result. The server then determines a field value difference score based on the difference in field values ​​between the first and second execution results. Finally, the server performs a weighted fusion of the duty cycle difference score and the field value difference score to obtain the statement execution test score corresponding to that target test statement.

[0168] The correspondence between the difference information and the difference score, as well as the weight of the weighted fusion, shall be set by the technicians according to the actual situation, and this application embodiment does not limit this.

[0169] Part Three: Based on the test results of the statement structure and the test results of the statement execution, the server determines the test results of the model.

[0170] In one possible implementation, the server performs a weighted fusion of the structure test score corresponding to the statement structure test result and the statement execution test score corresponding to the statement execution test result to obtain the model test score, which is used to represent the model test result.

[0171] The weights for weighted fusion are set by technical personnel according to the actual situation, and this application embodiment does not limit this.

[0172] 306. The server obtains the model performance parameters when generating test SQL statements for each target test statement from the SQL statement generation model. These model performance parameters include the model response time and the number of exception retries.

[0173] The model response time (RT) is the time it takes for the SQL statement generation model to output the test SQL statement from receiving the target test statement. The model response time can be the average of the processing time for processing multiple target test statements.

[0174] 307. Based on the model performance parameters, the structural differences between the test SQL statements and the labeled SQL statements of each target test statement, and the differences between the execution results of the first execution result and the second execution result, the server determines the model test result of the SQL statement generating the model.

[0175] In one possible implementation, the server determines a model performance score based on the model performance parameters. The server determines a structure test score and a statement execution test score based on structural differences between the test SQL statements and labeled SQL statements for each target test statement, as well as execution result differences between the first execution result and the second execution result. The server then weights and merges the model performance score, the structure test score, and the statement execution test score to obtain the model test score, which represents the model test result.

[0176] The method for determining the structure test score and the statement execution test score described above belongs to the same inventive concept as described in step 305 above. The implementation process is described in the relevant description of step 305 above, and will not be repeated here.

[0177] To provide a clearer explanation of the technical solutions provided in the embodiments of this application, the following is combined with... Figure 4 The technical solutions provided in the embodiments of this application will be described.

[0178] See Figure 4Multiple target test statements are obtained from the test dataset. These target test statements are then input into an SQL statement generation model to generate multiple test SQL statements. Each test SQL statement and its corresponding labeled SQL statement are executed in the target database to obtain the first execution result and the second execution result of each labeled SQL statement. The first and second execution results are then input into an SQL result analysis model to obtain the null value percentage difference information, the result deviation value, and the result deviation ratio for each target test statement. The multiple test SQL statements and their corresponding labeled SQL statements are then input into an SQL structure analysis model to obtain the table difference information, field difference information, and aggregation method difference information for each target test statement. Model performance parameters, including model response time and the number of abnormal retries, are obtained from the service logs of the SQL statement generation model. The above null value percentage difference information, result deviation value, result deviation ratio, table difference information, field difference information, aggregation method difference information, model response time, and number of abnormal retries are aggregated to obtain the model test results.

[0179] Additionally, during the construction of the test dataset, commonly used business queries can be input into the SQL statement generation model to obtain predicted SQL statements. Technical personnel can then refine these predicted SQL statements to obtain labeled SQL statements. Alternatively, commonly used business metrics can be input into the question to expand the model and generate multiple target test statements. These target test statements can then be input into the SQL statement generation model to obtain predicted SQL statements. Technical personnel can then refine these predicted SQL statements to obtain labeled SQL statements. Through these methods, a test dataset can be obtained.

[0180] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0181] The technical solution provided in this application, in response to a model testing request for an SQL statement generation model, converts multiple target test statements into multiple corresponding test SQL statements using the SQL statement generation model. The test SQL statements and annotated SQL statements of each target test statement are executed to obtain a first execution result of the test SQL statements and a second execution result of the annotated SQL statements, thereby increasing the testing at the SQL statement execution level. The model test result is determined by combining the structural differences between the test SQL statements and annotated SQL statements of each target test statement, as well as the differences in execution results between the first and second execution results. This approach considers not only the structural differences in SQL statements but also the differences in their execution results, resulting in highly accurate model test results.

[0182] Figure 5 This is a schematic diagram of a test device structure for an SQL statement generation model provided in an embodiment of this application. See also... Figure 5 The device includes: a processing module 501, an execution module 502, and a test module 503.

[0183] The processing module 501 is used to respond to the model test request of the SQL statement generation model, input multiple target test statements into the SQL statement generation model, process the multiple target test statements through the SQL statement generation model, and obtain the test SQL statements of each target test statement.

[0184] The execution module 502 is used to execute the test SQL statements and annotation SQL statements of each target test statement, and obtain the first execution result and the second execution result corresponding to each target test statement. The first execution result is the execution result of the test SQL statement, and the second execution result is the execution result of the annotation SQL statement.

[0185] The testing module 503 is used to determine the model test result of the SQL statement generation model based on the structural difference information between the test SQL statement and the labeled SQL statement of each target test statement, as well as the difference information between the execution result of the first execution result and the second execution result. The model test result is used to represent the accuracy of the SQL statement generated by the SQL statement generation model.

[0186] In one possible implementation, the processing module 501 is configured to, for any one of the plurality of target test statements, perform semantic encoding on the target test statement using the SQL statement generation model to obtain the test statement features of the target test statement. Then, the SQL statement generation model is used to perform multiple rounds of iterative decoding on the test statement features of the target test statement to obtain the test SQL statement for the target test statement.

[0187] In one possible implementation, the execution module 502 is configured to execute the test SQL statement and the annotation SQL statement of any one of the plurality of target test statements in the target database respectively, and obtain a first execution result and a second execution result corresponding to the target test statement. The first execution result and the second execution result both include field values ​​recalled from the target database.

[0188] In one possible implementation, the testing module 503 is used to determine the statement structure test result of the SQL statement generation model based on the structural difference information between the test SQL statements and the labeled SQL statements of each target test statement. Based on the execution result difference information between the first execution result and the second execution result, the module determines the statement execution test result of the SQL statement generation model. Finally, based on the statement structure test result and the statement execution test result, the module determines the model test result.

[0189] In one possible implementation, the structural difference information includes table difference information, field difference information, and aggregation method difference information. The testing module 503 is used to determine the structural test score corresponding to any one of the plurality of target test statements based on the table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of that target test statement. The average of the structural test scores corresponding to the plurality of target test statements is determined as the statement structure test result of the SQL statement generation model.

[0190] In one possible implementation, the apparatus further includes a structural difference information determination module, configured to, for any one of the plurality of target test statements, compare the table name, field name, and aggregation method of the data table indicated in the test SQL statement and the annotation SQL statement of the target test statement, respectively, to obtain table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of the target test statement. The table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of the target test statement are then concatenated to obtain the structural difference information between the test SQL statement and the annotation SQL statement of the target test statement.

[0191] In one possible implementation, the execution result difference information includes null value percentage difference information and field value difference information. The test module 503 is used to determine the statement execution test score corresponding to any one of the plurality of target test statements based on the null value percentage difference information and field value difference information between the first execution result and the second execution result. The average of the statement execution test scores corresponding to the plurality of target test statements is determined as the statement execution test result of the SQL statement generation model.

[0192] In one possible implementation, the apparatus further includes an execution result difference information determination module, configured to, for any one of the plurality of target test statements, determine the target field name in the second execution result corresponding to the target test statement, wherein both the first execution result and the second execution result include field values ​​retrieved from the target database. The target field name is used to perform field identification in the first execution result corresponding to the target test statement to obtain a reference field name in the first execution result, the reference field name being a field that matches the target field name. Based on the field value of the reference field name in the first execution result corresponding to the target test statement and the field value of the target field name in the second execution result corresponding to the target test statement, execution result difference information corresponding to the target test statement is determined.

[0193] In one possible implementation, the execution result difference information determination module is used to semantically encode the target field name to obtain the field name features of the target field name. It also performs semantic encoding on at least one candidate field name in the first execution result corresponding to the target test statement to obtain the field name features of the at least one candidate field name. Based on the field name features of the target field name and the field name features of the at least one candidate field name, it determines the reference field name from the at least one candidate field name.

[0194] In one possible implementation, the execution result difference information includes null value percentage difference information and field value difference information. The execution result difference information determination module is used to determine the first null value percentage of the first execution result corresponding to the target test statement and the second null value percentage of the second execution result corresponding to the target test statement. The difference between the first null value percentage and the second null value percentage is determined as the null value percentage difference information between the first and second execution results corresponding to the target test statement. The module also determines the result deviation value and result deviation ratio between the first and second execution results corresponding to the target test statement, and the field value difference information includes the result deviation value and the result deviation ratio. The null value percentage difference information, the result deviation value, and the result deviation ratio are determined as the field value difference information between the first and second execution results corresponding to the target test statement.

[0195] In one possible implementation, the apparatus further includes: a performance parameter acquisition module, used to acquire model performance parameters of the SQL statement generation model when generating test SQL statements for each target test statement, the model performance parameters including model response time and number of exception retries.

[0196] The test module 503 is also used to determine the model test result of the SQL statement generating model based on the model performance parameters, the structural difference information between the test SQL statements and the labeled SQL statements of each target test statement, and the execution result difference information between the first execution result and the second execution result.

[0197] It should be noted that the testing device for the SQL statement generation model provided in the above embodiments is only used as an example to illustrate the division of the above functional modules when testing the SQL statement generation model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the testing device for the SQL statement generation model provided in the above embodiments and the testing method embodiments for the SQL statement generation model belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0198] The technical solution provided in this application, in response to a model testing request for an SQL statement generation model, converts multiple target test statements into multiple corresponding test SQL statements using the SQL statement generation model. The test SQL statements and annotated SQL statements of each target test statement are executed to obtain a first execution result of the test SQL statements and a second execution result of the annotated SQL statements, thereby increasing the testing at the SQL statement execution level. The model test result is determined by combining the structural differences between the test SQL statements and annotated SQL statements of each target test statement, as well as the differences in execution results between the first and second execution results. This approach considers not only the structural differences in SQL statements but also the differences in their execution results, resulting in highly accurate model test results.

[0199] This application provides a computer device for performing the above-described method. This computer device can be implemented as a terminal or a server. The structure of the terminal will be described below:

[0200] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal 600 can be a smartphone, tablet computer, laptop computer, or desktop computer. The terminal 600 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0201] Typically, terminal 600 includes one or more processors 601 and one or more memories 602.

[0202] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0203] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one computer program, which is executed by the processor 601 to implement the test method of the SQL statement generation model provided in the method embodiments of this application.

[0204] In some embodiments, the terminal 600 may optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, and a power supply 608.

[0205] Peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0206] The radio frequency (RF) circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc.

[0207] Display screen 605 is used to display a user interface (UI). This UI may include graphics, text, icons, video, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard.

[0208] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal.

[0209] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 601 for processing, or input to the radio frequency circuit 604 to realize voice communication.

[0210] Power supply 608 is used to supply power to the various components in terminal 600. Power supply 608 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0211] In some embodiments, the terminal 600 further includes one or more sensors 609. The one or more sensors 609 include, but are not limited to, an accelerometer 610, a gyroscope 611, a pressure sensor 612, an optical sensor 613, and a proximity sensor 614.

[0212] Accelerometer 610 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established with terminal 600.

[0213] The gyroscope sensor 611 can detect the orientation and rotation angle of the terminal 600. The gyroscope sensor 611 can work in conjunction with the accelerometer sensor 610 to collect the user's 3D movements on the terminal 600.

[0214] The pressure sensor 612 can be installed on the side bezel of the terminal 600 and / or on the lower layer of the display screen 605. When the pressure sensor 612 is installed on the side bezel of the terminal 600, it can detect the user's grip signal on the terminal 600, and the processor 601 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 612. When the pressure sensor 612 is installed on the lower layer of the display screen 605, the processor 601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 605.

[0215] An optical sensor 613 is used to collect ambient light intensity. In one embodiment, a processor 601 can control the display brightness of a display screen 605 based on the ambient light intensity collected by the optical sensor 613.

[0216] The proximity sensor 614 is used to detect the distance between the user and the front of the terminal 600.

[0217] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on terminal 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0218] The aforementioned computer equipment can also be implemented as a server. The structure of a server is described below:

[0219] Figure 7This is a schematic diagram of a server structure provided in an embodiment of this application. The server 700 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 701 and one or more memories 702. The one or more memories 702 store at least one computer program, which is loaded and executed by the one or more processors 701 to implement the methods provided in the various method embodiments described above. Of course, the server 700 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 700 may also include other components for implementing device functions, which will not be elaborated upon here.

[0220] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to complete the testing method for the SQL statement generation model in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0221] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to execute the test method of the SQL statement generation model described above.

[0222] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0223] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0224] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A testing method for an SQL statement generation model, characterized in that, The method includes: In response to a model test request for the SQL statement generation model, multiple target test statements are input into the SQL statement generation model, and the multiple target test statements are processed by the SQL statement generation model to obtain test SQL statements for each target test statement. Execute the test SQL statement and annotation SQL statement for each of the target test statements to obtain the first execution result and the second execution result corresponding to each of the target test statements. The first execution result is the execution result of the test SQL statement, and the second execution result is the execution result of the annotation SQL statement. Based on the structural differences between the test SQL statements and the labeled SQL statements of each target test statement, as well as the differences between the execution results of the first execution result and the second execution result, the model test result of the SQL statement generation model is determined. The model test result is used to represent the accuracy of the SQL statements generated by the SQL statement generation model.

2. The method according to claim 1, characterized in that, The process of processing the multiple target test statements through the SQL statement generation model to obtain the test SQL statements for each target test statement includes: For any target test statement among the plurality of target test statements, the target test statement is semantically encoded using the SQL statement generation model to obtain the test statement features of the target test statement; The test SQL statement of the target test statement is obtained by performing multiple rounds of iterative decoding on the test statement features of the target test statement through the SQL statement generation model.

3. The method according to claim 1, characterized in that, The execution of the test SQL statements and annotation SQL statements for each of the target test statements yields a first execution result and a second execution result corresponding to each of the target test statements, including: For any one of the plurality of target test statements, the test SQL statement and the annotation SQL statement of the target test statement are executed in the target database respectively to obtain the first execution result and the second execution result corresponding to the target test statement. Both the first execution result and the second execution result include the field values ​​recalled from the target database.

4. The method according to claim 1, characterized in that, The model test result of the SQL statement generation model is determined based on the structural difference information between the test SQL statements and the labeled SQL statements of each of the target test statements, as well as the execution result difference information between the first execution result and the second execution result. This includes: Based on the structural differences between the test SQL statements and the labeled SQL statements of each target test statement, the statement structure test results of the SQL statement generation model are determined. Based on the difference information between the first execution result and the second execution result, the statement execution test result of the SQL statement generation model is determined; The test results of the model are determined based on the test results of the statement structure and the test results of the statement execution.

5. The method according to claim 4, characterized in that, The structural difference information includes table difference information, field difference information, and aggregation method difference information. The determination of the statement structure test result of the SQL statement generation model based on the structural difference information between the test SQL statements and the labeled SQL statements for each of the target test statements includes: For any target test statement among the plurality of target test statements, the structure test score corresponding to the target test statement is determined based on the table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of the target test statement; The average of the structure test scores corresponding to the multiple target test statements is determined as the statement structure test result of the SQL statement generation model.

6. The method according to claim 1 or 5, characterized in that, The method for determining the structural difference information includes: For any one of the multiple target test statements, compare the table name, field name, and aggregation method of the data table indicated in the test SQL statement and the annotation SQL statement of the target test statement, respectively, to obtain the table difference information, field difference information, and aggregation method difference information between the test SQL statement and the annotation SQL statement of the target test statement; By concatenating the table differences, field differences, and aggregation method differences between the test SQL statement and the annotation SQL statement of the target test statement, the structural differences between the test SQL statement and the annotation SQL statement of the target test statement are obtained.

7. The method according to claim 4, characterized in that, The execution result difference information includes null value percentage difference information and field value difference information. Determining the statement execution test result of the SQL statement generation model based on the execution result difference information between the first execution result and the second execution result includes: For any target test statement among the plurality of target test statements, the statement execution test score corresponding to the target test statement is determined based on the difference information of the proportion of null values ​​and the difference information of field values ​​between the first execution result and the second execution result; The average of the statement execution test scores corresponding to the multiple target test statements is determined as the statement execution test result of the SQL statement generation model.

8. A testing device for an SQL statement generation model, characterized in that, The device includes: The processing module is used to respond to a model test request for the SQL statement generation model, input multiple target test statements into the SQL statement generation model, process the multiple target test statements through the SQL statement generation model, and obtain test SQL statements for each target test statement. The execution module is used to execute the test SQL statements and annotation SQL statements of each of the target test statements, and obtain the first execution result and the second execution result corresponding to each of the target test statements. The first execution result is the execution result of the test SQL statement, and the second execution result is the execution result of the annotation SQL statement. The testing module is used to determine the model test result of the SQL statement generation model based on the structural difference information between the test SQL statement and the labeled SQL statement of each target test statement, as well as the difference information between the execution result of the first execution result and the second execution result. The model test result is used to represent the accuracy of the SQL statement generated by the SQL statement generation model.

9. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the test method for the SQL statement generation model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the test method for the SQL statement generation model as described in any one of claims 1 to 7.

Images