Structured query language migration test method and device and storage medium

Abstract

The invention relates to a structured query language migration test method and device and a storage medium, the method can extract configuration information based on a pre-migration structured query language application scene and a post-migration structured query language application scene, the configuration information comprises a control group configuration and at least one experimental group configuration, and the experimental group configuration comprises at least one experimental group configuration. Therefore, the to-be-migrated target structured query language can be executed according to the configuration information, a first execution result corresponding to the control group configuration and a second execution result corresponding to the experimental group configuration are obtained, and then a test result of structured query language migration is determined according to the first execution result and the second execution result. A large amount of migration development work does not need to be repeated every time there is an SQL migration demand, and the efficiency of SQL task migration is improved.

Description

Technical Field

[0001] The present application relates to the technical field of migration testing, and in particular to a structured query language migration testing method, device and storage medium. Background Art

[0002] With the development of Internet big data, the technology of online analytical processing (OLAP) engines and computing engines is changing very quickly. The introduction of new technologies, such as engine changes, engine version upgrades, and the introduction of optimization parameters, requires the migration of existing big data tasks. Due to the different characteristics of each engine, different usage scenarios, and even different Structured Query Language (SQL) dialects, the traditional solution is to do customized development and simulate dual-running, but in fact the general process framework of each development is similar, and a lot of development work is repeated. Summary of the invention

[0003] The present application provides a structured query language migration test method, device and storage medium to solve the problem that each SQL migration requirement requires customized development.

[0004] In a first aspect, the present application provides a structured query language migration testing method, the method comprising:

[0005] Extracting configuration information based on migration scenario information; wherein the migration scenario information is used to characterize the structured query language application scenario before migration and the structured query language application scenario after migration; the configuration information includes a control group configuration and at least one experimental group configuration;

[0006] Executing the target structured query language to be migrated according to the configuration information, obtaining a first execution result corresponding to the control group configuration, and obtaining a second execution result corresponding to the experimental group configuration;

[0007] A test result of the structured query language migration is determined according to the first execution result and the second execution result.

[0008] Optionally, extracting configuration information based on the migration scenario information includes:

[0009] Obtain migration scenario information;

[0010] Determine the original configuration before the migration and the scenario configuration of the target migration scenario according to the migration scenario information; wherein the original configuration before the migration includes the structured query language application scenario before the migration; and the scenario configuration of the target migration scenario includes the structured query language application scenario after the migration;

[0011] Determining the control group configuration according to the original configuration;

[0012] The experimental group configuration is determined according to the scenario configuration of the target migration scenario.

[0013] Optionally, determining the original configuration before migration and the scenario configuration of the target migration scenario according to the migration scenario information includes:

[0014] An experiment table and multiple configuration tables corresponding to the experiment table are extracted according to the migration scenario information; wherein the experiment table includes an experiment table identifier, and the experiment identifiers of the multiple configuration tables are the same as the experiment table identifier; the configuration table includes an original configuration before migration and a scenario configuration of at least one target migration scenario.

[0015] Optionally, executing the target structured query language to be migrated according to the configuration information to obtain a first execution result corresponding to the control group configuration and obtaining a second execution result corresponding to the experimental group configuration includes:

[0016] Parsing the target structured query language into an abstract syntax tree;

[0017] Rewriting the abstract syntax tree to obtain a target abstract syntax tree;

[0018] Converting the target abstract syntax tree into a first structured query language text based on a first structured query language engine configured for the control group, and converting the target abstract syntax tree into a second structured query language text based on a second structured query language engine configured for the experimental group;

[0019] Executing the first structured query language text according to the control group configuration to obtain a first temporary result table;

[0020] Execute the second structured query language text according to the experimental group configuration to obtain a second temporary result table;

[0021] The first temporary result table and the second temporary result table are verified. If the verification passes, the first temporary result table is determined to be the first execution result, and the second temporary result table is determined to be the second execution result.

[0022] Optionally, rewriting the abstract syntax tree to obtain a target abstract syntax tree includes:

[0023] Obtaining the node type of the parent node of the abstract syntax tree;

[0024] If the node type of the parent node of the abstract syntax tree is a target selection node, then adding a creation table to the parent node of the target selection node as a selection CTAS node;

[0025] If the node type of the parent node of the abstract syntax tree is a target insertion node, the target insertion node is replaced with a creation table as a selection CTAS node;

[0026] The rewritten abstract syntax tree is used as the target abstract syntax tree.

[0027] Optionally, verifying the first temporary result table and the second temporary result table includes:

[0028] Obtaining a first row number and a first cyclic redundancy check value of the first temporary result table;

[0029] Obtain a second row number and a second cyclic redundancy check value of the second temporary result table;

[0030] If the first row number is the same as the second row number, and the first cyclic redundancy check value is the same as the second cyclic redundancy check value, the check passes; otherwise, the check fails.

[0031] Optionally, the method further includes: after the first temporary result table and / or the second temporary result table is saved for more than a preset time period, deleting the first temporary result table and / or the second temporary result table.

[0032] In a second aspect, the present application provides a structured query language migration testing device, the device comprising:

[0033] An extraction module, configured to extract configuration information based on migration scenario information; wherein the migration scenario information is used to characterize a structured query language application scenario before migration and a structured query language application scenario after migration; and the configuration information includes a control group configuration and at least one experimental group configuration;

[0034] An execution module, configured to execute the target structured query language to be migrated according to the configuration information, obtain a first execution result corresponding to the control group configuration, and obtain a second execution result corresponding to the experimental group configuration;

[0035] A determination module is used to determine a test result of the structured query language migration according to the first execution result and the second execution result.

[0036] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0037] Memory, used to store computer programs;

[0038] The processor is used to implement the structured query language migration testing method described in any one of the embodiments of the first aspect when executing the program stored in the memory.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the structured query language migration testing method as described in any one of the embodiments of the first aspect is implemented.

[0040] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages compared with the prior art: the method provided by the embodiment of the present application extracts configuration information based on migration scenario information; wherein the migration scenario information is used to characterize the structured query language application scenario before migration and the structured query language application scenario after migration; the configuration information includes a control group configuration and at least one experimental group configuration; the target structured query language to be migrated is executed according to the configuration information, and a first execution result corresponding to the control group configuration is obtained, and a second execution result corresponding to the experimental group configuration is obtained; and the test result of the structured query language migration is determined according to the first execution result and the second execution result. The method can extract configuration information based on the structured query language application scenario before migration and the structured query language application scenario after migration, wherein the configuration information includes a control group configuration and at least one experimental group configuration, so that the target structured query language to be migrated can be executed according to the configuration information, and a first execution result corresponding to the control group configuration and a second execution result corresponding to the experimental group configuration can be obtained, and then the test result of the structured query language migration can be determined according to the first execution result and the second execution result. After the test result is obtained, the configuration information with the best effect can be compared, and the configuration information with a faster execution speed can be selected as the configuration information of the migration requirement for data migration, thereby improving the efficiency of data migration, eliminating the need to repeat a large amount of migration development work each time there is a SQL migration requirement, and saving development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0044] Figure 1 A system architecture diagram of a structured query language migration testing method provided for one embodiment of the present application;

[0045] Figure 2 A flow chart of a structured query language migration testing method provided by one embodiment of the present application;

[0046] Figure 3 A system block diagram of a structured query language migration testing method provided by another embodiment of the present application;

[0047] Figure 4 A schematic diagram of configuring a configuration module provided in one embodiment of the present application;

[0048] Figure 5 A schematic diagram of the structure of a structured query language migration test device provided by one embodiment of the present application;

[0049] Figure 6 A schematic diagram of the structure of an electronic device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0052] With the development of Internet big data, the technology of online analytical processing (OLAP) engines and computing engines is changing very fast. The introduction of new technologies, such as engine replacement, engine version upgrade, and optimization parameter introduction, requires the migration of existing big data tasks. Due to the different characteristics of each engine, different usage scenarios and even different Structured Query Language (SQL) dialects, for example, migration from Hive to Spark, Adhoc query Impala to Trino, and Spark version upgrade scenarios (Hive, Spark, Impala, Trino are different SQL engines), all require customized development. In fact, the general process framework of each development is similar, but due to the differences in SQL dialects between engines and the different SQL types of the previous dual-run (ETL tasks are INSERT statements, Adhoc is SELECT statements), a more general development method was not chosen to complete the migration requirements, and a lot of development work was repeated.

[0053] In order to solve the technical problem in the prior art that each SQL migration requirement requires customized development, the present application provides a structured query language migration testing method, device and storage medium, which eliminates the need to repeat a large amount of migration development work each time there is a SQL migration requirement, thereby improving the efficiency of SQL migration.

[0054] The first embodiment of the present application provides a structured query language migration test method, which can be applied to Figure 1 The system architecture shown includes at least a terminal 101 and a server 102, and the terminal 101 and the server 102 establish a communication connection.

[0055] The method can be applied to the terminal 101 or the server 102 in the system architecture, wherein the terminal 101 can be a desktop computer, a tablet computer, a notebook, a supercomputer or other devices, and the server 102 can be a local server, a cloud server, or a server cluster.

[0056] Next, based on the system architecture, the structured query language migration test method is described in detail, such as Figure 2 , the structured query language migration test method includes:

[0057] Step 201 , extracting configuration information based on migration scenario information; wherein the migration scenario information is used to characterize the structured query language application scenario before migration and the structured query language application scenario after migration; the configuration information includes a control group configuration and at least one experimental group configuration.

[0058] The migration scenario information may be, for example, migration from Hive to Spark, or an upgrade of the Spark version. Configuration information may be extracted based on the migration scenario information. The configuration information includes a control group configuration and at least one experimental group configuration. The control group configuration uses the SQL application scenario before migration, and the experimental group configuration uses the SQL application scenario after migration.

[0059] In one embodiment, configuration information is extracted based on migration scenario information, including: obtaining migration scenario information; determining an original configuration before migration and a scenario configuration of a target migration scenario based on the migration scenario information; wherein the original configuration before migration includes a structured query language application scenario before migration; the scenario configuration of the target migration scenario includes a structured query language application scenario after migration; determining a control group configuration based on the original configuration; and determining an experimental group configuration based on the scenario configuration of the target migration scenario.

[0060] In this embodiment, the Hive migration Spark scenario is used as an example. The SQL gateway can use Pilot. Pilot is a unified SQL access point and the entrance of all big data engines. The original configuration before migration and the scenario configuration of the target migration scenario can be determined according to the migration scenario information. For example, the original configuration includes the relevant SQL application scenario of Hive, and the scenario configuration of the target migration scenario includes the relevant SQL application scenario of Spark. The control group configuration can be determined according to the SQL application scenario before migration, and the experimental group configuration can be determined according to the SQL application scenario after migration. The migration scenario information can be engine migration, cluster migration, engine version switching, different table structure switching (Hive table upgraded to other types of tables, such as upgraded to Iceberg table), etc., without limitation. It should be noted that the experimental group configuration can include multiple to achieve the function of three runs or more runs. This general configuration scheme can decouple the configuration, design a general simulation experiment scheme for automatic execution, improve the efficiency of migration testing, and determine the migration scheme from the results of multiple runs.

[0061] In one embodiment, the original configuration before migration and the scene configuration of the target migration scene are determined based on the migration scenario information, including: extracting an experimental table and multiple configuration tables corresponding to the experimental table based on the migration scenario information; wherein the experimental table includes an experimental table identifier, and the experimental identifiers of multiple configuration tables are the same as the experimental table identifier; the configuration table includes an original configuration before migration and at least one scene configuration of the target migration scene.

[0062] In this embodiment, the experiment table and multiple configuration tables corresponding to the experiment table can be extracted according to the migration scenario information. The experiment table may include information such as the experiment table identifier (experiment table ID), the experiment name (name), the experiment description (description) and the owner. The configuration table may include the original configuration and other multiple scenario configurations. Specifically, each configuration table may include a configuration table ID, an experiment identifier (experiment ID), engine information, a group identifier (group), mapping rules, etc., wherein the configuration table can be associated with the experiment ID, that is, the experiment IDs of multiple configuration tables corresponding to the experiment table are the same, and the configuration tables can be distinguished by group. For example, the group identifier group of the control group configuration can be set to 0, the group of the experimental group configuration 1 can be set to 1, the group of the experimental group configuration 2 can be set to 2, and so on, without restriction.

[0063] Step 202 , executing the target structured query language to be migrated according to the configuration information, obtaining a first execution result corresponding to the control group configuration, and obtaining a second execution result corresponding to the experimental group configuration.

[0064] In one embodiment, executing the target structured query language to be migrated according to the configuration information, obtaining a first execution result corresponding to the control group configuration, and obtaining a second execution result corresponding to the experimental group configuration include:

[0065] Parse the target structured query language into an abstract syntax tree;

[0066] Rewrite the abstract syntax tree to obtain the target abstract syntax tree;

[0067] Converting the target abstract syntax tree into a first structured query language text based on a first structured query language engine configured for the control group, and converting the target abstract syntax tree into a second structured query language text based on a second structured query language engine configured for the experimental group;

[0068] Execute the first structured query language text according to the control group configuration to obtain a first temporary result table;

[0069] Execute the second structured query language text according to the experimental group configuration to obtain a second temporary result table;

[0070] The first temporary result table and the second temporary result table are verified. If the verification passes, the first temporary result table is determined to be the first execution result, and the second temporary result table is determined to be the second execution result.

[0071] In this embodiment, each target SQL to be migrated can be executed according to its corresponding configuration information. First, the target SQL is rewritten, and then the rewritten target SQL and its corresponding configuration information are submitted to the Pilot for execution. For the same SQL, the results of running under different configuration information are verified. After the verification passes, the corresponding execution results under different configuration information are obtained. After obtaining the results, the configuration with the best effect can be compared, and the configuration information with faster execution speed can be selected as the configuration information of the migration requirement for data migration, thereby improving the efficiency of data migration.

[0072] In one embodiment, an abstract syntax tree is rewritten to obtain a target abstract syntax tree, including: obtaining the node type of the parent node of the abstract syntax tree; if the node type of the parent node of the abstract syntax tree is a target selection node, adding a creation table as a selection CTAS node to the parent node of the target selection node; if the node type of the parent node of the abstract syntax tree is a target insertion node, replacing the target insertion node with a creation table as a selection CTAS node; and using the rewritten abstract syntax tree as the target abstract syntax tree.

[0073] In this embodiment, a general SQL parser can be used to parse SQL into an abstract syntax tree AST, and rewrite is performed on the abstract syntax tree. For example, if the node type of the parent node of the abstract syntax tree is a SELECT node, a create table as select CTAS (create table as select) node is added to the parent node of the SELECT node. If the node type of the parent node of the abstract syntax tree is an INSERT node, the INSERT node is replaced with a CTAS node, so that both the INSERT type and the SELECT type are uniformly rewritten into CTAS node types, and a temporary table is used to store the SELECT results. The rewritten abstract syntax tree is then converted into SQL text according to the SQL dialect of the engine in different configurations.

[0074] In one embodiment, the first temporary result table and the second temporary result table are verified, including: obtaining the first row number and the first cyclic redundancy check value of the first temporary result table; obtaining the second row number and the second cyclic redundancy check value of the second temporary result table; if the first row number is the same as the second row number, and the first cyclic redundancy check value is the same as the second cyclic redundancy check value, the verification passes, otherwise the verification fails.

[0075] In this embodiment, for the same SQL, data verification is performed on temporary result tables running under different configuration information. During verification, the number of rows and cyclic redundancy check (CRC) value of the temporary result table can be calculated. If the first number of rows and the second number of rows are the same, and the first CRC value and the second CRC value are the same, it means that the first temporary result table and the second temporary result table are consistent. If any one is different, the verification fails. After all SQL tasks are executed, the execution results under different configuration information can be counted, and the performance difference between the control group configuration and the experimental group configuration can be analyzed and compared.

[0076] In one embodiment, the method further includes: after the first temporary result table and / or the second temporary result table are saved for more than a preset time period, deleting the first temporary result table and / or the second temporary result table.

[0077] In this embodiment, the temporary result table can be saved for a preset period of time, such as 7 days, and the scheduled task is inspected daily. When the temporary result table expires, the temporary result table can be cleared and resources can be recycled.

[0078] Step 203: Determine a test result of the structured query language migration according to the first execution result and the second execution result.

[0079] The test result of the structured query language migration is determined according to the first execution result and the second execution result, so as to provide data support for the configuration information when the SQL migration requirement is finally selected for migration.

[0080] The method can extract configuration information based on the structured query language application scenario before migration and the structured query language application scenario after migration, wherein the configuration information includes a control group configuration and at least one experimental group configuration, so that the target structured query language to be migrated can be executed according to the configuration information, and a first execution result corresponding to the control group configuration and a second execution result corresponding to the experimental group configuration can be obtained, and then the test result of the structured query language migration can be determined according to the first execution result and the second execution result. After the test result is obtained, the configuration information with the best effect can be compared, and the configuration information with a faster execution speed can be selected as the configuration information of the migration requirement for data migration, thereby improving the efficiency of data migration, eliminating the need to repeat a large amount of migration development work each time there is a SQL migration requirement, and saving development costs.

[0081] In a specific embodiment, a system block diagram of a structured query language migration test method can be as follows: Figure 3It includes platform interface, background service, Pilot and underlying engine. The platform interface can be used for experiment import, experiment configuration and report generation. The background service can manage the experiment life cycle and clean up temporary tables. The Pilot can perform SQL auditing and routing forwarding. The underlying engine supports various SQL engines, such as Spark, ClickHouse, Trino, StarRocks, etc.

[0082] In this embodiment, the scenario of migrating Hive to Spark is used as an example. The SQL gateway can use Pilot, which is a unified SQL access point and the entrance to all big data engines. Submit SQL to Pilot and specify the cluster and engine type, and Pilot will forward the SQL to the corresponding engine for execution. In addition, Pilot has the audit information of all SQLs, and can directly filter out the SQLs that you want to simulate and run multiple times from the SQLs that have been historically executed by Pilot.

[0083] In this embodiment, the limitation of the dual-run model is broken away, the configuration is decoupled, and three runs or even more runs can be realized. Based on the existing Pilot functions, a general simulation experiment solution that can be automatically executed is designed. For example, it can be divided into four modules: experiment configuration module, task import module, task management module, and report generation module.

[0084] Previously, general configurations were recorded in a large table, with Hive configuration and Spark configuration in one row. This was highly coupled and could not be effectively expanded later. Currently, in the experiment configuration module, the experiment information and configuration information can be split into two tables. The configurations are associated with the experiment ID and distinguished by group. The schematic diagram of the configuration module can be shown as follows: Figure 4 The concepts of control group and experimental group correspond to the original configuration and new configuration in the migration scenario. There can be multiple new configurations. A SQL statement is simulated and run under multiple configurations. After the results are obtained, the configuration with the best effect can be compared.

[0085] The experiment table may include information such as the experiment table identifier (experiment table ID), experiment name (name), experiment description (description) and owner. The configuration table may include the original configuration and other multiple scenario configurations. Specifically, each configuration table may include the configuration table ID, experiment identifier (experiment ID), engine information (engine), group identifier (group), mapping rules, cluster (cluster), distributed computing user (hadoopUser), pilot property information (pilot properties), etc. Among them, the configuration table can be associated with the experiment ID, that is, the experiment IDs of multiple configuration tables corresponding to the experiment table are the same, and the configuration tables can be distinguished by group. For example, the group identifier group of the control group configuration can be set to 0, the group of the experimental group configuration 1 can be set to 1, and the group of the experimental group configuration 2 can be set to 2, etc., without restriction.

[0086] In the task import module, Pilot has the audit information of all SQL statements, so you only need to filter out qualified tasks by engine, cluster, user, time, etc. in Pilot's historical execution records and import them into the experiment you just configured.

[0087] In the task management module, after you have the SQL to be migrated and the experimental configuration, you can perform the following steps for each SQL according to different configurations, and the entire process will be automated:

[0088] 1. SQL rewriting. Use a general SQL parser to parse SQL into an abstract syntax tree AST, rewrite on the abstract syntax tree, extract the SELECT node, add a CTAS node to its parent node position, rewrite both the INSERT type and the SELECT type into CTAS node types, use a temporary table to store the SELECT results, and then convert the rewritten abstract syntax tree into SQL text according to the SQL dialect of the engine in different configurations.

[0089] 2. Submit the rewritten SQL and corresponding configuration to Pilot for execution.

[0090] 3. For the same SQL statement, perform data verification on the temporary result table running under different configurations, calculate the number of rows in the result table and the total value of the CRC of each row. If both values ​​are the same as the control group results, the results are consistent.

[0091] 4. After all SQL tasks are executed, the results are counted, such as the time consumption under different configuration information, and the differences between each experimental group and the control group are analyzed. For example, the time consumption statistics are as follows:

[0092] Control group: The average time was 419.49 seconds, including 173.06 seconds for P50, 893.58 seconds for P90, and 3116.63 seconds for P99.

[0093] Experimental group 1: The average time consumed was 25.34 seconds, including 9.02 seconds for P50, 65.04 seconds for P90, and 11.65 seconds for P99.

[0094] It can be seen that under the configuration of experimental group 1, the efficiency of SQL migration is significantly higher.

[0095] 5. The temporary result table is retained for 7 days, and the scheduled task is inspected daily to clean up the expired temporary table and recycle resources.

[0096] In this embodiment, in production practice, the development processes in historical migration scenarios are sampled and extracted, and a universal solution is designed (universal task configuration, universal task import, universal SQL rewriting, universal data verification, and universal report output). In future migration scenarios, only simple configuration and batch import of SQL are required to perform migration testing with one click, and the solution with better migration effect is selected for data migration, which eliminates repeated development work and saves manpower.

[0097] Based on the same technical concept, the second embodiment of the present application provides a structured query language migration test device, such as Figure 5 , the device comprises:

[0098] An extraction module 501 is used to extract configuration information based on migration scenario information; wherein the migration scenario information is used to characterize the structured query language application scenario before migration and the structured query language application scenario after migration; the configuration information includes a control group configuration and at least one experimental group configuration;

[0099] An execution module 502 is used to execute the target structured query language to be migrated according to the configuration information, obtain a first execution result corresponding to the control group configuration, and obtain a second execution result corresponding to the experimental group configuration;

[0100] The determination module 503 is configured to determine a test result of the structured query language migration according to the first execution result and the second execution result.

[0101] The device can extract configuration information based on the structured query language application scenario before migration and the structured query language application scenario after migration, wherein the configuration information includes a control group configuration and at least one experimental group configuration, so that the target structured query language to be migrated can be executed according to the configuration information, and a first execution result corresponding to the control group configuration and a second execution result corresponding to the experimental group configuration can be obtained, and then the test result of the structured query language migration can be determined according to the first execution result and the second execution result. After obtaining the test result, the configuration information with the best effect can be compared, and the configuration information with a faster execution speed can be selected as the configuration information of the migration requirement for data migration, thereby improving the efficiency of data migration, eliminating the need to repeat a large amount of migration development work each time there is a SQL migration requirement, and saving development costs.

[0102] Optionally, the extraction module specifically includes:

[0103] A first acquisition unit, used to acquire migration scenario information;

[0104] A first determination unit is used to determine an original configuration before migration and a scenario configuration of a target migration scenario according to the migration scenario information; wherein the original configuration before migration includes a structured query language application scenario before migration; and the scenario configuration of the target migration scenario includes a structured query language application scenario after migration;

[0105] A second determining unit, configured to determine the control group configuration according to the original configuration;

[0106] The third determining unit is used to determine the experimental group configuration according to the scenario configuration of the target migration scenario.

[0107] Optionally, the first determination unit is specifically used to extract an experimental table and multiple configuration tables corresponding to the experimental table based on the migration scenario information; wherein, the experimental table includes an experimental table identifier, and the experimental identifiers of the multiple configuration tables are the same as the experimental table identifier; the configuration table includes an original configuration before migration and a scenario configuration of at least one target migration scenario.

[0108] Optionally, the execution module specifically includes:

[0109] A parsing unit, used for parsing the target structured query language into an abstract syntax tree;

[0110] A rewriting unit, used for rewriting the abstract syntax tree to obtain a target abstract syntax tree;

[0111] a conversion unit, configured to convert the target abstract syntax tree into a first structured query language text based on a first structured query language engine configured for the control group, and to convert the target abstract syntax tree into a second structured query language text based on a second structured query language engine configured for the experimental group;

[0112] A first execution unit, configured to execute the first structured query language text according to the control group configuration to obtain a first temporary result table;

[0113] A second execution unit, configured to execute the second structured query language text according to the experimental group configuration to obtain a second temporary result table;

[0114] A verification unit is used to verify the first temporary result table and the second temporary result table. If the verification passes, the first temporary result table is determined to be the first execution result, and the second temporary result table is determined to be the second execution result.

[0115] Optionally, the rewriting unit is specifically used to obtain the node type of the parent node of the abstract syntax tree; if the node type of the parent node of the abstract syntax tree is a target selection node, then add a creation table to the parent node of the target selection node as a selection CTAS node; if the node type of the parent node of the abstract syntax tree is a target insertion node, then replace the target insertion node with a creation table as a selection CTAS node; and use the rewritten abstract syntax tree as the target abstract syntax tree.

[0116] Optionally, the verification unit is specifically used to obtain a first row number and a first cyclic redundancy check value of the first temporary result table; obtain a second row number and a second cyclic redundancy check value of the second temporary result table; if the first row number is the same as the second row number, and the first cyclic redundancy check value is the same as the second cyclic redundancy check value, the verification passes, otherwise the verification fails.

[0117] Optionally, the execution module further includes a deleting unit, configured to delete the first temporary result table and / or the second temporary result table after the first temporary result table and / or the second temporary result table are saved for more than a preset time period.

[0118] like Figure 6 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0119] Memory 113, used for storing computer programs;

[0120] In one embodiment of the present application, the processor 111 is used to implement the structured query language migration test method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113, including:

[0121] Extracting configuration information based on migration scenario information; wherein the migration scenario information is used to characterize the structured query language application scenario before migration and the structured query language application scenario after migration; the configuration information includes a control group configuration and at least one experimental group configuration;

[0122] Executing the target structured query language to be migrated according to the configuration information, obtaining a first execution result corresponding to the control group configuration, and obtaining a second execution result corresponding to the experimental group configuration;

[0123] A test result of the structured query language migration is determined according to the first execution result and the second execution result.

[0124] Optionally, extracting configuration information based on the migration scenario information includes:

[0125] Obtain migration scenario information;

[0126] Determine the original configuration before the migration and the scenario configuration of the target migration scenario according to the migration scenario information; wherein the original configuration before the migration includes the structured query language application scenario before the migration; and the scenario configuration of the target migration scenario includes the structured query language application scenario after the migration;

[0127] Determining the control group configuration according to the original configuration;

[0128] The experimental group configuration is determined according to the scenario configuration of the target migration scenario.

[0129] Optionally, determining the original configuration before migration and the scenario configuration of the target migration scenario according to the migration scenario information includes:

[0130] An experiment table and multiple configuration tables corresponding to the experiment table are extracted according to the migration scenario information; wherein the experiment table includes an experiment table identifier, and the experiment identifiers of the multiple configuration tables are the same as the experiment table identifier; the configuration table includes an original configuration before migration and a scenario configuration of at least one target migration scenario.

[0131] Optionally, executing the target structured query language to be migrated according to the configuration information to obtain a first execution result corresponding to the control group configuration and obtaining a second execution result corresponding to the experimental group configuration includes:

[0132] Parsing the target structured query language into an abstract syntax tree;

[0133] Rewriting the abstract syntax tree to obtain a target abstract syntax tree;

[0134] Converting the target abstract syntax tree into a first structured query language text based on a first structured query language engine configured for the control group, and converting the target abstract syntax tree into a second structured query language text based on a second structured query language engine configured for the experimental group;

[0135] Executing the first structured query language text according to the control group configuration to obtain a first temporary result table;

[0136] Execute the second structured query language text according to the experimental group configuration to obtain a second temporary result table;

[0137] The first temporary result table and the second temporary result table are verified. If the verification passes, the first temporary result table is determined to be the first execution result, and the second temporary result table is determined to be the second execution result.

[0138] Optionally, rewriting the abstract syntax tree to obtain a target abstract syntax tree includes:

[0139] Obtaining the node type of the parent node of the abstract syntax tree;

[0140] If the node type of the parent node of the abstract syntax tree is a target selection node, then adding a creation table to the parent node of the target selection node as a selection CTAS node;

[0141] If the node type of the parent node of the abstract syntax tree is a target insertion node, the target insertion node is replaced with a creation table as a selection CTAS node;

[0142] The rewritten abstract syntax tree is used as the target abstract syntax tree.

[0143] Optionally, verifying the first temporary result table and the second temporary result table includes:

[0144] Obtaining a first row number and a first cyclic redundancy check value of the first temporary result table;

[0145] Obtain a second row number and a second cyclic redundancy check value of the second temporary result table;

[0146] If the first row number is the same as the second row number, and the first cyclic redundancy check value is the same as the second cyclic redundancy check value, the check passes; otherwise, the check fails.

[0147] Optionally, the method further includes: after the first temporary result table and / or the second temporary result table is saved for more than a preset time period, deleting the first temporary result table and / or the second temporary result table.

[0148] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0149] The communication interface is used for communication between the above terminal and other devices.

[0150] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0151] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0152] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the structured query language migration testing method provided by any of the aforementioned method embodiments is implemented.

[0153] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0155] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0156] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. In the description, the suffixes such as "module", "component" or "unit" used to represent the elements are only used to facilitate the description of the present application and have no specific meaning in themselves. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0157] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A structured query language migration test method, characterized in that: The method comprises: Extracting configuration information based on migration scenario information; wherein the migration scenario information is used to characterize the structured query language application scenario before migration and the structured query language application scenario after migration; the configuration information includes a control group configuration and at least one experimental group configuration; Executing the target structured query language to be migrated according to the configuration information, obtaining a first execution result corresponding to the control group configuration, and obtaining a second execution result corresponding to the experimental group configuration; A test result of the structured query language migration is determined according to the first execution result and the second execution result.

2. The method according to claim 1, characterized in that Extract configuration information based on migration scenario information, including: Obtain migration scenario information; Determine the original configuration before the migration and the scenario configuration of the target migration scenario according to the migration scenario information; wherein the original configuration before the migration includes the structured query language application scenario before the migration; and the scenario configuration of the target migration scenario includes the structured query language application scenario after the migration; Determining the control group configuration according to the original configuration; The experimental group configuration is determined according to the scenario configuration of the target migration scenario.

3. The method according to claim 2, characterized in that Determining the original configuration before the migration and the scenario configuration of the target migration scenario according to the migration scenario information includes: An experiment table and multiple configuration tables corresponding to the experiment table are extracted according to the migration scenario information; wherein the experiment table includes an experiment table identifier, and the experiment identifiers of the multiple configuration tables are the same as the experiment table identifier; the configuration table includes an original configuration before migration and a scenario configuration of at least one target migration scenario.

4. The method according to claim 1, characterized in that: Executing the target structured query language to be migrated according to the configuration information to obtain a first execution result corresponding to the control group configuration and a second execution result corresponding to the experimental group configuration, including: Parsing the target structured query language into an abstract syntax tree; Rewriting the abstract syntax tree to obtain a target abstract syntax tree; Converting the target abstract syntax tree into a first structured query language text based on a first structured query language engine configured for the control group, and converting the target abstract syntax tree into a second structured query language text based on a second structured query language engine configured for the experimental group; Executing the first structured query language text according to the control group configuration to obtain a first temporary result table; Execute the second structured query language text according to the experimental group configuration to obtain a second temporary result table; The first temporary result table and the second temporary result table are verified. If the verification passes, the first temporary result table is determined to be the first execution result, and the second temporary result table is determined to be the second execution result.

5. The method according to claim 4, characterized in that Rewriting the abstract syntax tree to obtain a target abstract syntax tree includes: Obtaining the node type of the parent node of the abstract syntax tree; If the node type of the parent node of the abstract syntax tree is a target selection node, then adding a creation table to the parent node of the target selection node as a selection CTAS node; If the node type of the parent node of the abstract syntax tree is a target insertion node, the target insertion node is replaced with a creation table as a selection CTAS node; The rewritten abstract syntax tree is used as the target abstract syntax tree.

6. The method according to claim 4, characterized in that Verifying the first temporary result table and the second temporary result table includes: Obtaining a first row number and a first cyclic redundancy check value of the first temporary result table; Obtain a second row number and a second cyclic redundancy check value of the second temporary result table; If the first row number is the same as the second row number, and the first cyclic redundancy check value is the same as the second cyclic redundancy check value, the check passes; otherwise, the check fails.

7. The method according to claim 4, characterized in that The method further includes: after the first temporary result table and / or the second temporary result table are saved for more than a preset time period, deleting the first temporary result table and / or the second temporary result table.

8. A structured query language migration test device, characterized in that: The device comprises: An extraction module, configured to extract configuration information based on migration scenario information; wherein the migration scenario information is used to characterize a structured query language application scenario before migration and a structured query language application scenario after migration; and the configuration information includes a control group configuration and at least one experimental group configuration; An execution module, configured to execute the target structured query language to be migrated according to the configuration information, obtain a first execution result corresponding to the control group configuration, and obtain a second execution result corresponding to the experimental group configuration; A determination module is used to determine a test result of the structured query language migration according to the first execution result and the second execution result.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the structured query language migration test method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the structured query language migration testing method according to any one of claims 1 to 7 is implemented.

Images