SQL (Structured Query Language) statement test method, system, device, equipment, medium and product

Abstract

The invention provides an SQL (Structured Query Language) statement test method, system, device and equipment, a medium and a product, and relates to the technical field of data processing. The method is applied to a drive server in a Spark cluster, the Spark cluster further comprises a plurality of working servers, and the method comprises the steps of obtaining a DDL statement representing a data structure of a to-be-generated data table; analyzing the DDL statement to obtain an analysis result; constructing a task sequence based on an analysis result by utilizing a directed acyclic graph scheduler component; issuing a task sequence to each working server by using a task scheduler component, generating a target data table containing original fields and data of the original fields by each working server according to the data generation task, and writing the target data table into a specified database according to the data writing task; the data of the original fields in any target data table are arranged according to the original arrangement sequence. According to the method, a large amount of data is stored in the local database of the server so as to test SQL statements related to query of a large amount of data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a SQL statement testing method, system, device, equipment, medium and product. Background Art

[0002] The server can manage data in the local database through Structured Query Language (SQL) statements. Developers need to test SQL statements to determine the execution efficiency of SQL statements and then optimize them.

[0003] In the prior art, in the process of testing SQL statements, the execution efficiency of the SQL statements can be determined based on the time it takes for the server to query data when executing the SQL statements. However, some SQL statements involve a large amount of data. For example, some SQL statements may need to query more data tables at the same time. Accordingly, when these SQL statements need to be tested, a large amount of data needs to be stored in the local database of the server. When a large amount of data is not stored in the local database of the server, such SQL statements cannot be tested. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a SQL statement testing method, system, device, equipment, medium and product to store a large amount of data in a local database on the server side to test SQL statements involving large amounts of data queries. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for testing an SQL statement, the method being applied to a driver server in a Spark cluster, the Spark cluster further comprising a plurality of working servers, the method comprising:

[0006] Obtaining a data definition language DDL statement representing a data structure of a data table to be generated; wherein the data structure of the data table to be generated is the same as the data structure of a data table in a specified database;

[0007] Parsing the DDL statement to obtain a parsing result; wherein the parsing result includes: each original field contained in the data table to be generated, and the arrangement order of each original field in the data table to be generated;

[0008] Using the directed acyclic graph scheduler component in the driving server, a task sequence is constructed based on the parsing result; wherein the task sequence includes a data generation task and a data writing task; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field;

[0009] Utilize the task scheduler component in the driving server to send the task sequence to each working server respectively, so that each working server generates a target data table containing each original field and the data belonging to each original field according to the data generation task, and writes the generated target data table into the designated database according to the data writing task; wherein, the data belonging to each original field in any target data table is arranged according to the original arrangement order.

[0010] In a second aspect, an embodiment of the present invention provides a SQL statement testing method, the method being applied to each of a plurality of working servers included in a testing system, the testing system further comprising a driving server, the method comprising:

[0011] The task sequence sent by the driving server is received through the execution process in each working server; wherein the task sequence is: the driving server parses the acquired data definition language DDL statement representing the data structure of the data table to be generated, constructs it according to the parsing result using the directed acyclic graph scheduler component in the driving server, and issues it to the multiple working servers respectively using the task scheduler component in the driving server; the data structure of the data table to be generated is the same as the data structure of the data table in the specified database; the task sequence includes data generation tasks and data writing tasks; the parsing result includes: the original fields contained in the data table to be generated, and the arrangement order of the original fields in the data table to be generated; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the original field;

[0012] According to the data generation task, a target data table is generated, which includes each original field and the data belonging to each original field; wherein the data belonging to each original field in any target data table is arranged according to the original arrangement order;

[0013] According to the data writing task, the generated target data table is written into the designated database.

[0014] In a third aspect, an embodiment of the present invention provides a SQL statement testing system, comprising a plurality of working servers and a driver server in a Spark cluster;

[0015] The driver server is used to execute the aforementioned SQL statement testing method;

[0016] Any working server among the multiple working servers is used to execute the aforementioned SQL statement testing method.

[0017] In a fourth aspect, an embodiment of the present invention provides a SQL statement testing device, the device is applied to a driver server in a Spark cluster, the Spark cluster also includes multiple working servers, and the device includes:

[0018] A first acquisition module is used to acquire a data definition language DDL statement representing a data structure of a data table to be generated; wherein the data structure of the data table to be generated is the same as the data structure of a data table in a specified database;

[0019] A first parsing module is used to parse the DDL statement to obtain a parsing result; wherein the parsing result includes: each original field contained in the data table to be generated, and the arrangement order of each original field in the data table to be generated;

[0020] A construction module is used to use the directed acyclic graph scheduler component in the driving server to construct a task sequence based on the parsing result; wherein the task sequence includes a data generation task and a data writing task; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field;

[0021] A sending module is used to use the task scheduler component in the driving server to send the task sequence to each working server respectively, so that each working server generates a target data table containing each original field and the data belonging to each original field according to the data generation task, and writes the generated target data table into the designated database according to the data writing task; wherein the data belonging to each original field in any target data table is arranged according to the original arrangement order.

[0022] In a fifth aspect, an embodiment of the present invention provides a SQL statement testing device, the device is applied to each working server of a plurality of working servers included in a testing system, the testing system further includes a driving server, and the device includes:

[0023] A receiving module, used for receiving a task sequence sent by the driving server through an execution process in each working server; wherein the task sequence is: the driving server parses the acquired data definition language DDL statement representing the data structure of the data table to be generated, constructs it according to the parsing result using the directed acyclic graph scheduler component in the driving server, and issues it to the multiple working servers respectively using the task scheduler component in the driving server; the data structure of the data table to be generated is the same as the data structure of the data table in the specified database; the task sequence includes a data generation task and a data writing task; the parsing result includes: each original field contained in the data table to be generated, and the arrangement order of each original field in the data table to be generated; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the original field;

[0024] A generating module, used for generating a target data table including each original field and data belonging to each original field according to the data generating task; wherein the data belonging to each original field in any target data table is arranged according to the original arrangement order;

[0025] A writing module is used to write the generated target data table into the specified database according to the data writing task.

[0026] An embodiment of the present invention further provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0027] Memory, used to store computer programs;

[0028] The processor is used to implement the aforementioned SQL statement testing method when executing the program stored in the memory.

[0029] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned SQL statement testing method is implemented.

[0030] An embodiment of the present invention further provides a computer program product, including a computer program, wherein the computer program implements the aforementioned SQL statement testing method when executed by a processor.

[0031] Beneficial effects of the embodiments of the present invention:

[0032] An SQL statement testing method provided by an embodiment of the present invention is applied to a driver server in a Spark cluster, wherein the Spark cluster also includes multiple working servers, and can obtain a data definition language DDL statement representing a data structure of a data table to be generated; parse the DDL statement to obtain a parsing result; use a directed acyclic graph scheduler component in the driver server to build a task sequence based on the parsing result; use the task scheduler component in the driver server to send the task sequence to each working server respectively, so that each working server generates a target data table containing each original field and data belonging to each original field according to the data generation task, and writes the generated target data table into a designated database according to the data writing task.

[0033] Based on the above processing, the parsing result includes the original fields contained in the data table to be generated and the arrangement order of the original fields in the data table to be generated. The data generation task in the task sequence constructed by the directed acyclic graph scheduler component based on the parsing result can represent the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field. Therefore, the data generation task is used to instruct each working server to generate the data of the column corresponding to each original field according to the original arrangement order, that is, the data generation task can be used to generate a data table with the same data structure as the data table to be generated. In the embodiment of the present invention, a task scheduler component is used to send a task sequence to each working server of the Spark cluster, and multiple working servers generate a target data table with the same data structure as the data table to be generated in parallel. The data structure of the data table to be generated is the same as the data structure of the data table in the specified database. The target data table is the same as the data structure of the data table in the specified database. The target data table is written into the specified database, which can ensure that a large number of target data tables written by multiple working servers are stored in the specified database, simulating the situation of storing a large amount of data in the specified database, so as to realize the storage of a large amount of data in the local database of the server, and thus can test SQL statements involving queries of large amounts of data.

[0034] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0036] Figure 1 A flowchart of a first SQL statement testing method provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a flow chart of a second SQL statement testing method provided in an embodiment of the present invention;

[0038] Figure 3 A flowchart of a third SQL statement testing method provided by an embodiment of the present invention;

[0039] Figure 4 A flowchart of a fourth SQL statement testing method provided by an embodiment of the present invention;

[0040] Figure 5 A flowchart of a fifth SQL statement testing method provided by an embodiment of the present invention;

[0041] Figure 6 A flowchart of a sixth SQL statement testing method provided by an embodiment of the present invention;

[0042] Figure 7 A flowchart of a seventh SQL statement testing method provided by an embodiment of the present invention;

[0043] Figure 8 A schematic diagram of the principle of the first SQL statement testing method provided by an embodiment of the present invention;

[0044] Fig. 9 A schematic diagram of the principle of a second SQL statement testing method provided by an embodiment of the present invention;

[0045] Fig.10 A schematic diagram of the structure of a DataFrame provided in an embodiment of the present invention;

[0046] Fig.11 A schematic diagram of the principle of a third SQL statement testing method provided by an embodiment of the present invention;

[0047] Fig.12 A schematic diagram of the principle of a fourth SQL statement testing method provided by an embodiment of the present invention;

[0048] Fig.13 A schematic diagram of the structure of a SQL statement testing system provided by an embodiment of the present invention;

[0049] Fig.14 A schematic diagram of the structure of a first SQL statement testing device provided by an embodiment of the present invention;

[0050] Fig.15 A schematic diagram of the structure of a second SQL statement testing device provided by an embodiment of the present invention;

[0051] Fig.16 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.

[0053] When it is necessary to test SQL statements involving large amounts of data, a large amount of data needs to be stored in the local database on the server. Generally speaking, during the SQL statement testing phase, for security reasons, the real data of users in the database will not be directly used as test data for testing. For example, in industries such as banking and securities, the real data in the database is sensitive data of users and cannot be used as test data. In order to test SQL statements involving large amounts of data, a large amount of test data (non-sensitive data of users, i.e., desensitized data) needs to be stored in the local database.

[0054] In order to better understand the embodiments of the present invention, the Spark cluster is introduced below.

[0055] Spark cluster is a distributed computing framework, which consists of multiple nodes, including a master node server (MasterNode) and multiple worker node (Worker Node) servers. The worker node servers are collectively referred to as worker servers in the embodiment of the present invention. The master node server is used to schedule the computing resources of each worker node server. The worker server is responsible for actually executing the computing tasks.

[0056] Resilient Distributed Datasets (RDD) is a data storage format for Spark clusters. RDD can be divided into multiple memory partitions, and each memory partition is distributed on different worker servers for parallel computing. There are multiple executors on each worker server. The worker server uses each of its own executors to perform calculations in the memory partitions first. If the memory space is close to the memory limit, temporary storage locations or disk space may be used for calculations to avoid memory overflow.

[0057] In order to store a large amount of data in a local database on a server side so as to test SQL statements involving large amounts of data queries, the present invention provides a SQL statement testing method, system, device, equipment, medium and product.

[0058] The SQL statement testing method provided in the embodiment of the present invention can be applied to a driver server in a Spark cluster, and the Spark cluster also includes multiple working servers. The driver server can be an electronic device installed with a Spark application, and the driver server can define the data processing process and the division of tasks. Optionally, if the master node server is installed with a Spark application, the master node server can be used as a driver server.

[0059] The SQL statement testing method provided by the embodiment of the present invention may include the following steps:

[0060] Obtaining a data definition language DDL statement representing a data structure of a data table to be generated; wherein the data structure of the data table to be generated is the same as the data structure of a data table in a specified database;

[0061] Parse the DDL statement to obtain a parsing result; wherein the parsing result includes: each original field contained in the data table to be generated, and the arrangement order of each original field in the data table to be generated;

[0062] The directed acyclic graph scheduler component in the driver server is used to construct a task sequence based on the parsing results; wherein the task sequence includes a data generation task and a data writing task; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field;

[0063] Utilize the task scheduler component in the driver server to send task sequences to each working server respectively, so that each working server generates a target data table containing each original field and the data belonging to each original field according to the data generation task, and writes the generated target data table into the designated database according to the data writing task; wherein the data belonging to each original field in any target data table are arranged in the original arrangement order.

[0064] Based on the above processing, the parsing result includes the original fields contained in the data table to be generated and the arrangement order of the original fields in the data table to be generated. The data generation task in the task sequence constructed by the directed acyclic graph scheduler component based on the parsing result can represent the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field. Therefore, the data generation task is used to instruct each working server to generate the data of the column corresponding to each original field according to the original arrangement order, that is, the data generation task can be used to generate a data table with the same data structure as the data table to be generated. In the embodiment of the present invention, a task scheduler component is used to send a task sequence to each working server of the Spark cluster, and multiple working servers generate a target data table with the same data structure as the data table to be generated in parallel. The data structure of the data table to be generated is the same as the data structure of the data table in the specified database. The target data table is the same as the data structure of the data table in the specified database. The target data table is written into the specified database, which can ensure that a large number of target data tables written by multiple working servers are stored in the specified database, simulating the situation of storing a large amount of data in the specified database, so as to realize the storage of a large amount of data in the local database of the server, and thus can test SQL statements involving queries of large amounts of data.

[0065] The following describes a SQL statement testing method provided by an embodiment of the present invention in conjunction with the accompanying drawings. Figure 1 , Figure 1 A flowchart of a SQL statement testing method provided by an embodiment of the present invention, the method may include steps S101-S104.

[0066] S101, obtaining a data definition language DDL statement representing a data structure of a data table to be generated.

[0067] It can be understood that the data table to be generated has the same data structure as that in the specified database, that is, the data structure of the data table to be generated is the same as that of the existing data table stored in the specified database, and the generated data table can be written into the specified database later. The specified database can be a My Structured Query Language Database Management System (MySQL) database or an Oracle database, and the embodiments of the present invention are only used as examples for illustration and are not specifically limited.

[0068] The developer can save the code of the SQL statement test method required to be executed by the driving server and the data definition language (DDL) statement required to execute the SQL statement test method in a configuration file, and then store the configuration file in the driving server. The process of storing the configuration file in the driving server may include: saving the code of the SQL statement test method required to be executed by the driving server and the DDL statement required to execute the SQL statement test method in a Scalable Language (Scala) file, using a Simple Build Tool (SBT) or other building tools to package the Scala file into a Java Archive File (JAR), i.e., a JAR file, and storing the JAR file as the configuration file in the driving server.

[0069] When the driving server receives an instruction to start executing the SQL statement test method, the driving server may obtain the DDL statement from a pre-configured configuration file that records the DDL statement.

[0070] Optionally, the driver server may access a specified database, query the data structure of an existing data table in the specified database, and obtain a DDL statement. In this regard, the embodiment of the present invention is only used as an example for illustration and is not specifically limited.

[0071] Exemplarily, the DDL statement of the data table to be generated defines the table structure of the data table to be generated in the specified database and the data type of each field. For example, the DDL statement can define the first column in a data table as a name field, the second column as an age field, and the third column as a weight field, and the data type of the data under the name field is a string type, and the data type of the data under the age field and the weight field is a numeric type.

[0072] S102, parsing the DDL statement to obtain a parsing result.

[0073] The parsing result includes: each original field included in the data table to be generated, and the arrangement order of each original field in the data table to be generated.

[0074] It is understandable that by parsing the DDL statement representing the data structure of the data table to be generated, the original fields contained in the data table to be generated and the arrangement order of the original fields in the data table to be generated can be obtained.

[0075] Exemplarily, a parsing function for each data type can be defined in the manner of a regular expression parser (RegexParsers) class in a configuration file (for example, a parsing function for an integer type, a parsing function for a string type, and a parsing function for a date type can be defined) to create a DDL statement parser. The DDL statement is parsed based on the DDL syntax parser to obtain the data type of each column of the original field in the data table to be generated and each original field. The parsing result can be a summary (Schema) of the table data structure (DataFrame) of the data table to be generated. In the embodiment of the present invention, the parsing and generation of fields of multiple data types can be supported by the parsing of DDL statements and the distributed computing capabilities of each work server in the subsequent Spark cluster.

[0076] S103, using the directed acyclic graph scheduler component in the driver server to construct a task sequence based on the parsing result.

[0077] Among them, the task sequence includes data generation tasks and data writing tasks; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field.

[0078] It can be understood that by driving the directed acyclic graph scheduler component in the server, a task sequence including data generation tasks and data writing tasks can be constructed, and when each working server executes the task sequence, the data generation task is executed first, and then the data writing task is executed, to ensure that each working server can generate data before writing data.

[0079] The data table to be generated has rows and columns. Each column contains all the data of an original field, and each row contains a data belonging to each original field. The original arrangement order of each original field in the data table to be generated refers to the arrangement order of the original fields to which the data in each column of the data table to be generated belongs. For example, the first column of data in the data table to be generated belongs to the name field, the second column of data belongs to the gender field, and the third column of data belongs to the height field. In the original arrangement order, the first column to the third column are the name field, the gender field, and the height field.

[0080] In the process of constructing the data generation task, the driver server can define the data generation subtask corresponding to each original field in turn according to the original arrangement order. The data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field.

[0081] Specifically, the driver server can determine the conversion operation that the subsequent work server needs to perform for each original field in the data table to be generated. The conversion operation may include: calling a function for generating a new column table (for example, withcolumn function), and calling a function for generating data belonging to each original field; wherein, the data generation subtask corresponding to each original field can also be called the conversion operation corresponding to the original field. Each conversion operation is arranged in the original arrangement order. For example, the first column in the data table to be generated is the name field, and the second column is the age field. The function for generating the data of the name field is function A, the function for generating the data of the age field is function B, and the function for generating a new column table is function C; then in the plan of the data generation task, each work server first calls function C to generate the first column table, and then calls function A to generate the data belonging to the name field in the first column table; then calls function C to generate the second column table, and then calls function B to generate the data belonging to the age field in the second column table.

[0082] In the process of constructing the data writing task, the driver server can determine the calculation operation to be performed after the conversion operation. Specifically, the calculation operation can be to call an interface that can communicate with the database, such as a JAVA database connector (Java Database Connectivity, JDBC); when a working server calls the JDBC connector, it can connect to the database and use the write (write.jdbc) function defined in the JDBC connector to write the target data table to the specified database. Optionally, in one implementation, the uniform resource locator (Uniform Resource Location, URL) of the specified database to be written, the user name and password for logging in to the specified database can be pre-configured in the configuration file of the JDBC connector. When the JDBC connector is called, a connection can be established with the specified database through the URL, and the target data table can be safely and accurately written to the specified database through the verification of the specified database according to the user name and password.

[0083] Exemplarily, the driver server can construct a directed acyclic graph (DAG) containing the conversion operations of the data table to be generated using a directed acyclic graph scheduler component, and split the DAG graph into different execution stages (Stages), that is, each conversion operation is divided into multiple execution stages. Among them, if each work server does not need to redistribute data, each conversion operation will be divided into the same execution stage. In an embodiment of the present invention, each work server does not need to move data when generating data, so that each conversion operation in this hair style embodiment belongs to the same execution stage. A task sequence is created based on the execution stage using a directed acyclic graph scheduler component, and the created task sequence includes a data generation task and a data writing task for the conversion operation. Among them, the data generation task and the data writing task are physical execution tasks.

[0084] S104, using the task scheduler component in the driving server, sends a task sequence to each working server respectively, so that each working server generates a target data table containing each original field and the data belonging to each original field according to the data generation task, and writes the generated target data table into the specified database according to the data writing task.

[0085] The data belonging to each original field in any target data table are arranged in the original arrangement order.

[0086] It is understandable that the driver server can receive an instruction to start executing the SQL statement test method, and the instruction can include the amount of data required to be generated for each original field. Among them, the amount of data required to generate for each original field is the number of data required to be generated under the original field. For example, if the amount of data required to generate the age field is 10, then 10 values ​​need to be generated; the amount of data can also be referred to as the number of data entries. The driver server can use the task scheduler component to send a task sequence to each work server. The amount of data required to generate for each work server is determined based on the amount of data required to generate for each original field and the number of memory partitions of the RDD. Among them, the task scheduler component can interact with each work server, for example, send a task sequence to each work server, and determine the task execution status of each work server so that the task can be restarted when the task execution fails.

[0087] It is understandable that the data generated by the working server can be stored in its own storage space.

[0088] Optionally, in one implementation, each working server preferentially generates and writes data in its own memory space through each execution process. Based on the issued task sequence, each working server can generate a target data table in its own memory space, and write the generated target data table from its own memory space to a designated database. Data generation and writing operations are performed directly in the memory space, thereby avoiding frequent disk read and write operations, improving response speed, maintaining high-precision data generation and writing operations, and can improve the processing efficiency and accuracy of storing a large number of target data tables in a designated database, and can improve the efficiency of SQL statement testing.

[0089] Optionally, in another implementation, when the remaining storage space in the memory space is less than a preset threshold (i.e., the remaining storage space in the memory space is small and close to the memory limit), data will be generated in a temporary storage area or disk space to avoid memory overflow.

[0090] It is understandable that in the Spark cluster, there are also cluster managers and query servers. For example, the cluster manager can be Yet Another Resource Negotiator ResourceManager (YARN ResourceManager) or Spark Standalone Master. The cluster manager can obtain the number of cores and memory size of each working server by querying the resource manager in the server. Among them, the number of cores and memory size can reflect the performance of each execution process. For example, the number of cores can be the number of cores of the central processing unit (CPU). The cluster manager can arrange the amount of data required to be generated by each working server according to the number of cores and memory size of each working server; after each working server receives the task sequence issued by the driver server through the task scheduler component, it can generate the target data table according to the amount of data arranged by the cluster manager. In other words, the cluster manager can be responsible for resource allocation, and the driver server is responsible for the scheduling of task issuance.

[0091] Optionally, in one implementation, if the available resources of each work server are similar, for example, the number of cores of each work server is the same, the amount of data to be generated can be evenly distributed to each work server for processing.

[0092] Optionally, in another implementation, if a working server has more resources, such as a larger number of cores, then you can try to arrange more data to be generated for the working server. For example, if working server a has 8 cores and working server b has 4 cores, then you can try to assign more tasks to working server a with more cores. In the embodiment of the present invention, by allocating tasks considering the number of cores of the execution process, the flexibility of each working server in the Spark cluster in generating the target data table and the efficiency of resource management can be improved. In this regard, the embodiment of the present invention is only used as an example for illustration and is not specifically limited.

[0093] It is understandable that the driver server can call a function to create an RDD with the same amount of data as the required amount of data to be generated, for example, calling the function spark.range(1,10000001) to create an RDD that can contain 10 million integers. The number of memory partitions of the RDD can be a default value or determined according to the number of working servers.

[0094] Exemplarily, when submitting a job through the command line tool spark-submit for submitting a Spark application, the parameter in the function .repartition() can be set to the number of each worker server to set the number of memory partitions of the RDD to the number of each worker server. For example, if the number of each worker server is 100, the number of memory partitions of the RDD can be set to 100 by using .repartition(100). When the number of each worker server is consistent with the number of memory partitions of the RDD, if the available resources of each worker server are the same, such as the number of cores, the amount of data required to be generated in each worker server can be determined based on the ratio of the amount of data required to be generated for each original field to the number of memory partitions, so as to evenly distribute the amount of data required to be generated to each worker server. Exemplarily, the amount of data required to be generated is 10 million, the number of each worker server is 100, the number of memory partitions is 100, and each worker server is responsible for generating 10 million / 100=100,000 data. The task scheduler component sends the task sequence to each work server respectively. After each work server receives the task sequence, each execution process in the work server can execute the task sequence and generate the data required by the work server in parallel.

[0095] In an embodiment of the present invention, the number of RDD partitions is set to the number of working servers. Through the memory partitioning of RDD, the required amount of data is evenly divided to each working server for parallel processing, which can improve the efficiency of generating the target data table and the efficiency of writing into the specified database.

[0096] Optionally, under the premise of preliminarily distributing the required amount of data to each work server evenly according to the amount of data to be generated and the number of each work server, the cluster manager can also try to distribute more data to the work server with a larger number of cores capable of operating according to the number of cores capable of operating in each work server. In the embodiment of the present invention, the required amount of data to be generated can be distributed to each work server based on the scheduling mechanism of Spark and the specific conditions of the available resources of each work server, so as to make full use of the computing resources of the Spark cluster and improve the efficiency of generating the target data table and writing into the specified database.

[0097] Based on the above processing, the parsing result includes the original fields contained in the data table to be generated and the arrangement order of the original fields in the data table to be generated. The data generation task in the task sequence constructed by the directed acyclic graph scheduler component based on the parsing result can represent the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field. Therefore, the data generation task is used to instruct each working server to generate the data of the column corresponding to each original field according to the original arrangement order, that is, the data generation task can be used to generate a data table with the same data structure as the data table to be generated. In the embodiment of the present invention, a task scheduler component is used to send a task sequence to each working server of the Spark cluster, and multiple working servers generate a target data table with the same data structure as the data table to be generated in parallel. The data structure of the data table to be generated is the same as the data structure of the data table in the specified database. The target data table is the same as the data structure of the data table in the specified database. The target data table is written into the specified database, which can ensure that a large number of target data tables written by multiple working servers are stored in the specified database, simulating the situation of storing a large amount of data in the specified database, so as to realize the storage of a large amount of data in the local database of the server, and thus can test SQL statements involving queries of large amounts of data.

[0098] In addition, based on the distributed computing capabilities of each working server in the Spark cluster, different amounts of data generation can be supported, and the SQL statement testing method provided in the embodiment of the present invention has high scalability and adaptability.

[0099] Optionally, in another embodiment, Figure 1 The SQL statement test method shown is as follows Figure 2 As shown, the method also includes steps S201-S202.

[0100] S201, obtaining an SQL statement to be tested for processing a data table in a specified database.

[0101] It is understandable that the driver server can obtain the SQL statement to be tested from the pre-configured configuration file, and the SQL statement to be tested can be processed for the data table in the specified database. Among them, the field to be processed is the field to be processed when executing the SQL statement to be tested, and the specific processing method depends on the processing type to which the SQL statement to be tested belongs. For example, if the SQL statement to be tested is a query statement, the field to be processed is the field to be queried; if the SQL statement to be tested is a delete statement, the field to be processed is the field to be deleted.

[0102] S202, parsing the SQL statement to be tested, and determining the fields to be processed in the SQL statement to be tested as the fields to be processed;

[0103] For each original field that does not belong to the determined fields to be processed, the data generation subtask corresponding to the original field is used to instruct each working server to generate a data table in which the data belonging to the original field is empty.

[0104] It is understandable that the SQL statement to be tested may be parsed to determine the expression containing the field to be processed, and the field in the determined expression may be used as the field to be processed.

[0105] Exemplarily, if the SQL statement to be tested is a query statement, the SQL statement to be tested contains a query expression, and the query expression contains fields required to be queried by the SQL statement to be tested. The SQL statement to be tested can be parsed to determine the query expression, and the fields in the query expression are used as fields to be processed. Specifically, an abstract syntax tree (AST) of the SQL statement to be tested can be established, and the query expression can be determined according to the AST, and the fields in the query expression can be used as fields to be processed.

[0106] It should be noted that when constructing the data generation task in the task sequence, if the original field in the data table to be generated does not belong to the field to be processed, the data generation subtask corresponding to the original field is used to instruct each working server to generate a data table with empty data belonging to the original field. In other words, test data is generated for the original fields that need to be processed by the SQL statement to be tested, and null values ​​are generated for the original fields that do not need to be processed by the SQL statement to be tested. For example, the fields to be processed determined by parsing the SQL statement to be tested include the ID number, and the original fields in the data table to be generated include: ID number, name, the name does not belong to the field to be processed, and the data generation subtask corresponding to the name field is used to instruct each working server to generate a data table with empty data belonging to the name field.

[0107] When a data generation subtask corresponding to an original field is used to instruct each working server to generate a data table with empty data belonging to the original field, a function for calling to generate an empty value may be defined in the data generation subtask.

[0108] In the embodiment of the present invention, test data can be generated for the original fields that need to be processed by the SQL statement to be tested, and null values ​​can be generated for the original fields that do not need to be processed by the SQL statement to be tested, thereby reducing the computing resources of the working server and improving the efficiency of the generated target data table.

[0109] Optionally, in one implementation, the parsing result further includes: the original data type of each original field;

[0110] For each original field that belongs to the determined field to be processed, the data generation subtask corresponding to the original field represents: generating data belonging to the original field, and the generated data is the target data type of the original field; the target data type of each original field represents: the data type supported by multiple working servers, corresponding to the original data type of the original field.

[0111] It can be understood that when the original field in the data table to be generated belongs to the field to be processed, the original field is the field that needs to be processed by the SQL statement to be tested, and the data generation subtask corresponding to the original field means: generate data belonging to the original field, and the generated data is the target data type of the original field.

[0112] The parsing result may also include the original data type of each original field. The original field is a field in the data table to be generated. The data table to be generated is in the specified database, and the original data type of the original field is the data type supported by the specified database. For example, the specified database is a MySQL database, and the original data type is the data type supported by the MySQL database. In order to ensure that the constructed task sequence can be executed in the work server in the Spark cluster, it is necessary to determine the target data type of the original field according to the original data type of the original field before constructing the task sequence, and the determined target data type is supported by the work server, for example, the target data type is supported by the Spark.SQL tool in the Spark cluster. Exemplarily, the target data type corresponding to the original data type can be determined by the DataTypes function. For example, the original data type is INTEGER, and the corresponding target data type is IntegerType; the original data type is FLOAT, and the corresponding target data type is FloatType. The type of data represented by the original data type and the target data type is the same, for example, INTEGER and IntegerType both represent the type of data as an integer, the difference is that the original data type is the data type supported by the specified database, and the target data type is the data type supported by the work server in the Spark cluster.

[0113] Optionally, in one implementation, when the original field in the data table to be generated matches the field to be processed, the original field is the field that needs to be processed by the SQL statement to be tested. At this time, the data generation subtasks corresponding to the original fields of different data types are different. For each original field belonging to the determined field to be processed, if the original data type of the original field is a numeric type, a first function is defined in the data generation subtask corresponding to the original field; the first function is used to instruct each working server to generate data belonging to the original field within a preset numerical range. The preset numerical range can be a pre-set reasonable numerical range. For example, if the original field is age, the preset numerical range can be 18-70; if the original field is a 6-digit customer number, the preset numerical range can be 1-999999.

[0114] If the original data type of the original field is a string type, a second function is defined in the data generation subtask corresponding to the original field; the second function is used to instruct each working server to select a string from a preset string set to generate data belonging to the original field. The preset string set may include multiple pre-set strings. For example, if the original field is gender, the preset string set may include the strings "male" and "female".

[0115] If the original data type of the original field is a date type, a third function is defined in the data generation subtask corresponding to the original field; the third function is used to instruct each working server to select a date in the specified date range to generate data belonging to the original field. The original field of date type can be an integer of yyyyMMdd, where yyyy represents year, MM represents month, and dd represents day.

[0116] It is understandable that the first function, the second function, and the third function are all set in the pre-configured configuration file, in which the preset data range, the preset string set, and the specified date range are also set. When constructing the data generation task, the first function is defined in the data generation subtask corresponding to the original field of the numeric type, the second function is defined in the data generation subtask corresponding to the original field of the string type, and the third function is defined in the data generation subtask corresponding to the original field of the date type. For example, the first function can be the rand() function, the second function can be the expr function, and the third function can be the current_date function and the current_timestamp function. Among them, the rand() function is used to generate random numbers within the preset data range, the expr function is used to randomly select a string from the preset string set, the current_date function is used to generate the current date, and the current_timestamp function is used to generate the current time.

[0117] In the embodiment of the present invention, the functions defined for the data generation subtasks corresponding to the original fields of different data types are different, so that data of different original fields can be flexibly generated.

[0118] Optionally, in one embodiment, Figure 1 The SQL statement test method shown is as follows Figure 3 As shown, the method also includes steps S301-S305.

[0119] S301, obtaining the current SQL statement to be tested.

[0120] It is understandable that the driver server can obtain the SQL statement to be tested typed by the developer. Alternatively, the developer can pre-configure the expression of the SQL statement to be tested and the parameters of the field to be processed in the configuration file; the driver server can inject the parameters of the field to be processed in the configuration file into the expression of the SQL statement to be tested to obtain the SQL statement to be tested. The embodiment of the present invention only provides an example for obtaining the SQL statement to be tested and does not make any specific limitation.

[0121] The current SQL statement to be tested may be a query statement for querying data, an update statement for updating data, or an insert statement for inserting data, etc. In subsequent embodiments of the present invention, the execution time of the query operation, update operation, or insert operation of the current SQL statement to be tested may be detected.

[0122] Optionally, in another implementation, the driving server may send a test instruction to the test server, so that the test server executes steps S301 to S305. In this regard, the embodiment of the present invention is only used as an example and is not specifically limited.

[0123] S302: Send a prediction instruction to a designated database.

[0124] The prediction instruction is used to instruct the specified database to execute a task for a current SQL statement to be tested, and the execution task for a current SQL statement to be tested indicates executing a first preset number of data operations in sequence according to the current SQL statement to be tested.

[0125] It is understandable that a prediction instruction can be sent to a designated database so that after receiving the prediction instruction, the designated database performs a first preset number of data operations in the designated database in sequence according to a current SQL statement to be tested. Exemplarily, a test tool program can be pre-installed in the driver server. The test tool program can also be called a stress testing tool service. When the test work program is run, a current SQL statement to be tested can be generated, and a first preset number of data operation requests can be sent to the designated database in sequence. The data operation request includes the current SQL statement to be tested. After receiving the data operation request, the designated database performs the first preset number of data operations in sequence according to the current SQL statement to be tested in the data operation request. Among them, the prediction instruction can be the first preset number of data operation requests sent to the designated database in sequence.

[0126] S303, obtaining the execution time of each data operation.

[0127] It is understandable that the driver server can record the execution time of each data operation. For example, each execution time can be written into a log file or collected in real time through a monitoring tool. Since the data operation is executed a first preset number of times, the first preset number of execution times can be obtained. Among them, the first preset number can be a preset number of executions, for example, 200 times.

[0128] S304, calculating the average of multiple execution times as the predicted average response time of the current SQL statement to be tested; and determining the maximum time among the multiple execution times as the predicted maximum response time of the current SQL statement to be tested.

[0129] It can be understood that the sum of the first preset number of execution times can be divided by the first preset number to obtain the predicted average response time of the current SQL statement to be tested; the maximum time among the first preset number of execution times can be used as the predicted maximum response time of the current SQL statement to be tested.

[0130] S305: If the current predicted average response time is not greater than the first threshold, and the current predicted maximum response time is not greater than the second threshold, a stress test instruction is sent to the designated database.

[0131] The stress testing instruction is used to instruct the specified database to execute the execution tasks for a second preset number of SQL statements to be tested in parallel.

[0132] It is understandable that the execution time of the data operation can reflect the efficiency of the specified database in executing the SQL statement to be tested. The longer the execution time, the lower the execution efficiency. When the current predicted average response time is not greater than the first threshold, and the current predicted maximum response time is not greater than the second threshold, the execution efficiency of the current SQL statement to be tested is relatively high, and a large-scale test of executing a second preset number of SQL statements simultaneously can be performed. The second preset number is the number of SQL statements to be tested required for large-scale testing, for example, the second preset number can be 500.

[0133] In the embodiment of the present invention, the test phase of the SQL statement is divided into a prediction phase and a stress test phase. In the prediction phase, only the execution efficiency of one SQL statement to be tested is tested, and in the stress test phase, the execution efficiency of multiple SQL statements to be tested is tested. If the execution efficiency of one SQL statement is high in the prediction phase, the stress test phase can be performed to perform stress testing of multiple SQL statements executed simultaneously, which can reduce the impact of the low execution efficiency of a single SQL statement on the execution efficiency of multiple SQL statements executed simultaneously, and obtain accurate stress test results.

[0134] For special cases where the execution time is extremely low, for example, the specified database can utilize computing resource fluctuations and can call more computing resources for execution, in which case the execution time is relatively low. If an extremely low execution time is used to reflect the execution efficiency, there will be misjudgments. The execution efficiency can be accurately and stably reflected by presetting the average response time. If the execution efficiency of the SQL statement to be tested is determined to be high only by relying on the preset average response time, the situation where the execution time is long will be ignored, resulting in the inability to accurately obtain the execution efficiency of the stress testing phase. In the embodiment of the present invention, by presetting the average response time and predicting the maximum response time, it can be accurately determined that the execution efficiency of the current SQL statement to be tested is high, thereby improving the accuracy of the results of the subsequent stress testing phase.

[0135] Optionally, in one implementation, Figure 3 Based on the SQL statement testing method shown, the method also includes steps A1 and A2.

[0136] A1, after sending a stress test instruction to a designated database, obtain the execution time of each parallel execution of a second preset number of SQL statements to be tested by the designated database to obtain a third preset number of execution times.

[0137] It is understandable that, in the process of sending a stress test instruction to a designated database, the driver server can call a stress test tool to send a third preset number of data operation requests to the designated database, the data operation request includes a second preset number of current SQL statements to be tested, that is, the second preset number of SQL statements to be tested are executed in parallel in the designated database, and such parallel execution needs to be performed a third preset number of times. For example, the second preset number is 5, that is, the number of SQL statements to be tested is 5, and the third preset number is 1, that is, the number of parallel executions is 1. The driver server can send a data operation request to the designated database, the data operation request includes 5 SQL statements to be tested, and the designated database executes in parallel according to these 5 SQL statements to be tested, and a total of one parallel execution is completed through these 5 SQL statements to be tested. For another example, the second preset number is 5, that is, the number of SQL statements to be tested is 5, and the third preset number is 2, that is, the number of parallel executions is 2. The driver server can send a data operation request to the specified database, and the data operation request includes 5 SQL statements to be tested, and the first parallel execution is completed through these 5 SQL statements to be tested; then the driver server sends another data operation request to the specified database, and the data operation request includes 5 SQL statements to be tested, and the second parallel execution is completed through these 5 SQL statements to be tested, and a total of 2 parallel executions are completed.

[0138] After receiving each data operation request, the designated database executes the second preset number of SQL statements to be tested in the data operation request in parallel, and records the execution time of the current SQL statement to be tested. Since such parallel execution will be executed a third preset number of times, a third preset number of processing times can be obtained. Among them, the stress testing instruction can be a third preset number of data operation requests sent to the designated database in sequence. The third preset number and the first preset number are the number of executions for each SQL statement to be tested in the prediction phase and the stress testing phase. The third preset number and the first preset number may be the same or different. For example, the third preset number may be 200 or 100. In this regard, the embodiments of the present invention are only used as examples for illustration and are not specifically limited.

[0139] Exemplarily, the stress testing tool called in the stress testing phase may be: JMeter, a stress testing tool developed by a third-party company based on Java, and the driver server may call an interface of JMeter to call JMeter.

[0140] A2, calculating the average of the third preset number of execution times as the average response time of the stress test of the current SQL statement to be tested; and determining the maximum time among the multiple execution times as the maximum response time of the stress test of the current SQL statement to be tested.

[0141] It can be understood that the sum of the third preset number of processing times of any SQL statement in the second preset number of SQL statements to be tested is divided by the third preset number to obtain the average response time of the stress test of the current SQL statement to be tested; the maximum time among the third preset number of processing times can be used as the maximum response time of the stress test of the current SQL statement to be tested.

[0142] In the embodiment of the present invention, the execution efficiency when multiple SQL statements to be tested are executed simultaneously in a specified database can be accurately determined by stress testing the average response time and the maximum response time.

[0143] Optionally, in one embodiment, Figure 3 Based on the SQL statement test method shown in FIG. 1 , the current SQL statement to be tested is a query statement, such as Figure 4 As shown, the method also includes steps S401-S403.

[0144] S401: If the current predicted average response time is greater than a first threshold, and / or the current predicted maximum response time is greater than a second threshold, it is determined whether there is a preset low-efficiency query expression in the query expression in the current SQL statement to be tested.

[0145] Among them, each low-efficiency query expression has a corresponding high-efficiency query expression, and the execution efficiency of the SQL statement containing the low-efficiency query expression is lower than the execution efficiency of the SQL statement containing the high-efficiency query expression corresponding to the low-efficiency query expression; the query function implemented by any low-efficiency query expression is the same as that implemented by the corresponding high-efficiency query expression.

[0146] S402: If there is a preset low-efficiency query expression in the query expression in the current SQL statement to be tested, the high-efficiency query expression corresponding to the low-efficiency query expression in the SQL statement to be tested is used to replace the low-efficiency query expression in the SQL statement to be tested to obtain the current SQL statement to be tested; and return to execute the step of sending a prediction instruction to the specified database until the current predicted average response time is not greater than the first threshold, and the current predicted maximum response time is not greater than the second threshold.

[0147] It can be understood that the current SQL statement to be tested is a query statement, the current SQL statement to be tested is used to query data, and the current SQL statement to be tested contains a query expression.

[0148] There may be three situations where the execution efficiency of the current SQL statement to be tested is low: the current predicted average response time is greater than the first threshold, the current predicted maximum response time is greater than the second threshold, and the current predicted average response time is greater than the first threshold and the current predicted maximum response time is greater than the second threshold. If any of the above three situations is met, it can be reflected that the execution efficiency of the current SQL statement to be tested is low.

[0149] In order to improve the execution efficiency of the SQL statement to be tested, it is necessary to find the reason why the execution efficiency of the SQL statement to be tested is low. It can be determined whether there is a preset low-efficiency query expression in the query expression in the current SQL statement to be tested. If there is a low-efficiency query expression, the high-efficiency query expression corresponding to the low-efficiency query expression can be used to replace the low-efficiency query expression in the SQL statement to be tested. Since any low-efficiency query expression has the same query function as the corresponding high-efficiency query expression, and the execution efficiency of the SQL statement containing the low-efficiency query expression is less than the execution efficiency of the SQL statement containing the high-efficiency query expression corresponding to the low-efficiency query expression, the replaced SQL statement to be tested can achieve the function of the original SQL statement to be tested, and the execution efficiency is higher.

[0150] For example, for a MySQL database, the inefficient query expression for performing mathematical expression operations is WHERE column1+column2=100, and the high-efficiency query expression corresponding to the inefficient query expression is WHEREcolumn1=100-column2. Wherein, the WHERE clause is a clause used to specify the query condition in the SQL statement to be tested, column1 is field 1, and column2 is field 2.

[0151] It should be noted that you can use functions that compile regular expressions, such as Pattern.compile("WHERE\\s+(\\w+\\s*\\+\\s*\\w+\\s*=\\s*\\d+)"), to create low-efficiency query expressions and high-efficiency query expressions. Among them, \\s+ matches one or more spaces, which is used to match the spaces that may exist after the WHERE keyword. (\\w+\\s*\\+\\s*\\w+\\s*=\\s*\\d+): (...) represents a capture group used to extract the matched substring. \\w+ matches one or more alphanumeric characters, indicating matching field names. \\s* matches zero or more spaces, indicating spaces that may exist after the field name. \\+ matches the plus sign, indicating matching the addition operator. \\s* matches zero or more spaces, indicating spaces that may exist after the plus sign. \\w+ matches one or more alphanumeric characters again, indicating matching another field name. \\s* matches zero or more spaces, indicating spaces that may exist after the second field name. = matches the equal sign, indicating matching the equal sign. \\s* matches zero or more spaces, indicating that there may be spaces after the equal sign. \\d+ matches one or more digits, indicating that it matches a constant value.

[0152] The low-efficiency query expression used for function operation is WHERE UPPER(column) = 'VALUE', and the high-efficiency query expression corresponding to the low-efficiency query expression is WHERE column = 'value'. Among them, UPPER() is a function used to convert all letters in a field to uppercase, column is the field, and VALUE is the value.

[0153] The low-efficiency query expression for string operation is WHERE SUBSTRING(column,1,3)='ABC', and the high-efficiency query expression corresponding to this low-efficiency query expression is WHERE column LIKE'ABC%'. Among them, SUBSTRING() is a function used to extract a part of a string, column is the field to be extracted, 1 is the starting position of the string to be extracted, 3 is the length of the string to be extracted, ABC is the extracted string, and columnLIKE'ABC%' is a function to extract the string starting with ABC.

[0154] The low-efficiency query expression used to find the specified value is WHERE column = '123'. If the specified value 123 to be queried is an integer type, and 123 in the specified database is also an integer type, the high-efficiency query expression corresponding to the low-efficiency query expression is WHERE column = 123. Using this high-efficiency query expression can avoid full table scans caused by converting data types.

[0155] Optionally, in one implementation, the SQL statement to be tested can be parsed to check whether there is any improper use of the wildcard % in the WHERE clause. For example, if WHERE column LIKE'%value%' exists in the SQL statement to be tested, any value containing value at any position in the string can be searched. If value is used to match the first few characters in the string (prefix matching), the required data can also be found. In this case, the first % in %value% can be considered to be improperly used, and WHERE column LIKE'value%' can be used as the WHERE clause in the SQL statement to be tested. Among them, column LIKE'value%' means a string that starts with value and has any length.

[0156] Optionally, in another implementation, the SQL statement to be tested may be parsed to check whether a non-index column is used in an ORDER BY clause used to sort the query structure. If the use of a non-index column in the ORDER BY clause may result in poor sorting performance, the non-index column may be replaced with an index column of an existing index in the ORDER BY clause, or an index may be created for the non-index column.

[0157] In the embodiment of the present invention, by replacing the query expression in the WHERE clause, analyzing whether the wildcard is used improperly, and analyzing whether the non-index column is used in the ORDER BY clause, and by parsing the syntax of the SQL statement under the MySQL database, the reasons for the defects in the execution efficiency of the SQL statement can be output, and the MySQL database can be prevented from abandoning the use of the index and performing a full table scan when querying data, thereby improving the execution efficiency of the SQL statement.

[0158] Optionally, in one implementation, an interface of a third-party SQL performance analysis tool may be called to analyze the WHERE clause and the ORDER BY clause in the SQL statement to be tested through the third-party SQL performance analysis tool. To this end, the embodiment of the present invention is only used as an example for illustration and is not specifically limited.

[0159] It is understandable that after the aforementioned replacement of the current SQL statement to be tested, the execution efficiency of the new SQL statement to be tested may be improved. At this time, the new SQL statement to be tested can be processed in the prediction phase, that is, returning to the step of sending the prediction instruction to the specified database. If the predicted average response time of the new current SQL statement to be tested is not greater than the first threshold, and the current predicted maximum response time is not greater than the second threshold, the stress testing phase can be entered.

[0160] S403: If the query expression in the current SQL statement to be tested does not contain a preset low-efficiency query expression, a stress test instruction is sent to the designated database.

[0161] It is understandable that if there is no preset inefficient query expression in the query expression in the current SQL statement to be tested, the SQL statement to be tested has no room for improvement and directly enters the stress testing phase, that is, sending a stress testing instruction to the specified database.

[0162] In the embodiment of the present invention, a two-stage SQL statement testing method of a prediction stage and a stress testing stage is applied, and query expressions in SQL statements to be tested with low execution efficiency can be replaced in the prediction stage, thereby improving the execution efficiency of the SQL statements to be tested and improving the processing efficiency of subsequent SQL statements to be tested in the stress testing stage.

[0163] In order to store a large amount of data in the local database of the server to test the SQL statement involving a large amount of data query, the present invention also provides a SQL statement testing method, which is applied to each of the multiple working servers included in the test system, and the test system also includes a driver server, such as Figure 5 As shown, the method includes steps S501-S503.

[0164] S501, receiving a task sequence sent by a driving server through an execution process in each working server.

[0165] Among them, the task sequence is: the driver server parses the data definition language DDL statement representing the data structure of the data table to be generated, uses the directed acyclic graph scheduler component in the driver server to build it according to the parsing result, and uses the task scheduler component in the driver server to issue it to multiple working servers respectively; the task sequence includes data generation tasks and data writing tasks; the parsing results include: the original fields contained in the data table to be generated, and the arrangement order of the original fields in the data table to be generated; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the original field.

[0166] It is understandable that each working server can receive the task sequence issued by the driving server. The task sequence issued by the driving server has been described in the above embodiment, and will not be described in detail here.

[0167] S502: Generate a target data table including each original field and data belonging to each original field according to the data generation task.

[0168] The data belonging to each original field in any target data table are arranged in the original arrangement order.

[0169] It can be understood that the task sequence is the computing task that the work server needs to perform, and the data generation task in the task sequence represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated. After receiving the task sequence, each work server can execute the data generation subtask corresponding to each original field in sequence according to the original arrangement order of each original field when executing the data generation task in the task sequence. Exemplarily, according to the original arrangement order of each original field, the function that generates the data belonging to each original field is called to generate a target data table containing each original field and the data belonging to each original field. Specifically, each execution process in each work server generates a partial row of the target data table containing each original field and the data belonging to each original field.

[0170] S503: Write the generated target data table into a designated database according to the data writing task.

[0171] It is understandable that the working server can call the write function in the JDBC connector to write the generated target data table into the specified database. Specifically, each execution process in each working server can write part of the rows of the generated target data table into the specified database.

[0172] Based on the above processing, the parsing result includes the original fields contained in the data table to be generated and the arrangement order of the original fields in the data table to be generated. The data generation task in the task sequence constructed by the directed acyclic graph scheduler component based on the parsing result can represent the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field. Therefore, the data generation task is used to instruct each working server to generate the data of the column corresponding to each original field according to the original arrangement order, that is, the data generation task can be used to generate a data table with the same data structure as the data table to be generated. In the embodiment of the present invention, a task scheduler component is used to send a task sequence to each working server of the Spark cluster, and multiple working servers generate a target data table with the same data structure as the data table to be generated in parallel. The data structure of the data table to be generated is the same as the data structure of the data table in the specified database. The target data table is the same as the data structure of the data table in the specified database. The target data table is written into the specified database, which can ensure that a large number of target data tables written by multiple working servers are stored in the specified database, simulating the situation of storing a large amount of data in the specified database, so as to realize the storage of a large amount of data in the local database of the server, and thus can test SQL statements involving queries of large amounts of data.

[0173] Optionally, in another embodiment, Figure 5 Based on the SQL statement test method shown, Figure 6 As shown, step S502 includes steps S5021-S5023.

[0174] S5021: Generate an initial data table containing various original fields according to the data generation task.

[0175] In the initial data table, each original field corresponds to a column, and the original fields are arranged in the original arrangement order.

[0176] It can be understood that each original field corresponds to a column in the initial data table, and the original fields are arranged in the original order. That is, the initial data table has the same table structure as the data table to be generated, but there is no data under the original fields in the initial data table.

[0177] S5022: Determine the original fields of the data to be generated according to the original arrangement order.

[0178] S5023, generate a specified number of data belonging to the original field of the current data to be generated as the data in the column corresponding to the original field of the current data to be generated, and return to execute the step of determining the original field of the current data to be generated according to the original arrangement order until the data in the column corresponding to each original field is generated.

[0179] It is understandable that the original field of the data to be generated can be determined according to the original arrangement order, and the data in the column corresponding to the original field can be generated for the original field of the data to be generated. Then, according to the original arrangement order, the next original field of the original field of the generated data is used as the original field of the data to be generated, and data generation is performed until the data in the columns corresponding to each original field are generated.

[0180] In the embodiment of the present invention, data in the columns corresponding to the original fields are generated in sequence according to the original arrangement order, thereby ensuring the integrity of the generated target data table and improving the accuracy of SQL statement testing.

[0181] Optionally, in one implementation, step S5023 includes steps S50231-S50234.

[0182] S50231, for the original field of the data to be generated, if the original field does not belong to the field to be processed, a specified number of empty data are used as the data in the column corresponding to the original field.

[0183] The fields to be processed are obtained by driving the server to obtain the SQL statement to be tested for processing the data table in the specified database, parsing the SQL statement to be tested, and determining the fields to be processed by the SQL statement to be tested.

[0184] It is understandable that if the original field of the data to be generated does not belong to the field to be processed, the data in the column corresponding to the original field can be assigned a null value. The field to be processed has been described in the above embodiment and will not be described in detail here.

[0185] S50232, if the original field belongs to the field to be processed and the original data type of the original field is a numeric type, then generate a specified number of data according to the first function defined in the data generation subtask corresponding to the original field as the data in the column corresponding to the original field.

[0186] The first function is used to instruct each working server to generate data belonging to the original field within a preset value range.

[0187] S50233, if the original field belongs to the field to be processed and the original data type of the original field is a string type, then generate a specified number of data according to the second function defined in the data generation subtask corresponding to the original field as the data in the column corresponding to the original field.

[0188] The second function is used to instruct each working server to select a character string from a preset character string set to generate data belonging to the original field.

[0189] S50234, if the original field belongs to the field to be processed and the original data type of the original field is a date type, then generate a specified number of data according to the third function defined in the data generation subtask corresponding to the original field as the data in the column corresponding to the original field.

[0190] The third function is used to instruct each working server to select a date in the specified date range to generate data belonging to the original field.

[0191] It is understandable that if the original field of the data to be generated currently belongs to the field to be processed, then the function can be called to generate the data in the column corresponding to the original field according to the data type of the original field. Among them, the first function, the second function and the third function have been described in the aforementioned embodiment, and will not be repeated here.

[0192] In the embodiment of the present invention, data of corresponding columns are generated according to different situations of original fields, so as to simulate various types of data in a specified database and improve the test accuracy of SQL statements.

[0193] Optionally, in one embodiment, Figure 6 Based on the SQL statement test method shown in the figure, the initial data table also contains a primary key field, such as Figure 7 As shown, the method also includes step S701.

[0194] S701, using a specified value as an initial primary key value, and sequentially generating a specified number of primary key values ​​belonging to a primary key field in a manner of increasing by 1 row by row.

[0195] The initial primary key values ​​corresponding to the working servers are different, and the primary key values ​​generated by the working servers do not have the same value;

[0196] For each original field, the data of the last row in the column corresponding to the generated original field belongs to the same row as the generated maximum primary key value.

[0197] It is understandable that the parsing result obtained by the driver server parsing includes the original fields of the data table to be generated. The driver server can also add a primary key field to the parsing result, and then when creating the data generation task in the task sequence, it can also create a data generation subtask representing the data of the column corresponding to the primary key field. The data generation subtask of the primary key field can define a function for generating data within a specified range, and the specified range can be represented by the initial primary key value of each working server and a specified number. The data generation subtask of the primary key field can be before the data generation subtask of each original field. After receiving the task sequence, each working server can generate an initial data table containing each original field and the primary key field.

[0198] Optionally, after generating each original field, the working server can call a removal function to remove the primary key field and the data under the primary key field in the initial data table. In the embodiment of the present invention, the primary key field can be used to ensure the order and uniqueness of data when generating data, and after generating the initial data table, the primary key field is removed to ensure that the generated initial data table structure does not contain the primary key field and is consistent with the table structure of the data table in the specified database. The removal function can be .drop("id"), where id represents the primary key field.

[0199] Each working server can use the specified value as the initial primary key value, and increase it by 1 row by row, to generate a specified number of primary key values ​​belonging to the primary key field at a time. For example, the specified value is 1, the specified number is 10, and the primary key values ​​of each row are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Exemplarily, the spark.range function defined in the data generation subtask of the primary key field can be called to use the specified value as the initial primary key value, and increase it by 1 row by row to generate a specified number of primary key values ​​belonging to the primary key field in sequence.

[0200] For each original field, the data in the last row of the column corresponding to the original field generated belongs to the same row as the generated maximum primary key value, that is, the specified number is the number of data required to be generated for each original field. For example, if the specified number is 5, there are 5 primary key values ​​under the primary key field and 5 data under the original field.

[0201] The initial primary key values ​​corresponding to each work server are different, and the primary key values ​​generated by each work server do not have the same value. For example, the initial primary key value of work server a is 1, the specified number is 10, and the initial primary key value of work server b is 11. The primary key values ​​of each row generated by work server a are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, and the primary key values ​​of each row generated by work server b are: 11, 12, 13, 14, 15, 16, 17, 18, 19, 20.

[0202] In the embodiment of the present invention, by generating different primary key values ​​for each working server, the primary key value written into the target data table in the specified database is guaranteed to be unique, the data volume of each original field generated by the working server is ensured to meet the requirements, and the accuracy of SQL statement testing is improved.

[0203] In order to better understand the embodiments of the present invention, Figure 8 The present invention introduces a SQL statement testing method provided by an embodiment of the present invention.

[0204] like Figure 8As shown, the original fields in the data table to be generated include: mobile phone number, capital account number and customer number, and the mobile phone number, capital account number and customer number are the fields to be processed in the SQL statement to be tested. Since the mobile phone number, capital account number and customer number involve sensitive customer privacy, the generated mobile phone number, capital account number and customer number can simulate the data of real customers, so the mobile phone number, capital account number and customer number can also be collectively referred to as desensitized fields. Generating data under the mobile phone number field, capital account field and customer number field in the data table to be generated can also be called desensitized field generation.

[0205] The driver server can obtain the DDL statement of the data structure of the data table to be generated; parse the DDL statement to obtain the parsing result; the parsing result includes: the original fields (mobile phone number, financial account number and customer number) contained in the data table to be generated, and the arrangement order of the original fields in the data table to be generated; use the directed acyclic graph scheduler component to build a task sequence based on the parsing result, and use the task scheduler component to send the task sequence to each working server respectively.

[0206] Each working server can generate a target data table including each original field and the data belonging to each original field according to the data generation task by executing the process after receiving the task sequence sent by the driving server.

[0207] Specifically, for the mobile phone number field, the data type of the mobile phone number field is a numeric type, and a specified number of data is generated according to the first function defined in the data generation subtask corresponding to the mobile phone number field as the data in the column corresponding to the mobile phone number field, for example, the rand() function is called to generate random numbers as the data under the mobile phone number field. That is, the process of generating a mobile phone number.

[0208] For the fund account field, the data type of the fund account field is a numeric type. According to the first function defined in the data generation subtask corresponding to the fund account field, a specified number of data is generated as the data in the column corresponding to the fund account field. For example, the rand() function is called to generate random numbers as the data under the fund account field. That is, the process of generating a fund account.

[0209] For the customer number field, the data type of the customer number field is a numeric type. According to the first function defined in the data generation subtask corresponding to the customer number field, a specified number of data is generated as the data in the column corresponding to the customer number field. For example, the rand() function is called to generate random numbers as the data under the customer number field. That is, the process of generating a customer number.

[0210] Each working server writes the generated target data table into the designated database according to the data writing task. In the embodiment of the present invention, the mobile phone number, fund account number and customer number are used as the data in the user's virtual account to simulate the real data storage situation of the securities company, thereby ensuring the test accuracy of the SQL statement to be tested.

[0211] In one embodiment, Fig. 9 A schematic diagram of the principles of a SQL statement testing method provided by an embodiment of the present invention.

[0212] like Fig. 9 As shown, the driver server can obtain the DDL statement representing the data structure of the data table to be generated; parse the DDL statement to obtain the parsing result. A DDL parser can be created based on the configuration file. By parsing the DDL statement through the parsing function for the original fields of different data types in the DDL parser, the original fields contained in the data table to be generated, the arrangement order of the original fields in the data table to be generated, and the data type of the original fields can be obtained. Among them, the original fields contained in the data table to be generated and the arrangement order of the original fields in the data table to be generated can also be called table structure information.

[0213] The driver server can use the directed acyclic graph scheduler component to build a task sequence based on the parsing results. Specifically, the schema of the table data structure (DataFrame) of the data table to be generated can be defined based on the parsing results. According to the schema of the DataFrame (i.e., the Spark DataFrame Schema), the transformation operation of the DataFrame is divided into tasks of a stage to obtain the data generation task. Then, a task sequence including data generation tasks and data writing tasks is constructed. Among them, the data generation task can be to call different functions to generate data for the original fields of different data types. Then, the task sequence is sent to each working server respectively.

[0214] The structure diagram of DataFrame is as follows: Fig.10 As shown, there are three columns. The first column is the name field, and the data type of the name field is string type. The second column is the age field, and the age field is integer type. The third column is the height field, and the height field is double precision floating point type. Among them, integer type and double precision floating point type are both numeric types.

[0215] Each working server can receive the task sequence sent by the driving server through the execution process; according to the data generation task, a target data table containing each original field and the data belonging to each original field is generated, that is, test data is generated. According to the data writing task, the generated target data table is written to the specified database. The specific data writing process can be: calling the interface of the JDBC connector, connecting to the MySQL database and writing the generated target data table to the specified database in the MySQL database.

[0216] For the adjustment and expansion process, developers can adjust the parsing functions of different data types in the DDL parser according to actual needs, that is, adjust the way to create the DDL parser in the configuration file. It is also possible to add a function called by the original field of a data type or adjust the function called by the original field of a data type according to actual needs, that is, adjust the way to build the task sequence in the configuration file to expand and adjust the data generation logic. The code and related parameters (such as DDL statements, preset value ranges, preset string sets, specified dates, etc.) required for the driver server to execute the SQL statement test method can be saved as a Scala file, and the Scala file can be packaged into a Java Archive File (JAR) using a simple build tool (Simple Build Tool, SBT) or other build tools. The driver server can submit the Spark application to the Spark cluster through the spark-submit command to implement the task sequence to each working server in the Spark cluster, so that each working server generates a target data table, connects to the MySQL database, and writes the target data table into the MySQL database. The driver server can obtain the current SQL statement to be tested, and query the data in the MySQL database through the obtained SQL statement to be tested.

[0217] In one embodiment, Fig.11 A schematic diagram of the principles of a SQL statement testing method provided by an embodiment of the present invention.

[0218] like Fig.11 As shown, the driver server obtains the current SQL statement to be tested and sends a prediction instruction to the specified database; obtains the execution time of each data operation; calculates the average of multiple execution times as the predicted average response time of the current SQL statement to be tested; and determines the maximum time among the multiple execution times as the predicted maximum response time of the current SQL statement to be tested; if the current predicted average response time is greater than the first threshold, and / or the current predicted maximum response time is greater than the second threshold, it is determined whether the query expression in the current SQL statement to be tested contains a preset inefficient query expression.

[0219] Specifically, for the query expression in the current SQL statement to be tested, determine whether the query expression in the WHERE clause that performs mathematical expression operations on the field is an inefficient query expression. The inefficient query expression for performing mathematical expression operations is WHERE column1+column2=100, and the high-efficiency query expression corresponding to the inefficient query expression is WHERE column1=100-column2.

[0220] If the query expression for performing mathematical expression operation on the field is an inefficient query expression, the high-efficiency query expression WHERE column1=100-column2 is output as a modification suggestion, that is, an optimization suggestion is output.

[0221] If the query expression for performing mathematical expression operation on the field is not an inefficient query expression, determine whether the query expression for performing function operation on the field in the WHERE clause is an inefficient query expression. The inefficient query expression for performing function operation is WHERE UPPER(column) = 'VALUE', and the high-efficiency query expression corresponding to the inefficient query expression is WHERE column = 'value'.

[0222] If the query expression that performs a function operation on a field is an inefficient query expression, a high-efficiency query expression WHERE column = 'value' is output as a modification suggestion, that is, an optimization suggestion is output.

[0223] If the query expression that performs function operation on the field is not an inefficient query expression, determine whether the query expression that performs string operation on the field in the WHERE clause is an inefficient query expression. The inefficient query expression used for string operation is WHERE SUBSTRING(column,1,3)='ABC', and the high-efficiency query expression corresponding to the inefficient query expression is WHERE column LIKE'ABC%'.

[0224] If the query expression for performing string operations on the field is an inefficient query expression, the high-efficiency query expression WHERE column LIKE'ABC%' is output as a modification suggestion, that is, an optimization suggestion is output.

[0225] If the query expression for string operation on the field is not an inefficient query expression, when the data type in the SQL statement to be tested is consistent with the data type in the specified database, determine whether the query expression for searching for the specified value in the WHERE clause is an inefficient query expression. The inefficient query expression for searching for the specified string is WHERE column = '123', and the high-efficiency query expression corresponding to the inefficient query expression is WHERE column = 123.

[0226] If the query expression for finding the specified value is an inefficient query expression, the high-efficiency query expression WHEREcolumn=123 is output as a modification suggestion, that is, an optimization suggestion is output.

[0227] If the query expression for finding the specified value is not an inefficient query expression, determine whether the wildcard % is used improperly in the WHERE clause.

[0228] If there is an improper use of the wildcard %, the WHERE clause in which the wildcard % is improperly used is output as a modification suggestion, that is, an optimization suggestion is output.

[0229] If there is no improper use of the wildcard character %, determine whether a non-index column is used in the ORDER BY clause for sorting the query structure.

[0230] If the non-index column is not used, the output is the correct result of the SQL statement to be tested.

[0231] If non-index columns are used, the ORDER BY clause of the non-index columns will be used as the modification suggestion output, that is, the optimization suggestion is output.

[0232] Developers can optimize the SQL statement to be tested according to the optimization suggestions. The driver server uses the optimized SQL statement to be tested as the current SQL statement to be tested and returns to execute the step of sending the prediction instruction to the specified database.

[0233] The embodiment of the present invention can avoid the MySQL database abandoning the use of indexes and performing a full table scan when querying data by replacing query expressions in the WHERE clause, analyzing whether wildcards are used improperly, and analyzing whether non-index columns are used in the ORDER BY clause, thereby improving the execution efficiency of SQL statements.

[0234] In one embodiment, Fig.12 A schematic diagram of a SQL statement testing method provided by an embodiment of the present invention

[0235] like Fig.12 As shown, developers can write test tool programs and install the test tool programs in the driver server.

[0236] After each working server writes the target data table to the specified database, you can use JMeter to perform large-scale stress testing directly after generating the desensitized data. Or the driver server can obtain the current SQL statement to be tested and send a prediction instruction to the specified database. The prediction instruction is used to instruct the specified database to perform a query task for a current SQL statement to be tested, that is, to enter the prediction stage and perform the prediction. Then, obtain the query duration of each data query processing (collect performance data). Calculate the average of multiple query durations as the predicted average response time of the current SQL statement to be tested; and determine the maximum duration among the multiple query durations as the predicted maximum response time of the current SQL statement to be tested (analyze performance data).

[0237] If the current predicted average response time is greater than the first threshold, and / or the current predicted maximum response time is greater than the second threshold, determine whether there is a preset low-efficiency query expression in the query expression in the current SQL statement to be tested. If there is a preset low-efficiency query expression in the query expression in the current SQL statement to be tested, replace the low-efficiency query expression in the SQL statement to be tested with a high-efficiency query expression corresponding to the low-efficiency query expression in the SQL statement to be tested to obtain the current SQL statement to be tested; and return to execute the step of sending a prediction instruction to the specified database until the current predicted average response time is not greater than the first threshold, and the current predicted maximum response time is not greater than the second threshold. That is, optimize and repeat the prediction.

[0238] In one embodiment, the present invention also provides a SQL statement testing system, such as Fig.13 As shown, the system includes multiple worker servers 1320 and a driver server 1310 in a Spark cluster;

[0239] The driver server 1310 is used to obtain the DDL statement representing the data structure of the data table to be generated; parse the DDL statement to obtain the parsing result; construct a task sequence based on the parsing result; and send the task sequence to each working server 1320 respectively.

[0240] The driver server 1310 is further used to obtain the SQL statement to be tested for processing the data table in the specified database; parse the SQL statement to be tested, and determine the fields to be processed by the SQL statement to be tested as the fields to be processed.

[0241] The driver server 1310 is also used to obtain the current SQL statement to be tested; send a prediction instruction to a specified database; obtain the execution time of each data operation; calculate the average of multiple execution times as the predicted average response time of the current SQL statement to be tested; and determine the maximum time among multiple query times as the predicted maximum response time of the current SQL statement to be tested; if the current predicted average response time is not greater than the first threshold, and the current predicted maximum response time is not greater than the second threshold, send a stress test instruction to the specified database.

[0242] Each working server 1320 among the multiple working servers is used to receive the task sequence sent by the driving server 1310; generate a target data table containing each original field and the data belonging to each original field according to the data generation task; and write the generated target data table into the specified database according to the data writing task.

[0243] Any working server 1320 is further configured to perform an execution task for a current SQL statement to be tested after receiving a prediction instruction. The execution task for a current SQL statement to be tested indicates executing a first preset number of data operations in sequence according to the current SQL statement to be tested.

[0244] Any working server 1320 is further configured to execute a second preset number of current SQL statements to be tested in parallel after receiving a stress testing instruction.

[0245] Based on the above processing, the parsing result includes the original fields contained in the data table to be generated and the arrangement order of the original fields in the data table to be generated. The data generation task in the task sequence constructed by the directed acyclic graph scheduler component based on the parsing result can represent the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field. Therefore, the data generation task is used to instruct each working server to generate the data of the column corresponding to each original field according to the original arrangement order, that is, the data generation task can be used to generate a data table with the same data structure as the data table to be generated. In the embodiment of the present invention, a task scheduler component is used to send a task sequence to each working server of the Spark cluster, and multiple working servers generate a target data table with the same data structure as the data table to be generated in parallel. The data structure of the data table to be generated is the same as the data structure of the data table in the specified database. The target data table is the same as the data structure of the data table in the specified database. The target data table is written into the specified database, which can ensure that a large number of target data tables written by multiple working servers are stored in the specified database, simulating the situation of storing a large amount of data in the specified database, so as to realize the storage of a large amount of data in the local database of the server, and thus can test SQL statements involving queries of large amounts of data.

[0246] In one embodiment, if Fig.14 As shown, an embodiment of the present invention further provides a SQL statement testing device, which is applied to a driver server in a Spark cluster, wherein the Spark cluster also includes multiple working servers, and the device includes:

[0247] The first acquisition module 1410 is used to acquire a data definition language DDL statement representing a data structure of a data table to be generated; wherein the data structure of the data table to be generated is the same as the data structure of a data table in a specified database;

[0248] The first parsing module 1420 is used to parse the DDL statement to obtain a parsing result; wherein the parsing result includes: each original field included in the data table to be generated, and the arrangement order of each original field in the data table to be generated;

[0249] A construction module 1430 is used to construct a task sequence based on the parsing result by using the directed acyclic graph scheduler component in the driving server; wherein the task sequence includes a data generation task and a data writing task; the data generation task represents a data generation subtask corresponding to each original field and an original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents generating data belonging to a column corresponding to the original field;

[0250] The sending module 1440 is used to use the task scheduler component in the driving server to send the task sequence to each working server respectively, so that each working server generates a target data table containing each original field and the data belonging to each original field according to the data generation task, and writes the generated target data table into the designated database according to the data writing task; wherein the data belonging to each original field in any target data table is arranged according to the original arrangement order.

[0251] Optionally, the device further comprises:

[0252] A second acquisition module is used to acquire a SQL statement to be tested for querying a data table in the specified database;

[0253] A second parsing module is used to parse the SQL statement to be tested and determine the fields to be processed in the SQL statement to be tested as the fields to be processed;

[0254] For each original field that does not belong to the determined fields to be processed, the data generation subtask corresponding to the original field is used to instruct each working server to generate a data table in which the data belonging to the original field is empty.

[0255] Optionally, the parsing result further includes: the original data type of each original field;

[0256] For each original field that belongs to the determined field to be processed, the data generation subtask corresponding to the original field represents: generating data belonging to the original field, and the generated data is the target data type of the original field; the target data type of each original field represents: the data type supported by the multiple working servers and corresponding to the original data type of the original field.

[0257] Optionally, for each original field belonging to the determined to-be-processed field, if the original data type of the original field is a numerical type, a first function is defined in the data generation subtask corresponding to the original field; the first function is used to instruct each working server to generate data belonging to the original field within a preset numerical range;

[0258] If the original data type of the original field is a string type, a second function is defined in the data generation subtask corresponding to the original field; the second function is used to instruct each working server to select a string from a preset string set to generate data belonging to the original field;

[0259] If the original data type of the original field is a date type, a third function is defined in the data generation subtask corresponding to the original field; the third function is used to instruct each working server to select a date in a specified date range to generate data belonging to the original field.

[0260] Optionally, the device further comprises:

[0261] The third acquisition module is used to obtain the current SQL statement to be tested;

[0262] A first sending module is used to send a prediction instruction to the designated database; wherein the prediction instruction is used to instruct the designated database to execute a task for a current SQL statement to be tested, and the execution task of the current SQL statement to be tested indicates that a first preset number of data operations are executed in sequence according to the current SQL statement to be tested;

[0263] The fourth acquisition module is used to obtain the execution time of each data query processing;

[0264] The first calculation module is used to calculate the average of multiple execution times as the predicted average response time of the current SQL statement to be tested; and determine the maximum time among the multiple execution times as the predicted maximum response time of the current SQL statement to be tested;

[0265] The second sending module is used to send a stress testing instruction to the designated database if the current predicted average response time is not greater than the first threshold and the current predicted maximum response time is not greater than the second threshold; wherein the stress testing instruction is used to instruct the designated database to execute the execution tasks for a second preset number of current SQL statements to be tested in parallel.

[0266] Optionally, the current SQL statement to be tested is a query statement, and the device further includes:

[0267] A judgment module, used for judging whether there is a preset low-efficiency query expression in the query expression of the current SQL statement to be tested if the current predicted average response time is greater than a first threshold value, and / or the current predicted maximum response time is greater than a second threshold value; wherein each low-efficiency query expression has a corresponding high-efficiency query expression, and the execution efficiency of the SQL statement containing the low-efficiency query expression is less than the execution efficiency of the SQL statement containing the high-efficiency query expression corresponding to the low-efficiency query expression; and the query function implemented by any low-efficiency query expression is the same as that implemented by the corresponding high-efficiency query expression;

[0268] A replacement module is used for replacing the inefficient query expression in the SQL statement to be tested with a high-efficiency query expression corresponding to the inefficient query expression in the SQL statement to be tested, if there is a preset inefficient query expression in the query expression in the current SQL statement to be tested, to obtain the current SQL statement to be tested; and returning to execute the step of sending the prediction instruction to the designated database until the current predicted average response time is not greater than the first threshold value, and the current predicted maximum response time is not greater than the second threshold value;

[0269] The third sending module is used to send the stress testing instruction to the designated database if there is no preset low-efficiency query expression in the query expression in the current SQL statement to be tested.

[0270] Optionally, the device further comprises:

[0271] A fifth acquisition module is used to acquire, after sending the stress testing instruction to the designated database, the execution time of each parallel execution of a second preset number of current SQL statements to be tested by the designated database, to obtain a third preset number of execution times;

[0272] The second calculation module is used to calculate the average of the third preset number of execution times as the average response time of the stress test of the current SQL statement to be tested; and determine the maximum time among the multiple execution times as the maximum response time of the stress test of the current SQL statement to be tested.

[0273] Based on the above processing, the parsing result includes the original fields contained in the data table to be generated and the arrangement order of the original fields in the data table to be generated. The data generation task in the task sequence constructed by the directed acyclic graph scheduler component based on the parsing result can represent the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field. Therefore, the data generation task is used to instruct each working server to generate the data of the column corresponding to each original field according to the original arrangement order, that is, the data generation task can be used to generate a data table with the same data structure as the data table to be generated. In the embodiment of the present invention, a task scheduler component is used to send a task sequence to each working server of the Spark cluster, and multiple working servers generate a target data table with the same data structure as the data table to be generated in parallel. The data structure of the data table to be generated is the same as the data structure of the data table in the specified database. The target data table is the same as the data structure of the data table in the specified database. The target data table is written into the specified database, which can ensure that a large number of target data tables written by multiple working servers are stored in the specified database, simulating the situation of storing a large amount of data in the specified database, so as to realize the storage of a large amount of data in the local database of the server, and thus can test SQL statements involving queries of large amounts of data.

[0274] In one embodiment, if Fig.15 As shown, an embodiment of the present invention further provides a SQL statement testing device, which is applied to each working server of a plurality of working servers included in a testing system, wherein the testing system further includes a driving server, and the device includes:

[0275] The receiving module 1510 is used to receive the task sequence sent by the driving server through the execution process in each working server; wherein the task sequence is: the driving server parses the acquired data definition language DDL statement representing the data structure of the data table to be generated, constructs it according to the parsing result using the directed acyclic graph scheduler component in the driving server, and issues it to the multiple working servers respectively using the task scheduler component in the driving server; the data structure of the data table to be generated is the same as the data structure of the data table in the specified database; the task sequence includes a data generation task and a data writing task; the parsing result includes: each original field contained in the data table to be generated, and the arrangement order of each original field in the data table to be generated; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the original field;

[0276] A generating module 1520 is used to generate a target data table including each original field and data belonging to each original field according to the data generating task; wherein the data belonging to each original field in any target data table is arranged according to the original arrangement order;

[0277] The writing module 1530 is used to write the generated target data table into the designated database according to the data writing task.

[0278] Optionally, the generating module 1520 includes:

[0279] A first generating unit is used to generate an initial data table including various original fields according to the data generating task; wherein, in the initial data table, each original field corresponds to a column, and the original fields are arranged according to the original arrangement order;

[0280] A determination unit, used to determine the original fields of the data to be generated according to the original arrangement order;

[0281] The second generating unit is used to generate a specified number of data belonging to the original field of the data to be generated currently as the data in the column corresponding to the original field of the data to be generated currently, and return to execute the step of determining the original field of the data to be generated currently according to the original arrangement order until the data in the column corresponding to each original field is generated.

[0282] Optionally, the second generating unit is specifically configured to, for an original field of the data to be generated, if the original field does not belong to the field to be processed, use a specified number of empty data as data in a column corresponding to the original field; wherein the field to be processed is obtained by the driving server obtaining a SQL statement to be tested for querying a data table in the specified database, parsing the SQL statement to be tested, and determining the field to be processed by the SQL statement to be tested;

[0283] If the original field belongs to the field to be processed, and the original data type of the original field is a numerical type, a specified number of data are generated according to the first function defined in the data generation subtask corresponding to the original field as the data in the column corresponding to the original field; wherein the first function is used to instruct each working server to generate data belonging to the original field within a preset numerical range;

[0284] If the original field belongs to the field to be processed, and the original data type of the original field is a string type, a specified number of data are generated according to the second function defined in the data generation subtask corresponding to the original field as the data in the column corresponding to the original field; the second function is used to instruct each working server to select a string from a preset string set to generate data belonging to the original field;

[0285] If the original field belongs to the field to be processed and the original data type of the original field is a date type, then a specified number of data are generated according to the third function defined in the data generation subtask corresponding to the original field as the data in the column corresponding to the original field; the third function is used to instruct each working server to select a date in the specified date range to generate data belonging to the original field.

[0286] Optionally, the initial data table also includes a primary key field;

[0287] The device also includes:

[0288] A primary key generation module, used to use a specified value as an initial primary key value, and to sequentially generate a specified number of primary key values ​​belonging to the primary key field in a manner of increasing by 1 row by row; wherein the initial primary key values ​​corresponding to the respective working servers are different, and the primary key values ​​generated by the respective working servers do not have the same value;

[0289] For each original field, the data of the last row in the column corresponding to the generated original field belongs to the same row as the generated maximum primary key value.

[0290] Based on the above processing, the parsing result includes the original fields contained in the data table to be generated and the arrangement order of the original fields in the data table to be generated. The data generation task in the task sequence constructed by the directed acyclic graph scheduler component based on the parsing result can represent the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field. Therefore, the data generation task is used to instruct each working server to generate the data of the column corresponding to each original field according to the original arrangement order, that is, the data generation task can be used to generate a data table with the same data structure as the data table to be generated. In the embodiment of the present invention, a task scheduler component is used to send a task sequence to each working server of the Spark cluster, and multiple working servers generate a target data table with the same data structure as the data table to be generated in parallel. The data structure of the data table to be generated is the same as the data structure of the data table in the specified database. The target data table is the same as the data structure of the data table in the specified database. The target data table is written into the specified database, which can ensure that a large number of target data tables written by multiple working servers are stored in the specified database, simulating the situation of storing a large amount of data in the specified database, so as to realize the storage of a large amount of data in the local database of the server, and thus can test SQL statements involving queries of large amounts of data.

[0291] The embodiment of the present invention further provides an electronic device, such as Fig.16As shown, it includes a processor 1601 , a communication interface 1602 , a memory 1603 and a communication bus 1604 , wherein the processor 1601 , the communication interface 1602 , and the memory 1603 communicate with each other via the communication bus 1604 .

[0292] Memory 1603, used for storing computer programs;

[0293] The processor 1601 is used to implement the SQL statement testing method in the above-mentioned embodiment when executing the program stored in the memory 1603.

[0294] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0295] The communication interface is used for communication between the above electronic device and other devices.

[0296] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0297] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0298] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned SQL statement testing methods are implemented.

[0299] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any one of the SQL statement testing methods in the above embodiments.

[0300] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0301] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0302] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment and the system embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0303] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A SQL statement testing method, characterized in that: The method is applied to a driver server in a Spark cluster, wherein the Spark cluster further includes a plurality of working servers, and the method includes: Obtaining a data definition language DDL statement representing a data structure of a data table to be generated; wherein the data structure of the data table to be generated is the same as the data structure of a data table in a specified database; Parsing the DDL statement to obtain a parsing result; wherein the parsing result includes: each original field contained in the data table to be generated, and the arrangement order of each original field in the data table to be generated; Using the directed acyclic graph scheduler component in the driving server, a task sequence is constructed based on the parsing result; wherein the task sequence includes a data generation task and a data writing task; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field; Utilize the task scheduler component in the driving server to send the task sequence to each working server respectively, so that each working server generates a target data table containing each original field and the data belonging to each original field according to the data generation task, and writes the generated target data table into the designated database according to the data writing task; wherein, the data belonging to each original field in any target data table is arranged according to the original arrangement order.

2. The method according to claim 1, characterized in that Before using the directed acyclic graph scheduler component in the driver server to construct a task sequence based on the parsing result, the method further includes: Obtaining a SQL statement to be tested for processing a data table in the specified database; Parsing the SQL statement to be tested, and determining the fields to be processed in the SQL statement to be tested as the fields to be processed; Wherein, for each original field that does not belong to the determined to-be-processed fields, the data generation subtask corresponding to the original field is used to instruct each working server to generate a data table in which the data belonging to the original field is empty; The parsing result also includes: the original data type of each original field; For each original field that belongs to the determined field to be processed, the data generation subtask corresponding to the original field represents: generating data belonging to the original field, and the generated data is the target data type of the original field; the target data type of each original field represents: the data type supported by the multiple working servers and corresponding to the original data type of the original field.

3. The method according to claim 2, characterized in that For each original field belonging to the determined to-be-processed field, if the original data type of the original field is a numerical type, a first function is defined in the data generation subtask corresponding to the original field; the first function is used to instruct each working server to generate data belonging to the original field within a preset numerical range; If the original data type of the original field is a string type, a second function is defined in the data generation subtask corresponding to the original field; the second function is used to instruct each working server to select a string from a preset string set to generate data belonging to the original field; If the original data type of the original field is a date type, a third function is defined in the data generation subtask corresponding to the original field; the third function is used to instruct each working server to select a date in a specified date range to generate data belonging to the original field.

4. The method according to claim 1, characterized in that After using the task scheduler component in the driver server to send the task sequence to each working server respectively, the method further includes: Get the current SQL statement to be tested; Sending a prediction instruction to the designated database; wherein the prediction instruction is used to instruct the designated database to execute a task for a current SQL statement to be tested, and the execution task of the current SQL statement to be tested indicates executing a first preset number of data operations in sequence according to the current SQL statement to be tested; Get the execution time of each data operation; Calculate the average of multiple execution times as the predicted average response time of the current SQL statement to be tested; and determine the maximum time among the multiple execution times as the predicted maximum response time of the current SQL statement to be tested; If the current predicted average response time is not greater than the first threshold, and the current predicted maximum response time is not greater than the second threshold, a stress test instruction is sent to the designated database; wherein the stress test instruction is used to instruct the designated database to execute execution tasks for a second preset number of current SQL statements to be tested in parallel; Obtaining the execution time of the second preset number of current SQL statements to be tested being executed in parallel by the designated database each time, to obtain a third preset number of execution times; Calculate the average of the third preset number of execution times as the average response time of the stress test of the current SQL statement to be tested; and determine the maximum time among the multiple execution times as the maximum response time of the stress test of the current SQL statement to be tested.

5. The method according to claim 4, characterized in that The current SQL statement to be tested is a query statement, and the method further includes: If the current predicted average response time is greater than the first threshold, and / or the current predicted maximum response time is greater than the second threshold, it is determined whether there is a preset low-efficiency query expression in the query expression in the current SQL statement to be tested; wherein each low-efficiency query expression has a corresponding high-efficiency query expression, and the execution efficiency of the SQL statement containing the low-efficiency query expression is less than the execution efficiency of the SQL statement containing the high-efficiency query expression corresponding to the low-efficiency query expression; any low-efficiency query expression and the corresponding high-efficiency query expression implement the same query function; If there is a preset low-efficiency query expression in the query expression of the current SQL statement to be tested, the high-efficiency query expression corresponding to the low-efficiency query expression in the SQL statement to be tested is used to replace the low-efficiency query expression in the SQL statement to be tested, so as to obtain the current SQL statement to be tested; and returning to execute the step of sending the prediction instruction to the designated database until the current predicted average response time is not greater than the first threshold and the current predicted maximum response time is not greater than the second threshold; If the query expression in the current SQL statement to be tested does not contain a preset low-efficiency query expression, the stress testing instruction is sent to the designated database.

6. A SQL statement testing method, characterized in that: The method is applied to each working server of a plurality of working servers included in a test system, wherein the test system further includes a driving server, and the method includes: The task sequence sent by the driving server is received through the execution process in each working server; wherein the task sequence is: the driving server parses the acquired data definition language DDL statement representing the data structure of the data table to be generated, constructs it according to the parsing result using the directed acyclic graph scheduler component in the driving server, and issues it to the multiple working servers respectively using the task scheduler component in the driving server; the data structure of the data table to be generated is the same as the data structure of the data table in the specified database; the task sequence includes data generation tasks and data writing tasks; the parsing result includes: the original fields contained in the data table to be generated, and the arrangement order of the original fields in the data table to be generated; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the original field; According to the data generation task, a target data table is generated, which includes each original field and the data belonging to each original field; wherein the data belonging to each original field in any target data table is arranged according to the original arrangement order; According to the data writing task, the generated target data table is written into the designated database.

7. The method according to claim 6, characterized in that The step of generating a target data table including each original field and data belonging to each original field according to the data generation task includes: According to the data generation task, an initial data table including various original fields is generated; wherein, in the initial data table, each original field corresponds to a column, and the original fields are arranged according to the original arrangement order; the initial data table also includes a primary key field; Determine the original fields of the data to be generated according to the original arrangement order; Taking the specified value as the initial primary key value, sequentially generate a specified number of primary key values ​​belonging to the primary key field in a manner of increasing by 1 row by row; wherein the initial primary key values ​​corresponding to the working servers are different, and the primary key values ​​generated by the working servers do not have the same value; Generate a specified number of data belonging to the original field of the data to be generated currently as the data in the column corresponding to the original field of the data to be generated currently, and return to execute the step of determining the original field of the data to be generated currently according to the original arrangement order until the data in the column corresponding to each original field is generated; wherein, for each original field, the data of the last row in the column corresponding to the original field generated belongs to the same row as the generated maximum primary key value; The step of generating a specified number of data belonging to the original field of the data to be generated as data in a column corresponding to the original field of the data to be generated includes: For the original field of the data to be generated, if the original field does not belong to the field to be processed, a specified number of empty data are used as the data in the column corresponding to the original field; wherein the field to be processed is obtained by the driving server obtaining the SQL statement to be tested for processing the data table in the specified database, parsing the SQL statement to be tested, and determining the field to be processed by the SQL statement to be tested; If the original field belongs to the field to be processed, and the original data type of the original field is a numerical type, a specified number of data are generated according to the first function defined in the data generation subtask corresponding to the original field as the data in the column corresponding to the original field; wherein the first function is used to instruct each working server to generate data belonging to the original field within a preset numerical range; If the original field belongs to the field to be processed, and the original data type of the original field is a string type, a specified number of data are generated according to the second function defined in the data generation subtask corresponding to the original field as the data in the column corresponding to the original field; the second function is used to instruct each working server to select a string from a preset string set to generate data belonging to the original field; If the original field belongs to the field to be processed and the original data type of the original field is a date type, then a specified number of data are generated according to the third function defined in the data generation subtask corresponding to the original field as the data in the column corresponding to the original field; the third function is used to instruct each working server to select a date in the specified date range to generate data belonging to the original field.

8. A SQL statement testing system, characterized in that: Contains multiple worker servers and driver servers in a Spark cluster; The drive server is used to execute the method according to any one of claims 1 to 5; Any working server among the multiple working servers is used to execute the method according to claim 6 or 7.

9. A SQL statement testing device, characterized in that: The device is applied to a driving server in a Spark cluster, and the Spark cluster also includes a plurality of working servers. The device includes: A first acquisition module is used to acquire a data definition language DDL statement representing a data structure of a data table to be generated; wherein the data structure of the data table to be generated is the same as the data structure of a data table in a specified database; A first parsing module is used to parse the DDL statement to obtain a parsing result; wherein the parsing result includes: each original field contained in the data table to be generated, and the arrangement order of each original field in the data table to be generated; A construction module is used to use the directed acyclic graph scheduler component in the driving server to construct a task sequence based on the parsing result; wherein the task sequence includes a data generation task and a data writing task; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the column corresponding to the original field; A sending module is used to use the task scheduler component in the driving server to send the task sequence to each working server respectively, so that each working server generates a target data table containing each original field and the data belonging to each original field according to the data generation task, and writes the generated target data table into the designated database according to the data writing task; wherein the data belonging to each original field in any target data table is arranged according to the original arrangement order.

10. A SQL statement testing device, characterized in that: The device is applied to each working server of a plurality of working servers included in a test system, the test system further includes a driving server, and the device includes: A receiving module, used for receiving a task sequence sent by the driving server through an execution process in each working server; wherein the task sequence is: the driving server parses the acquired data definition language DDL statement representing the data structure of the data table to be generated, constructs it according to the parsing result using the directed acyclic graph scheduler component in the driving server, and issues it to the multiple working servers respectively using the task scheduler component in the driving server; the data structure of the data table to be generated is the same as the data structure of the data table in the specified database; the task sequence includes a data generation task and a data writing task; the parsing result includes: each original field contained in the data table to be generated, and the arrangement order of each original field in the data table to be generated; the data generation task represents the data generation subtask corresponding to each original field and the original arrangement order of each original field in the data table to be generated; the data generation subtask corresponding to each original field represents the generation of data belonging to the original field; A generating module, used for generating a target data table including each original field and data belonging to each original field according to the data generating task; wherein the data belonging to each original field in any target data table is arranged according to the original arrangement order; A writing module is used to write the generated target data table into the specified database according to the data writing task.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images