SQL (Structured Query Language) task processing method and device and electronic equipment

Abstract

The invention relates to the field of data query processing, in particular to an SQL task processing method and device and electronic device.The processing method comprises the steps that an SQL task is obtained; obtaining corresponding operator information and operator indexes according to the SQL task; the operator information at least comprises one operator operation; respectively predicting the cost of each calculation engine in a preset calculation engine library according to the operator information and the operator indexes; determining a target engine of the SQL task according to the cost of each calculation engine; and executing the SQL task according to the target engine. Compared with the prior art, by calculating the cost of each calculation engine in the calculation engine library, selecting the optimal calculation engine to execute the SQL task and intelligently selecting the better calculation engine to execute according to the actual SQL task, the execution efficiency of the query task can be effectively improved, invalid consumption of a CPU (Central Processing Unit) and a memory can be reduced, and the execution efficiency of the query task is improved. The calculation cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data query processing, and more particularly, to a method, apparatus, and electronic device for processing SQL tasks. Background Art

[0002] With the explosive growth of data volume, Spark, as a distributed computing framework, has been widely used in the processing of large-scale data SQL (Structured Query Language). However, in the prior art, the Spark framework defaults to using a single computing engine, making it difficult to fully utilize the performance advantages of the CPU (Central Processing Unit), especially when dealing with complex operators, the efficiency is relatively low. Summary of the Invention

[0003] The present invention aims to overcome at least one defect of the above prior art, and provides a method, apparatus, and electronic device for processing SQL tasks, which can effectively utilize the CPU performance and improve the query efficiency.

[0004] According to the first aspect of the present application, a method for processing SQL tasks is provided, and the processing method includes:

[0005] Obtain an SQL task;

[0006] Obtain the corresponding operator information and operator metrics according to the SQL task; the operator information includes at least one operator operation;

[0007] According to the operator information and the operator metrics, respectively predict the costs of each computing engine in a preset computing engine library;

[0008] Determine the target engine of the SQL task according to the costs of each computing engine;

[0009] Execute the SQL task using the target engine.

[0010] Optionally, the step of respectively predicting the costs of each computing engine in a preset computing engine library according to the operator information and the operator metrics specifically includes:

[0011] Initialize the operation scores of each operator operation;

[0012] According to the operator metrics, respectively predict the operation scores of each computing engine for executing each operator operation in the operator information;

[0013] Respectively count the operation scores of all operator operations in the operator information executed by each computing engine to obtain the cost scores of each computing engine, and use the cost scores as the costs of the corresponding computing engines.

[0014] Optionally, the operator metrics at least include the number of input items and the number of output items;

[0015] According to the operator metrics, respectively predict the operation scores of each operator operation in the operator information executed by each computing engine, specifically:

[0016] According to the number of input items and the number of output items, call an operation calculation function to predict the operation scores of each operator operation in the operator information executed by each computing engine.

[0017] Optionally, the operation calculation function is specifically:

[0018] Calculate the ratio of the input number of input items to the number of output items;

[0019] If the ratio is greater than or equal to a first preset ratio, increase the operation score corresponding to the computing engine with the highest first change index by a preset standard value; the first change index is the computing performance of the computing engine at the ratio when the ratio of the number of input items to the number of output items is greater than or equal to the first preset ratio;

[0020] If the ratio is equal to a second preset ratio, increase the operation score corresponding to the computing engine with the highest second change index by a preset standard value; the second change index is the computing performance of the computing engine at the ratio when the ratio of the number of input items to the number of output items is equal to the second preset ratio.

[0021] Optionally, the operator metrics further include the data table input size;

[0022] The method for respectively predicting the costs of each computing engine in a preset computing engine library according to the operator information and the operator metrics further includes:

[0023] Statistically calculate the data table input sizes corresponding to each operator operation to obtain a total data value;

[0024] Use the total data value and the cost score of the computing engine as the cost corresponding to the computing engine.

[0025] Optionally, the method for determining the target engine of the SQL task according to the costs of each computing engine specifically includes:

[0026] Judge the magnitudes of the cost scores in the costs of each computing engine, and use the computing engine with the largest cost score as the target engine.

[0027] Optionally, the method for determining the target engine of the SQL task according to the costs of each computing engine specifically includes:

[0028] If the total data value is less than or equal to a preset data threshold, obtain the computing engine with the lowest first change index and the highest second change index as the target engine;

[0029] If the total data value is greater than the preset data threshold, judge the cost scores in the costs of each computing engine, and take the computing engine with the highest cost score as the target engine.

[0030] Optionally, the operator operations in the operator information include one or a combination of sorting operations, aggregation operations, broadcast join operations, and sorting join operations; the step of calling the operation calculation function to predict the operation scores of each computing engine for executing each operator operation in the operator information according to the number of input rows and the number of output rows specifically includes:

[0031] If the operator operation is a sorting operation, directly call the operation calculation function;

[0032] If the operator operation is an aggregation operation, directly call the operation calculation function;

[0033] If the operator operation is a broadcast join operation, obtain the computing engine with the highest second change index, and increase the operation score corresponding to the broadcast join operation of the obtained computing engine by a preset standard value;

[0034] If the operator operation is a sorting join operation, directly call the operation calculation function.

[0035] According to a second aspect of the present application, there is provided an SQL task processing device, and the processing device includes:

[0036] A task acquisition module, configured to acquire an SQL task;

[0037] An operator acquisition module, configured to acquire corresponding operator information and operator metrics according to the SQL task; the operator information includes at least one operator operation;

[0038] A cost calculation module, configured to respectively predict the costs of each computing engine in a preset computing engine library according to the operator information and the operator metrics;

[0039] An engine determination module, configured to determine the target engine of the SQL task according to the costs of each computing engine;

[0040] A task execution module, configured to execute the SQL task by using the target engine.

[0041] According to a third aspect of the present application, there is provided an electronic device, including a memory and a processor. A computer-readable instruction is stored on the memory, and the processor executes the computer-readable instruction to implement a SQL task processing method described in the first aspect above.

[0042] According to a fourth aspect of the present application, there is provided a computer storage medium, on which a computer-readable program is stored. When the computer-readable program is executed, it implements a SQL task processing method described in the first aspect above.

[0043] Based on any one of the above aspects provided, a SQL task processing method, apparatus and electronic device provided by the present application, after obtaining a SQL task, obtain corresponding operator information and operator metrics according to the SQL task, and then calculate the cost of each computing engine pre-stored in the computing engine library according to the obtained operator information and operator metrics, and then can select the optimal computing engine to execute the SQL task according to the calculated cost, and intelligently select a better computing engine to execute according to the actual SQL task, which can effectively improve the execution efficiency of the query task, and then can reduce the ineffective consumption of the CPU and memory and reduce the computing cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 It is a schematic application scenario diagram of the processing method provided in this embodiment.

[0046] Figure 2 It is a flowchart of the steps of the processing method provided in this embodiment.

[0047] Figure 3 It is the step flow of cost calculation provided in this embodiment Figure 1 .

[0048] Figure 4 It is the step flow of cost calculation provided in this embodiment Figure 2 .

[0049] Figure 5 It is a flowchart of the steps of the operation calculation function provided in this embodiment.

[0050] Figure 6 It is the step flow of target engine determination provided in this embodiment Figure 1 .

[0051] Figure 7 The step flow determined for the target engine provided in this embodiment Figure 2 。

[0052] Figure 8 The schematic diagram of the functional modules of the processing device provided in this embodiment.

[0053] Figure 9 The schematic diagram of the structure of the electronic device provided in this embodiment. Detailed implementation manners

[0054] The accompanying drawings of this application are only for illustrative purposes and should not be construed as a limitation to this application. To better illustrate the following embodiments, some components in the drawings may be omitted, enlarged, or reduced, which do not represent the dimensions of actual products; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0055] In order to enable those skilled in the art to better understand the solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0056] It should be noted that the terms "first", "second", etc. in the specification, claims and above-mentioned drawings of this application are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0057] With the explosive growth of data volume, Spark, as a distributed computing framework, is widely used in large-scale data SQL processing. Spark is an open-source cluster computing environment that enables in-memory distributed datasets and can provide interactive queries. In the actual usage process, users communicate with the driver in Spark and query the required table data through a specified interface. Spark distributes the data to be queried to each worker node for calculation, and then aggregates the query results of each worker node through the driver and feeds them back to the customer. Therefore, Spark has very accurate query capabilities for large-scale data.

[0058] However, in the existing technical solutions, although it can ensure accurate query results for data, it cannot guarantee the query efficiency. The reason is that the Spark framework is usually implemented using a single computing engine. Since different computing engines have different performances for different operators and data processing, the single-computing-engine approach cannot make good use of the advantages of the CPU to accelerate the operation of the calculation. Especially in the case of processing complex operators, the computing timing is low.

[0059] In the prior art, although the Spark framework supports the extension of computing engines, due to the lack of an intelligent engine selection mechanism, it is impossible to dynamically adapt to the optimal engine according to the task characteristics, resulting in resource waste and performance bottlenecks.

[0060] This embodiment provides a technical solution that can solve the above problems. The following will describe the specific implementation manners of the present application in detail with reference to the accompanying drawings.

[0061] Exemplarily, it is a schematic diagram of an application scenario of a SQL task processing method provided by an embodiment of the present application. As Figure 1 shown, the application scenario at least includes a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has functions such as SQL task parsing, operator information and metric acquisition, and cost calculation; the terminal 200 has functions such as SQL task parsing, operator information and operator metric acquisition, and cost calculation, or the terminal 200 can receive SQL tasks, operator information and operator metrics, cost and other information sent by the server 100.

[0062] It can be understood that the server 100 can be an independent electronic device or a cluster composed of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a vehicle-mounted terminal, etc., but is not limited thereto.

[0063] In an implementable manner, the server 100 and the terminal 200 can respectively execute the SQL task processing method provided by the embodiments of the present application. Alternatively, optionally, part of the SQL task processing method provided by the embodiments of the present application is executed in the server 100, and part is executed in the terminal 200.

[0064] As Figure 2 shown, this embodiment provides a SQL task processing method. The SQL task processing method of this embodiment can be applied to the Spark framework. The following describes the specific implementation manner of this embodiment based on the Spark framework structure.

[0065] In this embodiment, the processing method may include the following steps:

[0066] S1: Obtain a SQL task;

[0067] Specifically, in this step, a SQL task request from a user can be received through the user interface of the Spark framework, or a SQL task of the user can be obtained from a third-party application through the interface of the Spark framework; the obtained SQL task is sent to the Spark framework for analysis and processing.

[0068] S2: Obtain corresponding operator information and operator metrics according to the SQL task;

[0069] In this step, by parsing the SQL task of the user, the operator information of the SQL task is extracted. The operator information includes the operator operations involved in the SQL task. The operator operations mainly include sorting operations, broadcast join operations, sort join operations, aggregation operations, and filtering operations, etc.; it can be understood that at least one operator operation is included in the operator information, that is, the operator information can be any one or a combination of multiple of the sorting operation, broadcast join operation, sort join operation, aggregation operation, and filtering operation.

[0070] It can be understood that by processing the large-scale data through the specific operator operations in the operator information of the SQL task, the data required by the user can be output. Among them, the large-scale data is the database or data table where the data required by the user is located, and this database or data table will be continuously updated.

[0071] The operator metrics can represent the relevant data parameters recorded by the computer recently when processing the large-scale data through the operator operations. Specifically, the operator metrics mainly can include the number of input records, the number of output records, and the input size of the data table. In addition, the operator metrics can also include the memory usage situation, the number of disk read and write operations, etc.

[0072] Among them, the number of input items represents the quantity of data input to the operator operation for execution under the corresponding operator operation, and the number of output items represents the quantity of data output after the execution of the corresponding operator operation; it can be understood that for the case where the operator information contains a single operator operation, the number of input items usually equals the data volume of the database or data table; for the case where the operator information contains multiple operator operations, the number of input items and output items corresponding to each operator operation will change with the execution order of the operator operations.

[0073] In this embodiment, each operator operation and the corresponding operator metrics involved in the SQL task can be saved to an operator runtime library. Preferably, the operator runtime library can be set as a readable file of a Spark framework. Since the Spark framework is implemented based on the Java system, the readable file can include, but is not limited to, json files, txt files, csv files, etc.

[0074] It can be understood that in order to ensure the accuracy of the obtained operator metrics, the operator runtime library will be continuously updated, that is, in the operator runtime library, the operator metrics corresponding to each operator operation will also be continuously updated accordingly.

[0075] S3: According to the operator information and the operator metrics, predict the costs of each computing engine in the preset computing engine library respectively;

[0076] In this embodiment, the cost of each computing engine represents the performance of each computing engine in executing all operator operations in the operator information under the operator metrics; and as described above, in this embodiment, the obtained operator metrics can be the data parameters corresponding to the operator operation recorded most recently in the operator runtime library. That is, in this embodiment, in fact, the operator metrics corresponding to the most recent operator operation are used to predict the computing cost required for the current SQL task, and then the optimal computing engine is selected according to this computing cost, thereby effectively improving the execution efficiency of the SQL task.

[0077] Specifically, as Figure 3 shown, the calculation of the cost of each computing engine can specifically include:

[0078] S31: Initialize the operation scores of each operator operation;

[0079] S32: According to the operator metrics, predict the operation scores of each computing engine in executing each operator operation in the operator information respectively;

[0080] S33: Respectively count the operation scores of all operator operations in the operator information executed by each of the computing engines to obtain the cost scores of each of the computing engines, and use the cost scores as the costs corresponding to the computing engines.

[0081] It is understandable that different computing engines have different performances for different operator operations. At the same time, different computing engines also have different computing performances under different operator metrics. Therefore, in order to effectively obtain the performance of each computing engine for the SQL task, in this embodiment, it is necessary to calculate the operation scores of each computing engine corresponding to each operator operation in the operator information under the operator metrics, and then respectively count the operation scores of each computing engine to obtain the cost scores of each computing engine for the SQL task, which are used as the costs of the computing engines. In the actual application process, the computing performance of the computing engine is related to the change of the input data volume and the output data volume, that is, related to the change of the input number of rows and the output number of rows. Some computing engines may have better computing performance for the case where the input number of rows and the output number of rows change greatly, while some other computing engines may have better computing performance for the case where the input number of rows and the output number of rows change less. Therefore, in this embodiment, the calculation of the operation score can be specifically as follows:

[0082] According to the input number of rows and the output number of rows, call the operation calculation function to predict the operation scores of each computing engine for each operator operation in the operator information.

[0083] Among them, as Figure 5 shown, the operation calculation function is specifically:

[0084] B1: Calculate the ratio of the input number of rows and the output number of rows.

[0085] B11: If the ratio is greater than or equal to the first preset ratio, increase the operation score corresponding to the computing engine with the highest first change index by a preset standard value.

[0086] B12: If the ratio is equal to the second preset ratio, increase the operation score corresponding to the computing engine with the highest second change index by a preset standard value.

[0087] As described above, the computing performance of different computing engines is related to the change of the number of input records and the number of output records, and the ratio of the number of input records to the number of output records reflects the change of the number of input records and the number of output records, which can be used to judge the corresponding computing performance of each computing engine, and further as a reference for determining the computing engine. Therefore, in this embodiment, by setting the first preset ratio and the second preset ratio, as the computing index of the computing engine for the change of the number of input records and the number of output records; wherein, the first preset ratio is greater than the second preset ratio.

[0088] Specifically, the first change index is the computing performance of the computing engine when the ratio of the number of input records to the number of output records is greater than or equal to the first preset ratio; preferably, the first preset ratio can be set to 1.5-1.6; it can be understood that when the ratio of the number of input records to the number of output records is greater than or equal to the first preset ratio, it means that the difference or change between the number of input records and the number of output records is relatively large at this time. Therefore, the higher the first change index, the better the computing performance of the computing engine when the change between the number of input records and the number of output records is large.

[0089] Correspondingly, the second change index is the computing performance of the computing engine when the ratio of the number of input records to the number of output records is equal to the second preset ratio; it can be understood that the second preset ratio is usually set to 1 or a value close to 1, that is, when the ratio of the number of input records to the number of output records is equal to the second preset ratio, it means that the difference or change between the number of input records and the number of output records is relatively small at this time. Therefore, the higher the second change index, the better the computing performance of the computing engine when the change between the number of input records and the number of output records is small.

[0090] In one example, for the convenience of description, the computing engine library includes the Gluten computing engine and the Blaze computing engine. The Gluten computing engine and the Blaze computing engine are implemented based on the Spark engine extension. Among them, the Gluten computing engine has better computing performance when the change between the number of input records and the number of output records is small, and the Blaze computing engine has better computing performance when the change between the number of input records and the number of output records is large. Therefore, the second change index of the Gluten computing engine is higher than that of the Blaze computing engine, and the first change index of the Gluten computing engine is lower than that of the Blaze computing engine.

[0091] S4: Determine the target engine of the SQL task according to the cost of each computing engine;

[0092] In an implementation manner of this embodiment, when only the number of input items and the number of output items are included in the operator metrics and other operator metrics that affect the computing performance of the computing engine are not included, as Figure 6 shown, this step may specifically be:

[0093] S411: Determine the cost scores in the costs of each computing engine, and use the computing engine with the largest cost score as the target engine.

[0094] In the actual application process, the computing performance of the computing engine is also affected by other operator metrics. For example, as described above, the input size of the data table. Therefore, in another implementation manner of this embodiment, the operator metrics include the number of input items, the number of output items, and the input size of the data table. At this time, as Figure 4 shown, the calculation of the costs of each computing engine described above may further include:

[0095] S34: Count the input sizes of the data tables corresponding to each operator operation to obtain a total data value;

[0096] S35: Use the total data value and the cost score of the computing engine as the cost corresponding to the computing engine.

[0097] At the same time, since in this implementation manner, in addition to the cost score size, the cost also includes the total data value. Therefore, in this implementation manner, as Figure 7 shown, step S4 may specifically include the following steps:

[0098] S421: If the total data value is less than or equal to a preset data threshold, obtain the computing engine with the lowest first change index and the highest second change index as the target engine;

[0099] S422: If the total data value is greater than the preset data threshold, determine the cost scores in the costs of each computing engine, and use the computing engine with the highest cost score as the target engine.

[0100] Specifically, the higher the second change index of the computing engine, the higher the computing performance for small-scale data, and the higher the first change index of the computing engine, the higher the computing performance for large-scale data. Therefore, by counting the input sizes of the data tables corresponding to each operator operation in the operator information, the total data value of the input of the SQL task is obtained, and then the optimal computing engine can be selected according to the total data value.

[0101] It is understandable that although the operator operations may include sorting operations, broadcast join operations, sort-merge join operations, aggregation operations, and filtering operations, etc., in fact, the operator operations with relatively large overhead are sorting operations, broadcast join operations, sort-merge join operations, and aggregation operations. Moreover, for different said operator operations, the corresponding operator metrics are different. Therefore, in this embodiment, the step of calling the operation calculation function to predict the operation scores of each operator operation in the operator information executed by each computing engine according to the number of input records and the number of output records may specifically include:

[0102] If the operator operation is a sorting operation, directly call the operation calculation function;

[0103] If the operator operation is an aggregation operation, directly call the operation calculation function;

[0104] If the operator operation is a broadcast join operation, obtain the computing engine with the highest second change metric, and increase the operation score corresponding to the broadcast join operation of the obtained computing engine by a preset standard value;

[0105] If the operator operation is a sort-merge join operation, directly call the operation calculation function.

[0106] In a specific example, assume that the operator information includes the sorting operation, aggregation operation, broadcast join operation, and sort-merge join operation; the first preset ratio is set to 1.5, the second preset ratio is set to 1, the preset data threshold is 4GB, and the preset standard value is set to 1; the computing engine library includes the Gluten computing engine and the Blaze computing engine.

[0107] For the sorting operation, the corresponding operator metrics are: the number of input records is 100, the number of output records is 100, and the input size of the data table is 1GB; for the sorting operation, directly call the operation calculation function. The ratio of the number of input records to the number of output records is 100 / 100 = 1, which is equal to the second preset ratio. Therefore, increase the operation score of the Gluten computing engine corresponding to the sorting operation by 1;

[0108] For the aggregation operation, the corresponding operator metrics are: the number of input records is 1000, the number of output records is 100, and the input size of the data table is 2GB; for the aggregation operation, directly call the operation calculation function. The ratio of the number of input records to the number of output records is 1000 / 100 = 10, which is greater than the first preset ratio. Therefore, increase the operation score of the Blaze computing engine corresponding to the aggregation operation by 1;

[0109] For the broadcast join operation, the corresponding operator metrics are: the number of input rows is 500, the number of output rows is 500, and the input size of the data table is 1.5 GB; for the broadcast join operation, increment the operation score of the Gluten computing engine corresponding to the broadcast join operation by 1;

[0110] For the sort-merge join operation, the corresponding operator metrics are: the number of input rows is 500, the number of output rows is 500, and the input size of the data table is 1 GB; for the sort-merge join operation, directly call the operation calculation function, and the ratio of the number of input rows to the number of output rows is 1, which is equal to the second preset ratio. Therefore, increment the operation score of the Gluten computing engine corresponding to the sort-merge join operation by 1;

[0111] Statistically calculate each of the operation scores of the Gluten computing engine and the Blaze computing engine as the cost scores of the Gluten computing engine and the Blaze computing engine. After statistics, the cost score of the Gluten computing engine is 3, and the cost score of the Blaze computing engine is 1;

[0112] Statistically calculate the input size of the data table for each operator operation in the above operator information, and obtain the total data value of 1 + 2 + 1.5 + 1 = 5.5, which is greater than the preset data threshold of 4 GB. Therefore, use the Gluten computing engine with the highest cost score as the target engine.

[0113] It should be noted that the specific values of the above number of input rows, number of output rows, and input size of the data table are only for reference as examples and may not conform to the actual usage situation.

[0114] S5: Execute the SQL task using the target engine.

[0115] Therefore, in this embodiment, by setting up the computing engine library, the expansion of the computing engine can be flexibly achieved, and thus more task scenarios can be adapted, effectively improving the execution efficiency of SQL tasks.

[0116] As Figure 8 shown, the embodiment of the present application also provides an SQL task processing device, and the device includes:

[0117] A task acquisition module 11, configured to acquire an SQL task;

[0118] In this embodiment, the task acquisition module 11 can be used to execute Figure 2 the step S1 shown. For the specific description of the task acquisition module 11, reference can be made to the description of the step S1.

[0119] An operator acquisition module 12, configured to obtain corresponding operator information and operator metrics according to the SQL task; the operator information includes at least one operator operation;

[0120] In this embodiment, the operator acquisition module 12 can be used to execute Figure 2 Step S2 shown. For the specific description of the operator acquisition module 12, reference can be made to the description of step S2.

[0121] A cost calculation module 13, configured to predict the costs of respective computing engines in a preset computing engine library according to the operator information and the operator metrics;

[0122] In this embodiment, the cost calculation module 13 can be used to execute Figure 2 Step S3 shown. For the specific description of the cost calculation module 13, reference can be made to the description of step S3.

[0123] An engine determination module 14, configured to determine a target engine for the SQL task according to the costs of the respective computing engines;

[0124] In this embodiment, the engine determination module 14 can be used to execute Figure 2 Step S4 shown. For the specific description of the engine determination module 14, reference can be made to the description of step S4.

[0125] A task execution module 15, configured to execute the SQL task by using the target engine;

[0126] In this embodiment, the task execution module 15 can be used to execute Figure 2 Step S5 shown. For the specific description of the task execution module 15, reference can be made to the description of step S5.

[0127] An embodiment of the present application provides an electronic device, the structure of which is as Figure 9 shown. The electronic device can be the server 100 or the terminal 200 shown in this embodiment Figure 1 shown.

[0128] As Figure 9 shown, the electronic device includes a memory 21, a processor 22, a communication module 23, an input / output interface 24, etc. Optionally, the memory 21, the processor 22, the communication module 23, and the input / output interface 24 can be connected and communicate through a bus 25.

[0129] The memory 21 is used to store one or more computer programs and transmit the code of the computer programs to the processor 22; when the one or more computer programs are executed by the processor 22, the SQL task processing method in the embodiments of the present application is implemented.

[0130] Optionally, the electronic device can be connected to a network through the communication module 23 to communicate with other devices, such as terminals or servers, through the network to achieve data interaction. The electronic device can be various forms of digital computers, exemplarily, such as desktop computers, servers, workbenches, mainframe computers or other types of computers. The electronic device can also be various forms of mobile terminals, exemplarily, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0131] Optionally, the electronic device can be connected to the required input / output devices, such as keyboards, display devices, etc., through the input / output interface 24. The electronic device itself can have a display device and can also externally connect other display devices through the input / output interface 24. Optionally, a storage device, such as a hard disk, etc., can also be connected through the input / output interface 24, so as to store the data in the electronic device into the storage device, or read the data in the storage device, and can also store the data in the storage device into the memory 21. It can be understood that the input / output interface 24 can be a wired interface or a wireless interface. According to different actual application scenarios, the devices connected to the input / output interface 24 can be components of the electronic device or external devices connected to the electronic device when needed.

[0132] Optionally, the memory 21 can be a volatile memory and / or a non-volatile memory. The volatile memory can be a random access memory, etc., and the non-volatile memory can be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0133] Optionally, the computer program stored in the processor 22 can be divided into one or more modules. The one or more modules are stored in the memory 21 and executed by the processor 22 to complete the method provided in this embodiment. The one or more modules can be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device.

[0134] Optionally, the processor 22 may be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 22 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special artificial intelligence computing chips, various processors running machine learning model algorithms, and may also be any suitable controller, microcontroller, processor, etc. The processor 22 executes the various methods and processes of this embodiment. Exemplarily, it is, for example, a SQL task processing method of an embodiment of this application.

[0135] Optionally, the bus 25 may include a path for transmitting information. The bus 25 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 25 may be divided into an address bus, a data bus, a control bus, etc.

[0136] In an alternative implementation, an embodiment of this application also provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to execute the methods of the above method embodiments. Part or all of the computer program may be loaded and / or installed on the memory 21 of the electronic device. When the computer program is executed by the processor 22, one or more steps of a SQL task processing method of an embodiment of this application may be executed.

[0137] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.

[0138] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, rather than limitations on the specific implementation manners of the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the claims of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. A method for processing SQL tasks, characterized in that, The processing method includes: Obtain an SQL task; Obtain corresponding operator information and operator metrics according to the SQL task; the operator information includes at least one operator operation; Predict the costs of each computing engine in a preset computing engine library respectively according to the operator information and the operator metrics; Determine the target engine of the SQL task according to the costs of each computing engine; Execute the SQL task using the target engine.

2. The SQL task processing method according to claim 1, wherein The step of predicting the costs of each computing engine in a preset computing engine library respectively according to the operator information and the operator metrics specifically includes: Initialize the operation scores of each operator operation; Predict the operation scores of each operator operation in the operator information executed by each computing engine respectively according to the operator metrics; Statistically calculate the operation scores of all operator operations in the operator information executed by each computing engine respectively to obtain the cost scores of each computing engine, and use the cost scores as the costs of the corresponding computing engines.

3. The SQL task processing method according to claim 2, wherein The operator metrics include at least the number of input rows and the number of output rows; The step of predicting the operation scores of each operator operation in the operator information executed by each computing engine respectively according to the operator metrics is specifically: Call an operation calculation function to predict the operation scores of each operator operation in the operator information executed by each computing engine according to the number of input rows and the number of output rows.

4. The SQL task processing method according to claim 3, characterized in that, The operation calculation function is specifically: Calculate the ratio of the input number of input rows and the number of output rows; If the ratio is greater than or equal to a first preset ratio, increase the operation score corresponding to the computing engine with the highest first change index by a preset standard value; the first change index is the computing performance of the computing engine at the ratio when the ratio of the number of input rows and the number of output rows is greater than or equal to the first preset ratio; If the ratio is equal to a second preset ratio, increase the operation score corresponding to the computing engine with the highest second change index by a preset standard value; The second change index is the computing performance of the computing engine at the ratio when the ratio of the number of input rows and the number of output rows is equal to the second preset ratio.

5. The SQL task processing method according to claim 3, wherein The operator metrics further include the input size of the data table; The step of predicting the costs of each computing engine in a preset computing engine library respectively according to the operator information and the operator metrics further includes: Statistically calculate the input sizes of the data tables corresponding to each operator operation to obtain a total data value; Use the total data value and the cost score of the computing engine as the cost of the corresponding computing engine.

6. A SQL task processing method according to any one of claims 2-4, characterized in that, The step of determining the target engine of the SQL task according to the costs of each computing engine specifically includes: Judge the magnitudes of the cost scores in the costs of each computing engine, and use the computing engine with the largest cost score as the target engine.

7. A SQL task processing method according to claim 5, characterized in that, The step of determining the target engine of the SQL task according to the costs of each computing engine specifically includes: If the total data value is less than or equal to a preset data threshold, obtain the computing engine with the lowest first change index and the highest second change index as the target engine; If the total value of the data is greater than a preset data threshold, judge the cost scores in the costs of each of the computing engines, and use the computing engine with the highest cost score as the target engine.

8. A SQL task processing method according to any one of claims 3, 4, 5 or 7, characterized in that The operator operations in the operator information include one or a combination of sorting operations, aggregation operations, broadcast join operations, and sorting join operations; the step of calling the operation calculation function to predict the operation scores of each computing engine for executing each operator operation in the operator information according to the number of input items and the number of output items specifically includes: If the operator operation is a sorting operation, directly call the operation calculation function; If the operator operation is an aggregation operation, directly call the operation calculation function; If the operator operation is a broadcast join operation, obtain the computing engine with the highest second change index, and increase the operation score corresponding to the broadcast join operation of the obtained computing engine by a preset standard value; If the operator operation is a sorting join operation, directly call the operation calculation function.

9. An SQL task processing device, characterized in that, The processing device includes: A task acquisition module, configured to acquire an SQL task; An operator acquisition module, configured to acquire corresponding operator information and operator metrics according to the SQL task; the operator information includes at least one operator operation; A cost calculation module, configured to respectively predict the costs of each computing engine in a preset computing engine library according to the operator information and the operator metrics; An engine determination module, configured to determine the target engine of the SQL task according to the costs of each computing engine; A task execution module, configured to execute the SQL task by using the target engine.

10. An electronic device, comprising a memory and a processor, characterized in that Computer-readable instructions are stored on the memory, and the processor executes the computer-readable instructions to implement the method for processing an SQL task according to any one of claims 1-8 above.

Images