Training method of computing engine selection model, computing engine selection method and device

Abstract

The application discloses a training method and device of a computing engine selection model and a computing engine selection method, and belongs to the technical field of big data. The method comprises the following steps: obtaining a structured query language (SQL) training set, and text features, vectorized data and data features of data tables in each SQL statement in the SQL training set; generating an engine label of each SQL statement, wherein the engine label is used for indicating a computing engine with the shortest execution time corresponding to each SQL statement; taking the text features, the vectorized data and the data features of the data tables in each SQL statement as training data of a computing engine selection model, taking the engine label corresponding to each SQL statement as a label value, and training the computing engine selection model; wherein the trained computing engine selection model is used for outputting a computing engine with the shortest execution time of a target SQL statement according to the input target SQL statement.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of big data, and particularly relates to a training method of a computing engine selection model, a computing engine selection method and device. BACKGROUND

[0002] Structured Query Language (SQL) is a query and programming language of a database, which is convenient for accessing data and querying, updating and managing a relational database system.

[0003] When a technical person uses a big data platform to perform SQL interactive query, the execution efficiency of different computing engines is different. Specifically, which computing engine to select often needs to be determined by an experienced engineer or manually tried, and there is no automatic method, which greatly increases the time cost and resource cost. SUMMARY

[0004] The purpose of the embodiments of the application is to provide a training method of a computing engine selection model, a computing engine selection method and device, which can solve the problem of high time cost and resource cost of the existing computing engine selection method.

[0005] In a first aspect, the embodiments of the application provide a training method of a computing engine selection model, and the method comprises the following steps.

[0006] Obtaining a structured query language (SQL) training set and text features, vectorized data and data features of data tables in each SQL statement in the SQL training set;

[0007] Generating an engine label of each SQL statement, the engine label being used to indicate a computing engine corresponding to each SQL statement and having the shortest execution time;

[0008] Taking the text features, vectorized data and data features of data tables in each SQL statement as training data of a computing engine selection model, and taking the engine label corresponding to each SQL statement as a label value, the computing engine selection model is trained;

[0009] The trained computing engine selection model is used to output a computing engine having the shortest execution time of a target SQL statement according to the input target SQL statement.

[0010] Optionally, the text features, vectorized data of each of the SQL statements, and data features of data tables in each of the SQL statements are taken as training data of a computing engine selection model, and engine labels corresponding to the SQL statements are taken as label values, and the computing engine selection model is trained, comprising:

[0011] The vectorized data of each of the SQL statements and the engine labels corresponding to the SQL statements are input into the computing engine selection model as label values, and a one-dimensional hidden layer vector representation is obtained.

[0012] The computing engine selection model is trained based on the one-dimensional hidden layer vector representation, the text features of each of the SQL statements, the data features of data tables in each of the SQL statements, and the engine labels corresponding to each of the SQL statements.

[0013] Optionally, the SQL training set is obtained, and the text features, vectorized data of each of the SQL statements in the SQL training set, and the data features of data tables in each of the SQL statements are obtained, comprising:

[0014] Text analysis is performed on each of the SQL statements in the SQL training set to determine the text features of each of the SQL statements, and the text features include the number of data tables in the SQL statements and information for representing whether an aggregation function exists in the SQL statements.

[0015] Based on natural language processing (NLP) technology, text vectorization is performed on each of the SQL statements in the SQL training set to obtain vectorized data of each of the SQL statements.

[0016] Based on the text features of the SQL statements, table names of data tables involved in the SQL statements are determined.

[0017] Based on the table names of the data tables, historical data of the data tables is obtained from a statistical table, and a maximum value of the historical data is taken as the data features of the data tables in the SQL statements.

[0018] Optionally, the computing engine selection model is trained based on the one-dimensional hidden layer vector representation, the text features of each of the SQL statements, the data features of data tables in each of the SQL statements, and the engine labels corresponding to each of the SQL statements, comprising:

[0019] The one-dimensional hidden layer vector representation, the text features of each SQL statement, and the data features of the data table in each SQL statement are taken as training data, and the engine label corresponding to each SQL statement is taken as a label value, and a distributed gradient boosting library XGBoost or a gradient boosting decision tree GBDT or a light gradient boosting machine LightGBM is used for training.

[0020] In a second aspect, the embodiments of the present application provide a method for selecting a computing engine, the method comprising:

[0021] obtaining a computing engine selection request, and obtaining a SQL statement in the computing engine selection request;

[0022] inputting the SQL statement into a computing engine selection model, and obtaining first data output by the computing engine selection model, the first data being used to indicate a computing engine with the shortest execution time for the SQL statement;

[0023] The computing engine selection model is a model trained based on the method of the first aspect.

[0024] Optionally, the inputting of the SQL statement into the computing engine selection model comprises:

[0025] obtaining text features of the SQL statement, vectorized data, and data features of a data table in the SQL statement;

[0026] inputting the vectorized data of the SQL statement into the computing engine selection model, and obtaining a one-dimensional hidden layer vector representation;

[0027] inputting the one-dimensional hidden layer vector representation, the text features of the SQL statement, and the data features of the data table in the SQL statement into the computing engine selection model.

[0028] Optionally, the obtaining of the data features of the data table in the SQL statement comprises:

[0029] determining a table name of the data table in the SQL statement based on the text features;

[0030] determining historical data of the data table based on the table name, wherein the historical data comprises a Hadoop distributed file system (HDFS) file size;

[0031] determining current data of the data table by using a time series prediction method on the historical data, the current data being the data features of the data table.

[0032] In a third aspect, the embodiments of the present application provide a training device of a computing engine selection model, the device comprising:

[0033] The first obtaining module is configured to obtain a structured query language (SQL) training set and text features, vectorized data and data features of data tables in each SQL statement in the SQL training set.

[0034] The generating module is configured to generate an engine label of each SQL statement, the engine label being used to indicate a respective calculation engine with the shortest execution time of each SQL statement.

[0035] The training module is configured to train a calculation engine selection model by taking the text features, vectorized data and data features of data tables in each SQL statement as training data and taking the engine label corresponding to each SQL statement as a label value, and train the calculation engine selection model.

[0036] The calculation engine selection model after being trained is used to output a calculation engine with the shortest execution time of a target SQL statement according to the target SQL statement.

[0037] In a fourth aspect, an embodiment of the present application provides a calculation engine selection device, and the device comprises:

[0038] The second obtaining module is configured to obtain a calculation engine selection request and obtain an SQL statement in the calculation engine selection request.

[0039] The selecting module is configured to input the SQL statement into a calculation engine selection model and obtain first data output by the calculation engine selection model, the first data being used to indicate a calculation engine with the shortest execution time of the SQL statement.

[0040] The calculation engine selection model is a model trained based on the method of the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides an electronic device, and the electronic device comprises a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the training method of the calculation engine selection model of the first aspect or the steps of the calculation engine selection method of the second aspect.

[0042] In a sixth aspect, an embodiment of the present application provides a readable storage medium, and the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the training method of the calculation engine selection model of the first aspect or the steps of the calculation engine selection method of the second aspect.

[0043] In the embodiment of the present application, the text features of the SQL statement, the vectorized data and the data features of the data table in the SQL statement are taken as the training data of the computing engine selection model, the engine label corresponding to the SQL statement is taken as the label value, the computing engine selection model is trained, and as the training data and the corresponding label value increase continuously, the accuracy of the output result of the trained computing engine selection model will be continuously improved. After the training is completed, when a computing engine needs to be selected for a SQL statement, the SQL statement only needs to be input into the computing engine selection model, and the computing engine selection model can select the computing engine with the shortest execution time, without manual attempts, automation is realized, and the time cost and resource cost are reduced. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 One of the flowcharts of the training method of the computing engine selection model provided in the embodiment of the present application;

[0045] Figure 2 One of the flowcharts of the computing engine selection method provided in the embodiment of the present application;

[0046] Figure 3 The second flowchart of the training method of the computing engine selection model provided in the embodiment of the present application;

[0047] Figure 4 The second flowchart of the computing engine selection method provided in the embodiment of the present application;

[0048] Figure 5 The structural schematic diagram of the training selection device of the computing engine selection model provided in the embodiment of the present application;

[0049] Figure 6 The structural schematic diagram of the computing engine selection device provided in the embodiment of the present application;

[0050] Figure 7 The structural schematic diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0052] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0053] The methods provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0054] like Figure 1 As shown in the embodiments of this application, the training method for the computational engine selection model includes the following steps:

[0055] Step S11: Obtain the Structured Query Language (SQL) training set, as well as the text features, vectorized data, and data features of the data tables in each SQL statement within the training set.

[0056] The SQL training set includes multiple SQL statements. To ensure the accuracy of the output results after the computing engine selects the model for training, the number of SQL statements needs to be as large as possible, and the types of SQL statements should be diverse. The textual features of SQL statements can be the data table information relationships within the SQL statement, or information about the presence of aggregate functions within the SQL statement.

[0057] Based on Natural Language Processing (NLP) technology, the SQL statement is vectorized into text, resulting in vectorized SQL statement data. The data table containing the SQL statement is then retrieved, and the corresponding data table information is found from the statistical table to obtain the data characteristics of the data table.

[0058] Step S12: Generate an engine tag for each SQL statement. The engine tag indicates the computing engine with the shortest execution time for each SQL statement.

[0059] Execute each SQL statement in the SQL training set in different computing engines in turn (the resource configuration of each computing engine is balanced), record the execution time of the SQL statement in different computing engines, and obtain the engine label corresponding to the computing engine with the shortest SQL statement execution time. The computing engine can be a first-generation batch processing based on MapReduce Hive engine, a second-generation stream processing Spark engine, a third-generation batch and stream integrated data processing Flink engine, an interactive query Presto, a Massively Parallel Processing (MPP) architecture GreenPlum engine. The execution time of different SQL statements in different computing engines is inconsistent.

[0060] Step S13, training the computing engine selection model based on the one-dimensional hidden layer vector representation, the text features of each SQL statement, the data features of the data table in each SQL statement, and the engine label corresponding to each SQL statement.

[0061] The trained computing engine selection model is used to output the computing engine with the shortest execution time of the target SQL statement according to the input target SQL statement.

[0062] The text features, vectorized data and data features of the data table in the SQL statement obtained in the above steps are used as the training data of the computing engine selection model, and the engine label corresponding to the SQL statement is used as the label value. The computing engine selection model is trained. With the continuous increase of training data and corresponding label values, the accuracy of the output result of the trained computing engine selection model will continuously improve. After training is completed, when a SQL statement needs to be selected for a computing engine, the SQL statement only needs to be input into the computing engine selection model, and the computing engine selection model can select the computing engine with the shortest execution time, without manual trial, achieving automation and reducing time cost and resource cost.

[0063] Optionally, in step S13, the text features of each SQL statement, the vectorized data and the data features of the data table in each SQL statement are used as the training data of the computing engine selection model, and the engine label corresponding to each SQL statement is used as the label value. The computing engine selection model is trained, comprising:

[0064] The vectorized data of each SQL statement and the engine label corresponding to each SQL statement are input into the computing engine selection model as label values to obtain a one-dimensional hidden layer vector representation.

[0065] The computing engine selection model is trained based on the one-dimensional hidden layer vector representation, a text feature of each of the SQL statements, a data feature of a data table in each of the SQL statements, and an engine label corresponding to each of the SQL statements.

[0066] The computing engine selection model of the embodiment includes a first sub-model and a second sub-model. Vectorized data of a SQL statement is input into the first sub-model as training data, and an engine label corresponding to the SQL statement is input into the first sub-model as a label value. The first sub-model is trained, and each training improves the accuracy of the output result of the first sub-model. At the same time, the first sub-model generates a corresponding one-dimensional hidden layer vector representation. It should be noted that the one-dimensional hidden layer vector representation is not the output result of the first sub-model, but a parameter corresponding to the first sub-model. The one-dimensional hidden layer vector representation, a text feature of the SQL statement, and a data feature of a data table in the SQL statement are input into the second sub-model as training data, and the engine label corresponding to the SQL statement is input into the second sub-model as a label value. The second sub-model is trained to output a computing engine with the shortest execution time of the SQL statement according to the input one-dimensional hidden layer vector representation, the text feature of the SQL statement, and the data feature of the data table in the SQL statement. In the above steps, the one-dimensional hidden layer vector representation is obtained through the vectorized data of the SQL statement and the first sub-model. The one-dimensional hidden layer vector representation is combined with other related features of the text feature of the SQL statement and the data feature of the data table to train the second sub-model, and finally the computing engine selection model is obtained. Compared with only using the related features of the SQL statement, the accuracy of the output result of the trained computing engine selection model is improved.

[0067] Optionally, the vectorized data of each of the SQL statements is input into the first sub-model as training data, and the engine label corresponding to each of the SQL statements is input into the first sub-model as a label value. The first sub-model is trained using a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0068] Optionally, in step S11, the SQL training set, a text feature, vectorized data of each of the SQL statements in the SQL training set, and a data feature of a data table in each of the SQL statements are obtained, including:

[0069] The text feature of each of the SQL statements in the SQL training set is determined by performing text analysis on each of the SQL statements in the SQL training set. The text feature includes the number of data tables in the SQL statement and information indicating whether an aggregation function exists in the SQL statement.

[0070] Text analysis refers to the representation of the text and the selection of its features. The text features of the SQL statement include the number of data tables (whether there is one data table or multiple data tables), the name of the data table, information for representing whether there is an aggregation function in the SQL statement, and information for representing whether an aggregation function is used in the SQL statement.

[0071] Based on the natural language processing (NLP) technology, the text vectorization is performed on each SQL statement in the SQL training set to obtain vectorized data of each SQL statement.

[0072] Based on the text features of the SQL statement, the table name of the data table involved in the SQL statement is determined.

[0073] Based on the table name of the data table, the historical data of the data table is obtained from a statistical table, and the maximum value of the historical data is taken as the data feature of the data table in the SQL statement.

[0074] Optionally, the maximum value of the historical data is subjected to data standardization (min-max standardization), and the maximum value of the historical data after the data standardization is taken as the data feature of the data table in the SQL statement.

[0075] It should be noted that the data of the statistical table is obtained by querying the Hive table metadata, and this method has high efficiency in obtaining the record quantity of the statistical table. Preferably, the data of the statistical table can be obtained in a non-interactive query period such as the evening, and a part of the telecommunication data table can be extracted for real data statistics. Other types of data tables can be calculated according to the data features of the data table, such as the business type and the size of the underlying file (the statistical time is shorter). The above data features, text features, and vectorized data are used to train the computing engine selection model, which further ensures the accuracy of the output result of the computing engine selection model.

[0076] Optionally, the one-dimensional hidden layer vector representation, the text features of each SQL statement, the data features of the data table in each SQL statement, and the engine label corresponding to each SQL statement are used to train the computing engine selection model, including:

[0077] The one-dimensional hidden layer vector representation, the text features of each SQL statement, and the data features of the data table in each SQL statement are used as training data. The engine label corresponding to each SQL statement is used as the label value. The computing engine selection model is trained using a distributed gradient boosting library (eXtreme Gradient Boosting, XGBoost), a gradient boosting decision tree (GBDT), or a lightweight gradient boosting machine (LightGBM).

[0078] XGBoost is characterized by its high efficiency and flexibility. Furthermore, training the computational engine selection model with different types of algorithms will result in varying accuracy of the output results. Therefore, during the actual computational engine selection process, the model with the highest accuracy among the different algorithms can be determined based on the accuracy of the actual results.

[0079] like Figure 3 The diagram illustrates the training process of the computational engine selection model. It begins by obtaining the original SQL samples, i.e., the SQL training set. Using NLP techniques, the SQL statements are vectorized to obtain vectorized data. The SQL statements are then executed in different computational engines, and the engine with the shortest execution time and its corresponding engine label are obtained. Based on a text analysis program, textual and data features of the SQL statements are acquired. Textual features include the number of data tables in the SQL statement and information indicating the presence of aggregation functions. Data features include standardized data representing the historical maximum values ​​of the data tables in the SQL statement. The engine labels and vectorized data are then combined, and an RNN / CNN neural network is used to train Classifier 1 and its corresponding one-dimensional hidden layer vector representation. Classifier 1 is the first sub-model in the computational engine selection model. Finally, the engine labels, one-dimensional hidden layer vector representation, textual features, and data features are combined, and XGBoost is used for training to obtain Classifier 2, the second sub-model in the computational engine selection model.

[0080] like Figure 2 As shown in the embodiments of this application, a method for selecting a computing engine is also provided, including the following steps:

[0081] Step S21: Obtain the computing engine selection request, and obtain the SQL statement in the computing engine selection request.

[0082] The execution subject of the computing engine selection method acquires and responds to the computing engine selection request while acquiring the SQL statement contained in the computing engine selection request.

[0083] In step S22, the SQL statement is input into the computing engine selection model, and first data output by the computing engine selection model is acquired, the first data being used to indicate the computing engine with the shortest execution time for the SQL statement.

[0084] The computing engine selection model is a model trained based on the training method of the computing engine selection model of the above embodiment.

[0085] In this embodiment, the model of the computing engine selection model trained based on the training method of the computing engine selection model is used to input the SQL statement into the computing engine selection model, and the corresponding first data is output by the model, the first data being the engine label data in the training method of the computing engine selection model, and the corresponding computing engine can be selected according to the engine label without manual attempts, thereby greatly saving the time cost and resource cost.

[0086] Optionally, in step S22, inputting the SQL statement into the computing engine selection model comprises:

[0087] acquiring the text feature, the vectorized data of the SQL statement, and the data feature of the data table in the SQL statement;

[0088] inputting the vectorized data of the SQL statement into the computing engine selection model, and acquiring a one-dimensional hidden layer vector representation;

[0089] inputting the one-dimensional hidden layer vector representation, the text feature of the SQL statement, and the data feature of the data table in the SQL statement into the computing engine selection model.

[0090] It should be noted that the computing engine selection model of this embodiment comprises a first sub-model and a second sub-model, the vectorized data of the SQL statement is input into the first sub-model to obtain a one-dimensional hidden layer vector representation, and the one-dimensional hidden layer vector representation, the text feature of the SQL statement, and the data feature of the data table in the SQL statement are input into the second sub-model to obtain the first data used to determine the computing engine with the shortest execution time. The computing engine selection model determines the first data according to the text feature, the vectorized data of the SQL statement, and the data feature of the data table in the SQL statement, so that the first data can accurately indicate the computing engine with the shortest execution time.

[0091] Optionally, in step S22, acquiring the data feature of the data table in the SQL statement comprises:

[0092] determining the table name of the data table in the SQL statement based on the text feature.

[0093] determine historical data of the data table based on the table name, wherein the historical data comprises a Hadoop Distributed File System (HDFS) file size;

[0094] determine current data of the data table by using a time series prediction method on the historical data, wherein the current data is a data feature of the data table.

[0095] determine the current data of the data table by using an Auto Regression Moving Average (ARMA) algorithm or an Auto Regressive Integrated Moving Average (ARIMA) algorithm. For example, based on the table name and the data type of the data table in the SQL statement, the historical data of the data table on the T-th day is obtained from a statistical table, and the data on the T+1-th day is obtained by using the above algorithm. Assuming that the data increment is consistent every hour, the current data size of the data table can be calculated according to the current time of the data table, without the time-consuming step of obtaining real-time data, thereby improving the efficiency of the computing engine selection method.

[0096] As shown in FIG. 1, Figure 4 The SQL statement is obtained, vectorized data of the SQL statement is obtained based on NLP technology, the vectorized data of the SQL statement is input into a classifier 1, the classifier 1 is a first sub-model in the computing engine selection model, and a one-dimensional hidden layer vector representation is obtained. Based on a text analysis method, text features and data features of the SQL statement are obtained. The text features are, for example, whether the SQL statement is a single table or multiple tables, that is, the number of data tables, whether isSingle, isMult, isGroup, and other aggregation functions are used. Historical data of the data table in the SQL statement is obtained, an ARMA model is used for time fitting, current data is obtained, the current data is normalized by using a min-max standardization method, data features are obtained, the data features, the text features, and the one-dimensional hidden layer vector representation are input into a classifier 2, that is, a second sub-model in the computing engine selection model, first data is output, and a computing engine with the shortest execution time is selected according to the first data.

[0097] The training device of the computing engine selection model is also provided in the embodiments of the present application. In the embodiments of the present application, the training device of the computing engine selection model is taken as an example to execute the training method of the computing engine selection model, and the accompanying drawings are combined to illustrate the training device of the computing engine selection model. Figure 5 The training device of the computing engine selection model is also provided in the embodiments of the present application. In the embodiments of the present application, the training device of the computing engine selection model is taken as an example to execute the training method of the computing engine selection model, and the accompanying drawings are combined to illustrate the training device of the computing engine selection model.

[0098] The first obtaining module 501 is configured to obtain a structured query language (SQL) training set and text features, vectorized data and data features of data tables in each SQL statement in the SQL training set;

[0099] The generating module 502 is configured to generate an engine label of each SQL statement, where the engine label is used to indicate a respective computing engine with the shortest execution time of each SQL statement.

[0100] The training module 503 is configured to train a computing engine selection model by taking the text features, the vectorized data and the data features of the data tables in each SQL statement as training data and taking the engine label corresponding to each SQL statement as a label value, and train the computing engine selection model.

[0101] The trained computing engine selection model is configured to output a computing engine with the shortest execution time of a target SQL statement according to the target SQL statement.

[0102] Optionally, the training module 503 is further configured to:

[0103] input the vectorized data of each SQL statement and the engine label corresponding to each SQL statement as a label value into the computing engine selection model to obtain a one-dimensional hidden layer vector representation;

[0104] train the computing engine selection model by taking the engine label corresponding to each SQL statement as a label value based on the one-dimensional hidden layer vector representation, the text features of each SQL statement and the data features of the data tables in each SQL statement.

[0105] Optionally, the first obtaining module 501 is further configured to:

[0106] perform text analysis on each SQL statement in the SQL training set to determine text features of each SQL statement, where the text features include the number of data tables in the SQL statement and information used to represent whether an aggregation function exists in the SQL statement;

[0107] perform text vectorization on each SQL statement in the SQL training set based on natural language processing (NLP) technology to obtain vectorized data of each SQL statement;

[0108] determine table names of data tables involved in the SQL statement based on the text features of the SQL statement;

[0109] Based on the table name of the data table, historical data of the data table is obtained from a statistical table, and a maximum value of the historical data is taken as the data feature of the data table in the SQL statement.

[0110] Optionally, the training module 503 is further configured to:

[0111] The one-dimensional hidden layer vector representation, the text feature of each SQL statement, and the data feature of the data table in each SQL statement are taken as training data, and the engine label corresponding to each SQL statement is taken as a label value, and distributed gradient boosting library XGBoost or gradient boosting decision tree GBDT or light gradient boosting machine LightGBM is used for training.

[0112] The generation apparatus of the training model tool can train a computing engine selection model based on the text feature of the SQL statement, the vectorized data, the data feature of the data table in the SQL statement, and the engine label corresponding to the SQL statement, so as to improve the accuracy of the output result of the computing engine selection model.

[0113] It should be noted that the generation apparatus of the training model tool provided in the embodiments of the present application can implement all the technical processes of the above-mentioned generation method of the training model tool, and can achieve the same technical effects. To avoid repetition, details are not repeated here.

[0114] The embodiments of the present application also provide a computing engine selection apparatus. In the embodiments of the present application, the computing engine selection apparatus executes the computing engine selection method as an example, and the accompanying drawings are combined to Figure 6 The computing engine selection apparatus 600 provided in the embodiments of the present application is described. The computing engine selection apparatus 600 comprises:

[0115] The second acquisition module 601 is configured to acquire a computing engine selection request and acquire a SQL statement in the computing engine selection request.

[0116] The selection module 602 is configured to input the SQL statement into a computing engine selection model, and acquire first data output by the computing engine selection model, wherein the first data is used to indicate a computing engine with the shortest execution time for the SQL statement.

[0117] The computing engine selection model is a model trained based on the above-mentioned training method of the computing engine selection model.

[0118] Optionally, the selection module 602 is further configured to:

[0119] The text feature of the SQL statement, the vectorized data, and the data feature of the data table in the SQL statement are acquired.

[0120] The vectorization data of the SQL statement is input into the computing engine selection model, and a one-dimensional hidden layer vector representation is obtained;

[0121] The one-dimensional hidden layer vector representation, the text features of the SQL statement, and the data features of the data table in the SQL statement are input into the computing engine selection model.

[0122] Optionally, the selection module 602 is further configured to:

[0123] Based on the text features, a table name of the data table in the SQL statement is determined;

[0124] Based on the table name, historical data of the data table is determined, wherein the historical data includes a Hadoop Distributed File System (HDFS) file size;

[0125] A time series prediction method is used on the historical data to determine current data of the data table, and the current data is the data feature of the data table.

[0126] The computing engine selection apparatus provided in the application can automatically select a computing engine with the shortest execution time according to an input SQL statement, without manual operation.

[0127] It should be noted that the computing engine selection apparatus provided in the embodiments of the application can implement all the technical processes of the computing engine selection method and achieve the same technical effects. To avoid repetition, details are not described herein.

[0128] The apparatus in the embodiments of the application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like. The non-mobile electronic device can be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like. The embodiments of the application are not limited in this regard.

[0129] Optionally, as shown in Figure 7 The electronic device 700 includes a processor 701 and a memory 702. The memory 702 stores programs or instructions executable by the processor 701. When the programs or instructions are executed by the processor 701, the programs or instructions implement each step of the training method of the computing engine selection model or the computing engine selection method embodiment described above, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0130] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0131] The embodiments of the present application also provide a readable storage medium, which stores programs or instructions. When the programs or instructions are executed by a processor, the programs or instructions implement each process of the training method of the computing engine selection model or the computing engine selection method embodiment described above, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0132] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0133] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element. In addition, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to the order of performing the functions shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in a reverse order, for example, the described method can be performed in an order different from that described, and various steps can be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0134] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, also can be through hardware, but in many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art to make contributions can be embodied in the form of computer software products, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc), including a number of instructions to make a terminal (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0135] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this, any skilled in the art within the scope of the technology disclosed in the present application, can easily think of changes or replacement, should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A training method for a computational engine selection model, characterized in that, The method includes: Obtain a Structured Query Language (SQL) training set, as well as the text features, vectorized data, and data features of the data tables in each SQL statement within the training set; Generate an engine tag for each of the SQL statements, the engine tag being used to indicate the computing engine with the shortest execution time for each of the SQL statements; The text features, vectorized data, and data features of the data table in each SQL statement are used as training data for the computing engine selection model, and the engine label corresponding to each SQL statement is used as the label value to train the computing engine selection model. The trained computing engine selection model is used to output the computing engine with the shortest execution time of the target SQL statement based on the input target SQL statement. The step of using the text features, vectorized data, and data features of the data tables in each SQL statement as training data for the computational engine selection model, and using the engine label corresponding to each SQL statement as a label value, to train the computational engine selection model includes: The vectorized data of each SQL statement and the engine label corresponding to each SQL statement are input into the computing engine selection model as label values ​​to obtain a one-dimensional hidden layer vector representation. Based on the one-dimensional hidden layer vector representation, the text features of each SQL statement, and the data features of the data table in each SQL statement, the engine label corresponding to each SQL statement is used as the label value to train the computing engine selection model.

2. The training method for the computational engine selection model as described in claim 1, characterized in that, Obtain the SQL training set, as well as the text features, vectorized data, and data features of the data tables in each SQL statement within the training set, including: Text analysis is performed on each SQL statement in the SQL training set to determine the text features of each SQL statement. The text features include the number of data tables in the SQL statement and information used to characterize whether there are aggregate functions in the SQL statement. Based on Natural Language Processing (NLP) technology, each SQL statement in the SQL training set is vectorized into text to obtain vectorized data for each SQL statement. Based on the textual features of the SQL statement, determine the table names of the data tables involved in the SQL statement; Based on the table name of the data table, historical data of the data table is obtained from the statistics table, and the maximum value of the historical data is used as the data feature of the data table in the SQL statement.

3. The training method for the computational engine selection model as described in any one of claims 1 to 2, characterized in that, The training of the computational engine selection model based on the one-dimensional hidden layer vector representation, the text features of each SQL statement, the data features of the data table in each SQL statement, and the engine label corresponding to each SQL statement includes: The training data consists of the one-dimensional hidden layer vector representation, the text features of each SQL statement, and the data features of the data table in each SQL statement. The engine label corresponding to each SQL statement is used as the label value. The training is performed using the distributed gradient boosting library XGBoost, the gradient boosting decision tree GBDT, or the lightweight gradient boosting machine LightGBM.

4. A method for selecting a computing engine, characterized in that, The method includes: Obtain the computing engine selection request, and obtain the SQL statement in the computing engine selection request; The SQL statement is input into the computing engine selection model, and the first data output by the computing engine selection model is obtained. The first data is used to indicate the computing engine with the shortest execution time for the SQL statement. The computing engine selects a model that is trained based on the method described in any one of claims 1-3.

5. The computing engine selection method as described in claim 4, characterized in that, The step of inputting the SQL statement into the computing engine to select a model includes: Obtain the text features, vectorized data, and data features of the data tables in the SQL statement; The vectorized data of the SQL statement is input into the computing engine to select a model and obtain a one-dimensional hidden layer vector representation; The one-dimensional hidden layer vector representation, the text features of the SQL statement, and the data features of the data table in the SQL statement are input into the computing engine selection model.

6. The computing engine selection method as described in claim 5, characterized in that, The step of obtaining the data characteristics of the data table in the SQL statement includes: Based on the text features, determine the table name of the data table in the SQL statement; Based on the table name, determine the historical data of the data table, wherein the historical data includes the file size of the distributed file system HDFS; A time series forecasting method is used on the historical data to determine the current data of the data table, and the current data is the data feature of the data table.

7. A training apparatus for a computational engine selection model, characterized in that, The device includes: The first acquisition module is used to acquire a Structured Query Language (SQL) training set, as well as the text features, vectorized data, and data features of the data tables in each SQL statement in the SQL training set. A generation module is used to generate an engine tag for each of the SQL statements, wherein the engine tag is used to indicate the computing engine with the shortest execution time for each of the SQL statements. The training module is used to train the computing engine selection model by using the text features, vectorized data, and data features of the data table in each SQL statement as training data, and the engine label corresponding to each SQL statement as the label value. The trained computing engine selection model is used to output the computing engine with the shortest execution time of the target SQL statement based on the input target SQL statement. The training module is also used for: The vectorized data of each SQL statement and the engine label corresponding to each SQL statement are input into the computing engine selection model as label values ​​to obtain a one-dimensional hidden layer vector representation. Based on the one-dimensional hidden layer vector representation, the text features of each SQL statement, and the data features of the data table in each SQL statement, the engine label corresponding to each SQL statement is used as the label value to train the computing engine selection model.

8. A computing engine selection device, characterized in that, The device includes The second acquisition module is used to acquire the computing engine selection request and acquire the SQL statement in the computing engine selection request; The selection module is used to input the SQL statement into the computing engine selection model and obtain the first data output by the computing engine selection model. The first data is used to indicate the computing engine with the shortest execution time for the SQL statement. The computing engine selects a model that is trained based on the method described in any one of claims 1-3.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the training method for the computational engine selection model as claimed in any one of claims 1 to 3 or the computational engine selection method as claimed in any one of claims 4 to 6.

Images