A query task slicing method and device

Abstract

This invention provides a query task slicing method and apparatus, relating to the field of data processing technology. The method includes: generating prediction results for multiple query tasks, where the prediction results indicate whether slicing of the multiple query tasks is necessary; if the prediction results indicate that slicing of the multiple query tasks is necessary, determining multiple atomic tables corresponding to each query task among the multiple query tasks to be processed, where the multiple atomic tables store information including the query results corresponding to the respective query tasks; constructing a query table based on all the atomic tables corresponding to the multiple query tasks, where the query table stores the multiple query tasks and represents the correspondence between each query task and the atomic tables; determining slicing nodes based on the query table; and slicing the multiple query tasks based on the slicing nodes. This invention can reduce the latency of executing query tasks.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a query slicing method and apparatus. Background Technology

[0002] With the development of information technology, production data has increased to a new order of magnitude. To handle large-scale query tasks, related technologies often employ slicing, dividing a long query task into multiple shorter tasks for sequential querying, thus reducing query latency. However, in these technologies, query task slicing is typically done using manually pre-configured parameters. In actual production, different query tasks contain varying content, making it impossible to effectively slice a long task into multiple shorter tasks of roughly equal length. This results in some sliced ​​tasks still being quite long, leading to higher latency for subsequent query execution.

[0003] It is evident that the relevant technologies suffer from high latency when executing query tasks after slicing them. Summary of the Invention

[0004] This invention provides a method and apparatus for querying task slices to solve the problem of high query latency after querying task slices in related technologies.

[0005] To solve the above problems, the present invention is implemented as follows: In a first aspect, embodiments of the present invention provide a method for querying task slices, including: Generate prediction results for multiple query tasks, the prediction results being used to characterize whether the multiple query tasks need to be sliced, the multiple query tasks being structured query language (SQL) tasks; When the prediction result characterization requires slicing the multiple query tasks, multiple atomic tables are determined for each query task among the multiple query tasks to be processed. The information stored in the multiple atomic tables includes the query results corresponding to the respective query tasks. A query table is constructed based on all atomic tables corresponding to the multiple query tasks. The query table is used to store the multiple query tasks and to represent the correspondence between each query task and an atomic table. The slice nodes are determined based on the query table; The multiple query tasks are sliced ​​based on the slice nodes.

[0006] Secondly, embodiments of the present invention provide a query task slicing device, comprising: A generation module is used to generate prediction results for multiple query tasks. The prediction results are used to characterize whether the multiple query tasks need to be sliced. The multiple query tasks are structured query language (SQL) tasks. The first determining module is used to determine multiple atomic tables corresponding to each query task among the multiple query tasks to be processed when the prediction result characterization requires slicing of the multiple query tasks. The information stored in the multiple atomic tables includes the query results corresponding to the respective query tasks. A construction module is used to construct a query table based on all atomic tables corresponding to the multiple query tasks. The query table is used to store the multiple query tasks and to characterize the correspondence between each query task and an atomic table. The second determining module is used to determine slice nodes based on the query table; The slicing module is used to slice the multiple query tasks based on the slice nodes.

[0007] Thirdly, embodiments of the present invention also provide an electronic device, including a transceiver and a processor. The processor is configured to generate prediction results corresponding to multiple query tasks. The prediction results are used to characterize whether the multiple query tasks need to be sliced. The multiple query tasks are Structured Query Language (SQL) tasks. The processor is configured to, when the prediction result indicates that slicing of the multiple query tasks is required, determine multiple atomic tables corresponding to each query task among the multiple query tasks to be processed, wherein the information stored in the multiple atomic tables includes the query results corresponding to the respective query tasks, and the multiple query tasks are Structured Query Language (SQL) tasks. The processor is further configured to construct a query table based on all atomic tables corresponding to the plurality of query tasks, the query table being used to store the plurality of query tasks and to characterize the correspondence between each query task and an atomic table; The processor is further configured to determine slice nodes based on the query table; The processor is further configured to slice the plurality of query tasks based on the slice nodes respectively.

[0008] Fourthly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the query task slicing method described in the first aspect above.

[0009] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the query task slicing method described in the first aspect.

[0010] In a sixth aspect, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the query task slicing method described in the first aspect.

[0011] In this embodiment of the invention, prediction results corresponding to multiple query tasks are generated. These prediction results indicate whether slicing of the multiple query tasks (Structured Query Language, SQL) is required. If the prediction results indicate that slicing is necessary, multiple atomic tables corresponding to each query task are determined. These atomic tables store information including the query results corresponding to the respective query tasks. A query table is constructed based on all the atomic tables corresponding to the multiple query tasks. This query table stores the multiple query tasks and represents the correspondence between each query task and the atomic tables. Slicing nodes are determined based on the query table. The multiple query tasks are then sliced ​​based on these sliceing nodes. This method of constructing a query table, storing multiple query tasks and their correspondence with atomic tables, and determining sliceing nodes based on the query table allows for slicing of the multiple query tasks according to their correspondence with atomic tables. Slicing the multiple query tasks based on the sliceing nodes prevents the resulting tasks from being excessively long, thus effectively reducing the latency of executing the query tasks. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a query task slicing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the connection diagram provided in the embodiment of the present invention; Figure 3 This is a schematic diagram of the encoding structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the model structure provided in the embodiment of the present invention; Figure 5This is a schematic diagram of model training provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a horizontal slice provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of a longitudinal slice provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the composite slice provided in an embodiment of the present invention; Figure 9 This is an overall flowchart of the query task slice provided in the embodiments of the present invention; Figure 10 This is a structural diagram of a query task slicing device provided in an embodiment of the present invention; Figure 11 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Please see Figure 1 , Figure 1 This is a flowchart of a query task slicing method provided in an embodiment of the present invention, such as... Figure 1 As shown, it includes the following steps: Step 101: Generate prediction results for multiple query tasks. The prediction results are used to characterize whether the multiple query tasks need to be sliced. The multiple query tasks are Structured Query Language (SQL) tasks.

[0016] Step 102: Determine the multiple atomic tables corresponding to each query task in the multiple query tasks to be processed. The information stored in the multiple atomic tables includes the query results corresponding to the query tasks.

[0017] The aforementioned multiple query tasks are tasks uploaded by one or more terminals. The determination of whether slicing is necessary is made by generating prediction results corresponding to multiple query tasks. It should be understood that if the multiple query tasks consume relatively few computing resources and the load on the cluster or server platform executing the query tasks is low, slicing is not required; the tasks can be directly forwarded to the cluster or server platform executing the query tasks. However, if the multiple query tasks consume a significant amount of computing resources and the load on the cluster or server platform executing the query tasks is high, then slicing is necessary before the cluster or server platform executing the query tasks executes the sliced ​​tasks.

[0018] In some implementations, the multiple atomic tables corresponding to each query task among the multiple query tasks to be processed can be uploaded to the data management platform by one or more terminals. The data management platform then determines whether slicing is required. If it is determined that slicing of multiple query tasks is required, the data management platform forwards the multiple query tasks to the data slicing processing platform, which then slices the multiple query tasks.

[0019] The aforementioned atomic tables are the atomic tables corresponding to query tasks. By determining the multiple atomic tables corresponding to each query task, it is possible to determine the situation in which atomic tables are called when executing a query task, so as to subsequently determine the slice node for slicing the query task by using all the atomic tables corresponding to multiple query tasks.

[0020] The atomic table stores information including the query results corresponding to the query task, so that the query results corresponding to the query task can be determined through the atomic table when the query task is executed.

[0021] Step 103: Construct a query table based on all atomic tables corresponding to the multiple query tasks. The query table is used to store the multiple query tasks and to represent the correspondence between each query task and an atomic table.

[0022] The query table above stores multiple query tasks and the correspondence between each query task and an atomic table. The query table can be used to determine the status of the query tasks to be processed and the correspondence between different query tasks and different atomic tables. Then, the slice nodes for slicing multiple query tasks can be determined based on the query table.

[0023] For example, a threshold for the number of atomic tables corresponding to the sliced ​​tasks can be pre-configured, and then the slice nodes can be determined based on the number of atomic tables, ensuring that the number of atomic tables corresponding to the sliced ​​tasks is less than the pre-configured threshold. For instance, setting the atomic table threshold to 2 will ensure that the number of atomic tables corresponding to the sliced ​​tasks does not exceed 2.

[0024] For example, a model can be pre-trained, and the query table can be input into the model, which will then determine the slice nodes for slicing multiple query tasks.

[0025] Step 104: Determine the slice nodes based on the query table.

[0026] Step 105: Slice the multiple query tasks based on the slice nodes.

[0027] The aforementioned slice nodes are used to slice multiple query tasks. The slice nodes are determined by querying the table, and multiple query tasks are sliced ​​based on the slice nodes. This allows the sliced ​​tasks to be executed in batches, and the execution results are returned to the terminal that initiated the query task in sequence. This effectively avoids the problem of long response time caused by excessively large query tasks.

[0028] The slice nodes are determined by querying a table. This can be done by combining a pre-configured threshold for the number of atomic tables corresponding to the sliced ​​tasks with the query table, or by using a pre-trained model. By determining slice nodes through a query table, the resulting tasks after slicing multiple query tasks based on slice nodes are less likely to be very long, thus reducing the latency of subsequent query task execution.

[0029] In this embodiment of the invention, prediction results corresponding to multiple query tasks are generated. These prediction results indicate whether slicing of the multiple query tasks (Structured Query Language, SQL) is required. If the prediction results indicate that slicing is necessary, multiple atomic tables corresponding to each query task are determined. These atomic tables store information including the query results corresponding to the respective query tasks. A query table is constructed based on all the atomic tables corresponding to the multiple query tasks. This query table stores the multiple query tasks and represents the correspondence between each query task and the atomic tables. Slicing nodes are determined based on the query table. The multiple query tasks are then sliced ​​based on these sliceing nodes. This method of constructing a query table, storing multiple query tasks and their correspondence with atomic tables, and determining sliceing nodes based on the query table allows for slicing of the multiple query tasks according to their correspondence with atomic tables. Slicing the multiple query tasks based on the sliceing nodes prevents the resulting tasks from being excessively long, thus effectively reducing the latency of executing the query tasks.

[0030] In some implementations, data query tasks can be initiated from multiple terminals and submitted synchronously to the task management platform. The task management platform will first set a task code for the task and query the cluster status to obtain the current number of tasks being processed (NUM).proc Number of queued tasks (NUM) queue CPU resource utilization ratio Memory resource usage ratio Disk space usage ratio Other metrics; the task management platform calls the task parser to parse the submitted tasks and obtain the total number of atomic tables X. table Total number of script lines R tot Number of table association types (Y), and the maximum number of records in the atomic table for a query (Record). max And the average number of records in the atomic table being queried (Record) avg The parameters are used to evaluate whether multiple query tasks need to be submitted to the data slicing processing platform for processing.

[0031] The cluster evaluation function and the task evaluation function can be expressed by the following formulas: ; ; ; in the formula This is the cluster evaluation function, where K is the current task number and m is the cluster node. For task evaluation function, and These represent the weighting coefficients for cluster evaluation and task evaluation, respectively. .

[0032] The following formula is used to determine whether to submit the query task to the data slicing processing platform: ; AVG in the formula Ev_his To set a threshold.

[0033] In one embodiment, generating prediction results corresponding to multiple query tasks includes: Extract the query features corresponding to the multiple query tasks and the cluster features of the processing cluster. The processing cluster is the cluster that executes the multiple query tasks. The cluster features are used to characterize the task queuing status and / or resource usage in the processing cluster. Prediction results are generated based on the query features and the cluster features.

[0034] The query features mentioned above are used to characterize the situation of multiple query tasks, such as the number of multiple query tasks and the field length of multiple query tasks. The cluster features mentioned above are used to characterize the task queuing situation and / or resource usage of the processing cluster that executes multiple query tasks. By using the query features and cluster features, it can be determined whether the processing cluster can directly execute multiple query tasks. If multiple query tasks cannot be processed directly, then the multiple query tasks need to be sliced, and then the processing cluster executes the sliced ​​tasks.

[0035] The query features may include: the total number of atomic tables involved, the total number of script lines, the number of table join types, the maximum number of records in the queried atomic table, the number of evaluation records in the queried atomic table, whether there is aggregation processing in multiple query tasks, whether there is sorting processing in multiple query tasks, the total number of "predicate structures", the number of atomic tables with duplicate primary keys, the number of table joins, and whether there is Cartesian product join processing.

[0036] Cluster characteristics can include: the number of currently processed tasks, the number of queued tasks, the proportion of CPU resources used, the proportion of memory resources used, the proportion of disk space used, and the number of cluster nodes.

[0037] The above prediction results are used to characterize whether the multiple query tasks need to be sliced. By extracting query features and cluster features, different prediction results can be generated based on the query features and cluster features. The prediction results can be used to determine whether the multiple query tasks need to be sliced.

[0038] In some implementations, generating prediction results based on the query features and the cluster features may involve calculating a first set of computing resources required to execute multiple query tasks based on the query features, and determining a second set of computing resources remaining in the processing cluster based on the cluster features. If the first set of computing resources is less than the second set of computing resources, a prediction result is generated to indicate that multiple query tasks do not need to be sliced. If the first set of computing resources is greater than or equal to the second set of computing resources, a prediction result is generated to indicate that multiple query tasks need to be sliced.

[0039] In some implementations, the prediction result is generated based on the query features and the cluster features, or a prediction model is pre-trained and the prediction result is generated through the prediction model.

[0040] Specifically, generating prediction results based on the query features and the cluster features includes: The query features and cluster features are input into the target evaluation model, and the first number of query result entries is output based on the target evaluation model. The target evaluation model is used to predict the number of query result entries under different query features and cluster features. The query features, the cluster features, and the number of the first query result entries are input into the slicing judgment model, and the prediction result is output based on the slicing judgment model. The slicing judgment model is used to predict whether the query needs to be sliced ​​under different query features, cluster features, and the number of query result entries.

[0041] The aforementioned target evaluation model is used to predict the number of query result entries under different query features and clustering features, while the aforementioned slicing judgment model is used to predict whether the query needs to be sliced ​​under different query features, clustering features, and the number of query result entries. In this embodiment of the invention, after extracting the query features and clustering features, the target evaluation model and the slicing judgment model are sequentially used to generate the prediction results.

[0042] The target evaluation model and the slice judgment model are obtained in the following ways: Obtain multiple sample tasks, sample query features and sample cluster features corresponding to the multiple sample tasks, and sample prediction results corresponding to the multiple sample tasks; Each of the multiple sample tasks is encoded to obtain the encoded data corresponding to each sample task. The encoded data is used to characterize the correspondence between the corresponding sample task and different atomic tables, the connection conditions of different fields included in the corresponding sample task, and at least one of the different fields, different operators, and different values ​​included in the corresponding sample task. The initial evaluation model is trained based on the sample query features, the sample cluster features, and the encoded data corresponding to the multiple sample tasks to obtain the intermediate evaluation model and the number of intermediate entries output by the intermediate evaluation model. The initial evaluation model is used to determine the number of query result entries under different query features and cluster features. The initial judgment model is trained based on the sample query features, the sample cluster features, the encoded data corresponding to the multiple sample tasks, and the number of intermediate entries to obtain the intermediate judgment model and the intermediate results output by the intermediate judgment model. If the intermediate result matches the sample prediction result, the intermediate evaluation model is set as the target evaluation model, and the intermediate judgment model is set as the slice judgment model.

[0043] The aforementioned multiple sample tasks can be tasks selected from historical query tasks. The aforementioned sample query features and sample cluster features are the features corresponding to multiple sample tasks. The prediction result of whether to slice multiple sample tasks can be obtained through the sample query features and sample cluster features; the aforementioned sample prediction result is the result of whether to slice multiple sample tasks. By extracting multiple sample tasks, the corresponding sample query features and sample cluster features of multiple sample tasks, and the corresponding sample prediction results of multiple sample tasks from historical query tasks, the target evaluation model and the slicing judgment model can be trained.

[0044] Furthermore, after obtaining multiple sample tasks, sample query features and sample cluster features corresponding to the multiple sample tasks, and sample prediction results corresponding to the multiple sample tasks, each sample task is then encoded to obtain encoded data. Through the encoded data, the correspondence between the corresponding sample task and different atomic tables, the connection conditions of different fields included in the corresponding sample task, and at least one of the different fields, different operators, and different numerical values ​​included in the corresponding sample task can be determined.

[0045] Specifically, encoding each sample task separately can include at least one of the following: Encode the atomic table and attributes corresponding to each sample task; The connection conditions for each sample task are encoded. The predicate structure of each sample task is encoded.

[0046] The encoding process for the atomic table and attributes corresponding to each sample task can be achieved by determining the atomic table corresponding to each sample task based on all pre-stored atomic tables, and then encoding the atomic table and attributes corresponding to each sample task. In this way, by encoding the atomic table and attributes corresponding to each sample task, the encoding result can characterize the correspondence between the corresponding sample task and different atomic tables.

[0047] For example, assuming the total number of atomic tables stored in the database table is N, an atomic table A with n attributes can be represented as A = {a1} R a2 R ,...,a n RTo encode the atomic table A, one-hot encoding can be used, encoding A into a one-hot vector of length N. The attribute identifiers (IDs) in atomic table A are also encoded using one-hot vectors, with a length equal to the total number of distinct attribute names in all tables, M. Therefore, for each sample task's corresponding atomic table, the encoding result is a vector of length M+N. For example, the encoded atomic table and attributes can be represented as FE(A.id)=[1,0,0,0,1,0,...,0,0].

[0048] The above join conditions refer to the joins of different fields. It should be noted that both the sample task and the query task include predicate structures, which are triples (col, op, val) composed of different fields (col), different operators (op), and different values ​​(val). For both the sample task and the query task, the join conditions are sequence-independent; that is, changing the left and right attributes of the equi-join should maintain the same feature vector, which can be represented as: A.id and B.film_id are fields of two attributes, respectively. In this embodiment, the connection conditions of each sample task are encoded to capture the correlation between different atomic tables, thereby improving the accuracy of subsequent determination of whether slicing is needed.

[0049] In some implementations, the connection conditions for each sample task are encoded, which can be based on the connection relationships to establish, for example... Figure 2 The connection graph shown has different nodes corresponding to different fields, and the edges between nodes correspond to whether there is a connection relationship between the two fields. Then, the adjacency matrix is ​​obtained by encoding the connection graph, and the connection conditions of each sample task can be determined through the adjacency matrix.

[0050] Furthermore, the predicate structure includes different fields (col), different operators (op), and different values ​​(val). `col` represents the column / attribute involved in the current predicate, `op` is a comparison operator in {>,=,<,≥,≤,!} used to determine the size between `col` and `val`, and `val` is a random value selected from the range of attribute values ​​corresponding to `col`. It should be understood that `col` encoding is essentially a combination of the base table and the attribute; the `op` operator can also be processed using one-hot encoding, for example, using [0,0,0,0,1,0,...,0,0] to represent the operator ≤. In this way, by encoding the predicate structure of each sample task, the sample task itself can be encoded, allowing the encoding result to determine the different fields, operators, and values ​​included in the corresponding sample task.

[0051] In some implementations, each of the plurality of sample tasks is encoded separately to obtain the encoded data corresponding to each sample task. This can be achieved by concatenating the results of encoding the atomic tables and attributes corresponding to each sample task, encoding the connection conditions of each sample task, and / or encoding the predicate structure of each sample task.

[0052] For example, such as Figure 3 As shown, the encoded data for each sample task includes the result of encoding the atomic table and attributes corresponding to each sample task (i.e., table feature encoding), the result of encoding the connection conditions of each sample task (i.e., connection feature encoding), and / or the result of encoding the predicate structure of each sample task (i.e., predicate feature encoding).

[0053] Furthermore, three type codes can be added to the header of the encoded data corresponding to each sample task. These codes represent the presence of results for encoding the atomic tables and attributes corresponding to each sample task, encoding the join conditions of each sample task, and / or encoding the predicate structure of each sample task. For example, the type codes are 000, 001, 010, 011, 100, 101, 110, and 111. The first bit of the code represents the state of the corresponding encoding result. 0 indicates no, and 1 indicates yes. For example, 011 indicates that the encoded data corresponding to the sample task includes results for encoding the join conditions and predicate structure of each sample task.

[0054] After obtaining the encoded data, the initial evaluation model is trained based on sample query features, sample cluster features, and encoded data corresponding to multiple sample tasks, resulting in an intermediate evaluation model and the number of intermediate entries output by the intermediate evaluation model. Then, the initial judgment model is trained based on the same sample query features, sample cluster features, encoded data corresponding to multiple sample tasks, and the number of intermediate entries, resulting in an intermediate judgment model and its intermediate output. Thus, by simultaneously training both the initial evaluation model and the initial judgment model, a judgment can be made by combining the intermediate output of the final intermediate judgment model with the sample prediction results to determine whether the intermediate evaluation model and the intermediate judgment model have been successfully trained.

[0055] In some implementations, the initial evaluation model and the initial judgment model can be constructed using models built with various algorithms as the base model. Specifically, the base model can be constructed using convolutional neural networks, backpropagation (BP) neural networks, extreme gradient boosting (XGboost), and gradient boosting decision tree (GBDT) algorithms.

[0056] Furthermore, such as Figure 4 As shown, after constructing a basic model using multiple algorithms, the outputs of different basic models can be fused through a linear layer, a rectified linear unit (ReLU), and an activation function (such as the sigmoid function) to obtain the final number of entries and prediction results.

[0057] Thus, the overall process of training the model in this embodiment is as follows: Figure 5 As shown, by determining the atomic table, connection conditions, and / or predicate structure corresponding to the sample task, feature encoding is performed to obtain encoded data. For example, the table features, connection graph, and predicate tree of the atomic table are extracted by a feature extractor, and then encoded respectively to obtain table feature encoding representing the correspondence between the corresponding sample task and different atomic tables, connection feature encoding representing the connection conditions of different fields included in the corresponding sample task, and / or predicate feature encoding representing the different fields, different operators, and different numerical values ​​included in the corresponding sample task. In this way, the target evaluation model and the slice judgment model are trained.

[0058] In one embodiment, determining the slice node based on the query table includes at least one of the following: A first slice node is determined based on the set sliding window and the query table, and the first slice node is located between the two set sliding windows in the query table; The second slice node is determined based on the query table, and the second slice node corresponds to the fields of the query task corresponding to different atomic tables; The step of slicing the multiple query tasks based on the slice nodes includes at least one of the following: The multiple query tasks are horizontally sliced ​​based on the first slice node, and the horizontal slicing is used to slice different query tasks. The multiple query tasks are vertically sliced ​​based on the second slice node, and the vertical slicing is used to slice different fields included in a query task.

[0059] The first slice node is used to perform horizontal slicing of multiple query tasks, and the second slice node is used to perform vertical slicing of multiple query tasks. Horizontal slicing of multiple query tasks can be performed as follows: Figure 6 As shown, vertical slicing of multiple query tasks can be performed as follows: Figure 7 As shown, query tasks can be sliced ​​through horizontal slicing and / or vertical slicing. In some implementations, horizontal slicing or vertical slicing can be used alone, or as shown in the example below. Figure 8 The diagram shows the use of horizontal and vertical slicing.

[0060] It should be understood that the query table can be adjusted before slicing multiple query tasks. Specifically, the multiple query tasks stored in the query table can be shown in Table 1 below: Table 1 queries multiple query tasks stored in the table.

[0061] In Table 1, N represents yes and Y represents no.

[0062] Furthermore, the correspondence between each query task and the atomic table in the query table storage can be shown in Table 2 below: Table 2 shows the correspondence between each query task and the atomic table stored in the query table.

[0063] In Table 2, association type 1 represents left association, 2 represents inner association, 3 represents full association, and 4 represents right association.

[0064] The structures of Tables 1 and 2 are adjusted to obtain Table 3 as shown below: Table 3: Adjusted Query Table

[0065] Table 3 can be used to determine the atom tables and their associations for different fields, as well as whether a field can be used as a slice node. The first and second slice nodes can then be determined using Table 3.

[0066] In some implementations, a first slice node is determined based on a set sliding window and a query table, such that the first slice node is located between two set sliding windows in the query table, thereby allowing multiple query tasks to be horizontally sliced ​​through the first slice node.

[0067] If multiple query tasks are stored in the query table using fields such as sequential ID, horizontal slicing can be performed based on the sequential ID field as the slicing dimension. If fields such as sequential ID do not exist, horizontal slicing can be performed by taking the modulo of the hash value of the primary key.

[0068] In addition, the sliding window can be a pre-configured static window, such as for the number of 3 query tasks; or it can be a dynamic sliding window calculated based on the total number of records in the atomic tables of the query tasks, the number of partitions, and other related factors.

[0069] Specifically, assume the starting row number of the j-th sliding window is row_numb s_j The terminating row number is row_numb e_j Sliding window length Related to the total number of records and partitions in the atomic table, it can be calculated using the following formula: ; In the formula, Record represents the number of records, and Num represents the number of records. part This represents the number of partitions. It should be understood that if the query table contains a sequential ID field (e.g., a "row_id" field), then the "row_id" of the sliced ​​task will be located between i and... Between. If the sequential ID field does not exist, the hash modulo method is used for segmentation, and the first task (starting task) in the nth sliding window after slicing is row_numb. s_n =row_numb e_(n-1) +1, row_numb e_(n-1) The last task in the (n-1)th sliding window is row_numb. The last task (the terminating task) in the nth sliding window is row_numb. e_n =row_numb s_n + .

[0070] Furthermore, set the step size of the sliding window. It can also be calculated using the number of records and the number of partitions, specifically using the following formula: ; In the formula, The sliding coefficient takes values ​​in the interval (0, 1). In this embodiment, the sliding coefficient can be initialized. The value is 0.1. Step size, The sliding window is determined by its length. The first slice node is determined by the sliding window. Horizontal slicing is performed using the first slice node to obtain a set of multiple tasks, denoted as subset. i .

[0071] Furthermore, if the number of remaining entries to be segmented in the atomic table (Record) l <= Then use Record l As the step length of this wheel Generate the final slice subset of the dataset for this round. p Then all the sliced ​​subsets are subset1, subset2, subset3, ..., subset... p .

[0072] In some implementations, the query time for each slice subset of the dataset can be recorded, specifically represented as Tim1, Tim2, Tim3, ..., Tim... p The first performance evaluation metric for slicing based on the first slice node. Defined as: ; The first performance evaluation index corresponding to the currently determined first slice node is calculated using the above formula.

[0073] Furthermore, let A new sliding coefficient is determined to generate a new first slice node for horizontal slicing, and the first performance evaluation index corresponding to the new first slice node is calculated. For example, n sliding coefficients are generated, corresponding first slice nodes are generated for each, and the corresponding first performance evaluation index is calculated. Furthermore, the first performance evaluation metric that is minimized (i.e., The first slice node corresponding to the sliding coefficient is used for horizontal slicing.

[0074] In one embodiment, determining the second slice node based on the query table includes: The association parameters of the multiple atomic tables are determined based on the query table, and the association parameters are used to characterize the association between different atomic tables; At least one second slice node is determined based on the association parameters and the number of query tasks associated with each of the multiple atomic tables.

[0075] It should be noted that since multiple atomic tables corresponding to multiple query tasks are related, when performing vertical slicing on query tasks, slicing can be performed based on the correlation parameters of the multiple atomic tables corresponding to multiple query tasks. This ensures that the fields of the related atomic tables are in the same subset as much as possible after slicing, so that the subsequent processing cluster can quickly execute the sliced ​​tasks.

[0076] Among them, the associated parameters can be extracted by querying the correspondence between different query tasks stored in the query table.

[0077] In some implementations, the atomic tables associated with each query task can be determined by association parameters, and the field corresponding to the atomic table with the most associated query tasks in the atomic table associated with each query task is used as the first slice node.

[0078] In one embodiment, determining at least one second slice node based on the association parameters and the number of query tasks associated with each of the plurality of atomic tables includes: Input the association parameters and the number of query tasks corresponding to each atomic table in the plurality of atomic tables into the node confirmation model, and output at least one second slice node based on the node confirmation model; The node confirmation model is obtained in the following way: Obtain a sample lookup table, multiple sample atom tables corresponding to the sample lookup table, and sample slice nodes corresponding to multiple sample tasks included in the sample lookup table; Based on the multiple sample atomic tables, the sample association parameters of the multiple sample tasks are determined. The sample association parameters are used to characterize the sample atomic tables that correspond to the multiple sample tasks, as well as the association between the multiple sample atomic tables. The sample lookup table is adjusted based on the sample association parameters, and the adjusted sample lookup table includes parameter columns corresponding to the sample association parameters; The initial confirmation model is trained based on the sample slice nodes and the adjusted sample query table to obtain the node confirmation model. The initial confirmation model is used to predict the slice nodes corresponding to different query tasks.

[0079] In this embodiment of the invention, a node confirmation model is trained to determine the second slice node corresponding to the query task, thereby enabling vertical slicing of the query task based on the second slice node. A simple initial confirmation model can be constructed using the LightGBM ensemble algorithm, and then trained using a sample lookup table and multiple sample atom tables corresponding to the sample lookup table.

[0080] It should be noted that the sample lookup table and the multiple sample atom tables corresponding to the sample lookup table can be obtained by outputting the target evaluation model and the slice judgment model respectively. This enables the joint confirmation of at least one second slice node through the target evaluation model, the slice judgment model and the node confirmation model.

[0081] In one embodiment, the vertical slicing of the plurality of query tasks based on the second slice node includes: Based on the at least one second slice node, each query task is divided into at least one first field and multiple second fields. The at least one first field is a field among the multiple initial fields included in each query task that has a corresponding relationship with the at least one second slice node. The multiple second fields are fields other than the at least one first field among the multiple initial fields. The multiple query tasks are vertically sliced ​​based on the at least one first field and the multiple second fields. Each subtask corresponding to each sliced ​​query task includes a first field and at least one second field.

[0082] The first and second fields mentioned above are obtained by slicing through the second slice node. That is, after vertically slicing the query task through the second slice node, the dataset X of the first field is obtained. key ={key _1 ,key2,key3,..,key m} and the dataset X of the second field non-key ={col1,col2,col3,..,col p Then, based on at least one first field and multiple second fields, the multiple query tasks are vertically sliced, so that each subtask corresponding to each sliced ​​query task includes one first field and at least one second field, that is, each subtask corresponds to at least one atomic table.

[0083] For example, each round can randomly select from X in a manner without replacement. non-key Select k non-related second fields and X key The first field of each element together forms a new subtask. Here, k is calculated using the following formula: ; In the formula, Y is the adjustment coefficient, and its value is located in the interval (0,1). In this embodiment, Y can be initialized to 0.1.

[0084] If X non-key If the number of remaining fields is less than k, then all remaining fields are used to generate the last subtask of this round. The set of subtasks for the generated vertical slices is defined as sub-dataset1, sub-dataset2, sub-dataset3, ..., sub-dataset m .

[0085] Furthermore, the query times TM1, TM2, TM3, ..., TM of the subtask set of the vertical slice are statistically analyzed. n And calculate the second performance evaluation index based on the second slice node for slicing. for: ; In the formula, TM comb The query time for each subtask.

[0086] Furthermore, it can make This allows for further vertical slicing of the query task using different adjustment coefficients. Specifically, n adjustment coefficients can be pre-set, and a second performance evaluation metric can be calculated for each adjustment coefficient after slicing. Choose the one with the lowest second performance evaluation metric (i.e., The adjustment coefficient is used to perform longitudinal slicing.

[0087] In some implementations, horizontal and vertical slices can be combined to achieve composite slicing for multiple query tasks (i.e., slicing based on the first slice node and the second slice node). The third performance evaluation metric T corresponding to the composite slice is: ; In the formula, TME ll For the query time of the ll-th subtask after slicing, TME comb The query time for each subtask.

[0088] Furthermore, the minimum performance evaluation index is determined from the first performance evaluation index corresponding to the horizontal slice, the second performance evaluation index corresponding to the vertical slice, and the third performance evaluation index corresponding to the composite slice. The slicing method corresponding to the minimum performance evaluation index is then used as the slicing method for the final slicing of multiple query tasks.

[0089] Specifically, in one embodiment, determining the slice node based on the query table includes: The first slice node is determined based on the set sliding window and the query table, and the second slice node is determined based on the query table; The step of slicing the multiple query tasks based on the slice nodes includes: Calculate a first performance evaluation index, a second performance evaluation index, and a third performance evaluation index. The first performance evaluation index is used to represent the slicing situation based on the first slice node, the second performance evaluation index is used to characterize the slicing situation based on the second slice node, and the third performance evaluation index is used to characterize the slicing situation based on both the first slice node and the second slice node. The multiple query tasks are sliced ​​based on the slice nodes corresponding to the target performance evaluation index, wherein the target performance evaluation index is the smallest performance evaluation index among the first performance evaluation index, the second performance evaluation index, and the third performance evaluation index.

[0090] It should be understood that the first performance evaluation index, the second performance evaluation index, and the third performance evaluation index in this embodiment can be calculated using the formulas described above, or a calculation model can be pre-trained to calculate the first performance evaluation index, the second performance evaluation index, and the third performance evaluation index.

[0091] In this embodiment, the minimum performance evaluation index is determined from the first performance evaluation index corresponding to the horizontal slice, the second performance evaluation index corresponding to the vertical slice, and the third performance evaluation index corresponding to the composite slice. The slicing method corresponding to the minimum performance evaluation index is used as the slicing method for the final slicing of multiple query tasks, so as to achieve a better slicing effect.

[0092] The overall process of this invention is as follows: Figure 9 As shown, the terminal first submits the query task to the platform. The platform has a target evaluation model and a slicing judgment model to determine whether slicing is required. If slicing is required, the slicing node is determined, and horizontal slicing, vertical slicing, or composite slicing is performed based on the slicing node, and the final slicing method is evaluated. Finally, the slicing subtasks are sent to the processing cluster, and the processing cluster executes the slicing subtasks.

[0093] In some implementations, after slicing the multiple query tasks based on the slice nodes, the sliced ​​query tasks are submitted to the executing server (e.g., HiveServer2 server). The executing server extracts the results of the sliced ​​query tasks and returns the results, so that the results of all query tasks can be integrated to form the final query result.

[0094] Please see Figure 10 , Figure 10 This is a structural diagram of a query task slicing device provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the query task slicing device 1000 includes: The generation module 1001 is used to generate prediction results corresponding to multiple query tasks. The prediction results are used to characterize whether the multiple query tasks need to be sliced. The multiple query tasks are structured query language (SQL) tasks. The first determining module 1002 is used to determine multiple atomic tables corresponding to each query task among the multiple query tasks to be processed when the prediction result characterizes that the multiple query tasks need to be sliced. The information stored in the multiple atomic tables includes the query results corresponding to the respective query tasks. The construction module 1003 is used to construct a query table based on all atomic tables corresponding to the multiple query tasks. The query table is used to store the multiple query tasks and to characterize the correspondence between each query task and the atomic table. The second determining module 1004 is used to determine slice nodes based on the query table; The slicing module 1005 is used to slice the multiple query tasks based on the slicing nodes.

[0095] In one embodiment, the generation module 1001 includes: An extraction unit is used to extract query features corresponding to the multiple query tasks and cluster features of the processing cluster, wherein the processing cluster is a cluster that executes the multiple query tasks, and the cluster features are used to characterize the task queuing status and / or resource usage in the processing cluster. A generation unit is used to generate prediction results based on the query features and the cluster features.

[0096] In one embodiment, the generation unit is specifically used for: The query features and cluster features are input into the target evaluation model, and the first number of query result entries is output based on the target evaluation model. The target evaluation model is used to predict the number of query result entries under different query features and cluster features. The query features, the cluster features, and the number of the first query result entries are input into the slicing judgment model, and the prediction result is output based on the slicing judgment model. The slicing judgment model is used to predict whether the query needs to be sliced ​​under different query features, cluster features, and the number of query result entries.

[0097] In one embodiment, the target evaluation model and the slice judgment model are obtained in the following manner: Obtain multiple sample tasks, sample query features and sample cluster features corresponding to the multiple sample tasks, and sample prediction results corresponding to the multiple sample tasks; Each of the multiple sample tasks is encoded to obtain the encoded data corresponding to each sample task. The encoded data is used to characterize the correspondence between the corresponding sample task and different atomic tables, the connection conditions of different fields included in the corresponding sample task, and at least one of the different fields, different operators, and different values ​​included in the corresponding sample task. The initial evaluation model is trained based on the sample query features, the sample cluster features, and the encoded data corresponding to the multiple sample tasks to obtain the intermediate evaluation model and the number of intermediate entries output by the intermediate evaluation model. The initial evaluation model is used to determine the number of query result entries under different query features and cluster features. The initial judgment model is trained based on the sample query features, the sample cluster features, the encoded data corresponding to the multiple sample tasks, and the number of intermediate entries to obtain the intermediate judgment model and the intermediate results output by the intermediate judgment model. If the intermediate result matches the sample prediction result, the intermediate evaluation model is set as the target evaluation model, and the intermediate judgment model is set as the slice judgment model.

[0098] In one embodiment, the second determining module 1004 includes at least one of the following: The second determining unit is used to determine a first slice node based on a set sliding window and the query table, wherein the first slice node is located between two set sliding windows in the query table; The third determining unit is used to determine the second slice node based on the query table, wherein the second slice node corresponds to the fields of the query task corresponding to different atomic tables; The slicing module 1005 includes at least one of the following: The first slicing unit is used to perform horizontal slicing on the plurality of query tasks based on the first slicing node, wherein the horizontal slicing is used to slice different query tasks; The second slicing unit is used to perform vertical slicing on the multiple query tasks based on the second slicing node. The vertical slicing is used to slice different fields included in a query task.

[0099] In one embodiment, the third determining unit is specifically used for: The association parameters of the multiple atomic tables are determined based on the query table, and the association parameters are used to characterize the association between different atomic tables; At least one second slice node is determined based on the association parameters and the number of query tasks associated with each of the multiple atomic tables.

[0100] In one embodiment, determining at least one second slice node based on the association parameters and the number of query tasks associated with each of the plurality of atomic tables includes: Input the association parameters and the number of query tasks corresponding to each atomic table in the plurality of atomic tables into the node confirmation model, and output at least one second slice node based on the node confirmation model; The node confirmation model is obtained in the following way: Obtain a sample lookup table, multiple sample atom tables corresponding to the sample lookup table, and sample slice nodes corresponding to multiple sample tasks included in the sample lookup table; Based on the multiple sample atomic tables, the sample association parameters of the multiple sample tasks are determined. The sample association parameters are used to characterize the sample atomic tables that correspond to the multiple sample tasks, as well as the association between the multiple sample atomic tables. The sample lookup table is adjusted based on the sample association parameters, and the adjusted sample lookup table includes parameter columns corresponding to the sample association parameters; The initial confirmation model is trained based on the sample slice nodes and the adjusted sample query table to obtain the node confirmation model. The initial confirmation model is used to predict the slice nodes corresponding to different query tasks.

[0101] In one embodiment, the second slicing unit is specifically used for: Based on the at least one second slice node, each query task is divided into at least one first field and multiple second fields. The at least one first field is a field among the multiple initial fields included in each query task that has a corresponding relationship with the at least one second slice node. The multiple second fields are fields other than the at least one first field among the multiple initial fields. The multiple query tasks are vertically sliced ​​based on the at least one first field and the multiple second fields. Each subtask corresponding to each sliced ​​query task includes a first field and at least one second field.

[0102] In one embodiment, the second determining module 1004 includes: The fourth determining unit is used to determine the first slice node based on the set sliding window and the query table, and to determine the second slice node based on the query table; The slicing module 1005 includes: The calculation unit is used to calculate a first performance evaluation index, a second performance evaluation index and a third performance evaluation index. The first performance evaluation index is used to represent the slicing situation based on the first slice node, the second performance evaluation index is used to characterize the slicing situation based on the second slice node, and the third performance evaluation index is used to characterize the slicing situation based on the first slice node and the second slice node. The third slicing unit is used to slice the multiple query tasks based on the slicing nodes corresponding to the target performance evaluation index, wherein the target performance evaluation index is the smallest performance evaluation index among the first performance evaluation index, the second performance evaluation index, and the third performance evaluation index.

[0103] The query task slicing device provided in this embodiment of the invention can implement each process of the above-described query task slicing method, with one-to-one correspondence of technical features and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0104] It should be noted that the query task slicing device in the embodiments of the present invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0105] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the above-described functionality. Figure 1 The various processes of the query task slicing method embodiment shown herein can achieve the same technical effect, and will not be described again here to avoid repetition.

[0106] For details, see Figure 11 As shown, this embodiment of the invention also provides an electronic device, including a bus 1101, a transceiver 1102, an antenna 1103, a bus interface 1104, a processor 1105, and a memory 1106.

[0107] The processor 1105 is used to generate prediction results corresponding to multiple query tasks. The prediction results are used to characterize whether the multiple query tasks need to be sliced. The multiple query tasks are structured query language (SQL) tasks. The processor 1105 is configured to, when the prediction result indicates that the multiple query tasks need to be sliced, determine multiple atomic tables corresponding to each query task among the multiple query tasks to be processed, wherein the information stored in the multiple atomic tables includes the query results corresponding to the respective query tasks. The processor 1105 is further configured to construct a query table based on all atomic tables corresponding to the plurality of query tasks, the query table being used to store the plurality of query tasks and to characterize the correspondence between each query task and an atomic table; The processor 1105 is also configured to determine slice nodes based on the query table; The processor 1105 is also configured to slice the plurality of query tasks based on the slice nodes respectively.

[0108] In one embodiment, generating prediction results corresponding to multiple query tasks includes: Extract the query features corresponding to the multiple query tasks and the cluster features of the processing cluster. The processing cluster is the cluster that executes the multiple query tasks. The cluster features are used to characterize the task queuing status and / or resource usage in the processing cluster. Prediction results are generated based on the query features and the cluster features.

[0109] In one embodiment, generating a prediction result based on the query features and the cluster features includes: The query features and cluster features are input into the target evaluation model, and the first number of query result entries is output based on the target evaluation model. The target evaluation model is used to predict the number of query result entries under different query features and cluster features. The query features, the cluster features, and the number of the first query result entries are input into the slicing judgment model, and the prediction result is output based on the slicing judgment model. The slicing judgment model is used to predict whether the query needs to be sliced ​​under different query features, cluster features, and the number of query result entries.

[0110] In one embodiment, the target evaluation model and the slice judgment model are obtained in the following manner: Obtain multiple sample tasks, sample query features and sample cluster features corresponding to the multiple sample tasks, and sample prediction results corresponding to the multiple sample tasks; Each of the multiple sample tasks is encoded to obtain the encoded data corresponding to each sample task. The encoded data is used to characterize the correspondence between the corresponding sample task and different atomic tables, the connection conditions of different fields included in the corresponding sample task, and at least one of the different fields, different operators, and different values ​​included in the corresponding sample task. The initial evaluation model is trained based on the sample query features, the sample cluster features, and the encoded data corresponding to the multiple sample tasks to obtain the intermediate evaluation model and the number of intermediate entries output by the intermediate evaluation model. The initial evaluation model is used to determine the number of query result entries under different query features and cluster features. The initial judgment model is trained based on the sample query features, the sample cluster features, the encoded data corresponding to the multiple sample tasks, and the number of intermediate entries to obtain the intermediate judgment model and the intermediate results output by the intermediate judgment model. If the intermediate result matches the sample prediction result, the intermediate evaluation model is set as the target evaluation model, and the intermediate judgment model is set as the slice judgment model.

[0111] In one embodiment, determining the slice node based on the query table includes at least one of the following: A first slice node is determined based on the set sliding window and the query table, and the first slice node is located between the two set sliding windows in the query table; The second slice node is determined based on the query table, and the second slice node corresponds to the fields of the query task corresponding to different atomic tables; The step of slicing the multiple query tasks based on the slice nodes includes at least one of the following: The multiple query tasks are horizontally sliced ​​based on the first slice node, and the horizontal slicing is used to slice different query tasks. The multiple query tasks are vertically sliced ​​based on the second slice node, and the vertical slicing is used to slice different fields included in a query task.

[0112] In one embodiment, determining the second slice node based on the query table includes: The association parameters of the multiple atomic tables are determined based on the query table, and the association parameters are used to characterize the association between different atomic tables; At least one second slice node is determined based on the association parameters and the number of query tasks associated with each of the multiple atomic tables.

[0113] In one embodiment, determining at least one second slice node based on the association parameters and the number of query tasks associated with each of the plurality of atomic tables includes: Input the association parameters and the number of query tasks corresponding to each atomic table in the plurality of atomic tables into the node confirmation model, and output at least one second slice node based on the node confirmation model; The node confirmation model is obtained in the following way: Obtain a sample lookup table, multiple sample atom tables corresponding to the sample lookup table, and sample slice nodes corresponding to multiple sample tasks included in the sample lookup table; Based on the multiple sample atomic tables, the sample association parameters of the multiple sample tasks are determined. The sample association parameters are used to characterize the sample atomic tables that correspond to the multiple sample tasks, as well as the association between the multiple sample atomic tables. The sample lookup table is adjusted based on the sample association parameters, and the adjusted sample lookup table includes parameter columns corresponding to the sample association parameters; The initial confirmation model is trained based on the sample slice nodes and the adjusted sample query table to obtain the node confirmation model. The initial confirmation model is used to predict the slice nodes corresponding to different query tasks.

[0114] In one embodiment, the vertical slicing of the plurality of query tasks based on the second slice node includes: Based on the at least one second slice node, each query task is divided into at least one first field and multiple second fields. The at least one first field is a field among the multiple initial fields included in each query task that has a corresponding relationship with the at least one second slice node. The multiple second fields are fields other than the at least one first field among the multiple initial fields. The multiple query tasks are vertically sliced ​​based on the at least one first field and the multiple second fields. Each subtask corresponding to each sliced ​​query task includes a first field and at least one second field.

[0115] In one embodiment, determining the slice node based on the query table includes: The first slice node is determined based on the set sliding window and the query table, and the second slice node is determined based on the query table; The step of slicing the multiple query tasks based on the slice nodes includes: Calculate a first performance evaluation index, a second performance evaluation index, and a third performance evaluation index. The first performance evaluation index is used to represent the slicing situation based on the first slice node, the second performance evaluation index is used to characterize the slicing situation based on the second slice node, and the third performance evaluation index is used to characterize the slicing situation based on both the first slice node and the second slice node. The multiple query tasks are sliced ​​based on the slice nodes corresponding to the target performance evaluation index, wherein the target performance evaluation index is the smallest performance evaluation index among the first performance evaluation index, the second performance evaluation index, and the third performance evaluation index.

[0116] exist Figure 11In this document, a bus architecture (represented by bus 1101) is used. Bus 1101 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 1105 and memory represented by memory 1106. Bus 1101 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 1104 provides an interface between bus 1101 and transceiver 1102. Transceiver 1102 may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 1105 is transmitted over a wireless medium via antenna 1103, which further receives data and transmits it to processor 1105.

[0117] Processor 1105 is responsible for managing bus 1101 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 1106 can be used to store data used by processor 1105 during operation.

[0118] Optionally, the processor 1105 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU).

[0119] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the above-described functions. Figure 1 The various processes corresponding to the query task slicing method embodiments achieve the same technical effect, and will not be described again here to avoid repetition. The computer-readable storage medium mentioned includes, for example, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0120] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The various processes in the corresponding query task slicing method implementation examples can achieve the same technical effect, and will not be described again here to avoid repetition.

[0121] In the embodiments of this invention, the terms "first," "second," etc., are used to distinguish similar object parameters and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected object parameters, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0122] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, etc.) to execute the methods of the various embodiments of this application.

[0124] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A query task slicing method, characterized in that, include: Generate prediction results for multiple query tasks, the prediction results being used to characterize whether the multiple query tasks need to be sliced, the multiple query tasks being structured query language (SQL) tasks; When the prediction result characterization requires slicing the multiple query tasks, multiple atomic tables are determined for each query task among the multiple query tasks to be processed. The information stored in the multiple atomic tables includes the query results corresponding to the respective query tasks. A query table is constructed based on all atomic tables corresponding to the multiple query tasks. The query table is used to store the multiple query tasks and to represent the correspondence between each query task and an atomic table. The slice nodes are determined based on the query table; The multiple query tasks are sliced ​​based on the slice nodes.

2. The method as described in claim 1, characterized in that, The prediction results generated for multiple query tasks include: Extract the query features corresponding to the multiple query tasks and the cluster features of the processing cluster. The processing cluster is the cluster that executes the multiple query tasks. The cluster features are used to characterize the task queuing status and / or resource usage in the processing cluster. Prediction results are generated based on the query features and the cluster features.

3. The method as described in claim 2, characterized in that, The generation of prediction results based on the query features and the cluster features includes: The query features and cluster features are input into the target evaluation model, and the first number of query result entries is output based on the target evaluation model. The target evaluation model is used to predict the number of query result entries under different query features and cluster features. The query features, the cluster features, and the number of the first query result entries are input into the slicing judgment model, and the prediction result is output based on the slicing judgment model. The slicing judgment model is used to predict whether the query needs to be sliced ​​under different query features, cluster features, and the number of query result entries.

4. The method as described in claim 3, characterized in that, The target evaluation model and the slice judgment model are obtained in the following way: Obtain multiple sample tasks, sample query features and sample cluster features corresponding to the multiple sample tasks, and sample prediction results corresponding to the multiple sample tasks; Each of the multiple sample tasks is encoded to obtain the encoded data corresponding to each sample task. The encoded data is used to characterize the correspondence between the corresponding sample task and different atomic tables, the connection conditions of different fields included in the corresponding sample task, and at least one of the different fields, different operators, and different values ​​included in the corresponding sample task. The initial evaluation model is trained based on the sample query features, the sample cluster features, and the encoded data corresponding to the multiple sample tasks to obtain the intermediate evaluation model and the number of intermediate entries output by the intermediate evaluation model. The initial evaluation model is used to determine the number of query result entries under different query features and cluster features. The initial judgment model is trained based on the sample query features, the sample cluster features, the encoded data corresponding to the multiple sample tasks, and the number of intermediate entries to obtain the intermediate judgment model and the intermediate results output by the intermediate judgment model. If the intermediate result matches the sample prediction result, the intermediate evaluation model is set as the target evaluation model, and the intermediate judgment model is set as the slice judgment model.

5. The method according to any one of claims 1 to 4, characterized in that, The determination of slice nodes based on the query table includes at least one of the following: A first slice node is determined based on the set sliding window and the query table, and the first slice node is located between the two set sliding windows in the query table; The second slice node is determined based on the query table, and the second slice node corresponds to the fields of the query task corresponding to different atomic tables; The step of slicing the multiple query tasks based on the slice nodes includes at least one of the following: The multiple query tasks are horizontally sliced ​​based on the first slice node, and the horizontal slicing is used to slice different query tasks. The multiple query tasks are vertically sliced ​​based on the second slice node, and the vertical slicing is used to slice different fields included in a query task.

6. The method as described in claim 5, characterized in that, The step of determining the second slice node based on the query table includes: The association parameters of the multiple atomic tables are determined based on the query table, and the association parameters are used to characterize the association between different atomic tables; At least one second slice node is determined based on the association parameters and the number of query tasks associated with each of the multiple atomic tables.

7. The method as described in claim 6, characterized in that, Determining at least one second slice node based on the association parameters and the number of query tasks associated with each of the multiple atomic tables includes: Input the association parameters and the number of query tasks corresponding to each atomic table in the plurality of atomic tables into the node confirmation model, and output at least one second slice node based on the node confirmation model; The node confirmation model is obtained in the following way: Obtain a sample lookup table, multiple sample atom tables corresponding to the sample lookup table, and sample slice nodes corresponding to multiple sample tasks included in the sample lookup table; Based on the multiple sample atomic tables, the sample association parameters of the multiple sample tasks are determined. The sample association parameters are used to characterize the sample atomic tables that correspond to the multiple sample tasks, as well as the association between the multiple sample atomic tables. The sample lookup table is adjusted based on the sample association parameters, and the adjusted sample lookup table includes parameter columns corresponding to the sample association parameters; The initial confirmation model is trained based on the sample slice nodes and the adjusted sample query table to obtain the node confirmation model. The initial confirmation model is used to predict the slice nodes corresponding to different query tasks.

8. The method as described in claim 5, characterized in that, The vertical slicing of the multiple query tasks based on the second slice node includes: Based on the at least one second slice node, each query task is divided into at least one first field and multiple second fields. The at least one first field is a field among the multiple initial fields included in each query task that has a corresponding relationship with the at least one second slice node. The multiple second fields are fields other than the at least one first field among the multiple initial fields. The multiple query tasks are vertically sliced ​​based on the at least one first field and the multiple second fields. Each subtask corresponding to each sliced ​​query task includes a first field and at least one second field.

9. The method according to any one of claims 1 to 4, characterized in that, The step of determining the slice node based on the query table includes: The first slice node is determined based on the set sliding window and the query table, and the second slice node is determined based on the query table; The step of slicing the multiple query tasks based on the slice nodes includes: Calculate a first performance evaluation index, a second performance evaluation index, and a third performance evaluation index. The first performance evaluation index is used to represent the slicing situation based on the first slice node, the second performance evaluation index is used to characterize the slicing situation based on the second slice node, and the third performance evaluation index is used to characterize the slicing situation based on both the first slice node and the second slice node. The multiple query tasks are sliced ​​based on the slice nodes corresponding to the target performance evaluation index, wherein the target performance evaluation index is the smallest performance evaluation index among the first performance evaluation index, the second performance evaluation index, and the third performance evaluation index.

10. A query task slicing device, characterized in that, include: A generation module is used to generate prediction results for multiple query tasks. The prediction results are used to characterize whether the multiple query tasks need to be sliced. The multiple query tasks are structured query language (SQL) tasks. The first determining module is used to determine multiple atomic tables corresponding to each query task among the multiple query tasks to be processed when the prediction result characterization requires slicing of the multiple query tasks. The information stored in the multiple atomic tables includes the query results corresponding to the respective query tasks. A construction module is used to construct a query table based on all atomic tables corresponding to the multiple query tasks. The query table is used to store the multiple query tasks and to characterize the correspondence between each query task and an atomic table. The second determining module is used to determine the slice node based on the query table; The slicing module is used to slice the multiple query tasks based on the slice nodes.

Images