Data processing method and device

Abstract

The invention discloses a data processing method and device. The method is applied to a coordination node in the distributed database system, the distributed database system further comprises a first data node and at least one second data node, and the first data node is used for performing data interaction with the at least one second data node through a data redistribution operator when data query is executed; the method comprises the following steps: confirming that a first data node has N to-be-executed data redistribution operators, wherein N is a positive integer greater than or equal to 1; when N is larger than a threshold value, the N data redistribution operators are divided into M operator groups, each operator group comprises at least one data redistribution operator in the N data redistribution operators, and M is a positive integer larger than 1; and indicating the first data node to execute the M operator groups in sequence. The method can avoid the problem that the CPU utilization rate is too high due to the fact that the data nodes execute too many data redistribution operators at the same time.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing method and device. Background Art

[0002] A distributed database system usually includes at least one coordinate node (CN) and multiple data nodes (DN). Data nodes are also called data storage nodes and are used to store data. Among them, data nodes in distributed databases mostly adopt a share-nothing architecture, and data is stored in the storage engine of each data node using a hash distribution method.

[0003] Distributed database systems perform data queries by executing structured query language (SQL). The data associated with the query is often distributed on different data nodes, which requires data redistribution. When a data node performs data redistribution, the data node may transmit data to other data nodes through multiple data redistribution operators. In related technologies, a data node executes multiple data redistribution operators at the same time, resulting in excessive utilization of the central processing unit (CPU) of the data node, affecting the execution of other services. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and device, which can avoid problems such as excessive CPU utilization caused by data nodes in a distributed database system executing too many data redistribution operators at the same time.

[0005] In a first aspect, a data processing method is provided, which is applied to a coordination node in a distributed database system, wherein the distributed database system also includes a first data node and at least one second data node, wherein the first data node is used to interact with the at least one second data node through a data redistribution operator when executing a data query; the method includes: confirming that the first data node has N data redistribution operators to be executed, where N is a positive integer greater than or equal to 1; when N is greater than a threshold, dividing the N data redistribution operators into M operator groups, wherein the operator group includes at least one data redistribution operator among the N data redistribution operators, and M is a positive integer greater than 1; instructing the first data node to execute the M operator groups in sequence.

[0006] When there are many data redistribution operators to be executed on a data node, the method groups operators to divide the data redistribution operators into different operator groups. Then, the data node executes the data redistribution operators to be executed serially by serially executing the operator groups, thereby avoiding problems such as excessive CPU utilization caused by the data node executing too many data redistribution operators at the same time.

[0007] In a possible implementation, the data redistribution operator is a thread, and the threshold is determined by the number of available threads of the first data node.

[0008] In this implementation, the threshold is determined by the number of available threads of the data node. Therefore, when the number of available threads of the data node cannot meet the simultaneous execution requirements of the N data redistribution operators to be executed, the N data redistribution operators can be divided into different operator groups, so that the data node can execute the data redistribution operators to be executed serially by serially executing the operator groups.

[0009] In a possible implementation, the N data redistribution operators are divided into M operator groups, including: grouping the N data redistribution operators based on the number of materialized operators in the first data node to obtain the M operator groups. The materialized operators are used to cache the execution results of the lower layer operators.

[0010] In this implementation, operators are grouped based on the materialized operator, so that the data redistribution operator in the operator group can be used as the lower-level operator of the materialized operator, and the intermediate results can be cached in the materialized operator to wait for the call of the merge operator after all operator groups are executed.

[0011] In one possible implementation, N data redistribution operators are divided into M operator groups, including: based on the number of materialized operators in the first data node, the N data redistribution operators are grouped to obtain K operator groups, where K is a positive integer greater than 1; confirming that the number of data redistribution operators included in at least one operator group of the K operator groups is greater than a threshold; and grouping the data redistribution operators in at least one operator group.

[0012] In this implementation, if the number of data redistribution operators in the operator group obtained by grouping operators based on the materialized operator still exceeds the threshold, the operator group can be further grouped until the number of data redistribution operators in the operator group is less than or equal to the threshold.

[0013] In a possible implementation, the operator groups except the last executed operator group among the M operator groups include at least one materialized operator.

[0014] In this implementation, the operator group that is not executed last includes a materialization operator to cache data output by the operators in the operator group so that the data can wait for the call of the merge operator after all operator groups are executed.

[0015] In a second aspect, a data processing device is provided, which is configured in a coordination node in a distributed database system, and the distributed database system also includes a first data node and at least one second data node, wherein the first data node is used to interact with the at least one second data node through a data redistribution operator when executing a data query; the device includes: a confirmation module, used to confirm that the first data node has N data redistribution operators to be executed, N is a positive integer greater than or equal to 1; a division module, used to divide the N data redistribution operators into M operator groups when N is greater than a threshold, wherein the operator group includes at least one data redistribution operator among the N data redistribution operators, and M is a positive integer greater than 1; an indication module, used to instruct the first data node to execute the M operator groups in sequence.

[0016] In a possible implementation, the data redistribution operator is a thread, and the threshold is determined by the number of available threads of the first data node.

[0017] In a possible implementation, the partitioning module is used to: group the N data redistribution operators based on the number of materialized operators in the first data node to obtain M operator groups.

[0018] In one possible implementation, the partitioning module is used to: group N data redistribution operators based on the number of materialized operators in the first data node to obtain K operator groups, where K is a positive integer greater than 1; confirm that the number of data redistribution operators included in at least one of the K operator groups is greater than a threshold; and group the data redistribution operators in at least one operator group.

[0019] In a possible implementation, the operator groups except the last executed operator group among the M operator groups include at least one materialized operator.

[0020] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method provided in the first aspect.

[0021] In a fourth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0022] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computer device cluster, the computer device cluster executes the method provided in the first aspect.

[0023] The beneficial effects of the second to fifth aspects can be referred to the above introduction to the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the structure of a distributed database system provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of a distributed database system provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of a distributed database system provided in an embodiment of the present application;

[0027] Figure 4 A flowchart of a data processing method provided in an embodiment of the present application;

[0028] Figure 5 A flowchart of a data processing method provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of operator grouping provided in an embodiment of the present application;

[0030] Fig. 7A A schematic diagram of operator grouping provided in an embodiment of the present application;

[0031] Figure 7B A schematic diagram of operator grouping provided in an embodiment of the present application;

[0032] Fig. 8A A schematic diagram of operator grouping provided in an embodiment of the present application;

[0033] Figure 8B A schematic diagram of operator grouping provided in an embodiment of the present application;

[0034] Figure 8C A schematic diagram of an operator group execution provided in an embodiment of the present application;

[0035] Fig. 9 A schematic diagram of operator grouping provided in an embodiment of the present application;

[0036] Fig.10 A schematic diagram of operator grouping provided in an embodiment of the present application;

[0037] Fig.11A structural schematic diagram of a data processing device is provided for an embodiment of the present application;

[0038] Fig.12 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0039] Fig.13 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0040] Fig.14 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The scheme provided by the embodiment of the present application will be described below in conjunction with the accompanying drawings. In the embodiment of the present application, "plurality" refers to two or more than two. "First", "second", etc. are only used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.

[0042] To facilitate understanding of the solutions provided by the embodiments of the present application, some technical terms that may be involved in the embodiments of the present application are first introduced before the solutions provided by the embodiments of the present application are introduced.

[0043] Shared nothing architecture: A distributed computing architecture in which each node (such as a data node in a distributed data system using this architecture) is independent and has its own nodes, and there is no single point of competition between different nodes in the system. More specifically, the nodes do not share resources such as CPU, memory, and hard disk, and each node manages its own data.

[0044] Online analytical processing (OLAP): A distributed database system, which is another type of database system relative to online transaction processing (OLTP). The characteristics of OLAP are read-more and write-more. Data warehouse service (DWS) and reports belong to OLAP.

[0045] Data Warehouse Service: An enterprise system for analyzing and reporting on structured and semi-structured data from multiple sources, providing an out-of-the-box, scalable, and fully managed analytical database service.

[0046] Coordinator node (CN): A node in a distributed database system that is mainly responsible for executing tasks such as plan generation and resource management. A coordinator node can communicate with any data node in the distributed database system.

[0047] Data node (DN): A node in a distributed database system that is primarily responsible for data storage and computing.

[0048] Data redistribution operator: also known as stream thread, is a thread used for network data exchange between nodes in a distributed database system.

[0049] Material operator: An execution operator in a node in a distributed database, whose main function is to cache the execution results of the lower-level operators.

[0050] Append operator: An execution operator in a distributed database system, mainly used to process union all statements, concatenating the results of each branch of the union all statement, and executing each branch sequentially.

[0051] In a distributed data management system, data is stored with a distribution key. When performing a data query, the user issues a query statement (such as an SQL statement) to the coordination node of the distributed data system through the client of the distributed database system. The query optimization module of the coordination node receives the query statement, parses the query statement, generates a data query plan, and distributes the generated data query plan to each data node. After receiving the data query plan, each data node performs a data query.

[0052] If the distribution keys of the associated data tables are the same, the associated data can be associated within the same node, and the results of the association operation are sent to the coordination node, which can improve the efficiency of data query. If the distribution keys of the associated data tables are different, that is, the conditions for associating two or more data tables are inconsistent with their distribution columns (that is, the data to be associated are not on the same data node. In this case, at least one of the data tables needs to be redistributed, that is, the data stored in the at least one data table on different data nodes is aggregated to the same data node. Among them, the conditions for associating the data tables are also called the association keys of the data tables.

[0053] For example, the distribution key of table A1 is aa, the distribution key of table A2 is aa, and the distribution key of table A3 is bb. When the data in table A1 and table A2 are equally associated with A1.aa=A2.aa, the association keys of table A1 and table A2 are both aa, that is, table A1 and table A2 are both associated on the distribution key aa, so table A1 and table A2 can be associated in the same data node. When table A2 and table A3 are equally associated with A2.bb=A3.aa, the association key of table A2 is bb, and the association key of table A3 is aa. Since the distribution keys of table A2 and table A3 are different from their association keys, the data node will redistribute table A3 with the association key aa, and at the same time redistribute table A2 with the association key bb, so as to achieve the purpose of completing the association in the same data node.

[0054] Data redistribution requires data distribution between data nodes through the network. When a data node performs data query, if the aforementioned data redistribution is required, the data node starts the data redistribution operator to process the data communication between data nodes. Each data redistribution operator is a separate thread, which is used to calculate the hash value of the data to be sent according to the associated key, and then send the data to be sent to the corresponding data node based on the calculation result.

[0055] In one scenario, a query statement contains multiple union all clauses. The union all clause is also called a branch. In the related art, when a data node executes data queries for such query statements, the data redistribution operators in multiple branches are started simultaneously to perform parallel queries. The append operator at the upper level of the query statement is executed serially to calculate the data of each branch.

[0056] Since the amount of data being queried is not large, the execution time of each branch is within a few seconds. However, when multiple branches are executed concurrently, the data redistribution operator will be run simultaneously on the data node, which will lead to excessive CPU usage of the data node and affect the execution of other query statements. In addition, since the append operator schedules the data of each branch serially, this causes the data of many branches to be in a waiting state (that is, the execution results of many data redistribution operators are in a waiting state), resulting in a waste of thread resources.

[0057] The embodiment of the present application provides a data processing method. When a data node has a large number of redistribution operators corresponding to the same query statement, the method divides the redistribution operators into different operator groups and executes each operator group in series, thereby avoiding problems such as excessive CPU usage caused by the simultaneous operation of multiple data redistribution operators.

[0058] Next, the data processing method provided in the embodiment of the present application is introduced.

[0059] Figure 1 FIG. 1 shows a distributed database system 100. Figure 1 As shown, the distributed database system 100 may include at least one client 110, at least one coordination node 120, and multiple data nodes, for example, data node 131, data node 132, and the like.

[0060] The user can send a query statement, such as an SQL statement, to the coordination node 120 through the client 110. The coordination node 120 can generate a query plan based on the query statement and send the query plan to the corresponding data node. The data node can execute the data query plan to perform data query. Among them, some data query plans require data redistribution. When executing the data query plan, the data node (such as data node 131) interacts with other data nodes (such as data node 132) through the data redistribution operator to perform data redistribution. Among them, the data node executing the data query plan is also called the data node executing the data query.

[0061] In an embodiment of the present application, taking data node 131 as an example of a data node that executes a data query plan, the coordination node 120 can also obtain the number of data redistribution operators to be executed of data node 131 based on the query statement. The data redistribution operator to be executed is a data redistribution operator that needs to be executed by data node 131 for data redistribution, that is, when data node 131 performs data redistribution, it interacts with other data nodes (such as data node 132) through the data redistribution operator to be executed. If the data redistribution operator to be executed exceeds the preset threshold value Y1, the coordination node 120 groups the data redistribution operators to be executed so that the data node 131 executes the data redistribution operator serially, thereby avoiding problems such as excessive CPU usage caused by the simultaneous execution of all data redistribution operators to be executed.

[0062] In some embodiments, the data redistribution operator is a thread, and the threshold value Y1 is determined by the number of available threads of the data node 131. The number of available threads refers to the number of threads remaining and not yet used in the data node. The number of threads that a data node can run is limited. The number of threads that can be run on a data node is subtracted from the number of threads that the data node has run to obtain the number of available threads on the data node. In one example, the threshold value Y1 may be equal to the number of available threads of the data node 131. In another example, the threshold value Y1 is fifty percent to ninety percent of the number of available threads of the data node 131. In some embodiments, the threshold value Y1 may be set by a user or other personnel (e.g., an operation and maintenance personnel of a distributed database system). For example, the threshold value Y1 may be 10, 100, etc. The embodiment of the present application does not limit the specific value of the threshold value Y1.

[0063] In some embodiments, Figure 2 As shown, the coordination node 120 includes a query optimization module. The query optimization module can perform operations such as syntax parsing, semantic parsing, logic optimization, physical optimization, path generation, and plan generation on the query statement to obtain a data query plan. By parsing the data query plan, the number of data redistribution operators to be executed of the data node corresponding to the data query plan can be determined. The data node corresponding to the data query plan refers to the data node that executes the data query plan, that is, the coordination node 120 sends the data query plan to the data node corresponding to the data query plan, and then the data node can execute the data query plan.

[0064] like Figure 2 As shown, when the number of data redistribution operators to be executed on a data node is greater than a threshold value Y1, the query optimization module performs operator grouping. Taking data node 131 as a data node corresponding to a data query plan as an example, if the number of data redistribution operators to be executed on data node 131 is greater than the threshold value Y1, the query optimization module performs operator grouping on the data redistribution operators to be executed on data node 131. The grouping result is included in the data query plan to update the data query plan.

[0065] The coordination node 120 may send the updated data query plan to the data node, such as the data node 131. When executing the data query plan, the data node 131 sequentially executes the operator groups obtained by the operator grouping. For example, the query optimization module divides the data redistribution operator to be executed of the data node 131 into the operator group B1 and the operator group B2. The data node 131 sequentially executes the operator group B1 and the operator group B2. That is, the operator group B1 is executed first, and when the operator group B1 is executed, the operator group B2 is executed.

[0066] In some embodiments, Figure 3 As shown, the coordination node 120 may include an operator grouping module. The operator grouping module is used to group operators when the number of data redistribution operators to be executed by the data nodes is greater than a threshold value Y1. The details are as follows.

[0067] In step 301, the query optimization module may send the data query plan to the operator grouping module. In step 302, the operator grouping module determines the number of data redistribution operators to be executed of the data node corresponding to the data query plan based on the data query plan. Taking data node 131 as a data node corresponding to the data query plan as an example, if the number of data redistribution operators to be executed of data node 131 is greater than the threshold value Y1, in step 303, the operator grouping module performs operator grouping on the data redistribution operators to be executed of data node 131. If the number of data redistribution operators in the operator group obtained by the grouping result of step 303 is still greater than the threshold value Y1, then in step 304, the data redistribution operators in the operator group are recursively grouped, that is, the data redistribution operators in the operator group are grouped again until the number of data redistribution operators in each operator group is less than or equal to the threshold value Y1. After steps 303 and 304, in step 305, the operator grouping module obtains the grouping result. The grouping result includes two or more operator groups, such as operator group B1, operator group B2, etc. The number of data redistribution operators in each operator group is less than or equal to a threshold value Y1.

[0068] Then, in step 306, the operator grouping module sends the grouping result to the query optimization module. In step 307, the query optimization module can send the grouping result to the data node 131. When performing data query, the data node 131 executes the operator groups in the grouping result in sequence. For example, the operator group B1 is executed first, and when the operator group B1 is executed, the operator group B2 is executed.

[0069] The above examples introduce the distributed database system provided by the embodiment of the present application. Next, in conjunction with the distributed database system, the data processing method provided by the embodiment of the present application is introduced.

[0070] The method may be executed by the coordination node 120 in the distributed database system. The first data node in the distributed database is used to perform data interaction with at least one second data node through a data redistribution operator when executing a data query. The first data node may be data node 131, and one of the at least one second data node may be data node 132. Figure 4 As shown, the method includes the following steps.

[0071] Step 401: confirm that the first data node has N data redistribution operators to be executed, where N is a positive integer greater than or equal to 1.

[0072] The coordination node 120 can obtain the number of data redistribution operators to be executed for each data node corresponding to the data query plan through the data query plan. The data query plan corresponds to a first data node and at least one second data node. The data redistribution operator to be executed refers to the data redistribution operator to be executed when the data node corresponding to the data query plan executes the data query plan to perform data query. In the following description, the first data node is taken as an example for introduction.

[0073] In step 401, the coordination node 130 obtains, according to the data query plan, that the first data node has N data redistribution operators to be executed. The first data node is a data node corresponding to the data query plan, and the N data redistribution operators to be executed are all data redistribution operators executed when the first data node executes the data query plan.

[0074] Step 402, when N is greater than a threshold value Y1, the N data redistribution operators are divided into M operator groups, wherein the operator group includes at least one data redistribution operator among the N data redistribution operators, and M is a positive integer greater than 1. That is, when the number of data redistribution operators to be executed by the first data node when executing the data query plan for data query is greater than the threshold value Y1, the data redistribution operators to be executed by the first data node when executing the data query plan for data query are grouped to obtain a plurality of operator groups. The threshold value Y1 can be referred to the above description and will not be repeated here.

[0075] In some embodiments, grouping the data redistribution operators to be executed by the first data node when executing the data query plan to perform data query can be achieved by grouping the query tree corresponding to the first data node. That is, when the number of data redistribution operators to be executed by the first data node when executing the data query plan to perform data query is greater than the threshold value Y1, the query tree corresponding to the first data node is replanned for grouping. The query tree corresponding to the first data node here is the operator to be executed by the first data node when executing the data query plan (for example, a data redistribution operator, a join operator, an append operator, a materialization operator, etc.).

[0076] In some embodiments, grouping can be performed based on materialized operators in the query tree. When grouping the query tree, the query tree is traversed from the root to the top to determine whether grouping is possible, that is, whether grouping is possible is determined from the root.

[0077] See also Figure 5Step 402 includes steps 501 and 502, and it is determined whether N is greater than a threshold value Y1. If N is greater than the threshold value Y1, step 502 is executed to perform grouping based on the existing materialized operators in the query tree.

[0078] The grouping based on the existing materialized operators in the query tree can be divided into the following cases.

[0079] Case 1: If the query tree has one or more materialized operators, each materialized operator and the operators under the materialized operator are divided into an operator group, and the merge operator and the operators under the merge operator except the materialized operator are divided into an operator group. In this way, one or more operator groups can be obtained. In one example, refer to Figure 6 , the query tree can be set to include operator C1, operator C2, operator C3, operator C4, operator C5, operator C6, operator C7, operator C8, and operator C9. Among them, C2 can be a merge operator, and C9 is a materialized operator. In the query tree, the operator C2 connected to operator C9 (i.e., the operator under operator C9) and operator C9 can be divided into an operator group B1, and other operators under operator C2 and operator C2 can be divided into another operator group B2. Among them, each operator group can include at least one data redistribution operator in the query tree.

[0080] In the embodiment of the present application, the join operator refers to a join operator or an append operator. The operator under operator X is an operator that outputs data to operator X.

[0081] Case 2: If there is no materialized operator in the query tree, add one or more materialized operators at appropriate locations in the query tree. Then, divide each materialized operator and the operators under the materialized operator into an operator group, so that one or more operator groups can be obtained. In an example, see Fig. 7A , the query tree can be set to include operator C1, operator C2, operator C3, operator C4, operator C5, operator C6, operator C7, and operator C8. Among them, C2 can be a merge operator. Operator C1, operator C2, operator C3, operator C4, operator C5, operator C6, operator C7, and operator C8 are all non-materialized operators. In this case, operator C9, which is used as a materialized operator, can be added under operator C2 to obtain the following: Figure 7B The query tree is shown in Figure 1. Then, Figure 7B In the query tree shown, the connection to operator C9 (i.e., the operator under operator C9) and operator C9 under operator C2 are divided into an operator group B1, and other operators under operator C2 are divided into another operator group B2. Each operator group may include at least one data redistribution operator in the query tree.

[0082] In this way, through step 502, K operator groups can be obtained, wherein each operator group includes at least one data redistribution operator in the query tree. K is an integer greater than 1.

[0083] See also Figure 5 , step 402 also includes step 503, determining whether the number of data redistribution operators in each of the K operator groups obtained in step 502 is greater than a threshold value Y1.

[0084] If the number of data redistribution operators in each of the K operator groups obtained in step 502 is less than or equal to the threshold Y1, the operator grouping is terminated and the K operator groups obtained in step 502 are used as the above-mentioned M operator groups, that is, M equals K.

[0085] If the number of data redistribution operators in at least one of the K operator groups obtained in step 502 is greater than the threshold value Y1, step 504 is executed to further group the data redistribution operators in the at least one operator group until the number of data redistribution operators in each of the divided operator groups is less than or equal to the threshold value Y1. In step 504, a materialized operator may be added to the at least one operator group, and then the materialized operators are further grouped.

[0086] In some embodiments, taking operator group B2 in the at least one operator group as an example, a materialized operator is added at an appropriate position in operator group B2, and then the materialized operator and the operators under the materialized operator are divided into a new operator group. In this way, a new operator group can be split from operator group B2, and operator group B2 can be split into two operator groups. One of the two operator groups can still be called operator group B2, and the other operator group is called operator group B3.

[0087] In one example, see Fig. 8A , the operator group B2 may be set to include operator C2, operator C3, operator C5, operator C6, operator C11, operator C12, and operator C13. Among them, any one or more of operator C3, operator C5, and operator C6 may be a data redistribution operator, and any one or more of operator C11, operator C12, and operator C13 may be a data redistribution operator. Figure 8B , a materialized operator C10 can be added between operator C5 and operator C11. Then, the materialized operator C10, operator C11, operator C12, and operator C13 are divided into operator group B3, and operator C2, operator C3, operator C5, and operator C6 form a new operator group B2.

[0088] In this way, through the above method, the N data redistribution operators to be executed by the first data node can be divided into M operator groups, each operator group includes at least one data redistribution operator among the N data redistribution operators, and the number of data redistribution operators in each operator group is less than or equal to the threshold value Y1. In addition, the M operator groups may include an operator group that does not include a materialization operator. The operator group that does not include a materialization operator includes a merging operator. The merging operator is used to perform calculations based on the data output by other operators in the operator group where the merging operator is located and the data stored by the materialization operators in other operator groups. When the M operator groups are executed in sequence, the operator group that does not include a materialization operator is executed last.

[0089] Back to Figure 4 In step 403, the coordination node 120 instructs the first data node to execute the M operator groups in sequence to ensure the serial execution of the M operator groups.

[0090] The coordination node 120 may send the grouping result of step 402 to the first data node, where the grouping result indicates the M operator groups, so that the first data node executes the M operator groups in sequence. In one example, the M operator groups may be set as Figure 8B The first data node can be configured as follows: Figure 8C The execution order shown is to execute operator group B1, operator group B2 and operator group B3, that is, operator group B3 is executed first, then operator group B1, and finally operator group B2.

[0091] The first data node executes the M operator groups serially, that is, only the operators in one operator group of the M operator groups are executed at the same time, and after all the operators in one operator group are executed, the operators in the next operator group are executed. In this way, the serial execution of the data redistribution operators in different operator groups is guaranteed, avoiding problems such as excessive CPU utilization of the first data node.

[0092] In addition, after completing the data query, the first data node may send the query result to the client 110 .

[0093] The above takes the first data node as an example to introduce operator grouping and execution. Referring to the above introduction, the coordination node 120 can group operators for other data nodes corresponding to the data query plan or query statement and instruct the data nodes to execute the operator groups in sequence, which will not be repeated here.

[0094] Next, the data processing method provided in the embodiment of the present application is introduced by way of examples in combination with different execution operators.

[0095] In some embodiments, see Fig. 9, the execution operator is specifically a join operator. In one example, the join operator can be specifically a hash join operator. The data tables to be queried can be set to include data table t1, data table t2, and data table 3, and the association keys and distribution keys of data table t1, data table t2, and data table 3 are all inconsistent. In this case, the coordination node 120 can generate Fig. 9 The query tree shown. Figure 5 In step 502, based on the materialized operator, the operators corresponding to data table t1 and data table t2 (i.e., the execution flow of data table t1 and data table t2) are divided into one operator group, and the operators corresponding to data table t3 are divided into operator group B3. Then, after Figure 5 In step 503 and step 504, the operators corresponding to data table t1 are divided into operator group B2, and the operators corresponding to data table t2 are divided into operator group B1. Operator group B2 and operator group B1 both include materialized operators for storing intermediate calculation results. Fig. 9 When the query tree is shown, operator group B1, operator group B2 and operator group B3 can be executed in sequence. This makes the number of data redistribution operators executed simultaneously 1, reducing the CPU utilization of the data node.

[0096] In some embodiments, see Fig.10 , the execution operator is specifically the append operator, which is used to merge the query results of subquery G1, subquery G2 and subquery G3. In the prior art, multiple subqueries are executed in parallel, and the Append operator calls the query results of each subquery in series, which causes the query results of the subqueries to be in a waiting state, and the parallel execution of multiple subqueries causes problems such as excessive CPU utilization of data nodes. Through the data processing method provided by the embodiment of the present application, operators in different subqueries can be divided into different operator groups (for example, the operator of subquery G1 is divided into operator group B1, the operator of subquery G2 is divided into operator group B2, and the operator of subquery G3 is divided into operator group B3), and data nodes execute different operator groups in series, that is, only one subquery is executed at the same time, thereby avoiding problems such as excessive CPU utilization caused by too many operators executed at the same time.

[0097] Furthermore, if there are too many data redistribution operators in a subquery, the data processing method provided in the embodiment of the present application can also group the operators in the subquery. Fig.10 As shown, some operators in subquery G1 can be divided into operator group B1, and other operators can be divided into operator group B2. Then, the data nodes execute different operator groups in series to avoid problems such as excessive CPU utilization caused by too many operators being executed at the same time.

[0098] The embodiment of the present application also provides a data processing device 1100. The device 1100 is configured in a coordination node (such as the above-mentioned coordination node 120) in a distributed database system, and the distributed database system also includes a first data node (such as the above-mentioned data node 131) and at least one second data node (such as the above-mentioned data node 132), wherein the first data node is used to perform data interaction with the at least one second data node through a data redistribution operator when executing a data query. Fig.11 As shown, the device 1100 includes:

[0099] A confirmation module 1110 is used to confirm that the first data node has N data redistribution operators to be executed, where N is a positive integer greater than or equal to 1;

[0100] A partitioning module 1120, configured to, when N is greater than a threshold, partition the N data redistribution operators into M operator groups, wherein the operator group includes at least one data redistribution operator among the N data redistribution operators, and M is a positive integer greater than 1;

[0101] The instruction module 1130 is used to instruct the first data node to execute the M operator groups in sequence.

[0102] In some embodiments, the data redistribution operator is a thread, and the threshold is determined by the number of available threads of the first data node.

[0103] In some embodiments, the partitioning module 1120 is used to: group the N data redistribution operators based on the number of materialized operators in the first data node to obtain the M operator groups.

[0104] In some embodiments, the partitioning module 1120 is used to: group the N data redistribution operators based on the number of materialized operators in the first data node to obtain K operator groups, where K is a positive integer greater than 1; confirm that the number of data redistribution operators included in at least one of the K operator groups is greater than the threshold; and group the data redistribution operators in the at least one operator group.

[0105] In some embodiments, the operator groups other than the last executed operator group among the M operator groups include at least one materialized operator.

[0106] Among them, the confirmation module 1110, the division module 1120 and the indication module 1130 can all be implemented by software, or can be implemented by hardware. Exemplarily, the following takes the confirmation module 1110 as an example to introduce the implementation of the confirmation module 1110. Similarly, the implementation of the division module 1120 and the indication module 1130 can refer to the implementation of the confirmation module 1110.

[0107] As an example of a software functional unit, the confirmation module 1110 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the confirmation module 1110 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0108] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0109] As an example of a hardware functional unit, the confirmation module 1110 may include at least one computing device, such as a server, etc. Alternatively, the confirmation module 1110 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0110] The multiple computing devices included in the confirmation module 1110 can be distributed in the same region or in different regions. The multiple computing devices included in the confirmation module 1110 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the confirmation module 1110 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0111] It should be noted that, in other embodiments, the confirmation module 1110 may be used to execute Figure 4 In any step of the method shown, the partitioning module 1120 can be used to perform Figure 4 In any step of the method shown, the instruction module 1130 can be used to perform Figure 4 The steps implemented by the confirmation module 1110, the division module 1120 and the indication module 1130 can be specified as needed, and the steps implemented by the confirmation module 1110, the division module 1120 and the indication module 1130 can be specified as needed. Figure 4 The different steps in the method shown are used to implement the overall functions of the apparatus 1100 .

[0112] The present application also provides a computing device 1200. Fig.12 As shown, the computing device 1200 includes: a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate through the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1200.

[0113] The bus 1202 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 The bus 1202 may include a path for transmitting information between various components of the computing device 1200 (eg, the memory 1206, the processor 1204, and the communication interface 1208).

[0114] The processor 1204 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0115] The memory 1206 may include a volatile memory, such as a random access memory (RAM). The memory 1206 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0116] The memory 1206 stores executable program codes, and the processor 1204 executes the executable program codes to respectively implement the functions of the aforementioned confirmation module 1110, the division module 1120, and the indication module 1130, thereby implementing Figure 4 That is, the memory 1206 stores the method for executing Figure 4 Instructions for the method shown.

[0117] The communication interface 1208 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1200 and other devices or communication networks.

[0118] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0119] like Fig.13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 in one or more computing devices 1200 in the computing device cluster may store the same Figure 4 Instructions for the method shown.

[0120] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store a program for executing Figure 4In other words, the combination of one or more computing devices 1200 can jointly execute instructions for executing Figure 4 Instructions for the method shown.

[0121] It should be noted that the memory 1206 in different computing devices 1200 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the apparatus 1100. That is, the instructions stored in the memory 1206 in different computing devices 1200 may implement the functions of one or more modules of the confirmation module 1110, the division module 1120 and the indication module 1130.

[0122] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.14 A possible implementation is shown. Fig.14 As shown, two computing devices 1200A and 1200B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1206 in the computing device 1200A stores instructions for executing the functions of the confirmation module 1110. At the same time, the memory 1206 in the computing device 1200B stores instructions for executing the functions of the division module 1120 and the indication module 1130.

[0123] It should be understood that Fig.14 The functions of the computing device 1200A shown in FIG. 1200A may also be completed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B may also be completed by multiple computing devices 1200.

[0124] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.13 and Fig.14 The difference is that the memory 1206 in one or more computing devices 1200 in the computing device cluster may store the same memory for executing Figure 4 Instructions for the method shown.

[0125] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store a program for executing Figure 4 In other words, the combination of one or more computing devices 1200 can jointly execute instructions for executing Figure 4 Instructions for the method shown.

[0126] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes Figure 4 The method shown.

[0127] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a host migration device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute Figure 4 The method shown.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: A coordination node applied to a distributed database system, the distributed database system further comprising a first data node and at least one second data node, wherein the first data node is used to perform data interaction with the at least one second data node through a data redistribution operator when executing a data query; the method comprising: Confirming that the first data node has N data redistribution operators to be executed, where N is a positive integer greater than or equal to 1; When N is greater than a threshold, the N data redistribution operators are divided into M operator groups, wherein the operator group includes at least one data redistribution operator among the N data redistribution operators, and M is a positive integer greater than 1; Instruct the first data node to execute the M operator groups in sequence.

2. The method according to claim 1, characterized in that The data redistribution operator is a thread, and the threshold is determined by the number of available threads of the first data node.

3. The method according to claim 1 or 2, characterized in that: The dividing the N data redistribution operators into M operator groups includes: The N data redistribution operators are grouped based on the number of materialized operators in the first data node to obtain the M operator groups.

4. The method according to claim 1 or 2, characterized in that: The dividing the N data redistribution operators into M operator groups includes: Based on the number of materialized operators in the first data node, the N data redistribution operators are grouped to obtain K operator groups, where K is a positive integer greater than 1; Confirming that the number of data redistribution operators included in at least one of the K operator groups is greater than the threshold; The data redistribution operators in the at least one operator group are grouped.

5. The method according to any one of claims 1 to 4, characterized in that The operator groups except the last executed operator group among the M operator groups include at least one materialized operator.

6. A data processing device, characterized in that: A coordination node configured in a distributed database system, the distributed database system further comprising a first data node and at least one second data node, wherein the first data node is used to perform data interaction with the at least one second data node through a data redistribution operator when executing a data query; the device comprises: A confirmation module, used to confirm that the first data node has N data redistribution operators to be executed, where N is a positive integer greater than or equal to 1; a partitioning module, configured to, when N is greater than a threshold, partition the N data redistribution operators into M operator groups, wherein the operator group includes at least one data redistribution operator among the N data redistribution operators, and M is a positive integer greater than 1; The instruction module is used to instruct the first data node to execute the M operator groups in sequence.

7. The device according to claim 6, characterized in that The data redistribution operator is a thread, and the threshold is determined by the number of available threads of the first data node.

8. The device according to claim 6 or 7, characterized in that The partitioning module is used for: The N data redistribution operators are grouped based on the number of materialized operators in the first data node to obtain the M operator groups.

9. The device according to claim 6 or 7, characterized in that The partitioning module is used for: Based on the number of materialized operators in the first data node, the N data redistribution operators are grouped to obtain K operator groups, where K is a positive integer greater than 1; Confirming that the number of data redistribution operators included in at least one of the K operator groups is greater than the threshold; The data redistribution operators in the at least one operator group are grouped.

10. The device according to any one of claims 6 to 9, characterized in that: The operator groups except the last executed operator group among the M operator groups include at least one materialized operator.

11. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 5.

13. A computer program product comprising instructions, characterized in that When the instructions are executed by a computer device cluster, the computer device cluster executes the method according to any one of claims 1 to 5.

Images