Data redistribution method and database

Abstract

A data redistribution method, applied to a database comprising a coordinator node, a plurality of computing nodes, and at least one storage node. The method comprises: a coordinator node determines that the optimal execution plan generated by the coordinator node comprises a redistribution plan, the redistribution plan being used for redistributing the first data stored in a first storage node in a database; the coordinator node transmits the redistribution plan to the first storage node; and on the basis of the redistribution plan, the first storage node transmits the first data to at least one of a plurality of computing nodes in the database. Therefore, when a redistribution plan exists in an optimal execution plan, data transmission is subjected to from pull by a destination end to active push by a source end, so that the network transmission hop count can be reduced, and the amount of data transmission in the network is reduced, reducing the consumption of resources such as network resources and computing resources, and improving the data query performance.

Description

Data redistribution method and database

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on November 22, 2023, with application number 202311574829.8 and application name “A Data Redistribution Method and Database”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of information technology (IT), and in particular to a data redistribution method and a database. Background Art

[0003] As databases move toward cloud computing, database architectures are evolving toward a storage-computing separation, separating the storage layer from the computing layer. In this storage-computing separation, storage and computing resources are independently requested and released on demand. However, retrieving data from the storage layer often requires extensive network interaction, resulting in excessive network traffic and reduced data query performance. Summary of the Invention

[0004] The present application provides a data redistribution method, a database, a computer storage medium, and a computer product, which can reduce network transmission volume during data query and improve data query performance.

[0005] In a first aspect, the present application provides a data redistribution method, applied to a database, the database comprising: a coordinator node, multiple computing nodes, and at least one storage node. The method comprises: the coordinator node determining that an optimal execution plan generated by the coordinator node includes a redistribution plan, the redistribution plan being used to redistribute first data, wherein the first data is stored in a first storage node among at least one storage node; the coordinator node transmitting the redistribution plan to the first storage node; and the first storage node transmitting the first data to at least one computing node among the multiple computing nodes based on the redistribution plan.

[0006] In this way, when there is a redistribution plan in the optimal execution plan, the data transmission is changed from being pulled from the destination end to being actively pushed from the source end, which can reduce the number of network transmission hops, reduce the amount of data transmitted in the network, and reduce the consumption of network and computing resources, thereby improving data query performance.

[0007] In one possible implementation, the first storage node transmits the first data to at least one computing node from among the plurality of computing nodes based on a redistribution plan. This specifically includes: the first storage node filters out the first data based on the redistribution plan; the first storage node filters out at least one computing node from among the plurality of computing nodes based on the first data; and the first storage node transmits the first data to the filtered computing node. In this way, the storage node can filter out the computing node that needs to process the data and transmit the data to the computing node, allowing the computing node to process the data.

[0008] In one possible implementation, before the first storage node transmits the first data to at least one of the plurality of computing nodes, the method further includes: establishing a data transmission channel between the first storage node and each of the at least one of the plurality of computing nodes. In this way, data can be transmitted between the storage node and the computing node via the data transmission channel.

[0009] In one possible implementation, the coordinator node determines that the optimal execution plan generated by the coordinator node includes a redistribution plan, specifically including: the coordinator node determines that the aggregation key included in the query statement is inconsistent with the distribution key used when storing data in the database; or the coordinator node determines that the join key included in the query statement is inconsistent with the distribution key used when storing data in the database.

[0010] In a second aspect, the present application provides a database comprising: a coordinator node, multiple computing nodes, and at least one storage node. The coordinator node is configured to determine whether an optimal execution plan generated by the coordinator node includes a redistribution plan, wherein the redistribution plan is configured to redistribute first data, wherein the first data is stored in a first storage node among the at least one storage node. The coordinator node is further configured to transmit the redistribution plan to the first storage node. The first storage node is configured to transmit the first data to at least one computing node among the multiple computing nodes based on the redistribution plan.

[0011] In one possible implementation, when the first storage node transmits the first data to at least one computing node among multiple computing nodes based on the redistribution plan, it is specifically used to: filter out the first data based on the redistribution plan; filter out at least one computing node from multiple computing nodes based on the first data; and transmit the first data to the filtered computing node.

[0012] In a possible implementation, before transmitting the first data to at least one computing node among the multiple computing nodes, the first storage node is further configured to: establish a data transmission channel with each computing node in at least one computing node among the multiple computing nodes.

[0013] In one possible implementation, when the coordinator node determines that the optimal execution plan generated by the coordinator node includes a redistribution plan, it is specifically used to: determine that the aggregation key included in the query statement is inconsistent with the distribution key used when storing data in the database; or determine that the join key included in the query statement is inconsistent with the distribution key used when storing data in the database.

[0014] In a third aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect. The computing device cluster may include one or more computing devices.

[0015] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, causes the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect. The computing device cluster may include one or more computing devices.

[0016] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG1 is a schematic diagram of the logical architecture of a database management system provided in an embodiment of the present application;

[0018] FIG2 is a schematic diagram of the physical architecture of a database provided in an embodiment of the present application;

[0019] FIG3 is a schematic diagram of a data redistribution process provided by an embodiment of the present application;

[0020] FIG4 is a flow chart of a data redistribution method provided in an embodiment of the present application;

[0021] FIG5 is a schematic diagram of the structure of a database provided in an embodiment of the present application;

[0022] FIG6 is a schematic diagram of the architecture of a database system provided in an embodiment of the present application;

[0023] FIG7 is a schematic diagram of an execution step of the system shown in FIG6;

[0024] FIG8 is a schematic diagram of another execution step of the system shown in FIG6;

[0025] FIG9 is a schematic diagram of the architecture of a database system provided in an embodiment of the present application;

[0026] FIG10 is a schematic diagram illustrating an example of a redistribution scenario of an aggregation operation provided in an embodiment of the present application;

[0027] FIG11 is a schematic diagram illustrating an example of a data redistribution scenario of a distributed Join provided in an embodiment of the present application;

[0028] FIG12 is a schematic diagram illustrating an example of a data redistribution scenario provided by an embodiment of the present application;

[0029] FIG13 is a schematic diagram illustrating an example of execution steps of a data redistribution scenario provided in an embodiment of the present application;

[0030] FIG14 is a schematic diagram of a database system architecture using distributed storage or centralized shared storage provided in an embodiment of the present application;

[0031] FIG15 is a schematic diagram of an execution step of the system shown in FIG14;

[0032] FIG16 is a schematic diagram of a database system architecture in which state and persistence are not separated, provided in an embodiment of the present application;

[0033] FIG17 is a schematic diagram of an execution step of the system shown in FIG16 . DETAILED DESCRIPTION

[0034] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0035] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0036] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0037] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0038] First, some technical terms involved in this application are introduced.

[0039] (1) Query

[0040] A query is the process of retrieving data from a database. This can involve retrieving data from a single table or joining data from multiple tables. Queries can be written using SQL statements. Through queries, users can easily retrieve the data they need for data analysis, report generation, and decision support.

[0041] (2) Operator

[0042] Operators, also known as operators, are used to process data. Operators are categorized as logical operators and physical operators. Logical operators describe the semantics of an operation but do not cover its specific implementation. Physical operators describe its execution. For example, a join is a logical operator, while the corresponding hash join, nested loop join, and sort-merge join are physical operators.

[0043] (3) Query Optimizer

[0044] The query optimizer is a key component of a database management system, responsible for parsing, analyzing, optimizing, and generating execution plans for SQL statements. Its goal is to find the optimal execution plan to meet user query requirements with minimal time and resource costs. The query optimizer is primarily responsible for converting logical operators into physical operators to generate an efficient execution plan. The execution plan displays the physical operators.

[0045] (4) Execution plan

[0046] An execution plan describes the specific steps and execution order for a database management system to execute a query statement. The basic unit of operation in an execution plan is called a (physical) operator, representing a specific operation, such as a table scan or hash join. Operators in an execution plan form a tree structure in the order in which they are executed. The root node of the tree is the outermost operator, and the leaf nodes are the innermost operators.

[0047] (5) Connection operation

[0048] A join operation generates a new table by matching data from two or more tables according to the join conditions. Joins allow you to retrieve related information from multiple tables. Joins are typically implemented using SQL statements, including keywords such as INNER JOIN, LEFT JOIN, and RIGHT JOIN.

[0049] (6) Connection conditions

[0050] A join condition specifies the conditions for connecting two tables, typically based on comparing certain columns in the two tables to join related rows. For example, the query statement SELECT * FROM t1 INNER JOIN t2 ON t1.c1 = t2.c2 has a join condition of t1.c1 = t2.c2. When the value of column c1 in table t1 equals the value of column c2 in table t2, the related rows of t1 and t2 are joined. Join conditions are typically categorized as equijoins and non-equijoins. Equijoins use only the equal sign, while non-equijoins typically use comparison operators such as greater than, greater than or equal to, less than, less than or equal to, and not equal to.

[0051] (7) Connection key

[0052] A join key is a column or combination of columns used to connect two tables. It associates rows with identical or related values ​​in the two tables to generate the joined result set. In SQL queries, the join key is typically specified using the JOIN clause. For example, in the join condition t1.c1 = t2.c2, the join key for table t1 is c1, and the join key for table t2 is c2; in the join condition t1.c1 > t2.c2 AND t1.c3 > t2.c4, the join key for table t1 is (c1, c3), and the join key for table t2 is (c2, c4).

[0053] (8) Polymerization bond

[0054] An aggregation key is a column or attribute used to perform an aggregation operation. An aggregation operation is an operation used to calculate summary values ​​(such as sum, average, count, maximum, minimum, etc.) of data. In a data query, when an aggregation calculation is performed based on a data field, this data field becomes the aggregation key. For example, a query might ask: sum the values ​​of column B for rows with the same data in column A of a table. Column A is the aggregation key. To facilitate the summation, rows with the same data in column A need to be aggregated.

[0055] (9) Distribution Key

[0056] A distribution key is a concept used in distributed databases to determine how data is distributed across different nodes. A distribution key is a key attribute or column in a distributed database that influences how data is distributed and stored. Typically, data in a table is distributed across different storage nodes based on the data characteristics of a specific column in the table. This column serves as the distribution key.

[0057] (10) Redistribute

[0058] Redistribution is the process of redistributing data across different nodes to improve query performance or data management. It is a common operator that requires data to be redistributed according to new rules for subsequent calculations or query input by other operators in the plan tree.

[0059] (11) PushDown

[0060] Pushdown is an optimization method in data query, which means that the actions originally executed in the upper-level components are put into the lower-level components for early execution to achieve the purpose of improving query efficiency.

[0061] (12) Predicate

[0062] A predicate refers to a characteristic description of the required data in a data query, and data can be filtered based on the predicate information.

[0063] (13) Data distribution

[0064] Data distribution means that in a distributed database, data is distributed on different storage nodes according to certain rules.

[0065] Next, the technical solution provided by this application is introduced.

[0066] For example, Figure 1 illustrates a logical architecture diagram of a database management system provided by an embodiment of the present application. As shown in Figure 1 , the database management system may include a client 100 and a database 200. Database 200 may include an SQL engine 210 and a storage engine 220. Client 100 refers to various forms of database connection, such as the ActiveX Data Objects (ADO) connection used in .Net and the Java Database Connect (JDBC) connection used in Java.

[0067] The SQL engine 210 is primarily responsible for generating an efficient execution plan for SQL statements entered by the client 110 under the current load scenario, and for executing that execution plan. The SQL engine 210 may include a connector 211, a query cache 212, a parser 213, an optimizer 214, and an executor 215. The connector 211 is primarily responsible for communicating with the client 110 and for business logic processing such as connection authentication, connection count determination, and connection pool management. The query cache 212 primarily serves to improve query efficiency. The cache stores data in the form of a hash table of keys and values, where the key is the specific SQL statement and the value is the result set. When a SQL statement arrives, if the query cache function is enabled, the SQL engine 210 will first check the query cache 212 for matching data. If a match is found, the matching data is directly returned to the client 110 without further parsing the corresponding SQL statement. However, if the SQL statement contains user-defined functions, stored functions, user variables, or temporary tables, the query cache 212 will not be used. If no match is found in the query cache 212, the corresponding SQL statement is parsed using the parser 213. The parser 213 is primarily responsible for parsing the SQL statement according to grammatical rules and generating an internally recognizable parse tree. The optimizer 214 is responsible for optimizing the parse tree generated by the parser 213 to find an optimal execution plan. After the optimizer 214 finds the optimal execution plan, the executor 215 is primarily responsible for calling the storage engine 220 interface to execute the query or other operations based on the optimal execution plan, ultimately returning the query result set to the client 110.

[0068] The storage engine 220 is primarily responsible for the storage, retrieval, and management of data. It defines important characteristics of the database management system, such as how to organize data, how to execute queries and transactions, and the security and reliability of data.

[0069] For example, FIG2 shows a schematic diagram of the physical architecture of a database provided in an embodiment of the present application. As shown in FIG2 , the database 200 may include at least one coordinator node, several computing nodes, and several storage nodes. Each coordinator node and computing node in the database 200 may be any device or apparatus with computing capabilities, such as a server or a virtual machine running on general-purpose hardware. A storage node may be any device or apparatus with storage capabilities, such as memory, a hard disk, a disk array, a cloud storage pool (such as object storage, a distributed file system, a cloud block storage, etc.). A computing node is associated with a storage node, and the two may have a binding relationship; of course, there may not be a binding relationship between the computing node and the storage node. The computing nodes interact with each other through a network or other communication methods, and have high parallel processing and expansion capabilities. Each computing node processes its own data separately, and summarizes the processed results to the upper layer or circulates them between other nodes. In addition, the coordinator node and each computing node may interact through a network or other communication methods. In this embodiment, the coordinator node is mainly used to communicate with the client 110, and to generate an optimal execution plan and send the optimal execution plan to the computing node. The computing node is mainly used to execute the execution plan received from the coordinator node. When executing the plan, it can read data from the storage node bound to it, process the data, or forward the data to other nodes to achieve data redistribution. The storage node is mainly used to store data. In some embodiments, the coordinator node can be integrated into the computing node or arranged separately from the computing node, which is not limited here. In some embodiments, the coordinator node can be configured with the connector 211, query cache 212, parser 213 and optimizer 214 shown in Figure 1; each computing node can be configured with an executor 215; each storage node can be configured with a storage engine 220.

[0070] The physical architecture of the database 200 shown in FIG2 can be understood as a database architecture with storage and computing separated. This database architecture with storage and computing separated is derived from the traditional database architecture with storage and computing integrated. In the traditional database architecture with storage and computing integrated, storage resources and computing resources are bound. When evolving to the database architecture with storage and computing separated, there is also a certain binding relationship between computing nodes and storage nodes. This means that when the plan executed by a certain computing node includes a redistribution plan, the computing node needs to first read data from the storage node with which it has a binding relationship, and then transmit the read data to other computing nodes. For example, as shown in FIG3, computing node A and storage node A have a binding relationship, computing node B and storage node B have a binding relationship, storage node A stores the data of column P1 in table T1, and storage node B stores the data of column P2 in table T1; if computing node A needs to operate on the data of column P2 in table T1, and computing node B needs to operate on the data of column P1 in table T1, then both computing nodes A and B need to perform a redistribution operation. At this point, compute node A must first read the data in column P1 of table T1 from storage node A and then transfer that data to compute node B. Similarly, compute node B must first read the data in column P2 of table T1 from storage node B and then transfer that data to compute node A. As can be seen, when executing the redistribution plan, the number of network transmission hops is large, and the data transmission volume across the entire network is also large, which increases the consumption of network and computing resources and reduces data query performance.

[0071] In view of this, an embodiment of the present application provides a data redistribution method. When the redistribution plan is included in the optimal execution plan, the redistribution plan can be pushed down to the storage node, so that the storage node can directly send the data to the corresponding computing node, thereby changing the data transmission from pulling from the destination end to actively pushing from the source end, reducing the number of network transmission hops, reducing the amount of data transmission in the network, and reducing the consumption of network and computing resources, thereby improving data query performance.

[0072] The following describes the data redistribution method provided in the embodiments of the present application.

[0073] For example, FIG4 shows a flow chart of a data redistribution method provided in an embodiment of the present application. The method can be applied to a database, which may include: a coordinator node, multiple computing nodes, and at least one storage node. For example, the database may be, but is not limited to, a cloud database. As shown in FIG4, the data redistribution method may include the following steps:

[0074] S401: The coordinator node generates an optimal execution plan based on the query statement received from the client.

[0075] In this embodiment, when a coordinator node in the database receives a query statement from a client, it can process the query statement to generate an optimal execution plan. For example, the coordinator node can be, but is not limited to, the coordinator node described in FIG2 .

[0076] S402: The coordinator node determines that the optimal execution plan includes a redistribution plan, where the redistribution plan is used to redistribute the first data stored in the first storage node.

[0077] In this embodiment, in a scenario where data needs to be redistributed, the coordinator node can determine that the optimal execution plan it generates includes a redistribution plan. For example, when the aggregation key included in the query statement is inconsistent with the distribution key used when storing data in the database, the coordinator node can determine that the optimal execution plan includes a redistribution plan, or when the join key included in the query statement is inconsistent with the distribution key used when storing data in the database, the coordinator node can determine that the optimal execution plan includes a redistribution plan. Exemplarily, the redistribution plan can be used to redistribute the first data stored in the first storage node. Exemplarily, the first storage node can be, but is not limited to, a storage node described in FIG. 2 .

[0078] S403: The coordinator node transmits the redistribution plan to the first storage node.

[0079] In this embodiment, the coordinator node can transmit the redistribution plan to the first storage node to push the redistribution plan down to the first storage node. For example, the coordinator node can learn about the data that needs to be redistributed through the redistribution plan; further, by querying the metadata in the database, it can be known on which storage node the data is stored. In some embodiments, the coordinator node can directly send the redistribution plan to the first storage node, or forward the redistribution plan to the first storage node through a computing node. The specific method can be determined according to actual conditions and is not limited here.

[0080] S404: The first storage node transmits the first data to at least one computing node among the multiple computing nodes based on the redistribution plan.

[0081] In this embodiment, after obtaining the redistribution plan, the first storage node may execute the redistribution plan and transmit the first data to at least one of the multiple computing nodes so that the first data can be processed by the at least one of the multiple computing nodes. Exemplarily, after obtaining the redistribution plan, the first storage node executes the redistribution plan and traverses and scans the data in each partition on it to obtain the first data. The first storage node may then perform calculations based on a partitioning algorithm (such as a hash partitioning algorithm) to filter out at least one computing node from the multiple computing nodes for processing the first data. In this way, at least one computing node is filtered out from the multiple computing nodes in the database. Finally, the first storage node may transmit the first data to the filtered computing node. In some embodiments, the first storage node may first establish a data transmission channel with each of the filtered computing nodes, and then transmit data to each of the filtered computing nodes. The first storage node may perform a handshake operation with the computing node to establish a data transmission channel between the two. After the computing node obtains the first data, or after obtaining data from another storage node, it may perform calculations on the obtained data. Illustratively, the computing node may be, but is not limited to, a computing node described in FIG. 2 .

[0082] In this way, when there is a redistribution plan in the optimal execution plan, the data transmission is changed from being pulled from the destination end to being actively pushed from the source end, which can reduce the number of network transmission hops, reduce the amount of data transmitted in the network, and reduce the consumption of network and computing resources, thereby improving data query performance.

[0083] In some embodiments, the database may include multiple storage nodes, and one storage node is associated with one computing node, for example, in a binding relationship, etc. In other embodiments, the computing nodes and storage nodes included in the database may not have an association relationship.

[0084] In some embodiments, as shown in Figure 4 , the storage node in the database can logically be a single node, for example, composed of several thread pools. In this scenario, there can be no binding relationship between data and storage nodes, meaning that the storage node can read data from all partitions. In this scenario, data can still be pushed from the storage node, which can also improve data query efficiency. Since data is stored according to distribution key rules, for example, a partition may contain a single file or a group of files, the compute nodes need to redistribute this data before aggregating it for computation. In this case, having the storage node read the files in a particular partition and then distribute them to different compute nodes is also more efficient. Compared to having compute nodes pull data individually, a storage node-driven push solution can effectively reduce data scanning. For example, if three compute nodes need to pull data from a storage node, the compute node-driven pull method requires each compute node to scan the data from the storage node once. However, the solution in this embodiment only requires the storage node to scan the data once, significantly reducing the number of data scans and lowering disk input / output (I / O).

[0085] In some embodiments, as databases move towards cloud computing, traditional storage and computing-integrated database architectures cannot achieve layered, elastic, on-demand allocation and use due to the binding of storage resources and computing resources.

[0086] The separation of the storage and computing layers, with storage and computing resources independently requested and released on demand, also introduces new challenges. The computing layer needs to interact extensively with the storage layer to retrieve data, making it much less efficient than local data access in traditional integrated storage and computing architectures.

[0087] To reduce the amount of network data transmission in a storage-computing separation architecture, there are several mature technologies in the industry. As shown in Figure 6, this system architecture includes the following software modules: Spark (i.e., the computing layer) and ZNBase (and the storage layer). The computing layer pushes operators down to the storage layer, which returns only the final results rather than the raw data, significantly reducing the amount of data transmitted over the network. The figure above illustrates an example of predicate pushdown. The predicate, here represented by PushDownExpr, describes the filtering conditions for the required data in the computing layer. When a data read request does not contain a predicate, the processing flow is as shown in Figure 7: Step 1: The computing node sends a data read request to the storage (without the predicate); Step 2: The storage node reads the full data; Step 3: The storage node sends the full data to the computing node; Step 4: The computing node receives all the data and filters it according to the predicate; Step 5: The computing node proceeds to the next processing step. When the data read request contains predicate information, the processing flow can be shown as in Figure 8. Step 1: The computing node sends a data read request to the storage (including predicate information); Step 2: The storage node reads the required data according to the predicate information; Step 3: The storage node sends the filtered data to the computing node; Step 4: The computing node receives the data; Step 5: Enter the next processing flow of the computing node.

[0088] Comparing Figures 7 and 8, we can see that after applying this technique, by pushing down the predicates, the data sent from the storage nodes to the compute nodes is exactly what the compute nodes need. If the amount of data filtered by the predicate is large, this technique will inevitably greatly reduce ineffective network data transmission.

[0089] The above analysis shows that the system shown in Figure 6 can push predicate descriptions down to storage nodes, filtering out large amounts of data through predicate pushdown and reducing network transmission. However, its applicable scenarios only represent a small subset of all database query scenarios. No corresponding solutions are provided for more complex computational layer operators. For example, multi-table distributed joins or aggregation queries are not suitable for direct pushdown to the storage layer due to their complex computational logic.

[0090] In addition, as shown in Figure 9, to reduce the amount of network data transmission in a storage-computing separation architecture, high-speed networks such as remote direct memory access (RDMA) can be used between compute nodes and storage nodes to reduce network latency and improve data transmission efficiency. In this way, by using special hardware to accelerate the network, the problem of large network transmission volume leading to reduced request throughput after storage-computing separation is solved to a certain extent. However, this method has special requirements for hardware and cannot completely solve the problem of large data volumes in a storage-computing separation architecture. Therefore, it is necessary to optimize the database process and logic more from the software process or algorithm level.

[0091] In view of the problems existing in the above-mentioned technologies, this application provides a method or device for improving query performance in a storage-computation separation architecture. In queries involving aggregation operations or distributed joins, data redistribution operations between compute nodes are pushed down to storage nodes, reducing the total amount of network transmission of redistributed table data. By pushing the redistribution operator down, the network and computing resource consumption of the redistribution operation on the compute nodes is greatly optimized, reducing the number of network data transmission hops and the total amount of transmission.

[0092] The application scenario of this application is to execute query scenarios that require data redistribution in databases or data services with a storage and computing separation architecture. This includes:

[0093] 1. Aggregate data query request: In a distributed system, when the distribution key and aggregation key are inconsistent, data redistribution is required according to the aggregation key to facilitate subsequent aggregation calculations.

[0094] 2. Distributed Join query request: In a distributed system, when the Join key is inconsistent with the distribution key, data must be redistributed according to the Join key to facilitate subsequent Join operations.

[0095] An example of a redistribution scenario for aggregation operations is shown in Figure 10. In Figure 10, T1 identifies the data table named T1; T1-P1, T1-P2, and T1-P3 identify partitions of table T1, stored on storage nodes 1, 2, and 3, respectively; and T1-redistributeP1, T1-redistributeP2, and T1-redistributeP3 are the partitions after redistribution based on the aggregation key.

[0096] For an aggregate query request that requires redistribution, the execution process before applying this application is as follows:

[0097] 1. The computing nodes read the P1, P2, and P3 partition data of the T1 table from the storage nodes respectively.

[0098] 2. The computing node calculates the final destination computing node of the data in each partition based on the aggregation key and sends it to the corresponding computing node. If the final destination node is the current computing node, no sending is required.

[0099] 3. After completing step 2, each computing node obtains the data it needs and can proceed to the next step of calculation.

[0100] An example of a data redistribution scenario for a distributed join can be shown in Figure 11. Figure 11 is similar to Figure 10. In Figure 11, the distributed join involves two tables, T1 and T2. T1 does not need to be redistributed because its distribution key is consistent with its join key. However, T2 does need to be redistributed because its distribution key is different from its join key. In the current scenario, the execution process before applying this application is as follows:

[0101] 1) The computing nodes read the data of partitions P1, P2, and P3 of table T1 and the data of partitions P1, P2, and P3 of table T2 from the storage nodes respectively.

[0102] 2) The computing node calculates the final destination computing node of the data in each partition of the T2 table based on the aggregation key of the T2 table and sends it to the corresponding computing node. If the final destination node is the current computing node, no sending is required.

[0103] 3) After completing step 2, each computing node obtains the required T1 and T2 table partition data and can proceed to the next step of Join calculation.

[0104] In this application, as shown in Figure 12, by pushing the redistribution action down to the storage nodes, the network transmission of the redistributed data is reduced. Data is sent directly from the storage nodes to the target compute nodes, bypassing intermediate compute nodes. The data transmission process is as follows: 1) The storage node traverses the data and calculates the destination compute node based on the data distribution characteristics, then sends the data to the destination compute node. 2) After receiving data from different storage nodes, the compute node proceeds to the next step of the computation process.

[0105] The specific execution steps are shown in Figure 13. 1) The coordinator node generates a redistribute push execution plan and sends it to all involved computing nodes and storage nodes. The plan can be sent directly to the storage by the coordinator node or sent to the storage by the computing nodes.

[0106] a) The coordinator node is a logical concept. Physically, this role can be integrated with a computing node or deployed independently.

[0107] b) At the same time, the redistribute pushdown plan can be broadcast by the coordinator node to the storage nodes participating in the request, or it can be issued through the computing nodes, and is not limited to the scenario described in the above figure.

[0108] 2) After receiving the pushed-down redistribute plan, the storage node establishes a data flow channel with all involved computing nodes.

[0109] 3) The storage node traverses and scans the partitioned data, and at the same time calculates the target computing node of the scanned data according to the partitioning algorithm, puts the data into the corresponding data flow channel, and sends the data to the target computing node.

[0110] 4) For each computing node, data is collected from multiple storage data flow channels and the next step of data calculation is performed according to the execution plan.

[0111] Furthermore, storage in this application can be categorized as shared storage or non-shared storage, depending on how the stored data is shared. The physical storage medium is not limited and can be memory, disk, or remote distributed storage. The deployment of compute and storage nodes in this application is also unrestricted and can be deployed on host machines, virtual machines, or containers.

[0112] For example, as shown in FIG14 , distributed storage or centralized shared storage is used, and the upper-level computing nodes are not aware of the distribution details of the T1 table, which is reflected as a whole. The execution process at this time can be shown in FIG15 . In FIG15 , first, the coordinator node generates an execution plan. Then, the coordinator node synchronizes the plan to all computing nodes and distributed storage (or centralized shared storage) participating in the current query. Then, the distributed storage (or centralized shared storage) establishes a data transmission channel with all computing nodes participating in the current query. Then, the distributed storage starts data scanning and sends the qualified data to the target computing node according to the redistribution rules. Finally, after receiving the T1 table data, the target computing node proceeds to the next step of the calculation process.

[0113] As shown in Figure 16, this architecture integrates state and persistence. The difference between this architecture and the architecture shown in Figure 14 lies in the different storage formats. The execution process is shown in Figure 17. In Figure 17, the coordinator node first generates an execution plan. Next, the coordinator node synchronizes the plan with all compute nodes and storage nodes participating in the query. Next, the storage node establishes a data transmission channel with the compute nodes participating in the query. The storage node then begins data scanning, sending eligible data to the target compute node according to the redistribution rules. Finally, after receiving the data from table T1, the target compute node proceeds to the next step in the computation process.

[0114] The main technical approach adopted in this application is to push down the redistribute operator, which redistributes the operators or plans to the storage nodes in a storage-computation separation architecture. The entire redistribution operation is completed through negotiation between the coordinator node, storage nodes, and computing nodes. The beneficial effects achieved by this approach are as follows:

[0115] 1. Reduce network transmission: Reduce the network transmission process of table data, reduce one round of table data replication process, reduce CPU / memory resource consumption, and improve query latency and throughput.

[0116] 2. Reduce disk IO: Data transmission is changed from being pulled from multiple destinations to being actively pushed from the source. The data file scan at the source can be optimized from multiple times to one, reducing disk IO.

[0117] 3. Safer: Pushing down reduces the number of network transmission hops that table data undergoes, making it safer.

[0118] It should be understood that the order of execution of the steps in the above embodiments does not necessarily imply a specific order of execution. The order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments described above can be combined according to actual circumstances, and the combined solutions are still within the scope of protection of this application.

[0119] Based on the method in the above embodiment, an embodiment of the present application provides a database.

[0120] For example, FIG5 shows a schematic diagram of the structure of a database provided in an embodiment of the present application. As shown in FIG5 , the database 500 may include: a coordinator node 510, multiple computing nodes 520, and at least one storage node 530. The coordinator node 510 is configured to determine whether the optimal execution plan generated by the coordinator node 510 includes a redistribution plan, where the redistribution plan is used to redistribute first data, wherein the first data is stored in a first storage node 530 among the at least one storage node 530 and processed by the first computing node 520 among the multiple computing nodes 520. The coordinator node 510 is also configured to transmit the redistribution plan to the first storage node 530. The first storage node 530 is configured to transmit the first data to at least one computing node 520 among the multiple computing nodes based on the redistribution plan. For example, the coordinator node 510 may communicate with the storage node 530 indirectly through the computing node 520, or directly. Furthermore, the computing node 520 and the storage node 530 may or may not have a binding relationship, depending on the actual situation. Exemplarily, the storage node 530 may be a memory, a hard disk, a disk array, or a cloud storage pool (eg, object storage, a distributed file system, a cloud block storage, etc.).

[0121] In some embodiments, when the first storage node 530 transmits the first data to at least one computing node 520 based on the redistribution plan, it is specifically used to: filter out the first data based on the redistribution plan; filter out at least one computing node 520 from multiple computing nodes 520 based on the first data; and transmit the first data to at least one computing node 520.

[0122] In some embodiments, before transmitting the first data to at least one computing node 520 , the first storage node 530 is further configured to establish a data transmission channel with each of the screened computing nodes 520 .

[0123] In some embodiments, when the coordinator node 510 determines that the optimal execution plan generated by the coordinator node 510 includes a redistribution plan, it is specifically used to: determine that the aggregation key included in the query statement is inconsistent with the distribution key used when storing data in the database; or determine that the connection key included in the query statement is inconsistent with the distribution key used when storing data in the database.

[0124] It should be understood that the above-mentioned database is used to execute the method in the above-mentioned embodiment. The implementation principles and technical effects of the corresponding nodes in the database are similar to those described in the above-mentioned method. The working process of the database can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0125] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computing device cluster including at least one computing device, the computing device cluster executes the method described in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), a semiconductor medium (for example, a solid-state drive), or a cloud storage pool (for example, an object storage, a distributed file system, a cloud block storage, etc.), etc.

[0126] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions. When the computer program product is run on a computing device cluster including at least one computing device, the computing device cluster executes the method in the above embodiment.

[0127] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0128] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0129] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0130] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data redistribution method, characterized in that: Applied to a database, the database includes: a coordinator node, multiple computing nodes and at least one storage node, the method includes: The coordinator node determines that the optimal execution plan generated by the coordinator node includes a redistribution plan, where the redistribution plan is used to redistribute the first data, wherein the first data is stored in a first storage node among the at least one storage node; The coordinator node transmits the redistribution plan to the first storage node; The first storage node transmits the first data to at least one computing node among the plurality of computing nodes based on the redistribution plan.

2. The method according to claim 1, characterized in that The first storage node transmits the first data to the at least one computing node based on the redistribution plan, specifically including: The first storage node filters out the first data based on the redistribution plan; The first storage node selects the at least one computing node from the plurality of computing nodes based on the first data; The first storage node transmits the first data to the at least one computing node.

3. The method according to claim 1 or 2, characterized in that: Before the first storage node transmits the first data to the first computing node, the method further includes: The first storage node establishes a data transmission channel with each computing node in the at least one computing node.

4. The method according to any one of claims 1 to 3, characterized in that: The coordinator node determines that the optimal execution plan generated by the coordinator node includes a redistribution plan, specifically including: The coordinator node determines that the aggregation key included in the query statement is inconsistent with the distribution key used when storing data in the database; Alternatively, the coordinator node determines that the connection key included in the query statement is inconsistent with the distribution key used when storing data in the database.

5. A database, characterized in that: include: A coordinator node, multiple computing nodes, and at least one storage node; The coordinator node is used to determine that the optimal execution plan generated by the coordinator node includes a redistribution plan, and the redistribution plan is used to redistribute the first data, wherein the first data is stored in a first storage node among the at least one storage node; The coordinator node is further used to transmit the redistribution plan to the first storage node; The first storage node is used to transmit the first data to at least one computing node among the multiple computing nodes based on the redistribution plan.

6. The database according to claim 5, characterized in that When the first storage node transmits the first data to at least one computing node among the plurality of computing nodes based on the redistribution plan, the first storage node is specifically configured to: filter out the first data based on the redistribution plan; Based on the first data, selecting the at least one computing node from the plurality of computing nodes; The first data is transmitted to the at least one computing node.

7. The database according to claim 5 or 6, characterized in that: Before transmitting the first data to the at least one computing node, the first storage node is further configured to: A data transmission channel is established with each computing node in the at least one computing node.

8. The database according to any one of claims 5 to 7, characterized in that: When the coordinator node determines that the optimal execution plan generated by the coordinator node includes a redistribution plan, the coordinator node is specifically used to: Determining that an aggregation key included in the query statement is inconsistent with a distribution key used when storing data in the database; Alternatively, it is determined that the join key included in the query statement is inconsistent with the distribution key used when storing data in the database.

9. A computer-readable storage medium storing a computer program, which, when executed on a computing device cluster comprising at least one computing device, enables the computing device cluster to execute the method according to any one of claims 1 to 4.

10. A computer program product, characterized in that When the computer program product is run on a computing device cluster including at least one computing device, the computing device cluster is enabled to execute the method according to any one of claims 1 to 4.

Images