A database execution engine adaptive push-pull model combination method and component

Abstract

The present invention discloses a method and component for combining a database execution engine with an adaptive push-pull model, the method comprising: building BridgePushIn and BridgePushOut nodes of Push type and BridgePullNode node of Pull type in an executor execution link; judging the node type of the node on the execution link and marking it, the node type including nodes that can be replaced with and cannot be replaced with Push type; assembling the marked nodes on the execution link, and the defined BridgePushIn node, BridgePushOut node and BridgePullNode node to construct the execution link; initializing the assembled execution link, and performing data flow processing on the initialized execution link. The present invention combines the advantages of the Pull and Push models, and effectively improves the execution efficiency of the database system.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing, and mainly to a database execution engine adaptive push-pull model combination method and component. Background Art

[0002] In database systems, see Figure 1 , mainly divided into three modules: optimizer, execution engine and storage. Among them, the optimizer is responsible for calculating the optimal query plan based on SQL (Structured Query Language), the execution engine is responsible for the entire query execution process, including multiple subcomponents such as query parser, query optimizer, executor, etc., converting the query plan into a series of executable steps and managing the execution of the steps, including data access, calculation, concurrency control, transaction management, etc.; the storage is responsible for the storage of user data and metadata.

[0003] In the database system, the pull model and the push model are two different data transmission and processing modes.

[0004] In the Pull model, the data flow is mainly in the form of pull, and the downstream node pulls the data from the upstream node. The data flow and control flow are consistent, but more virtual functions need to be added for calling, the execution efficiency is low, and the concurrent processing is limited. The retrieval of data in the model is initiated by the consumer (usually the client application), and the consumer will periodically poll the database to obtain the latest data changes. For example, a client may periodically execute a SELECT query to check whether there are new records or whether existing records have been updated. In summary, the advantage of this model is that it is simple and easy to control, but the disadvantage is that it may cause unnecessary polling, especially when the data does not change frequently, which will lead to a waste of resources.

[0005] In the Push model, the data flow is mainly in the form of push, and the data flow is pushed from the upstream node to the downstream node. The data flow and control flow are opposite. In addition, in the process of processing, in order to improve efficiency, data redistribution is generally required, which is relatively complex, but reduces virtual function calls and has high execution efficiency. In the Push model, data changes are actively pushed to consumers by producers (databases or services). When data changes, the database will immediately notify consumers and push the changed data. For example, some databases support the publish / subscribe (pub / sub) mechanism, allowing applications to subscribe to data change events. When an event occurs, the database will actively push the event and related data to subscribers. This model has stronger real-time performance, can respond to data changes immediately, and reduce unnecessary polling, but it is more complicated to implement and requires the database to support this push mechanism.

[0006] In actual applications, the above two models have their own characteristics and can be selected according to different scenarios and requirements. For example, for systems that need to respond to data changes in real time, the Push model may be more suitable; while for systems where data changes infrequently or real-time requirements are not high, the Pull model may be simpler and more efficient. However, in some complex scenarios, such as mixed scenarios with filtering, sorting, and aggregation, if the Pull model is used alone, the efficiency of sorting and aggregation is average due to too many virtual function calls and limited concurrent processing capabilities. For the Push model, the efficiency is relatively high, but it requires the configuration of a precise scheduling system, which is too complicated to implement. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method for improving the efficiency of a database execution engine by combining a Pull model and a Push model.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions.

[0009] In a first aspect, the present invention provides a database execution engine adaptive push-pull model combination method, comprising:

[0010] Build BridgePushIn and BridgePushOut nodes in the executor execution link and define the node type as Push type;

[0011] Build the BridgePullNode node in the executor execution link and define the node type as Pull type;

[0012] Determine the node type of the node on the execution link in the executor and mark it. The node type includes nodes that can be replaced with Push type and nodes that cannot be replaced with Push type;

[0013] Assemble the nodes marked on the execution link, as well as the defined BridgePushIn node, BridgePushOut node, and BridgePullNode node to build the execution link;

[0014] The assembled execution link is initialized, and the initialized execution link performs data flow processing.

[0015] Furthermore, the BridgePushIn node serves as an input node of the Push model execution link, responsible for driving the input of data and pushing the data into the execution link; the BridgePushOut node serves as an output node of the Push model execution link, responsible for the output of data, and outputs the processed data outside the execution link.

[0016] Furthermore, the BridgePullNode node is an access node of the Pull model execution link and is connected to the Push model execution link.

[0017] Further, the determining the node type of the node on the execution link in the executor and marking it includes:

[0018] Assume that all nodes on the entire execution chain are Pull type nodes;

[0019] Traverse from the top-level node to the bottom-level node from top to bottom according to the query plan;

[0020] Determine whether the current node needs to be replaced with a Push type node based on the whitelist;

[0021] When encountering a node that needs to be replaced with a Push type, mark it as cur and continue to traverse recursively downward. When encountering a node that does not need to be replaced with a Push type, mark it as end and continue to traverse recursively downward until all nodes are traversed.

[0022] Furthermore, the whitelist includes a group of custom-set nodes that have been implemented as a Push model.

[0023] Further, the assembling of the nodes marked on the execution link, and the defined BridgePushIn nodes, BridgePushOut nodes, and BridgePullNode nodes includes:

[0024] By using the BridgePushIn node and the BridgePushOut node as the input node and the output node of the Push model execution link, and wrapping the entire execution link from BridgePushIn to BridgePushOut with the BridgePullNode node, a new execution link is constructed.

[0025] Furthermore, the initialization of the assembled execution link includes calling the Init interface of each node, setting the internal variables corresponding to each node, and initializing the state of the global runtime.

[0026] In a second aspect, the present invention provides a database execution engine adaptive push-pull model combination component, comprising:

[0027] The BridgePushIn and BridgePushOut node building modules are used to build BridgePushIn and BridgePushOut nodes in the actuator execution link and define the node type as Push type;

[0028] The BridgePullNode node building module is used to build the BridgePullNode node in the executor execution link and define the node type as the Pull type;

[0029] The node type marking module is used to determine the node type on the execution link in the executor and mark it. The node types include nodes that can be replaced with Push type and nodes that cannot be replaced with Push type.

[0030] The execution link building module is used to assemble the nodes marked on the execution link, as well as the defined BridgePushIn node, BridgePushOut node and BridgePullNode node to build the execution link;

[0031] The initialization module is used to initialize the assembled execution link, and the initialized execution link performs data flow processing.

[0032] Beneficial effects of the present invention:

[0033] The present invention combines the advantages of the Pull and Push models. On some complex nodes, such as aggregation nodes, Join nodes, sorting nodes, etc., the Push model is used to improve the processing rate, and the Pull model is used to control the flow of data. Under the premise of simplifying the complexity of the database system, the execution efficiency of the database system is improved, and the speed of data processing is effectively improved. At the same time, since the present invention controls the replacement of Pull and Push based on the whitelist rule, in actual implementation, the whitelist can be modified according to the actual situation of the database operation, so as to achieve the control of the specific node to be replaced. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A schematic diagram of the location of the execution engine in the database system;

[0036] Figure 2 A flowchart of a database execution engine adaptive push-pull model combination method provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of data flow of newly added BridgePushIn and BridgePushOut nodes provided in an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of data flow of a newly added BridgePullNode node provided in an embodiment of the present invention;

[0039] Figure 5 A schematic diagram of assembling BridgePullNode provided in an embodiment of the present invention;

[0040] Figure 6 A schematic diagram of a method for combining a database execution engine adaptive push-pull model to perform data flow processing provided by an embodiment of the present invention;

[0041] Figure 7 A schematic diagram of a database execution engine adaptive push-pull model combined component provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to clearly illustrate the purpose, technical solutions and advantages of the present invention, the technical solutions of the present invention will be clearly and completely described through implementation methods with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Terminology explanation:

[0044] Executor: The executor is a key part of the execution engine, executing the sequence of operations defined in the query plan one by one, such as scanning tables, applying filters, performing joins, sorting, and aggregation. The executor usually consists of a series of nodes (also called operators) that define the specific steps of query execution. Each node is responsible for executing a specific part of the query, and the nodes can be linked together in a specific order in the query plan to form an execution tree. Each node receives the output of the previous node as input, then performs its own operation and passes the result to the next node. In this way, the entire query is executed and finally generates the result set required by the user.

[0045] agg: In the database, agg is usually the abbreviation of aggregate, which refers to an aggregate function or an aggregate operation. An aggregate function calculates a set of values ​​and returns a single value.

[0046] Join: In database, Join is an operation used to combine rows from two or more tables into a single table or result set based on certain related columns.

[0047] Scan: In the database, Scan usually refers to the operation of scanning the entire database table. Full Table Scan means that the database execution engine reads the data of the entire table row by row in order to perform query or maintenance operations.

[0048] Embodiment 1:

[0049] See also Figure 2 This embodiment provides a database execution engine adaptive push-pull model combination method, including:

[0050] S1: Build BridgePushIn and BridgePushOut nodes in the executor execution link and define the node type as Push type;

[0051] S2: Build the BridgePullNode node in the executor execution link and define the node type as Pull type;

[0052] S3: Determine the node type on the execution link in the executor and mark it. The node types include push type nodes and pull type nodes.

[0053] S4: Based on the node type marked on the execution link, the BridgePushIn node, the BridgePushOut node, and the BridgePullNode node are assembled to construct the execution link;

[0054] S5: Initialize the assembled link, and perform data flow processing on the initialized execution link.

[0055] Specifically:

[0056] In step S1, BridgePushIn and BridgePushOut nodes are built in the executor execution link and defined as Push type nodes. As the input node of the Push model execution link, the BridgePushIn node is responsible for driving the input of data, and is used to receive data from external data sources, or to respond to certain events (such as data changes in the database), and then push the data to the execution link. As the output node of the Push model execution link, BridgePushOut is responsible for data output, and outputs the processed data outside the execution link, see Figure 3 .

[0057] In step S2, a BridgePullNode node is built in the executor execution link and defined as a Pull type node; the BridgePullNode node is the access node of the Pull model execution link and connects to the Push model execution link. The node acts as a bridge between the pull type node and the push type node to drive the flow of data. For a schematic diagram in the executor, see Figure 4 .

[0058] Through the BridgePullNode node, the data flow processing mode can be switched from the Pull mode to the Push mode. The database execution engine can achieve the best performance and efficiency according to different data processing requirements and workloads. That is, the advantages of both the Pull and Push models can be combined in the execution link, and the execution efficiency of the database system can be improved while simplifying the complexity of the database system, effectively improving the speed of data processing.

[0059] In step S3, determining the node type of the node on the execution link in the executor and marking it includes the following contents:

[0060] During the executor initialization phase, assuming that all nodes on the entire execution link are Pull type nodes, the query plan is traversed from the top node to the bottom node from top to bottom. During the traversal process, the whitelist is used to determine whether the current node needs to be replaced with a Push type node. Nodes whose node IDs are in the whitelist are considered to need to be replaced with Push type nodes. The whitelist is a list of node IDs that can be added or deleted, and can be set manually according to the actual situation of database operation. For nodes that have been implemented as Push models, their IDs can be added to the whitelist. Nodes that belong to the whitelist can be replaced with Push nodes. For example, if the Agg node has implemented the Push version, the Agg node ID can be added to the whitelist, and the whitelist is used to determine whether the current node needs to be replaced with a Push type node.

[0061] When encountering a node that needs to be replaced with a Push type, mark it as cur and continue to traverse recursively downward. When encountering a node that cannot be replaced with a Push type, mark it as end and continue to traverse recursively downward until all nodes are traversed.

[0062] It should be noted that the assumption that the entire execution link is a Pull-type node is based on the following considerations:

[0063] First, in practical applications, the pull model is usually used more frequently to transfer data between nodes when executing plans;

[0064] Second, as introduced in the background technology, compared with the push model, the pull model has a simple structure, is easy to control, and is more convenient to implement;

[0065] However, the data flow processing of the Pull model depends on the drive of control instructions, which causes the system to spend more time waiting for control instructions, resulting in a waste of machine resources. The Push model does not rely on the drive of control instructions. Instead, after the upstream node processes the data, it automatically transfers the data to the downstream node, which can make full use of machine resources.

[0066] In step S4, based on the data transfer types supported by each node marked in step S3, and the BridgePushIn node, BridgePushOut node, and BridgePullNode node defined in steps S1 and S2, the above nodes are assembled to build an execution link. Specifically:

[0067] See also Figure 5 , assemble cur (Push node 2 in the figure) and end (push node 1 in the figure) marked in step S3 with BridgePushIn, BridgePushOut, and BridgePullNode. That is, BridgePushIn is the input node of end, BridgePushOut is the output node of cur, and BridgePullNode encapsulates the entire link from BridgePushIn to BridgePushOut.

[0068] In the new execution link, for BridgePullNode, when it receives the control instruction of GetNext, it will pull data from the upstream node and pass the pulled data to the BridgePushIn node. For all nodes between BridgePushIn and BridgePushOut, there is no need to wait for the GetNext control instruction, but after the upstream node is processed, it can be directly pushed to the downstream, and the processed result will be passed to BridgePushOut. When the downstream node of BridgePullNode pulls data, it takes out the data from BridgePushOut and gives it to the downstream node. In step S5, the assembled link is initialized. Initialization usually refers to preparing and setting each node and operation in the plan after the query execution plan is built, including calling the Init interface of each node, setting the corresponding internal variables of each node, and initializing the state of the global runtime, specifically including initializing memory, setting the corresponding variables and states inside the node, to ensure that the system can execute instructions correctly and efficiently. After the initialization is completed, the link can start to execute the query plan and return the result after processing.

[0069] A specific example of using the assembled link to process data is as follows:

[0070] See also Figure 6 , taking the scenario with ResultNode node, agg node (Pushnode2 in the figure), join node (Pushnode1 in the figure), and scan node (Pullnode in the figure) as an example for explanation, as follows:

[0071] Driven by the top-level nodes, data flow processing begins;

[0072] Since the data cache locality of agg nodes and join nodes in the Push model is good and has a relatively high execution efficiency, when traversing the node types on the execution link, they are marked as supporting push types;

[0073] ResultNode sends a GetNext control instruction to BridgePullNode. After receiving the GetNext control instruction, BridgePullNode sends a GetNext control instruction to the upstream Scan node. After receiving the control instruction, Scan starts to scan the data. After the Scan node scans the data, the data is transmitted to BridgePullNode. BridgePullNode pushes the data to BridgePushIn node. BridgePushIn node serves as the input node of the Push model execution link and inputs the data to the Push execution link.

[0074] The push execution link is assembled with join nodes and agg nodes, so the join nodes and agg nodes are executed in turn to process the data. The data flow between join and agg does not need to wait for the GetNext control instruction.

[0075] After being processed by the join node, the data is transmitted to the BridgePushOut node. As the output node in the Push model execution link, the BridgePushOut node is responsible for data output and outputs the processed data outside the link, thereby completing the data flow of the entire link.

[0076] The above method combines the Pull and Push models in the execution link, uses the Push model to increase the data processing rate, and uses the Pull model to control the flow of data, which has higher execution efficiency.

[0077] Embodiment 2:

[0078] See also Figure 7 This embodiment provides a database execution engine adaptive push-pull model combination component, including:

[0079] The BridgePushIn and BridgePushOut node building modules are used to build BridgePushIn and BridgePushOut nodes in the executor execution link, and define the node type as Push type; the BridgePushIn node is the input node of the Push model execution link, responsible for driving the input of data, used to receive data from external data sources, or used to respond to certain events (such as data changes in the database), and then push the data to the execution link. BridgePushOut is the output node in the Push model execution link, responsible for data output, and outputs the processed data outside the link.

[0080] The BridgePullNode node building module is used to build the BridgePullNode node in the executor execution link and define the node type as the Pull type; the BridgePullNode node is connected to the Pull model execution link and docked with the Push model execution link, and plays a bridging role between the pull type node and the push type node to drive the data flow processing. Through the BridgePullNode node, the data flow processing mode can be switched from the Pull mode to the Push mode, and the database execution engine can achieve optimal performance and efficiency according to different data processing requirements and workloads.

[0081] The node type marking module is used to determine the node type on the execution link in the executor and mark it. The node types include nodes that can be replaced with Push type nodes and nodes that cannot be replaced with Push type nodes. Specifically, assuming that all nodes on the entire execution link are Pull type nodes, traverse from the top node to the bottom node from top to bottom according to the query plan. During the traversal process, determine whether the current node needs to be replaced with a Push type node according to the whitelist. The whitelist is a list of node IDs that can be added or deleted. For nodes whose node IDs are in the whitelist, they are considered to need to be replaced with Push type nodes.

[0082] The execution link construction module is used to assemble the nodes marked on the execution link, as well as the defined BridgePushIn node, BridgePushOut node and BridgePullNode node, to construct the execution link; that is, BridgePushIn is an input node that cannot be replaced by a Push type node, BridgePushOut is an output node that needs to be replaced by a Push type node, and BridgePullNode packages the entire link from BridgePushIn to BridgePushOut. For BridgePullNode, when it receives the GetNext control instruction, it will pull data from the upstream node and pass the pulled data to the BridgePushIn node. For all nodes between BridgePushIn and BridgePushOut, there is no need to wait for the GetNext control instruction, but after the upstream node completes the processing, it can be directly pushed to the downstream, and the processed results will be passed to BridgePushOut; when the downstream node of BridgePullNode pulls data, the data is taken out from BridgePushOut and given to the downstream node.

[0083] The initialization module is used to initialize the assembled execution link, and the initialized execution link performs data flow processing. It includes calling the Init interface of each node, setting the corresponding internal variables of each node, and initializing the state of the global runtime, specifically including initializing the memory, setting the corresponding variables and states inside the node, to ensure that the system can execute instructions correctly and efficiently.

[0084] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0085] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0086] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

Claims

1. A database execution engine adaptive push-pull model combination method, characterized in that: include, Build BridgePushIn and BridgePushOut nodes in the executor execution link, and define the node type as Push type; the BridgePushIn node is used as the input node of the Push model execution link, responsible for driving the input of data and pushing the data into the execution link; the BridgePushOut node is used as the output node of the Push model execution link, responsible for the output of data, and outputs the processed data outside the execution link; A BridgePullNode node is built in the executor execution link, and the node type is defined as the Pull type; the BridgePullNode node is the access node of the Pull model execution link and is connected to the Push model execution link; Assuming that all nodes on the entire execution link are Pull type nodes, query whether the Pull type nodes on the execution link can be replaced with Push type nodes according to the whitelist traversal and mark them. The whitelist includes a set of custom-set nodes that have been implemented as Push models. Assemble the nodes marked on the execution link, as well as the defined BridgePushIn node, BridgePushOut node, and BridgePullNode node, including using the BridgePushIn node and BridgePushOut node as the input node and output node of the Push model execution link, and wrapping the entire execution link from BridgePushIn to BridgePushOut with the BridgePullNode node to build a new execution link; The assembled execution link is initialized, and the initialized execution link performs data flow processing.

2. The database execution engine adaptive push-pull model combination method according to claim 1, characterized in that: The querying and marking of whether the pull-type node on the execution link can be replaced with the push-type node according to the whitelist traversal includes: Traverse from the top-level node to the bottom-level node from top to bottom according to the query plan; Determine whether the current node needs to be replaced with a Push type node based on the whitelist; When encountering a node that needs to be replaced with a Push type, mark it as cur and continue to traverse recursively downward. When encountering a node that does not need to be replaced with a Push type, mark it as end and continue to traverse recursively downward until all nodes are traversed.

3. The database execution engine adaptive push-pull model combination method according to claim 1, characterized in that: Initializing the assembled execution link includes calling the Init interface of each node, setting the internal variables corresponding to each node, and initializing the state of the global runtime.

4. A database execution engine adaptive push-pull model combination component, comprising: The BridgePushIn and BridgePushOut node building modules are used to build BridgePushIn and BridgePushOut nodes in the executor execution link, and define the node type as Push type; the BridgePushIn node is used as the input node of the Push model execution link, responsible for driving the input of data and pushing the data into the execution link; the BridgePushOut node is used as the output node of the Push model execution link, responsible for the output of data, and outputs the processed data outside the execution link; The BridgePullNode node building module is used to build a BridgePullNode node in the executor execution link and define the node type as the Pull type; the BridgePullNode node is the access node of the Pull model execution link and connects to the Push model execution link; A node type marking module is used to assume that all nodes on the entire execution link are Pull type nodes, and query whether the Pull type nodes on the execution link can be replaced with Push type nodes according to the whitelist traversal and mark them. The whitelist includes a set of custom-set nodes that have been implemented as Push models; An execution link construction module is used to assemble the nodes marked on the execution link, as well as the defined BridgePushIn node, BridgePushOut node, and BridgePullNode node, including using the BridgePushIn node and BridgePushOut node as the input node and output node of the Push model execution link, and wrapping the entire execution link from BridgePushIn to BridgePushOut with the BridgePullNode node to construct a new execution link; The initialization module is used to initialize the assembled execution link, and the initialized execution link performs data flow processing.

Images