Event-driven database asynchronous execution engine, execution method, equipment and storage medium
By introducing an event-driven asynchronous execution mechanism into the database execution engine, it supports parallel processing at the operator level and data level, solving the problem of insufficient concurrency support for volcanic models, improving query execution efficiency and simplifying code complexity.
Patent Information
- Application Number
- CN202510144667.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-10
- Publication Date
- 2025-05-30
AI Technical Summary
The volcanic model of the existing database execution engine is not designed for concurrent execution. It has poor concurrency support between data and operators, resulting in complex modifications and extensions when supporting concurrent execution, which is large in work and complex in code.
The event-driven database asynchronous execution engine is adopted, and parallel processing at the operator level and data level is supported through event-driven methods, including operator Process, Runner, Stream and asynchronous event processing engine. Runner processes data through a simple state machine to respond to external event scheduling operators.
Through an event-driven method, parallel processing between operators and data is better supported, query execution efficiency is improved, the workload of new operators is simplified, and data processing and control is decoupled, which is convenient for the function expansion of the execution engine.
Smart Images
Figure CN120067133A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of database asynchronous execution engines, and in particular, to an event-driven database asynchronous execution engine, an execution method, a device, and a storage medium. Background Art
[0002] The execution engine of a database is responsible for executing the execution plan given by the optimizer, reading data and processing it, and then returning the result to the client, which directly affects the execution performance of queries.
[0003] Currently, the mainstream design of database execution engines is the Volcano Model (also known as the Iterator Model, Pipeline Model, etc.). The characteristics of the Volcano Model are as follows: all relational algebra operators implement the same iterator interface and are organized in a tree structure. Starting from the root node of the tree, the current operator calls the next() interface of the next operator to obtain data until the next() interface of the bottommost operator in the chain is called. The operation results (i.e., data) of the operators flow from the bottommost of the tree to the topmost of the tree in the reverse order, and each operator on the way can operate on it.
[0004] One major problem with the Volcano Model is that its native architecture is not designed for concurrent execution, and it has poor support for concurrency between data and between operators. If concurrent execution is to be supported, complex modifications and extensions need to be made at various levels, and this process needs to be repeated for each operator of the Volcano Model, resulting in a large workload and complex code. Summary of the Invention
[0005] Object of the Invention: The present invention provides an event-driven database asynchronous execution engine, an execution method, a device, and a storage medium, which support parallel processing at the operator level and data level through an event-driven manner, thereby improving query efficiency.
[0006] Technical solution: An event-driven database asynchronous execution engine according to the present invention includes: an operator Process for implementing database functions, a Runner for encapsulating the Process processing logic, a Stream for transmitting data between Runners, an asynchronous event processing engine for driving the execution of Runners, and a QueryBuilder for generating a Runner structure according to a query plan; the operator Process encapsulates the state of operator execution and the input and output Streams, and provides a standard interface for the upper-layer Runner to call. Inside the Runner, a simple state machine responds to external events to mobilize the operator for data processing. The Stream encapsulates several Channels for one-to-one communication. Different types of Streams support different functions. The asynchronous event processing engine supports waking up the registered Runners through events to process the latest data, thereby realizing event-driven. In addition, it provides a certain degree of thread safety guarantee for the execution of the Process. The Query Builder converts the input query plan into an EventGraph. When constructing the EventGraph, the required Process and Runner are constructed in sequence according to the query plan, and the Runner and the internal Process are connected with the Stream. At the same time, the corresponding event state machine is constructed.
[0007] Further, the operator Process is responsible for reading data, performing joins, and sorting data.
[0008] Further, the Runner supports serial processing, parallel processing, and streaming processing of different processing methods of the Process, and nests several sub-Runners inside, thereby forming a chain structure or a tree structure of the Runner.
[0009] Correspondingly, an event-driven database asynchronous execution method includes the following steps:
[0010] Step 1: When the database generates a query plan for a query and waits for execution, first input the query plan into the Query Builder. The Query Builder will convert the query plan into an EventGraph according to the content of the query plan. The EventGraph will construct the required Process and Runner in sequence according to the query plan, and connect each Runner through the Stream. At the same time, all the generated Runners (including the parent Runner and all internal Runners) will be registered in the event processing engine. The EventGraph will be placed in a top-level parent Runner and driven by the parent Runner;
[0011] Step 2: After the query plan is converted into the parent Runner, an event is sent to the parent Runner by the outside of the parent Runner through the event processing engine, thereby starting the execution of the query;
[0012] Step 3: Obtain the information of the target Runner from this event, find the target Runner from the registered Runners, and take an idle thread from the thread pool to let the target Runner process this event;
[0013] Step 4: The target Runner records the source Runner of this event as its own parent Runner, and processes the data according to the status information carried by this event in combination with the status information of the Runner itself.
[0014] Further, in Step 1, the Runner has the following five states:
[0015] (1) Running, indicating that the Runner is processing data;
[0016] (2) WaitInput, indicating that the Runner is waiting for data input from the upstream Runner;
[0017] (3) Finished, indicating that the Runner has completed the processing of a batch of data;
[0018] (4) Destroying, indicating that the Runner is being destroyed;
[0019] (5) Destroyed, indicating that the Runner has completed the destruction.
[0020] Further, in Step 2, the event has the following five states:
[0021] (1) Proc indicates that the target Runner starts running for the first time, sent from the parent Runner to the child Runner, or sent from outside the Runner;
[0022] (2) Post indicates that the target Runner needs to continue running;
[0023] (3) Finished indicates that the source Runner has completed the processing of a batch of data, sent from the child Runner to the parent Runner; The Runner will batch and split the output data internally to control the sending timing of the Finished event, that is, initiate the event according to the data flow control. At this time, the target Runner may need to drive the subsequent Runner of the source Runner to start running, or send the Finished event to its own parent Runner when it has also completed running;
[0024] (4)Destroy indicates that the source Runner starts to be destroyed and is sent from the parent Runner to the child Runner. At this time, the target Runner should send the Destroy event to all its child Runners and start to destroy itself.
[0025] (5)Destroyed indicates that the source Runner has completed destruction and is sent from the child Runner to the parent Runner. At this time, the target Runner may need to perform the final work of destroying itself and send the Destroyed event to its parent Runner after completion.
[0026] Furthermore, in step 4, if data needs to be read from the InputStream, it is read from the InputStream. After processing, if there is output, the output data is written to the OutputStream, and the next Runner is driven to execute according to the status of the Runner itself through the event processing engine. When the Runner itself finishes execution, an event is sent to the parent Runner to notify the parent Runner, and so on until the top-level Runner finishes execution.
[0027] Correspondingly, an event-driven database asynchronous execution device, characterized in that it includes: one or more processors;
[0028] A storage device for storing one or more programs and user data;
[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the event-driven database asynchronous execution method as described in any one of claims 4 to 7.
[0030] Correspondingly, an event-driven database asynchronous execution storage medium, characterized in that a computer program is stored thereon, and when the program is executed by a processor, it implements the event-driven database asynchronous execution method as described in any one of claims 4 to 7.
[0031] Beneficial effects: Compared with the prior art, the present invention has the following remarkable advantages: By means of event-driven, it better supports parallel processing between operators and data, improving the query execution efficiency; Through the encapsulation of the Runner, the parallel, serial, streaming and other control logics inside the operator are abstracted, so that the same type of control logic operators can reuse the corresponding Runner, thus simplifying the workload of adding new operators; Decoupling elements such as data processing, data control, data sources and data streams, and concurrent control is convenient for the functional expansion of the execution engine and makes the code more concise. Description of the Drawings
[0032] Figure 1 Schematic diagram of the database asynchronous execution engine structure of the present invention.
[0033] Figure 2 Schematic diagram of the database asynchronous execution method process of the present invention.
[0034] Figure 3 Schematic diagram of the Runner state machine of the database asynchronous execution engine of the present invention.
[0035] Figure 4 Schematic diagram of the event processing engine of the database asynchronous execution engine of the present invention. Detailed implementation manners
[0036] Event-driven is a programming paradigm, and the data processing logic it contains is driven to execute as events are triggered, rather than at a specified time. In the demand-driven paradigm, the data processing logic is usually initiated from top to bottom, and the top-level processing logic actively pulls data from the bottom-level logic and processes it. The volcano model is a representative of the demand-driven paradigm, which actively pulls data from the bottom layer for processing by calling the bottom-layer iterator at the top level.
[0037] The following describes in detail the event-driven database asynchronous execution engine provided by the embodiments of the present application with reference to the accompanying drawings.
[0038] Figure 1 Schematic diagram of the query object structure of the event-driven database asynchronous execution engine provided by the embodiments of the present application, where:
[0039] S101: The query object is responsible for driving the outermost Runner to execute, obtaining the query result and error information from the outermost Runner;
[0040] S102: The outermost Runner is responsible for driving the EventGraph and executing specific business logic. The EventGraph will pass events to the corresponding sub-Runners according to the current state of the query to drive the query execution.
[0041] S103: The Runner is responsible for managing the execution of internal Runners or Processes. Several Runners or Processes can be nested within a Runner. There is a simple event state machine inside the Runner, which schedules the execution of internal Runners or Processes based on the event status sent by the event engine and its own status. There are different types of Runners, and each type of Runner represents a scheduling and execution logic, such as sequential execution, concurrent execution, streaming execution, MapReduce execution, etc., which can match operators with various different execution logics. With the encapsulation of the Runner, the complex execution logic and status inside the operator can be hidden, and only a small part of the status exposed by the Runner needs to be known externally, greatly simplifying the control code; at the same time, the abstraction of the Runner also enables operators with similar execution logics to directly reuse the Runner, reducing the workload of adding new operators.
[0042] S104: The Stream is responsible for the communication between Runners, between Processes, and between Runners and Processes. The Stream encapsulates several Channels for one-to-one communication inside. By combining and controlling the Channels, the Stream can implement functions such as one-to-many, many-to-one, read-only control, data persistence, etc., and can also hide details such as local transmission and network transmission.
[0043] S105: The Process is responsible for encapsulating the status and input / output of the operator and providing interfaces for operations such as data reading, writing, and execution. For a Runner with a simple structure, the input and output of the Process are actually the input and output stream of the Runner; for some Runners with a complex structure, the internal Processes communicate with each other through Streams that are completely hidden inside the Runner.
[0044] Figure 2 It is a schematic diagram of the working process of the event-driven database asynchronous execution engine provided by the embodiments of this application, where:
[0045] S201: Query Builder converts the input query plan into EventGraph, encapsulates Process with Runner, and connects Runner and query objects of internal Process with Stream. EventGraph can be constructed in a top-down and outside-in order, that is, the outermost Runner is constructed first, and then the sub-Runners and Processes contained in the Runner are constructed one by one according to the execution order of the query, and Stream is used to transfer data between Runners. EventGraph is wrapped with the top-level Runner and query objects.
[0046] S202: By executing query statements, calling APIs and other methods, the query object initiates an event to the outermost Runner via the event processing engine, thereby mobilizing the event state machine of the Runner to start working, and then starting the entire query. In addition to carrying status information, information about the event initiator and the event receiver, the event also carries a callback function so that the event initiator can control the subsequent processing logic according to the processing status of the event. The parameter of the callback function is also an event, which contains the processing status of the previous event. In the event initiated by the query to the outermost Runner, the callback function is the processing function of the query object itself, which returns the query results or error information and other outputs to the outside according to the query status.
[0047] S203: The event state machine of each Runner mobilizes the sub-Runner and / or the internally encapsulated Process to process data according to the state of the Runner and the input event state, and selects subsequent operations according to the processing results, which generally include passing the processing results to the lower-level Runner through the Stream, requesting new input from the upper-level Runner when the input is exhausted, and returning error information to the upper-level Runner when processing errors.
[0048] S204: When the internal Runner and Process have completed execution, or an error occurs during execution, or sufficient data has been obtained in this round of query, the outermost Runner calls the callback function provided by the query object, and this round of query is completed. To continue querying, just re-execute step S202.
[0049] Figure 3 : is a schematic diagram of the Runner state machine of the event-driven database asynchronous execution engine provided in an embodiment of the present application, wherein:
[0050] S301: Initial running state of the Runner. First, it is necessary to set data such as the error handling callback and caller information of the Runner, and then regularly execute the internal Runner / Process once. After one execution is completed, the Runner determines whether the execution logic of the entire Runner is completed based on the result of this execution and the internal state, updates its own state of the Runner, and executes the callback function. If an error occurs during the internal execution of the Runner, it is handed over to the error handling callback for processing.
[0051] S302: Continuing running state of the Runner. Continuously execute the internal Runner / Process once.
[0052] S303: Completion state of the Runner. After performing some finishing work, through the caller information recorded during the initial run, execute the callback function provided by the caller to notify the caller.
[0053] S304: Starting destruction state of the Runner. After updating the caller information, notify the internal Runner / Process to start destruction and deconstruction.
[0054] S305: Completion state of the destruction of the Runner. Execute the callback function provided by the new caller to notify that the destruction has been completed.
[0055] Figure 4 It is a schematic diagram of the event processing engine of the event-driven database asynchronous execution engine provided by the embodiments of the present application, where:
[0056] S401: In the event object, in addition to the event state and callback function, there is also an ID that identifies the event processor, and each event processor is identified by a unique ID.
[0057] S402: EventQueue is a global event queue that stores all event objects that have been initiated and are pending distribution to event processors. Optionally, EventQueue can encapsulate multiple event queues with different priorities, and when retrieving events, it preferentially retrieves from the queue with the highest priority. Only when the high-priority queue is empty will it retrieve event objects from the lower-priority queue. EventQueue also maintains a list of registered event processors. EventQueue is responsible for matching the event object with the corresponding processor by ID when adding an event, so as to determine the target of event distribution. The Runner is also an event processor, and the registration of the Runner is performed when the QueryBuilder creates the Runner.
[0058] S403: The event processing thread is responsible for continuously retrieving event objects from EventQueue for distribution and executing the processing logic of the event processor.
[0059] In the present invention, the encapsulation of the Runner for scheduling execution of the Process, the encapsulation of the event engine for concurrency control, and the encapsulation of the Stream for data splitting and aggregation can effectively support the operators for parallel computing and reduce the workload.
Claims
1. An event-driven database asynchronous execution engine, characterized in that: include: Operator Process that implements database functions, Runner for encapsulating Process processing logic, Stream for transferring data between Runners, asynchronous event processing engine for driving Runner execution, and Query Builder that generates Runner structure according to query plan; The operator Process encapsulates the execution status and input and output Streams of the operator, and provides a standard interface for the upper-level Runner to call. The Runner responds to external events through a simple state machine to mobilize the operator to process data. The Stream encapsulates several Channels for one-to-one communication. Different types of Streams support different functions. The asynchronous event processing engine supports waking up the registered Runner through events to process the latest data to achieve event-driven. In addition, it also provides a certain degree of thread safety guarantee for the execution of the Process. The Query Builder converts the input query plan into an EventGraph. When building the EventGraph, the required Process and Runner are built in sequence according to the query plan, and the Runner and internal Process are connected with the Stream, and the corresponding event state machine is built at the same time.
2. The event-driven database asynchronous execution engine according to claim 1, characterized in that: The operator Process is responsible for reading data, performing connections, and sorting data.
3. The event-driven database asynchronous execution engine according to claim 1, characterized in that: Runner supports processes with different processing methods, such as serial processing, parallel processing, and streaming processing. It nests several sub-Runners inside to form a chain structure or tree structure of Runner.
4. A method for executing the event-driven database asynchronous execution engine according to claim 1, characterized in that: The steps include: Step 1: When the database generates a query plan for a query and waits for execution, the query plan is first input into the Query Builder. The Query Builder converts the query plan into an EventGraph based on the content of the query plan. The EventGraph sequentially builds the required Process and Runner according to the query plan, and connects each Runner through a Stream. At the same time, all generated Runners are registered with the event processing engine. The EventGraph is placed in a top-level parent Runner and driven by the parent Runner. Step 2: After the query plan is converted to the parent Runner, an event is sent to the parent Runner through the event processing engine outside the parent Runner, thereby starting the query execution; Step 3: Get the target Runner information from the event, find the target Runner from the registered Runners, and take an idle thread from the thread pool to let the target Runner process the event; Step 4: The target Runner records the source Runner of the event as its parent Runner, and processes the data based on the status information carried by the event and the Runner's own status information.
5. The execution method of the event-driven database asynchronous execution engine according to claim 1, characterized in that: Furthermore, in step 1, the Runner has the following five states: (1) Running, indicating that the Runner is processing data; (2) WaitInput, indicating that the Runner is waiting for data input from the upstream Runner; (3) Finished, indicating that the Runner has completed processing a batch of data; (4)Destroying, indicating that the Runner is being destroyed; (5) Destroyed, indicating that the Runner has been destroyed.
6. The execution method of the event-driven database asynchronous execution engine according to claim 1, characterized in that: In step 2, the event has the following five states: (1) Proc indicates that the target Runner starts running for the first time, and is sent from the parent Runner to the child Runner, or from outside the Runner; (2) Post indicates that the target Runner needs to continue running; (3) Finished indicates that the source Runner has completed processing a batch of data, which is sent by the child Runner to the parent Runner. The Runner will batch and split the output data internally to control the timing of sending the Finished event, that is, to initiate an event based on data flow control. At this time, the target Runner may need to drive the subsequent Runners of the source Runner to start running, or send a Finished event to its parent Runner when it has completed running itself. (4) Destroy indicates that the source Runner begins to be destroyed. It is sent by the parent Runner to the child Runner. At this time, the target Runner should send a Destroy event to all its child Runners and begin to destroy itself. (5) Destroyed indicates that the source Runner has been destroyed and is sent by the child Runner to the parent Runner. At this time, the target Runner may need to perform the finishing work of destroying itself and send a Destroyed event to its parent Runner after completion.
7. The execution method of the event-driven database asynchronous execution engine according to claim 1, characterized in that: In step 4, if data needs to be read from the InputStream, it is read from the InputStream. After processing, if output is generated, the output data is written to the OutputStream, and according to the state of the Runner itself, the event processing engine drives the next Runner to execute. When the Runner itself finishes executing, an event is sent to the parent Runner to notify the parent Runner, and so on, until the top-level Runner finishes executing.
8. An event-driven database asynchronous execution device, characterized in that: include: one or more processors; A storage device for storing one or more programs and user data; When the one or more programs are executed by one or more processors, the one or more processors implement the event-driven database asynchronous execution method as described in any one of claims 4 to 7.
9. An event-driven database asynchronous execution storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the event-driven database asynchronous execution method as claimed in any one of claims 4 to 7 is implemented.