Data processing methods, electronic devices, computer-readable media and computer program products
By introducing an asynchronous coroutine mechanism into the database system, the production and consumption of batch data are decoupled, which solves the problems of CPU resource waste and pipeline lag in the Volcano model, and improves data processing efficiency and system stability.
Patent Information
- Application Number
- CN202611615917.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
Traditional volcano-based database engines suffer from high CPU resource consumption, low instruction and data cache hit rates, difficulty in utilizing SIMD acceleration capabilities, and pipeline stuttering and low resource utilization due to blocking operators when processing big data.
An asynchronous coroutine mechanism is adopted. The first coroutine extracts and reconstructs batch data from the target storage space and stores it in a shared queue. The second coroutine asynchronously consumes data from the shared queue, thereby decoupling the production and consumption of batch data and reducing pipeline lag and context switching overhead.
It significantly improves CPU utilization and execution efficiency, reduces pipeline lag, increases data processing throughput and system fairness, and solves the problem of low resource utilization in the traditional volcano model.
Smart Images

Figure CN123114801A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, electronic device, computer-readable medium, and computer program product. Background Technology
[0002] With the development of big data technology, enterprise data scale is growing exponentially, placing higher demands on the real-time analysis capabilities of database systems. Traditional database engines based on the Volcano Model employ a row-by-row processing mode, resulting in a significant consumption of CPU resources on query tree traversal and frequent function calls. This leads to excessively deep call stacks and low cache hit rates for instructions and data, making it difficult to effectively utilize the acceleration capabilities of Single Instruction Multiple Data (SIMD). Therefore, engineers have developed a vectorized execution engine specifically for the Volcano Model.
[0003] In the batch data processing flow of a vectorized execution engine, operators such as sorting and materialization (referred to as blocking operators in this paper) typically store data in memory or spill it to disk in row form. After performing the relevant processing steps, these data are then reconstructed into batch data for consumption by upper-level operators. Because the vectorized engine relies on the batch processing mechanism, upper-level operators often need to synchronously wait for the blocking operators to complete the reconstruction of the batch data when pulling data, causing the batch data processing pipeline to stall. Summary of the Invention
[0004] This disclosure provides a data processing method, an electronic device, a computer-readable medium, and a computer program product.
[0005] In a first aspect, embodiments of this disclosure provide a data processing method, including:
[0006] The first coroutine extracts the data to be processed from the target storage space and reconstructs it to obtain batch data, and stores the batch data into a shared queue. The first coroutine is a coroutine in an asynchronous thread.
[0007] The second coroutine retrieves the batch data from the shared queue for consumption. The second coroutine is a coroutine within the mainline thread. The mainline thread and the asynchronous thread are independent threads. The mainline thread is used to perform structured query language computation. The mainline thread includes its own independent second coroutine. The asynchronous thread is used to asynchronously construct the batch data. The shared queue is used to implement data transfer between the first coroutine and the second coroutine.
[0008] Secondly, embodiments of this disclosure provide an electronic device, including a memory and a processor; the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements the data processing method as described above.
[0009] Thirdly, embodiments of this disclosure provide a computer-readable medium storing a computer program that, when executed by a processor, implements the data processing method as described above.
[0010] Fourthly, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the data processing method as described above.
[0011] The data processing method provided in this embodiment involves a first coroutine extracting the data to be processed from the target storage space and reconstructing it to obtain batch data, which is then stored in a shared queue. A second coroutine retrieves the batch data from the shared queue for consumption. The first and second coroutines can run asynchronously. The first coroutine generates batch data and stores it in the shared queue, while the second coroutine can directly retrieve the batch data from the shared queue without waiting. This reduces or even avoids pipeline blockages in data processing, eliminates the need for thread scheduling, avoids additional overhead, significantly reduces context switching overhead, and thus greatly improves CPU utilization and execution efficiency. Attached Figure Description
[0012] Figure 1 A flowchart illustrating thread execution in a database system based on the volcano model is shown.
[0013] Figure 2 A schematic diagram illustrating the data processing method performed according to embodiments of the present disclosure is shown.
[0014] Figure 3 A flowchart of a data processing method provided by an embodiment of this disclosure is shown.
[0015] Figure 4 A schematic diagram of the first and second coroutines cooperating to process tasks is shown in an embodiment of this disclosure.
[0016] Figure 5 A schematic diagram of the scheduling strategy for asynchronous threads in an embodiment of this disclosure is shown.
[0017] Figure 6 A flowchart illustrating the interaction between the first and second coroutines in an embodiment of this disclosure is shown.
[0018] Figure 7 A schematic diagram of the scheduling strategy for mainstream waterline threads in an embodiment of this disclosure is shown.
[0019] Figure 8 This is a schematic diagram of the composition of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0020] To enable those skilled in the art to better understand the technical solutions of this disclosure, the data processing method, electronic device, computer-readable medium, and computer program product provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.
[0021] The present disclosure will be described more fully below with reference to the accompanying drawings; however, the embodiments shown may be embodied in different forms, and the present disclosure should not be construed as limited to the embodiments set forth below. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of the disclosure.
[0022] The accompanying drawings are provided to further illustrate this disclosure and form part of the specification. They are used together with the detailed embodiments to explain this disclosure and do not constitute a limitation thereof. These and other features and advantages will become more apparent to those skilled in the art from the description of detailed embodiments with reference to the accompanying drawings.
[0023] Unless otherwise specified, each embodiment and feature of this disclosure may be used individually or in combination with other embodiments and features thereof.
[0024] Those skilled in the art will understand that various changes in form and detail may be made to the embodiments of this disclosure without departing from the scope of this disclosure as set forth by the appended claims.
[0025] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. The term "and / or" as used in this disclosure includes any and all combinations of one or more of the associated enumerated entries. The singular forms "a" and "the" as used in this disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. The terms "comprising," "made of," etc., as used in this disclosure specify the presence of the stated feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.
[0026] Unless otherwise specified, all terms used in this disclosure (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined in this disclosure.
[0027] This disclosure is not limited to the embodiments shown in the accompanying drawings, but includes modifications to the configuration based on the manufacturing process. Therefore, the areas illustrated in the drawings are schematic, and the shapes of the areas shown illustrate specific shapes of areas of an element, but are not intended to be limiting.
[0028] Figure 1 A flowchart illustrating thread execution in a database system based on the volcano model is shown. Figure 1 As shown, the user-submitted Structured Query Language (SQL) request is processed by parser 10 to generate a syntax tree, then by optimizer 20 to convert it into an execution plan tree, and finally executed by executor 30. Data reading during execution is driven by scan operators and retrieved directly from storage engine 40.
[0029] The execution plan tree consists of a root node and at least one level of child nodes. Each child node includes an operator, and different operators consume different amounts of CPU resources. The traditional volcano model uses a classic iterative execution mechanism, where upper-level child nodes in the execution plan tree recursively call the get_next() interface to pull data down. After the scanning operator scans the data, it then calculates the data along the execution plan tree through each child node and returns up layer by layer.
[0030] The steps executed by executor 30 of the volcano model include:
[0031] Step 1: Aggregation operator 31 calls get_next() to request batch data from sorting operator 32;
[0032] Step 2: The sorting operator 32 requests batch data from the scanning operator 33;
[0033] Step 3: After the scanning operator 33 retrieves the data to be processed from the disk, it pushes it to the sorting operator 32.
[0034] Step 4: After the sorting operator 32 obtains the data to be processed, it sorts the data and stores the sorted data in the form of rows in memory or on disk.
[0035] Step 5: After sorting by sorting operator 32, the data to be processed is retrieved from memory or disk and reconstructed into batch data, which is then pushed to aggregation operator 31.
[0036] Step 6: After receiving the batch data, the aggregation operator 31 performs aggregation calculations.
[0037] After sorting is complete, sorting operator 32 needs to retrieve the data to be processed from memory or disk in row format and reconstruct it into batch data in column format before pushing it to aggregation operator 31. Therefore, the pipeline must wait for the data reconstruction to complete. This situation, where upper-level operators need to wait for lower-level operators to complete data reconstruction before execution, can easily cause pipeline stuttering, especially when dealing with variable-length fields such as Text, where the high memory copy overhead will further exacerbate the stuttering.
[0038] In a multi-core processor architecture, multiple threads within the same process can be scheduled by the operating system to different processor cores, enabling parallel hardware execution. While asynchronous multi-threaded processing can alleviate pipeline stuttering to some extent, the context switching caused by thread scheduling (typically with time slices of 10ms-30ms) introduces significant overhead. Moreover, time-consuming operators monopolize CPU resources for extended periods, leading to starvation of other tasks and limiting concurrent processing capabilities.
[0039] This disclosure addresses the problems of the classic iterative execution mechanism of the volcano model by proposing a data processing method that can reduce or even avoid pipeline stalls.
[0040] Figure 2 A schematic diagram illustrating the data processing method performed according to embodiments of the present disclosure is shown. Figure 2 As shown, the main thread scheduling module 100 provides an independent service thread for each concurrent SQL request. This service thread serves as the main pipeline thread, which is used to execute SQL calculations. Each main pipeline thread has its own independent coroutine (referred to as the second coroutine in this embodiment). The second coroutine is used to execute tasks in the main pipeline and to actively pause SQL calculations, relinquishing the CPU used for SQL calculations. It should be noted that although only three main pipeline threads, SQL1, SQL2, and SQL3, and their corresponding second coroutines C1, C2, and C3 are shown in the figure, this does not imply a limitation on the number of main pipeline threads and second coroutines.
[0041] The data construction module 200 provides a coroutine for asynchronous threads (referred to as the first coroutine in this embodiment) to serve the blocking operator for each SQL thread. This allows for the asynchronous construction of batch data, such as the batch data required by sorting and materialization operators, and the pushing of the batch data to the corresponding shared queue. In this embodiment, operators that consume CPU resources exceeding a certain threshold are called blocking operators. The batch data meets the columnar data structure required by the vectorized execution engine.
[0042] In this embodiment, the asynchronous thread and the mainline thread are independent and separate threads that can be executed in parallel to asynchronously build batch data, thereby decoupling the construction of batch data from the mainline thread.
[0043] In practical applications, those skilled in the art can run the first and second coroutines in other independent scheduling units / thread models as needed. For example, the first coroutine can be hosted in an asynchronous thread, and the second coroutine in a mainstream thread. The first coroutine is used to build batch data; therefore, it can also be called a producer coroutine. The second coroutine is used to consume the batch data built by the first coroutine; therefore, it can also be called a consumer coroutine.
[0044] Task scheduling queue 300 is used to mark the current task's state as YIELD_PENDING. When mainstream pipeline threads implement a proactive yielding strategy for high CPU-intensive operators such as Join, Aggregate, and Sort, they mark the current task's state as yielding and add it to the task scheduling queue. This queue records the consumption tasks that the second coroutine needs to execute; that is, it's the queue where the consumer coroutine consumes the batch data generated by the first coroutine.
[0045] This disclosure provides a data processing method, mainly deployed in... Figure 1 The executor shown addresses the issues of pipeline stalling and low resource utilization caused by blocking operators in traditional volcano model vectorized execution engines for massive data processing scenarios.
[0046] This data processing method can be applied to a single server or a distributed database built on multiple servers with a multi-core CPU architecture. These servers' CPUs support SIMD instruction sets such as AVX2 and AVX-512 to accelerate batch processing of data using a vectorized execution engine. These servers run on Linux distributions with kernels that support coroutines and asynchronous I / O mechanisms (such as io_uring or epoll) to support coroutine suspension and resumption.
[0047] Figure 3 A flowchart of a data processing method provided by an embodiment of this disclosure is shown. Figure 3 As shown, the data processing methods include:
[0048] In step S301, the first coroutine extracts the data to be processed from the target storage space and reconstructs it to obtain batch data, and stores the batch data into a shared queue.
[0049] The target storage space includes, but is not limited to, the memory and disk of the blocking operator. When the amount of data to be processed is small, it is stored in the memory of the blocking operator. When the amount of data to be processed is large and the memory of the blocking operator cannot meet the storage requirements, the overflowing data is stored on the disk.
[0050] In this embodiment, each blocking operator is configured with a shared queue. When the first coroutine generates batch data, the batch data is stored in the corresponding shared queue. The shared queue is used to transfer data between the first coroutine and the second coroutine.
[0051] In some embodiments, blocking operators include one or more of the following: join operators, sorting operators, scan operators, aggregation operators, and materialization operators. The join operator is used to merge data from two or more tables based on association conditions; the sorting operator is used to sort resource-intensive data; the scan operator is the starting point for query operations and is responsible for reading data from disk or memory; the aggregation operator is used to group and statistically analyze the data using a hash table or sorting structure; and the materialization operator is used for nested loop join operations to store intermediate results so that subsequent operators can repeatedly read or randomly access them.
[0052] In step S302, the second coroutine retrieves batch data from the shared queue for consumption.
[0053] In this embodiment, the first coroutine is used to asynchronously construct the batch data required by the blocking operator; the second coroutine is used to complete the calculation of the main watershed of the execution plan. The first and second coroutines have the advantages of low context switching cost and fine-grained control with arbitrary yield / Await.
[0054] In this embodiment, the second coroutine directly pulls the pre-built batch data from the shared queue without waiting for the blocking operator to complete the full data processing. This achieves complete decoupling of batch data production and consumption, enabling the mainstream pipeline threads to quickly complete consumption tasks. This significantly reduces CPU idle time caused by I / O waiting or data reconstruction, greatly improves the throughput of consumption tasks, reduces stuttering in the mainstream pipeline threads, and reduces the occupation of CPU resources.
[0055] The data processing method provided in this embodiment involves a first coroutine extracting and reconstructing data to be processed from a target storage space to obtain batch data, which is then stored in a shared queue. A second coroutine retrieves the batch data from the shared queue for consumption. The first and second coroutines can run asynchronously. The first coroutine generates the batch data and stores it in the shared queue, while the second coroutine can directly retrieve the batch data from the shared queue without waiting. This reduces or even avoids pipeline blockages in data processing, eliminates the need for thread scheduling, avoids additional overhead, significantly reduces context switching overhead, and thus greatly improves CPU utilization and execution efficiency.
[0056] In some embodiments, the first coroutine is a coroutine in an asynchronous thread, and the second coroutine is a coroutine in a mainline thread. The mainline thread and the asynchronous thread are independent threads, therefore, the first coroutine and the second coroutine can be executed asynchronously. For example, the first coroutine is used to asynchronously construct the batch data required by the blocking operator, and the second coroutine is used to perform the computational tasks in the mainline thread using the batch data.
[0057] In some embodiments, the data to be processed is row data, and the batch data is column data. Row data and column data are data formats stored in memory or on disk. Row data stores all field values of a row record together in sequence, and column data stores all values of a column together in sequence.
[0058] The first coroutine extracts and reconstructs the data to be processed from the target storage space to obtain batch data, and stores the batch data into a shared queue. This includes: in response to the target storage space storing data to be processed, the first coroutine extracts row data from the target storage space and reconstructs it to obtain column data, and stores the column data into a shared queue.
[0059] When the mainline thread detects that the target storage space corresponding to the blocking operator has stored data to be processed, that is, the blocking operator has completed row-format storage, the first goroutine reads a certain number of row data from the target storage space in sequence, performs row-to-column conversion, reconstructs the row data into column data required by the vectorization engine, and then stores the column data into the shared queue.
[0060] For example, Figure 4 A schematic diagram illustrating the collaborative task processing of a first coroutine and a second coroutine in an embodiment of this disclosure is shown. Figure 4 As shown, the execution plan tree consists of multiple child nodes, each corresponding to an operator. Operators include scan operators, materialization operators, nested loop join operators, sorting operators, and aggregation operators. The solid lines with solid arrows in the diagram represent the driving flow. The main pipeline thread traverses the execution plan tree starting from the aggregation operator. The aggregation operator requests data to be processed from the sorting operator, the sorting operator requests data to the nested loop join operator, the nested loop join operator requests data to either the scan operator or the aggregation operator, and the aggregation operator requests data to the scan operator. The solid lines with simplified arrows in the diagram represent the data flow. After the scan operator receives the data to be processed, it passes the data upwards layer by layer, finally passing it to the aggregation operator.
[0061] For example, a materialization operator extracts data to be processed from a scanning operator and stores it in row format in its memory. If the materialization operator's memory cannot hold the data, the overflowing data can be written to disk. Similarly, a sorting operator extracts data to be processed from a nested loop join operator and stores it in row format in its memory. If the sorting operator's memory cannot hold the data, the overflowing data can be written to disk.
[0062] When the main stream thread detects that an operator has stored the data to be processed in row format in the operator's memory and disk, the first goroutine reads a certain amount of data to be processed from memory and disk, reconstructs it, and stores the reconstructed column data into the shared queue corresponding to the sorting operator.
[0063] For example, taking the sorting operator as an example, when the main stream thread detects that the nested loop join operator has stored the data to be processed in row format in the memory and disk of the sorting operator, the first goroutine reads a certain amount of data to be processed from the memory and disk of the sorting operator, reconstructs it, and stores the reconstructed column data into the shared queue corresponding to the sorting operator.
[0064] For example, taking the materialization operator as an example, when the main waterline thread detects that the scan operator has stored the data to be processed in row format in the memory and disk of the materialization operator, the first coroutine reads a certain amount of data to be processed from the memory and disk of the materialization operator, reconstructs it, and stores the reconstructed column data into the shared queue corresponding to the scan operator.
[0065] In some embodiments, when the shared queue reaches its maximum capacity, the first coroutine voluntarily suspends itself, marks the current refactoring task as yielded, and records the first suspension point. In response to a first event wake-up command initiated by the second coroutine, the first coroutine resumes execution of the refactoring task from the first suspension point.
[0066] For example, when the shared queue corresponding to the sorting operator reaches its capacity limit, the first coroutine immediately performs a suspension operation, pauses the current reconstruction task, and yields the CPU to avoid long waits caused by invalid CPU polling. Simultaneously, the state of the current reconstruction task is marked as yielded, and the first suspension point is recorded so that the current reconstruction task can be resumed from the first suspension point later.
[0067] Figure 5 A schematic diagram of the scheduling strategy for asynchronous threads in an embodiment of this disclosure is shown. For example... Figure 5As shown, assume that the asynchronous threads include the first coroutine M1, the first coroutine M2, and the first coroutine M3. Each first coroutine corresponds to a shared queue, that is, the first coroutine M1 corresponds to shared queue A, the first coroutine M2 corresponds to shared queue B, and the first coroutine M3 corresponds to shared queue C. Each shared queue serves one mainstream thread, that is, shared queue A serves mainstream thread SQL1, shared queue B serves mainstream thread SQL2, and shared queue C serves mainstream thread SQL3.
[0068] When the shared queue A corresponding to the mainstream thread SQL1 reaches its capacity limit, the first goroutine M1 is immediately suspended, marking the current refactoring task status as yielded and recording the first suspension point A, voluntarily yielding the CPU. The CPU then executes the refactoring task of the first goroutine M2. When the shared queue B corresponding to the mainstream thread SQL2 reaches its capacity limit, the first goroutine M2 is immediately suspended, marking the current refactoring task status as yielded and recording the first suspension point B, voluntarily yielding the CPU. The CPU then executes the refactoring task of the first goroutine M3. When the shared queue C corresponding to the mainstream thread SQL3 reaches its capacity limit, the first goroutine M3 is immediately suspended, marking the current refactoring task status as yielded and recording the first suspension point C, voluntarily yielding the CPU. The CPU then executes the refactoring task of either the first goroutine M1 or M2.
[0069] When the amount of data in shared queue A reaches the restart condition, the mainline thread SQL1 sends a first event wake-up command to the first goroutine M1, and the first goroutine M1 continues to execute the reconstruction task from the first suspended point A. Similarly, when the amount of data in shared queue B reaches the restart condition, the mainline thread SQL2 sends a first event wake-up command to the first goroutine M2, and the first goroutine M2 continues to execute the reconstruction task from the first suspended point B. When the amount of data in shared queue C reaches the restart condition, the mainline thread SQL3 sends a first event wake-up command to the first goroutine M3, and the first goroutine M3 continues to execute the reconstruction task from the first suspended point C.
[0070] In some embodiments, the restart conditions include, but are not limited to, the shared queue being empty or the data volume being less than a preset first data volume threshold. The first data volume threshold can be set by the user, and this embodiment does not limit it.
[0071] When the shared queue is empty or the amount of data is less than the preset first data volume threshold, the second coroutine sends a first event wake-up command to the first coroutine.
[0072] During the process of the second coroutine consuming batch data in the shared queue, the batch data in the shared queue continuously decreases. When the shared queue is empty or the data volume is less than the preset first data volume threshold, the first event wake-up command is initiated so that the first coroutine can continue to execute the reconstruction task from the first suspension point and fill the shared queue with the newly generated batch data.
[0073] In some embodiments, when the shared queue is empty, the second coroutine actively suspends itself, marks the current consumption task status as yielded, and records the second suspension point; in response to the second event wake-up command initiated by the first coroutine, the second coroutine continues to execute the consumption task from the second suspension point.
[0074] When the shared queue is empty, the second coroutine has no batch data to consume. The second coroutine voluntarily suspends itself, releasing the CPU, which can then execute the consumption tasks of other second coroutines. When the second coroutine receives the second event wake-up command, it resumes the consumption task from the second suspension point.
[0075] In some embodiments, the second event wake-up instruction is an instruction initiated by the first coroutine when the shared queue reaches its maximum capacity.
[0076] Figure 6 A flowchart illustrating the interaction between the first and second coroutines in an embodiment of this disclosure is shown. Figure 6 As shown, the interaction steps between the first and second coroutines include:
[0077] Step S601: The first coroutine reads the data to be processed from the memory or disk of the blocking operator.
[0078] The data to be processed stored in the memory or disk of the blocking operator is row data.
[0079] In step S602, the first coroutine reconstructs the data to be processed, converting row data into batch data and storing the batch data in a shared queue.
[0080] The batch data obtained from the first coroutine reconstruction is column data.
[0081] In step S603, when the shared queue is full, the first coroutine sends a second event wake-up command to the second coroutine, and the first coroutine suspends the reconstruction task, actively yields the CPU, marks the status of the reconstruction task as yielded, and records the position of the first suspension point.
[0082] The second event wake-up command can be either epoll or io_uring.
[0083] In step S604, the second coroutine retrieves batch data from the shared queue and executes the corresponding consumption task.
[0084] The consumption task can be to directly transmit batch data to the upper-level operator, or it can be to perform the corresponding calculations before uploading to the upper-level operator. The consumption task is determined by the function of the blocking operator, and this embodiment does not limit it.
[0085] In step S605, when the shared queue is empty, the second coroutine sends a first event wake-up command to the first coroutine, suspends the consumption task, actively yields the CPU, marks the status of the consumption task as yielded, and records the position of the second suspension point.
[0086] In step S606, after receiving the first event wake-up command, the first coroutine continues to execute the reconstruction task from the first hangup point and stores the reconstructed batch data into the shared queue.
[0087] In step S607, when the shared queue is full again, the first coroutine sends a second event wake-up command to the second coroutine, and the first coroutine suspends the reconstruction task, actively yields the CPU, marks the status of the reconstruction task as yielded, and records the position of the first suspension point.
[0088] In step S608, the second coroutine retrieves batch data from the shared queue and continues to execute the consumption task from the second hang-up point.
[0089] The interaction process between the first and second coroutines realizes a closed loop of "stopping when full and calling when empty" for the shared queue, actively yielding the CPU for other tasks to use, thereby improving the utilization of CPU resources.
[0090] To prevent time-consuming blocking operators such as join, aggregation, and sorting operators from occupying the CPU for extended periods, the second coroutine can proactively suspend itself while these blocking operators are executing, thus relinquishing the CPU and preventing other concurrent consumer tasks from "starving" due to prolonged CPU deprivation. Consumer tasks proactively suspended by the second coroutine can be re-added to the task scheduling queue. During the next scheduling cycle, these tasks will be retrieved and resumed from the second suspension point. The task scheduling queue records the consumer tasks that the second coroutine needs to execute.
[0091] If the amount of data processed by the second coroutine reaches the preset second data volume threshold, or the running time reaches the preset time slice threshold, the second coroutine will be actively suspended, and the second suspension start point will be recorded. The consumption task will be put back into the task scheduling queue. The mainline thread will periodically traverse the task scheduling queue, take out the consumption tasks in the yielded state, and continue to execute the consumption tasks from the second suspension start point.
[0092] The duration of the time slice threshold can be set by the user. This embodiment does not limit this, such as a time slice threshold of 10ms-30ms.
[0093] When the amount of data processed by the second coroutine reaches a preset second data volume threshold, or when the running time reaches a preset time slice threshold, the second coroutine proactively suspends the consumption task and places it in the task scheduling queue, no longer exclusively occupying the CPU, thus achieving fine-grained fair scheduling. Under high-concurrency mixed loads, long-tail consumption tasks proactively release the CPU, preventing high-time-consuming blocking operators from occupying the CPU for extended periods, ensuring that short-term consumption tasks can be processed promptly, and reducing latency. While ensuring the performance of blocking operators, this effectively solves the task starvation problem caused by resource contention, significantly improving the system's fairness, stability, and service quality.
[0094] Figure 7 A schematic diagram of the scheduling strategy for mainstream waterline threads in an embodiment of this disclosure is shown. For example... Figure 7 As shown in the figure, there are three main waterline threads, SQL1, SQL2, and SQL3. Each main waterline thread corresponds to a second coroutine. For example, the main waterline thread SQL1 corresponds to the second coroutine C1, the main waterline thread SQL2 corresponds to the second coroutine C2, and the main waterline thread SQL3 corresponds to the second coroutine C3. Shared queues A, B, and C correspond to the main waterline threads SQL1, SQL2, and SQL3, respectively.
[0095] Suppose that when the mainline thread SQL1 is executing a CPU-intensive consumption task, such as a task where it performs a sorting operator, when SQL1 finishes processing a batch of data or its execution time reaches a preset time slice threshold, the corresponding second coroutine C1 is proactively suspended, pausing the sorting task. The sorting task's state is marked as YIELD_PENDING, and it is re-added to the task scheduling queue. In the next scheduling cycle, the executor traverses this task scheduling queue; if it finds a task with a YIELD_PENDING state, it retrieves it and resumes execution. This time slice mechanism ensures that long-tail tasks do not monopolize the CPU for extended periods, guaranteeing that other concurrent tasks can obtain execution resources in a timely manner, thus completely solving the task "starvation" problem.
[0096] In some embodiments, before the first coroutine extracts and reconstructs the data to be processed from the target storage space to obtain the batch data, the method further includes: traversing from the root node of the execution plan tree, creating a shared queue if it is determined that the child nodes of the execution plan tree contain blocking operators, and obtaining an idle coroutine from the coroutine pool of the asynchronous thread as the first coroutine. Each blocking operator corresponds to a shared queue and a first coroutine. The execution plan tree is obtained based on the parsing result of the SQL statement.
[0097] For example, taking a database system based on the volcano model, a user submits an SQL statement, which is parsed by a parser to generate a syntax tree. The optimizer then transforms the syntax tree to obtain an execution plan tree. The executor traverses the execution plan tree starting from the root node. When it encounters a blocking operator, it creates an independent shared queue for that blocking operator and obtains an idle coroutine from the coroutine pool of asynchronous threads as the first coroutine for that blocking operator.
[0098] In some embodiments, during the database startup phase, the executor can be initialized, a coroutine pool can be created, and a shared queue can be initialized. In this embodiment, based on the underlying thread model, a dedicated user-mode coroutine pool is created specifically for handling asynchronous data reconstruction tasks of blocking operators. The size of the coroutine pool, i.e., the maximum number of concurrent first coroutines, can be dynamically adjusted through user configuration options to adapt to different scenarios.
[0099] In some embodiments, after the database starts, a mainline thread is started. This mainline thread recursively calls `get_next_batch()` from top to bottom, starting from the root node of the execution plan tree, to drive the execution of child nodes. When a blocking operator is detected, the following preprocessing is performed: receiving batch data from the lower-level operator, unpacking the columnar batch data, writing it to memory in row format, and writing the overflowing data to a temporary disk file when memory is full; and retrieving an idle coroutine from the coroutine pool as the first coroutine for the blocking operator.
[0100] For example, taking the Materialize operator as an example, the Materialize operator receives columnar batch data from the right table of the Nested Loop Join, unpacks the columnar batch data, and writes it into the materialize operator's dedicated memory in row format. When the memory is full, the remaining data is written to a temporary disk file.
[0101] For example, taking a sorting operator as an example, the sorting operator receives columnar batch data from the connection operator, quickly sorts it in memory, and stores the sorted results in row format in the sorting operator's dedicated memory. If the amount of data in memory exceeds the memory threshold, external sorting is performed, and the sorting results are written to a temporary file on disk.
[0102] This disclosure also provides an electronic device, Figure 8 This is a schematic diagram of the composition of an electronic device provided in an embodiment of this disclosure, such as... Figure 8As shown, the electronic device includes: at least one processor 801; at least one memory 802; and one or more I / O interfaces 803 connected between the processor 801 and the memory 802; wherein the memory 802 stores one or more computer programs that can be executed by at least one processor 801, and the one or more computer programs are executed by at least one processor 801 to enable at least one processor 801 to perform any of the above-described data processing methods.
[0103] In this embodiment of the disclosure, the processor 801 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory is a device with data storage capabilities, including but not limited to random access memory (RAM), more specifically such as SDRAM, DDR, etc., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface, or read-write interface, is connected between the processor and the memory, enabling information interaction between the memory and the processor, including but not limited to a data bus (Bus).
[0104] This disclosure also provides a computer-readable medium, which includes a computer program that, when executed by a processor, implements any of the aforementioned data processing methods.
[0105] This disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the aforementioned data processing methods.
[0106] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0107] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.
Claims
1. A data processing method, characterized in that, It includes: The first coroutine extracts the data to be processed from the target storage space and reconstructs it to obtain batch data, and stores the batch data into a shared queue. The first coroutine is a coroutine in an asynchronous thread. The second coroutine retrieves the batch data from the shared queue for consumption. The second coroutine is a coroutine within the mainline thread. The mainline thread and the asynchronous thread are independent threads. The mainline thread is used to perform structured query language computation. The mainline thread includes its own independent second coroutine. The asynchronous thread is used to asynchronously construct the batch data. The shared queue is used to implement data transfer between the first coroutine and the second coroutine.
2. The method according to claim 1, characterized in that, Also includes: When the shared queue reaches its maximum capacity, the first coroutine is actively suspended, the current reconstruction task is marked as yielded, and the first suspension point is recorded. In response to the first event wake-up command initiated by the second coroutine, the first coroutine continues to execute the reconstruction task from the first suspension point.
3. The method according to claim 2, characterized in that, When the shared queue is empty or the amount of data is less than a preset first data volume threshold, the second coroutine sends the first event wake-up command to the first coroutine.
4. The method according to claim 1, characterized in that, Also includes: When the shared queue is empty, the second coroutine is actively suspended, marking the current consumption task's status as yielded and recording the second suspension point; In response to the second event wake-up command initiated by the first coroutine, the second coroutine continues to execute the consumption task from the second suspension point.
5. The method according to claim 4, characterized in that, The second event wake-up instruction is an instruction initiated by the first coroutine when the shared queue reaches its maximum capacity.
6. The method according to claim 2, characterized in that, Also includes: If the amount of data processed by the second coroutine reaches a preset second data volume threshold, or the running time reaches a preset time slice threshold, the second coroutine is actively suspended, the second suspension point is recorded, and the consumption task is put back into the task scheduling queue. The task scheduling queue is used to record the queue of the consumption tasks that the second coroutine needs to execute; The mainline thread periodically traverses the task scheduling queue, retrieves the consumer tasks in the yielded state, and continues to execute the consumer tasks from the second suspended starting point.
7. The method according to claim 1, characterized in that, The data to be processed is row data, and the batch data is column data; The first coroutine extracts the data to be processed from the target storage space and reconstructs it to obtain batch data, and stores the batch data into a shared queue, including: In response to the target storage space storing data to be processed, the first coroutine extracts the row data from the target storage space and reconstructs it to obtain the column data, and stores the column data into the shared queue.
8. The method according to claim 1, characterized in that, Before the first coroutine extracts the data to be processed from the target storage space and reconstructs it to obtain batch data, it also includes: Starting from the root node of the execution plan tree, if it is determined that the child node of the execution plan tree contains a blocking operator, the shared queue is created, and an idle coroutine is obtained from the coroutine pool of the asynchronous thread as the first coroutine. Each blocking operator corresponds to one shared queue and one first coroutine. The execution plan tree is obtained based on the parsing result of the SQL statement.
9. An electronic device, characterized in that, It includes a memory and a processor; the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements the data processing method according to any one of claims 1 to 8.
10. A computer-readable medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1 to 8.
11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1 to 8.
