Data processing method and device of database node, computer device, storage medium and program product

Abstract

The application relates to a data processing method and device of a database node, computer equipment, a storage medium and a program product. The method comprises the following steps: obtaining a target execution plan fragment corresponding to a coordination node according to a database query request received by the coordination node; performing task encapsulation on the target execution plan fragment based on global snapshot information to obtain a target execution task corresponding to the target execution plan fragment; in the case that state information of a first resource group of the coordination node meets a preset state condition, performing local asynchronous execution on the target execution task according to the global snapshot information in the target execution task through a local thread pool of the coordination node to obtain a local execution result of the target execution task; and fusing the local execution result with execution results of each execution plan fragment associated with the database query request to obtain a target execution result corresponding to the database query request. The method can improve the node resource utilization rate of a distributed database.

Description

Technical Field

[0001] This application relates to the field of distributed database technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and computer program product for a database node. Background Technology

[0002] In recent years, the scale of data and the number of concurrent requests have grown exponentially, and more and more scenarios have begun to use distributed database technology to improve the efficiency of data querying.

[0003] Traditional distributed databases typically employ a Shared-Nothing architecture (a type of distributed computing architecture), where the roles of different database nodes are strictly separated. Even if the queried data resides locally on the coordinating node, the coordinating node in a traditional distributed database architecture still needs to communicate with itself via network protocols before transmitting the data to the other data nodes for processing. This results in the coordinating node's computing resources often remaining idle during query intervals, not participating in actual data processing, leading to resource waste. Therefore, improving the node resource utilization rate of distributed databases has become one of the most pressing issues to be addressed. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product for database nodes that enables the coordination node of a distributed database to perform tasks and improves the utilization rate of node resources in the distributed database, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a data processing method for a database node. The method includes:

[0006] Based on the database query request received by the coordinating node, the target execution plan fragment corresponding to the coordinating node is obtained;

[0007] Based on global snapshot information, the target execution plan fragment is encapsulated into a task to obtain the target execution task corresponding to the target execution plan fragment;

[0008] When the status information of the first resource group of the coordination node is detected to meet the preset status conditions, the local thread pool of the coordination node performs local asynchronous execution processing on the target execution task according to the global snapshot information in the target execution task, and obtains the local execution result of the target execution task.

[0009] The local execution result is fused with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

[0010] In one embodiment, after performing task encapsulation processing on the target execution plan fragment based on global snapshot information to obtain the target execution task corresponding to the target execution plan fragment, the method further includes:

[0011] The task queue depth of the local thread pool is continuously monitored to obtain the current queue depth.

[0012] If the current queue depth exceeds a preset depth threshold, then it is confirmed that the status information of the first resource group does not meet the preset status condition.

[0013] The target execution task is converted into a remote execution request, and the remote execution request is sent to the data node for execution.

[0014] Receive the downgraded execution result returned by the data node based on the remote execution request.

[0015] In one embodiment, after sending the remote execution request to the data node for execution, the method further includes:

[0016] Upon receiving the error report information from the data node, the target execution task in the local thread pool is rolled back to obtain the first rollback result of the local thread pool, and the remote execution request in the data node is rolled back to obtain the second rollback result of the data node.

[0017] The error report information is generated based on the error events that occur when the data node executes the remote execution request.

[0018] In one embodiment, before continuously monitoring the task queue depth of the local thread pool to obtain the current queue depth, the method further includes:

[0019] The local resources of the coordination node are allocated to obtain a first resource group and a second resource group; the first resource group represents the local resources of the coordination node used to process local execution tasks; the second resource group represents the local resources of the coordination node used to process coordination tasks.

[0020] The first resource group and the second resource group are continuously monitored and processed to obtain the first resource usage information of the first resource group and the second resource usage information of the second resource group;

[0021] Based on the first resource usage information and the second resource usage information, the first resource group and the second resource group are reassigned to obtain the updated first resource group and the updated second resource group.

[0022] In one embodiment, based on the database query request received by the coordinating node, the target execution plan fragment corresponding to the coordinating node is obtained, including:

[0023] Based on the database query request received by the coordinating node, a global execution plan corresponding to the database query request is generated;

[0024] The global execution plan is split into multiple execution plan segments.

[0025] From the plurality of execution plan fragments, identify the target execution plan fragment that belongs to the local execution of the coordination node.

[0026] In one embodiment, the method further includes:

[0027] When the coordinating node recovers from the crash state, it obtains transaction information for unfinished pending transactions;

[0028] Based on the transaction information, a transaction status query request for the transaction to be processed is generated;

[0029] Receive transaction status information returned based on the transaction status query request;

[0030] Based on the transaction status information, the pending transaction is processed to obtain the transaction processing result.

[0031] Secondly, this application also provides a data processing apparatus for a database node. The apparatus includes:

[0032] The fragment acquisition module is used to obtain the target execution plan fragment corresponding to the coordination node based on the database query request received by the coordination node;

[0033] The task encapsulation module is used to encapsulate the target execution plan fragment based on global snapshot information to obtain the target execution task corresponding to the target execution plan fragment;

[0034] An asynchronous execution module is used to perform local asynchronous execution processing on the target execution task based on the global snapshot information in the target execution task through the local thread pool of the coordinating node when the status information of the first resource group of the coordinating node meets the preset status conditions, so as to obtain the local execution result of the target execution task.

[0035] The result fusion module is used to fuse the local execution result with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

[0036] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0037] Based on the database query request received by the coordinating node, the target execution plan fragment corresponding to the coordinating node is obtained;

[0038] Based on global snapshot information, the target execution plan fragment is encapsulated into a task to obtain the target execution task corresponding to the target execution plan fragment;

[0039] When the status information of the first resource group of the coordination node is detected to meet the preset status conditions, the local thread pool of the coordination node performs local asynchronous execution processing on the target execution task according to the global snapshot information in the target execution task, and obtains the local execution result of the target execution task.

[0040] The local execution result is fused with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

[0041] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0042] Based on the database query request received by the coordinating node, the target execution plan fragment corresponding to the coordinating node is obtained;

[0043] Based on global snapshot information, the target execution plan fragment is encapsulated into a task to obtain the target execution task corresponding to the target execution plan fragment;

[0044] When the status information of the first resource group of the coordination node is detected to meet the preset status conditions, the local thread pool of the coordination node performs local asynchronous execution processing on the target execution task according to the global snapshot information in the target execution task, and obtains the local execution result of the target execution task.

[0045] The local execution result is fused with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

[0046] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0047] Based on the database query request received by the coordinating node, the target execution plan fragment corresponding to the coordinating node is obtained;

[0048] Based on global snapshot information, the target execution plan fragment is encapsulated into a task to obtain the target execution task corresponding to the target execution plan fragment;

[0049] When the status information of the first resource group of the coordination node is detected to meet the preset status conditions, the local thread pool of the coordination node performs local asynchronous execution processing on the target execution task according to the global snapshot information in the target execution task, and obtains the local execution result of the target execution task.

[0050] The local execution result is fused with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

[0051] The aforementioned data processing methods, apparatus, computer equipment, storage media, and computer program products for database nodes obtain the target execution plan fragment corresponding to the coordinating node based on the database query request received by the coordinating node; based on global snapshot information, the target execution plan fragment is encapsulated into a task to obtain the target execution task corresponding to the target execution plan fragment; when the status information of the first resource group of the coordinating node is detected to meet the preset status conditions, the target execution task is executed asynchronously locally through the local thread pool of the coordinating node based on the global snapshot information in the target execution task to obtain the local execution result of the target execution task; the local execution result is fused with the execution results of each execution plan fragment associated with the database query request to obtain the target execution result corresponding to the database query request. This method encapsulates the target execution plan fragments of the coordinating node based on global snapshot information, ensuring the consistency of data reading by nodes during distributed queries. It also constructs an asynchronous execution mechanism using a local thread pool to execute tasks, avoiding main thread blocking and improving the task scheduling throughput of the coordinating node. Simultaneously, it achieves precise utilization of local resources through resource group status information detection, reducing resource idleness. Furthermore, it obtains the final target execution result by fusing local execution results with the execution results of each execution plan fragment. This ensures the integrity of single-node execution results and provides an efficient and consistent foundation for the final aggregation of distributed query results. While ensuring that the coordinating node of the distributed database can execute tasks, it also effectively improves the execution efficiency, resource utilization, and data consistency level of the distributed database coordinating node. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an application environment diagram of a data processing method for a database node in one embodiment;

[0054] Figure 2 This is a flowchart illustrating a data processing method for a database node in one embodiment;

[0055] Figure 3 This is a timing diagram illustrating the processing of a database query request in one embodiment;

[0056] Figure 4 This is a flowchart illustrating the steps of task encapsulation processing for a target execution plan fragment in one embodiment;

[0057] Figure 5 This is a schematic diagram illustrating the process of error recovery during the execution phase and the crash recovery of the coordination node in one embodiment;

[0058] Figure 6 This is a flowchart illustrating the data processing method for a database node in another embodiment;

[0059] Figure 7 This is a flowchart illustrating asynchronous execution processing and dynamic degradation processing in one embodiment;

[0060] Figure 8 This is a structural block diagram of a data processing device for a database node in one embodiment;

[0061] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0064] The data processing method for database nodes provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the nodes in the distributed database include a coordinator node 101 and data nodes 102. The coordinator node 101 is the component in the distributed system that receives client query requests, generates a global execution plan, distributes plan fragments, and finally aggregates the plan execution results. The data node 102 is the component in the distributed system that actually stores data shards and executes the plan fragments. The coordinator node 101 communicates with the data node 102 via a network. The data storage system can store the data that the data node 102 needs to process. The data storage system can be integrated on a server or located in the cloud or on other network servers. Coordinating node 101 obtains the target execution plan fragment corresponding to the coordinating node based on the database query request received by the coordinating node; based on global snapshot information, it performs task encapsulation processing on the target execution plan fragment to obtain the target execution task corresponding to the target execution plan fragment; when it detects that the status information of the first resource group of the coordinating node meets the preset status conditions, it performs local asynchronous execution processing on the target execution task through the local thread pool of the coordinating node according to the global snapshot information in the target execution task to obtain the local execution result of the target execution task; it merges the local execution result with the execution results of each execution plan fragment associated with the database query request to obtain the target execution result corresponding to the database query request.

[0065] In one embodiment, such as Figure 2 As shown, a data processing method for database nodes is provided, which can be applied to... Figure 1 Taking the coordination node in the example, the following steps are included:

[0066] Step S201: Based on the database query request received by the coordinating node, obtain the target execution plan fragment corresponding to the coordinating node.

[0067] Among them, the target execution plan fragment refers to the sub-plan that is assigned to a single data node for execution after the global execution plan has been split.

[0068] A database query request is an instruction sent to a database using a specific database language (such as Structured Query Language (SQL)). Database query requests are used to retrieve, manipulate, or manage data from a (distributed) database. For example, a database query request can be a SELECT (select or retrieve) request or an INSERT (insert) request.

[0069] Specifically, after receiving a database query request (such as an SQL request) initiated by an external source (such as a client), the coordinating node can generate a global execution plan. This global execution plan is then broken down into multiple execution plan fragments, and the local fragment to be executed (e.g., Scan(table_A), i.e., scanning table A) is identified. The coordinating node then obtains its local target execution plan fragment.

[0070] Step S202: Based on the global snapshot information, perform task encapsulation processing on the target execution plan fragment to obtain the target execution task corresponding to the target execution plan fragment.

[0071] Global snapshot information refers to a globally unique transaction version number or timestamp that identifies a specific point in time. Global snapshot information is used to ensure that the data read by worker threads in the local thread pool is consistent with the data read by other data nodes.

[0072] Specifically, the coordinating node does not process the target execution plan fragment in its main thread. Instead, it encapsulates the target execution plan fragment into an execution task containing the target execution plan fragment, global snapshot information, and parameters, i.e., the target execution task, so that the target execution task can be submitted to a dedicated, configurable local thread pool for processing.

[0073] For example, suppose the database query request is "SELECT SUM(orders.value) FROM ordersJOIN customers ON orders.cid = customers.id WHERE customers.region = 'Asia';". The coordinating node detects that the data shard leader of the customers table is located locally. Therefore, it allocates the Scan(customers WHERE region='Asia') and the subsequent HashJoin (a hash join used to load data from a table into memory and build a hash table to prepare for subsequent join matching) construction phase as the target execution plan fragment for local execution. Then, the target execution task corresponding to the target execution plan fragment is submitted to the local thread pool. Worker threads retrieve the target execution task from the thread pool and begin the scanning and hash table construction operations.

[0074] Step S203: When the status information of the first resource group of the coordinating node is detected to meet the preset status conditions, the local thread pool of the coordinating node performs local asynchronous execution processing on the target execution task according to the global snapshot information in the target execution task, and obtains the local execution result of the target execution task.

[0075] Among them, the status information is used to characterize whether the resource utilization of a resource group (such as the first resource group or the second resource group) is saturated.

[0076] Specifically, Figure 3 A timing diagram for processing database query requests, such as Figure 3 As shown, the Resource Monitor monitors the resource usage of the first resource group in real time. The coordinating node can request the status information of the first resource group from the Resource Monitor through the asynchronous execution controller to determine whether the status information of the first resource group meets the preset status conditions. For example, if the status information of the first resource group is detected to be in an unsaturated state, it is considered that the status information of the first resource group meets the preset status conditions. When it is detected that the status information of the first resource group of the coordinating node meets the preset status conditions, the coordinating node can submit the target execution task to the coordinating node's local thread pool and return immediately, so that the coordinating node can continue to process other client requests, node coordination, and other coordination tasks, thereby achieving non-blocking coordination. In other words, while the worker threads in the local thread pool are asynchronously executing the target execution task, the main thread of the coordinating node can continue to process new database query requests without being affected by the task processing operations of the worker threads in the local thread pool.

[0077] The worker threads in the local thread pool obtain the target execution task from the local thread pool. The worker threads can call the execution engine to asynchronously execute the processing operation corresponding to the target execution task, obtain the local execution result of the target execution task, and return the local execution result to the coordinating node.

[0078] Furthermore, during the asynchronous execution of the target task, worker threads can also access the local storage engine carrying global snapshot information (S_i). The local storage engine's MVCC (Multi-Version Concurrency Control) module filters out the visible data versions based on the global snapshot information. For local write operations, worker threads generate a Write Ahead Logging (WAL) log, which contains a global transaction identifier (e.g., an ID identifier) ​​and uses a two-phase commit protocol (2PC) to ensure consistency in write operations performed by all nodes participating in the write operation.

[0079] Continuing with the example in step S202 above, the global snapshot information corresponding to the database query request is the timestamp T_snapshot = 2023-11-07 10:00:00.000. When the local worker thread scans the customers table, for each row, it checks its xmin (the transaction identifier of the current data row version) and xmax (the transaction identifier of the current data row version that has expired) system columns, only reading rows committed before T_snapshot that have not been deleted. Even if another transaction happens to insert or modify the region field of the customers table after the global snapshot information T_snapshot during the execution of this query, these changes are not visible to this query, thus ensuring the consistency of the entire query result.

[0080] Step S204: The local execution result is merged with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

[0081] Specifically, the coordinating node receives execution results (or downgraded execution results) from other data nodes or from the coordinating node itself for each execution plan segment via the communication component. The coordinating node invokes the result merger to aggregate the local execution results (or downgraded execution results) of the target execution plan segment with the execution results of the remaining execution plan segments, obtaining a set of execution results. This set of execution results is then set as the target execution result corresponding to the database query request. The coordinating node returns the final target execution result to an external source (such as a client).

[0082] In the data processing method of the aforementioned database nodes, the target execution plan fragments of the coordinating node are encapsulated based on global snapshot information, ensuring the consistency of data reading by nodes during distributed queries. An asynchronous execution mechanism using a local thread pool is also constructed to execute tasks, avoiding main thread blocking and improving the task scheduling throughput of the coordinating node. Simultaneously, accurate utilization of local resources is achieved through resource group status information detection, reducing resource idleness. Furthermore, the final target execution result is obtained by fusing local execution results with the execution results of each execution plan fragment. This ensures the integrity of single-node execution results and provides efficient and consistent support for the final aggregation of distributed query results. While ensuring that the coordinating node of the distributed database can execute tasks, it also effectively improves the execution efficiency, resource utilization, and data consistency level of the distributed database coordinating node.

[0083] In one embodiment, such as Figure 4 As shown, step S202 above, based on global snapshot information, performs task encapsulation processing on the target execution plan fragment to obtain the target execution task corresponding to the target execution plan fragment, specifically including the following:

[0084] Step S401: Continuously monitor the task queue depth of the local thread pool to obtain the current queue depth.

[0085] The task queue depth refers to the number of tasks contained in the task queue of the local thread pool.

[0086] Specifically, by continuously monitoring the task queue depth of the local thread pool, the coordinating node can obtain the current queue depth of the task queue in real time.

[0087] Step S402: If the current queue depth exceeds the preset depth threshold, then confirm that the status information of the first resource group does not meet the preset status conditions.

[0088] Specifically, when the current queue depth is detected to exceed a preset depth threshold (for example, the preset depth threshold can be twice the size of the local thread pool), the coordinating node confirms that the state information representation of the first resource group is saturated, that is, the state information of the first resource group does not meet the preset state conditions, thereby triggering the dynamic degrade mechanism.

[0089] Dynamic degradation refers to a fault-tolerance and load balancing mechanism in which the database system automatically converts execution plan fragments that should be executed locally into remote execution requests and sends them to other data nodes for execution when local asynchronous execution resources are saturated.

[0090] Step S403: Convert the target execution task into a remote execution request and send the remote execution request to the data node for execution.

[0091] Specifically, the dynamic degradation mechanism works as follows: For newly submitted target execution plan fragments, instead of attempting to convert them into target execution tasks and add them to the task queue of the local thread pool, they are intercepted. The coordinating node then invokes plan rewriting to convert the newly submitted target execution plan fragment into an equivalent remote execution request. This remote execution request contains all necessary execution information (such as operators, predicates, global snapshot information, etc.). Based on the database's data distribution information, the coordinating node sends this remote execution request to other data nodes (OtherDatanodes) that store replicas of the same data shards for execution.

[0092] Step S404: Receive the downgrade execution result returned by the data node based on the remote execution request.

[0093] Specifically, after the data node completes the data processing corresponding to the remote execution request, it obtains the degraded execution result and returns the degraded execution result to the coordinating node.

[0094] For example, during peak business hours, the local thread pool's task queue is full, say, with 10 target execution tasks backlogged. If a new database query request arrives, a dynamic degradation mechanism is immediately activated: Assuming the new query request requests a scan of the local `log_table` (a database table), the execution plan fragment `Scan(log_table WHERE ...)` can be obtained. The coordinating node calls the plan rewriter to rewrite this execution plan fragment into a remote execution request with the same filtering conditions, i.e., `Remote_Exec_Request`. The coordinating node queries metadata to determine that a certain Follower replica of `log_table` resides on a data node (marked as `DataNode-5`). The coordinating node then sends the remote execution request `Remote_Exec_Request` and the global snapshot information `S_i` to `DataNode-5` over the network. After completing the scan of `log_table`, `DataNode-5` returns the scan results (i.e., the degradation execution results) to the coordinating node. It should be noted that although this introduces network overhead, it prevents the coordinating node from crashing due to overload or its response from slowing down drastically, thus ensuring the overall availability of the distributed database system.

[0095] In this embodiment, by continuously monitoring the task queue depth of the local thread pool, the load status of the first resource group is accurately perceived. When the task queue depth exceeds a preset depth threshold, the plan rewriter is promptly invoked to convert the target execution task into a remote execution request and distribute it to other data nodes for re-execution. Then, the degraded execution results returned by other data nodes are received, effectively avoiding task blocking or node crashes caused by overload of local resources of the coordinating node, and ensuring the stable operation of the coordinating node. At the same time, the remote distribution of tasks realizes the balanced utilization of resources within the distributed system, reduces task waiting time, improves the overall execution efficiency and resource utilization of database tasks, and provides reliable support for high-concurrency query scenarios of distributed databases.

[0096] In one embodiment, after sending the remote execution request to the data node for execution in step S403, the method further includes: upon receiving error report information from the data node, performing rollback processing on the target execution task in the local thread pool to obtain a first rollback result of the local thread pool, and performing rollback processing on the remote execution request in the data node to obtain a second rollback result of the data node; wherein the error report information is generated based on error events generated by the data node when executing the remote execution request.

[0097] It should be noted that error report information can also be generated based on errors produced by the local thread pool when executing local plan segments.

[0098] Specifically, Figure 5 This is a flowchart illustrating the error recovery during the execution phase and the crash recovery of the coordination node, as shown below. Figure 5 As shown, in a failure scenario where an error occurs during the execution phase, if an error (such as a unique key conflict or data validation failure) occurs while executing a local plan segment on the data node or the asynchronous worker thread of the coordinating node's local thread pool, it will immediately report the error to the main thread of the coordinating node, which acts as the transaction coordinator. Upon receiving the error report, the coordinating node will immediately initiate a global rollback process. For example, it may send a rollback instruction (ROLLBACK) to all data nodes participating in the transaction via a two-phase commit protocol (2PC), obtaining the second rollback result from the data nodes. Simultaneously, it will roll back all operations executed by the asynchronous worker threads of the local thread pool within the transaction, obtaining the first rollback result from the local thread pool.

[0099] For example, suppose the target execution plan segment (labeled transaction T1) needs to update table A on the coordinating node's local machine and table B on the remote node. The asynchronous worker thread in the local thread pool successfully updates table A, but an error occurs when the remote node updates table B. After receiving the error report about table B from the remote node, the coordinating node first sends a rollback instruction to the asynchronous worker thread in the local thread pool to roll back the update to table A, and simultaneously sends a rollback instruction to the remote node (even though it may have failed) to roll back the update to table B. Ultimately, tables A and B are restored to their state before distributed transaction T1 occurred.

[0100] In this embodiment, upon receiving an error report, the target execution task in the local thread pool and the remote execution request in the data node are rolled back respectively. By obtaining the first rollback result locally and the second rollback result of the data node, the data inconsistency problem caused by some tasks succeeding and some tasks failing is avoided. At the same time, the error event is quickly responded to and a precise rollback is triggered, which reduces the scope of error impact, lowers the risk of data corruption, and ensures the data consistency and data integrity of the distributed database system, providing strong support for highly available distributed data processing scenarios.

[0101] In one embodiment, before continuously monitoring the task queue depth of the local thread pool to obtain the current queue depth in step S401, the method further includes: performing resource allocation processing on the local resources of the coordinating node to obtain a first resource group and a second resource group; the first resource group represents the local resources of the coordinating node used to process locally executed tasks; the second resource group represents the local resources of the coordinating node used to process coordinated tasks; continuously monitoring the first resource group and the second resource group to obtain first resource usage information of the first resource group and second resource usage information of the second resource group; and performing resource reallocation processing on the first resource group and the second resource group based on the first resource usage information and the second resource usage information to obtain an updated first resource group and an updated second resource group.

[0102] It should be noted that the tasks handled by the coordinating node are usually coordination tasks. However, the coordinating node in this application can also execute tasks asynchronously using a local thread pool. In other words, in this application, the tasks handled by the coordinating node include coordination tasks and locally executed tasks.

[0103] Specifically, through the internal resource controller, separate resource groups are set up for the local execution tasks and coordination tasks of the coordination node, namely the first resource group and the second resource group. The first resource group is used to limit the number of CPU (Central Processing Unit) cores (e.g., up to 50% of CPU), memory bandwidth and maximum memory usage (e.g., up to 40% of total memory) that the local thread pool can use.

[0104] Furthermore, the built-in monitoring module collects real-time usage metrics (such as CPU utilization, memory pressure, and thread pool queue depth) of the first and second resource groups for coordinating resources, obtaining first resource usage information for the first resource group and second resource usage information for the second resource group. Based on the first and second resource usage information, the system dynamically reallocates resources between the first and second resource groups, resulting in updated first and second resource groups. For example, when the coordination node is found to have heavy coordination tasks (such as CPU utilization exceeding 80%), the dynamic scheduler can automatically reduce the priority of the local thread pool, reduce its CPU quota, or slow down the task delivery speed, that is, reduce the resource configuration of the first resource group and increase the resource configuration of the second resource group, prioritizing the stability of the coordination tasks of the coordination node.

[0105] In this embodiment, by dividing the local resources of the coordination node into a first resource group dedicated to handling local execution tasks and a second resource group for handling coordination tasks, resource isolation of different types of tasks is achieved, avoiding resource contention between tasks. At the same time, by continuously monitoring the resource usage information of the two groups, resources are dynamically reallocated, enabling resource allocation to accurately match the real-time load requirements of the two types of tasks. This ensures the stable execution of coordination tasks to maintain the collaborative scheduling capability of the distributed system, while also improving the resource utilization efficiency of local execution tasks. It effectively avoids resource idleness or overload issues, and overall optimizes the rationality of resource configuration and task processing throughput of the coordination node, enhancing the operational stability and elastic adaptability of the distributed database system.

[0106] In one embodiment, step S201 above, which obtains the target execution plan fragment corresponding to the coordinating node based on the database query request received by the coordinating node, specifically includes the following: generating a global execution plan corresponding to the database query request based on the database query request received by the coordinating node; performing plan splitting on the global execution plan to obtain multiple execution plan fragments of the global execution plan; and identifying the target execution plan fragment belonging to the local execution of the coordinating node from the multiple execution plan fragments.

[0107] Specifically, such as Figure 3 As shown, the coordinating node receives the SQL text of a database query request initiated by the client and passes it to the SQL Parser component (a tool for parsing SQL statements into an abstract syntax tree). The SQL Parser performs syntax parsing on the SQL text, generates an abstract syntax tree (AST), and performs logical optimization on the AST to generate a global execution plan, which is then returned to the coordinating node. The coordinating node sends the global execution plan to the Optimizer for splitting into multiple execution plan fragments that can be processed in parallel. From these multiple execution plan fragments, the coordinating node identifies the target execution plan fragment that belongs to its local execution.

[0108] In this embodiment, by generating a global execution plan for the received database query request and reasonably splitting the plan into multiple execution plan fragments that can be processed in parallel, and then identifying the target execution plan fragment suitable for coordinating local execution on the node, the decomposition of the global execution plan for the database query request and the precise allocation of the load are effectively realized. This not only fully leverages the parallel processing capabilities of each node in the distributed architecture, but also reduces the communication overhead and data transfer costs between nodes by executing specific target execution plan fragments locally. As a result, the overall response speed of the database query and the utilization efficiency of computing resources are improved, providing reliable technical support for high performance and low latency in large-scale data query scenarios.

[0109] In one embodiment, the data processing method for the database node further includes: when the coordinating node recovers from a crash state, obtaining transaction information of unfinished pending transactions; generating a transaction status query request for pending transactions based on the transaction information; receiving transaction status information returned based on the transaction status query request; and processing the pending transactions according to the transaction status information to obtain the transaction processing result of the pending transactions.

[0110] A transaction is a collection of database operations (such as INSERT, UPDATE, and DELETE) that are treated as a single logical unit of work.

[0111] like Figure 5 As shown, in a failure scenario involving a coordinating node crash and recovery, the coordinating node might be managing a distributed transaction before crashing; for example, it might crash shortly after issuing a PREPARE request. The coordinating node's recovery process needs to be integrated with the state of the Global Transaction Manager (GTM). Specifically, when the crashed coordinating node restarts, the recovery mechanism is activated. It reads the local transaction log, filters out all pending transactions (T2) that haven't been finally committed (COMMIT or ROLLBACK), and obtains transaction information such as transaction identifiers (e.g., IDs). For each pending transaction, the coordinating node sends a transaction status query request to the GTM via the communication component. This request contains transaction information. The GTM receives the transaction status query request, queries the final state of the pending transaction (e.g., COMMIT or ROLLBACK) based on the transaction information, and returns the transaction status information to the recovering coordinating node. If the global transaction manager returns a COMMIT transaction status, the coordinating node resends the commit instruction to all data nodes involved in the pending transaction, synchronously executing the commit operation (including redoing the log application after commit for asynchronous worker threads in the local thread pool). If the status is ROLLBACK, a rollback instruction is sent, executing the rollback operation. After each data node completes the commit or rollback operation, it returns the transaction processing result (e.g., success or failure) to the coordinating node.

[0112] For example, the coordinating node crashes after issuing a PREPARE request. Upon restarting, the coordinating node finds that transaction T2 is in a "preparing" state. The coordinating node queries the global transaction manager and learns that transaction T2 has been marked as "pending rollback" due to a timeout. Therefore, the coordinating node sends a ROLLBACK instruction to all data nodes and rolls back all data modifications made by the asynchronous worker threads in its local thread pool for transaction T2, ensuring global consistency.

[0113] In this embodiment, when the coordinating node recovers from a crash, it accurately obtains the transaction information of unfinished pending transactions, generates corresponding transaction status query requests, and receives feedback transaction status information. Then, it performs targeted processing on pending transactions based on the status information. This ensures the integrity of unfinished transactions before and after the crash, avoids the problems of transaction loss or execution chaos, and quickly restores the transaction processing flow through clear status query and targeted processing logic. This reduces the impact of node crash on system operation and effectively guarantees the distributed transaction reliability and data consistency of the database system.

[0114] In one embodiment, such as Figure 6 As shown, another data processing method for database nodes is provided, which can be applied to... Figure 1 Taking the coordination node in the example, the following steps are included:

[0115] Step S601: Generate a global execution plan corresponding to the database query request based on the database query request received by the coordinating node.

[0116] Step S602: Perform plan splitting on the global execution plan to obtain multiple execution plan fragments of the global execution plan.

[0117] Step S603: Identify the target execution plan fragment that belongs to the local execution of the coordination node from multiple execution plan fragments.

[0118] Step S604: Based on the global snapshot information, perform task encapsulation processing on the target execution plan fragment to obtain the target execution task corresponding to the target execution plan fragment.

[0119] Step S605: Continuously monitor the task queue depth of the local thread pool to obtain the current queue depth.

[0120] Step S606: If the current queue depth exceeds the preset depth threshold, confirm that the status information of the first resource group does not meet the preset status conditions; convert the target execution task into a remote execution request and send the remote execution request to the data node for execution; receive the downgraded execution result returned by the data node based on the remote execution request.

[0121] Step S607: When the status information of the first resource group of the coordinating node is detected to meet the preset status conditions, the local thread pool of the coordinating node is used to perform local asynchronous execution processing on the target execution task according to the global snapshot information in the target execution task, so as to obtain the local execution result of the target execution task.

[0122] Step S608: The local execution result or the downgraded execution result is merged with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

[0123] The data processing method for the aforementioned database nodes achieves the following beneficial effects: Based on global snapshot information, the target execution plan fragments of the coordinating node are encapsulated, ensuring the consistency of data reading by nodes during distributed queries. An asynchronous execution mechanism using a local thread pool is also constructed to execute tasks, avoiding main thread blocking and improving the task scheduling throughput of the coordinating node. Simultaneously, accurate utilization of local resources is achieved through resource group status information detection, reducing resource idleness. Furthermore, the final target execution result is obtained by fusing local execution results with the execution results of each execution plan fragment. This ensures the integrity of single-node execution results and provides efficient and consistent support for the final aggregation of distributed query results. While ensuring that the coordinating node of the distributed database can execute tasks, it also effectively improves the execution efficiency, resource utilization, and data consistency level of the distributed database coordinating node.

[0124] To more clearly illustrate the data processing method for database nodes provided in this disclosure, a specific embodiment is used to describe the data processing method for database nodes. This application also provides another data processing method for database nodes, specifically including the following:

[0125] (1) Non-blocking asynchronous execution unit: Figure 7 This is a flowchart illustrating asynchronous execution and dynamic degradation processing, as shown below. Figure 7 As shown, the coordinating node receives database query requests (SQL requests) from clients, generates a global execution plan, and breaks it down into multiple execution plan fragments. It then identifies the local fragment it needs to execute, i.e., the target execution plan fragment. This target execution plan fragment is encapsulated into a target execution task and submitted to a dedicated local thread pool, enabling non-blocking execution of the coordinating node's main thread.

[0126] (2) Snapshot Consistency Processing Unit: Based on global snapshot information, ensure that the data read by the asynchronous worker thread of the local thread pool is consistent with the data read by other data nodes;

[0127] (3) Resource isolation and scheduling unit: By dividing resource groups, the resource usage of locally executed tasks is restricted, and the resource allocation and task priority of the first and second resource groups are dynamically adjusted in combination with monitoring;

[0128] (4) Dynamic degradation unit: such as Figure 7 As shown, resource usage is monitored through a resource monitor, as well as the task queue depth of the local thread pool. When the task queue depth exceeds a preset threshold, a dynamic degradation mechanism is triggered, intercepting new target execution tasks and adding them to the queue. These new target execution tasks are then converted into remote execution requests and sent to other data nodes that share the same target execution plan fragment for execution. The data nodes return the degradation execution results. The degradation execution results of the target execution plan fragment and the execution results of the remaining execution plan fragments are aggregated to obtain an execution result set. This execution result set is then set as the target execution result corresponding to the database query request. The coordinating node returns the final target execution result to the client.

[0129] (5) Fault-tolerant recovery unit: After handling errors in the execution phase and failures of the coordination node, the atomicity of transactions can be guaranteed through global rollback, status query and decision execution.

[0130] In this embodiment, the following beneficial effects can be achieved: (1) Fully exploit the idle computing resources of the coordinating node, eliminate the network communication overhead of local queries, improve the resource utilization and query throughput of the entire distributed database cluster, solve the problem of resource waste in traditional architecture, make full use of the idle resources of the coordinating node, and reduce the demand for additional data nodes, which is conducive to reducing the hardware procurement and operation and maintenance costs of the distributed database cluster; (2) Combine global snapshot consistency with local asynchronous execution, break the role barrier of the coordinating node, and ensure that the data view of local asynchronous execution is consistent with other data nodes through global snapshot information, meet the transaction isolation requirements, and avoid the data consistency risk brought by mixed role nodes; (3) The system has the ability to protect itself from overload through dynamic degradation mechanism. When the local load of the coordinating node is too high, the task is automatically diverted to avoid the avalanche effect that may occur under pressure of mixed architecture nodes, which greatly enhances the robustness and production availability of the system; (4) Combine with differentiated fault recovery mechanism to further ensure data consistency and system reliability.

[0131] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0132] Based on the same inventive concept, this application also provides a database node data processing apparatus for implementing the data processing method for the database node described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more database node data processing apparatus embodiments provided below can be found in the limitations of the database node data processing method described above, and will not be repeated here.

[0133] In one embodiment, such as Figure 8 As shown, a data processing device 800 for a database node is provided, comprising: a fragment acquisition module 801, a task encapsulation module 802, an asynchronous execution module 803, and a result fusion module 804, wherein:

[0134] The fragment acquisition module 801 is used to obtain the target execution plan fragment corresponding to the coordinating node based on the database query request received by the coordinating node.

[0135] The task encapsulation module 802 is used to encapsulate the target execution plan fragment based on global snapshot information to obtain the target execution task corresponding to the target execution plan fragment.

[0136] The asynchronous execution module 803 is used to perform local asynchronous execution processing on the target execution task based on the global snapshot information in the target execution task through the local thread pool of the coordinating node when the status information of the first resource group of the coordinating node meets the preset status conditions, so as to obtain the local execution result of the target execution task.

[0137] The result fusion module 804 is used to fuse the local execution results with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

[0138] In one embodiment, the data processing device 800 of the database node further includes a dynamic degradation module, which is used to continuously monitor and process the task queue depth of the local thread pool to obtain the current queue depth of the task queue; if the current queue depth exceeds a preset depth threshold, it is confirmed that the status information of the first resource group does not meet the preset status conditions; the target execution task is converted into a remote execution request and the remote execution request is sent to the data node for execution; and the degradation execution result returned by the data node based on the remote execution request is received.

[0139] In one embodiment, the data processing device 800 of the database node further includes an error stage rollback module, which is used to roll back the target execution task in the local thread pool to obtain a first rollback result of the local thread pool and to roll back the remote execution request in the data node to obtain a second rollback result of the data node when receiving error report information from the data node; wherein, the error report information is generated based on the error event generated by the data node when executing the remote execution request.

[0140] In one embodiment, the data processing device 800 of the database node further includes a resource allocation module, used to perform resource allocation processing on the local resources of the coordination node to obtain a first resource group and a second resource group; the first resource group represents the local resources of the coordination node used to process local execution tasks; the second resource group represents the local resources of the coordination node used to process coordination tasks; the first resource group and the second resource group are continuously monitored to obtain first resource usage information of the first resource group and second resource usage information of the second resource group; based on the first resource usage information and the second resource usage information, the first resource group and the second resource group are reallocated to obtain an updated first resource group and an updated second resource group.

[0141] In one embodiment, the fragment acquisition module 801 is further configured to generate a global execution plan corresponding to the database query request based on the database query request received by the coordinating node; perform plan splitting on the global execution plan to obtain multiple execution plan fragments of the global execution plan; and identify the target execution plan fragment belonging to the local execution of the coordinating node from the multiple execution plan fragments.

[0142] In one embodiment, the data processing device 800 of the database node further includes a node crash recovery module, which is used to obtain transaction information of unfinished pending transactions when the coordinating node recovers from a crash state; generate a transaction status query request for pending transactions based on the transaction information; receive transaction status information returned based on the transaction status query request; and process the pending transactions according to the transaction status information to obtain the transaction processing result of the pending transactions.

[0143] Each module in the data processing device of the aforementioned database node can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0144] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores relevant task data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method for a database node.

[0145] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0147] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0148] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method for a database node, characterized in that, The method includes: Generate a global execution plan based on the database query requests received by the coordinating node; The global execution plan is split into multiple execution plan fragments, and the local fragments that the coordinating node itself needs to execute are identified, thus obtaining the target execution plan fragments of the coordinating node. Based on global snapshot information, the target execution plan fragment is encapsulated into a task to obtain the target execution task corresponding to the target execution plan fragment; When the status information of the first resource group of the coordination node is detected to meet the preset status conditions, the local thread pool of the coordination node performs local asynchronous execution processing on the target execution task according to the global snapshot information in the target execution task to obtain the local execution result of the target execution task; wherein, the status information is used to characterize whether the resource utilization of the first resource group is saturated; The local execution result is fused with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

2. The method according to claim 1, characterized in that, After performing task encapsulation processing on the target execution plan fragment based on global snapshot information to obtain the target execution task corresponding to the target execution plan fragment, the process further includes: The task queue depth of the local thread pool is continuously monitored to obtain the current queue depth. If the current queue depth exceeds a preset depth threshold, then it is confirmed that the status information of the first resource group does not meet the preset status condition. The target execution task is converted into a remote execution request, and the remote execution request is sent to the data node for execution. Receive the downgraded execution result returned by the data node based on the remote execution request.

3. The method according to claim 2, characterized in that, After sending the remote execution request to the data node for execution, the process also includes: Upon receiving the error report information from the data node, the target execution task in the local thread pool is rolled back to obtain the first rollback result of the local thread pool, and the remote execution request in the data node is rolled back to obtain the second rollback result of the data node. The error report information is generated based on the error events that occur when the data node executes the remote execution request.

4. The method according to claim 2, characterized in that, Before continuously monitoring the task queue depth of the local thread pool to obtain the current queue depth, the process further includes: The local resources of the coordination node are allocated to obtain a first resource group and a second resource group; the first resource group represents the local resources of the coordination node used to process local execution tasks; the second resource group represents the local resources of the coordination node used to process coordination tasks. The first resource group and the second resource group are continuously monitored and processed to obtain the first resource usage information of the first resource group and the second resource usage information of the second resource group; Based on the first resource usage information and the second resource usage information, the first resource group and the second resource group are reassigned to obtain the updated first resource group and the updated second resource group.

5. The method according to claim 1, characterized in that, The step of obtaining the target execution plan fragment corresponding to the coordinating node based on the database query request received by the coordinating node includes: Based on the database query request received by the coordinating node, a global execution plan corresponding to the database query request is generated; The global execution plan is split into multiple execution plan segments. From the plurality of execution plan fragments, identify the target execution plan fragment that belongs to the local execution of the coordination node.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: When the coordinating node recovers from the crash state, it obtains transaction information for unfinished pending transactions; Based on the transaction information, a transaction status query request for the transaction to be processed is generated; Receive transaction status information returned based on the transaction status query request; Based on the transaction status information, the pending transaction is processed to obtain the transaction processing result.

7. A data processing device for a database node, characterized in that, The device includes: The fragment acquisition module is used to generate a global execution plan based on the database query request received by the coordinating node; the global execution plan is split into multiple execution plan fragments, and the local fragments that the coordinating node itself needs to execute are identified, thus obtaining the target execution plan fragments of the coordinating node. The task encapsulation module is used to encapsulate the target execution plan fragment based on global snapshot information to obtain the target execution task corresponding to the target execution plan fragment; An asynchronous execution module is used to perform local asynchronous execution processing on the target execution task based on the global snapshot information in the target execution task through the local thread pool of the coordinating node when the status information of the first resource group of the coordinating node is detected to meet the preset status conditions, so as to obtain the local execution result of the target execution task; wherein, the status information is used to characterize whether the resource utilization of the first resource group is saturated; The result fusion module is used to fuse the local execution result with the execution results of each execution plan segment associated with the database query request to obtain the target execution result corresponding to the database query request.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images