Multi-level group submission method and device for distributed database, equipment and medium
By employing a multi-level group commit method, data nodes and compute nodes work together to solve the problems of high write latency and low resource utilization in distributed databases, achieving efficient concurrent transaction processing and resource optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-03-24
AI Technical Summary
Existing distributed database group commit mechanisms suffer from high write latency and low resource utilization, especially under low and high load conditions.
A multi-level group commit method is adopted, in which transaction requests are generated by data nodes in chronological order, merged into local transaction groups, and then integrated into global transaction groups by compute nodes. The two-phase commit process ensures transaction atomicity and dynamically adjusts the merge threshold to adapt to load changes.
It reduces the number of I/O operations, improves concurrent processing efficiency, reduces the response time of a single transaction, optimizes resource utilization, and adapts to the needs of different load scenarios.
Smart Images
Figure CN121722844A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed database technology, and in particular to a multi-level group commit method, apparatus, device, and storage medium for distributed databases. Background Technology
[0002] In the field of distributed databases, group commit is a core technology for optimizing high-concurrency write performance. Its core principle lies in integrating multiple concurrent small transactions into a large-scale batch transaction. By uniformly executing write-ahead log (WAL) flushing and commit confirmation operations, the number of disk I / O operations is significantly reduced, thereby effectively improving the overall system throughput. However, in practical applications, existing group commit mechanisms have revealed two prominent drawbacks: First, high write latency. In low-load scenarios, group commit needs to wait for a sufficient number of transactions to accumulate or for a timeout mechanism to be triggered before execution. This directly leads to a significant increase in the response time of a single transaction, especially in business scenarios involving small-batch data writing, where the time spent waiting for the group commit condition to be met can even far exceed the actual data processing time. Second, low resource utilization. Existing mechanisms generally use static parameter configuration, which cannot dynamically adjust the number and time interval of group commit transactions according to the real-time system load. This results in severe resource contention pressure during high-load periods and significant resource idleness and waste during low-load periods. Summary of the Invention
[0003] The main objective of this invention is to provide a multi-level group commit method, apparatus, device, and storage medium for distributed databases, aiming to solve the problems of high write latency and low resource utilization in existing group commit methods.
[0004] To achieve the above objectives, the present invention provides a multi-level group commit method for a distributed database, comprising: Obtain transaction requests sent by the client, and arrange the transaction requests in chronological order through data nodes to generate a transaction queue; The data node merges the transaction queues into a local transaction group and sends the local transaction group to the compute node; When the computing node receives a local transaction group sent by the data node, it integrates the local transaction group to generate a global transaction group; The compute node sends a global group commit request to the data node, the data node returns a ready status based on the global group commit request, and the compute node executes a global transaction group commit operation based on the ready status.
[0005] Furthermore, to achieve the above objectives, the present invention provides a multi-level group commit device for a distributed database, comprising: The transaction queue module is used to obtain transaction requests sent by the client and arrange the transaction requests in chronological order through data nodes to generate a transaction queue; The local transaction group module is used by the data node to merge the transaction queues into a local transaction group and send the local transaction group to the compute node; The global transaction group module is used to integrate the local transaction groups sent by the data nodes and generate a global transaction group when the computing node receives the local transaction groups sent by the data nodes. The phase commit module is used to send a global group commit request to the data node through the compute node, the data node provides feedback on the ready status based on the global group commit request, and the compute node executes the global transaction group commit operation based on the ready status.
[0006] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a multi-level group commit program for a distributed database stored in the memory and executable on the processor, wherein when the multi-level group commit program for the distributed database is executed by the processor, it implements the steps of the multi-level group commit method for a distributed database as described above.
[0007] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a multi-level group commit program for a distributed database, wherein the multi-level group commit program for a distributed database, when executed by a processor, implements the steps of the multi-level group commit method for a distributed database as described above.
[0008] Beneficial Effects: This invention relates to the field of distributed database technology and discloses a multi-level group commit method for distributed databases, comprising: obtaining transaction requests issued by clients and arranging the transaction requests into a transaction queue by data nodes in chronological order; merging the transaction queues into local transaction groups by data nodes and sending them to compute nodes; when a compute node receives a local transaction group, integrating the local transaction groups to generate a global transaction group; sending a global group commit request to the data nodes through the compute nodes, with the data nodes providing feedback on the ready status based on the global group commit request, and the compute nodes executing the global transaction group commit operation based on the ready status. This invention first generates a transaction queue by data nodes in chronological order and merges it into a local transaction group, and then integrates it into a global transaction group by compute nodes. The operation is completed through a two-stage commit, reducing I / O operations through batch processing, ensuring transaction atomicity through global coordination, and improving concurrent processing efficiency. Attached Figure Description
[0009] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1This is a schematic diagram of an application environment for a multi-level group submission method for a distributed database according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the multi-level group submission method for distributed databases according to the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multi-level group submission device for the distributed database of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0010] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0011] The multi-level group commit method for distributed databases provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain transaction requests from the client and arrange them into a transaction queue according to time order through data nodes. The data nodes merge the transaction queues into local transaction groups and send them to the compute nodes. When the compute nodes receive the local transaction groups, they integrate them to generate a global transaction group. The compute nodes send a global group commit request to the data nodes, and the data nodes provide a ready status feedback based on the global group commit request. The compute nodes then execute the global transaction group commit operation based on the ready status. This invention first generates a transaction queue by data nodes according to time order and merges it into a local transaction group, and then integrates it into a global transaction group by the compute nodes. The operation is completed through two-stage commit, batch processing reduces I / O operations, global coordination ensures transaction atomicity, and improves concurrent processing efficiency. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers. The invention will be described in detail below through specific embodiments.
[0012] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the multi-level group commit method for a distributed database provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.
[0013] like Figure 2 As shown, the multi-level group commit method for distributed databases proposed in this invention includes the following steps: S100: Obtain transaction requests sent by the client, and arrange the transaction requests in chronological order through data nodes to generate a transaction queue; S200: The data node merges the transaction queue into a local transaction group and sends the local transaction group to the computing node; S300. When the computing node receives a local transaction group sent by the data node, it integrates the local transaction group to generate a global transaction group. S400: The computing node sends a global group commit request to the data node, the data node provides feedback on the ready status based on the global group commit request, and the computing node executes the global transaction group commit operation based on the ready status.
[0014] In this embodiment, when a client issues a transaction request, the data node (data writing layer) first accesses these requests to the local transaction buffer and sorts them according to the arrival time of the transactions, forming an ordered transaction queue. This sorting method ensures the sequentiality of transactions during the local processing phase, laying the foundation for consistency in subsequent merging and committing.
[0015] Data nodes do not directly process individual transactions. Instead, they use an adaptive merging strategy to combine multiple transactions in the transaction queue into a local transaction group. The merging is based on a dynamically adjusted threshold, taking into account factors such as time window, number of transactions, and data volume. The merged data node generates a unified pre-commit version containing information such as version number, timestamp, number of transactions, data volume, and transaction time span, and sends this local transaction group to the compute nodes.
[0016] As the global coordination core, the compute node receives local transaction groups from multiple data nodes and further integrates them into a global transaction group. To ensure data consistency when global transaction groups execute concurrently, each global transaction group is assigned a globally incrementing SEQUENCE ID, which functions through a conflict resolution mechanism based on SEQUENCE COLUMN.
[0017] The global transaction group commit achieves atomicity through an extended two-phase commit: First, in the preparation phase, the compute node sends a global group commit request to all involved data nodes. Upon receiving the request, the data node locks its corresponding local transaction group and reports its readiness status to the compute node. After all data nodes confirm readiness, the commit phase begins. The compute node only needs to perform one persistence operation to write the metadata of the global transaction group (such as version number and timestamp), which can significantly reduce its own load. Subsequently, the data nodes will execute a unified commit of their local transaction groups to complete the entire transaction processing flow.
[0018] In one embodiment, S100 includes: S101, Receive transaction requests from clients through the data nodes of the distributed database; S102. The data nodes store the transaction requests to the local memory queue in chronological order to form a transaction queue.
[0019] In this embodiment, the data nodes of the distributed database act as the receiving terminals for transaction requests, directly accepting various transaction requests initiated by clients to ensure the directness and efficiency of request transmission. The data nodes store these requests one by one into a local memory queue according to the arrival time order, ultimately forming an ordered transaction queue. This time-ordered storage method provides a clear sequential basis for subsequent transaction merging and ensures the logical continuity of transaction processing. Choosing a local memory queue as a temporary storage medium reduces the intermediate steps of writing data to disk, lowers the latency of the transaction temporary storage stage, and lays the foundation for fast transaction processing in high-concurrency scenarios. Specifically, the data node is the core node in the distributed database responsible for data storage and local data processing, while the local memory queue is the memory space within the data node used to temporarily cache transaction requests. It features fast read / write speeds and low resource consumption, adapting to the rapid access requirements of high-concurrency small transactions.
[0020] In one embodiment, S200 includes: S201. Determine whether the transaction requests in the transaction queue meet the merging threshold; S202. If the transaction request meets the merging threshold, the data node will merge the transaction requests that meet the merging threshold into a local transaction group. S203. Generate a unified pre-commit version for the local transaction group and send the pre-commit version of the local transaction group to the compute node.
[0021] In this embodiment, the data node continuously monitors the status of the transaction queue and determines whether the transaction requests in the queue meet the preset merging threshold. The merging threshold here is not a fixed value, but is dynamically adjusted according to the real-time load of the data node based on an adaptive merging strategy.
[0022] When transaction requests in the transaction queue meet the dynamically adjusted merge threshold, the data node initiates a local transaction merge process, integrating multiple eligible transaction requests in the queue into a local transaction group. After the merge, the data node generates a unified pre-commit version for this local transaction group. This version contains complete core metadata, specifically covering key information such as version number, timestamp, number of transactions, data volume, and transaction time span. This information provides the basis for subsequent global coordination and conflict resolution among compute nodes. Finally, the data node synchronously sends the local transaction group carrying the pre-commit version to the compute nodes, completing the first level of local merge and data transmission, preparing for the subsequent integration and commit of global transaction groups. The merge threshold refers to the critical condition that triggers the transaction merge operation, and the pre-commit version is a standardized data carrier generated after the transaction group is merged, used to ensure the consistency of transaction information transmission and processing between nodes.
[0023] In one embodiment, S300 includes: S301. When the computing node receives a local transaction group sent by the data node, it initiates a global transaction group merging process. S302. The computing node integrates the local transaction groups according to the pre-committed version information to generate a global transaction group; S303. The computing node assigns a globally unique transaction ID to each global transaction group.
[0024] In this embodiment, when the compute node receives local transaction groups from various data nodes, it immediately initiates a global transaction group merging process. This operation is the core of the second level in the distributed two-level group commit architecture. Its core objective is to achieve global coordination and unified commit of transactions across multiple data nodes, breaking the limitations of single-node transaction processing and improving transaction processing efficiency in a distributed environment. As the coordinating core of global transactions, the compute node fully leverages the pre-commit version information generated by each local transaction group when merging local transaction groups. This information includes key details such as version number, timestamp, number of transactions, data volume, and transaction time span. Through the integration and analysis of this information, the scattered local transaction groups from different data nodes are further merged into a unified global transaction group. This integration method significantly reduces the number of interactions during cross-node transaction commits, avoiding the cumbersome communication overhead caused by separate commits from multiple local transaction groups, thereby significantly improving the overall transaction processing efficiency of the entire distributed database.
[0025] Meanwhile, considering that during concurrent execution of global transaction groups, different transaction groups may contain updates to the same primary key, leading to data inconsistency, compute nodes assign a globally unique transaction ID to each generated global transaction group. This ID is a globally incrementing SEQUENCE ID, whose globally incrementing characteristic ensures the uniqueness and order of each global transaction group. As the core identifier of global transactions, the SEQUENCE ID effectively guarantees the sequential consistency of different transaction groups when involving updates to the same primary key.
[0026] In one embodiment, S400 includes: S4011. Send a global group commit request to all data nodes through the computing node; S4012. When the data node receives a global group commit request, it performs a locking operation on the local transaction group and feeds back the ready status to the compute node. S4013. After the computing node receives the ready status feedback from all data nodes, it persists the metadata of the global transaction group to the write-ahead log. S4014. After the metadata of the global transaction group is persistently written to the write-ahead log, the final commit instruction is sent to all data nodes through the compute node. S4015. When the data node receives the final commit instruction, it executes the unified commit operation of the local transaction group.
[0027] In this embodiment, the global transaction group commit mechanism of this distributed database is implemented through an extended two-phase commit process to ensure transaction atomicity and data consistency. First, the compute node, acting as the global coordination core, sends a global group commit request to all data nodes participating in the global transaction group, initiating the commit preparation process. Upon receiving the request, a data node immediately performs a locking operation on its local transaction group to prevent interference from other concurrent operations, ensuring the independence of transaction execution. After locking, the data node reports its readiness status to the compute node, indicating that it is ready to commit.
[0028] Only after the compute node confirms that it has received readiness status feedback from all data nodes (i.e., all data nodes have completed local transaction group locking) will the persistence operation of the global transaction group metadata be performed—writing the global transaction group metadata (including version number, timestamp, and other key information) into the write-ahead log (WAL). The write-ahead log is a core mechanism for ensuring data reliability in databases. It requires that key metadata be written to the log before a transaction is committed. Even if a system failure occurs later, the transaction can be recovered through the log, ensuring that no data is lost.
[0029] Once the metadata of the global transaction group is successfully persisted to the write-ahead log, the compute node sends a final commit instruction to all data nodes, explicitly informing them that the final commit of their local transaction groups is ready. Upon receiving this instruction, the data nodes execute the unified commit operation for their local transaction groups, completing the actual writing and application of data, thus ending the entire global transaction group commit process. This process ensures the atomicity of transactions in a distributed environment (either all succeed or all fail) through a two-phase "prepare-commit" design, and significantly reduces the load on compute nodes and improves overall commit efficiency by requiring only one metadata persistence operation per compute node.
[0030] In one embodiment, S400 further includes: S4021. During the commit process of a global transaction group, the compute node determines whether there are update operations with the same primary key among the global transaction groups based on the globally unique transaction ID. S4022. When there are update operations with the same primary key among the global transaction groups, obtain the database type and determine the version control strategy based on the database type and the data mode of the table. S4023. Handle data conflicts in concurrent execution of the global transaction group according to the version control policy.
[0031] In this embodiment, during the submission process of a global transaction group, the computing node uses the allocated globally unique transaction ID (i.e., the globally incrementing SEQUENCE ID) to determine whether there are update operations with the same primary key between different global transaction groups. This ID is the core identifier that ensures the consistency of transaction order. Its globally incrementing characteristic can clearly define the execution order logic of each transaction group, thereby accurately identifying potential data conflict scenarios.
[0032] When an update operation with the same primary key is detected within a global transaction group, the system first obtains the specific database type and then determines the corresponding version control strategy based on the table's data schema. For OLTP (Online Transaction Processing) databases, multi-version control is automatically performed based on the SEQUENCE ID to ensure that the transaction execution order is consistent with the data update logic. For OLAP (Online Analytical Processing) databases, the system flexibly adapts to different table data schemas. Aggregate tables are updated directly without additional version control; detail tables do not require maintaining data versions and can be written directly; unique key tables require a version control mechanism to avoid consistency issues caused by uncertain write order, ultimately ensuring the accuracy and integrity of data after all transactions are committed in a distributed environment. This process echoes the conflict resolution concept based on SEQUENCE COLUMN and adapts to the business needs of different database types through differentiated version control strategies, effectively solving the data inconsistency problem when concurrent transactions update the same primary key.
[0033] In one embodiment, S201 further includes: S2011. Monitor the load status in real time through the data node and obtain load status data; S2012. Compare the load status data with a preset load threshold; S2013. When the load status data is greater than or equal to the preset load threshold, it is determined that the load status is high, and the local transaction merging threshold is reduced. S2014. When the load status data is less than the preset load threshold, it is determined that the load status is low, and the local transaction merging threshold is increased.
[0034] In this embodiment, in the multi-level dynamic group commit mechanism of this distributed database, the data nodes continuously monitor their own load status in real time. The core load status data acquired includes key indicators such as node CPU utilization, memory utilization, and transaction queue depth. These data directly reflect the current processing pressure and resource consumption of the data nodes and are the core basis for dynamically adjusting the merge threshold.
[0035] Data nodes compare real-time collected load status data with preset load thresholds. These preset load thresholds are baseline values set based on the optimal performance target of the database system and are used to define the high and low load ranges of the nodes. When the monitored load status data is greater than or equal to the preset load threshold, the system determines that the data node is under high load. If the original merge threshold is maintained at this time, it will lead to transaction queue backlog and increased latency in waiting for commit. Therefore, the system automatically lowers the merge threshold for local transactions, reduces the number of local transactions in a single group, and allows transactions to meet the merge conditions and commit more quickly. This rapidly releases system resources, alleviates resource contention pressure under high load, and reduces the response latency of individual transactions.
[0036] When the load status data is less than the preset load threshold, the system determines that it is in a low-load state. At this time, data node resources have idle space. In order to maximize disk I / O efficiency and reduce the number of transaction versions, the system will increase the local transaction merge threshold accordingly, allowing data nodes to integrate as many transactions in the transaction queue as possible, forming a larger local transaction group before committing. This method of dynamically adjusting the merge threshold based on load status breaks the limitations of static parameter configuration in traditional group commit mechanisms, achieving the optimal balance between resource utilization and transaction processing efficiency under different load scenarios, which fully conforms to the core design of the adaptive merge strategy of this invention. Among them, "load status data" refers to various indicator data reflecting the node's operating pressure, "preset load threshold" is the benchmark parameter for classifying high and low loads, and "merge threshold" is the critical condition for triggering transaction merge operations. Its dynamic adjustment is one of the key links in improving the multi-concurrency small transaction processing capability of distributed databases.
[0037] In one embodiment, a multi-level group commit device for a distributed database is provided, which corresponds one-to-one with the multi-level group commit method for the distributed database described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multi-level group commit device for the distributed database of the present invention. The modules include a transaction queue module 10, a local transaction group module 20, a global transaction group module 30, and a phase commit module 40. Detailed descriptions of each functional module are as follows: The transaction queue module 10 is used to obtain transaction requests sent by the client and arrange the transaction requests in chronological order through data nodes to generate a transaction queue. The local transaction group module 20 is used by the data node to merge the transaction queue into a local transaction group and send the local transaction group to the compute node; The global transaction group module 30 is used to integrate the local transaction groups and generate a global transaction group when the computing node receives the local transaction group sent by the data node. The phase commit module 40 is used to send a global group commit request to the data node through the computing node, the data node provides feedback on the ready status based on the global group commit request, and the computing node performs a global transaction group commit operation based on the ready status.
[0038] In one embodiment, the transaction queue module 10 includes: Receive transaction requests from clients through data nodes in a distributed database; The data nodes store the transaction requests in a local memory queue in chronological order, forming a transaction queue.
[0039] In one embodiment, the local transaction group module 20 includes: Determine whether the transaction requests in the transaction queue meet the merging threshold; If the transaction request meets the merging threshold, the data node will merge the transaction requests that meet the merging threshold into a local transaction group. A unified pre-commit version is generated for the local transaction group, and the local transaction group of the pre-commit version is sent to the compute node.
[0040] In one embodiment, the global transaction group module 30 includes: When the compute node receives a local transaction group sent by the data node, it initiates a global transaction group merging process; The computing nodes integrate local transaction groups based on the pre-committed version information to generate a global transaction group; Each global transaction group is assigned a globally unique transaction ID by the computing node.
[0041] In one embodiment, the stage submission module 40 includes: The computing node sends a global group commit request to all data nodes; When the data node receives a global group commit request, it performs a locking operation on the local transaction group and reports the ready status to the compute node. Once the compute node receives the ready status feedback from all data nodes, it persists the metadata of the global transaction group to the write-ahead log. After the metadata of the global transaction group is persistently written to the write-ahead log, the compute node sends a final commit instruction to all data nodes. Once the data node receives the final commit instruction, it executes a unified commit operation for the local transaction group.
[0042] In one embodiment, the stage submission module 40 further includes: During the global transaction group's commit process, the compute node determines whether there are update operations with the same primary key among the global transaction groups based on the globally unique transaction ID; When there are update operations with the same primary key in the global transaction group, obtain the database type and determine the version control strategy based on the database type and the data mode of the table; Data conflicts in concurrent execution of global transaction groups are handled according to the version control strategy.
[0043] In one embodiment, determining whether the transaction requests in the transaction queue meet the merging threshold further includes: The load status is monitored in real time through the data nodes to obtain load status data; The load status data is compared with a preset load threshold. When the load status data is greater than or equal to the preset load threshold, it is determined that the load is high, and the local transaction merging threshold is reduced. When the load status data is less than the preset load threshold, it is determined that the load status is low, and the local transaction merging threshold is increased.
[0044] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multi-level group submission method for a distributed database on the server side.
[0045] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the client-side functions or steps of a multi-level group submission method for a distributed database. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain transaction requests sent by the client, and arrange the transaction requests in chronological order through data nodes to generate a transaction queue; The data node merges the transaction queues into a local transaction group and sends the local transaction group to the compute node; When the computing node receives a local transaction group sent by the data node, it integrates the local transaction group to generate a global transaction group; The compute node sends a global group commit request to the data node, the data node returns a ready status based on the global group commit request, and the compute node executes a global transaction group commit operation based on the ready status.
[0046] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain transaction requests sent by the client, and arrange the transaction requests in chronological order through data nodes to generate a transaction queue; The data node merges the transaction queues into a local transaction group and sends the local transaction group to the compute node; When the computing node receives a local transaction group sent by the data node, it integrates the local transaction group to generate a global transaction group; The compute node sends a global group commit request to the data node, the data node returns a ready status based on the global group commit request, and the compute node executes a global transaction group commit operation based on the ready status.
[0047] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0048] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0049] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0050] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A multi-level group commit method for a distributed database, characterized in that, Includes the following steps: Obtain transaction requests sent by the client, and arrange the transaction requests in chronological order through data nodes to generate a transaction queue; The data node merges the transaction queues into a local transaction group and sends the local transaction group to the compute node; When the computing node receives a local transaction group sent by the data node, it integrates the local transaction group to generate a global transaction group; The compute node sends a global group commit request to the data node, the data node returns a ready status based on the global group commit request, and the compute node executes a global transaction group commit operation based on the ready status.
2. The multi-level group commit method for distributed databases as described in claim 1, characterized in that, The step of obtaining transaction requests sent by the client and arranging the transaction requests in chronological order through data nodes to generate a transaction queue includes: Receive transaction requests from clients through data nodes in a distributed database; The data nodes store the transaction requests in a local memory queue in chronological order, forming a transaction queue.
3. The multi-level group commit method for distributed databases as described in claim 1, characterized in that, The step of merging the transaction queues into local transaction groups by the data nodes and sending the local transaction groups to the compute nodes includes: Determine whether the transaction requests in the transaction queue meet the merging threshold; If the transaction request meets the merging threshold, the data node will merge the transaction requests that meet the merging threshold into a local transaction group. A unified pre-commit version is generated for the local transaction group, and the local transaction group of the pre-commit version is sent to the compute node.
4. The multi-level group commit method for distributed databases as described in claim 1, characterized in that, When the computing node receives a local transaction group sent by the data node, it integrates the local transaction groups to generate a global transaction group, including: When the compute node receives a local transaction group sent by the data node, it initiates a global transaction group merging process; The computing nodes integrate local transaction groups based on the pre-committed version information to generate a global transaction group; Each global transaction group is assigned a globally unique transaction ID by the computing node.
5. The multi-level group commit method for distributed databases as described in claim 1, characterized in that, The process of sending a global group commit request from the compute node to the data node, the data node responding with a ready status based on the global group commit request, and the compute node executing a global transaction group commit operation based on the ready status includes: The computing node sends a global group commit request to all data nodes; When the data node receives a global group commit request, it performs a locking operation on the local transaction group and reports the ready status to the compute node. Once the compute node receives the ready status feedback from all data nodes, it persists the metadata of the global transaction group to the write-ahead log. After the metadata of the global transaction group is persistently written to the write-ahead log, the compute node sends a final commit instruction to all data nodes. Once the data node receives the final commit instruction, it executes a unified commit operation for the local transaction group.
6. The multi-level group commit method for distributed databases as described in claim 1, characterized in that, The process includes sending a global group commit request from the compute node to the data node, the data node providing a readiness status response based on the global group commit request, and the compute node executing a global transaction group commit operation based on the readiness status. The process also includes: During the global transaction group's commit process, the compute node determines whether there are update operations with the same primary key among the global transaction groups based on the globally unique transaction ID; When there are update operations with the same primary key in the global transaction group, obtain the database type and determine the version control strategy based on the database type and the data mode of the table; Data conflicts in concurrent execution of global transaction groups are handled according to the version control strategy.
7. The multi-level group commit method for distributed databases as described in claim 3, characterized in that, Determining whether the transaction requests in the transaction queue meet the merging threshold also includes: The load status is monitored in real time through the data nodes to obtain load status data; The load status data is compared with a preset load threshold. When the load status data is greater than or equal to the preset load threshold, it is determined that the load is high, and the local transaction merging threshold is reduced. When the load status data is less than the preset load threshold, it is determined that the load status is low, and the local transaction merging threshold is increased.
8. A multi-level group commit device for a distributed database, characterized in that, The multi-level group commit device for the distributed database includes: The transaction queue module is used to obtain transaction requests sent by the client and arrange the transaction requests in chronological order through data nodes to generate a transaction queue; The local transaction group module is used by the data node to merge the transaction queues into a local transaction group and send the local transaction group to the compute node; The global transaction group module is used to integrate the local transaction groups sent by the data nodes and generate a global transaction group when the computing node receives the local transaction groups sent by the data nodes. The phase commit module is used to send a global group commit request to the data node through the compute node, the data node provides feedback on the ready status based on the global group commit request, and the compute node executes the global transaction group commit operation based on the ready status.
9. A computer device, characterized in that, The computer device includes a memory, a processor, and a multi-level group commit program for a distributed database stored on the memory and executable on the processor. When executed by the processor, the multi-level group commit program for a distributed database implements the steps of the multi-level group commit method for a distributed database as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a multi-level group commit program for a distributed database, which, when executed by a processor, implements the steps of the multi-level group commit method for a distributed database as described in any one of claims 1-7.