Distributed database transaction processing optimization method and device

Abstract

The invention discloses a distributed database transaction processing optimization method and device. The method comprises the following steps of: aggregating operation instructions and data of all participant nodes related to a distributed transaction by a coordinator node, and constructing a transaction request aggregation block; building an RDMA work request chain based on the aggregation block, wherein each chain entry corresponds to the RDMA transmission operation of a single participant; the whole WR chain is submitted to an RDMA network card through one-time doorbell pushing, and requests are issued asynchronously in batches by hardware; the participant node receives the data through the RDMA network card, asynchronously executes a local transaction and feeds back a response result; and the coordination node makes a global decision after aggregation response, and issues final instructions in batches through the WR chain again to complete transaction execution. According to the technical scheme, the number of node interaction times can be reduced, transaction delay can be reduced, and CPU loads can be unloaded.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for optimizing distributed database transaction processing. Background Technology

[0002] With the popularization of cloud computing and big data, distributed databases have become the core infrastructure for massive data processing due to their high scalability and high availability. However, distributed transaction processing is limited by the traditional TCP / IP communication architecture and has significant performance bottlenecks: frequent interactions between nodes based on protocols such as 2PC need to go through the kernel protocol stack, resulting in high network latency, and the CPU needs to bear non-business overhead such as serialization and interrupt handling, resulting in serious waste of computing power.

[0003] RDMA technology supports direct memory access across nodes, enabling low-latency, low-CPU-consumption communication and providing the possibility for optimizing transaction processing. However, existing RDMA application solutions have limitations: small request granularity and frequent interactions mean that complex transactions still face the problem of accumulated round-trip latency; the coordinator needs to build and submit each RDMA work request separately, which does not effectively reduce CPU load; and the work request chain (WR Chain) feature of the RDMA network card is not systematically utilized, and the hardware potential is not fully released.

[0004] Therefore, there is an urgent need in this field for a solution that deeply integrates the features of RDMA WR Chain, starting from the underlying communication mechanism, to reduce the number of node interactions, reduce transaction latency, and offload CPU load, thereby breaking through the bottlenecks of existing technologies. Summary of the Invention

[0005] This invention provides a method and apparatus for optimizing distributed database transaction processing, so as to reduce the number of node interactions, reduce transaction latency, and offload CPU load.

[0006] According to one aspect of the present invention, a distributed database transaction processing optimization method is provided, comprising:

[0007] After the coordinating node parses the transaction requirements, it constructs a transaction request aggregation block in local memory. The transaction request aggregation block contains the operation instructions and corresponding data that the transaction needs to send to all participant nodes. The operation instructions include prepare request, commit or abort instructions, and the corresponding data includes key-value pairs to be processed and log records.

[0008] The coordinating node constructs an RDMA work request chain based on the transaction request aggregation block. The work request chain consists of multiple consecutive work request entries. Each work request entry corresponds to an RDMA transmission operation of a participant node, and each work request entry is associated with the pre-registered receive buffer address of the target participant node and the corresponding data segment in the aggregation block.

[0009] The coordinating node submits the work request chain to the local RDMA network card and executes all transmission operations in batches asynchronously in sequence. It sends transaction requests to each participant node. Each participant node generates a completion queue event (CQE) through the RDMA network card, parses the request by polling the CQE, executes local transaction operations, and generates a processing response result.

[0010] The coordinating node collects and processes the responses from all participating nodes, makes a global commit or abort decision, and distributes the decision instructions in batches to each participating node through the work request chain to complete the transaction execution.

[0011] Optionally, constructing the transaction request aggregation block in local memory includes:

[0012] By using the metadata service of the distributed database, we can obtain the memory layout information, address range of the pre-registered receive buffer, and permission configuration of all participant nodes involved in the transaction.

[0013] According to the preset data alignment standard and storage format, the local pre-allocated memory area is divided into independent data segments, and the operation instructions and data payloads corresponding to each participant node are stored in the corresponding data segments respectively;

[0014] A mapping table is established between the unique identifier of the participant node and the memory offset and data length of the corresponding data segment within the aggregation block. The mapping table is used to associate and locate the RDMA work request entry with the data of the transaction request aggregation block, and the memory address of the transaction request aggregation block is pre-registered with RDMA.

[0015] Optionally, the construction of the RDM work request chain includes:

[0016] The RDMA aggregation engine of the coordinating node retrieves a pre-allocated idle work request chain structure from the cache pool of the work request chain, the work request chain structure containing a preset number of configurable work request entry slots.

[0017] For each participant node, parameter information is configured in the corresponding work request entry slot. The parameter information includes the unique identifier of the target participant node, the mapping relationship between the physical address and virtual address of the pre-registered receive buffer, the data transmission length, the RDMA operation type identifier, and the memory offset of the corresponding data segment in the transaction request aggregation block.

[0018] The work request entries are sorted according to a preset priority rule, and the transmission operations corresponding to each work request entry are executed sequentially by the RDMA network card according to the sorting result; and each work request entry is also configured with a timeout retransmission flag. If the corresponding transmission operation times out and is not completed, the RDMA network card is triggered to automatically retransmit the request data corresponding to the work request entry.

[0019] Optionally, the coordinating node is further configured with a work request chain cache pool and an aggregation block cache pool. The work request chain cache pool pre-allocates multiple work request chain structures of fixed length, and the aggregation block cache pool pre-allocates multiple memory blocks of different sizes. Correspondingly, the method further includes:

[0020] When constructing the transaction request aggregation block, a memory block of the appropriate specification is selected from the aggregation block cache pool for reuse;

[0021] When constructing the work request chain, an idle work request chain structure is retrieved from the work request chain cache pool. After the transaction is completed, the corresponding aggregate block memory block and work request chain structure are reset to an idle state to achieve cyclic reuse and reduce memory allocation and release overhead.

[0022] Optionally, each participant node generates a completion queue event (CQE) via an RDMA network card, parses requests by polling the CQE, executes local transaction operations, and generates a processing response result, including:

[0023] Participant nodes pre-register a fixed-size receive buffer via RDMA driver and synchronize the buffer address and permission information to the coordinating node;

[0024] After the RDMA network card completes the data writing, the transaction execution engine of the participant node directly accesses the transaction data in the pre-registered buffer through memory mapping.

[0025] When performing the local transaction operation, an exclusive lock is applied to the target data, the transaction log is written to the persistent storage medium, the lock resource is released after the data update is completed, and a response result containing the lock status, log writing result and data consistency verification result is generated.

[0026] Optionally, the collection and processing of response results from all participant nodes through the coordinating node includes:

[0027] After each participant node generates a response result, the response data is pushed to the pre-registered response buffer of the coordinating node through an RDMA SEND operation. The coordinating node obtains the response data by polling its own RDMA completion queue, and at the same time, establishes a response result summary table based on the association between the participant node identifier and the response data.

[0028] When all the participant nodes' response results are collected within the preset timeout period, the global decision-making process is initiated; if not all response results are collected within the timeout period, a global termination instruction is automatically generated.

[0029] Optionally, the method further includes:

[0030] The coordinating node dynamically adjusts the structure of the transaction request aggregation block and the configuration parameters of the WR entries according to the type of the upper-layer distributed transaction protocol. The upper-layer distributed transaction protocol includes the two-phase commit protocol 2PC, the Paxos Commit protocol, and the Raft Commit protocol.

[0031] During RDMA transmission operations, the system dynamically selects whether to enable RDMA memory zero-copy technology and data compression algorithms based on the amount of data transmitted, in order to optimize transmission efficiency.

[0032] According to another aspect of the present invention, a distributed database transaction processing optimization apparatus is provided, comprising:

[0033] The transaction request aggregation unit is used to construct a transaction request aggregation block in local memory after the coordinating node parses the transaction request. The transaction request aggregation block contains the operation instructions and corresponding data that the transaction needs to be sent to all participant nodes. The operation instructions include preparation request, commit or abort instructions, and the corresponding data includes key-value pairs to be processed and log records.

[0034] The work request chain construction unit is used to construct an RDMA work request chain based on the transaction request aggregation block by the coordinating node. The work request chain consists of multiple consecutive work request entries, each work request entry corresponds to an RDMA transmission operation of a participant node, and each work request entry is associated with the pre-registered receive buffer address of the target participant node and the corresponding data segment in the aggregation block.

[0035] The chained request submission unit is used to submit the work request chain to the local RDMA network card through the coordinating node to execute all transmission operations in sequence asynchronously in batches, send transaction requests to each participant node, and each participant node generates a completion queue event (CQE) through the RDMA network card, parses the request by polling the CQE and executes local transaction operations to generate a processing response result.

[0036] The decision-making unit is used to collect and process the response results of all participant nodes through the coordination node, make a global commit or abort decision, and send the decision instructions to each participant node in batches through the work request chain to complete the transaction execution.

[0037] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0038] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the distributed database transaction processing optimization method according to any embodiment of the present invention.

[0039] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the distributed database transaction processing optimization method according to any embodiment of the present invention.

[0040] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the distributed database transaction processing optimization method according to any embodiment of the present invention.

[0041] The technical solution of this invention involves the coordinating node first aggregating the operation instructions and data of all participant nodes involved in the distributed transaction to construct a transaction request aggregation block. Then, based on this aggregation block, an RDMA work request chain is built, with each chain entry corresponding to the RDMA transmission operation of a single participant. The entire WR chain is submitted to the RDMA network card via a single push notification, and the hardware asynchronously sends requests in batches. Participant nodes receive data through the RDMA network card and asynchronously execute local transactions, providing feedback on the response results. After aggregating the responses, the coordinating node makes a global decision and again sends the final instructions in batches through the WR chain to complete the transaction execution. This technical solution, by aggregating multiple network operations in the distributed transaction protocol into an RDMA work request chain, which is then executed asynchronously by the RDMA network card hardware in a single operation, significantly reduces the number of interactions with the server, lowers transaction latency, and significantly offloads the CPU load.

[0042] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a distributed database transaction processing optimization method provided in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of a method for constructing a transaction request aggregation block according to an embodiment of the present invention;

[0046] Figure 3This is a schematic diagram illustrating the construction of an RDMA work request chain applicable to an embodiment of the present invention;

[0047] Figure 4 This is a timing diagram of a two-phase commit process based on a WR chain applicable to an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of a distributed database transaction processing optimization device provided in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the distributed database transaction processing optimization method of this invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] Figure 1 This is a flowchart of a distributed database transaction processing optimization method provided by an embodiment of the present invention. This embodiment is applicable to situations where CPU load is reduced during distributed database transaction execution. The method can be executed by a distributed database transaction processing optimization device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0053] S110. After the coordinating node parses the transaction requirements, it constructs a transaction request aggregation block in local memory. The transaction request aggregation block contains the operation instructions and corresponding data that the transaction needs to send to all participant nodes. The operation instructions include preparation requests, commit or abort instructions, and the corresponding data includes key-value pairs to be processed and log records.

[0054] In a distributed database system, the coordinating node is responsible for coordinating transaction execution. It handles functions such as transaction parsing, request aggregation, response collection, and global decision-making, serving as the central coordinator connecting clients and participant nodes. Participant nodes, which store specific business data in the distributed database, receive transaction requests from the coordinating node, perform local data read / write operations, lock management, log persistence, and other operations, and then report the execution results back to the coordinating node.

[0055] Specifically, the coordinating node receives and parses the distributed transaction requests from clients, determining which participant nodes need to collaborate and what operations each node needs to perform. To avoid frequent interactions caused by sending requests to each node individually in traditional solutions, the coordinating node creates a unified transaction request aggregation block in its local memory, centrally storing the operation instructions and business data required by all participant nodes. The transaction request aggregation block is a unified data block built by the coordinating node in its local memory, used to centrally store all instructions and data that need to be sent to all participant nodes for this transaction, achieving one-time aggregation and batch distribution. The operation instructions include: a prepare request, which is an instruction during the transaction preparation phase, requiring participant nodes to complete preprocessing operations such as data locking, transaction log persistence, and data consistency verification; a commit instruction, which is the final execution instruction of the transaction, issued only when all participants are successfully prepared, requiring participant nodes to confirm data modifications and release resources; and abort instruction, which is a transaction rollback instruction, issued when any participant fails to prepare or times out without response, requiring participant nodes to undo the preprocessing operations they have performed and restore the data to its original state. The key-value pairs to be processed are the core business data that needs to be modified in the transaction. They are stored in key-value format and are the specific objects for participant nodes to perform data update operations.

[0056] S120. The coordinating node constructs an RDMA work request chain based on the transaction request aggregation block. The work request chain consists of multiple consecutive work request entries. Each work request entry corresponds to the RDMA transmission operation of a participant node, and each work request entry is associated with the pre-registered receive buffer address of the target participant node and the corresponding data segment in the aggregation block.

[0057] The RDMA Work Request Chain (WR Chain) is a logical link in RDMA technology consisting of multiple consecutive WR entries. It supports batch submission of all entries to the network interface card (NIC) via a single doorbell push operation, with hardware executing them sequentially and asynchronously. This eliminates the need for CPU to process and schedule each entry individually, achieving efficient batch data transmission. A WR entry is the basic unit of the RDMA Work Request Chain, containing the complete parameter configuration required for a single RDMA transmission operation. It serves as the carrier linking transmitted data to the target node, with each entry corresponding to an independent transmission task for a single participant node. RDMA transmission operations are cross-node data transmission behaviors implemented based on RDMA technology, including two types: RDMA SEND (data push to the target node's receive queue) and RDMA WRITE (direct writing to a specified memory area on the target node), neither of which requires intervention from the target node's operating system kernel. The pre-registered receive buffer address is a dedicated memory area address pre-allocated in the participant node's local memory and registered with the RDMA driver. This address has completed permission configuration and address mapping, ensuring direct access from the coordinating node's RDMA NIC, avoiding kernel protocol stack intervention and additional address resolution overhead during data transmission. The corresponding data fragment in the aggregation block is a data portion within the transaction request aggregation block that is exclusively allocated to a single participant node. It contains the operation instructions and business data required by the node to execute the transaction, and is associated with the corresponding WR entry through memory offset.

[0058] Specifically, based on the transaction request aggregation block already created by the coordinating node, the coordinating node actively completes the construction of the RDMA work request chain, ensuring that the data in the link and the aggregation block are completely matched. The work request chain is an ordered set of multiple consecutive WR entries, with the link length consistent with the number of participant nodes involved in the transaction, and each WR entry corresponds to a single participant node, carrying the RDMA transmission task of that node. Each WR entry contains: the target location pointing to the pre-registered receive buffer address of the corresponding participant node, ensuring that the data can be accurately delivered to the target memory area; and data association, which binds the exclusive data fragment of the participant in the transaction request aggregation block through memory offset to determine the specific content of this transmission.

[0059] S130. The work request chain is submitted to the local RDMA network card through the coordinating node. All transmission operations are executed asynchronously in batches in sequence. Transaction requests are sent to each participant node. Each participant node generates a completion queue event (CQE) through the RDMA network card. The request is parsed by polling the CQE and the local transaction operation is executed to generate the processing response result.

[0060] The Completion Queue Event (CQE) is an event notification generated in the RDMA completion queue of a participant node after the RDMA network card completes a single data transmission. It is used to inform the participant node that the corresponding transaction request has been successfully received and written to the pre-registered buffer, and is a signal indicating that the request to receive has been completed.

[0061] Specifically, the coordinating node does not need to submit individual transmission requests one by one. Instead, it submits the complete chain of work requests to its local RDMA network card in a single operation. The network card hardware independently handles all transmission tasks, sending the transaction requests corresponding to each participant node to the target node according to the WR entry order. During this process, the coordinating node's CPU is completely removed from transmission scheduling, achieving offloading of computing power. After receiving the corresponding data, each participant node's RDMA network card directly writes it to the pre-registered buffer and automatically generates a CQE, signaling to the node that the request has been received, ensuring the reliability and timeliness of data transmission. Participant nodes actively detect the arrival of requests by polling CQEs. The mechanism of active querying to check whether there is a CQE in the completion queue replaces the traditional interrupt notification method, avoiding CPU overhead caused by frequent interrupts and improving the efficiency and stability of request processing. After detecting a CQE, the participant node parses the transaction request from the pre-registered buffer and executes the local transaction operation, ensuring the consistency and atomicity of the transaction locally. After completing local processing, the participant node generates the corresponding processing response result based on the execution status.

[0062] S140. Collect and process the response results of all participant nodes through the coordinating node, make a global commit or abort decision, and send the decision instructions to each participant node in batches through the work request chain to complete the transaction execution.

[0063] Specifically, the coordinating node receives the response results from all participating nodes via the RDMA SEND operation through its own RDMA communication interface. It verifies the integrity and validity of the responses, excludes abnormal responses, and statistically analyzes the state distribution of all participants. Based on the atomicity and consistency requirements of distributed transactions, the coordinating node executes decision rules: if all participating nodes return "preparation successful" and there are no timeouts or anomalies, the transaction is deemed to meet the global execution conditions, and a global commit decision is generated; if any participant returns "preparation failed" or "data abnormal," or if all responses are not collected after exceeding a preset timeout threshold, the transaction is deemed to be at risk, and a global abort decision is generated to avoid data inconsistency caused by some nodes executing while others do not. The coordinating node reuses the batch transmission capability of the RDMA work request chain, integrating commit or abort decision instructions into a new transaction request aggregation block, constructing a corresponding WR chain, and submitting it to the RDMA network card via a single doorbell push. The hardware asynchronously batches the decision instructions to all participating nodes, avoiding the cumulative delay caused by issuing decisions to each node individually in traditional solutions. After receiving the decision instruction, each participant node executes the corresponding operation. Upon receiving the commit instruction, it confirms the data modification, persists the final log, and releases the lock resources. Upon receiving the abort instruction, it performs data rollback, deletes temporary data, and releases resources based on the transaction log. Finally, all participant nodes synchronously complete the transaction state closure, ensuring that the atomicity and consistency requirements of distributed transactions are met.

[0064] Figure 2 This is a flowchart of a transaction request aggregation block construction method provided by an embodiment of the present invention, such as... Figure 2 As shown, the method includes:

[0065] S210. Obtain the memory layout information, address range of pre-registered receive buffer, and permission configuration of all participant nodes involved in the transaction through the metadata service of the distributed database.

[0066] S220. According to the preset data alignment standard and storage format, divide the local pre-allocated memory area into independent data segments, and store the operation instructions and data payloads corresponding to each participant node into the corresponding data segments.

[0067] S230. Establish a mapping table between the unique identifier of the participant node and the memory offset and data length of the corresponding data segment in the aggregation block. The mapping table is used to associate and locate the RDMA work request entry with the transaction request aggregation block data, and the memory address of the transaction request aggregation block is pre-registered with RDMA.

[0068] Specifically, the coordinating node obtains the memory layout information, pre-registered receive buffer address range, and permission configuration of all participant nodes involved in the transaction through the metadata service of the distributed database, thereby determining the data transmission target and permission basis. According to the preset data alignment standard and storage format, it divides an independent data segment for each participant node in the locally pre-allocated memory area, and stores the operation instructions and data payloads corresponding to each node into the corresponding data segment to ensure the standardization and identifiability of data storage. It establishes a mapping table between the unique identifier of the participant node and the memory offset and data length of the corresponding data segment in the aggregate block. This mapping table is used to quickly associate and locate the target data in the aggregate block with RDMA work request entries. At the same time, it pre-registers the memory address of the transaction request aggregate block with RDMA to ensure that the RDMA network card can directly access the aggregate block.

[0069] In this embodiment of the invention, constructing the RDM work request chain includes:

[0070] The RDMA aggregation engine of the coordinating node retrieves a pre-allocated idle work request chain structure from the work request chain cache pool. The work request chain structure contains a preset number of configurable work request entry slots.

[0071] For each participant node, parameter information is configured in the corresponding work request entry slot. The parameter information includes the unique identifier of the target participant node, the mapping relationship between the physical address and virtual address of the pre-registered receive buffer, the data transmission length, the RDMA operation type identifier, and the memory offset of the corresponding data segment in the transaction request aggregation block.

[0072] The work request entries are sorted according to a preset priority rule, and the transmission operations corresponding to each work request entry are executed sequentially by the RDMA network card according to the sorting result. Each work request entry is also configured with a timeout retransmission flag. If the corresponding transmission operation times out and is not completed, the RDMA network card is triggered to automatically retransmit the request data corresponding to the work request entry.

[0073] Figure 3 This is a schematic diagram illustrating the construction of an RDMA work request chain applicable to an embodiment of the present invention. For example... Figure 3 As shown, the RDMA aggregation engine of the coordinating node, as the execution unit for building the RDMA work request chain, prioritizes retrieving idle work request chain structures from the pre-created work request chain cache pool. All work request chain structures in this cache pool have already completed memory allocation and basic initialization in advance, so there is no need to create them temporarily during transaction execution, which can avoid the performance overhead caused by repeated memory allocation and release. The work request chain structure contains a preset number of configurable work request entry slots, and the number of slots can be set in advance according to the range of the number of transaction participant nodes commonly found in distributed databases.

[0074] For each participant node involved in the transaction, complete parameter information is filled into the corresponding work request entry slot in the link to ensure that the RDMA network card can accurately execute data transmission operations. Specifically, the unique identifier of the target participant node is used to identify the object of data transmission and avoid node confusion; the mapping relationship between the physical and virtual addresses of the pre-registered receive buffer ensures that data is accurately written to the preset area; the data transmission length indicates the number of bytes transmitted in a single transmission, ensuring the integrity of data transmission; the RDMA operation type identifier determines the data transmission mode, adapting to the transmission requirements of different types of data; and the memory offset of the corresponding data segment in the transaction request aggregation block can quickly locate the storage location of the operation instructions and data payload required by the participant node in the aggregation block.

[0075] To optimize overall transmission efficiency and avoid resource contention or latency accumulation caused by disordered transmission, all configured work request entries are sorted according to a preset priority rule. This priority rule can be set based on factors such as the current load status of participant nodes, data transmission volume, or transaction correlation. After receiving the work request chain, the RDMA network card will execute the transmission operation corresponding to each entry in sequence according to the sorting result. At the same time, each work request entry is configured with a timeout retransmission flag. When the RDMA network card detects that the transmission operation corresponding to a certain entry has not been completed after exceeding the preset timeout threshold, it will automatically trigger the retransmission mechanism to resend the request data corresponding to that entry, so as to avoid the entire transaction execution being blocked due to a single transmission failure.

[0076] In this embodiment of the invention, the coordinating node is further configured with a work request chain cache pool and an aggregation block cache pool. The work request chain cache pool pre-allocates multiple work request chain structures of fixed length, and the aggregation block cache pool pre-allocates multiple memory blocks of different sizes. Correspondingly, the method further includes:

[0077] When constructing a transaction request aggregation block, a memory block of the appropriate specification is selected from the aggregation block cache pool for reuse;

[0078] When constructing a work request chain, an idle work request chain structure is retrieved from the work request chain cache pool. After the transaction is completed, the corresponding aggregate block memory block and work request chain structure are reset to an idle state to achieve cyclic reuse and reduce memory allocation and release overhead.

[0079] Specifically, to improve the response speed of distributed transaction processing and reduce memory management overhead, the coordinating node is configured with a work request chain cache pool and an aggregation block cache pool. The coordinating node first evaluates the specifications of the required memory blocks based on the actual needs of the current transaction, selects free memory blocks from the aggregation block cache pool that perfectly match the evaluation results, and uses them as the request aggregation block carrier for this transaction, filling in the operation instructions and data payloads of each participating node.

[0080] When constructing the RDMA work request chain, an idle work request chain structure is retrieved from the work request chain cache pool. Basic parameter configuration is completed during system initialization. After retrieval, the specific parameters of each WR entry can be configured directly based on the participant node information of the current transaction. After the entire distributed transaction completes its final execution, the coordinating node performs a reset operation on the aggregated block memory block and work request chain structure used in this transaction. For the aggregated block memory block, the stored operation instructions, data payloads, and other business data are cleared, while the block's specifications and RDMA registration information are retained. For the work request chain structure, the parameter configuration of each WR entry is reset, and its association with the current transaction is severed. The reset memory block and link structure are re-marked as idle and stored back in the corresponding cache pool for use by new distributed transactions. This not only reduces CPU overhead in memory management but also reduces memory fragmentation, ensuring long-term system stability and enhancing the low-latency, high-efficiency characteristics of distributed transaction processing.

[0081] In this embodiment of the invention, each participating node generates a completion queue event (CQE) through its RDMA network card, parses the request by polling the CQE, executes local transaction operations, and generates a processing response result, including:

[0082] Participant nodes pre-register a fixed-size receive buffer via RDMA driver and synchronize the buffer address and permission information to the coordinating node;

[0083] After the RDMA network card completes the data writing, the transaction execution engine of the participant node directly accesses the transaction data in the pre-registered buffer through memory mapping.

[0084] When performing local transaction operations, an exclusive lock is applied to the target data, the transaction log is written to the persistent storage medium, the lock resource is released after the data update is completed, and a response result containing the lock status, log writing result and data consistency verification result is generated.

[0085] Specifically, after a participant node connects to the distributed database system and enables transaction processing capabilities, it allocates a fixed-size dedicated area in its local memory as a receive buffer via the RDMA driver. The participant node declares the buffer's memory address, storage capacity, and access permissions to the RDMA driver, ensuring that the buffer is directly accessible by the remote coordinating node's RDMA network card. The participant node then synchronizes the physical address, virtual address mapping, and permission configuration information of this pre-registered receive buffer to the coordinating node through the distributed database's metadata service.

[0086] Once the coordinating node issues a transaction request through the RDMA work request chain, the RDMA network interface card (NIC) of the participant node will write the transaction request into the pre-registered receive buffer according to the address and data range specified in the WR entry. This process requires no intervention from the participant node's operating system kernel and does not consume CPU resources. After the RDMA NIC completes the data writing and generates a CQE, the participant node's transaction execution engine accesses the transaction data in the pre-registered buffer through memory mapping. This allows the transaction execution engine to directly map to the buffer's memory address space, reducing the latency of local transaction processing and minimizing CPU computational consumption during data copying.

[0087] After the transaction execution engine of the participant node parses the transaction request in the pre-registered buffer, it executes an exclusive lock on the target data involved in the transaction to prevent other concurrent transactions from modifying the same data at the same time. The transaction log is written to the persistent storage medium, so that even if the node fails later, the transaction can be rolled back or redone through the log to restore the data to the correct state. After the data is updated, the previously locked exclusive lock resources are released to avoid long-term lock occupation that may block other transactions. The transaction execution engine summarizes the key states of the entire local operation process and generates a complete response result. The lock state is used to inform the coordinating node whether the lock has been successfully acquired and released, the log writing result is used to confirm whether the transaction log has been reliably persisted, and the data consistency verification result is used to prove that the local data modification meets the transaction requirements.

[0088] In this embodiment of the invention, the coordination node collects and processes the response results of all participating nodes, including:

[0089] After each participant node generates a response result, the response data is pushed to the pre-registered response buffer of the coordinating node through the RDMA SEND operation. The coordinating node obtains the response data by polling its own RDMA completion queue, and at the same time, establishes a response result summary table based on the association between the participant node identifier and the response data.

[0090] When all participant nodes' responses are collected within the preset timeout period, the global decision-making process is initiated; if not all responses are collected within the timeout period, a global termination instruction is automatically generated.

[0091] Specifically, after each participant node completes its local transaction operation and generates a response result, it uses an RDMA SEND operation to push the response data to the response buffer pre-registered by the coordinating node. The coordinating node then obtains the response data by polling its own RDMA completion queue, rather than relying on interrupt notifications. This reduces the waste of CPU computing power caused by frequent interrupts and improves the efficiency and stability of response collection. At the same time, the coordinating node will build a response result summary table based on the association between the unique identifier of the participant node and the corresponding response data, clearly recording the processing status of each node and avoiding response data confusion or omission.

[0092] To prevent distributed transactions from being blocked indefinitely due to issues such as failure of some participant nodes or network congestion, the coordinating node pre-sets a reasonable timeout threshold when initiating the response collection process. When the coordinating node successfully collects the response results from all participant nodes within the preset timeout period, it initiates a global decision-making process to determine whether the transaction meets the global commit conditions based on the processing status of all nodes. If the timeout period has expired but the response results from all participant nodes have not been collected, the coordinating node automatically generates a global abort instruction. Allowing transactions to wait indefinitely for unresponsive nodes may result in data being locked for a long time, affecting system availability. Directly triggering the abort process allows all participant nodes that have completed their preparation operations to roll back their data in a timely manner, avoiding data inconsistency caused by the loss of connectivity of some nodes, while also releasing relevant resources to ensure the overall stability of the system.

[0093] In this embodiment of the invention, the method may further include the following steps:

[0094] The coordinating node dynamically adjusts the structure of the transaction request aggregation block and the configuration parameters of the WR entries according to the type of the upper-layer distributed transaction protocol. The upper-layer distributed transaction protocols include the two-phase commit protocol 2PC, the Paxos Commit protocol, and the Raft Commit protocol.

[0095] During RDMA transmission operations, the system dynamically selects whether to enable RDMA memory zero-copy technology and data compression algorithms based on the amount of data transmitted, in order to optimize transmission efficiency.

[0096] Figure 4This is a sequence diagram of a two-phase commit process based on a WR chain applicable to embodiments of the present invention. The upper-layer distributed transaction protocol is the logical framework for ensuring the consistency of distributed transactions. However, different protocols have significant differences in execution flow, instruction types, and data interaction requirements. To make the optimization scheme based on RDMA WR Chain universal, the coordinating node is designed with a protocol adaptive adjustment mechanism. When different upper-layer transaction protocols are accessed, the structure of the transaction request aggregation block and the configuration parameters of the WR entries are dynamically adjusted. Specifically, when adjusting the aggregation block structure, instruction fields are added or deleted, and the data storage order is adjusted according to protocol requirements; when adjusting the WR entry configuration parameters, the interaction flow of the protocol is adapted, and the timeout threshold is optimized. Without modifying the underlying RDMA transmission logic, it can connect to mainstream distributed transaction protocols, improving universality and compatibility.

[0097] The benefits of RDMA memory zero-copy technology vary depending on the data transfer volume. For small data transfers, the saved copy overhead is far less than the configuration cost of enabling the technology. Data compression algorithms can reduce bandwidth usage, but they require CPU resources for compression and decompression, which can lead to a decrease in overall efficiency for small data transfers. Therefore, a dynamic optimization selection mechanism is introduced during RDMA transmission. This mechanism makes decisions based on real-time data transfer volume. When the data transfer volume exceeds a preset threshold, both RDMA memory zero-copy technology and data compression algorithms are activated simultaneously. Zero-copy technology reduces latency, and compression algorithms reduce bandwidth usage, working together to maximize transmission efficiency. When the data transfer volume is below the preset threshold, only basic RDMA transmission functionality is retained, and zero-copy and compression algorithms are not activated to avoid unnecessary computational consumption and configuration overhead.

[0098] Figure 5 This is a schematic diagram of the structure of a distributed database transaction processing optimization device provided in an embodiment of the present invention. Figure 5 As shown, the device includes:

[0099] The transaction request aggregation unit 510 is used to construct a transaction request aggregation block in local memory after the coordinating node parses the transaction request. The transaction request aggregation block contains the operation instructions and corresponding data that the transaction needs to send to all participant nodes. The operation instructions include preparation request, commit or abort instructions, and the corresponding data includes key-value pairs to be processed and log records.

[0100] The work request chain construction unit 520 is used to construct an RDMA work request chain based on a transaction request aggregation block through a coordinating node. The work request chain consists of multiple consecutive work request entries. Each work request entry corresponds to an RDMA transmission operation of a participant node, and each work request entry is associated with the pre-registered receive buffer address of the target participant node and the corresponding data segment in the aggregation block.

[0101] The chained request submission unit 530 is used to submit the work request chain to the local RDMA network card through the coordinating node, execute all transmission operations in sequence asynchronously in batches, send transaction requests to each participant node, and each participant node generates a completion queue event (CQE) through the RDMA network card, parses the request by polling the CQE and executes local transaction operations to generate a processing response result.

[0102] The decision-making unit 540 is used to collect and process the response results of all participant nodes through the coordination node, make a global commit or abort decision, and send the decision instructions to each participant node in batches through the work request chain to complete the transaction execution.

[0103] The distributed database transaction processing optimization device provided in this embodiment of the invention can execute the distributed database transaction processing optimization method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0104] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0105] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0107] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as distributed database transaction processing optimization methods.

[0108] In some embodiments, the distributed database transaction processing optimization method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the distributed database transaction processing optimization method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the distributed database transaction processing optimization method by any other suitable means (e.g., by means of firmware).

[0109] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0111] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0113] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0114] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0115] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0116] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A distributed database transaction processing optimization method, characterized in that, include: After the coordinating node parses the transaction requirements, it constructs a transaction request aggregation block in local memory. The transaction request aggregation block contains the operation instructions and corresponding data that the transaction needs to send to all participant nodes. The operation instructions include prepare request, commit or abort instructions, and the corresponding data includes key-value pairs to be processed and log records. The coordinating node constructs an RDMA work request chain based on the transaction request aggregation block. The work request chain consists of multiple consecutive work request entries. Each work request entry corresponds to an RDMA transmission operation of a participant node, and each work request entry is associated with the pre-registered receive buffer address of the target participant node and the corresponding data segment in the aggregation block. The coordinating node submits the work request chain to the local RDMA network card and executes all transmission operations in batches asynchronously in sequence. It sends transaction requests to each participant node. Each participant node generates a completion queue event (CQE) through the RDMA network card, parses the request by polling the CQE, executes local transaction operations, and generates a processing response result. The coordinating node collects and processes the responses from all participating nodes, makes a global commit or abort decision, and distributes the decision instructions in batches to each participating node through the work request chain to complete the transaction execution.

2. The method according to claim 1, characterized in that, The construction of the transaction request aggregation block in local memory includes: By using the metadata service of the distributed database, we can obtain the memory layout information, address range of the pre-registered receive buffer, and permission configuration of all participant nodes involved in the transaction. According to the preset data alignment standard and storage format, the local pre-allocated memory area is divided into independent data segments, and the operation instructions and data payloads corresponding to each participant node are stored in the corresponding data segments respectively; A mapping table is established between the unique identifier of the participant node and the memory offset and data length of the corresponding data segment within the aggregation block. The mapping table is used to associate and locate the RDMA work request entry with the data of the transaction request aggregation block, and the memory address of the transaction request aggregation block is pre-registered with RDMA.

3. The method according to claim 1, characterized in that, The construction of the RDM work request chain includes: The RDMA aggregation engine of the coordinating node retrieves a pre-allocated idle work request chain structure from the cache pool of the work request chain, the work request chain structure containing a preset number of configurable work request entry slots. For each participant node, parameter information is configured in the corresponding work request entry slot. The parameter information includes the unique identifier of the target participant node, the mapping relationship between the physical address and virtual address of the pre-registered receive buffer, the data transmission length, the RDMA operation type identifier, and the memory offset of the corresponding data segment in the transaction request aggregation block. The work request entries are sorted according to a preset priority rule, and the transmission operations corresponding to each work request entry are executed sequentially by the RDMA network card according to the sorting result; and each work request entry is also configured with a timeout retransmission flag. If the corresponding transmission operation times out and is not completed, the RDMA network card is triggered to automatically retransmit the request data corresponding to the work request entry.

4. The method according to claim 1, characterized in that, The coordinating node is also configured with a work request chain cache pool and an aggregation block cache pool. The work request chain cache pool pre-allocates multiple work request chain structures of fixed length, and the aggregation block cache pool pre-allocates multiple memory blocks of different sizes. Correspondingly, the method further includes: When constructing the transaction request aggregation block, a memory block of the appropriate specification is selected from the aggregation block cache pool for reuse; When constructing the work request chain, an idle work request chain structure is retrieved from the work request chain cache pool. After the transaction is completed, the corresponding aggregate block memory block and work request chain structure are reset to an idle state to achieve cyclic reuse and reduce memory allocation and release overhead.

5. The method according to claim 1, characterized in that, Each participating node generates a completion queue event (CQE) through its RDMA network card, parses requests by polling the CQE, executes local transaction operations, and generates a processing response result, including: Participant nodes pre-register a fixed-size receive buffer via RDMA driver and synchronize the buffer address and permission information to the coordinating node; After the RDMA network card completes the data writing, the transaction execution engine of the participant node directly accesses the transaction data in the pre-registered buffer through memory mapping. When performing the local transaction operation, an exclusive lock is applied to the target data, the transaction log is written to the persistent storage medium, the lock resource is released after the data update is completed, and a response result containing the lock status, log writing result and data consistency verification result is generated.

6. The method according to claim 1, characterized in that, The process of collecting and processing the response results of all participating nodes through the coordinating node includes: After each participant node generates a response result, the response data is pushed to the pre-registered response buffer of the coordinating node through an RDMA SEND operation. The coordinating node obtains the response data by polling its own RDMA completion queue, and at the same time, establishes a response result summary table based on the association between the participant node identifier and the response data. When all the participant nodes' response results are collected within the preset timeout period, the global decision-making process is initiated; if not all response results are collected within the timeout period, a global termination instruction is automatically generated.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: The coordinating node dynamically adjusts the structure of the transaction request aggregation block and the configuration parameters of the WR entries according to the type of the upper-layer distributed transaction protocol. The upper-layer distributed transaction protocol includes the two-phase commit protocol 2PC, the Paxos Commit protocol, and the Raft Commit protocol. During RDMA transmission operations, the system dynamically selects whether to enable RDMA memory zero-copy technology and data compression algorithms based on the amount of data transmitted, in order to optimize transmission efficiency.

8. A distributed database transaction processing optimization device, characterized in that, include: The transaction request aggregation unit is used to construct a transaction request aggregation block in local memory after the coordinating node parses the transaction request. The transaction request aggregation block contains the operation instructions and corresponding data that the transaction needs to be sent to all participant nodes. The operation instructions include preparation request, commit or abort instructions, and the corresponding data includes key-value pairs to be processed and log records. The work request chain construction unit is used to construct an RDMA work request chain based on the transaction request aggregation block by the coordinating node. The work request chain consists of multiple consecutive work request entries, each work request entry corresponds to an RDMA transmission operation of a participant node, and each work request entry is associated with the pre-registered receive buffer address of the target participant node and the corresponding data segment in the aggregation block. The chained request submission unit is used to submit the work request chain to the local RDMA network card through the coordinating node to execute all transmission operations in sequence asynchronously in batches, send transaction requests to each participant node, and each participant node generates a completion queue event (CQE) through the RDMA network card, parses the request by polling the CQE and executes local transaction operations to generate a processing response result. The decision-making unit is used to collect and process the response results of all participant nodes through the coordination node, make a global commit or abort decision, and send the decision instructions to each participant node in batches through the work request chain to complete the transaction execution.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a distributed database transaction processing optimization method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the distributed database transaction processing optimization method according to any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements a distributed database transaction processing optimization method according to any one of claims 1-7.

Images