Distributed database system based on RDMA (Remote Direct Memory Access) and storage and calculation separation and data processing method thereof

Abstract

The invention relates to the field of databases, in particular to a distributed database system based on RDMA and storage and calculation separation and a data processing method of the distributed database system. According to the method, a distributed architecture is constructed through stateless processing calculation of a working node WN and fragmented data storage of a data node DN; the core of the method is that after a WN receives an SQL (Structured Query Language), intelligent pruning is carried out in combination with a distributed partition table and a consistent Hash ring to generate an accurate push-down data node table and an execution plan, and then data transmission with a target DN is carried out through a hybrid communication channel based on RDMA (Remote Direct Memory Access); according to the hybrid channel, RDMA bilateral operation is carried out on small messages, RDMA unilateral read operation is carried out on a large-scale result set, and zero-copy transmission is achieved with the assistance of a global memory pool. According to the scheme, remarkable performance improvement is realized, including microsecond-level network delay, query throughput improved by several times, minute-level elastic capacity expansion and low-delay submission of cross-fragmentation transactions, and the performance bottleneck of a traditional distributed database is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of databases, and more specifically, to a distributed database system based on RDMA and storage-compute separation, and its data processing method. Background Technology

[0002] With the continuous growth of data volume, distributed databases have become a key infrastructure supporting modern applications. Their mainstream architecture is shifting from tight coupling of storage and computation to separation of storage and computation, in order to achieve better elasticity and resource utilization.

[0003] However, existing technical solutions still face several serious challenges. First, the traditional TCP / IP network protocol stack inherently suffers from kernel overhead and multiple data copying issues, becoming a major bottleneck restricting the performance of distributed databases and making it difficult to further reduce query latency. Second, when processing cross-shard transactions, the traditional two-phase commit protocol, due to its serial communication mode, introduces multiple network round trips, resulting in high latency and causing transaction processing performance to significantly decrease as the number of nodes increases. Furthermore, during data node expansion, the traditional consistent hashing algorithm still requires migrating a large amount of data, leading to lengthy expansion times and difficulty in achieving true elastic scaling. Moreover, improper resource management during this process can easily lead to service instability.

[0004] Therefore, there is an urgent need in this field for a new distributed database data processing solution that can fundamentally reduce network communication overhead, optimize distributed transaction processing efficiency, and achieve rapid elastic scaling. Summary of the Invention

[0005] The purpose of this invention is to provide a database system and data processing method that effectively solves the performance bottleneck of traditional distributed databases.

[0006] According to a first aspect of the present invention, a distributed database system based on RDMA and storage-compute separation is proposed, comprising: At least one working node WN, and multiple data nodes DN connected via network communication; The worker node WN is used to receive query requests, perform SQL parsing, optimization and execution plan generation, and maintain a distributed partition table containing sharding strategies and sharding mapping relationships; the data node DN is used to store data shards and respond to data operation commands from the worker node WN. Data transmission is conducted between the working node WN and the data node DN, as well as between multiple data nodes DN, through a hybrid communication channel based on RDMA. The system manages data distribution through a sharding mapping mechanism based on a consistent hash ring and supports elastic expansion of data nodes (DNs).

[0007] According to some embodiments, in the database system of the first aspect of the present invention, the RDMA-based hybrid communication channel is configured as follows: For data packets less than or equal to a preset threshold, RDMA bilateral operation is used for transmission; For data packets or result sets exceeding a preset threshold, RDMA single-sided read operation is used for transmission; Worker node WN and data node DN register a global memory pool during system initialization and reuse the registered memory regions for RDMA communication during operation.

[0008] According to some embodiments, in the database system of the first aspect of the present invention, the elastic expansion process of data nodes (DNs) includes: Create a new data node DN; Reassign half of the adjacent virtual nodes on the original consistent hash ring to the newly added data node DN; Update the mapping relationship between virtual nodes and data nodes (DNs) in the distributed partitioning table. In the background, asynchronously, the fragmented data to be migrated is written directly from the old data node DN to the buffer of the newly added data node DN through RDMA one-sided read operation.

[0009] According to some embodiments, in the database system of the first aspect of the present invention, the elastic expansion process of data nodes (DNs) further includes: Rename the fragment files locally on the old data node DN; Update the shard location metadata in the distributed partition table to complete the route switch.

[0010] According to a second aspect of the present invention, a distributed database data processing method based on RDMA and storage-compute separation is proposed, applied to a database system as described in the first aspect of the present invention, comprising the following steps: S1. Receive SQL statements sent by clients through worker node WN and allocate connection resources from the global memory pool; S2. Worker node WN performs intelligent pruning based on distributed partitioned tables and consistent hash rings to determine the pushdown data node table associated with the SQL statement; S3. Generate an execution plan containing pushdown operators based on the pushdown decision optimizer; S4. Data is transmitted between the working node WN and one or more target data nodes DN in the pushdown data nodes through a hybrid communication channel based on RDMA to execute the pushdown plan and generate execution results; S5. Return the execution result to the client.

[0011] According to some embodiments, in the method of the second aspect of the present invention, step S2 includes: Extract one or more shard key constants from an SQL statement; The distributed partitioning table is queried based on the sharding key constant to determine the sharding strategy corresponding to the target data node DN; Calculate the target virtual node corresponding to the sharding key constant based on the sharding strategy and consistent hash ring; Map the target virtual node to the corresponding physical data node (DN) to generate an initial node set; The initial node set is pruned to exclude data nodes (DNs) that do not contain relevant data, and a pushdown data node table is generated.

[0012] According to some embodiments, in the method of the second aspect of the present invention, the push-down data node table includes data node identifiers, virtual node ranges, sharding key values, and RDMA buffer information; the step of pruning the initial node set to exclude data node DNs that do not contain relevant data, and generating the push-down data node table includes: Based on the calculation results of the fragmentation key constant, determine the actual data node DN bitmap that needs to be accessed; Compare the initial node set with the data node DN bitmap to identify and eliminate irrelevant data node DNs; Record the data node identifier, virtual node range, and sharding key value corresponding to each target data node DN in the pushdown data node table; Allocate an RDMA communication buffer for each target data node (DN) and update the buffer address and size information to the RDMA buffer information in the pushed-down data node table.

[0013] According to some embodiments, in the method of the second aspect of the present invention, step S4 includes: S41. Serialize the pushdown plan into a byte stream and send the serialized pushdown plan to the input buffer of each target data node (DN) via RDMA-send operation; S42. Each target data node (DN) reads and deserializes the pushdown plan from the input buffer and executes the pushdown operator locally; S43. For small-scale intermediate results, each target data node (DN) sends the results directly to the working node (WN) via RDMA-send operation; S44. For large-scale result sets, the worker node WN reads the result data directly from the output buffer of each target data node DN via RDMA-read operation; S45. The worker node WN merges the intermediate results returned by each target data node DN to generate the final execution result.

[0014] According to some embodiments, in the method of the second aspect of the present invention, when the transaction corresponding to the SQL statement is a cross-shard transaction, step S4 further includes a lock-free globally ordered write mechanism: Worker node WN obtains write ticket sequence numbers from each target data node DN through the RDMA atomic operation Fetch-and-Add. The write ticket sequence number is used to ensure the global order when multiple worker nodes WN concurrently write to the same data shard. Each target data node (DN) performs data operations in the order of the write ticket number, achieving lock-free ordered disk write.

[0015] According to some embodiments, in the method of the second aspect of the present invention, step S4 further includes an optimized two-phase commit processing mechanism: For cross-shard transactions, a two-phase commit protocol 2PC is used, in which both prepare and commit messages are broadcast in parallel to all relevant target data nodes (DNs) via RDMA-send; Both prepare and commit messages are sent by completing communication between all target data nodes (DNs) in a single network round trip.

[0016] The solution proposed in this invention solves the problems of network transmission latency and low efficiency of elastic scaling in existing database systems, and has the following beneficial effects: 1. RDMA Hybrid Channel Protocol Stack: The hybrid mode proposed in this invention, which uses RDMA bilateral operation for small packets and RDMA unilateral reading for large result sets, solves the kernel overhead and multiple copying problems of the traditional TCP / IP protocol stack, significantly reduces network latency, increases throughput and greatly reduces CPU usage.

[0017] 2. Intelligent pruning and full operator pushdown engine: This invention solves the problems of broadcast storms and invalid data transmission in cross-shard queries by dynamically calculating the target DN bitmap during the parsing stage and supporting the pushdown of multiple operators, which greatly reduces network traffic and data transmission volume and improves query response speed.

[0018] 3. Lock-free globally ordered write mechanism: Utilizing RDMA atomic operations (Fetch-and-Add) to provide a global sequence number when multiple worker nodes concurrently write to the same shard, it solves the performance bottleneck and single point of failure risk of distributed locks, and achieves low-latency ordered write under high concurrency.

[0019] 4. Hash ring split expansion and RDMA accelerated migration: When expanding the consistent hash ring, the strategy of migrating only about half of the adjacent virtual nodes is combined with RDMA-read to achieve zero CPU participation in data migration. This aims to solve the problems of large data migration volume and long service interruption time in traditional expansion, and achieve smooth elastic expansion at the minute or even second level.

[0020] 5. Parallel two-phase commit optimization: For cross-shard transactions, the multi-round communication latency of traditional 2PC is reduced to a single round trip, directly solving the core latency bottleneck of distributed transactions.

[0021] This invention deeply integrates RDMA network capabilities with core distributed database technologies, achieving near-hardware-limit performance while maintaining the ease of use of traditional databases. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without exceeding the scope of protection claimed by the present invention.

[0023] Figure 1 This is a schematic diagram of the architecture of an embodiment 1000 of the distributed database system based on RDMA and storage-compute separation of the present invention; Figure 2 This is a flowchart illustrating an embodiment 2000 of the data processing method for a distributed database system based on RDMA and storage-compute separation according to the present invention. Figure 3 for Figure 2 A flowchart illustrating step S2 in Example 2000; Figure 4 for Figure 3 A flowchart illustrating step S25; Figure 5 for Figure 2 A flowchart illustrating step S4 in Example 2000. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Figure 1 This is a schematic diagram of the architecture of embodiment 1000 of the distributed database system based on RDMA and storage-compute separation of the present invention. Figure 1 As shown, Embodiment 1000 includes a working node WN101, a data node DN102, and a data node DN103.

[0026] Optionally, worker node WN101 receives query requests, performs SQL parsing, optimization, and execution plan generation, and maintains a distributed partition table containing sharding strategies and sharding mapping relationships. Data nodes DN102 and DN103 store data shards and respond to data operation commands from worker node WN.

[0027] Among them, data transmission between working node WN101 and data nodes DN102 and DN103, as well as between data nodes DN102 and DN103, is carried out through a hybrid communication channel based on RDMA.

[0028] Optionally, in embodiment 1000, the RDMA-based hybrid communication channel includes: For data packets less than or equal to the preset threshold, RDMA bilateral operation is used for transmission; for data packets or result sets greater than the preset threshold, RDMA unilateral read operation is used for transmission. Worker node WN101 and data nodes DN102 and DN103 register a global memory pool during system initialization and reuse the registered memory area for RDMA communication during operation. Optionally, the preset threshold is 32KB.

[0029] Optionally, in some specific embodiments, in a hybrid communication channel based on RDMA: For small messages, RDMA bilateral operation is used. Specifically, the worker node (WN) uses RDMA-Send to push the request to the target data node (DN). The data node (DN) then needs to prepare and place the RDMA-Recv request in its network card's work queue to receive the incoming message. This is a communication mode that requires cooperation from the receiving CPU. Although it still uses kernel bypass and zero-copy, the receiving CPU needs to consume cycles to prepare the receive descriptor. This is suitable for high-frequency, small-batch interactions, such as control commands and data write requests.

[0030] For large result sets, a single-sided RDMA read operation is employed. The specific process is as follows: the worker node (WN) directly sends an RDMA-Read command to its local network interface card (NIC), containing the memory address and access key of the target data node (DN). Subsequently, the NIC of the worker node (WN) directly reads the data from the specified area in the memory of the remote data node (DN) over the network and writes it directly to the local memory buffer specified by the application layer of the worker node (WN). Throughout the entire data transmission process, the CPU of the data node requires no intervention; it neither needs to be aware of the read operation nor execute any instructions to process it. This process is crucial to the performance of the hybrid communication channel, achieving zero-CPU access to remote data and completely freeing up the CPU resources of the data node (DN) for processing local computing tasks.

[0031] To support the specific implementation of RDMA-based hybrid communication channels, this invention maintains a global memory pool, the main functions of which include: Eliminate the overhead of duplicate registration, avoid registering memory regions separately for each connection and each communication, and reduce system calls and CPU overhead for memory registration / deregistration; achieve zero-copy transmission, pre-registered memory regions can be directly accessed by remote nodes, and data is transmitted directly between application layer memory and network without copying the kernel buffer; ensure memory access security, control the access permissions of remote nodes through an access key mechanism, and only nodes holding a valid rkey can access the specified memory region.

[0032] Optionally, the global memory pool is registered during the system initialization phase. The specific registration steps include: when the worker node WN starts, registering a memory pool for receiving query results; when the data node DN starts, registering a memory pool for storing sharded data.

[0033] Memory pool location distribution: Each node independently maintains its own global memory pool; the memory pool is located in user space, but is registered as remotely accessible via RDMA network card; the size is dynamically adjusted according to the node role, with WN focusing on result caching and DN focusing on data storage.

[0034] In some specific embodiments, the database system of the present invention includes at least one working node WN and multiple data nodes DN connected via network communication.

[0035] In some specific embodiments, the system proposed in this invention manages data distribution through a sharding mapping mechanism based on a consistent hash ring and supports elastic scaling of data nodes (DNs).

[0036] Optionally, the elastic expansion process of the data node DN in this invention includes: creating a new data node DN; reallocating half of the adjacent virtual nodes on the original consistent hash ring to the new data node DN; updating the mapping relationship between virtual nodes and data nodes DN in the distributed partition table; and asynchronously writing the fragmented data to be migrated out directly from the old data node DN to the buffer of the new data node DN through RDMA one-sided read operation in the background.

[0037] Optionally, the elastic scaling process of a data node (DN) also includes: renaming the shard files locally on the old data node DN; and updating the shard location metadata in the distributed partition table to complete the routing switch.

[0038] In traditional distributed databases, the data migration process during scaling typically involves: reading data from the source node; serializing the data and transmitting it over the network; and deserializing and writing it to the target node. This process involves significant amounts of CPU computation, memory copying, and network I / O.

[0039] The core principle of the elastic scaling process in this invention lies in decoupling logical data distribution from physical file storage, and achieving efficient scaling through the mathematical properties of consistent hash rings and the atomicity of file system operations. Logical layer: The consistent hash ring determines which virtual node the data should belong to; Physical layer: The fragmented file determines on which data node's disk the data is actually stored on.

[0040] The mathematical principle of a consistent hash ring includes its symmetry. In some specific embodiments, during the scaling process, the consistent hash ring: Initial state: Virtual nodes are evenly distributed on the ring. Suppose there are N virtual nodes: V0, V1, V2, ..., V_{N-1} Allocation before expansion: DN1: [V0, V1, V2, ..., V_{N / 2-1}] / / First half DN2: [V_{N / 2}, V_{N / 2+1}, ..., V_{N-1}] / / Second half After expansion, the capacity will be redistributed: DN1: [V0, V1, ..., V_{N / 4-1}] / / first 1 / 4 DN2: [V_{N / 2}, V_{N / 2+1}, ..., V_{3N / 4-1}] / / The first half of the middle half DN3: [V_{N / 4}, V_{N / 4+1}, ..., V_{N / 2-1}, / / The latter half of the original DN1 V_{3N / 4}, V_{3N / 4+1}, ..., V_{N-1}] / / The second half of the original DN2 In some specific embodiments, the present invention transforms the traditional view of data migration as equivalent to data movement into a view where data migration is essentially a change in data ownership: In the traditional approach, import and export require network transmission, with the source node completing data reading -> network sending -> deleting local files, and the target node completing network receiving -> writing data; while in the present invention, ownership is transferred only through metadata updates, with the source node renaming files, and the target node creating new fragment file references after RDMA direct reading is completed.

[0041] Optionally, in some specific embodiments, in the flexible expansion scheme of the present invention, while the file is being renamed, the background performs actual data synchronization via RDMA-read; Source node: After the file is renamed, the file becomes read-only, but it can still be read by the new node via RDMA; New node: Reads data directly from the source node's memory cache via RDMA-read. This scheme has the advantage of zero copy, that is, data is directly transferred from the source node's memory to the new node's memory, bypassing the operating system kernels of both nodes.

[0042] Specifically, in a database system that includes DN1 and DN2, the process of adding a new data node DN3 includes: 1. Hash ring fission and metadata update Initial ring: [V0(DN1), V1(DN1), V2(DN2), V3(DN2), V4(DN1), V5(DN1), V6(DN2), V7(DN2)] Added DN3 rear ring: [V0(DN1), V1(DN3), V2(DN2), V3(DN3), V4(DN1), V5(DN1),V6(DN2), V7(DN2)] DN3 connectors: V1 (originally DN1) and V3 (originally DN2) Migrating amount: 2 / 8 = 1 / 4 of the virtual nodes, but since each virtual node carries a different amount of data, the total amount is approximately 1 / 2 of the data. 2. Atomicity of file operations Source node preparation: Rename the shard file to be migrated from shard_v1.active to shard_v1.migrating. This renaming operation is atomic, ensuring that no new writes will occur during the migration process.

[0043] RDMA data synchronization: The new node DN3 reads the data corresponding to shard_v1.migrating from the source node's memory via RDMA-read. Due to the zero-copy characteristic of RDMA, the source node's CPU is almost uninvolved.

[0044] Ownership switch: After data synchronization is complete, DN3 creates the shard_v1.active file, and the source node deletes the shard_v1.migrating file.

[0045] Figure 2 This is a flowchart illustrating an embodiment 2000 of the data processing method for a distributed database system based on RDMA and storage-compute separation according to the present invention. Figure 2 As shown, Example 2000 includes steps S1-S5.

[0046] In step S1, the SQL statement sent by the client is received through the worker node WN, and connection resources are allocated from the global memory pool.

[0047] Optionally, in step S1, the worker node WN listens for client connection requests through the standard database port. After the operating system completes the TCP three-way handshake, the WN's network framework receives new connection events and then ensures that the connections are evenly distributed to each WN instance through the load balancer.

[0048] For example, in step S1, the specific allocation process for connection resources includes: Connection context allocation: Obtain a connection block from the free list of the connection resource pool and initialize the connection ID, client address, and authentication status; Buffer allocation: Input buffer: 64KB, used to receive SQL requests from clients; Output buffer: 128KB, divided into small response block (4KB), medium response block (16KB), and large response block (64KB); RDMA memory registration: For clients that support RDMA, pre-register the output buffer, generate the access key (rkey) and remote address.

[0049] In step S2, worker node WN performs intelligent pruning based on the distributed partition table and consistent hash ring to determine the pushdown data node table associated with the SQL statement.

[0050] Optionally, the intelligent pruning process in step S2 specifically includes: extracting one or more sharding key constants from the SQL statement; querying the distributed partitioning table based on the sharding key constants to determine the sharding strategy corresponding to the target data node DN; calculating the target virtual node corresponding to the sharding key constant based on the sharding strategy and the consistent hash ring; mapping the target virtual node to the corresponding physical data node DN to generate an initial node set; pruning the initial node set to exclude data node DNs that do not contain relevant data, and generating a push-down data node table.

[0051] Optionally, the distributed partitioned table stores the following key information: Sharding Key: Which column(s) are used as the sharding basis; Sharding Strategy Type: Hash, Range, or List; Strategy Parameters: Specific sharding rule parameters; Sharding Mapping Relationship: The mapping from logical shards to virtual nodes.

[0052] In step S3, an execution plan containing pushdown operators is generated based on the pushdown decision optimizer.

[0053] Optionally, in step S3, the pushdown plan is formulated based on two core principles: maximizing data locality and minimizing network transmission. Cost-benefit analysis is used to determine which operations should be pushed down to data nodes for execution, and the evaluation metrics include: Data reduction rate = (Original data volume - Processed data volume) / Original data volume Network cost savings = Reduced data volume × Network transmission latency Computational load balancing = Data node computational overhead - Worker node computational overhead In step S4, data is transmitted between the working node WN and one or more target data nodes DN in the pushdown data nodes through a hybrid communication channel based on RDMA to execute the pushdown plan and generate execution results.

[0054] Optionally, step S4 specifically includes: serializing the pushdown plan into a byte stream and sending the serialized pushdown plan to the input buffer of each target data node (DN) via an RDMA-send operation; each target data node (DN) reads and deserializes the pushdown plan from the input buffer and executes the pushdown operator locally; for small-scale intermediate results, each target data node (DN) sends the results directly to the worker node (WN) via an RDMA-send operation; for large-scale result sets, the worker node (WN) reads the result data directly from the output buffer of each target data node (DN) via an RDMA-read operation; and the worker node (WN) merges the intermediate results returned by each target data node (DN) to generate the final execution result.

[0055] In step S5, the execution result is returned to the client. Optionally, in step S5, the worker node DN converts the internal data structure into the client protocol format, adds metadata information, and serializes it into a network byte stream. The client then returns the result by directly reading the WN output buffer through RDMA path optimization.

[0056] Figure 3 for Figure 2 A flowchart illustrating step S2 in Example 2000. (See attached diagram.) Figure 3 As shown, step S2 includes steps S21-S25.

[0057] In step S21, worker node WN extracts one or more shard key constants from the SQL statement.

[0058] Optionally, in step S21, the parser of the worker node WN performs lexical and syntactic analysis on the received SQL statement to generate an Abstract Syntax Tree (AST); it traverses the AST to identify shard key references contained in WHERE conditions, JOIN conditions, INSERT VALUES, etc.; it extracts specific constant values ​​from the identified shard key references; and it performs data type conversion and normalization on the extracted constants.

[0059] For example, the SQL statement received in step S1 is: SELECT user_name, order_amount FROM orders WHERE user_id IN (1001,1002, 1003) AND create_date>= '2024-01-01' ORDER BY order_amount DESC LIMIT50 Therefore, in step S21, the sharding key user_id values ​​extracted from the query conditions are: 1001, 1002, and 1003.

[0060] In step S22, the worker node WN queries the distributed partition table based on the sharding key constant to determine the sharding strategy corresponding to the target data node DN.

[0061] In some specific embodiments, in step S22, the worker node WN queries the distributed partition table to obtain the sharding configuration information of the target table; determines the sharding strategy type used by the table; obtains the specific parameter configuration of the sharding strategy; and verifies the compatibility between the sharding key constant and the sharding strategy.

[0062] Optionally, strategy type processing includes: for hash shards, obtaining the hash function name and parameters; for range shards, obtaining the range boundary definition; and for list shards, obtaining the list of mappings from values ​​to shards.

[0063] Optionally, the distributed partition table stores static table-level metadata, including: sharding key, sharding strategy, and mapping of shards to virtual nodes. In step S22, worker node WN queries the distributed partition table for the sharding strategy and mapping relationship corresponding to the sharding key constant.

[0064] In step S23, the worker node WN calculates the target virtual node corresponding to the sharding key constant according to the sharding strategy. Specifically, this includes: applying the corresponding calculation logic to each sharding key constant according to the sharding strategy type; mapping the calculation results to the virtual node space of the consistent hash ring; and merging identical virtual nodes to reduce redundant calculations.

[0065] Optionally, in step S23, the sharding key constant user_id = 1001, and the corresponding sharding strategy is a hash sharding strategy. The hash function hash(user_id) is applied to the sharding key, and the hash result is mapped to the range of virtual nodes, resulting in virtual node = hash(user_id) % 1024. For example, user_id = 1001 -> hash(1001) = 357 -> virtual node = 357.

[0066] Optionally, in step S23, if the sharding strategy corresponding to the sharding key constant is a range sharding strategy, then the virtual node is determined by directly comparing the sharding key value with the range boundary. Optionally, if the sharding strategy is a list sharding strategy, the virtual node is determined by direct mapping based on the enumeration value.

[0067] In step S24, the working node WN maps the target virtual node to the corresponding physical data node DN based on a consistent hashing ring, generating an initial node set. Optionally, the consistent hashing ring is a dynamic data distribution algorithm that includes the mapping relationship from virtual nodes to physical data nodes.

[0068] In some specific embodiments, in step S24, the working node WN queries the current state information of the consistent hash ring; maps the calculated virtual nodes to physical data nodes DN; collects all involved physical nodes to generate an initial node set; and merges multiple virtual nodes mapped to the same physical node.

[0069] In step S25, the working node WN prunes the initial node set, excluding data nodes DN that do not contain relevant data, and generates a pushdown data node table. Specifically, this includes: determining the bitmap of data nodes DN that actually need to be accessed based on the calculation results of the shard key constant; comparing the initial node set with the data node DN bitmap to identify and exclude irrelevant data nodes DN; recording the data node identifier, virtual node range, and shard key value corresponding to each target data node DN in the pushdown data node table; allocating an RDMA communication buffer for each target data node DN, and updating the buffer address and size information to the RDMA buffer information in the pushdown data node table.

[0070] Optionally, the pushdown data node table includes the following fields: Data node: The physical identifier of the target node; Virtual node range: The range of virtual nodes that this node is responsible for; Sharding key value: The specific data key value that this node needs to process; RDMA buffer: Input buffer, used to receive the memory address of the execution plan; Output buffer, used to return the memory address of the query result; Buffer size: pre-allocated memory space; Status flag, the current status of the node, such as: ready, busy, faulty.

[0071] Specifically, a concrete implementation of pushing down the data node table includes: Example of pushdown data node representation

[0072] According to such Figure 3 The implementation method shown in this invention, through precise data positioning and routing optimization, achieves orders-of-magnitude performance improvement and resource utilization efficiency enhancement in distributed database systems, providing a solid technical foundation for large-scale distributed applications.

[0073] Figure 4 for Figure 3 A flowchart illustrating step S25. (See attached diagram.) Figure 4 As shown, step S25 includes steps S251-S254.

[0074] Step S251: Based on the calculation results of the fragmentation key constant, determine the actual data node DN bitmap that needs to be accessed.

[0075] Optionally, the initialization process of the data node (DN) bitmap is as follows: a bitmap with the same number of data nodes as the cluster is created, with all initial values ​​set to 0. In step S251, for each node in the initial node set, the corresponding bit is set in the bitmap.

[0076] For example: Cluster node data: [DN1, DN2, DN3, DN4, DN5, DN6] Initial node set: {DN3, DN4} The generated bitmap is: [0, 0, 1, 1, 0, 0] / / 1 indicates access is required, 0 indicates exclusion. Step S252: Compare the initial node set with the data node DN bitmap to identify and exclude irrelevant data nodes DN.

[0077] Optionally, in step S252, the working node WN compares the initial set of closed cases with the data node bitmap, excludes the nodes marked as 0 in the bitmap, and confirms that the remaining nodes do indeed contain the data required for the query.

[0078] Step S253: Record the data node identifier, virtual node range, and sharding key value corresponding to each target data node DN in the push-down data node table.

[0079] Optionally, in step S253, the working node WN creates the data structure of the pushdown data node table to initialize the table structure, then records detailed information for each target data node DN, and establishes the association between the shard key value and the target data node DN.

[0080] Step S254: Allocate an RDMA communication buffer for each target data node (DN) and update the buffer address and size information in the RDMA buffer information of the pushed-down data node table. Specifically, this includes: Memory pool allocation: Allocate the memory region required for RDMA communication from the global memory pool; Buffer registration: Registers the requested memory region as an RDMA-accessible region; Information logging: Records buffer information into the pushed-down data node table; Resource reservation: Reserve sufficient buffer space to support subsequent data transmission.

[0081] According to such Figure 5 The embodiments shown in this invention have the following technical innovations and advantages: 1. Performance optimization: Precise routing: Intelligent pruning ensures that queries are only sent to nodes that actually contain the relevant data; Resource conservation: Avoid sending queries to irrelevant nodes, saving network bandwidth and computing resources; Parallel processing: lays the foundation for subsequent parallel query execution.

[0082] 2. Scalability: Dynamic Adaptation: Consistent hashing rings support dynamic node scaling. Load balancing: Achieving uniform data distribution through virtual nodes; Fault tolerance: The failure of a single node does not affect the query processing of other nodes.

[0083] 3. Resource Management: Memory optimization: Avoid runtime memory allocation overhead by pre-allocating RDMA buffers; Connection reuse: The global memory pool supports multiple queries sharing communication resources; Traffic control: Provides foundational information for subsequent credit-based traffic control.

[0084] Figure 5 for Figure 2 A flowchart illustrating step S4 in Example 2000. (See attached diagram.) Figure 5 As shown, step S4 includes steps S41-S45.

[0085] In step S41, the working node WN serializes the pushdown plan into a byte stream and sends the serialized pushdown plan to the input buffer of each target data node DN through an RDMA-send operation.

[0086] The pushdown plan serialization performed in step S41 is the process of converting the structured execution plan into a compact binary format. Optionally, the execution plan contains complex data structures such as operation sequences, parameter configurations, and execution contexts, which are converted into a continuous byte stream through serialization for easy network transmission and storage. Optionally, the serialization process employs an efficient binary encoding scheme, using type marking, length prefixes, and value encoding techniques to flatten the tree-like operation plan structure into a linear byte sequence. This encoding method significantly reduces the amount of data transmitted while maintaining sufficient information integrity to support accurate deserialization.

[0087] In some specific embodiments, the principle of the RDMA-send transmission operation in step S41 is as follows: the working node WN sends the serialized byte stream to the pre-registered input buffer of the target data node DN through the RDMA-send operation. RDMA-send is a bilateral operation, requiring cooperation between the sender and receiver. WN's RDMA network card directly accesses the serialized data in its local user-space memory, encapsulates it through the RDMA protocol, and sends it to the target DN's RDMA network card via the network. After receiving the data, the target DN's network card directly writes it into the pre-registered input buffer memory area. The entire process completely bypasses the operating system kernel, achieving zero-copy data transmission.

[0088] In step S41, the input buffer pool is maintained by each Data Node (DN) and registered with the RDMA network card during system initialization. Optionally, the Network Node (WN) obtains the input buffer address and access key of each DN by querying the pushed-down data node table. Credit-based flow control is used during transmission. The WN maintains a credit counter for each DN to ensure that incomplete transmission requests do not exceed the preset window size, preventing memory resource overload at the DN end. Optionally, the preset window is 32 entries by default.

[0089] In step S42, each target data node (DN) reads and deserializes the pushdown plan from the input buffer and executes the pushdown operator locally.

[0090] In some specific embodiments, in step S42, after the data node DN detects that new data has arrived in the input buffer through polling or event notification mechanisms, it begins the deserialization process. The deserializer parses the byte stream according to a predefined format specification and gradually reconstructs the complete data structure of the execution plan. This process includes steps such as parsing the operation sequence, restoring parameter configuration, and reconstructing the execution context, ultimately restoring a structured representation in the DN memory that is semantically equivalent to the original execution plan at the WN end.

[0091] Optionally, the principle of the pushdown operator in step S42 is as follows: According to the reconstruction execution plan, DN sequentially executes each pushdown operator on the local data shard. The execution process fully leverages the advantages of data locality, performing calculations directly at the data storage location: Filtering operator: Apply WHERE conditions to filter local sharded data, retaining only records that meet the conditions; Projection operator: Extracts specific fields needed for the query from the complete record, reducing the amount of data to be processed later; Aggregation operators: perform local aggregation calculations on records with the same grouping key to generate intermediate aggregation results; Sorting operator: performs partial sorting of intermediate results based on the ORDER BY condition; Limitation operator: Apply the LIMIT condition to extract a local Top-N record.

[0092] In step S43, for small-scale intermediate results, each target data node (DN) directly sends the results to the worker node (WN) via RDMA-send operation. In step S44, for large-scale result sets, the worker node (WN) directly reads the result data from the output buffers of each target data node (DN) via RDMA-read operation.

[0093] Optionally, in steps S43 and S44, a transmission strategy is intelligently selected based on the size of the result set. Specifically, in step S43, small-scale intermediate sets are directly returned to the worker node WN using an RDMA-send operation. This strategy is suitable for scenarios where the amount of data is greatly reduced after filtering, results are aggregated, and queries are restricted. Since the result set size is small, direct sending results in lower overall overhead. In step S44, large-scale result sets are processed using an RDMA-read operation, where the worker node WN actively pulls data from the output buffer of the data node DN. This mode is suitable for query scenarios that generate a large number of intermediate results, such as full table scans and complex joins. The pull mode delegates the initiative for data transmission to WN, allowing it to control the data inflow rate based on its processing capacity and network conditions.

[0094] Optionally, RDMA-send return mechanism: The DN encapsulates small-scale result data directly in an RDMA-send message and sends it to the WN's receive buffer via bilateral operations. The WN needs to pre-publish sufficient receive descriptors to receive these return messages. During transmission, the DN's RDMA network card directly accesses the memory area where the result data resides, encapsulates it into an RDMA packet, and sends it to the network. After receiving the data, the WN's RDMA network card directly writes the data to the correct receive buffer location based on the destination address information in the packet.

[0095] Alternatively, the RDMA-read one-sided operation principle is as follows: RDMA-read is a one-sided operation, requiring only the participation of the initiating party (WN) and completely eliminating the need for CPU intervention from the target party (DN). WN directly issues a read command to its local RDMA network card, specifying the source address (DN's output buffer), the destination address (WN's local buffer), and the data size. WN's RDMA network card sends a read request to DN's RDMA network card over the network. After verifying authorization, DN's network card directly accesses its local output buffer memory, reads the requested data, and returns it to WN's network card. Finally, WN's network card writes the data directly to the specified local memory.

[0096] In step S45, the worker node WN merges the intermediate results returned by each target data node DN to generate the final execution result.

[0097] In some specific embodiments, the working node WN performs feature identification and analysis on the intermediate results of each DN in step S45, selects an appropriate merging strategy, and performs multi-way merge sorting, global aggregation calculation and deduplication. Finally, the WN organizes the merged data into the format expected by the client.

[0098] Optionally, step S4 also includes a lock-free globally ordered write mechanism and an optimized two-phase commit processing mechanism to further reduce latency and make full use of network bandwidth in the cross-shard transaction processing scenario of distributed database, while maintaining strong consistency semantic guarantees.

[0099] The lock-free globally ordered write mechanism includes: worker nodes WN obtain write ticket numbers from each target data node DN through RDMA atomic operations Fetch-and-Add. The write ticket numbers are used to ensure the global order when multiple worker nodes WN concurrently write to the same data fragment; each target data node DN executes data operations in the order of the write ticket numbers to achieve lock-free ordered disk write.

[0100] For example, in some specific embodiments, the write ticket allocation of the lock-free globally ordered write mechanism is implemented as follows: each data node maintains an independent 64-bit write ticket counter, and the worker node obtains the write ticket sequence number through an RDMA Fetch-and-Add atomic operation. This operation is completed at the RDMA network card level and does not involve the data node CPU, thus achieving a truly lock-free sequential allocation.

[0101] The optimized two-phase commit processing mechanism includes: using the two-phase commit protocol 2PC for cross-shard transactions, where both prepare and commit messages are broadcast in parallel to all relevant target data nodes (DNs) via RDMA-send; and the transmission of prepare and commit messages is completed in a single network round trip to communicate with all target data nodes (DNs).

[0102] In the two-phase commit processing mechanism, parallel broadcasting utilizes the worker node WN as the transaction coordinator, using RDMA-send to simultaneously send prepare or commit messages to all participating data nodes DN. This parallel communication mode breaks through the serial bottleneck of traditional 2PC.

[0103] Optionally, in the optimized two-phase commit processing mechanism, consistency is guaranteed as follows: in the preparation phase, all participants complete local preparation and write to the redo log; in the commit phase, all participants make the final commit based on the results of the preparation phase.

[0104] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of the present invention, its specific implementation methods, and its application scope, are all within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A distributed database system based on RDMA and storage-compute separation, characterized in that, include: At least one working node WN, and multiple data nodes DN connected via network communication; The worker node WN is used to receive query requests, perform SQL parsing, optimization and execution plan generation, and maintain a distributed partition table containing sharding strategies and sharding mapping relationships. The data node DN is used to store data fragments and respond to data operation commands from the worker node WN; The working node WN and the data node DN, as well as the multiple data nodes DN, transmit data through a hybrid communication channel based on RDMA. The system manages data distribution through a sharding mapping mechanism based on a consistent hash ring and supports elastic scaling of the data nodes (DNs).

2. The system according to claim 1, characterized in that, The RDMA-based hybrid communication channel is configured as follows: For data packets less than or equal to a preset threshold, RDMA bilateral operation is used for transmission; For data packets or result sets exceeding the preset threshold, RDMA single-sided read operation is used for transmission; The worker node WN and the data node DN register a global memory pool during system initialization and reuse the registered memory regions for RDMA communication during operation.

3. The system according to claim 1, characterized in that, The elastic expansion process of the data node (DN) includes: Create a new data node DN; The existing half of the adjacent virtual nodes on the consistent hash ring are reassigned to the newly added data node DN; Update the mapping relationship between virtual nodes and data nodes (DNs) in the distributed partitioning table. In the background, asynchronously, the fragmented data to be migrated is written directly from the old data node DN to the buffer of the newly added data node DN through RDMA one-sided read operation.

4. The system according to claim 3, characterized in that, The elastic expansion process of the data node (DN) also includes: The old data node DN was locally renamed to rename the fragment file; Update the shard location metadata in the distributed partition table to complete the route switch.

5. A distributed database data processing method based on RDMA and storage-compute separation, applied to the system described in any one of claims 1-4, characterized in that, Includes the following steps: S1. Receive SQL statements sent by clients through the worker node WN, and allocate connection resources from the global memory pool; S2. The worker node WN performs intelligent pruning based on the distributed partition table and the consistent hash ring to determine the pushdown data node table associated with the SQL statement; S3. Generate an execution plan containing pushdown operators based on the pushdown decision optimizer; S4. Data is transmitted between the working node WN and one or more target data nodes DN among the pushed-down data nodes through an RDMA-based hybrid communication channel to execute the push-down plan and generate execution results; S5. Return the execution result to the client.

6. The method according to claim 5, characterized in that, Step S2 includes: Extract one or more shard key constants from the SQL statement; The distributed partitioning table is queried based on the sharding key constant to determine the sharding strategy corresponding to the target data node DN; Calculate the target virtual node corresponding to the sharding key constant based on the sharding strategy and the consistent hash ring; Map the target virtual node to the corresponding physical data node DN to generate an initial node set; The initial node set is pruned to exclude data nodes (DNs) that do not contain relevant data, thereby generating the pushdown data node table.

7. The method according to claim 6, characterized in that, The pushdown data node table includes data node identifiers, virtual node ranges, sharding keys, and RDMA buffer information; the step of pruning the initial node set, excluding data node DNs that do not contain relevant data, and generating the pushdown data node table includes: Based on the calculation results of the fragmentation key constant, the actual data node DN bitmap that needs to be accessed is determined; By comparing the initial node set with the data node DN bitmap, irrelevant data node DNs are identified and excluded. The data node identifier, the virtual node range, and the sharding key value corresponding to each target data node DN are recorded in the pushdown data node table. An RDMA communication buffer is allocated for each target data node (DN), and the buffer address and size information are updated in the RDMA buffer information of the pushdown data node table.

8. The method according to claim 5, characterized in that, Step S4 includes: S41. Serialize the pushdown plan into a byte stream, and send the serialized pushdown plan to the input buffer of each target data node (DN) through an RDMA-send operation; S42. Each of the target data nodes (DNs) reads and deserializes the pushdown plan from the input buffer and executes the pushdown operator locally; S43. For small-scale intermediate results, each of the target data nodes DNs sends the results directly to the working node WN via an RDMA-send operation; S44. For large-scale result sets, the working node WN directly reads the result data from the output buffer of each target data node DN via RDMA-read operation; S45. The working node WN merges the intermediate results returned by each of the target data nodes DN to generate the final execution result.

9. The method according to claim 8, characterized in that, If the transaction corresponding to the SQL statement is a cross-shard transaction, step S4 also includes a lock-free globally ordered write mechanism: The worker node WN obtains the write ticket sequence number from each of the target data nodes DN through the RDMA atomic operation Fetch-and-Add. The write ticket sequence number is used to ensure the global order when multiple worker nodes WN concurrently write to the same data fragment. Each of the target data nodes (DNs) performs data operations in the order of the write ticket number, achieving lock-free ordered disk write.

10. The method according to claim 9, characterized in that, Step S4 also includes an optimized two-phase commit processing mechanism: The cross-shard transaction adopts a two-phase commit protocol 2PC, wherein both the prepare message and the commit message are broadcast in parallel to all relevant target data nodes (DNs) via RDMA-send; The sending of both the prepare and commit messages is accomplished through a single network round trip, completing communication between all the target data nodes (DNs).

Images