A distributed database system based on RDMA shared memory pool

Abstract

This invention discloses a distributed database system based on RDMA shared memory pools, relating to the field of distributed data management technology. The invention sets up two database cluster nodes as shared memory nodes, with the shared memory nodes serving as primary and backup nodes. A shared memory pool is created using the physical memory of the shared memory nodes, and metadata is stored in categories. Leveraging the low latency of RDMA, the number of copies and kernel intervention during data transmission is reduced, lowering CPU and memory utilization and significantly shortening data read / write time and query response time. This scheme significantly improves the performance of distributed databases under high-concurrency access scenarios.

Description

Technical Field

[0001] This invention discloses a distributed database system based on RDMA shared memory pool, which relates to the field of distributed data management technology. Background Technology

[0002] Shared-nothing architecture is a common distributed database architecture. Each node has its own private CPU, memory, hard drive, etc., with no shared resources. Processing units communicate via protocols, resulting in better parallel processing and scalability. Each node operates independently, processing its own data, with results potentially aggregated to higher layers or circulated between nodes. Shared-nothing distributed database systems continuously broadcast the latest metadata to other nodes in the cluster via a data synchronization protocol. However, this metadata synchronization process can present several challenges. As the cluster grows, data synchronization between the two furthest nodes may require passing through multiple intermediate nodes, causing synchronization delays. Poor network connectivity at intermediate nodes can even lead to data loss, impacting the high availability and high concurrency characteristics of the distributed database system. Summary of the Invention

[0003] This invention addresses the problems of existing technologies by providing a distributed database system based on RDMA shared memory pool, which greatly improves the database's operating efficiency and response speed, reduces system resource consumption, and enhances the performance of large-scale data processing and high-concurrency access.

[0004] The specific solution proposed in this invention is as follows:

[0005] This invention provides a distributed database system based on RDMA shared memory pool, including a node management module, an RDMA memory management module, a data transmission and synchronization module, and a query processing module.

[0006] The node management module selects two nodes within the distributed database cluster as shared memory nodes, designating one shared memory node as the master node and the other as the backup node.

[0007] The RDMA memory management module requests and locks physical memory for shared memory nodes based on the estimated database cluster node size and user data capacity, creating a shared memory pool. Within this pool, it divides the shared memory into node shared memory and data table shared memory for storing different types of metadata.

[0008] This enables the master node and backup nodes to provide services to the outside world through a shared memory pool listening address;

[0009] Initialize the shared memory pool: Before the distributed database cluster starts, the data transmission and synchronization module of the master node starts listening, the RDMA memory management module of the master node generates the corresponding shared memory pool memory and permission information, the data transmission and synchronization module of the backup node initiates an RDMA connection to the master node to obtain the shared memory pool memory and permission information of the master node, and at the same time, the RDMA memory management module of the backup node generates a shared memory pool memory and permission information of the same specifications.

[0010] The data transmission and synchronization modules of other database cluster nodes actively initiate RDMA connections to the shared memory pool listening address and initiate information synchronization operations to obtain the memory and permission information of the shared memory pool. During the operation of the database cluster nodes, the RDMA memory management module of the database cluster nodes writes newly generated metadata or changed metadata into the shared memory pool through RDMA write operations.

[0011] In a distributed database cluster, a randomly accessed node receives query requests through a query processing module. The query processing module constructs a distributed logical plan and a physical plan. Based on the logical plan and the physical plan, the randomly accessed node uses the data transmission and synchronization module to establish an RDMA connection with the master node and then reads the corresponding information from the shared memory pool.

[0012] Furthermore, in the aforementioned distributed database system based on RDMA shared memory pool, permission information is required when the shared memory pool performs RDMA read and write operations. The data transmission and synchronization module of the master node synchronizes the permission information to other distributed database cluster nodes for memory read and write use. At the same time, the master node and the backup node synchronize permission information with each other through the data transmission and synchronization module. After the backup node switches to become the new master node, it synchronizes the permission information to other distributed database cluster nodes.

[0013] Furthermore, in the aforementioned distributed database system based on an RDMA shared memory pool, the master node's RDMA memory management module divides the shared memory pool into multiple fixed-size memory blocks and records the status and location information of each memory block.

[0014] When a database cluster node initiates a write memory operation, the RDMA memory management module obtains the write address according to the address calculation rules and writes the metadata to the shared memory pool. When a database cluster node initiates a read memory operation, the RDMA memory management module reads the data from the corresponding memory block according to the memory virtual address and offset.

[0015] Furthermore, in the distributed database system based on RDMA shared memory pool, when the RDMA memory management module writes metadata into the shared memory pool, the types of metadata include node configuration information, node health status, node storage usage, and table information, and are assigned numbers 0-4 in sequence, or new numbers can be assigned to new types of metadata as needed.

[0016] Furthermore, in the distributed database system based on RDMA shared memory pool, the RDMA memory management module sequentially generates the ID of each node in the database cluster starting from sequence 1. The RDMA memory management module calculates the address block for writing the metadata of the database cluster node into the shared memory pool by adding the node's shared memory starting address + (node ​​ID-1) * metadata quantity + metadata number, and calculates the write address of the data table by adding the data table's shared memory starting address + table ID.

[0017] This invention also provides a distributed database management method based on RDMA shared memory pool. The method utilizes a distributed database system based on RDMA shared memory pool for distributed database management. The distributed database system includes a node management module, an RDMA memory management module, a data transmission and synchronization module, and a query processing module.

[0018] The node management module selects two nodes within the distributed database cluster as shared memory nodes, designating one as the master node and the other as the backup node.

[0019] The RDMA memory management module requests and locks physical memory for shared memory nodes based on the estimated database cluster node size and user data capacity, creating a shared memory pool. Within this pool, it divides the shared memory into node shared memory and data table shared memory for storing different types of metadata.

[0020] This enables the master node and backup nodes to provide services to the outside world through a shared memory pool listening address;

[0021] Initialize the shared memory pool: Before the distributed database cluster starts, the master node’s data transmission and synchronization module starts listening, the master node’s RDMA memory management module generates the corresponding shared memory pool memory and permission information, the backup node’s data transmission and synchronization module initiates an RDMA connection to the master node to obtain the master node’s shared memory pool memory and permission information, and at the same time, the backup node’s RDMA memory management module generates a shared memory pool of the same specifications with the same memory and permission information.

[0022] The data transmission and synchronization module of other database cluster nodes actively initiates an RDMA connection to the shared memory pool listening address and initiates information synchronization operation to obtain the memory and permission information of the shared memory pool. During the operation of the database cluster node, the RDMA memory management module of the database cluster node writes newly generated metadata or changed metadata into the shared memory pool through RDMA write operation.

[0023] In a distributed database cluster, a randomly accessed node receives query requests through a query processing module. The query processing module constructs a distributed logical plan and a physical plan. Based on the logical plan and the physical plan, the randomly accessed node uses the data transmission and synchronization module to establish an RDMA connection with the master node and then reads the corresponding information from the shared memory pool.

[0024] Furthermore, the aforementioned distributed database management method based on RDMA shared memory pool: when performing RDMA read and write operations in the shared memory pool, permission information is required. The permission information is synchronized to other distributed database cluster nodes for memory read and write use through the data transmission and synchronization module of the master node. At the same time, the master node and the backup node synchronize permission information with each other through the data transmission and synchronization module. After the backup node switches to become the new master node, it synchronizes the permission information to other distributed database cluster nodes.

[0025] Furthermore, the aforementioned distributed database management method based on RDMA shared memory pool: The shared memory pool is divided into multiple fixed-size memory blocks through the RDMA memory management module of the master node, and the status and location information of each memory block are recorded.

[0026] When a database cluster node initiates a write memory operation, the RDMA memory management module obtains the write address according to the address calculation rules and writes the metadata to the shared memory pool. When a database cluster node initiates a read memory operation, the RDMA memory management module reads the data from the corresponding memory block according to the memory virtual address and offset.

[0027] Furthermore, the distributed database management method based on RDMA shared memory pool: when writing metadata to the shared memory pool through the RDMA memory management module, the types of metadata include node configuration information, node health status, node storage usage, and table information, and are assigned numbers 0-4 in sequence, or new numbers can be assigned to new types of metadata as needed.

[0028] Furthermore, the distributed database management method based on RDMA shared memory pool is as follows: The RDMA memory management module sequentially generates the ID of each node in the database cluster starting from sequence 1. The RDMA memory management module calculates the address block for writing the metadata of the database cluster node into the shared memory pool by adding (node ​​ID-1)*data metadata quantity and metadata number to the node's shared memory starting address. The write address of the data table is calculated by adding the table ID to the data table's shared memory starting address.

[0029] The advantages of this invention are:

[0030] Two database cluster nodes are configured as shared memory nodes, serving as primary and backup nodes. A shared memory pool is created using the physical memory of the shared memory nodes, and metadata is stored in categories. Leveraging the low latency of RDMA, the number of copies and kernel intervention during data transfer is reduced, lowering CPU and memory utilization and significantly shortening data read / write and query response times. This solution significantly improves the performance of distributed databases under high-concurrency access scenarios. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a distributed database cluster architecture based on RDMA shared memory pool.

[0032] Figure 2 This is a flowchart illustrating the application of the shared memory pool in the method of this invention.

[0033] Figure 3 This is a schematic diagram illustrating the process of executing a query request using the method of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0035] Example 1

[0036] This invention provides a distributed database system based on RDMA shared memory pool, including a node management module, an RDMA memory management module, a data transmission and synchronization module, and a query processing module.

[0037] The node management module selects two nodes within the distributed database cluster as shared memory nodes, designating one of the shared memory nodes as the master node and the other as the backup node. When the current master node fails, it automatically switches to the backup node as the new master node.

[0038] The RDMA memory management module allocates and locks physical memory for shared memory nodes based on the estimated database cluster node size and user data capacity, creating a shared memory pool. Within this pool, it divides the shared memory into node shared memory and data table shared memory for storing different types of metadata. Simultaneously, when allocating physical memory, it generates a mapping between virtual and physical addresses and provides the virtual addresses to the application.

[0039] This enables the master node and backup nodes to provide services to the outside world through the listening address of the shared memory pool.

[0040] The master node's RDMA memory management module divides the shared memory pool into multiple fixed-size memory blocks and records the status and location information of each memory block.

[0041] When a database cluster node initiates a write operation to memory, the RDMA memory management module obtains the write address according to the address calculation rules and writes the metadata to the shared memory pool. When a database cluster node initiates a read operation to memory, the RDMA memory management module reads the data from the corresponding memory block in response based on the memory virtual address and offset. In the shared memory pool, the primary and backup nodes request memory blocks of the same size and allocate them consistently. The primary and backup nodes store the mapping relationship of memory block addresses. When the data in a memory block on the primary node changes, it is synchronized to the backup node.

[0042] Initialize the shared memory pool: Before the distributed database cluster starts, the data transmission and synchronization module of the master node starts listening, the RDMA memory management module of the master node generates the corresponding shared memory pool memory and permission information, the data transmission and synchronization module of the backup node initiates an RDMA connection to the master node to obtain the shared memory pool memory and permission information of the master node, and at the same time, the RDMA memory management module of the backup node generates a shared memory pool memory and permission information of the same specifications.

[0043] The data transmission and synchronization modules of other database cluster nodes actively initiate RDMA connections to the shared memory pool's listening address and perform information synchronization operations to obtain the shared memory pool's memory and permission information, such as memory address information, memory size and block information, and permission information. During the operation of the database cluster nodes, the RDMA memory management modules of the database cluster nodes write newly generated or changed metadata into the shared memory pool through RDMA write operations. Permission information is required when the shared memory pool performs RDMA read and write operations. The data transmission and synchronization module of the master node synchronizes the permission information to other distributed database cluster nodes for memory read and write operations. Simultaneously, the master node and backup nodes synchronize permission information with each other through the data transmission and synchronization module. After the backup node switches to become the new master node, it synchronizes the permission information to other distributed database cluster nodes.

[0044] When the RDMA memory management module writes metadata to the shared memory pool, the metadata types include node configuration information, node health status, node storage usage, and table information, and are assigned numbers 0-4 sequentially, or new numbers can be assigned to new types of metadata as needed. The RDMA memory management module generates the ID of each node in the database cluster sequentially, starting from sequence 1. It calculates the address block for writing the metadata of a database cluster node to the shared memory pool by adding the node's shared memory address to (node ​​ID - 1) * the number of metadata entries and the metadata number. It also calculates the write address of a data table by adding the table's shared memory address to the table's ID.

[0045] In a distributed database cluster, a randomly accessed node receives query requests through a query processing module. The query processing module constructs a distributed logical plan and a physical plan. Based on the logical plan and the physical plan, the randomly accessed node uses the data transmission and synchronization module to establish an RDMA connection with the master node and then reads the corresponding information from the shared memory pool.

[0046] Example 2

[0047] This invention also provides a distributed database management method based on RDMA shared memory pool. The method utilizes a distributed database system based on RDMA shared memory pool for distributed database management. The distributed database system includes a node management module, an RDMA memory management module, a data transmission and synchronization module, and a query processing module.

[0048] The node management module selects two nodes within the distributed database cluster as shared memory nodes, designating one as the master node and the other as the backup node.

[0049] The RDMA memory management module requests and locks physical memory for shared memory nodes based on the estimated database cluster node size and user data capacity, creating a shared memory pool. Within this pool, it divides the shared memory into node shared memory and data table shared memory for storing different types of metadata.

[0050] This enables the master node and backup nodes to provide services to the outside world through a shared memory pool listening address;

[0051] Initialize the shared memory pool: Before the distributed database cluster starts, the master node’s data transmission and synchronization module starts listening, the master node’s RDMA memory management module generates the corresponding shared memory pool memory and permission information, the backup node’s data transmission and synchronization module initiates an RDMA connection to the master node to obtain the master node’s shared memory pool memory and permission information, and at the same time, the backup node’s RDMA memory management module generates a shared memory pool of the same specifications with the same memory and permission information.

[0052] The data transmission and synchronization module of other database cluster nodes actively initiates an RDMA connection to the shared memory pool listening address and initiates information synchronization operation to obtain the memory and permission information of the shared memory pool. During the operation of the database cluster node, the RDMA memory management module of the database cluster node writes newly generated metadata or changed metadata into the shared memory pool through RDMA write operation.

[0053] In a distributed database cluster, a randomly accessed node receives query requests through a query processing module. The query processing module constructs a distributed logical plan and a physical plan. Based on the logical plan and the physical plan, the randomly accessed node uses the data transmission and synchronization module to establish an RDMA connection with the master node and then reads the corresponding information from the shared memory pool.

[0054] The information interaction and execution process involved in the above method flow are based on the same concept as the system embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0055] Similarly, the method of this invention sets two database cluster nodes as shared memory nodes, with the shared memory nodes serving as primary and backup nodes. A shared memory pool is created using the physical memory of the shared memory nodes, and metadata is stored in categories. Leveraging the low latency of RDMA, the number of copies and kernel intervention during data transmission is reduced, lowering CPU and memory utilization and significantly shortening data read / write and query response times. This scheme significantly improves the performance of distributed databases in high-concurrency access scenarios.

[0056] It should be noted that not all steps and modules in the above system structures and processes are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above embodiments can be physical or logical structures; that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0057] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A distributed database system based on RDMA shared memory pool, characterized in that: It includes a node management module, an RDMA memory management module, a data transmission and synchronization module, and a query processing module. The node management module selects two nodes within the distributed database cluster as shared memory nodes, designating one shared memory node as the master node and the other as the backup node. The RDMA memory management module requests and locks physical memory for shared memory nodes based on the estimated database cluster node size and user data capacity, creating a shared memory pool. Within this pool, it divides the shared memory into node shared memory and data table shared memory for storing different types of metadata. This enables the master node and backup nodes to provide services to the outside world through a shared memory pool listening address; Initialize the shared memory pool: Before the distributed database cluster starts, the data transmission and synchronization module of the master node starts listening, the RDMA memory management module of the master node generates the corresponding shared memory pool memory and permission information, the data transmission and synchronization module of the backup node initiates an RDMA connection to the master node to obtain the shared memory pool memory and permission information of the master node, and at the same time, the RDMA memory management module of the backup node generates a shared memory pool memory and permission information of the same specifications. The data transmission and synchronization modules of other database cluster nodes actively initiate RDMA connections to the shared memory pool listening address and initiate information synchronization operations to obtain the memory and permission information of the shared memory pool. During the operation of the database cluster nodes, the RDMA memory management module of the database cluster nodes writes newly generated metadata or changed metadata into the shared memory pool through RDMA write operations. In a distributed database cluster, a randomly accessed node receives query requests through a query processing module. The query processing module constructs a distributed logical plan and a physical plan. Based on the logical plan and the physical plan, the randomly accessed node uses the data transmission and synchronization module to establish an RDMA connection with the master node and then reads the corresponding information from the shared memory pool.

2. A distributed database system based on RDMA shared memory pool according to claim 1, characterized in that... When the shared memory pool performs RDMA read and write operations, it requires permission information. The data transmission and synchronization module of the master node synchronizes the permission information to other distributed database cluster nodes for memory read and write use. At the same time, the master node and the backup node synchronize permission information with each other through the data transmission and synchronization module. After the backup node switches to become the new master node, it synchronizes the permission information to other distributed database cluster nodes.

3. A distributed database system based on RDMA shared memory pool according to claim 1, characterized in that: The master node's RDMA memory management module divides the shared memory pool into multiple fixed-size memory blocks and records the status and location information of each memory block. When a database cluster node initiates a write memory operation, the RDMA memory management module obtains the write address according to the address calculation rules and writes the metadata to the shared memory pool. When a database cluster node initiates a read memory operation, the RDMA memory management module reads the data from the corresponding memory block according to the memory virtual address and offset.

4. A distributed database system based on RDMA shared memory pool according to claim 1, characterized in that: When the RDMA memory management module writes metadata to the shared memory pool, the types of metadata include node configuration information, node health status, node storage usage, and table information, and are assigned numbers 0-4 in sequence, or new numbers can be assigned to new types of metadata as needed.

5. A distributed database system based on RDMA shared memory pool according to claim 1, characterized in that: The RDMA memory management module generates the ID of each node in the database cluster sequentially starting from sequence 1. The RDMA memory management module calculates the address block for writing the metadata of the database cluster node to the shared memory pool by adding (node ​​ID-1) * metadata quantity + metadata number. The module also calculates the write address of the data table by adding the shared memory address of the data table to the table ID.

6. A distributed database management method based on RDMA shared memory pool, characterized in that: Distributed database management is achieved through a distributed database system based on RDMA shared memory pools. This distributed database system includes a node management module, an RDMA memory management module, a data transmission and synchronization module, and a query processing module. The node management module selects two nodes within the distributed database cluster as shared memory nodes, designating one as the master node and the other as the backup node. The RDMA memory management module requests and locks physical memory for shared memory nodes based on the estimated database cluster node size and user data capacity, creating a shared memory pool. Within this pool, it divides the shared memory into node shared memory and data table shared memory for storing different types of metadata. This enables the master node and backup nodes to provide services to the outside world through a shared memory pool listening address; Initialize the shared memory pool: Before the distributed database cluster starts, the master node’s data transmission and synchronization module starts listening, the master node’s RDMA memory management module generates the corresponding shared memory pool memory and permission information, the backup node’s data transmission and synchronization module initiates an RDMA connection to the master node to obtain the master node’s shared memory pool memory and permission information, and at the same time, the backup node’s RDMA memory management module generates a shared memory pool of the same specifications with the same memory and permission information. The data transmission and synchronization module of other database cluster nodes actively initiates an RDMA connection to the shared memory pool listening address and initiates information synchronization operation to obtain the memory and permission information of the shared memory pool. During the operation of the database cluster node, the RDMA memory management module of the database cluster node writes newly generated metadata or changed metadata into the shared memory pool through RDMA write operation. In a distributed database cluster, a randomly accessed node receives query requests through a query processing module. The query processing module constructs a distributed logical plan and a physical plan. Based on the logical plan and the physical plan, the randomly accessed node uses the data transmission and synchronization module to establish an RDMA connection with the master node and then reads the corresponding information from the shared memory pool.

7. A distributed database management method based on RDMA shared memory pool according to claim 6, characterized in that... When the shared memory pool performs RDMA read and write operations, it requires permission information. The permission information is synchronized to other distributed database cluster nodes through the data transmission and synchronization module of the master node for memory read and write use. At the same time, the master node and the backup node synchronize permission information with each other through the data transmission and synchronization module. After the backup node switches to become the new master node, it synchronizes the permission information to other distributed database cluster nodes.

8. A distributed database management method based on RDMA shared memory pool according to claim 6, characterized in that: The master node's RDMA memory management module divides the shared memory pool into multiple fixed-size memory blocks and records the status and location information of each memory block. When a database cluster node initiates a write memory operation, the RDMA memory management module obtains the write address according to the address calculation rules and writes the metadata to the shared memory pool. When a database cluster node initiates a read memory operation, the RDMA memory management module reads the data from the corresponding memory block according to the memory virtual address and offset.

9. A distributed database management method based on RDMA shared memory pool according to claim 6, characterized in that: When writing metadata to the shared memory pool through the RDMA memory management module, the types of metadata include node configuration information, node health status, node storage usage, and table information, and are assigned numbers 0-4 in sequence, or new numbers can be assigned to new types of metadata as needed.

10. A distributed database management method based on RDMA shared memory pool according to claim 6, characterized in that: The RDMA memory management module generates the ID of each node in the database cluster sequentially starting from sequence 1. The RDMA memory management module calculates the address block for writing the metadata of the database cluster node to the shared memory pool by adding (node ​​ID-1) * metadata quantity + metadata number. The module also calculates the write address of the data table by adding the shared memory address of the data table to the table ID.

Images