RDMA distributed data transmission method supporting high concurrent message transmission

Abstract

The present invention belongs to the field of distributed data access technology, and discloses an RDMA distributed data transmission method that supports high-concurrency message transmission, aiming to improve system performance in a high-concurrency message transmission environment. The method includes the following core steps: 1) peer-to-peer server design: each device is an independent server, which always monitors the connection channel to establish a connection; 2) pre-establish a connection resource pool; 3) RPC message creation and registration; 4) RDMA transmission to the target server memory mailbox pool; 5) high-concurrency processing on the client and server; 6) dynamic polling on the server; 7) zero-copy transmission of data through RDMA technology. The present invention uses the method of decoupling connection resources to improve the dynamic expansion capability of the architecture, and registering a continuous large page of memory at one time can reduce QP jitter and enhance the stability of the system. The use of a merged sending strategy effectively improves the message transmission efficiency under the distributed network architecture.

Description

Technical Field

[0001] The present invention relates to the field of distributed data access technology, and more particularly to an RDMA distributed data transmission method supporting high-concurrency message transmission. Background Art

[0002] In modern computer systems, data transfer technology plays a crucial role, not only affecting the efficiency of data exchange within a single computer but also directly impacting the performance of distributed systems across networks. From early parallel bus architectures to today's widely adopted high-speed serial interfaces, such as PCI Express (PCIe), the evolution of system buses has been a key driver of computing performance. These bus technologies integrate data, address, and control signals to enable efficient connections between the CPU, memory, and I / O devices. In modern computer design, buses not only connect I / O devices but also, through direct memory access (DMA) technology, offload the CPU's workload, freeing it to focus on complex computing tasks and thus improving overall system performance. DMA technology allows I / O devices to exchange data directly with memory without CPU intervention, significantly accelerating data transfer speeds. Furthermore, the introduction of interrupt systems has enabled real-time interaction between peripheral devices and the CPU, which is crucial for building multitasking systems.

[0003] As data transmission demands continue to grow, traditional parallel communication methods are increasingly facing limitations in speed and signal integrity, prompting a new trend in high-speed serial communication. PCIe is a prominent example of this trend. Its point-to-point connection method not only supports higher data rates but also effectively reduces latency and data conflicts. PCIe's multi-channel parallel transmission capabilities significantly increase system bandwidth, meeting the demands of high-performance computing applications.

[0004] In terms of inter-computer communication, the development of network technology has provided standardized protocols for data exchange between multiple devices. This has not only promoted the widespread use of the internet but also laid the foundation for the implementation of distributed computing. In modern distributed network architectures, efficient and flexible data transmission is crucial. Traditional network transmission models involve a large amount of redundant data copying between kernel space and user space, resulting in significant performance degradation when transmitting large amounts of data. To address this issue, Remote Direct Memory Access (RDMA) technology has emerged. RDMA allows one computer to directly access the memory of another computer without intervention from the operating system's network stack or the CPU, significantly reducing data transmission latency and CPU load. This technology is particularly suitable for scenarios with strict requirements for ultra-low latency and high throughput, such as high-performance computing, distributed storage, and data centers. Due to its low latency, high bandwidth, and high throughput, Remote Direct Memory Access (RDMA) has been widely adopted in high-performance computing and data center networking. By bypassing the traditional network protocol stack, RDMA enables direct data transmission below the network transport layer. This stack bypass mechanism avoids the processing of multiple layers of network protocols, such as the Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Protocol (IP), thereby improving transmission efficiency. In addition, RDMA technology supports kernel bypass, allowing applications to directly perform data transmission without entering kernel mode. This approach reduces the cost of context switching between kernel mode and user mode and reduces the frequency of data copying. RDMA technology inherits and develops the advantages of direct memory access (DMA) technology, allowing external devices (such as devices connected via the PCIe bus) to bypass the CPU and directly access system memory. RDMA further expands this concept, allowing local computers to directly access the memory of remote computers (including dynamic random access memory DRAM and non-volatile memory NVM), thereby improving the overall performance of distributed system communications.

[0005] In the research and development of network architecture, the integration of RDMA with modern high-speed network technologies such as Ethernet, Fibre Channel, and Infiniband has further enhanced its advantages. As one of the early applications of RDMA technology, Infiniband has been widely used in supercomputing centers and large-scale data centers due to its high bandwidth and low latency. In future network architecture design, RDMA technology is expected to be combined with next-generation network protocols such as IPv6 and Time-Sensitive Networking (TSN) to form a more efficient and real-time communication network to meet the needs of latency-sensitive applications such as the Internet of Things, industrial control, and autonomous driving. By reducing CPU and operating system intervention, RDMA and its derivative technologies provide a lighter-weight and faster solution for future network transmission, helping to further promote the data transmission speed and efficiency of distributed systems. Summary of the Invention

[0006] The present invention aims to provide an RDMA distributed data transmission method that supports high-concurrency message transmission. This method can be applied in high-concurrency message transmission environments, particularly in distributed network architectures, leveraging Remote Direct Memory Access (RDMA) technology to significantly improve data transmission performance. This invention proposes a series of optimized designs for data transmission, resource management, load balancing, and dynamic system scalability, addressing issues such as resource conflicts, transmission delays, and resource consumption that plague traditional distributed systems in high-concurrency environments.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] According to a first aspect of the present invention, a method for RDMA distributed data transmission supporting high-concurrency message transmission is provided. The method is based on an RDMA distributed architecture supporting high-concurrency message transmission, wherein the RDMA distributed architecture supporting high-concurrency message transmission includes a client and a server. The client and the server are connected via hardware RDMA and a high-speed switch, with the client or the server serving as a node. The method includes:

[0009] After each node is started, it obtains the configuration information of the management node through the CFG configuration file and first establishes a connection with the management node to obtain the metadata of other nodes; wherein, the configuration information of the management node includes the IP address, port number and initial node number of the management node;

[0010] Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node and places the connection resources into the connection manager;

[0011] The database responds to the application's query statements and obtains the connection resources bound to the client;

[0012] The database calls various data transmission function modules in the client to obtain a transmission request; wherein the various data transmission function modules include distributed indexes for querying and retrieving query conditions, distributed locks for ensuring cache consistency and high concurrency, and remote transmission of undo segments for recovery;

[0013] The client adds header information and RPC flag information to the transmission request;

[0014] The client places the transmission request, including the header information and RPC flag information, into a slot in the memory block corresponding to the connection in the local memory pool. All requests in the same memory block are combined and encapsulated into a message. The client thread then places the request information in the slot into the corresponding Mailbox slot on the server through a unilateral write operation on the RDMA network.

[0015] The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot;

[0016] The server distributes the message to different execution programs according to the message type to process the message type;

[0017] The server-side message executor places the processed response message and data smaller than 512B in the server's shared send buffer. The handler combines the response message and data, adds the message type and status, and forms an RPC message.

[0018] In the server-side message execution program, if the returned data is larger than 512B, it is directly written to the specified memory address using a one-way write. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the one-way write is completed, an RPC completion message is sent using the RPC mailbox system.

[0019] The server-side message handler sends the RPC message to the client's corresponding memory pool slot using one-sided write;

[0020] The client thread passes the return message to the functional module in the client to process the response message.

[0021] According to a second aspect of the present invention, there is provided an RDMA distributed data transmission component supporting high-concurrency message transmission, wherein the system includes a client, a server, and a management node, wherein the client and the server are connected via hardware RDMA and a high-speed switch, with the client or the server serving as a node, and the management node connected to the node; wherein:

[0022] After each node is started, it obtains the configuration information of the management node through the CFG configuration file and first establishes a connection with the management node to obtain the metadata of other nodes; wherein, the configuration information of the management node includes the IP address, port number and initial node number of the management node;

[0023] Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node and places the connection resources into the connection manager;

[0024] The database responds to the application's query statements and obtains the connection resources bound to the client;

[0025] The database calls various data transmission function modules in the client to obtain a transmission request; wherein the various data transmission function modules include distributed indexes for querying and retrieving query conditions, distributed locks for ensuring cache consistency and high concurrency, and remote transmission of undo segments for recovery;

[0026] The client adds header information and RPC flag information to the transmission request;

[0027] The client places the transmission request, including the header information and RPC flag information, into a slot in the memory block corresponding to the connection in the local memory pool. All requests in the same memory block are combined and encapsulated into a message. The client thread then places the request information in the slot into the corresponding Mailbox slot on the server through a unilateral write operation on the RDMA network.

[0028] The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot;

[0029] The server distributes the message to different execution programs according to the message type to process the message type;

[0030] The server-side message executor places the processed response message and data smaller than 512B in the server's shared send buffer. The handler combines the response message and data, adds the message type and status, and forms an RPC message.

[0031] In the server-side message execution program, if the returned data is larger than 512B, it is directly written to the specified memory address using a one-way write. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the one-way write is completed, an RPC completion message is sent using the RPC mailbox system.

[0032] The server-side message handler sends the RPC message to the client's corresponding memory pool slot using one-sided write;

[0033] The client thread passes the return message to the functional module in the client to process the response message.

[0034] The present invention has at least the following beneficial effects:

[0035] The present invention enables efficient, low-latency data transmission and parallel processing. By leveraging Remote Direct Memory Access (RDMA) technology, it establishes a high-speed, zero-copy data transmission channel between the client and server, significantly improving the system's data transmission and processing performance in high-concurrency environments. After receiving a transmission request, the client parses the request and generates a corresponding remote procedure call (RPC) message. The RPC message encapsulates the request content, call instructions, and related transmission metadata, making the data transmission process structured and efficient. To further improve transmission speed, the client pre-registers a memory pool on the RDMA network interface card (RNIC). This pre-registration directly associates the memory pool with the network card, enabling an efficient data access path. Using RDMA technology, the client transmits the encapsulated RPC message directly to the server's designated memory pool without requiring the involvement of the operating system kernel. This zero-copy transmission mode significantly reduces CPU involvement and data transmission latency, making it particularly suitable for high-concurrency scenarios with frequent transmission of small messages. The pre-registered memory pool also facilitates message reuse and rapid recovery, optimizing memory management and enabling the client to respond to continuous RPC requests more quickly. The server is responsible for receiving RPC messages from the client, parsing them, and executing the corresponding processing tasks based on the request. During RDMA transmission, the client's RPC messages directly enter the server's pre-registered memory pool, which the server can access instantly, allowing it to more quickly retrieve and process the message data. After receiving the message, the server decapsulates it and distributes the task to the corresponding handler for execution. After processing the request task, the server places the result into a mailbox buffer and returns the processing result to the client via the established RDMA channel. This structured request-process-return model enables efficient concurrent processing. The pre-registration of multiple memory pools and mailbox buffers also reduces lock contention and resource waiting time between threads, further enhancing the server's parallel processing capabilities. To support highly concurrent dynamic environments, the present invention designs a flexible connection mechanism that allows the client and server to dynamically add or release connection resources. Connection status is managed through heartbeat communication with the cluster decision node (Meta). After receiving information about other surviving devices, each client proactively establishes a pre-connection with them, forming a connection resource pool, thereby reducing the latency caused by dynamic connections. This design of the connection resource pool provides dynamic scalability, enabling the system to efficiently handle the addition or removal of client and server nodes. Furthermore, the present invention integrates an intelligent detection mechanism, allowing the server to dynamically switch between polling and event-driven modes based on load. Under high load, polling mode is automatically entered to quickly detect and process newly arrived messages; under low load, it switches to event-driven mode, reducing CPU usage and resource waste.Through the intelligent detection mechanism, the system can balance resource usage under different load conditions, achieving the dual advantages of high performance and low power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of an RDMA distributed architecture supporting high-concurrency message transmission provided by an embodiment of the present invention.

[0037] Figure 2 A data processing flow chart of the distributed data transmission method provided by an embodiment of the present invention.

[0038] Figure 3 A schematic diagram of a mailbox-based RDMA transmission optimization mechanism provided by an embodiment of the present invention.

[0039] Figure 4 This is a diagram of resource allocation management for connections between a server and a client provided by an embodiment of the present invention.

[0040] Figure 5 A structural diagram of an RDMA distributed data transmission component supporting high-concurrency message transmission provided by an embodiment of the present invention.

[0041] Figure 6 A diagram of a connection manager for a distributed network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are further described in detail below with reference to the accompanying drawings and specific embodiments, but are not intended to limit the present invention. For the various steps described herein, if there is no necessity for a contextual relationship between each other, the order in which they are described as examples herein should not be regarded as limiting, and those skilled in the art should know that they can be adjusted in order as long as the logic between them is not destroyed, resulting in the inability to implement the entire process.

[0043] The embodiment of the present invention provides an RDMA distributed data transmission method that supports high concurrent message transmission, based on the RDMA distributed architecture that supports high concurrent message transmission. Figure 1, which is a structural diagram of an RDMA distributed architecture supporting high-concurrency message transmission provided by an embodiment of the present invention. The RDMA distributed architecture supporting high-concurrency message transmission includes a first server 101, a second server 102, and a third server 103, wherein the first server 101 and the second server 102 serve as the server and the client, and the client and the server are connected via hardware RDMA and a high-speed switch. A client or a server serves as a node, and the third server 103 serves as a management node. Each first server 101 and the second server 102 are peers, and each device can be given the ability to serve as both a client and a server, thereby achieving the parity of each node in the system.

[0044] In this embodiment, a peer-to-peer server architecture is employed within an RDMA distributed architecture that supports high-concurrency message transmission. Each device is empowered to function as both a client and a server, thereby achieving parity among all nodes in the system. This means that all devices possess independent server functionality, actively listening for connection requests on the network and quickly responding and establishing connections upon receiving new connection requests. This design helps ensure smooth communication and stable response speeds in a dynamic, highly concurrent network environment. In the peer-to-peer server design, each node can not only initiate connection requests as a data requester (client), but also respond to connection requests from other nodes (server) and process data. This peer-to-peer architecture enables flexible device role transitions, enabling any node to assume the responsibility of receiving, processing, or forwarding data based on real-time load conditions, significantly improving system flexibility. All devices continuously listen for new connection requests on independent service channels and are able to instantly detect and respond to any connection request initiated by another node. In this way, the system can instantly establish connections and create data channels, significantly reducing connection establishment latency. At the same time, the distributed monitoring mechanism of nodes enables more efficient resource allocation, avoiding the single point of bottleneck that can occur in traditional centralized systems. The peer-to-peer server architecture provides the technical foundation for the system's elastic expansion and contraction capabilities. When the system needs to increase or decrease the number of nodes, newly added or removed devices simply join or leave the network, automatically adapting to the network environment and connection requests without requiring additional configuration or downtime. This flexible scalability ensures that the system maintains high performance even in high-concurrency scenarios and can dynamically respond to load changes and flexibly adapt to fluctuating business demands. By equipping each node with peer-to-peer server capabilities, the system effectively avoids the impact of single points of failure. When a node fails or the network fluctuates, other nodes can quickly take over and continue providing services, ensuring system stability and service continuity. Furthermore, each node can respond to requests and assume service tasks at any time, significantly improving the overall fault tolerance and reliability of the system. Therefore, through the peer-to-peer server design, the present invention achieves a distributed architecture with high flexibility, high responsiveness, and elastic scalability. This design ensures that each node can efficiently participate in data transmission and processing while maintaining load balancing, providing excellent scalability and high performance support for distributed systems.

[0045] See also Figure 2 , is a data processing flow chart of an RDMA distributed data transmission method supporting high-concurrency message transmission provided by an embodiment of the present invention. The RDMA distributed data transmission method supporting high-concurrency message transmission includes the following steps:

[0046] S201: After each node is started, it obtains the configuration information of the management node (including IP, port, and initial node number) through the CFG configuration file and first establishes a connection with the management node to obtain the meta information of other nodes.

[0047] In some embodiments, step S201 specifically includes: in the initial stage of starting the distributed database, each node in the system needs to obtain basic configuration information of the cluster in order to correctly join the cluster and work with other nodes. This step is completed by reading the CFG configuration file.

[0048] The specific steps are as follows:

[0049] Configuration file parsing: When each node starts, it will first read the local CFG configuration file, which contains the basic configuration information of the cluster, such as the IP address, port number and initial node number of the management node. These configuration information are necessary information for each node to join the cluster, ensuring that each node can successfully communicate with the management node initially after startup. The configuration file usually contains information about multiple management nodes (such as multiple IP addresses or ports) to support high-availability configuration. Once a management node fails, the system can automatically switch to other management nodes to ensure the reliability of the system. In a distributed architecture, the management node is the core component responsible for global metadata management. Its main responsibilities include maintaining the network topology, node status, data distribution and load of all nodes in the cluster. Through the unified management of the management node, the entire cluster can achieve consistency control of metadata, avoiding distributed consistency problems caused by the independent operation of multiple nodes.

[0050] During database startup, the management node dynamically updates node information based on connection requests from individual nodes and synchronizes the current cluster layout status with newly joined nodes to ensure they can quickly join the cluster and function. Initial connection between nodes and the management node: After parsing the CFG file, each node first attempts to establish an initial connection with the management node. This connection establishment process is crucial, as the node needs to obtain global network topology and metadata from the management node to understand the location and status of other nodes in the cluster. Once the connection is established, the node sends its node information (such as IP address, port number, and node ID) to the management node. The management node then adds this information to the cluster's metadata record and monitors the node's status. Retrieving network topology heartbeat information: Upon receiving a connection request from a new node, the management node sends the node metadata, including the current network topology, node list, and data distribution status. This allows the node to clearly understand the IP addresses and ports of other nodes in the cluster upon startup, facilitating subsequent point-to-point connection establishment. This centralized management approach effectively reduces the complexity of node discovery and avoids network resource-consuming operations such as broadcasts. Furthermore, because network topology information is centrally managed by the management node, the system can dynamically adjust the topology as needed and quickly update it when nodes join or leave. Optimized startup performance: Through a centralized configuration acquisition process, nodes do not need to perform network broadcasts or point-to-point connection tests on a node-by-node basis during startup. Instead, they obtain the overall topology information directly from the management node, speeding up the startup process and improving system startup performance and reliability.

[0051] S202: Each node pre-establishes a connection with other nodes based on the meta-information obtained from the management node, and places the connection resources into the connection manager.

[0052] In some embodiments, the specific implementation process of step S202 is as follows:

[0053] Metadata Parsing: After completing the initial connection with the management node and obtaining meta information, each node parses the node list issued by the management node. This information includes the IP addresses, port numbers, and node identifiers of other nodes, providing accurate data for subsequent connection preparation. Meta information may also include the node's role (e.g., data node, compute node, standby node), load information, etc., to facilitate the allocation of different resource priorities to different node types when establishing connections.

[0054] See also Figure 4, which is a diagram of the resource allocation management of the server and client connections provided by an embodiment of the present invention. The process of each node pre-establishing a connection with other nodes and placing the connection resources into the connection manager is as follows: RDMA establishes a single memory block and needs to create an RDMA connection resource and register resource exchange through a TCP socket or RDMA bilaterally, so the connection time is 30ms. However, the data transmission time of 3us is one ten-thousandth of the time to establish a connection. If the disconnection time is added, the round-trip delay will continue to increase; the connection resources required by the RDMA need to be created when establishing a connection, including (RDMA identifier, connection parameters). Pre-established connection: In order to improve the performance of distributed databases, nodes usually adopt a batch pre-connection method, that is, according to the node list in the cluster, establish connections with multiple nodes at the same time. This method is more efficient than connecting one by one, and can reduce delays in subsequent queries or data operations. When connecting in batches, the system will reasonably allocate resources based on network bandwidth, node performance and preset concurrency limits to avoid establishing too many connections at the same time, which may lead to system resource exhaustion or performance degradation. Allocation and management of connection resources: The client's message encapsulation handler creates an RPC message packet in the local Worker (sender connection resource object) memory pool; the client manages multiple actively established connection objects, and the Worker message memory pool has been registered in advance in the client's network card to support RDMA message transmission; using the pre-established RDMA connection, the RPC message is transmitted to the target memory pool corresponding to the Partition in the server; the target memory pool is a specific memory pool in the server's memory pool that has established a mapping relationship with the client and has been registered in advance on the server's network card to support RDMA message transmission; the server manages multiple passively established connection objects, Partition, and the handler thread in the server adaptively polls the mailbox flag to see if there is a message. If there is a message, the obtained message packet is parsed and handed over to the corresponding handler for processing. After processing, the server handler will put the processed message into the corresponding memory block in the Partition and return the message through the passively established connection. After the connection is established, each node will put the resources of the connection (such as sockets, connection parameters, etc.) into the local connection manager. As the core component for managing all connection resources, the connection manager is responsible for recording the status of each connection, maintaining the connection pool, and allocating connection resources. Connection resources in the connection pool are reused to avoid the performance overhead caused by frequent connect and disconnect operations. This allows the database to quickly retrieve an available connection from the connection pool when a new query request arrives, significantly reducing the time it takes to establish a connection.

[0055] See also Figure 6, a diagram of the connection manager for a distributed network provided by an embodiment of the present invention. This invention introduces a pre-established connection resource pool to improve the system's response speed and resource utilization efficiency. Upon receiving a heartbeat message from the Meta node (i.e., the cluster's decision-making device), each device proactively establishes connections with other surviving devices and adds these connections to its local connection resource pool. This design not only reduces dynamic connection latency but also ensures real-time system connectivity under high-concurrency access, enhancing the responsiveness and reliability of the distributed architecture. In traditional dynamic connection mechanisms, devices must establish a connection before each communication, introducing additional latency, especially in high-concurrency scenarios. By pre-establishing a connection pool, this design allows each device to maintain a persistent connection with other devices upon startup, significantly reducing connection establishment time during communication. When a device in the system detects a new surviving node via a heartbeat, it immediately establishes a connection with it and adds it to the connection pool. This eliminates the need to reestablish a connection each time data transmission is required, significantly improving data transmission response speed. To ensure system flexibility, the connection resource pool supports dynamic expansion and contraction. The addition and removal of devices triggers an update to the connection pool. When a new device joins the cluster, other devices automatically establish connections to it, ensuring that all nodes in the cluster maintain synchronized connectivity. Conversely, when a node exits or a failed node is detected, the system automatically clears the corresponding connection resources to avoid resource waste. This dynamic adjustment mechanism ensures stable operation in scenarios with high concurrency or frequent node changes, and adapts to cluster configurations of varying sizes. To ensure efficient management of the connection resource pool, the system incorporates a regular maintenance mechanism to optimize the allocation and use of connection resources. By regularly checking the connection status in the connection pool, the system removes redundant connection resources and promptly updates connection information for failed nodes, reducing unnecessary connection overhead. This maintenance mechanism effectively improves resource pool utilization while preventing the accumulation of invalid connections, helping to maintain long-term system stability and efficiency. The connection resource pool design provides robust support for high-concurrency access scenarios. By pre-establishing persistent connections, devices no longer face the bottleneck of dynamically establishing connections when a surge in access requests occurs, instead utilizing the existing resource pool for communication. This mechanism not only improves the system's processing speed but also reduces network latency or connection congestion that may occur in high-concurrency environments, ensuring that the system remains stable and responsive even during peak loads. Therefore, through the pre-established connection resource pool, the present invention achieves a distributed network architecture with low latency, high responsiveness, and high stability. While ensuring efficient connectivity between devices, this mechanism flexibly adapts to changes in system scale, providing strong support for the stability and high performance of distributed systems in high-concurrency environments.

[0056] S203: After the database receives the query statement from the application, the database thread first obtains the connection resource that has been bound to the client.

[0057] In some embodiments, in step S203, the database receives a query request and obtains connection resources. In a distributed database system, when the database receives a query statement from an application, it must first find a suitable connection resource for the request to perform data transmission.

[0058] The specific steps are as follows:

[0059] Obtaining a bound connection from the client: The database thread checks the connection resources already bound to the thread to quickly find a suitable connection for the query. These connection resources are typically maintained by a connection manager, which caches direct connections from the client to the target node. The connection manager prioritizes existing connection resources over establishing a new connection. This significantly reduces the time required to establish a connection, improving query processing responsiveness.

[0060] Checking the availability of the connection: After obtaining the bound connection resources, the database thread will first check the connection's availability to ensure that the connection is valid. If a connection failure is detected, the connection manager will automatically switch to a backup connection or re-establish the connection if necessary. This mechanism prevents query failures due to connection interruptions and further optimizes system performance through load balancing. Prepare resource scheduling for the connection resource pool: When a database node needs to execute multiple queries, the connection manager will schedule appropriate connection resources from the connection pool based on the load situation and allocate resources based on the query priority and execution frequency to ensure high-priority processing of critical queries.

[0061] S204: The database calls various data transmission function modules in the client to obtain a transmission request. The various data transmission function modules may include, for example, distributed indexes for querying and retrieving query conditions, distributed locks for ensuring cache consistency and high concurrency, and remote transmission of undo segments for recovery.

[0062] In step S204, the data transmission function module is called and a transfer request is generated. After successfully acquiring the bound connection resource, the database calls various data transmission function modules in the client to generate and obtain the corresponding transfer request. The main function of these function modules is to provide the database with a series of distributed system features, such as query distribution, cache consistency, and transaction recovery.

[0063] S205: The client adds the transmission request to the header information and the RPC flag information.

[0064] In step S205, after the transmission request is generated, the client adds header information to the request and marks it with the RPC flag, indicating that it is a remote procedure call request. To optimize data transmission efficiency, the client stores the request in the local memory pool. The memory pool allocates independent slots for each memory block, which are used to temporarily store the transmission request data. This design allows the system to effectively utilize the local memory cache, avoid frequent allocation and release of memory resources, and further reduce request processing latency.

[0065] The present invention achieves efficient remote procedure call (RPC) message transmission through RPC message creation and memory registration. The client generates an RPC message packet using a local memory pool based on the captured transmission request. Specifically, the client parses the transmission request and encapsulates the requested data and call information into a message format that complies with the RPC standard to maintain consistency during subsequent data transmission and processing. This message encapsulation process includes the integration of information such as the request type, target address, and data content, ensuring the integrity and reliability of the message when transmitted between different devices. By creating the RPC message packet in the local memory pool, the client can complete the message construction and preparation more quickly. Compared to the temporary memory allocation method, the memory pool provides reusable resources, thereby reducing the system overhead caused by memory allocation and release. This design enables the client to efficiently process continuous RPC requests and is particularly suitable for achieving fast responses in high-concurrency environments. In addition, the memory resources managed in the local memory pool are also convenient for unified planning, thereby further improving the overall performance of the system.

[0066] S206: The client then places the request into a slot in the memory block belonging to the connection in the local memory pool, merges all requests in the same memory block, and encapsulates it into a large message. The client thread places the request information in the slot into the corresponding Mailbox slot on the server through a unilateral write operation on the RDMA network.

[0067] In step S206, after the client thread stores the transmission request in the memory pool, the client uses a one-way write operation over the RDMA (Remote Direct Memory Access) network to write the requested data directly to the corresponding Mailbox slot on the server. This one-way write operation eliminates the need for direct server participation, allowing the client to directly access the server's memory area. This significantly reduces network transmission latency and avoids the overhead associated with frequent network interactions in traditional RPC.

[0068] In some embodiments, the detailed steps of merging and sending (i.e., merging all requests in the same memory block and encapsulating them into a large message, and the client thread putting the slot request information into the corresponding Mailbox slot on the server through a unilateral write operation on the RDMA network) are as follows:

[0069] As a mature, high-performance network communication technology, RDMA easily exhibits a relatively stable message sending frequency under different network configurations (number of QPs, selection of unilateral / bilateral operations, sufficiently large MTU, number of elements in a single poll completion queue, connections / datagrams, reliable / unreliable, etc.). The factor that has the greatest impact on the sending frequency is primarily the message size. When the bandwidth is not fully utilized, changing the message size has little effect on the message sending frequency, so bandwidth utilization is approximately proportional to message size. This characteristic can lead to low RDMA bandwidth utilization and poor concurrent throughput in scenarios where small messages are sent, namely:

[0070] Bandwidth utilization = message sending frequency × message size

[0071] To address this issue, this embodiment innovatively introduces a small message merging mechanism. This mechanism, through inter-thread synchronized send buffers, merges multiple small messages into appropriately large data packets for transmission, significantly improving RDMA bandwidth utilization. The size of the merged message can be adjusted based on the maximum transmission rate actually supported by the network card, thereby adapting to different network environments. To achieve message merging, the send buffer's write mechanism, memory layout, the way the remote host receives messages, and the processing algorithm for merged requests all require careful consideration. Designing a send buffer based on the principle of hierarchical management is suitable for high-concurrency message scenarios. First, different node-level send buffers should be set for different nodes. This prevents the same QP from being competed for by scanning threads in different send buffers. Multiple message-level send buffers should be set within the same node-level send buffer to facilitate non-blocking request transmission for more requesters. Requests should be directly entered into the corresponding message-level send buffer using an efficient method (such as hashing). Taking the common distributed key-value storage scenario as an example, consider a relatively simple remote key-value access scenario: the key occupies a fixed 8B space, and the value length is not fixed. Suppose we need to send an 8B key to Node B on a Mellanox ConnectX-5100Gbps NIC on Node A, requesting the remote host to return a 128B value. Actual testing shows that to achieve maximum bandwidth utilization on a 100Gbps NIC, the payload required for each message send must exceed 3kB. This means we need to fill the same message-level send buffer with 384 key values ​​(3kB / 8B) between each message send to achieve maximum request concurrency within the same message. These 384 key-value access requests directed to Node B are sent through the same QP corresponding to the node-level send buffer. If concurrent requests come from different threads, we also need to consider the concurrency control issue when each thread writes to the send buffer simultaneously. Since we are considering a high-concurrency scenario, assuming that the number of requests from each thread to each node is large enough, we can simply allocate a fixed space in each message-level send buffer for each thread as a thread-level send buffer. The advantage of this is that there is no need to synchronize the write operations of each thread. We only need to ensure that the entire message-level send buffer is locked during RDMA sends until all requests in the buffer are responded to and then released.

[0072] After the careful design of message sending mentioned above, Node B is able to receive the merged access request. If Node B needs to process these 384 remote read requests concurrently and still merge the replies according to the 3kB message size to achieve maximum bandwidth utilization. Here, there is an inconsistency between the request size (8B key) and the reply size (related to the value length), which means that one request message here will correspond to a large number of reply messages. This requires the indexing algorithm used to find the value to process the request quickly, and it cannot wait for all processing to be completed before packaging the data and returning it, but instead returns it whenever the 3kB value is obtained. The sending buffer of Node B's response to Node A can also be designed based on the hierarchical management method described above.

[0073] During the small message transmission phase, the present invention uses RDMA technology to directly send encapsulated RPC messages to the server's target memory pool, specifically, the server's partition. Each server-side memory pool is pre-registered with its network interface card (NIC) and mapped to the client's corresponding memory block. This one-to-one mapping ensures that client messages are accurately delivered to the corresponding server-side memory pool during high-concurrency access. Furthermore, multiple mailbox memory blocks within the server-side memory pool can simultaneously receive messages from different clients, avoiding performance bottlenecks caused by lock contention and queuing. During data transmission, the client's memory pool is pre-registered with the NIC, enabling data to be directly transmitted to the server via a Remote Direct Memory Access (RDMA) channel. Memory registration is a key step, as it associates addresses in the client's memory pool with the NIC, enabling the RDMA channel to directly access these memory addresses without intervention from the operating system kernel. This enables "zero-copy" data transmission, directly transferring data from the client's memory pool to the server's designated memory area, completely bypassing the operating system kernel's management and significantly reducing transmission latency. The memory pool preregistration strategy significantly optimizes the efficiency of RDMA connection establishment and data transmission. After preregistration, the client can immediately perform RDMA operations when data needs to be transferred, eliminating the need for temporary memory registration, reducing pre-communication preparation time. Furthermore, the preregistered memory pool establishes a persistent and effective communication channel between the client and server, eliminating the latency associated with multiple connection establishments. This optimization is particularly effective for frequent, small-data-volume RPC requests.

[0074] In addition, the present invention supports inline sending when processing a large number of small messages: in a high-concurrency network environment, the frequency of small message transmission is very high. In the traditional way, each small message needs to be sent independently, which increases the burden on the system and causes transmission delays. Inline send technology is an optimization method that allows small messages to be written directly into the send request of network transmission without the need for additional memory address pointers. In this way, data transmission can be completed through a single operation, reducing the overhead caused by memory addressing and copying. Inline sending avoids the transmission waiting time of multiple small messages, so it can effectively improve the system response speed. It is especially suitable for scenarios that need to process frequent, short data packets, such as small status information updates in distributed services. During the transmission process, the RDMA network interface card (NIC) supports direct reading of these inline data without loading from other memory addresses, thereby reducing transmission delays and improving performance.

[0075] Furthermore, in a high-concurrency environment, the present invention pairs multiple mailbox memory pools on the server side with different client memory pools using a one-to-one mapping, enabling each client to be directly routed to the corresponding memory pool when accessing the server. This design significantly optimizes resource allocation on the server side and reduces lock contention. When processing a request, the server selects a relatively idle Worker object based on the current load and writes the message to the designated memory pool mailbox via RDMA unilateral write operations, thereby achieving load balancing.

[0076] S207: The server-side polling thread polls and checks the flags of the Mailbox slots. If the flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program processes the data in the Mailbox slot.

[0077] In step S207, the server's polling thread continuously checks the status of the Mailbox slots. When a slot's flag is detected as ONFLIGHT, the server deems a new request to have arrived and triggers the message processing program to read the request data from the Mailbox slot. This polling mechanism allows the server to quickly respond to client requests and process new requests immediately upon detection, significantly reducing response latency.

[0078] S208: The server distributes the message to different execution programs according to the message type in the message to process the message type.

[0079] In step S208, after reading the request data, the server parses the message type and dispatches the request to the corresponding execution program based on the message type. Different message types typically correspond to different database operation modules, such as read, update, and transaction control. This dispatch mechanism allows the server to call specific execution programs based on the actual needs of the request, improving request processing efficiency while also ensuring modular operation and system scalability.

[0080] S209: The server-side message execution program stores the processed response message and data (less than 512B) in the shared sending buffer of the server, and the processing program combines the response message and data, assembles the message type and message status to form an RPC message.

[0081] S210: If the returned data is larger than 512 bytes, the server message execution program uses a single-side write to directly write to the specified memory address. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. Finally, when the single-side write is completed, an RPC completion message is sent using the RPC mailbox system.

[0082] In steps S209-S210, after processing the request, the server will assemble the processing results (including response message and data) into a new RPC message. For response data less than 512B, the server will store the response message and data together in the shared send buffer of the server, and add the message type and status information to form a complete RPC message. The message is waiting in the buffer to be sent to the client. For data larger than 512B, the server will use a unilateral write operation to directly write the data to the memory address specified by the client. If the response data exceeds 3KB, the system will unpack the data and write each package to the specified memory address to avoid delays and memory usage caused by transmitting a large amount of data at one time. When all data is written, the server sends an RPC completion message to the client through the RPC mailbox system to notify the client that the data is ready.

[0083] In some embodiments, see Figure 3, a schematic diagram of a mailbox-based RDMA transmission optimization mechanism provided by an embodiment of the present invention. A message processor configured on the server sends an RPC completion message to the client via the RPC mailbox system. The message processor employs an intelligent dynamic detection mechanism. In high-concurrency scenarios, when the message volume surges, the mechanism automatically switches to a polling method to check the mailbox flag. The advantage of this polling method is that it effectively reduces the overhead caused by frequent thread switching in high-load environments, thereby improving the overall processing efficiency of the system. Conversely, when the message volume is low, the dynamic detection mechanism automatically switches to an event-driven method to check the mailbox flag. The advantage of this event-driven method is that it can more accurately respond to actual message arrival events, avoiding unnecessary waste of CPU cycles and optimizing resource utilization. By dynamically adjusting the detection strategy based on message volume, the present invention can intelligently balance system resource usage under different load conditions, ensuring response speed and processing power in high-concurrency environments while avoiding unnecessary resource consumption when the message volume is low, achieving the dual goals of high performance and low power consumption. The implementation of this mechanism not only improves the flexibility and adaptability of the server in processing messages, but also brings users a smoother and more stable service experience.

[0084] The dynamic polling and event-driven detection mechanism designed in the present invention can optimize resource utilization. The message processor on the server side can switch to polling mode at high concurrency based on the system load to reduce the overhead of thread switching and improve response speed. When the load is low, it automatically switches to event-driven mode to avoid ineffective CPU resource consumption through accurate message arrival detection. This mechanism combining adaptive polling and event-driven ensures the resource utilization efficiency of the system under different load conditions, taking into account both the rapid response capability at high concurrency and the energy saving effect at low load.

[0085] S211: The server-side message processing program sends the RPC message to the slot of the memory block in the one-to-one corresponding memory pool of the client using a one-sided Write.

[0086] In step S211, the server sends the assembled RPC message to the client's designated memory pool using a unilateral write operation, allowing the client to directly read the server's response message. This design reduces frequent interactions between the client and server, ensuring that response data reaches the client as quickly as possible, and improving data transmission efficiency.

[0087] S212: The client thread passes the returned message to the functional module in the client to process the response message.

[0088] In step S212, after receiving the returned response message, the client's processing thread passes it to the corresponding functional module on the client to complete the parsing and processing of the response data. For example, for query request results, the client will directly return the data to the application; for update request responses, the client will check whether the operation was successful. After this step is completed, the entire request processing process ends, and both the client and server release the corresponding resources to prepare for the next request.

[0089] In some embodiments, the distributed data transmission method also incorporates a resilient connection mechanism, implemented by periodically sending connection check messages to the server. This design ensures that the connection between the client and server remains active and stable. Specifically, the client periodically sends connection check messages to the server to verify the connection status. If the server fails to respond to the connection check message within a preset timeframe—that is, if the client does not receive the server's check reply message—the client takes necessary measures to maintain system stability and data integrity. Specifically, the client terminates the mapping relationship with the target server's memory pool to avoid potential data inconsistencies. During the termination process, if any unsent RPC (remote procedure call) messages remain in the client's memory pool, the system takes special measures to ensure that these messages are not lost. Specifically, the system saves the pending transfer requests corresponding to these unsent RPC messages. Once the mapping relationship between the client's memory pool and the other server's memory pool is reestablished, the system generates new RPC messages based on these pending transfer requests. These new RPC messages are directly transmitted to the other server's memory pool using RDMA technology, ensuring efficient data transmission and low latency. RDMA technology allows data to be transferred directly between memories without the need for the operating system kernel, significantly improving data transfer efficiency. Through this flexible connection and message processing mechanism, the present invention effectively protects data integrity and consistency when network conditions are unstable or server-side anomalies occur, while ensuring high system availability and performance. The implementation of this mechanism not only improves the system's fault tolerance but also provides more reliable support for distributed computing across networks.

[0090] The present invention uses a flexible connection mechanism to maintain active connections between the client and the server. The client periodically sends connection detection messages to the server to confirm the connection status. If the server does not respond within a preset time, the client disconnects the mapping relationship and takes measures to protect the integrity of the data. Unsent RPC messages are saved. When the client reestablishes a connection, these messages are resent to the new server memory pool, ensuring continuous data transmission. This design enhances the system's fault tolerance and stability, providing more reliable support for distributed computing.

[0091] In some embodiments, the distributed data transmission method also includes connection decoupling. The step of connection decoupling includes: separating connection resources (such as queue pairs QP, network memory resources MR, etc.) from the actual task processing logic, so that the connection resources between the client and the server can be independently expanded and managed. Improved scalability: This decoupling design allows the system to dynamically add new connection resources when the load increases without affecting the stability and operating efficiency of existing connections. Dynamic management: After decoupling, the system can flexibly allocate connection resources dynamically, increase or decrease resources at any time to cope with instantaneous load changes, greatly improving the elasticity and scalability of the system. The registration method for large page memory reduces the queue's sensitivity to jitter. In a distributed system, high-frequency message transmission may cause frequent jitter of the network queue, affecting system performance. The present invention reduces the queue's sensitivity to jitter by registering large page memory, thereby enhancing the stability of the system. Advantages of large page memory: Compared with conventional memory pages, large page memory can reduce the number of memory management unit (MMU) cache failures and reduce the overhead of page table lookups. Enhanced stability: By registering large page memory, the system can reduce the frequency of accesses to individual page table entries, reducing the jitter caused by frequent read and write operations in the message queue. This large page memory registration method is particularly suitable for high-frequency RDMA message transmission, which can improve the stability and consistency of data transmission, allowing the system to maintain stable performance under high load. Small messages are merged and sent to improve system bandwidth utilization. Transmitting a large number of small messages individually not only increases network transmission overhead but also reduces network bandwidth utilization. To address this problem, the present invention adopts message merging technology to package multiple small messages together for a single transmission, thereby improving bandwidth utilization efficiency. When transmitting small messages, the system first combines several small messages into a single data packet, which is then transmitted via RDMA. This method can reduce the total number of transmissions and improve network bandwidth utilization. The merged transmission method reduces the protocol overhead (such as header information) of each transmission, reducing latency while also increasing the actual effective data transmission rate. This merging method is particularly suitable for scenarios where frequent and continuous transmission of small data packets is required, such as status reporting of IoT devices or data updates of real-time monitoring systems.

[0092] Embodiments of the present invention also provide an RDMA distributed data transmission component that supports high-concurrency message transmission. This RDMA distributed data transmission component optimizes data transmission and processing processes to improve efficiency and performance. It includes a client and a server and has the following features: Client portion: The client is responsible for obtaining transmission requests to be sent to the server. These requests are data or instructions that users or applications need the server to process. Based on these transmission requests, the client creates remote procedure call (RPC) messages in the client's memory pool. This step encapsulates the requests into RPC messages for transmission over the network. The client transmits the RPC messages to the target server memory pool in the server using Remote Direct Memory Access (RDMA). RDMA technology allows data to be transferred directly between client and server memory without the intervention of the operating system kernel, thereby reducing latency and improving transmission efficiency. The client memory pool is pre-registered with the client's network interface card (NIC), ensuring the availability and direct accessibility of the memory pool. Server portion: The server contains multiple server memory pools pre-registered with the server NIC. These memory pools are used to receive RPC messages from the client. The target server memory pool of the server is a specific memory pool that has a mapping relationship with the client, which means that a direct communication channel has been established between the memory pools of the client and the server. The server obtains RPC messages from the target server memory pool and obtains transmission requests based on these messages. The server executes these transmission requests to complete the data processing or computing tasks required by the user or application. Through this design, the distributed data access system of the present invention can achieve efficient data transmission and fast data processing while reducing system overhead and latency. The direct memory access capability between the client and the server makes data transmission faster and more secure, while the pre-registered memory pool ensures the scalability and flexibility of the system. Overall, this system provides users with a high-performance, high-reliability data access solution.

[0093] Specifically, see Figure 5 , which is a structural diagram of an RDMA distributed data transmission component supporting high-concurrency message transmission provided by an embodiment of the present invention. The RDMA distributed data transmission component supporting high-concurrency message transmission includes a client 501, a server 502, and a management node 503. The client 501 and the server 502 are connected via hardware RDMA and a high-speed switch. The client 501 or the server 502 serves as a node, and the management node 503 is connected to the node.

[0094] After each node is started, it obtains the configuration information of the management node through the CFG configuration file and first establishes a connection with the management node to obtain the metadata of other nodes; wherein, the configuration information of the management node includes the IP address, port number and initial node number of the management node;

[0095] Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node and places the connection resources into the connection manager;

[0096] The database responds to the application's query statements and obtains the connection resources bound to the client;

[0097] The database calls various data transmission function modules in the client to obtain a transmission request; wherein the various data transmission function modules include distributed indexes for querying and retrieving query conditions, distributed locks for ensuring cache consistency and high concurrency, and remote transmission of undo segments for recovery;

[0098] The client adds header information and RPC flag information to the transmission request;

[0099] The client places the transmission request, including the header information and RPC flag information, into a slot in the memory block corresponding to the connection in the local memory pool. All requests in the same memory block are combined and encapsulated into a message. The client thread then places the request information in the slot into the corresponding Mailbox slot on the server through a unilateral write operation on the RDMA network.

[0100] The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot;

[0101] The server distributes the message to different execution programs according to the message type to process the message type;

[0102] The server-side message executor places the processed response message and data smaller than 512B in the server's shared send buffer. The handler combines the response message and data, adds the message type and status, and forms an RPC message.

[0103] In the server-side message execution program, if the returned data is larger than 512B, it is directly written to the specified memory address using a one-way write. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the one-way write is completed, an RPC completion message is sent using the RPC mailbox system.

[0104] The server-side message handler sends the RPC message to the client's corresponding memory pool slot using one-sided write;

[0105] The client thread passes the return message to the functional module in the client to process the response message.

[0106] It should be noted that the various system structures described in this embodiment and the methods described previously belong to the same technical concept and achieve the same technical effects through the same principles, and will not be repeated here.

[0107] The above implementation modes are only used to illustrate the present application and are not intended to limit the present application. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions also fall within the scope of the present application, and the scope of patent protection of the present application shall be defined by the claims.

Claims

1. An RDMA distributed data transmission method supporting high concurrent message transmission, characterized in that: Based on an RDMA distributed architecture supporting high-concurrency message transmission, the RDMA distributed architecture supporting high-concurrency message transmission includes a client and a server, the client and the server are connected via hardware RDMA and a high-speed switch, and the client or the server is regarded as a node, the method includes: After each node is started, it obtains the configuration information of the management node through the CFG configuration file and first establishes a connection with the management node to obtain the metadata of other nodes; wherein, the configuration information of the management node includes the IP address, port number and initial node number of the management node; Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node and places the connection resources into the connection manager; The database responds to the application's query statements and obtains the connection resources bound to the client; The database calls various data transmission function modules in the client to obtain a transmission request; wherein the various data transmission function modules include distributed indexes for querying and retrieving query conditions, distributed locks for ensuring cache consistency and high concurrency, and remote transmission of undo segments for recovery; The client adds header information and RPC flag information to the transmission request; The client places the transmission request, including the header information and RPC flag information, into a slot in the memory block corresponding to the connection in the local memory pool. All requests in the same memory block are combined and encapsulated into a message. The client thread then places the request information in the slot into the corresponding Mailbox slot on the server through a unilateral write operation on the RDMA network. The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot; The server distributes the message to different execution programs according to the message type to process the message type; The server-side message executor places the processed response message and data smaller than 512B in the server's shared send buffer. The handler combines the response message and data, adds the message type and status, and forms an RPC message. In the server-side message execution program, if the returned data is larger than 512B, it is directly written to the specified memory address using a one-way write. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the one-way write is completed, an RPC completion message is sent using the RPC mailbox system. The server-side message handler sends the RPC message to the client's corresponding memory pool slot using one-sided write; The client thread passes the return message to the functional module in the client to process the response message.

2. The method according to claim 1, characterized in that After each node is started, it obtains the configuration information of the management node through the CFG configuration file and first establishes a connection with the management node to obtain the metadata of other nodes, including: When each node starts, it reads the local CFG configuration file, which includes the IP address, port number, and initial node number of the management node. The management node is a component responsible for global metadata management in the RDMA distributed architecture that supports high-concurrency message transmission. The global metadata management includes maintaining the network topology, node status, data distribution, and load status of all nodes in the cluster. After obtaining the CFG configuration file, each node establishes an initial connection with the management node. During the initial connection process, the node obtains the global network topology and metadata information from the management node to clarify the location and status of other nodes in the cluster. After the initial connection is successfully established, the node sends its node information to the management node. The management node adds the node information to the cluster's metadata record and monitors the node's status. After each node is started, the management node dynamically updates the node information based on the connection requests of each node, and synchronizes the current layout status of the cluster to the newly joined node.

3. The method according to claim 1, characterized in that Each node pre-establishes connections with other nodes based on the metadata obtained from the management node and places the connection resources into the connection manager, including: After completing the initial connection with the management node and obtaining the meta-information, each node parses the node list issued by the management node. The node list includes the IP addresses, port numbers, and node identifiers of other nodes. The meta-information includes the node role and load information, which is used to assign different resource priorities to different types of nodes when establishing a connection. Each node simultaneously establishes connections with multiple other nodes in advance; wherein each node establishes connections with other nodes in advance by creating RDMA connection resources and exchanging registration resources through TCP sockets or RDMA bilaterally, wherein the RDMA connection resources include RDMA identifiers and connection parameters; Place the connection resource into the connection manager as follows: The client's message encapsulation processing program creates an RPC message packet in a local sender connection resource object memory pool. The client manages multiple actively established connection resource objects, and the sender connection resource object message memory pool has been pre-registered on the client's network card to support RDMA message transmission. The RPC message is transmitted to the target memory pool corresponding to the partition on the server using the pre-established RDMA connection. The target memory pool is a memory pool in the server's memory pool that has established a mapping relationship with the client and has been pre-registered on the server's network card to support RDMA message transmission. The server manages multiple passively established connection objects. The handler thread on the server adaptively polls the mailbox flag to see if there is a message. If there is a message, the obtained message packet is parsed and handed to the corresponding server handler for processing. After processing, the server handler places the processed message into the corresponding target memory pool in the partition and returns the message to each node via the passively established connection. After the connection is established, the connection resources are placed in the local connection manager. The connection manager, as the core component for managing all connection resources, is used to record the status of each connection, maintain the connection pool, and allocate connection resources.

4. The method according to claim 1, wherein The database responds to the application's query statements and obtains the connection resources bound to the client, including: The database thread checks the connection resources bound to the thread. The connection resources are maintained by the connection manager, which caches direct connection information from the client to the target node. The connection manager gives priority to existing connection resources. After obtaining the bound connection resources, the database thread checks the available status of the connection. If a connection failure is found, the connection manager automatically switches to a backup connection or re-establishes the connection. Re-establishing the connection includes: when a database node needs to execute multiple queries, the connection manager schedules appropriate connection resources from the connection pool based on the load situation, and allocates resources based on the query priority and execution frequency.

5. The method according to claim 1, wherein The client places the transmission request with the header information and RPC flag information into a slot in the memory block of the corresponding connection in the local memory pool, merges all requests in the same memory block, and encapsulates them into a message. The client thread places the request information in the slot into the corresponding Mailbox slot on the server through a unilateral write operation on the RDMA network, including: After the client thread stores the transmission request in the local memory pool, the client uses the unilateral write operation of the RDMA network to write the request data directly into the corresponding Mailbox slot on the server; All requests in the same memory block are merged and encapsulated into a message. Different node-level send buffers are set for different nodes, and multiple message-level send buffers should be set in the same node-level send buffer. The transmission request is directly filled into the corresponding message-level send buffer through hashing.

6. The method according to claim 1, characterized in that The server distributes the message to different execution programs according to the message type to process the message type, including: After reading the request data, the server parses the message type of the request data and distributes the request to the corresponding execution program according to the message type; different message types correspond to different execution programs in the database operation module, and the execution programs in the different database operation modules are used to implement reading, updating and transaction control.

7. The method according to claim 1, characterized in that The server-side message handler sends the RPC message to the client's corresponding memory pool slot using a one-sided write, including: The server performs a unilateral write operation on the assembled RPC message and sends it to the specified slot in the client's corresponding memory pool, so that the client can directly read the server's response message.

8. The method according to claim 1, characterized in that The server uses the RPC mailbox system to send an RPC completion message, including: In the face of high concurrency, when the message volume surges, it automatically switches to polling mode to detect the flag in the mailbox; When the message volume is lower than the set threshold, it automatically switches to the event-driven method to detect the flag in the mailbox.

9. The method according to claim 1, characterized in that The method further comprises: The client periodically sends connection detection messages to the server to check the connection status between the client and the server; If the server fails to respond to the connection detection message within the preset time, the client terminates the mapping relationship between it and the target server's memory pool; during the termination of the mapping relationship, if there are still unsent RPC messages in the client's memory pool, the pending transmission request corresponding to the unsent RPC message will be saved. When the mapping relationship is re-established between the client and the server, a new RPC message will be generated based on the pending transmission request; the new RPC message will be directly transmitted to the server using RDMA technology.

10. An RDMA distributed data transmission component supporting high concurrent message transmission, characterized in that: The components include a client, a server, and a management node. The client and the server are connected via hardware RDMA and a high-speed switch. The client or the server serves as a node, and the management node is connected to the node. After each node is started, it obtains the configuration information of the management node through the CFG configuration file and first establishes a connection with the management node to obtain the metadata of other nodes; wherein, the configuration information of the management node includes the IP address, port number and initial node number of the management node; Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node and places the connection resources into the connection manager; The database responds to the application's query statements and obtains the connection resources bound to the client; The database calls various data transmission function modules in the client to obtain a transmission request; wherein the various data transmission function modules include distributed indexes for querying and retrieving query conditions, distributed locks for ensuring cache consistency and high concurrency, and remote transmission of undo segments for recovery; The client adds header information and RPC flag information to the transmission request; The client places the transmission request, including the header information and RPC flag information, into a slot in the memory block corresponding to the connection in the local memory pool. All requests in the same memory block are combined and encapsulated into a message. The client thread then places the request information in the slot into the corresponding Mailbox slot on the server through a unilateral write operation on the RDMA network. The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot; The server distributes the message to different execution programs according to the message type to process the message type; The server-side message executor places the processed response message and data smaller than 512B in the server's shared send buffer. The handler combines the response message and data, adds the message type and status, and forms an RPC message. In the server-side message execution program, if the returned data is larger than 512B, it is directly written to the specified memory address using a one-way write. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the one-way write is completed, an RPC completion message is sent using the RPC mailbox system. The server-side message handler sends the RPC message to the client's corresponding memory pool slot using one-sided write; The client thread passes the return message to the functional module in the client to process the response message.

Images