RDMA (Remote Direct Memory Access) distributed data transmission method supporting high-concurrency message transmission

Abstract

The invention belongs to the technical field of distributed data access, discloses an RDMA (Remote Direct Memory Access) distributed data transmission method supporting high-concurrency message transmission, and aims to improve system performance in a high-concurrency message transmission environment. The method comprises the following core steps: 1) peer-to-peer server design: each device is an independent server, and connection is established by a connection channel monitored at any time; 2) establishing a connection resource pool in advance; 3) creating and registering an RPC message; 4) transmitting the RDMA to a memory mailbox pool of a target server; 5) the client and the server perform high-concurrency processing; 6) dynamically polling the server; (7) zero-copy transmission of data is achieved through the RDMA technology. According to the invention, the dynamic expansion capability of the architecture is improved by adopting a connection resource decoupling mode, QP jitter can be reduced and the stability of the system can be enhanced by registering a memory of a continuous large page at one time, and the message transmission efficiency under the distributed network architecture is effectively improved by adopting a combined sending strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data access, and more specifically, to an RDMA distributed data transmission method supporting high concurrent message transmission. Background Art

[0002] In modern computer systems, data transmission technology plays a vital role. It is not only related to the efficiency of data exchange within a single computer, but also directly affects the performance of distributed systems across networks. From the early parallel bus architecture to the widely used high-speed serial interfaces such as PCI Express (PCIe), the evolution of system buses has been a key factor in promoting computing performance. These bus technologies integrate data, address and control signals to achieve efficient connections between CPU, memory and I / O devices. In modern computer design, the bus is not only used to connect I / O devices, but also through direct memory access (DMA) technology, it reduces the burden on the CPU, allowing it to focus more on performing complex computing tasks, thereby improving the overall performance of the system. DMA technology allows I / O devices to exchange data directly with memory without the intervention of the CPU, which significantly speeds up data transmission. In addition, the introduction of the interrupt system makes it possible for real-time interaction between peripheral devices and the CPU, which is essential for building systems that can handle multiple tasks.

[0003] As the demand for data transmission continues to grow, traditional parallel communication methods have gradually revealed their limitations in speed and signal integrity, and high-speed serial communication technology has become a new development trend. PCIe is an outstanding representative of this trend. It uses a point-to-point connection method, which not only supports higher data transmission rates, but also effectively reduces latency and data conflicts. PCIe's multi-channel parallel transmission capability greatly improves the system's transmission bandwidth and meets the needs of high-performance computing applications.

[0004] In terms of communication between computers, the development of network technology provides standardized protocol support for data interaction between multiple devices, which not only promotes the popularization of the Internet, but also provides a foundation for the implementation of distributed computing. In modern distributed network architecture, the efficiency and flexibility of data transmission are crucial. In the traditional network transmission model, due to the large number of redundant data copies between the kernel space and the user space, this will cause significant performance loss during large-scale data transmission. In order to solve this problem, remote direct memory access (RDMA) technology came into being. RDMA technology allows one computer to directly access the memory of another computer without the intervention of the operating system's network stack and CPU, thereby greatly reducing the delay and CPU load of data transmission. This technology is particularly suitable for scenarios with strict requirements for ultra-low latency and high throughput, such as high-performance computing, distributed storage, and data centers. Remote direct memory access (RDMA) technology has been widely used in the fields of high-performance computing and data center networks due to its low latency, high bandwidth, and high throughput characteristics during transmission. RDMA technology bypasses the traditional network protocol stack and realizes direct data transmission below the network transmission layer. This stack bypass mechanism avoids the processing of multiple layers of network protocols, such as the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), and the Internet Protocol (IP), thereby improving transmission efficiency. In addition, RDMA technology supports kernel bypass, which enables applications to directly perform data transmission without entering kernel state. This approach reduces the context switching cost between kernel state and user state and reduces the frequency of data copying. RDMA technology inherits and develops the advantages of direct memory access (DMA) technology, allowing external devices (such as devices connected via the PCIe bus) to bypass the CPU and directly access system memory. RDMA further expands this concept, allowing local computers to directly access the memory of remote computers (including dynamic random access memory DRAM and non-volatile memory NVM), thereby improving the overall performance of distributed system communications.

[0005] In the research and development of network architecture, the combination of RDMA and modern high-speed network technologies, such as Ethernet, Fibre Channel, and Infiniband, further enhances its advantages. As one of the early applications of RDMA technology, Infiniband has been widely used in supercomputing centers and large-scale data centers due to its high bandwidth and low latency. In the design of future network architectures, RDMA technology is expected to be combined with new-generation network protocols such as IPv6 and time-sensitive networking (TSN) to form a more efficient and real-time communication network to meet the needs of delay-sensitive applications such as the Internet of Things, industrial control, and autonomous driving. By reducing CPU and operating system intervention, RDMA and its derivative technologies provide a lighter and faster solution for future network transmission, which will help promote further improvements in data transmission speed and efficiency in distributed systems. Summary of the invention

[0006] The purpose of the present invention is to provide an RDMA distributed data transmission method that supports high-concurrency message transmission, which can be applied to high-concurrency message transmission environments, especially in distributed network architectures, using remote direct memory access (RDMA) technology to significantly improve data transmission performance. The present invention proposes a series of optimization designs in terms of data transmission, resource management, load balancing, and dynamic scalability of the system, solving the problems of resource conflicts, transmission delays, and resource consumption in traditional distributed systems in high-concurrency environments.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] According to a first aspect of the present invention, there is provided an RDMA distributed data transmission method supporting high concurrent message transmission, based on an RDMA distributed architecture supporting high concurrent message transmission, the RDMA distributed architecture supporting high concurrent message transmission includes a client and a server, the client and the server are connected via hardware RDMA and a high-speed switch, and the client or the server is used as a node, the method includes:

[0009] After each node is started, it obtains the configuration information of the management node through the CFG configuration file, and first establishes a connection with the management node to obtain the meta information of other nodes; wherein the configuration information of the management node includes the IP, port and initial node number of the management node;

[0010] Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node, and puts the connection resources into the connection manager;

[0011] The database responds to the application's query statements and obtains the connection resources that have been bound to the client;

[0012] The database calls each data transmission function module in the client to obtain a transmission request; wherein each data transmission function module includes a distributed index for querying and retrieving query conditions, a distributed lock for ensuring cache consistency and high concurrency, and an Undo segment remote transmission for recovery;

[0013] The client adds header information and RPC flag information to the transmission request;

[0014] The client puts the transmission request with the header information and RPC flag information into a slot in the memory block of the corresponding connection in the local memory pool, merges all requests in the same memory block, and encapsulates them into a message; the client thread puts the request information of the slot into the corresponding Mailbox slot of the server through the unilateral write operation of the RDMA network;

[0015] The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot;

[0016] The server distributes the message to different execution programs to process the message type according to the message type in the message;

[0017] The server-side message execution program puts the processed response message and data less than 512B in the shared send buffer of the server, and the processing program combines the response message and data together, assembles the message type and message status to form an RPC message;

[0018] In the server-side message execution program, if the returned data is larger than 512B, the single-side write is used to directly write it to the specified memory address. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the single-side write is fully executed, an RPC completion message is sent using the RPC mailbox system.

[0019] The server-side message handler sends the RPC message to the slot of the memory block in the client's one-to-one corresponding memory pool using one-sided write;

[0020] The client thread passes the return message to the functional module in the client to process the response message.

[0021] According to a second aspect of the present invention, there is provided an RDMA distributed data transmission component supporting high concurrent message transmission, the system comprising a client, a server, and a management node, the client and the server are connected via hardware RDMA and a high-speed switch, the client or the server is used as a node, and the management node is connected to the node; wherein:

[0022] After each node is started, it obtains the configuration information of the management node through the CFG configuration file, and first establishes a connection with the management node to obtain the meta information of other nodes; wherein the configuration information of the management node includes the IP, port and initial node number of the management node;

[0023] Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node, and puts the connection resources into the connection manager;

[0024] The database responds to the application's query statements and obtains the connection resources that have been bound to the client;

[0025] The database calls each data transmission function module in the client to obtain a transmission request; wherein each data transmission function module includes a distributed index for querying and retrieving query conditions, a distributed lock for ensuring cache consistency and high concurrency, and an Undo segment remote transmission for recovery;

[0026] The client adds header information and RPC flag information to the transmission request;

[0027] The client puts the transmission request with the header information and RPC flag information into a slot in the memory block of the corresponding connection in the local memory pool, merges all requests in the same memory block, and encapsulates them into a message; the client thread puts the request information of the slot into the corresponding Mailbox slot of the server through the unilateral write operation of the RDMA network;

[0028] The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot;

[0029] The server distributes the message to different execution programs to process the message type according to the message type in the message;

[0030] The server-side message execution program puts the processed response message and data less than 512B in the shared send buffer of the server, and the processing program combines the response message and data together, assembles the message type and message status to form an RPC message;

[0031] In the server-side message execution program, if the returned data is larger than 512B, the single-side write is used to directly write it to the specified memory address. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the single-side write is fully executed, an RPC completion message is sent using the RPC mailbox system.

[0032] The server-side message handler sends the RPC message to the slot of the memory block in the client's one-to-one corresponding memory pool using one-sided write;

[0033] The client thread passes the return message to the functional module in the client to process the response message.

[0034] The present invention has at least the following beneficial effects:

[0035] The present invention can realize efficient and low-latency data transmission and parallel processing. Through remote direct memory access (RDMA) technology, a high-speed, zero-copy data transmission channel is established between the client and the server, which greatly improves the data transmission and processing performance of the system in a high-concurrency environment. After receiving the transmission request, the client will parse the request and generate a corresponding remote procedure call (RPC) message. The RPC message encapsulates the content of the request, the call instruction and the related transmission metadata, making the data transmission process structured and efficient. In order to further improve the transmission speed, the client will register the memory pool on the RDMA network interface card (RNIC) in advance. This pre-registration operation directly associates the memory pool with the network card, realizing an efficient data access path. Through RDMA technology, the client directly transmits the encapsulated RPC message to the specified memory pool of the server without the participation of the operating system kernel. This zero-copy transmission mode greatly reduces the intervention of the CPU and reduces the delay of data transmission, and is particularly suitable for high-concurrency and small message frequent transmission scenarios. The pre-registered memory pool also facilitates the reuse and rapid recovery of messages, optimizes memory management, and enables the client to respond to continuous RPC requests at a faster speed. The server is responsible for receiving RPC messages from the client, parsing and executing corresponding processing tasks according to the request. During the RDMA transmission process, the client's RPC message directly enters the pre-registered memory pool in the server, and the server can access the memory pool immediately, so as to obtain and process the message data faster. After receiving the message, the server will decapsulate and distribute the task to the corresponding handler for execution. After processing the request task, the server will put the result into the mailbox buffer and return the processing result to the client through the established RDMA channel. Through this structured request-processing-return mode, efficient concurrent processing can be achieved. The pre-registration of multiple memory pools and mailbox buffers also reduces the lock contention and resource waiting time between threads, further improving the parallel processing capability of the server. In order to support a high-concurrency dynamic environment, the present invention designs an elastic connection mechanism so that the client and the server can dynamically increase or release connection resources. The connection state is managed by heartbeat communication with the cluster decision node (Meta). After receiving the information of other surviving devices, each client actively establishes a connection with it in advance to form a connection resource pool, thereby reducing the delay caused by dynamic connection. The connection resource pool designed in this way has dynamic expansion and contraction capabilities, so that the system can efficiently handle the addition or withdrawal of client and server nodes. In addition, the present invention integrates an intelligent detection mechanism, and the server can dynamically switch between polling and event-driven modes according to the load situation. Under high load, it automatically enters the polling mode to quickly detect and process newly arrived messages; under low load, it switches to the event-driven mode, thereby reducing CPU occupancy and resource waste.Through the intelligent detection mechanism, the system can balance resource usage under different load conditions, achieving the dual advantages of high performance and low power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of an RDMA distributed architecture supporting high-concurrency message transmission provided by an embodiment of the present invention.

[0037] Figure 2 A data processing flow chart of a distributed data transmission method provided in an embodiment of the present invention.

[0038] Figure 3 A schematic diagram of a mailbox-based RDMA transmission optimization mechanism provided by an embodiment of the present invention.

[0039] Figure 4 A resource allocation management diagram for connecting a server and a client provided in an embodiment of the present invention.

[0040] Figure 5 A structural diagram of an RDMA distributed data transmission component supporting high-concurrency message transmission provided by an embodiment of the present invention.

[0041] Figure 6 A diagram of a connection manager for a distributed network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments, but are not intended to limit the present invention. For the various steps described herein, if there is no necessity for a causal relationship between each other, the order in which they are described as examples herein should not be regarded as a limitation, and those skilled in the art should know that they can be adjusted in order, as long as the logic between them is not destroyed, resulting in the inability to implement the entire process.

[0043] The embodiment of the present invention provides an RDMA distributed data transmission method supporting high concurrent message transmission, based on the RDMA distributed architecture supporting high concurrent message transmission, see Figure 1, which is a structural diagram of an RDMA distributed architecture supporting high-concurrency message transmission provided by an embodiment of the present invention. The RDMA distributed architecture supporting high-concurrency message transmission includes a first server 101, a second server 102, and a third server 103, wherein the first server 101 and the second server 102 are used as a server and a client, and the client and the server are connected through hardware RDMA and a high-speed switch, a client or a server is used as a node, and the third server 103 is used as a management node. Each first server 101 and the second server 102 are peers, and each device can be given the ability to act as both a client and a server, thereby achieving the peer nature of each node in the system.

[0044] In this embodiment, in the RDMA distributed architecture that supports high-concurrency message transmission, the design of the peer-to-peer server architecture is adopted: each device is given the ability to act as both a client and a server, thereby achieving the equality of each node in the system. This means that all devices have independent server functions, can actively listen to connection requests in the network, and can quickly respond and establish connections when receiving new connection applications. This design helps to ensure that the system maintains smooth communication and stable response speed in a dynamic, high-concurrency network environment. In the design of the peer-to-peer server, each node can not only initiate a connection request as a data requester (client), but also has the ability to respond to connection requests (server) from other nodes and process data. Through this peer-to-peer architecture, the system realizes the elastic conversion of device roles, so that any node can assume the task of receiving, processing, or forwarding data according to the real-time load situation, significantly improving the flexibility of the system. All devices continuously listen for new connection requests in independent service channels, and can detect and respond immediately once other nodes initiate connection requests. In this way, the system can instantly complete the establishment of connections and the creation of data channels, greatly reducing the delay in connection establishment. At the same time, the distributed monitoring mechanism of the nodes also enables the system to allocate resources more efficiently, avoiding the single-point bottleneck problem that may occur in traditional centralized systems. The peer-to-peer server architecture provides a technical basis for the elastic expansion and shrinking capabilities of the system. When the system needs to increase or decrease the number of nodes, the newly added or exited devices only need to join or exit the network to automatically adapt to the network environment and connection requests without additional configuration or downtime. This flexible expansion capability ensures that the system can still maintain high performance in high-concurrency scenarios, and can dynamically respond to load changes and flexibly adapt to fluctuations in business needs. By having peer-to-peer server capabilities for each node, the system effectively avoids the impact of single point failures. When a node fails or the network fluctuates, other nodes can quickly take over its role and continue to provide services to ensure the stability of the system and the continuity of services. At the same time, each node can respond to requests and undertake service tasks at any time, which greatly improves the overall fault tolerance and reliability of the system. Therefore, through the peer-to-peer server design, the present invention realizes a distributed architecture with high flexibility, high response speed and elastic expansion capabilities. The design ensures that each node can efficiently participate in data transmission and processing under the premise of load balancing, providing good scalability and high-performance support for distributed systems.

[0045] See also Figure 2 , is a data processing flow chart of an RDMA distributed data transmission method supporting high concurrent message transmission provided by an embodiment of the present invention. The RDMA distributed data transmission method supporting high concurrent message transmission comprises the following steps:

[0046] S201: After each node is started, it obtains the configuration information of the management node (including IP, port, and initial node number) through the CFG configuration file and first establishes a connection with the management node to obtain the meta information of other nodes.

[0047] In some embodiments, step S201 specifically includes: in the initial stage of starting the distributed database, each node in the system needs to obtain basic configuration information of the cluster in order to correctly join the cluster and work with other nodes. This step is completed by reading the CFG configuration file.

[0048] The specific steps are as follows:

[0049] Configuration file parsing: When each node starts, it will first read the local CFG configuration file, which contains the basic configuration information of the cluster, such as the IP address, port number and initial node number of the management node. This configuration information is necessary for each node to join the cluster, ensuring that each node can successfully communicate with the management node initially after startup. The configuration file usually contains information about multiple management nodes (such as multiple IP addresses or ports) to support high-availability configuration. Once a management node fails, the system can automatically switch to other management nodes to ensure the reliability of the system. In a distributed architecture, the management node is the core component responsible for global metadata management. Its main responsibilities include maintaining the network topology, node status, data distribution and load of all nodes in the cluster. Through the unified management of the management node, the entire cluster can achieve consistency control of metadata, avoiding distributed consistency problems caused by the independent operation of multiple nodes.

[0050] During the database startup process, the management node will dynamically update the node information according to the connection request of each node, and synchronize the current layout status of the cluster to the newly added node to ensure that the new node can quickly join the cluster and play a role. Initial connection between the node and the management node: After parsing the CFG file, each node will first try to establish an initial connection with the management node. This connection establishment process is crucial because the node needs to obtain the global network topology and metadata information from the management node to clarify the location and status of other nodes in the cluster. After the connection is successfully established, the node will send its own node information (such as IP address, port number, node number, etc.) to the management node, and the management node will add the node information to the metadata record of the cluster and monitor the status of the node. Obtain network topology heartbeat packet information: After receiving the connection request of the new node, the management node will send the metadata information such as the current network topology information, node list, data distribution status, etc. of the cluster to the node. In this way, the node can clearly understand the IP addresses and ports of other nodes in the cluster when it starts, which is convenient for subsequent point-to-point connections. This centralized management method effectively reduces the complexity of nodes discovering other nodes in the network by themselves, and avoids operations that consume network resources such as broadcasting. In addition, since the network topology information is centrally managed by the management node, the system can dynamically adjust the topology structure as needed and quickly update it when a node joins or leaves. Optimize startup performance: Through the centralized configuration acquisition process, nodes do not need to perform network broadcasts or point-to-point connection tests one by one when starting up, but directly obtain the overall topology information from the management node, thereby speeding up the startup process and improving the startup performance and reliability of the system.

[0051] S202: Each node pre-establishes a connection with other nodes according to the meta-information obtained from the management node, and puts the connection resources into the connection manager.

[0052] In some embodiments, the specific implementation process of step S202 is as follows:

[0053] Parsing meta information: After completing the initial connection with the management node and obtaining the meta information, each node will parse the node list issued by the management node. This information includes the IP address, port number, and node identifier of other nodes, providing accurate data for subsequent connection preparation. The meta information may also include the role of the node (such as data node, computing node, standby node, etc.), load information, etc., to facilitate the allocation of different resource priorities to different types of nodes when establishing a connection.

[0054] See also Figure 4, which is a diagram of resource allocation management for connection between the server and the client provided in an embodiment of the present invention. Each node pre-establishes a connection with other nodes and places the connection resources into the connection manager in the following process: RDMA establishes a single memory block and needs to create RDMA connection resources and register resource exchange through TCP socket or rdma bilateral, so the connection time is 30ms, but the data transmission of 3us is one ten-thousandth of the time to establish a connection. If the disconnection time is added, the round-trip delay will continue to increase; the connection resources required by the RDMA need to be created when establishing a connection, including (rdma identifier, connection parameters) pre-established connection: In order to improve the performance of distributed databases, nodes usually adopt a batch pre-connection method, that is, according to the node list in the cluster, establish connections with multiple nodes at the same time. This method is more efficient than connecting one by one, and can reduce the delay in subsequent queries or data operations. When connecting in batches, the system will reasonably allocate resources according to network bandwidth, node performance and preset concurrency limits to avoid system resource exhaustion or performance degradation caused by establishing too many connections at the same time. Allocation and management of connection resources: The client's message encapsulation processing program creates an RPC message packet in the local Worker (sender connection resource object) memory pool; the client manages multiple active connection objects Worker, and the Worker message memory pool has been registered in the client's network card in advance to support RDMA message transmission; using the pre-established RDMA connection, the RPC message is transmitted to the target memory pool corresponding to the Partition in the server; the target memory pool is a specific memory pool in the server's memory pool that has established a mapping relationship with the client, and has been registered in advance on the server's network card to support RDMA message transmission; the server manages multiple passive connection objects Partition, and the handler thread in the server adaptively polls the mailbox flag to see if there is a message. If there is a message, the obtained message packet is parsed and handed over to the corresponding processing program for processing. After the processing is completed, the server processing program will put the processed message into the one-to-one corresponding memory block in the Partition and return the message through the passively established connection. After each node establishes the connection, the resources of the connection (such as sockets, connection parameters, etc.) will be placed in the local connection manager. As the core component for managing all connection resources, the connection manager is responsible for recording the status of each connection, maintaining the connection pool, and allocating connection resources. The connection resources in the connection pool will be reused to avoid the performance overhead caused by frequent connection and disconnection operations. In this way, when the database receives a new query request, it can quickly take out an available connection from the connection pool, greatly reducing the time to establish a connection.

[0055] See also Figure 6, is a connection manager diagram of a distributed network provided by an embodiment of the present invention. The present invention introduces a pre-established connection resource pool to improve the response speed and resource utilization efficiency of the system. After receiving the heartbeat information sent by the Meta node (i.e., the decision-making device of the cluster), each device actively establishes connections with other surviving devices and incorporates these connections into the local connection resource pool. Such a design not only reduces the delay of dynamic connections, but also ensures the real-time connection status of the system under high concurrent access, and enhances the response speed and reliability of the distributed architecture. In the traditional dynamic connection mechanism, the device needs to establish a connection before each communication, which will introduce additional delays, especially in the case of high concurrency. This design establishes a connection pool in advance, so that each device can maintain a persistent connection with other devices after startup, greatly shortening the connection establishment time during communication. When the heartbeat detects a new surviving node in the system, it will immediately establish a connection with it and incorporate it into the connection pool, so that there is no need to re-establish the connection each time data transmission is required, thereby greatly improving the response speed of data transmission. To ensure the flexibility of the system, the connection resource pool supports dynamic expansion and contraction. The joining and leaving of devices will trigger the update of the connection pool. When a new device joins the cluster, other devices will automatically establish connections with it to ensure that the connection relationship of all nodes in the cluster remains synchronized. On the contrary, when a node exits or a failed node is detected, the system will automatically clean up the corresponding connection resources to avoid resource waste. This dynamic adjustment mechanism ensures that the system can still operate stably in scenarios with high concurrency or frequent node changes, and adapt to cluster configurations of different sizes. To ensure efficient management of the connection resource pool, the system has designed a regular maintenance mechanism to optimize the allocation and use of connection resources. By regularly checking the connection status in the connection pool, the system can clear redundant connection resources and update the connection information of failed nodes in a timely manner, thereby reducing unnecessary connection overhead. This maintenance mechanism effectively improves the utilization rate of the resource pool, while also avoiding the accumulation of invalid connections, which helps to maintain the long-term stability and efficiency of the system. The design of the connection resource pool provides a solid support for the system in high-concurrency access scenarios. Through the persistent connection established in advance, each device no longer faces the bottleneck of dynamically establishing connections when there is a surge in instantaneous access requests, but directly uses the existing resource pool for communication. This mechanism not only improves the processing speed of the system, but also reduces the network delay or connection blockage that may occur in a high-concurrency environment, ensuring that the system can still respond stably during peak loads. Therefore, through the pre-established connection resource pool, the present invention realizes a distributed network architecture with low latency, high response, and high stability. While ensuring efficient connection between devices, this mechanism flexibly adapts to changes in system scale, providing strong support for the stability and high performance of distributed systems in a high-concurrency environment.

[0056] S203: After the database receives the query statement from the application, the database thread first obtains the connection resource that has been bound to the client.

[0057] In some embodiments, in step S203, the database receives a query request and obtains connection resources. In a distributed database system, when the database receives a query statement sent by an application, it must first find a suitable connection resource for the request to perform data transmission.

[0058] The specific steps are as follows:

[0059] Get the client's bound connection: The database thread will check the connection resources that the thread has bound to in order to find a suitable connection for the query as soon as possible. These connection resources are usually maintained by the connection manager, which caches the direct connection information from the client to the target node. The connection manager will give priority to existing connection resources instead of re-establishing a connection. In this way, the database can significantly reduce the time overhead of establishing a connection and improve the response speed of query processing.

[0060] Check the availability of the connection: After obtaining the bound connection resources, the database thread will first check the available status of the connection to ensure that the connection is valid. If a connection failure is found, the connection manager will automatically switch to the backup connection or re-establish the connection when necessary. This mechanism can prevent query failures due to connection interruptions and further optimize system performance through load balancing. Prepare resource scheduling for the connection resource pool: When a database node needs to execute multiple queries, the connection manager will schedule appropriate connection resources from the connection pool according to the load situation, and allocate resources according to the query priority and execution frequency to ensure high priority processing of key queries.

[0061] S204: The database calls each data transmission function module in the client to obtain a transmission request, wherein each data transmission function module may be, for example, a distributed index for querying and retrieving query conditions, a distributed lock for ensuring cache consistency and high concurrency, and an Undo segment remote transmission for recovery.

[0062] In step S204, the data transmission function module is called and a transmission request is generated. After successfully obtaining the bound connection resources, the database will call various data transmission function modules in the client to generate and obtain corresponding transmission requests. The main function of these function modules is to provide a series of distributed system features for the database, such as query distribution, cache consistency and transaction recovery.

[0063] S205: The client adds the transmission request to the header information and the RPC flag information.

[0064] In step S205, after the input request is generated, the client will add header information to the request and mark it with the RPC flag, indicating that this is a remote procedure call request. In order to optimize the efficiency of data transmission, the client will store the request in the local memory pool. In the memory pool, independent slots are divided for each memory block, which are used to temporarily store the data of the transmission request. Through this design, the system can effectively utilize the local memory cache, avoid frequent allocation and release of memory resources, and further reduce the delay of request processing.

[0065] The present invention realizes efficient remote procedure call (RPC) message transmission through RPC message creation and memory registration. The client generates an RPC message packet using a local memory pool according to a captured transmission request. Specifically, the client parses the transmission request and encapsulates the requested data and call information into a message format that complies with the RPC standard so as to maintain consistency in subsequent data transmission and processing. This message encapsulation process includes the integration of information such as request type, target address, data content, etc., ensuring the integrity and reliability of the message when it is transmitted between different devices. By creating an RPC message packet in a local memory pool, the client can complete the construction and preparation of the message more quickly. Compared with the temporary allocation of memory, the memory pool provides reusable resources, thereby reducing the system overhead caused by memory allocation and release. This design enables the client to efficiently process continuous RPC requests, and is particularly suitable for achieving fast response in a high-concurrency environment. In addition, the memory resources managed in the local memory pool are also convenient for unified planning, thereby further improving the overall performance of the system.

[0066] S206: The client then puts it into a slot in the memory block of the connection in the local memory pool, merges all requests in the same memory block, and encapsulates it into a large message. The client thread puts the slot request information into the corresponding Mailbox slot on the server through a unilateral write operation of the RDMA network.

[0067] In step S206, after the client thread stores the transmission request in the memory pool, the client uses the unilateral write operation of the RDMA (Remote Direct Memory Access) network to directly write the request data into the corresponding Mailbox slot of the server. This unilateral write operation does not require the direct participation of the server, and the client can directly operate the memory area of ​​the server, which greatly reduces the delay in network transmission and avoids the overhead caused by frequent network interactions in traditional RPC.

[0068] In some embodiments, the detailed steps of merging and sending (i.e., merging all requests in the same memory block and encapsulating them into a large message, and the client thread puts the slot request information into the corresponding Mailbox slot on the server through a unilateral write operation of the RDMA network) are as follows:

[0069] As a mature high-performance network communication technology, RDMA tends to show a relatively stable message sending frequency under different network configurations (number of QPs, selection of unilateral / bilateral operations, sufficiently large MTU, number of elements in a single polling completion queue, connection / datagram, reliable / unreliable, etc.), and the factor that has a greater impact on the sending frequency is mainly the message size. When the bandwidth is not fully occupied, changing the message size has little effect on the message sending frequency, so the bandwidth utilization is approximately proportional to the message size. This feature will lead to low RDMA bandwidth utilization and poor concurrent throughput in the scenario of sending small messages, namely:

[0070] Bandwidth utilization = message sending frequency × message size

[0071] To address this problem, this embodiment innovatively introduces a small message merging mechanism, which merges multiple small messages into appropriately large data packets for transmission through a synchronized send buffer between threads, which can significantly improve the utilization of RDMA bandwidth. The size of the merged message can be adjusted according to the maximum transmission rate actually supported by the network card to adapt to different network environments. In order to achieve message merging, the writing mechanism of the send buffer, the memory layout, the way the remote host receives messages, and the processing algorithm for the merged request all need to be carefully considered. Designing a send buffer based on the principle of hierarchical management is suitable for high-concurrency message scenarios. First, different node-level send buffers should be set for different nodes, so as to avoid the same QP being competed for by scanning threads of different send buffers. Multiple message-level send buffers should be set in the same node-level send buffer to facilitate more requesters to achieve non-blocking request transmission. Requests should be directly filled into the corresponding message-level send buffer in an efficient way (such as hashing). Taking the common distributed key-value storage scenario as an example, consider a simpler remote key-value access scenario: the key occupies a fixed 8B space, and the length of the value is not fixed. Assume that we need to send an 8B key to Node B with the support of a Mellanox ConnectX-5100Gbps NIC on Node A, and request the remote host to return a 128B value. According to actual tests, to achieve the maximum bandwidth utilization of a 100Gbps NIC, the payload size required for each message transmission should exceed 3kB. This means that we need to fill 3kB / 8B=384 key values ​​into the same message-level send buffer in the interval between every two message transmissions to achieve the maximum request concurrency in the same message. These 384 key-value access requests to Node B will be sent out via the same QP corresponding to the node-level send buffer. If concurrent requests come from different threads, we also need to consider the concurrency control problem when each thread writes to the send buffer at the same time. Since we are considering a high-concurrency scenario, assuming that the number of requests from each thread to each node is large enough, we only need to simply allocate a fixed space for each thread in each message-level send buffer as a thread-level send buffer. The advantage of this is that there is no need to synchronize the write operations of each thread. We only need to ensure that the entire message-level send buffer is locked when performing RDMA sending until all requests in the buffer are responded to and then released.

[0072] After the above careful design of message sending, node B is able to receive the merged access request. If node B needs to process these 384 remote read requests concurrently, and still merge the replies according to the 3kB message size to achieve maximum bandwidth utilization. Here, the request size (8B key) and the reply size (related to the value length) are inconsistent, which means that one request message here will correspond to a large number of reply messages. This makes the index algorithm used to find the value should realize fast processing of requests, and cannot wait for all processing to be completed before packaging and returning the data, but return it whenever the 3kB value is obtained. The sending buffer for node B to respond to node A can also be designed based on the hierarchical management method described above.

[0073] In the small message transmission stage of the present invention, the client uses RDMA technology to directly send the encapsulated RPC message to the target memory pool of the server, that is, the Partition of the server. Each memory pool of the server is pre-registered in its network card and mapped to the corresponding memory block of the client. This one-to-one mapping relationship ensures that the client's message can be accurately sent to the corresponding server memory pool during high concurrent access. In addition, multiple mailbox memory blocks in the server memory pool can simultaneously receive messages from different clients, avoiding performance bottlenecks caused by lock contention and queuing. During the data transmission process, the client's memory pool has been registered in the network card in advance, so that the data can be directly transmitted to the server through the remote direct memory access (RDMA) channel. Memory registration is a key step, which associates the address in the client's memory pool with the network card, so that the RDMA channel can directly access these memory addresses without the intervention of the operating system kernel. In this way, data transmission can achieve a "zero copy" mode, that is, directly from the client's memory pool to the memory area specified by the server, completely bypassing the management of the operating system kernel, thereby greatly reducing the delay in the transmission process. The pre-registration strategy of the memory pool significantly optimizes the efficiency of establishing RDMA connections and data transmission. After pre-registration, the client can perform RDMA operations immediately when data needs to be transmitted, without temporarily registering memory, reducing the preparation time before communication. At the same time, the pre-registered memory pool establishes a continuous and effective communication channel between the client and the server, thereby eliminating the delay caused by multiple connection establishment. For frequent RPC requests with small data volumes, the performance improvement brought by this optimization is particularly obvious.

[0074] In addition, the present invention supports inline sending when processing a large number of small messages: in a high-concurrency network environment, the frequency of small message transmission is very high. In the traditional way, each small message needs to be sent independently, which will increase the burden on the system and cause transmission delays. Inline send technology is an optimization method that allows small messages to be directly written into the sending request of network transmission without the need for additional memory address pointers. In this way, data transmission can be completed through a single operation, reducing the overhead caused by memory addressing and copying. Inline sending avoids the transmission waiting time of multiple small messages, so it can effectively improve the system response speed. It is particularly suitable for scenarios where frequent, short data packets need to be processed, such as small status information updates in distributed services. During the transmission process, the RDMA network interface card (NIC) supports direct reading of these inline data without loading from other memory addresses, thereby reducing transmission delays and improving performance.

[0075] At the same time, in a high-concurrency environment, the present invention uses a one-to-one mapping method to pair multiple mailbox memory pools on the server with different client memory pools, so that each client can be directly routed to the corresponding memory pool when accessing the server. This design greatly optimizes the resource allocation of the server and reduces the lock contention problem. When processing a request, the server selects a relatively idle Worker object based on the current load status, and writes the message to the specified memory pool mailbox through the unilateral write operation of RDMA, thereby achieving load balancing.

[0076] S207: The server-side polling thread polls and checks the flags of the Mailbox slots. If the flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program processes the data in the Mailbox slot.

[0077] In step S207, on the server side, the polling thread will continuously check the status of the Mailbox slot. When the flag of a slot is detected to be in the ONFLIGHT state, the server side will consider that a new request has arrived and trigger the message processing program to read the request data in the Mailbox slot. Through this polling mechanism, the server side can quickly respond to requests from the client side and process new requests immediately after detecting them, thereby greatly reducing response delays.

[0078] S208: The server distributes the message to different execution programs according to the message type in the message to process the message type.

[0079] In step S208, after reading the request data, the server will parse the message type of the request and distribute the request to the corresponding execution program according to the message type. Different message types usually correspond to different database operation modules, such as reading, updating, transaction control, etc. Through this distribution mechanism, the server can call a specific execution program according to the actual needs of the request, which improves the processing efficiency of the request and also ensures the modularity of the operation and the scalability of the system.

[0080] S209: The server-side message execution program stores the processed response message and data (less than 512B) in the shared sending buffer of the server, and the processing program combines the response message and data together with the message type and message status to form an RPC message.

[0081] S210: If the returned data is larger than 512B, the server message execution program uses a single-side write to directly write to the specified memory address. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. Finally, when the single-side write is fully executed, an RPC completion message is sent using the RPC mailbox system.

[0082] In steps S209-S210, after processing the request, the server will assemble the processing results (including response message and data) into a new RPC message. For response data less than 512B, the server will store the response message and data together in the shared send buffer of the server, and add the message type and status information to form a complete RPC message. The message is waiting in the buffer to be sent to the client. For data greater than 512B, the server will use a unilateral write operation to directly write the data to the memory address specified by the client. If the response data exceeds 3KB, the system will unpack the data and write each package to the specified memory address to avoid delays and memory usage caused by transmitting a large amount of data at one time. When all data is written, the server sends an RPC completion message to the client through the RPC mailbox system to notify the client that the data is ready.

[0083] In some embodiments, see Figure 3, a schematic diagram of a mailbox-based RDMA transmission optimization mechanism provided by an embodiment of the present invention. A message processor is configured in the server to send an RPC completion message to the client through the RPC mailbox system, and the message processor adopts an intelligent dynamic detection mechanism. In the face of high concurrency, when the message volume surges, the mechanism can automatically switch to a polling method to detect the flag in the mailbox. The advantage of this polling method is that it can effectively reduce the overhead caused by frequent thread switching in a high-load environment, thereby improving the overall processing efficiency of the system. On the contrary, when the message volume is low, the dynamic detection mechanism will automatically switch to an event-driven method to detect the flag in the mailbox. The advantage of the event-driven method is that it can respond to actual message arrival events more accurately, avoid unnecessary waste of CPU cycles, and optimize resource utilization. Through this method of dynamically adjusting the detection strategy according to the message volume, the present invention can intelligently balance the use of system resources under different load conditions, which not only ensures the response speed and processing power in a high-concurrency environment, but also avoids unnecessary resource consumption when the message volume is low, achieving the dual goals of high efficiency and low power consumption. The implementation of this mechanism not only improves the flexibility and adaptability of the server in processing messages, but also brings users a smoother and more stable service experience.

[0084] The dynamic polling and event-driven detection mechanism designed by the present invention can optimize resource utilization. The message processor on the server side can switch to polling mode at high concurrency according to the system load to reduce the overhead of thread switching and improve the response speed. When the load is low, it automatically switches to event-driven mode to avoid invalid CPU resource consumption through accurate message arrival detection. This mechanism combining adaptive polling and event-driven ensures the resource utilization efficiency of the system under different load conditions, taking into account the rapid response capability at high concurrency and the energy saving effect at low load.

[0085] S211: The server-side message processing program sends the RPC message to the slot of the memory block in the one-to-one corresponding memory pool of the client using a one-sided Write.

[0086] In step S211, the server sends the assembled RPC message to the specified slot in the client's corresponding memory pool using a unilateral write operation, so that the client can directly read the server's response message. This design reduces the frequent interaction between the client and the server, ensures that the response data can reach the client as soon as possible, and improves the efficiency of data transmission.

[0087] S212: The client thread sends the return message to the functional module in the client to process the response message.

[0088] In step S212, after receiving the returned response message, the client's processing thread will hand it over to the corresponding functional module of the client to complete the parsing and processing of the response data. For example, for the result data of the query request, the client will directly return the data to the application; for the response to the update request, the client will check whether the operation is successful, etc. After this step is completed, the entire request processing process ends, and both the client and the server release the corresponding resources to prepare for the next request.

[0089] In some embodiments, the distributed data transmission method also designs a flexible connection mechanism, which is implemented by regularly sending connection detection messages to the server. The purpose of this design is to ensure that the connection between the client and the server is always active and stable. Specifically, the client will periodically send connection detection messages to the server to verify the connection status between the two. If the server fails to respond to the connection detection message within a preset time, that is, the client does not receive the detection reply message from the server, then the client will take necessary measures to maintain the stability of the system and the integrity of the data. Specifically, the client will terminate the mapping relationship with the target server memory pool to avoid potential data inconsistency problems. In the process of terminating the mapping relationship, if there are still unsent RPC (remote procedure call) messages in the client memory pool, the system will take special measures to ensure that these messages are not lost. Specifically, the system will save the transmission requests to be sent corresponding to these unsent RPC messages. Once the mapping relationship between the client memory pool and other server memory pools is re-established, the system will generate new RPC messages based on these transmission requests to be sent. These new RPC messages will be directly transmitted to other server memory pools using RDMA technology, thereby ensuring high efficiency and low latency of data transmission. RDMA technology allows data to be transferred directly between memories without the intervention of the operating system kernel, which can significantly improve the efficiency of data transmission. Through this elastic connection and message processing mechanism, the present invention can effectively protect the integrity and consistency of data when the network environment is unstable or anomalies occur on the server side, while ensuring the high availability and high performance of the system. The implementation of this mechanism not only improves the fault tolerance of the system, but also provides more reliable support for distributed computing across networks.

[0090] The present invention adopts an elastic connection mechanism to maintain the connection activity between the client and the server. The client will periodically send a connection detection message to the server to confirm the connection status. If the server does not respond within a preset time, the client will disconnect the mapping relationship and take measures to protect the integrity of the data. Unsent RPC messages will be saved, and when the client re-establishes the connection, these messages will be resent to the new server memory pool to achieve continuity of data transmission. This design enhances the fault tolerance and stability of the system and provides more reliable support for distributed computing.

[0091] In some embodiments, the distributed data transmission method also includes connection decoupling. Among them, the step of connection decoupling includes: separating connection resources (such as queue pairs QP, network memory resources MR, etc.) from the actual task processing logic, so that the connection resources between the client and the server can be independently expanded and managed. Improved scalability: This decoupling design allows the system to dynamically add new connection resources when the load increases without affecting the stability and operating efficiency of existing connections. Dynamic management: After decoupling, the system can flexibly allocate connection resources dynamically, increase or decrease resources at any time to cope with instantaneous load changes, greatly improving the elasticity and scalability of the system. The registration method of large page memory reduces the sensitivity of the queue to jitter. In a distributed system, high-frequency message transmission may cause frequent jitter of the network queue, affecting system performance. The present invention reduces the sensitivity of the queue to jitter by registering a large page memory, thereby enhancing the stability of the system. Advantages of large page memory: Compared with conventional memory pages, large page memory can reduce the number of memory management unit (MMU) cache failures and reduce the overhead of page table lookups. Enhanced stability: By registering large page memory, the system can reduce the frequency of access to a single page table entry and reduce the jitter phenomenon caused by frequent read and write operations of the message queue. This large page memory registration method is particularly suitable for high-frequency RDMA message transmission, which can improve the stability and consistency of data transmission, so that the system can still maintain stable performance under high load. Small messages are sent together to improve system bandwidth utilization. Transmitting a large number of small messages separately not only increases the transmission overhead of the network, but also reduces the utilization of network bandwidth. In order to solve this problem, the present invention adopts message merging technology to package multiple small messages together for one-time transmission, thereby improving bandwidth utilization efficiency. When transmitting small messages, the system first combines several small messages into a data packet, and then transmits it through RDMA. This method can reduce the total number of transmissions and improve the utilization of network bandwidth. The combined transmission method reduces the protocol overhead (such as header information) of each transmission, while reducing delays and also improving the actual effective data transmission rate. This merging method is particularly suitable for scenarios where frequent and continuous small data packets need to be transmitted, such as status reporting of IoT devices or data updates of real-time monitoring systems.

[0092] The embodiment of the present invention also provides an RDMA distributed data transmission component supporting high-concurrency message transmission, which optimizes the data transmission and processing flow to improve efficiency and performance. It includes a client and a server, and has the following characteristics: Client part: The client is responsible for obtaining the transmission request to be sent to the server. These requests are data or instructions that users or applications need to be processed by the server. Based on these transmission requests, the client creates a remote procedure call (RPC) message in the client memory pool. This step is to encapsulate the request into an RPC message so that it can be sent over the network. The client transmits the RPC message to the target server memory pool in the server through remote direct memory access (RDMA). RDMA technology allows data to be transmitted directly between the client and the server's memory without the intervention of the operating system kernel, thereby reducing delays and improving transmission efficiency. The client memory pool is pre-registered in the client's network interface card (NIC), ensuring the availability and direct access capability of the memory pool. Server part: The server includes multiple server memory pools pre-registered in the server NIC, which are used to receive RPC messages from the client. The target server memory pool of the server is a specific memory pool with a mapping relationship with the client, which means that the memory pools between the client and the server have established a direct communication channel. The server obtains RPC messages from the target server memory pool and obtains transmission requests based on these messages. The server executes these transmission requests to complete the data processing or computing tasks required by the user or application. Through this design, the distributed data access system of the present invention can achieve efficient data transmission and fast data processing, while reducing system overhead and latency. The direct memory access capability between the client and the server makes data transmission faster and safer, and the pre-registered memory pool ensures the scalability and flexibility of the system. Overall, the system provides users with a high-performance, high-reliability data access solution.

[0093] Specifically, see Figure 5 , is a structural diagram of an RDMA distributed data transmission component supporting high-concurrency message transmission provided by an embodiment of the present invention, the RDMA distributed data transmission component supporting high-concurrency message transmission includes a client 501, a server 502 and a management node 503, the client 501 and the server 502 are connected through hardware RDMA and a high-speed switch, the client 501 or the server 502 is used as a node, and the management node 503 is connected to the node; wherein:

[0094] After each node is started, it obtains the configuration information of the management node through the CFG configuration file, and first establishes a connection with the management node to obtain the meta information of other nodes; wherein the configuration information of the management node includes the IP, port and initial node number of the management node;

[0095] Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node, and puts the connection resources into the connection manager;

[0096] The database responds to the application's query statements and obtains the connection resources that have been bound to the client;

[0097] The database calls each data transmission function module in the client to obtain a transmission request; wherein each data transmission function module includes a distributed index for querying and retrieving query conditions, a distributed lock for ensuring cache consistency and high concurrency, and an Undo segment remote transmission for recovery;

[0098] The client adds header information and RPC flag information to the transmission request;

[0099] The client puts the transmission request with the header information and RPC flag information into a slot in the memory block of the corresponding connection in the local memory pool, merges all requests in the same memory block, and encapsulates them into a message; the client thread puts the request information of the slot into the corresponding Mailbox slot of the server through the unilateral write operation of the RDMA network;

[0100] The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot;

[0101] The server distributes the message to different execution programs to process the message type according to the message type in the message;

[0102] The server-side message execution program puts the processed response message and data less than 512B in the shared send buffer of the server, and the processing program combines the response message and data together, assembles the message type and message status to form an RPC message;

[0103] In the server-side message execution program, if the returned data is larger than 512B, the single-side write is used to directly write it to the specified memory address. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the single-side write is fully executed, an RPC completion message is sent using the RPC mailbox system.

[0104] The server-side message handler sends the RPC message to the slot of the memory block in the client's one-to-one corresponding memory pool using one-sided write;

[0105] The client thread passes the return message to the functional module in the client to process the response message.

[0106] It should be noted that the various system structures described in this embodiment belong to the same technical concept as the previously described method, and achieve the same technical effect through the same principle, which will not be repeated here.

[0107] The above implementation modes are only used to illustrate the present application, and are not intended to limit the present application. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions also belong to the scope of the present application, and the scope of patent protection of the present application shall be limited by the claims.

Claims

1. An RDMA distributed data transmission method supporting high concurrent message transmission, characterized in that: Based on an RDMA distributed architecture supporting high concurrent message transmission, the RDMA distributed architecture supporting high concurrent message transmission includes a client and a server, the client and the server are connected via hardware RDMA and a high-speed switch, and the client or the server is used as a node, the method includes: After each node is started, it obtains the configuration information of the management node through the CFG configuration file, and first establishes a connection with the management node to obtain the meta information of other nodes; wherein the configuration information of the management node includes the IP, port and initial node number of the management node; Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node, and puts the connection resources into the connection manager; The database responds to the application's query statements and obtains the connection resources that have been bound to the client; The database calls each data transmission function module in the client to obtain a transmission request; wherein each data transmission function module includes a distributed index for querying and retrieving query conditions, a distributed lock for ensuring cache consistency and high concurrency, and an Undo segment remote transmission for recovery; The client adds header information and RPC flag information to the transmission request; The client puts the transmission request with the header information and RPC flag information into a slot in the memory block of the corresponding connection in the local memory pool, merges all requests in the same memory block, and encapsulates them into a message; the client thread puts the request information of the slot into the corresponding Mailbox slot of the server through the unilateral write operation of the RDMA network; The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot; The server distributes the message to different execution programs to process the message type according to the message type in the message; The server-side message execution program puts the processed response message and data less than 512B in the shared send buffer of the server, and the processing program combines the response message and data together, assembles the message type and message status to form an RPC message; In the server-side message execution program, if the returned data is larger than 512B, the single-side write is used to directly write it to the specified memory address. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the single-side write is fully executed, an RPC completion message is sent using the RPC mailbox system. The server-side message handler sends the RPC message to the slot of the memory block in the client's one-to-one corresponding memory pool using one-sided write; The client thread passes the return message to the functional module in the client to process the response message.

2. The method according to claim 1, characterized in that After each node is started, it obtains the configuration information of the management node through the CFG configuration file, and first establishes a connection with the management node to obtain the meta information of other nodes, including: When each node starts, it reads the local CFG configuration file, which includes the IP address, port number and initial node number of the management node. The management node is a component responsible for global metadata management in the RDMA distributed architecture that supports high-concurrency message transmission; wherein the global metadata management includes maintaining the network topology, node status, data distribution and load status of all nodes in the cluster; After obtaining the CFG configuration file, each node establishes an initial connection with the management node. During the initial connection process, the node obtains the global network topology and metadata information from the management node to clarify the location and status of other nodes in the cluster. After the initial connection is successfully established, the node sends its node information to the management node. The management node adds the node information to the metadata record of the cluster and monitors the node status. After each node is started, the management node dynamically updates the node information based on the connection requests of each node, and synchronizes the current layout status of the cluster to the newly added nodes.

3. The method according to claim 1, characterized in that Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node, and puts the connection resources into the connection manager, including: After completing the initial connection with the management node and obtaining the meta information, each node parses the node list issued by the management node; wherein the node list includes the IP address, port number and node identifier of other nodes, and the meta information includes the role and load information of the node, which is used to assign different resource priorities to different types of nodes when establishing a connection; Each node simultaneously establishes a connection with multiple other nodes in advance; wherein each node establishes a connection with other nodes in advance by creating an RDMA connection resource and exchanging registration resources through a TCP socket or an rdma bilateral, wherein the RDMA connection resource includes an rdma identifier and connection parameters; Place the connection resource into the connection manager as follows: The client's message encapsulation processing program creates an RPC message packet in the local sender connection resource object memory pool; the client manages multiple actively established connection resource objects, and the sender connection resource object message memory pool has been registered in advance in the client's network card to support RDMA message transmission; using the pre-established RDMA connection, the RPC message is transmitted to the target memory pool corresponding to the Partition in the server; the target memory pool is a memory pool in the server memory pool that has established a mapping relationship with the client, and has been registered in advance on the server's network card to support RDMA message transmission; the server manages multiple passively established connection objects, and the handler thread in the server adaptively polls the mailbox flag to see if there is a message. If there is a message, the obtained message packet is parsed and handed over to the corresponding server processing program for processing. After the processing is completed, the server processing program puts the processed message into the one-to-one corresponding target memory pool in the Partition and returns the message to each node through the passively established connection. After the connection is established, the connected resources will be put into the local connection manager. The connection manager is the core component for managing all connection resources, which is used to record the status of each connection, maintain the connection pool and allocate connection resources.

4. The method according to claim 1, characterized in that: The database responds to the application's query statements and obtains the connection resources bound to the client, including: The database thread checks the connection resources bound to the thread. The connection resources are maintained by the connection manager, and the direct connection information from the client to the target node is cached. The connection manager gives priority to the existing connection resources. After obtaining the bound connection resources, the database thread checks the available status of the connection. If a connection failure is found, the connection manager automatically switches to the backup connection or re-establishes the connection. Among them, re-establishing the connection includes: when a database node needs to execute multiple queries, the connection manager schedules appropriate connection resources from the connection pool according to the load situation, and allocates resources according to the priority and execution frequency of the query.

5. The method according to claim 1, characterized in that The client puts the transmission request with header information and RPC flag information into a slot in the memory block of the corresponding connection in the local memory pool, merges all requests in the same memory block, and encapsulates them into a message; the client thread puts the request information of the slot into the corresponding Mailbox slot of the server through a unilateral write operation of the RDMA network, including: After the client thread stores the transmission request in the local memory pool, the client uses the unilateral write operation of the RDMA network to directly write the request data into the corresponding Mailbox slot on the server; All requests in the same memory block are merged and encapsulated into a message. Different node-level sending buffers are set for different nodes, and multiple message-level sending buffers should be set in the same node-level sending buffer. The transmission request is directly filled into the corresponding message-level sending buffer through hashing.

6. The method according to claim 1, characterized in that The server distributes the messages to different executors to process the message types, including: After reading the request data, the server parses the message type of the request data and distributes the request to the corresponding execution program according to the message type; wherein different message types correspond to execution programs in different database operation modules, and the execution programs in the different database operation modules are used to implement reading, updating and transaction control.

7. The method according to claim 1, characterized in that The server-side message handler sends the RPC message to the client's corresponding memory pool slot using one-sided write, including: The server performs a unilateral write operation on the assembled RPC message and sends it to the specified slot in the client's corresponding memory pool, so that the client can directly read the server's response message.

8. The method according to claim 1, characterized in that The server uses the RPC mailbox system to send an RPC completion message, including: In the face of high concurrency, when the number of messages surges, it automatically switches to polling mode to detect the flag in the mailbox; When the message volume is lower than the set threshold, it automatically switches to the event-driven method to detect the flag in the mailbox.

9. The method according to claim 1, characterized in that: The method further comprises: The client periodically sends connection detection messages to the server to check the connection status between the client and the server; If the server fails to respond to the connection detection message within the preset time, the client terminates the mapping relationship with the target server memory pool; in the process of terminating the mapping relationship, if there are still unsent RPC messages in the client's memory pool, the pending transmission request corresponding to the unsent RPC message is saved, and when the mapping relationship is re-established between the client and the server, a new RPC message is generated based on the pending transmission request; the new RPC message is directly transmitted to the server using RDMA technology.

10. An RDMA distributed data transmission component supporting high concurrent message transmission, characterized in that: The components include a client, a server, and a management node. The client and the server are connected via hardware RDMA and a high-speed switch. The client or the server is regarded as a node, and the management node is connected to the node. After each node is started, it obtains the configuration information of the management node through the CFG configuration file, and first establishes a connection with the management node to obtain the meta information of other nodes; wherein the configuration information of the management node includes the IP, port and initial node number of the management node; Each node pre-establishes connections with other nodes based on the meta-information obtained from the management node, and puts the connection resources into the connection manager; The database responds to the application's query statements and obtains the connection resources that have been bound to the client; The database calls each data transmission function module in the client to obtain a transmission request; wherein each data transmission function module includes a distributed index for querying and retrieving query conditions, a distributed lock for ensuring cache consistency and high concurrency, and an Undo segment remote transmission for recovery; The client adds header information and RPC flag information to the transmission request; The client puts the transmission request with the header information and RPC flag information into a slot in the memory block of the corresponding connection in the local memory pool, merges all requests in the same memory block, and encapsulates them into a message; the client thread puts the request information of the slot into the corresponding Mailbox slot of the server through the unilateral write operation of the RDMA network; The server-side polling thread polls and checks the RPC flag information of each Mailbox slot. When the RPC flag information is in the ONFLIGHT state, it means that a message has arrived, and the message processing program is called to process the data in the Mailbox slot; The server distributes the message to different execution programs to process the message type according to the message type in the message; The server-side message execution program puts the processed response message and data less than 512B in the shared send buffer of the server, and the processing program combines the response message and data together, assembles the message type and message status to form an RPC message; In the server-side message execution program, if the returned data is larger than 512B, the single-side write is used to directly write it to the specified memory address. If the size is larger than 3KB, it is unpacked and distributed to the specified memory address. When the single-side write is fully executed, an RPC completion message is sent using the RPC mailbox system. The server-side message handler sends the RPC message to the slot of the memory block in the client's one-to-one corresponding memory pool using one-sided write; The client thread passes the return message to the functional module in the client to process the response message.

Images