A Redis system network data transmission method based on RDMA

Abstract

The present invention discloses a Redis system network data transmission method based on RDMA. The Redis server selects a network communication component based on RDMA or Socket according to the network scenario and registers the RDMA connection type. When a client sends a connection request to the Redis server, the Redis server receives the connection request and allocates a block to the client as the client's private block. The client sends a data request based on the Redis command format to the Redis server. The Redis server extracts data requests from the data request based on the Redis command format by continuously polling the receiving blocks of the private blocks of the clients connected to it. The Redis server processes the extracted data request and stores the processing result of the data request in the sending block of the private block of the corresponding client. The Redis server returns the processing result of the data request stored in the sending block of the private block of the client to the corresponding client. The present invention can support different communication modes and can improve system operation efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of Redis (Remote Dictionary Server) systems, and in particular to a Redis system network data transmission method based on RDMA. Background Art

[0002] For decades, relational database systems have been widely used in data storage due to their high reliability and stability. With the advent of the big data era, traditional relational databases, which require both reading data from disk and the time-consuming parsing of SQL statements, are struggling to cope with the massive amounts of data and the demands for rapid response. Redis, a memory-based key-value storage system and a NoSQL non-relational database, has become a very popular data storage method. Redis centrally stores data in the operating system's memory, eliminating the slow access times associated with disk or flash memory, allowing applications to access data more efficiently. While non-relational distributed key-value storage systems offer high concurrency, efficient queries, and high scalability compared to traditional relational databases, the increasing processing speed of individual processors has made distributed network transmission a major latency bottleneck for distributed applications. Summary of the Invention

[0003] The present invention aims to solve the problem that the performance overhead of network I / O becomes the performance bottleneck of the Redis system in the face of large amounts of dynamic data and the continuous expansion of the number of clusters, and provides a Redis system network data transmission method based on RDMA.

[0004] To solve the above problems, the present invention is achieved through the following technical solutions:

[0005] A Redis system network data transmission method based on RDMA includes the following steps:

[0006] (1) Redis server initialization phase:

[0007] Step 1.1. Load the communication module in the Redis server. This communication module consists of an RDMA-based network communication component and a traditional Socket-based network communication component. These two components are switched based on specific network scenarios by adding selection parameters in the Redis configuration file and judgment parameters in the redisServer structure.

[0008] Step 1.2: When the Redis server starts, it allocates a continuous memory area, which is divided into N blocks. Each block includes a receiving block and a sending block. N is a set value.

[0009] Step 1.3. Register the RDMA connection type in the connection layer framework of the Redis system.

[0010] (2) Redis system data transmission stage:

[0011] Step 2.1: When the client sends a connection request to the Redis server, the Redis server receives the connection request and establishes a connection with the client, creates a new file descriptor for network communication, and allocates a block to the client as the client's private block;

[0012] Step 2.2: The client adds a message header and a tail flag to the data request to obtain a data request in the Redis command format. The client then uses the RDMA Write operation to send the data request in the Redis command format to the Redis server.

[0013] Step 2.3: The Redis server continuously polls the receiving blocks of the private blocks of the clients connected to it. When it finds that the receiving block of a client's private block is no longer empty, the Redis server determines whether the unique identifier of the message header and the unique identifier of the tail flag in the data request based on the Redis command format stored therein are consistent, and whether the unique identifier is greater than the unique identifier of the previous data request recorded by the Redis server. If they are consistent and greater, the data request in the data request based on the Redis command format is extracted; otherwise, the Redis server continues to poll the receiving blocks of the private blocks of the clients connected to it.

[0014] Step 2.4: The Redis server calls the original Redis request command handler to process the extracted data request and stores the processing result of the data request in the sending block of the private block of the corresponding client;

[0015] Step 2.5: The Redis server uses the Send operation to return the processing result of the data request stored in the sending block of the client's private block to the corresponding client.

[0016] In step 1.1 above, add a communication_mode parameter as a selection parameter in the Redis configuration file, and add a member variable is_rdma as a judgment parameter in the redisServer structure;

[0017] If the Redis server reads the value of the communication_mode parameter in the Redis configuration file as rdma, it sets the member variable is_rdma in the global variable server of the redisServer structure type to 1, and the Redis server selects the RDMA-based network communication component;

[0018] If the Redis server reads the value of the communication_mode parameter in the Redis configuration file and it is tcp or empty, the member variable is_rdma in the global variable server of the redisServer structure type is set to 0, and the Redis server runs the traditional network communication component based on Socket.

[0019] In step 2.1 above, when the first Redis client initiates a connection request to the server, the Redis server allocates the first block of the memory area to the client when establishing a connection with it; when the second client initiates a connection request to the Redis server, the Redis server allocates the second block of the memory area to the client when establishing a connection with it; and so on, until all N blocks in the memory area allocated by the Redis server are allocated. At this time, when the (N+1)th client initiates a connection request to the Redis server, the Redis server allocates another continuous memory area containing N blocks.

[0020] In the above step 2.2, the data request based on the Redis command format consists of three parts: a message header, a data request, and a tail flag. The message header part consists of the length of the data request and the unique identifier of the data request, the data request part consists of the data request, and the tail flag part consists of the unique identifier of the data request.

[0021] In step 2.5 above, when the processing result of the data request sent by the Redis server is greater than the maximum value on the message transmission path, the Redis server first groups the processing results of the data request, sets a sequence number for each group, and adds an end flag at the end of each group. Then, the group is returned to the client in sequence using the Send operation; after the client receives all the groups, it processes and integrates all the groups according to the sequence number and end flag of the group, thereby obtaining a complete and correct processing result of the data request.

[0022] In the above step 2.5, before the Redis server returns the processing result of the data request to the corresponding client, it is necessary to first submit a receive work request to the client; when the client receives the processing result of the data request returned by the Redis server, the client will store the processing result of the data request in the memory area bound to the receive work request.

[0023] Compared with the prior art, the present invention has the following characteristics:

[0024] 1. In order to integrate high-performance network communication capabilities into the Redis system, the present invention registers the RDMA connection type in its connection layer framework, enabling the Redis system to support the RoCEv2 protocol;

[0025] 2. Considering the versatility of the system and to ensure that Redis can operate normally on traditional Ethernet networks, this paper proposes an alternative communication module that can ensure that the optimized Redis system supports not only RDMA-based network communication but also traditional Socket-based network communication.

[0026] 3. For concurrent scenarios, the present invention proposes a dynamic memory area allocation solution, which improves the system throughput and system operation efficiency by registering a buffer pool on the Redis server side. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is an architectural diagram of the improved Redis system of the present invention.

[0028] Figure 2 This is a schematic diagram of the Redis system buffer pool.

[0029] Figure 3 Schematic diagram of the Redis connection layer framework after RDMA connection is added to the Redis system.

[0030] Figure 4 This is a schematic diagram of the event callback of the improved Redis system of the present invention.

[0031] Figure 5 The data request format diagram after redesign for the Redis system. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific examples and the accompanying drawings.

[0033] A Redis system network data transmission method based on RDMA includes the following steps:

[0034] (1) Redis server initialization phase.

[0035] 1) Load the communication module in the Redis server

[0036] Considering that the interconnection provided by traditional Ethernet is lossy and best-effort, efficient interconnection of computing nodes requires a large amount of transaction acceleration in order to avoid software overhead and latency, and Ethernet switch equipment must be enhanced to reduce latency. RDMA (Remote Direct Memory Access) operations allow a machine to read or write from a pre-registered memory area of ​​another machine without involving the CPU on the remote side. Compared with traditional message passing, RDMA achieves minimal round-trip latency (microseconds), the highest throughput and the lowest CPU overhead. Therefore, the present invention combines RDMA into the Redis system to greatly improve the online processing rate and reduce data-intensive workloads. At the same time, the RoCE (RDMA over ConvergedEthernet) protocol supported by RDMA is an extension technology that allows RDMA hardware to run on the bottom layer of the Ethernet link, making RDMA high-performance hardware compatible with traditional Ethernet. The present invention will also have good adaptability by introducing it into the Redis system. Figure 1 shown.

[0037] The communication module framework introduced in this paper consists of two components: an RDMA-based network communication component and a traditional Socket-based network communication component. These two components can be switched based on specific network scenarios. Switching between these components is achieved by modifying the Redis configuration file and the redisServer structure. Specifically, a communication_mode parameter is added to the Redis configuration file as a selection parameter for the communication module, and a member variable is_rdma is added to the redisServer structure as a judgment parameter for the communication module.

[0038] If no specific configuration file is specified, the parameters of the Redis server will use the default values. If a specific configuration file is specified when starting the Redis server instance, the Redis server instance will use the parameters set in the configuration file during initialization. At this time: If the Redis server reads the value of the communication_mode parameter in the configuration file as rdma, the member variable is_rdma in the global variable server of the redisServer structure type is set to 1, and the Redis server instance will choose to run the RDMA-based network communication component in subsequent operations. If the Redis server reads the value of the communication_mode parameter in the configuration file as tcp or is empty, the member variable is_rdma in the global variable server of the redisServer structure type is set to 0, and the Redis server instance will choose to run the traditional Socket-based network communication component in subsequent operations.

[0039] 2) Initialize the buffer pool of the Redis server

[0040] During the RDMA-based Redis system network communication process, a memory area needs to be allocated in advance and registered with the host channel adapter (HCA). If the memory area is too small or too large, it will affect the use of the application itself. Therefore, the present invention proposes a dynamic memory area allocation method.

[0041] like Figure 2 As shown in the figure, the Redis server maintains a registered buffer pool. When the Redis server starts, it allocates a contiguous memory area, which is registered on the HCA. We call this registered memory area Memory Blocks. Memory Blocks are divided into N blocks, each of which is further divided into two parts: a receive block (Recv Block) and a send block (Send Block). The Recv Block is used to place client requests, while the Send Block serves as a buffer for server command processing results.

[0042] Each Block will be assigned to a client. When a client initiates a connection request to the server, the Redis server will allocate a Block to the client when establishing a connection with it. When other clients also initiate a connection request to the server, the Redis server will allocate a Block to the client when establishing a connection with it. This process continues in this way until all the Blocks in the Memory Blocks are allocated. At this time, if the server wants to process the connection of the N+1th client, the server needs to allocate another continuous Memory Blocks containing N Blocks. The present invention adopts this dynamic buffer allocation method to avoid memory waste caused by allocating too much memory at one time, effectively saving memory space, and also avoiding the reduction of system operating efficiency due to too little memory being allocated.

[0043] 3) Register the RDMA connection type in the connection layer framework of the Redis system

[0044] like Figure 3 As shown in the figure, the business layer in the Redis system accesses the various connection types (TCP, Unix, TLS or RDMA) of the Redis system through the connection layer framework. At the same time, the connection layer framework is also responsible for managing various connection types. For example, before a new connection type is used, it needs to be registered with the Redis system connection layer framework. The current connection management of the Redis system is managed around the connection object, which is similar to the management of sockets. It abstracts a connection layer framework that is higher than the socket to be compatible with different transport layers. Registering the RDMA connection type in the Redis system connection layer framework means implementing the function interface of the various callback functions for the unified connection type provided in the Redis system connection layer framework, such as Figure 4 As shown, the dotted box part is the function interface to be implemented.

[0045] (2) Redis system data transmission stage.

[0046] 1) When the nth client sends a connection request to the Redis server, the Redis server receives the connection request, establishes a connection with the nth client, creates a new file descriptor for network communication, and allocates a block to the client. Since blocks are private to clients, the block allocated to the nth client is called Block n.

[0047] 2) After the connection between the nth client and the Redis server is established, the nth client adds a message header and a tail marker to the data request to be sent, obtains a data request based on the Redis command format, and uses the RDMA Write operation to send the data request based on the Redis command format to the Redis server.

[0048] Data requests based on the Redis command format are as follows Figure 5 As shown in the figure, it mainly consists of three parts: the message header, the data request (Command), and the tail marker. The message header consists of the data request length (Length) and the unique identifier (ID) of the data request. The data request consists of the data request (Command), and the tail marker consists of the unique identifier (ID) of the data request. The unique identifier of the data request is incremented. The Redis server records the unique identifier of the previous data request. The unique identifier of each data request received by the Redis server is always greater than the unique identifier of the previous data request.

[0049] 3) Since the RDMAWrite operation is a one-side operation, the Redis server does not participate in the transmission process of the data request. Therefore, in order to determine whether a data request has arrived, the Redis server needs to continuously poll the Recv Block area of ​​the corresponding client's block to confirm that it has received the data request sent by the client based on the Redis command format.

[0050] When the Redis server finds that the Recv Block of Block n of the nth client is no longer empty, the Redis server determines whether the unique identifier of the message header and the unique identifier of the tail flag in the data request based on the Redis command format stored in the Recv Block of Block n are consistent, and whether the unique identifier is greater than the unique identifier of the previous data request recorded by the Redis server: If the unique identifier of the message header and the unique identifier of the tail flag are consistent, and the unique identifier is greater than the unique identifier of the previous data request recorded by the Redis server, the data request (Command) in the data request based on the Redis command format is extracted; otherwise, the Redis server continues to poll the receiving block of the private block of the client connected to it. By adding a tail flag to the end of the data request, the Recv Block area can be directly rewritten, and there is no need to clear the area again after extracting the request, and there is no need to worry that the data request polled by the Redis server has already been processed.

[0051] 4) Because the processing results of data requests must be stored in a pre-registered memory area before they can be returned to the corresponding client via RDMA technology, the Redis server calls the original Redis request command handler to process the extracted data request and stores the processing results in the Send Block of the private block of client n, Block n.

[0052] 5) The Redis server uses the Send operation to return the data request processing result in the Send Block area of ​​Block n to the nth client.

[0053] If the message size of the Redis server's processing result exceeds the maximum value for the message transmission path, the result will fail to be sent. This is primarily because the RDMA transmission type does not support grouping, and the size of data that can be sent in a single data transmission cannot exceed the maximum value. Therefore, to ensure that processing results larger than the maximum size can be successfully returned to the client, the Redis server should first group the processing results and return them to the client using the Send operation. Each group is assigned a sequence number and an end flag is added to the end of each group. A false end flag indicates that the group transmission has not yet completed, while a true end flag indicates that the group is the last. After receiving all the groups, the client processes and integrates them according to the sequence number and end flag to obtain the complete and correct processing result for the data request.

[0054] In order for the client to receive the processing results of its data request, the Redis server must submit a receive work request before the client sends its request. Only then, when the Redis server returns the processing results, will the QP in the client's HCA receive the processing results and place them in the memory area bound to the receive work request.

[0055] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention. Therefore, the present invention is not limited to the above-mentioned specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A Redis system network data transmission method based on RDMA, characterized in that: The steps are as follows: (1) Redis server initialization phase: Step 1.

1. Load the communication module in the Redis server. This communication module consists of an RDMA-based network communication component and a traditional Socket-based network communication component. These two components are switched based on specific network scenarios by adding selection parameters in the Redis configuration file and judgment parameters in the redisServer structure. Step 1.2: When the Redis server starts, it allocates a continuous memory area, which is divided into N blocks. Each block includes a receiving block and a sending block. N is a set value. Step 1.

3. Register the RDMA connection type in the connection layer framework of the Redis system. (2) Redis system data transmission stage: Step 2.1: When the client sends a connection request to the Redis server, the Redis server receives the connection request and establishes a connection with the client, creates a new file descriptor for network communication, and allocates a block to the client as the client's private block; Step 2.2: The client adds a message header and a tail flag to the data request to obtain a data request in the Redis command format. The client then uses the RDMA Write operation to send the data request in the Redis command format to the Redis server. Step 2.3: The Redis server continuously polls the receiving blocks of the private blocks of the clients connected to it. When it finds that the receiving block of a client's private block is no longer empty, the Redis server determines whether the unique identifier of the message header and the unique identifier of the tail flag in the data request based on the Redis command format stored therein are consistent, and whether the unique identifier is greater than the unique identifier of the previous data request recorded by the Redis server. If they are consistent and greater, the data request in the data request based on the Redis command format is extracted; otherwise, the Redis server continues to poll the receiving blocks of the private blocks of the clients connected to it. Step 2.4: The Redis server calls the original Redis request command handler to process the extracted data request and stores the processing result of the data request in the sending block of the private block of the corresponding client; Step 2.5: The Redis server uses the Send operation to return the processing result of the data request stored in the sending block of the client's private block to the corresponding client.

2. A Redis system network data transmission method based on RDMA according to claim 1, characterized in that: In step 1.1, add a communication_mode parameter as a selection parameter in the Redis configuration file, and add a member variable is_rdma as a judgment parameter in the redisServer structure; If the Redis server reads the value of the communication_mode parameter in the Redis configuration file as rdma, it sets the member variable is_rdma in the global variable server of the redisServer structure type to 1, and the Redis server selects the RDMA-based network communication component; If the Redis server reads the value of the communication_mode parameter in the Redis configuration file and it is tcp or empty, the member variable is_rdma in the global variable server of the redisServer structure type is set to 0, and the Redis server runs the traditional network communication component based on Socket.

3. A Redis system network data transmission method based on RDMA according to claim 1, characterized in that: In step 2.1, When the first Redis client initiates a connection request to the server, the Redis server will allocate the first block of the memory area to the client when establishing a connection with it; When the second client initiates a connection request to the Redis server, the Redis server will allocate the second block of the memory area to the client when establishing a connection with it; And so on, until all N blocks in the memory area allocated by the Redis server are allocated. At this time, when the N+1th client initiates a connection request to the Redis server, the Redis server allocates another continuous memory area containing N blocks.

4. A Redis system network data transmission method based on RDMA according to claim 1, characterized in that: In step 2.2, the data request based on the Redis command format consists of three parts: a message header, a data request, and a tail flag. The message header part consists of the length of the data request and the unique identifier of the data request, the data request part consists of the data request, and the tail flag part consists of the unique identifier of the data request.

5. The method for transmitting network data of a Redis system based on RDMA according to claim 1, wherein: In step 2.5, when the processing result of the data request sent by the Redis server is greater than the maximum value on the message transmission path, the Redis server first groups the processing results of the data request, sets a sequence number for each group, and adds an end flag at the end of each group. Then, the Redis server returns the groups to the client in sequence using the Send operation; After receiving all the packets, the client processes and integrates all the packets according to the sequence number and end flag of the packets to obtain the complete and correct processing results of the data request.

6. A Redis system network data transmission method based on RDMA according to claim 1, characterized in that: In step 2.5, before the Redis server returns the processing result of the data request to the corresponding client, it needs to submit a receive work request to the client in advance; when the client receives the processing result of the data request returned by the Redis server, the client will store the processing result of the data request in the memory area bound to the receive work request.

Images