RDMA message caching method based on programmable switch

Abstract

The invention relates to the technical field of data center network communication, and provides an RDMA (Remote Direct Memory Access) message caching method based on a programmable switch, which comprises the following steps: in a control plane, a first DCI (Downlink Control Information) switch establishes connection with an RDMA server and completes initialization of the first DCI switch; when a first DCI switch receives an RDMA message sent by a second DCI switch, the RDMA message is matched and screened through a flow table on a data plane, a corresponding mirror image type is set, hash flow characteristics are calculated based on an IP address and QP information of the RDMA message, and the hash flow characteristics and the mirror image type of the RDMA message are transmitted to an Egress pipeline; and in an Egress pipeline, combining the hash flow feature with the PSN of the RDMA message to further generate a first hash feature and a second hash feature of the message, calculating a storage address of the RDMA message by using the first hash feature and the second hash feature, adding header information, and sending the RDMA message to an RDMA server to realize storage of the RDMA message. Through interaction between the DCI switch and the RDMA server, active caching of the data is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data center network communication, and in particular relates to an RDMA message caching method based on a programmable switch. Background Art

[0002] Due to its advantages of low latency and high bandwidth data transmission, RDMA (Remote Direct Memory Access) technology has been widely used in scenarios such as high-performance computing (HPC), distributed storage, and data center networks. Through the RDMA protocol, data can be transferred between applications with zero copy, significantly reducing CPU usage and improving data throughput.

[0003] In the prior art, when data packet errors or loss occur in RDMA data transmission, the GBN (Go-Back-N) solution will trigger retransmission only after the receiving end detects packet loss and feeds back a retransmission request, so the response efficiency is low. Although the RDMA selective retransmission technology has been implemented on the RNIC network card, it can reduce the bandwidth waste and delay of the GBN algorithm, but this method requires at least one RTT (Round Trip Time) before retransmitting the lost message, resulting in a large transmission delay, especially in long-distance wide area networks. Summary of the invention

[0004] In view of the above-mentioned defects of the prior art, the present invention proposes an RDMA message caching method based on a programmable switch. The technical solution steps designed by the present invention include:

[0005] S10: On the control plane, the first DCI switch establishes a connection with the RDMA server and completes initialization of the first DCI switch;

[0006] S20: When the first DCI switch receives the RDMA message sent by the second DCI switch, it filters the RDMA message through flow table matching in the data plane and sets the corresponding mirror type, calculates the hash flow feature based on the IP address and QP information of the RDMA message, and passes the hash flow feature and the mirror type of the RDMA message to the Egress pipeline;

[0007] S30: In the Egress pipeline, the hash flow feature and the PSN of the RDMA message are combined to further generate the first hash feature and the second hash feature of the message, and the storage address of the RDMA message is calculated using the first hash feature and the second hash feature. After adding the header information, it is sent to the RDMA server to realize the storage of the RDMA message.

[0008] Preferably, the S10 includes:

[0009] S101: The first DCI switch initializes the target address and port configuration of the RDMA connection, generates an attribute structure for the connection, and after confirming that the target connection address is available, creates a connection endpoint and establishes a connection with the target RDMA server through RDMACM;

[0010] S102: After the RDMA connection is established, the first DCI switch creates a TCP Socket to listen to the connection request from the RDMA network card, receives metadata information from the RDMA server, and executes flow table delivery and register initialization according to the data plane logic.

[0011] Preferably, the metadata information of the RDMA server includes but is not limited to a memory address and rKey.

[0012] Preferably, the S10 further includes:

[0013] The RDMA server closes the ICRC field of the RDMA network card and initializes the connection configuration, sets the listening port, and establishes an RDMA endpoint to wait for a connection request from the first DCI switch.

[0014] Preferably, the S10 further includes:

[0015] The RDMA server allocates storage space for the data cache and registers the memory area. After completing the memory registration, the RDMA server sends the server's RDMA metadata to the client by creating a TCP connection.

[0016] Preferably, the S20 includes:

[0017] S201: When the RDMA message transmitted by the second DCI switch enters the first DCI switch, the field information is used as a matching condition through the flow table entry in the Ingress Control stage to identify the message to be cached, and set the mirror type of the corresponding metadata;

[0018] S202: Match the flow table according to the mirror type. If the match is successful, the hash flow feature value of the RDMA message is calculated and stored based on the custom CRC polynomial and combined with the source IP, destination IP and queue pair QP information, and the hash flow feature value is written into the metadata. The message enters the Traffic Manager to perform the mirroring operation and is passed to the Egress pipeline to provide input for subsequent address calculation.

[0019] Preferably, the field information in S2001 includes but is not limited to an operation code, a port number and a packet loss flag.

[0020] Preferably, the mirror type in S2001 includes but is not limited to SWITCH_MIRROR_TYPE_R DMA_LOST, PSN and Destination QP number.

[0021] Preferably, the S30 includes:

[0022] S301: In the Egress pipeline, according to the hash stream feature value and PSN of the RDMA message, the first hash feature value is first calculated and stored using the CRC32 algorithm, and then the second hash feature value is calculated and stored using the custom CRC polynomial;

[0023] S302: performing an XOR operation on the first hash feature value and the second hash feature value to synthesize a hash feature value, and clearing the lower 12 bits of the hash feature value to zero, and retaining the upper 20 bits as the storage address of the RDMA message;

[0024] S303: Add the header information of the operation code, PSN, virtual address, rKey and target QP number, and construct Ethernet and IP header information according to the source and target IP addresses and MAC addresses;

[0025] S304: Combine the constructed header information into a final RDMA WRITE message and send it to the RDMA server to implement storage of the RDMA message.

[0026] Preferably, the S30 further includes:

[0027] The RDMA server continuously waits for and receives RDMA messages, verifies whether the storage address and access key contained in the message are consistent with the local configuration, and after the verification is passed, the RDMA server writes the message data to the specified storage address to achieve caching of the RDMA message.

[0028] Beneficial effects:

[0029] 1. This application manages the connection between the switch and the RDMA server through the control plane, including applying for resources, configuring the target address, setting the listening port and creating a TCP connection, etc., to ensure normal RDMA data transmission between the DCI switch and the RDMA server;

[0030] 2. This application is based on the multi-level hash processing of the CRC algorithm and RDMA message features. By combining the hash stream feature value of the message with the PSN number to generate the first and second hash feature values, and performing an XOR operation to obtain the synthetic hash feature, the storage address is then extracted from it, which effectively disperses the data storage location and reduces the possibility of storage address conflicts;

[0031] 3. This application utilizes the Write operation of RDMA, and the network card hardware directly writes the data to the corresponding cache address without CPU intervention, so as to support 100Gbps / 400Gbps line speed transmission (which is difficult to achieve in the traditional TCP solution) and optimize the large-scale high-speed data transmission performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of a preferred embodiment of the present invention;

[0033] Figure 2 is a DCI switch processing flow chart of a preferred embodiment of the present invention;

[0034] Figure 3 This is a RDMA server processing flow chart of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0035] The embodiments of the present invention are described in detail below. The following embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the following embodiments.

[0036] The present invention designs a RDMA message caching method based on a programmable switch, which enables the DCI switch to actively cache the RDMA messages transmitted by other DCI switches so as to retransmit them when packet loss occurs, thereby significantly improving the reliability of cross-data center communication and data transmission efficiency. The technical solution includes the following steps, such as Figure 1 As shown, specifically including:

[0037] S10: On the control plane, the first DCI switch establishes a connection with the RDMA server and completes initialization of the first DCI switch;

[0038] S20: When the first DCI switch receives the RDMA message sent by the second DCI switch, it filters the RDMA message through flow table matching in the data plane and sets the corresponding mirror type, calculates the hash flow feature based on the IP address and QP information of the RDMA message, and passes the hash flow feature and the mirror type of the RDMA message to the Egress pipeline;

[0039] S30: In the Egress pipeline, the hash flow feature and the PSN of the RDMA message are combined to further generate the first hash feature and the second hash feature of the message, and the storage address of the RDMA message is calculated using the first hash feature and the second hash feature. After adding the header information, it is sent to the RDMA server to realize the storage of the RDMA message.

[0040] Preferably, if Figure 2As shown, S10 includes:

[0041] S101: The first DCI switch initializes the target address and port configuration of the RDMA connection, generates an attribute structure for the connection, and after confirming that the target connection address is available, creates a connection endpoint and establishes a connection with the target RDMA server through RDMACM;

[0042] S102: After the RDMA connection is established, the first DCI switch creates a TCP Socket to listen to the connection request from the RDMA network card, receives metadata information from the RDMA server, and executes flow table delivery and register initialization according to the data plane logic.

[0043] Preferably, the metadata information of the RDMA server includes but is not limited to a memory address and rKey.

[0044] Preferably, if Figure 3 As shown, S10 also includes:

[0045] The RDMA server closes the ICRC field of the RDMA network card and initializes the connection configuration, sets the listening port, and establishes an RDMA endpoint to wait for a connection request from the first DCI switch.

[0046] Preferably, S10 further includes:

[0047] The RDMA server allocates storage space for the data cache and registers the memory area. After completing the memory registration, the RDMA server sends the server's RDMA metadata to the client by creating a TCP connection.

[0048] Specifically, when the RDMA server fails to wait for the connection request from the first DCI switch, error information is printed and related resources are released; when the RDMA server fails to send RDMA metadata to the client, error information is printed and related resources are released.

[0049] Preferably, if Figure 2 As shown, S20 includes:

[0050] S201: When the RDMA message transmitted by the second DCI switch enters the first DCI switch, the field information is used as a matching condition through the flow table entry in the Ingress Control stage to identify the message to be cached, and set the mirror type of the corresponding metadata;

[0051] S202: Match the flow table according to the mirror type. If the match is successful, the hash flow feature value of the RDMA message is calculated and stored based on the custom CRC polynomial and combined with the source IP, destination IP and queue pair QP information, and the hash flow feature value is written into the metadata. The message enters the Traffic Manager to perform the mirroring operation and is passed to the Egress pipeline to provide input for subsequent address calculation.

[0052] Preferably, the field information in S2001 includes but is not limited to an operation code, a port number, and a packet loss flag.

[0053] Preferably, the mirror type in S2001 includes but is not limited to SWITCH_MIRROR_TYPE_RDMA_LOST, PSN and Destination QP number.

[0054] Preferably, if Figure 2 As shown, S30 includes:

[0055] S301: In the Egress pipeline, according to the hash stream feature value and PSN of the RDMA message, the first hash feature value is first calculated and stored using the CRC32 algorithm, and then the second hash feature value is calculated and stored using the custom CRC polynomial;

[0056] S302: performing an XOR operation on the first hash feature value and the second hash feature value to synthesize a hash feature value, and clearing the lower 12 bits of the hash feature value to zero, and retaining the upper 20 bits as the storage address of the RDMA message;

[0057] S303: Add the header information of the operation code, PSN, virtual address, rKey and target QP number, and construct Ethernet and IP header information according to the source and target IP addresses and MAC addresses;

[0058] S304: Combine the constructed header information into a final RDMA WRITE message and send it to the RDMA server to implement storage of the RDMA message.

[0059] Specifically, the custom CRC polynomial is 32w0X04C11DB7.

[0060] Preferably, if Figure 3 As shown, S30 also includes:

[0061] The RDMA server continuously waits for and receives RDMA messages, verifies whether the storage address and access key contained in the message are consistent with the local configuration, and after the verification is passed, the RDMA server writes the message data to the specified storage address to achieve RDMA message caching.

[0062] Specifically, when the verification message is wrong, the message is discarded.

[0063] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A RDMA message caching method based on a programmable switch, characterized in that: include: S10: On the control plane, the first DCI switch establishes a connection with the RDMA server and completes initialization of the first DCI switch; S20: When the first DCI switch receives the RDMA message sent by the second DCI switch, it filters the RDMA message through flow table matching in the data plane and sets the corresponding mirror type, calculates the hash flow feature based on the IP address and QP information of the RDMA message, and passes the hash flow feature and the mirror type of the RDMA message to the Egress pipeline; S30: In the Egress pipeline, the hash flow feature and the PSN of the RDMA message are combined to further generate the first hash feature and the second hash feature of the message, and the storage address of the RDMA message is calculated using the first hash feature and the second hash feature. After adding the header information, it is sent to the RDMA server to realize the storage of the RDMA message.

2. According to the RDMA message caching method based on a programmable switch according to claim 1, it is characterized in that: The S10 includes: S101: The first DCI switch initializes the target address and port configuration of the RDMA connection, generates an attribute structure for the connection, and after confirming that the target connection address is available, creates a connection endpoint and establishes a connection with the target RDMA server through RDMACM; S102: After the RDMA connection is established, the first DCI switch creates a TCP Socket to listen to the connection request from the RDMA network card, receives metadata information from the RDMA server, and executes flow table delivery and register initialization according to the data plane logic.

3. The RDMA message caching method based on a programmable switch according to claim 2, characterized in that: The metadata information of the RDMA server includes but is not limited to a memory address and an rKey.

4. The RDMA message caching method based on a programmable switch according to claim 2, characterized in that: The S10 further includes: The RDMA server closes the ICRC field of the RDMA network card and initializes the connection configuration, sets the listening port, and establishes an RDMA endpoint to wait for a connection request from the first DCI switch.

5. The RDMA message caching method based on a programmable switch according to claim 2, characterized in that: The S10 further includes: The RDMA server allocates storage space for the data cache and registers the memory area. After completing the memory registration, the RDMA server sends the server's RDMA metadata to the client by creating a TCP connection.

6. The RDMA message caching method based on a programmable switch according to claim 1, characterized in that: The S20 includes: S201: When the RDMA message transmitted by the second DCI switch enters the first DCI switch, the field information is used as a matching condition through the flow table entry in the Ingress Control stage to identify the message to be cached, and set the mirror type of the corresponding metadata; S202: Match the flow table according to the mirror type. If the match is successful, the hash flow feature value of the RDMA message is calculated and stored based on the custom CRC polynomial and combined with the source IP, destination IP and queue pair QP information, and the hash flow feature value is written into the metadata. The message enters the Traffic Manager to perform the mirroring operation and is passed to the Egress pipeline to provide input for subsequent address calculation.

7. The RDMA message caching method based on a programmable switch according to claim 1, characterized in that: The field information in S201 includes but is not limited to an operation code, a port number, and a packet loss flag.

8. The RDMA message caching method based on a programmable switch according to claim 1, characterized in that: The mirror type in S201 includes but is not limited to SWITCH_MIRROR_TYPE_RDMA_LOST, PSN and DestinationQP number.

9. The RDMA message caching method based on a programmable switch according to claim 1, characterized in that: The S30 includes: S301: In the Egress pipeline, according to the hash stream feature value and PSN of the RDMA message, the first hash feature value is first calculated and stored using the CRC32 algorithm, and then the second hash feature value is calculated and stored using the custom CRC polynomial; S302: performing an XOR operation on the first hash feature value and the second hash feature value to synthesize a hash feature value, and clearing the lower 12 bits of the hash feature value to zero, and retaining the upper 20 bits as the storage address of the RDMA message; S303: Add the header information of the operation code, PSN, virtual address, rKey and target QP number, and construct Ethernet and IP header information according to the source and target IP addresses and MAC addresses; S304: Combine the constructed header information into a final RDMA WRITE message and send it to the RDMA server to implement storage of the RDMA message.

10. The RDMA message caching method based on a programmable switch according to claim 9, characterized in that: The S30 further includes: The RDMA server continuously waits for and receives RDMA messages, verifies whether the storage address and access key contained in the message are consistent with the local configuration, and after the verification is passed, the RDMA server writes the message data to the specified storage address to achieve caching of the RDMA message.

Images