RDMA-based communication device and system

Abstract

The present application provides a communication device and system based on RDMA, which is applied to the request side of RDMA, including: a first conversion module, which is used to convert the read work request into a first RDMA write WQE when receiving a read work request; a first network interface card, which is used to convert the first RDMA write WQE into a first RDMA write request, and is also used to receive and parse the second RDMA write request sent by the RDMA response side in response to the first RDMA write request, obtain the request data corresponding to the read work request, write the request data into a storage address and generate a CQE; the first conversion module is also used to generate a WC according to the CQE, and send the WC. The present application implements the read operation through two RDMA write requests, and there is no need to generate a response, so there is no need to set CRRL and SRRL on the request side and the response side, saving memory resources.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a communication device and system based on RDMA. Background Art

[0002] In the related technology, when a Remote Direct Memory Access (RDMA) system communicates, the requesting side needs to generate a client-side RDMA read list (Client RDMA Read List, CRRL), and the responding side needs to generate a server-side RDMA read list (Service RDMA Read List, SRRL). When the responding side receives the RDMA read request sent by the requesting side, it generates read response information in the SRRL. The sending unit of the responding side can generate a read response according to the read response information in the SRRL, and the requesting side will receive the read response according to the information in the CRRL and perform corresponding operations.

[0003] Therefore, each queue pair (QP) on the request side needs to have a CRRL data structure; each QP on the response side needs to have an SRRL data structure. For example, if the entry of each CRRL is 511 bits and the depth is 118, and the entry of each SRRL is 256 bits and the depth is 118, if 1M QPs are supported, the required memory resources are 11 GB, which has a high memory usage and consumes a lot of memory resources. Summary of the invention

[0004] The present application provides a communication device and system based on RDMA, which is helpful to solve the above-mentioned technical problem of high memory usage.

[0005] To solve the above problems, the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a communication device based on RDMA, which is applied to the request side of RDMA, and the device includes: a first conversion module and a first network interface card; wherein,

[0007] The first conversion module is configured to, upon receiving a read work request, convert the read work request into a first RDMA write WQE, wherein the first RDMA write WQE at least includes a read address and a storage address of request data corresponding to the read work request;

[0008] The first network interface card is used to receive the first RDMA write WQE, convert the first RDMA write WQE into a first RDMA write request, and send the first RDMA write request;

[0009] The first network interface card is further configured to receive and parse a second RDMA write request sent by the RDMA response side in response to the first RDMA write request, obtain request data corresponding to the read work request, write the request data into the storage address, and generate a CQE;

[0010] The first conversion module is further configured to generate a WC according to the CQE and send the WC.

[0011] In some implementations, the first conversion module is specifically configured to:

[0012] The read work request information is used as the inline data of the WQE, the CQE identifier of the WQE is set to generate a CQE, the opcode of the WQE is set to a write request, and the type of the CQE in the WQE is set to a read-modify-write type, the first RDMA write WQE is generated, and the first RDMA write WQE is written into the first SQ cache unit.

[0013] In some embodiments, the first network interface card includes at least a first transmitting unit, a first receiving unit, and a first data cache unit;

[0014] The first transmitting unit is used to read the first RDMA write WQE, convert the first RDMA write WQE into a first RDMA write request, and send the first RDMA write request;

[0015] The first receiving unit is used to receive first ACK information sent by the RDMA response side, and generate a first CQE according to the first ACK information, wherein the first ACK information is used to indicate that the RDMA response side has received the first RDMA write request;

[0016] The first receiving unit is further used to receive and parse the second RDMA write request sent by the RDMA response side, obtain request data corresponding to the read work request, write the request data into the first data cache unit, and generate a second CQE;

[0017] The first conversion module is further configured to generate the WC according to the first CQE and the second CQE, and send the WC to the read work request sending end.

[0018] In some implementations, the first receiving unit is specifically configured to:

[0019] The first ACK information is received, a first RDMA write WQE corresponding to the first ACK information is read according to the PSN of the first ACK information, and a first CQE is generated according to the first RDMA write WQE, wherein the first CQE includes identification setting information of the first RDMA write WQE.

[0020] In some implementations, the first conversion module further includes:

[0021] A first CQE buffer unit, configured to buffer a first CQE generated by the first receiving unit;

[0022] A second CQE buffering unit, configured to buffer a second CQE generated by the first receiving unit;

[0023] A CQE processing unit is used to read the first CQE in the first CQE cache unit and read the second CQE in the second CQE cache unit. When the identification setting information of the first CQE is a read-modify-write type, if there is a second CQE corresponding to the first CQE in the second CQE cache unit, the WC is generated and sent to the read work request sending end.

[0024] In a second aspect, an embodiment of the present application provides an RDMA-based communication device, which is applied to an RDMA response side, and the device includes: a second conversion module and a second network interface card; wherein,

[0025] The second network interface card is used to receive and parse the first RDMA write request, and obtain payload information corresponding to the first RDMA write request, wherein the first RDMA write request is a write request converted from a read work request, and the payload information includes a read address of request data corresponding to the read work request and a storage address on the request side;

[0026] The second conversion module is used to read the payload information and generate a second RDMA write WQE based on the payload information;

[0027] The second network interface card is used to receive the second RDMA write WQE and read the request data according to the read address; it is also used to generate a second RDMA write request according to the request data and the second RDMA write WQE, and send the second RDMA write request to the request side.

[0028] In some implementations, the second conversion module is specifically configured to:

[0029] Read the payload information of the first RDMA write request, obtain the read address, data length and storage address of the request side of the request data corresponding to the read work request, determine the data length as the write data length of the second RDMA write WQE, use the storage address of the request side as the storage address of the second RDMA write WQE on the request side, set the operation code in the second RDMA write WQE to an RDMA write request with an immediate number, thereby generating the second RDMA write WQE.

[0030] In some implementations, the second network interface card further includes: an RDMA write WQE cache unit;

[0031] The RDMA write WQE cache unit is used to cache payload information;

[0032] The second conversion unit is used to query whether new payload information is written into the payload information cache unit, and read the payload information when it is queried that new payload information is written.

[0033] In some embodiments, the second network interface card includes: a second transmitting unit and a second receiving unit;

[0034] The second receiving unit is configured to send a doorbell to the second transmitting unit after receiving the first RDMA write request;

[0035] The second transmitting unit is used to send a first ACK to the first receiving unit of the requesting side when receiving the doorbell; wherein the first ACK information is used to indicate that the RDMA responding side has received the first RDMA write request;

[0036] The second transmitting unit is further used to receive the second RDMA write WQE, read the request data according to the read address, generate a second RDMA write request based on the request data and the second RDMA write WQE, and send the second RDMA write request to the requesting side.

[0037] In a third aspect, an embodiment of the present application provides an RDMA-based communication system, the system comprising: an RDMA-based communication device as in the first aspect and an RDMA-based communication device as in the second aspect.

[0038] The present application realizes the function of RDMA read request by converting the read work request into the first RDMA write request on the request side, and generating and sending the second RDMA write request in response to the first RDMA write request on the response side. Since the RDMA write request only generates Ack but not a response, the request side and the response side no longer need to set CRRL and SRRL, saving memory resources.

[0039] In addition, since read-modify-write is performed, there is no need to process the fields related to RDMA read and read response, so the fields related to RDMA read and read response in QPC can be removed, reducing the complexity of QPC.

[0040] Furthermore, because the read-modify-write operation is performed, the PSN is continuous, so there is no need to use (MSN,

[0041] IPSN) to determine whether the message is received, which is more conducive to hardware implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of an RDMA-based communication device on the request side provided in an embodiment of the present application.

[0043] Figure 2 It is a more detailed schematic diagram of an RDMA-based communication device on the request side provided in an embodiment of the present application.

[0044] Figure 3 It is a schematic diagram of an RDMA-based communication device on the response side provided in an embodiment of the present application.

[0045] Figure 4 It is a more detailed schematic diagram of an RDMA-based communication device on the response side provided in an embodiment of the present application.

[0046] Figure 5 It is a schematic diagram of an RDMA-based communication system provided in an embodiment of the present application.

[0047] Figure 6 It is a more detailed schematic diagram of the communication system provided in the embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0050] For the convenience of explanation, the Chinese terms corresponding to the English abbreviations in this application are introduced: Remote Direct Memory Access (RDMA), operation code (Opcode), inline data (inline data), send queue (SQ), receive queue (RQ), work queue element (WQE), network interface card (NIC), work request (WR), remote key (RKey), payload, queue pair (QP), and work completion instruction (WC).

[0051] See also Figure 1 , Figure 1 Schematic diagram of an RDMA-based communication device on the request side provided by an embodiment of the present application. Figure 1 As shown, the device is applied to the request side, and the request side device 100 includes: a first conversion module 110 and a first network interface card 120. When the first conversion module 110 receives a read work request, the read work request is converted into a first RDMA write WQE, wherein the first RDMA write WQE at least includes a read address and a storage address of the request data corresponding to the read work request. The first network interface card 120 is used to receive the first RDMA write WQE, convert the first RDMA write WQE into a first RDMA write request, and send the first RDMA write request. The first network interface card 120 is also used to receive and parse the second RDMA write request sent by the RDMA response side in response to the first RDMA write request, obtain the request data corresponding to the read work request, write the request data into the storage address and generate a CQE.

[0052] The request side device 100 can receive a read work request sent by an external device. In RDMA communication, when an external device needs to initiate a data transmission operation, the external device will create one or more work requests (Work Request, WR), which define the specific operation type to be performed, including but not limited to Send, Receive, Read or Write, etc. Each WR contains necessary parameters, such as when the request is a read request, including the read address of the requested data, the storage address of the requested data, the data length of the requested data, etc., to instruct the hardware on the network interface card (Network Interface Card, NIC) how to perform these operations.

[0053] It should be noted that, normally, a read work request will be converted into an RDMA read WQE, but in this application, a read work request will not be converted into an RDMA read WQE, but converted into a first RDMA write WQE to implement an RDMA read request through an RDMA write request.

[0054] Among them, the first network interface card 120 receives the first RDMA write WQE, converts the first RDMA write WQE into a first RDMA write request, and sends the first RDMA write request to the RDMA response side, so that the response side reads the request data based on the read address of the first RDMA write request and then feeds it back to the request side.

[0055] Among them, the first network interface card 120 receives and parses the second RDMA write request sent by the responding side in response to the first RDMA write request, obtains the request data corresponding to the read request, and writes the data into the storage address of the requesting side, and also generates a second CQE, which is used to indicate that the request data of the read request has been read into the first network interface card 120. Specifically, the CQE in RDMA is a data structure generated after the RDMA operation is completed, which is used to notify the external device that sends the work request of the status and result of the RDMA operation. When the RDMA operation, such as SEND, Write, READ, etc., is completed, the NIC will generate a CQE to indicate the completion status of the operation. Furthermore, the CQE can contain the status information of the RDMA operation, such as whether the operation is successful, whether an error occurs, etc.

[0056] The first conversion module is further used to generate a WC according to the second CQE, and send the WC to the external device that sends the read work request, so as to notify the external device that the request data of the read request has been read into the first network interface card and the external device can read it.

[0057] The request side device of the present application only needs to process RDMA write requests during operation, and RDMA write requests only generate Acks but no responses. Therefore, the request side no longer needs to set CRRL, thus saving memory resources.

[0058] In addition, since read-modify-write is performed, there is no need to process the fields related to RDMA read and read response, so the fields related to RDMA read and read response in QPC can be removed, reducing the complexity of QPC.

[0059] Furthermore, because the read-modify-write operation is performed, the PSN is continuous, so there is no need to use (MSN,

[0060] IPSN) to determine whether the message is received, which is more conducive to hardware implementation.

[0061] See also Figure 2 , Figure 2 It is a more detailed schematic diagram of an RDMA-based communication device on the request side provided in an embodiment of the present application.

[0062] like Figure 2 As shown, in a specific implementation, the first conversion module 110 is set in the RDMA driver, and the first conversion module 110 uses the read work request information as the inline data of the WQE, sets the CQE identifier of the WQE to generate the CQE, sets the opcode of the WQE to a write request, and sets the type of the CQE in the WQE to a read-modify-write type, generates a first RDMA write WQE, and writes the first RDMA write WQE into the first SQ cache unit 113. It can be seen that the first conversion module 110 implements the conversion from the read request information to the first RDMA write WQE by software.

[0063] Among them, inline data is a feature of RDMA communication that allows small blocks of data to be directly embedded in control information, such as WQE. After the inline data is embedded in WQE, when WQE is sent to the transmitting unit, the transmitting unit no longer needs to read data from the data cache unit.

[0064] The WQE operation code (Opcode) is used to specify the operation type represented by WQE. The operation code determines what action the hardware on the NIC should perform. Common WQE operation codes include but are not limited to the following: Send, Receive, Read, and Write. In this embodiment, it is necessary to rewrite the read request into a write request, so the Opcode needs to be set to a write request.

[0065] When generating the first RDMA write WQE, the flag of whether to generate CQE is set to generate CQE. At the same time, in order to distinguish the CQE generated by ordinary write requests from the CQE generated by read-modify-write type, a CQE type field is added to the WQE to indicate that the CQE is a CQE generated by converting a read work request into a write work request (i.e., a read-modify-write type CQE), rather than a CQE generated by an ordinary write request. For example, the field cqe_info is added to the WQE, where cqe_info carries at least 1 bit of information. When cqe_info is set to 1, it can be indicated that the CQE is a read-modify-write type CQE.

[0066] The read work request is converted into an RDMA write WQE by the first conversion module 110, and the network interface card in the request side device 100 no longer needs to process the RDMA read WQE.

[0067] In some embodiments, the first network interface card 120 includes at least a first transmitting unit 121, a first receiving unit 122, and a first data cache unit 123. The first transmitting unit 121 is used to read the first RDMA write WQE, convert the first RDMA write WQE into a first RDMA write request, and send the first RDMA write request. The first receiving unit 122 is used to receive the first ACK information sent by the RDMA response side, and generate a first CQE according to the first ACK information, and the first ACK information is used to indicate that the RDMA response side has received the first RDMA write request. The first receiving unit 122 is also used to receive and parse the second RDMA write request sent by the RDMA response side, obtain the request data corresponding to the read work request, write the request data into the storage address of the first data cache unit 123, and generate a second CQE. The first conversion module 110 is also used to generate a WC according to the first CQE and the second CQE, and send the WC to the read work request sending end.

[0068] Specifically, after receiving the first ACK information, the first receiving unit 122 reads the first RDMA write WQE from the first SQ cache unit 113 according to the PSN in the first ACK information. Since the CQE identifier of the first RDMA write WQE is set to generate CQE, the first CQE is generated when the first ACK information is received. At the same time, the type setting information of CQE is placed in the first CQE. When the first conversion module reads the first CQE, it can identify that the first CQE is the CQE generated after the RDMA write WQE converted from the read work request is completed, thereby distinguishing the CQE generated by the RDMA write WQE converted from the write work request and the CQE generated by the RDMA write WQE converted from the read work request.

[0069] Optionally, after receiving the second RDMA write request, the first receiving unit 122 sends a doorbell to the first transmitting unit 121, so that the first transmitting unit 121 feeds back second ACK information to the responding side, and the second ACK information is used to indicate that the RDMA requesting side has received the second RDMA write request.

[0070] In one embodiment, the second RDMA write request is an RDMA write request with an immediate number. According to the RDMA protocol, an RDMA write request with an immediate number needs to generate a CQE. At the same time, the second RDMA write request also carries CQE type information, which is used to indicate that the CQE is a CQE generated by an RDMA write WQE with an immediate number converted from a read request, so as to distinguish it from a CQE generated by an ordinary RDMA write WQE with an immediate number. Specifically, the first receiving unit 122 receives and parses the second RDMA write request, reads the RQ WQE from the RQ cache unit 112 according to the PSN obtained by the parsing, generates a second CQE based on the RQ WQE, carries the type information of the second CQE, and sends the second CQE to the second CQE cache unit 115.

[0071] In one embodiment, the device further includes a first CQE cache unit 114, a second CQE cache unit 115, and a CQE processing unit. The first CQE cache unit 114 is used to cache the first CQE generated by the first receiving unit 122. The second CQE cache unit 115 is used to cache the second CQE generated by the first receiving unit 122. The CQE processing unit 116 is used to read the first CQE in the first CQE cache unit and read the second CQE in the second CQE cache unit. When the identification setting information of the first CQE is a read-modify-write type, if there is a second CQE corresponding to the first CQE in the second CQE cache unit, a WC is generated and sent to the read work request sending end.

[0072] More specifically, when the CQE processing unit 116 reads that the type information of the first CQE is a read-modify-write type, the CQE processing unit stops reading the data in the first CQE cache unit until the second CQE corresponding to the first CQE is read from the second CQE cache unit, generates a WC, and then continues to read the CQE in the first CQE cache unit and continues processing.

[0073] The request side device of the present application only needs to process RDMA write requests during operation, and RDMA write requests only generate Acks but no responses. Therefore, the request side no longer needs to set CRRL, thus saving memory resources.

[0074] In addition, since read-modify-write is performed, there is no need to process the fields related to RDMA read and read response, so the fields related to RDMA read and read response in QPC can be removed, reducing the complexity of QPC.

[0075] Furthermore, because the read-modify-write operation is performed, the PSN is continuous, so there is no need to use (MSN,

[0076] IPSN) to determine whether the message is received, which is more conducive to hardware implementation.

[0077] See also Figure 3 , Figure 3 Schematic diagram of an RDMA-based communication device on the response side provided in an embodiment of the present application. Figure 3 As shown, Figure 3 The responding side device 200 includes: a second conversion module 210 and a second network interface card 220. The second conversion module 210 is used to read the payload information and generate a second RDMA write WQE based on the payload information. The second network interface card 220 is used to receive the second RDMA write WQE, read the request data according to the read address, and also to generate a second RDMA write request according to the request data and the second RDMA write WQE, and send the second RDMA write request to the requesting side.

[0078] Please refer to Figure 4 , Figure 4 It is a more detailed schematic diagram of an RDMA-based communication device on the response side provided in an embodiment of the present application.

[0079] like Figure 4 As shown, specifically, the second network interface card 220 includes an RDMA write WQE cache unit 223, which is used to cache the payload information. The RDMA write WQE cache unit 223 is a storage unit dedicated to caching the payload information of the first RDMA write request. When the second network interface card parses the payload information and the corresponding storage address information in the first RDMA write request, it can be known that the first RDMA write request is a write request converted from a read work request according to the storage address stored in the RDMA write WQE cache unit, and thus the payload obtained by parsing is stored at the corresponding address in the RDMA write WQE cache unit.

[0080] Specifically, the payload information includes a read address of request data corresponding to the read work request and a storage address on the request side. The payload information corresponding to the first RDMA write request is the inline data in the first RDMA write WQE.

[0081] In some embodiments, the second conversion module is arranged in the driver of the response device, and the second conversion module 210 is used to read the payload information of the first RDMA write request, obtain the read address, data length and storage address of the request data corresponding to the read work request according to the payload information, determine the data length as the write data length of the second RDMA write WQE, use the storage address of the request side as the storage address of the second RDMA write WQE on the request side, set the operation code in the second RDMA write WQE to the RDMA write request with an immediate number, thereby generating a second RDMA write WQE, and store the second RDMA write WQE in the second SQ cache unit 212.

[0082] Optionally, the second conversion module 210 uses a polling method to query whether there is new payload information written in the RDMA write WQE cache unit, and when it is queried that there is new payload information written, read the payload information, determine the data length in the payload information as the write data length of the second RDMA write WQE, use the storage address of the request side in the payload information as the storage address of the second RDMA write WQE on the request side, set the operation code in the second RDMA write WQE to an RDMA write request with an immediate value, and set the CQE type information in the second RDMA write WQE to a read-modify-write type.

[0083] A second RDMA write request is generated through a second RDMA write WQE with an immediate number, so that after the request side device 100 receives the second RDMA write request, a second CQE is generated. Since the second CQE carries the type information of the second CQE, it is convenient for the CQE processing unit 116 on the request side to identify and process the CQE.

[0084] In some embodiments, the second network interface card 220 includes: a second transmitting unit 221, a second receiving unit 222, and a second data cache unit 224. The second receiving unit 222 is used to send a doorbell to the second transmitting unit 221 after receiving the first RDMA write request. When the second transmitting unit 221 receives the doorbell, it sends a first ACK information to the requesting side; wherein the first ACK information is used to indicate that the RDMA responding side has received the first RDMA write request. The second transmitting unit 221 also receives a second RDMA write WQE, reads the request data according to the read address, generates a second RDMA write request based on the request data and the second RDMA write WQE, and sends the second RDMA write request to the requesting side.

[0085] In the first aspect, the response side device 200 of the present application only needs to process RDMA write requests during operation, and RDMA write requests only generate Acks but no responses. Therefore, the response side no longer needs to set SRRL, thus saving memory resources.

[0086] In the second aspect, the response side device 200 of the present application does not need to process RDMA read and read response related fields because of read-modify-write, so QPC related fields can be removed, reducing the complexity of QPC.

[0087] In the third aspect, the responding side device 200 of the present application performs read-modify-write operations to make the PSN continuous, so there is no need to use (MSN, IPSN) to determine whether a message is received, which is more conducive to hardware implementation.

[0088] Figure 5 Schematic diagram of a communication system based on RDMA provided in an embodiment of the present application. Figure 5 As shown, the communication system 1 includes the request side device 100 in any of the above embodiments and the response side device 200 in any of the above embodiments. After receiving the read work request, the request side device 100 converts the read work request into a first RDMA write request and sends the first RDMA write request to the response side device 200. After receiving the first RDMA write request, the responding side device parses it to obtain the payload information of the first RDMA write request, and generates a second RDMA write WQE based on the payload information. The responding side device sends a first ACK information to the requesting side device. In addition, the responding side device also generates a second RDMA write request based on the second RDMA write WQE and the read request data. The responding side device sends the second RDMA write request to the requesting side device. The requesting side device receives the first ACK information sent by the responding side device and generates a first CQE. The requesting side device also receives the second RDMA write request and parses it to obtain the request data, and stores the request data at the corresponding address, and generates a second CQE at the same time. The requesting side device generates a WC based on the first CQE and the second CQE, and sends the WC to the external device that sends the read request to notify the external device to read the data.

[0089] See also Figure 6 , Figure 6It is a more detailed schematic diagram of the communication system provided by the embodiment of the present application. After receiving the read work request sent by the sender of the read work request, the first conversion module 110 uses the read work request information as the inline data of the WQE, sets the CQE identifier of the WQE to generate the CQE, sets the opcode of the WQE to the write request, and sets the type of the CQE in the WQE to the read-modify-write type, generates a first RDMA write WQE, and writes the first RDMA write WQE into the first SQ cache unit 113. The first transmitting unit 121 reads the first RDMA write WQE, converts the first RDMA write WQE into a first RDMA write request, and sends the first RDMA write request to the second receiving unit 222. After receiving the first RDMA write request, the second receiving unit 222 parses it, obtains the payload information and storage address of the first RDMA write request, and stores the payload information at the corresponding storage address of the RDMA write WQE cache unit 223. When the second receiving unit 222 receives the second RDMA write request, it will also send a doorbell to the second transmitting unit 221 to notify the second transmitting unit 221 to send the first ACK information to the first receiving unit 122 to notify the request side device 100 that the response side device 200 has received the first RDMA write request. The second conversion module 210 polls the RDMA write WQE cache unit 223, and when it is found that the RDMA write WQE cache unit 223 has a new payload written, it reads the payload information, generates a second RDMA write WQE based on the payload information, and stores the second RDMA write WQE in the second SQ cache unit. The second transmitting unit 221 reads the second RDMA write WQE, reads the corresponding request data, generates a second RDMA write request based on the second RDMA write WQE and the request data, and sends the second RDMA write request.

[0090] The first receiving unit 122 receives the first ACK information sent by the second transmitting unit 221, generates a first CQE according to the first ACK information, and caches the first CQE in the first CQE cache unit. The first receiving unit 122 also receives and parses the second RDMA write request sent by the second transmitting unit 221, obtains the corresponding request data and stores the request data in the first data cache unit 123, generates a second CQE, and caches the second CQE in the second CQE cache unit 115. The CQE processing unit 116 reads the first CQE in the first CQE cache unit 114 and the second CQE in the second CQE cache unit. When the CQE processing unit 116 reads the type information of the first CQE as the read-modify-write type, the CQE processing unit 116 stops reading the data in the first CQE cache unit 114 until the second CQE corresponding to the first CQE is read from the second CQE cache unit 115, generates a WC, and sends the WC to the external device. Then, the new first CQE in the first CQE buffer unit 114 continues to be read, and the above processing flow is repeated.

[0091] For more details, please refer to the contents in the aforementioned implementation manner.

[0092] The present application realizes the function of RDMA read request by converting the read work request into the first RDMA write request on the request side, and generating and sending the second RDMA write request in response to the first RDMA write request on the response side. Since the RDMA write request only generates Ack but not a response, the request side and the response side no longer need to set CRRL and SRRL, saving memory resources.

[0093] In addition, since read-modify-write is performed, there is no need to process the RDMA read and read response related fields, so the QPC related fields can be removed, reducing the complexity of QPC.

[0094] Furthermore, because the read-modify-write operation is performed, the PSN is continuous, so there is no need to use (MSN, IPSN) to determine whether the message is received, which is more conducive to hardware implementation.

[0095] Finally, the present application is conducive to the implementation of congestion control. The read request in the prior art is bidirectional, and it is necessary to control the flow of request messages and the flow of response messages. In the present application, only the request message has a data message and needs flow control, while the response message only has an acknowledgement message and does not need flow control, thereby simplifying the implementation of congestion control.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A communication device based on RDMA, characterized in that: Applied to the request side of RDMA, the device comprises: a first conversion module and a first network interface card; wherein, The first conversion module is configured to, upon receiving a read work request, convert the read work request into a first RDMA write WQE, wherein the first RDMA write WQE at least includes a read address and a storage address of request data corresponding to the read work request; The first network interface card is used to receive the first RDMA write WQE, convert the first RDMA write WQE into a first RDMA write request, and send the first RDMA write request; The first network interface card is further configured to receive and parse a second RDMA write request sent by the RDMA response side in response to the first RDMA write request, obtain request data corresponding to the read work request, write the request data into the storage address, and generate a CQE; The first conversion module is further configured to generate a WC according to the CQE and send the WC.

2. The communication device according to claim 1, characterized in that The first conversion module is specifically used for: The read work request information is used as the inline data of the WQE, the CQE identifier of the WQE is set to generate a CQE, the opcode of the WQE is set to a write request, and the type of the CQE in the WQE is set to a read-modify-write type, the first RDMA write WQE is generated, and the first RDMA write WQE is written into the first SQ cache unit.

3. The communication device according to claim 1, characterized in that The first network interface card at least includes a first transmitting unit, a first receiving unit and a first data cache unit; The first transmitting unit is used to read the first RDMA write WQE, convert the first RDMA write WQE into a first RDMA write request, and send the first RDMA write request; The first receiving unit is used to receive first ACK information sent by the RDMA response side, and generate a first CQE according to the first ACK information, wherein the first ACK information is used to indicate that the RDMA response side has received the first RDMA write request; The first receiving unit is further used to receive and parse the second RDMA write request sent by the RDMA response side, obtain request data corresponding to the read work request, write the request data into the first data cache unit, and generate a second CQE; The first conversion module is further configured to generate the WC according to the first CQE and the second CQE, and send the WC to the read work request sending end.

4. The communication device according to claim 3, characterized in that: The first receiving unit is specifically configured to: The first ACK information is received, a first RDMA write WQE corresponding to the first ACK information is read according to the PSN of the first ACK information, and a first CQE is generated according to the first RDMA write WQE, wherein the first CQE includes identification setting information of the first RDMA write WQE.

5. The communication device according to claim 4, characterized in that: The device also includes: A first CQE buffer unit, configured to buffer a first CQE generated by the first receiving unit; A second CQE buffering unit, configured to buffer a second CQE generated by the first receiving unit; A CQE processing unit is used to read the first CQE in the first CQE cache unit and read the second CQE in the second CQE cache unit. When the identification setting information of the first CQE is a read-modify-write type, if there is a second CQE corresponding to the first CQE in the second CQE cache unit, the WC is generated and sent to the read work request sending end.

6. A communication device based on RDMA, characterized in that: Applied to the RDMA response side, the device comprises: a second conversion module and a second network interface card; wherein, The second network interface card is used to receive and parse the first RDMA write request, and obtain payload information corresponding to the first RDMA write request, wherein the first RDMA write request is a write request converted from a read work request, and the payload information includes a read address of request data corresponding to the read work request and a storage address on the request side; The second conversion module is used to read the payload information and generate a second RDMA write WQE based on the payload information; The second network interface card is used to receive the second RDMA write WQE and read the request data according to the read address; it is also used to generate a second RDMA write request according to the request data and the second RDMA write WQE, and send the second RDMA write request to the request side.

7. The communication device according to claim 6, characterized in that: The second conversion module is specifically used for: Read the payload information of the first RDMA write request, obtain the read address, data length and storage address of the request side of the request data corresponding to the read work request, determine the data length as the write data length of the second RDMA write WQE, use the storage address of the request side as the storage address of the second RDMA write WQE on the request side, set the operation code in the second RDMA write WQE to an RDMA write request with an immediate number, thereby generating the second RDMA write WQE.

8. The communication device according to claim 7, characterized in that: The second network interface card further includes: an RDMA write WQE cache unit; The RDMA write WQE cache unit is used to cache the payload information.

9. The communication device according to claim 8, characterized in that: The second network interface card includes: a second transmitting unit and a second receiving unit; The second receiving unit is configured to send a doorbell to the second transmitting unit after receiving the first RDMA write request; The second transmitting unit is used to send a first ACK information to the requesting side when receiving the doorbell; wherein the first ACK information is used to indicate that the RDMA responding side has received the first RDMA write request; The second transmitting unit is further used to receive the second RDMA write WQE, read the request data according to the read address, generate a second RDMA write request based on the request data and the second RDMA write WQE, and send the second RDMA write request to the requesting side.

10. A communication system based on RDMA, characterized in that: The system comprises: An RDMA-based communication device as described in any one of claims 1 to 5 and an RDMA-based communication device as described in any one of claims 6 to 9.

Images