RDMA-based requester, RDMA-based responder, and RDMA-based system

Abstract

The present application relates to an RDMA-based requester and an RDMA-based responder. The requester comprises a transmit processor and a receive processor, wherein the transmit processor is used for receiving a work queue element issued by a driver program, and if the work queue element is an RDMA read operation, converting the RDMA read operation, so as to obtain a request-side write request; the transmit processor is further used for transmitting the request-side write request to a responder; and the receive processor is used for receiving and parsing a response-side write request sent by the responder, so as to obtain target data and a destination address which correspond to the RDMA read operation, and writing the target data into a storage space corresponding to the destination address. The present application reduces QPC data structure fields and design difficulty, reduces the complexity of out-of-order processing of messages, and avoids the consumption of a large number of host memory and cache resources.

Description

RDMA-based request device, response device and system

[0001] Related applications

[0002] The present application claims priority from the Chinese patent application No. 202411363358.0, filed on September 27, 2024, and entitled "RDMA-based request device, response device and system", the contents of which are hereby incorporated by reference in their entirety. TECHNICAL FIELD

[0003] The present application relates to the field of remote direct memory access (RDMA) technology, and in particular to an RDMA-based request device, response device and system. BACKGROUND

[0004] As shown in FIG. 1, the method of FIG. 1 is a principle diagram of a currently used RDMA-based read data method, and the method of FIG. 1 includes the following steps:

[0005] (1) A driver issues a work queue element (WQE) to a send queue (SQ) of a requester (requester); the WQE is specifically an RDMA read operation, which is used to instruct the requester to request to read data from a response device (responder);

[0006] (2) A send processor (TXP) of the requester converts the WQE into an RDMA read request (RDMA read request), and writes related information for receiving a read response, such as a packet sequence number (Packet Sequence Number, PSN) and an SGE for placing the read response, into a client-side RDMA read list (Client RDMA read list, CRRL);

[0007] (3) The TXP of the requester sends the RDMA read request to the responder through a network;

[0008] (4) A receive end (RXP) of the responder receives the RDMA read request, parses the RDMA read request to obtain related information for generating a read response, and writes the related information for generating the read response into a server-side RDMA read list (SRRL), such as a read encyclopedia table header (RETH, which is a data structure), and generates a request queue doorbell (Receive Queue Doorbell, RQDB) to the TXP of the responder;

[0009] (5) RQDB is sent from the RXP of the responder to the TXP of the responder;

[0010] (6) The TXP of the responder receives the RQDB, reads the related information of generating a read response in a server-side RDMA read list (SRRL), and generates a read response (Read response) according to the related information;

[0011] (7) The TXP of the responder sends the Read response to the RXP of the requester through the network;

[0012] (8) The RXP of the requester receives the Read response, reads the related information of receiving the read response in a CRRL, writes the data of the Read response into a corresponding buffer according to the related information, and generates a CQE (CQE is a completion queue element) to the SQ;

[0013] (9) The CQE is returned to the driver, and the RDMA read operation is completed.

[0014] The above method has the following technical problems:

[0015] Each queue pair (QP) of the requester needs a CRRL data structure, and each QP of the responder needs a SRRL data structure. For example, each entry of the CRRL data structure is 512 bits, and the depth is 128; each entry of the SRRL data structure is 256 bits, and the depth is 128. If 1M QPs are supported, 12GB of memory resources are required, so a large amount of memory and cache resources are consumed. In addition, since one RDMA read operation may generate multiple Read responses, multiple Read responses may arrive out of order, the Read response PSN of different messages is not continuous, and the out-of-order processing of the Read response is relatively complex. Moreover, each QP has a queue pair context (QPC), and in the case of a large number of QPs, a large number of related QPCs need to be maintained, resulting in a large QPC bit width and an increase in complexity. SUMMARY

[0016] The present application aims to provide a RDMA-based request device, response device and system to solve the above technical problems.

[0017] To achieve the above object, according to a first aspect of the present application, a request device based on RDMA is provided, the request device comprising a sending processor and a receiving processor;

[0018] The sending processor is configured to receive a work queue element issued by a driver, and if the work queue element is an RDMA read operation, convert the RDMA read operation to obtain a request-side write request.

[0019] The sending processor is further configured to send the request-side write request to a response device.

[0020] The receiving processor is configured to receive and analyze the response-side write request sent by the response device to obtain target data and a destination address corresponding to the RDMA read operation, and write the target data into a storage space corresponding to the destination address.

[0021] According to a second aspect of the present application, a response device based on RDMA is further provided, the response device comprising a receiving processor, a sending processor and a data storage structure.

[0022] The response device comprises a receiving processor, a sending processor and a data storage structure.

[0023] The receiving processor is configured to receive a request-side write request sent by a request device, wherein the request-side write request is obtained by the request device converting an RDMA read operation when a work queue element issued by a driver to the request device is the RDMA read operation.

[0024] The receiving processor is further configured to analyze the request-side write request to obtain a source data address, a destination address and a data length, write the source data address, the destination address and the data length into the data storage structure, and generate a request queue doorbell.

[0025] The sending processor is configured to read the source data address, the destination address and the data length from the data storage structure according to the request queue doorbell, read target data according to the source data address and the data length, generate a response-side write request according to the target data and the destination address, and send the response-side write request to the request device, so that the request device writes the target data into a storage space corresponding to the destination address according to the response-side write request.

[0026] According to a third aspect of the present application, a system based on RDMA is further provided, comprising the request device according to the first aspect and the response device according to the second aspect.

[0027] The above request device, response device and system based on RDMA have the following beneficial effects:

[0028] The request device and the response device are used in pairs, and the same device can have the functions of the request device and the response device at the same time. The RDMA read operation issued by the driver to the request device is converted into a request-side write request, so that the request device does not need to set a data storage structure (CRRL) to store the relevant information of the received read response. Since the target data of the read is directly written into the storage space corresponding to the destination address in the request device by the response device, the request device does not need to perform complex out-of-order processing and rearrangement logic. Moreover, after the read operation is converted into a write operation, each write operation is independent and does not need to track multiple unfinished read operations, thereby reducing the maintenance of the QPC. In summary, a large amount of memory and cache resources can be avoided.

[0029] Other features and advantages of the above-mentioned RDMA-based request device, response device and system will be described in the subsequent description. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor based on the disclosed drawings.

[0031] FIG. 1 is an interaction flowchart of a conventional RDMA-based request device and response device.

[0032] FIG. 2 is an interaction flowchart of an RDMA-based request device and response device in an embodiment of the present application.

[0033] FIG. 3 is a message structure diagram of an IB protocol-based RDMA write request in an embodiment of the present application.

[0034] FIG. 4 is a message structure diagram of a request-side write request in an embodiment of the present application.

[0035] FIG. 5 is a message structure diagram of an IB protocol-based RDMA read response in an embodiment of the present application.

[0036] FIG. 6 is a message structure diagram of a response-side write request in an embodiment of the present application. DETAILED DESCRIPTION

[0037] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of the present application.

[0038] One embodiment of the present application provides an RDMA-based request device, and the working principle of the request device is shown in FIG. 2. Referring to FIG. 2, the request device (requester) includes a sending processor (TXP) and a receiving processor (RXP).

[0039] The sending processor is configured to receive a work queue element issued by a driver, and if the work queue element is an RDMA read operation, convert the RDMA read operation to obtain a request-side write request.

[0040] Specifically, in RDMA, a work queue element (WQE) is an instruction issued by a driver to a request device, used to perform a specific network operation. In this embodiment, an RDMA read operation is to be implemented. Because the WQE contains all the information required to perform an RDMA read operation, both the request device and the response device are provided with a network interface card (NIC) to implement RDMA communication.

[0041] Specifically, the sending processor (located in the NIC) of the request device receives a WQE issued by the driver, which is used to perform an RDMA read operation and contains information such as a source data address (i.e., a data location on the response device), a destination address (i.e., an address in the local memory for storing the read data), and a data length. The sending processor converts the WQE into a request-side write request (RDMA read convert write), so that the original RDMA read operation is changed into a write operation. The request-side write request contains the source data address, the destination address, and the data length, which indicate that the response device should read the target data from the source data address and write the target data to the destination address of the request device.

[0042] The sending processor is further configured to send the request-side write request to the response device, so that the response device reads the target data according to the source data address and the data length, and generates a response-side write request according to the target data, the destination address, and the data length.

[0043] Specifically, the sending processor of the request device sends the converted request-side write request to the response device through the network to inform the response device that the request device wants to read data of a specific length from the source data address of the response device and write it to the destination address of the request device. When the network interface card (NIC) of the response device receives the request-side write request, the receiving processor of the response device parses the request and reads the target data according to the source data address and data length information in the request, where the target data refers to the data that the request device wants to obtain, which is stored in the memory of the response device. Then, the sending processor of the response device generates a response-side write request (read response convert write) according to the read target data and the destination address and data length information in the request-side write request, and returns the response-side write request to the request device, so as to write the read target data to the destination address of the request device, thereby changing the RDMA read operation to an RDMA write operation.

[0044] The receiving processor is configured to receive and parse the response-side write request sent by the response device to obtain the target data and the destination address corresponding to the RDMA read operation, and write the target data to a storage space corresponding to the destination address.

[0045] Specifically, when the receiving processor (located in the network interface card (NIC) of the request device) of the request device receives the response-side write request, the receiving processor parses the response-side write request to extract the target data and the destination address information therefrom. After extracting the target data and the destination address, the receiving processor writes the target data to a storage space corresponding to the destination address in the memory of the request device, which means that the receiving processor of the request device directly places the data read from the response device to a memory location specified by the request device, thereby completing the reception of the data.

[0046] To sum up, the request device and the response device of the embodiment are used together, and the same device can simultaneously have the functions of the request device and the response device. The RDMA read operation issued by the driver to the request device is converted into a request-side write request, so that the request device does not need to set a data storage structure (CRRL) to store the relevant information of the read response. Since the target data read by the response device is directly written to the storage space corresponding to the destination address in the request device, the request device does not need to perform complex out-of-order processing and rearrangement logic. Moreover, after the read operation is converted into a write operation, each write operation is independent and does not need to track multiple unfinished read operations, thereby reducing the maintenance of the QPC. In summary, a large amount of memory and cache resources can be avoided.

[0047] In some embodiments, the request side write request comprises a Local Routing Header field, a Global Routing Header field, a Base Transport Header field, a Message payload field, an Invariant CRC field and a Variant CRC field; an OpCode in the Base Transport Header field of the request side write request uses a self-defined preset encoding, so that the response device can identify the operation type of the request side write request as a read request converted write request according to the self-defined preset encoding.

[0048] Specifically, as shown in FIG. 3, the message structure of a conventional RDMA write request based on the IB protocol, and as shown in FIG. 4, the message structure of the request side write request defined in the embodiment, in conventional RDMA communication, the sending processor of the request device converts the RDMA read operation issued by the driver to the SQ into a read request, but this will cause the technical problems mentioned in the background art, therefore, in the embodiment, the sending processor of the request device converts the RDMA read operation issued by the driver to the SQ into the request side write request with the message structure shown in FIG. 4, the message structure of the request side write request defined in the embodiment is different from the message structure of the conventional RDMA write request based on the IB protocol, the OpCode (operation code) in the Base Transport Header of the message structure of FIG. 3 uses the encoding defined by the IB protocol, while the OpCode (operation code) in the Base Transport Header of the message structure shown in FIG. 4 of the embodiment does not use the encoding already used by the IB protocol, but uses a self-defined preset encoding; for example, for RC type reset, the OpCode can be 8’b000_11000, to realize the conversion of the RDMA read operation to the write operation proposed in the embodiment. In addition, the Message payload of the message structure shown in FIG. 4 is the information of the original RDMA read operation issued by the driver, including but not limited to the address of the source data, the length of the source data, the storage address after the data return, etc.; the remaining fields of the message structure of FIG. 4 have the same content as the same fields of the message structure of FIG. 3, and therefore will not be described in detail.

[0049] In some embodiments, the device further comprises a counter;

[0050] The counter is used to record the number of received response side write requests which are in order and have not been reported to the driver when the receiving processor receives the response side write request sent by the response device;

[0051] The receiving processor is further configured to, when receiving the response-side Ack signal sent by the response device, if the work queue element corresponding to the response-side Ack signal is an RDMA read operation and the number of response-side write requests recorded by the counter is greater than or equal to the number of response-side write requests expected to be received by the RDMA read operation, generate a completion queue element corresponding to the RDMA read operation, and send the completion queue element to the driver.

[0052] Specifically, the receiving processor only checks the number of response-side write requests recorded by the counter when receiving the response-side Ack signal. Normally, when an RDMA read operation is completed, the number of response-side write requests recorded by the counter should be equal to the number of response-side write requests expected to be received by the RDMA read operation. However, in special cases, the number of response-side write requests recorded by the counter can be greater than the number of response-side write requests expected to be received by the RDMA read operation, for example, in the case of packet loss and retransmission. However, as long as the number of response-side write requests recorded by the counter is greater than or equal to the number of response-side write requests expected to be received by the RDMA read operation when the response-side Ack signal is received for checking, it can be determined that the read data collection has been completed and the RDMA read operation has been completed. At this time, a completion queue element corresponding to the RDMA read operation can be generated and sent to the driver to notify the driver. Meanwhile, the number of response-side write requests recorded by the counter also needs to be updated.

[0053] In some embodiments, the sending processor is further configured to store the request-side write request in a request-side sending chain table; wherein the request-side sending chain table stores each request-side write request in sending order;

[0054] The receiving processor is further configured to update the request-side sending chain table according to the response-side write request, and generate a request queue doorbell;

[0055] The sending processor is further configured to generate a request-side Ack signal according to the request queue doorbell, and send the request-side Ack signal to the response device to notify the response device that the response-side write request sent by the response device has been received by the request device.

[0056] Specifically, the receiving processor of the request device deletes the request-side write request corresponding to the response-side Ack signal from the request-side sending chain table (TXC) in response to the response-side Ack signal sent by the response device, indicating that the receiving processor of the response device has successfully received the request-side write request. The request-side sending chain table stores each request-side write request in sending order, and is used to track and manage the unfinished request-side write request.

[0057] The sending processor of the request device generates a request side Ack signal according to the request queue doorbell, and the request side Ack signal is used to confirm that the request device has received the response side write request. The sending processor of the request device sends the request side Ack signal to the response device, which can be understood as the request device sending a receiving confirmation to the response device.

[0058] In some embodiments, the request side write request, the response side write request, the request side Ack signal, and the response side Ack signal each carry a sequence number (PSN). The sequence number carried by the request side write request and the sequence number carried by the response side write request are independent of each other. The sequence number carried by the request side Ack signal is the same as the sequence number carried by the corresponding response side write request. The sequence number carried by the response side Ack signal is the same as the sequence number carried by the corresponding request side write request.

[0059] Specifically, in the conventional RDMA, because the PSNs of the request message and the returned response message are not continuous, a two-dimensional data structure of (MSN, IPSN) is generally used to describe the receiving state of the message, where IPSN is the PSN inside the message, and each message starts from 0. Therefore, the difficulty of processing out-of-order messages is increased. In addition, the returned response message includes a response with data and an Ack signal without data. The Ack signal and the response can aggregate the Ack signal in front of them, but the Ack signal cannot aggregate the response in front of it, which makes the in-sequence judgment of the Ack signal and the message complex. The meaning of aggregation is that the message can represent itself and the aggregated message in front of it. In view of this, the request device and the response device of the present embodiment are respectively configured with a set of independent PSNs, which are numbered in their own order and are not affected by the other party. For example, the PSN of the request device is PSN_a1-PSN_a100, and the PSN of the response device is PSN_b1-PSN_b100. Therefore, the PSNs of the multiple request side write requests sent in sequence by the request device are continuous, and the PSNs of the multiple response side write requests sent in sequence by the response device are also continuous. In addition, the sequence number carried by the request side Ack signal is the same as the sequence number carried by the corresponding response side write request, and the sequence number carried by the response side Ack signal is the same as the sequence number carried by the corresponding request side write request. Therefore, the request device can perform in-sequence operation according to the PSNs carried by the received response side write request and the response side Ack signal.

[0060] In some embodiments, the sending processor is further configured to read the request side write request from the request side sending link list and resend it to the response device when the receiving processor fails to successfully receive the response side Ack signal corresponding to the request side write request within a preset time.

[0061] Specifically, the sending processor of the requester stores the request-side write request into a request-side sending linked list, which stores each request-side write request in sending order, for tracking and managing the unfinished requests. When the receiving processor of the requester fails to successfully receive the response-side Ack signal corresponding to the request-side write request within a preset time, it indicates that some error may occur, such as network failure, data transmission failure, etc. In this case, the sending processor of the requester reads the request-side write request for which the response is not successfully received from the request-side sending linked list, and then the sending processor of the requester re-sends the request-side write request to the responder, so as to make the responder re-send the response-side write request corresponding to the request-side write request. Through this mechanism, even if the data transmission failure occurs, the RDMA communication can automatically attempt to re-transmit the unsuccessful request, thereby improving the reliability of data transmission and the robustness of the system. This error handling mechanism is another important component for ensuring the reliability of data transmission in RDMA communication.

[0062] Another embodiment of the present application provides an RDMA-based responder, as shown in FIG. 2, which includes a receiving processor (RXP), a sending processor (TXP) and a data storage structure (SRRL).

[0063] The receiving processor is configured to receive the request-side write request sent by the requester; wherein the request-side write request is a write request converted from a read operation request;

[0064] Specifically, the receiving processor (located in the NIC) of the responder receives the request-side write request (RDMAread convert write) sent by the requester, which is converted by the requester from the original RDMA read operation work queue element (WQE) issued by the driver. This conversion means that the original RDMA read operation is changed to a write operation, and the request-side write request contains information such as source data address, destination address and data length, which indicates that the responder should read how much data from the source data address and write the data to the destination address of the requester.

[0065] The receiving processor is further configured to parse the request-side write request to obtain the source data address, the destination address and the data length, write the source data address, the destination address and the data length into the data storage structure, and generate a request queue doorbell (Receive Queue Doorbell, RQDB).

[0066] Specifically, the receiving processor of the responder parses the read request, extracts the source data address, destination address and data length, which are the key parameters for performing the RDMA operation, and writes them into the data storage structure (SRRL) of the responder. The data storage structure is a data structure specially designed for storing and tracking the status of the RDMA operation. Then, the receiving processor of the responder generates an RQDB to inform the sending processor of the responder that there is new data in the request queue to be processed. Through the RQDB, the sending processor of the responder can know which data should be read from the data storage structure in order to generate the read response convert write request.

[0067] The sending processor is configured to read the source data address, destination address and data length from the data storage structure according to the request queue bell, read target data according to the source data address and data length, generate a read response convert write request according to the target data and destination address, and send the read response convert write request to the request device, so that the request device writes the target data into the storage space corresponding to the destination address according to the read response convert write request.

[0068] Specifically, the sending processor of the responder receives the RQDB and reads the source data address, destination address and data length from the data storage structure. These information tells the sending processor where to read data and where to write data. According to these information, the sending processor of the responder reads target data from the source data address. These target data are the data that the responder needs to send back to the request device. Then, the sending processor of the responder generates a read response convert write request according to the target data and destination address. This read response convert write request contains the target data and destination address information, which indicates the memory address where the request device should write the target data. Finally, the sending processor sends this read response convert write request back to the request device through the network, so that the request device can write the target data into the storage space corresponding to the destination address according to the read response convert write request.

[0069] In summary, the response device of the present embodiment and the request device of the above embodiment are used in combination, the same device can simultaneously have the functions of the request device of the above embodiment and the response device of the present embodiment, the RDMA read operation issued by the driver to the request device is converted into a request-side write request, so that it is not necessary to set a data storage structure (CRRL) in the request device to store the relevant information of the received read response, since the target data of the read is directly written into the storage space corresponding to the destination address in the request device by the response device, the request device does not need to perform complex out-of-order processing and rearrangement logic. Moreover, after the read operation is converted into a write operation, each write operation is independent, and there is no need to track multiple unfinished read operations, thereby reducing the maintenance of QPC. In summary, it is possible to avoid consuming a large amount of memory and cache resources.

[0070] In some embodiments, the response-side write request includes a Local Routing Header field, a Global Routing Header field, a Base Transport Header field, an Extended Transport Header field, a Message payload field, an Invariant CRC field, and a Variant CRC field; the OpCode in the Base Transport Header field of the response-side write request uses a custom preset encoding, so that the request device can recognize the operation type of the response-side write request as a response to the request-side write request according to the custom preset encoding.

[0071] Specifically, as shown in FIG. 5 is a message structure of a traditional RDMA read response based on IB protocol, as shown in FIG. 6 is a message structure of a response side write request defined in the embodiment, in traditional RDMA communication, the sending processor of the response device generates a read response for the read data to be returned, but this will cause the technical problem mentioned in the background art, therefore in the embodiment, the sending processor of the request device converts the RDMA read operation issued by the driver to the SQ into a request side write request of the message structure shown in FIG. 4, the response device reads data according to the request side write request and generates a response side write request of the message structure shown in FIG. 6, the message structure of the response side write request defined in the embodiment is different from the message structure of the traditional RDMA read response based on IB protocol, the OpCode (operation code) in the Base Transport Header of the message structure of FIG. 5 uses the code defined by IB protocol, while the OpCode (operation code) in the Base Transport Header of the message structure shown in FIG. 6 of the embodiment does not use the code used by IB protocol, but uses a preset code defined by itself; for example, for RC type reset, the OpCode can be 8’b000_11001, to realize the conversion of the RDMA read operation to the write operation proposed in the embodiment. In addition, the ExtendTransport header of the message structure shown in FIG. 6 is RETH, in which VA, R_KEY, DMALen are obtained from SRRL, the source is the request side write request sent by the request device, the remaining fields of the message structure of FIG. 6 are the same as the contents of the same fields of the message structure of FIG. 5, so they will not be described again.

[0072] In some embodiments, the receiving processor parsing the request side write request also obtains a sequence number (PSN), and writes the sequence number into the data storage structure;

[0073] The sending processor is further configured to read the sequence number (PSN) from the data storage structure according to the request queue doorbell, generate a response side Ack signal according to the sequence number, and after sending the response side write request to the request device, send the response side Ack signal to the request device to notify the request device that the request side write request has been received.

[0074] Specifically, a receiving processor of the responding device parses the request side write request and extracts a PSN therefrom, the PSN being an identifier for identifying the request side write request, and the receiving processor writes the extracted PSN into a data storage structure of the responding device. A sending processor of the responding device receives the RQDB and reads the PSN from the data storage structure, and the sending processor of the responding device generates a response side Ack signal according to the PSN, the response side Ack signal being a receiving acknowledgement of the sending data of the requesting device by the responding device and indicating that the request side write request corresponding to the PSN has completed receiving. After sending the response side write request to the requesting device, the sending processor of the responding device sends the response side Ack signal to the requesting device, notifying the requesting device that the request side write request has been received by the responding device. Based on the above manner, the RDMA communication parties can ensure reliable transmission and sequential processing of data.

[0075] In some embodiments, the sending processor is further configured to store the response side write request into a response side sending linked list; wherein the response side sending linked list stores each response side write request in sending order.

[0076] The receiving processor is further configured to, in response to the request side write request retransmitted by the requesting device, retransmit a corresponding response side write request stored in the response side sending linked list to the requesting device according to the retransmitted request side write request; and after retransmitting the response side write request, send a response side Ack signal to the requesting device to notify the requesting device that the retransmitted request side write request has completed receiving.

[0077] In particular, the sending processor of the responder stores the response-side write request into a response-side sending list (TXC) in which each response-side write request is stored in sending order, for tracking and managing the uncompleted responses. After the responder sends a response-side write request to the requester in response to the request-side write request from the requester, the responder sends a response-side Ack signal to the requester to inform the requester that the request-side write request sent by the requester has been received by the responder, and the response-side Ack signal should be received by the requester within a preset time. If the requester does not receive the response-side Ack signal within the preset time, it means that some error may occur, such as network failure, data transmission failure, etc., and the requester will retransmit the request-side write request to the responder. When the receiving processor of the responder receives the request-side write request, it generates an RQDB to instruct the sending processor to retransmit the request-side write request. The sending processor reads the corresponding response-side write request from the response-side sending list according to the RQDB, and the sending processor of the responder retransmits the response-side write request to the requester. Through the above mechanism, even if data transmission failure occurs, the RDMA communication can automatically attempt to retransmit the unsuccessful response, thereby improving the reliability of data transmission and the robustness of the system. This specific error handling mechanism is another important component for ensuring the reliability of data transmission in RDMA communication.

[0078] In some embodiments, the receiving processor is further configured to update the response-side sending list according to the request-side Ack signal sent by the requester.

[0079] In particular, the receiving processor of the responder responds to the request-side Ack signal sent by the requester, which means that the requester has successfully received the response-side write request. Then, the receiving processor of the responder deletes the response-side write request corresponding to the request-side Ack signal from the response-side sending list according to the request-side Ack signal. The response-side sending list stores each response-side write request in sending order, for tracking and managing the uncompleted responses.

[0080] The sending processor is further configured to retransmit the response-side write request to the requester when the receiving processor does not successfully receive the request-side Ack signal within a preset time.

[0081] In particular, when the receiving processor of the response device fails to successfully receive the request side Ack signal corresponding to the response side write request it has sent within a preset time, it indicates that some error may have occurred, such as network failure, data transmission failure, etc. In this case, the receiving processor of the response device generates an RQDB instruction to instruct the sending processor to retransmit the request side write request. The sending processor reads the corresponding response side write request from the response side sending linked list according to the RQDB, and the sending processor of the response device retransmits the response side write request to the request device. After retransmitting the response side write request, if the request device successfully receives the retransmitted response side write request, the request device sends a request side Ack signal to inform the response device that it has received the request side write request retransmitted by the response device. Through the above mechanism, even if data transmission failure occurs, RDMA communication can automatically attempt to retransmit the unsuccessful response, thereby improving the reliability of data transmission and the robustness of the system. This specific error handling mechanism is another important component to ensure the reliability of data transmission in RDMA communication.

[0082] In some embodiments, the request side write request, the response side write request, the request side Ack signal, and the response side Ack signal each carries a sequence number. The sequence number carried by the request side write request and the sequence number carried by the response side write request are independent of each other. The sequence number carried by the request side Ack signal is the same as the sequence number carried by the corresponding response side write request. The sequence number carried by the response side Ack signal is the same as the sequence number carried by the corresponding request side write request.

[0083] Specifically, in the conventional RDMA, since the PSN of the request message and the return response message is not continuous, a two-dimensional data structure of (MSN, IPSN) is generally used to describe the receiving state of the message, wherein the IPSN is the PSN inside the message, and each message starts from 0; thus, the difficulty of processing the out-of-order message is increased. In addition, the returned response message includes the response with data and the Ack signal without data; the Ack signal and the response can aggregate the previous Ack signal, but the Ack signal cannot aggregate the previous response, which leads to the complexity of the in-sequence judgment of the Ack signal and the message, and the aggregation means that the message can represent itself and the aggregated message in front of it. In view of this, the request device and the response device of the embodiment are respectively configured with a set of independent PSN, which are numbered in their own order and are not affected by each other, for example, the PSN of the request device is PSN_a1~PSN_a100, and the PSN of the response device is PSN_b1~PSN_b100, thus, the PSN of the multiple request-side write requests sent in sequence by the request device is continuous, the PSN of the multiple response-side write requests sent in sequence by the response device is also continuous, and the sequence number carried by the request-side Ack signal is the same as the sequence number carried by the corresponding response-side write request, and the sequence number carried by the response-side Ack signal is the same as the sequence number carried by the corresponding request-side write request, thus, the request device can perform the in-sequence operation according to the PSN carried by the received response-side write request and the response-side Ack signal.

[0084] Another embodiment of the present application provides an RDMA-based system, as shown in FIG. 2, which comprises the request device as described in the above embodiment and the response device as described in the above embodiment, and the working principle of the system of the embodiment can be obtained by referring to the content of the request device as described in the above embodiment and the response device as described in the above embodiment, and thus the description is not repeated here.

[0085] The technical features of the above-described embodiments can be combined arbitrarily, and in order to make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0086] The above-described embodiments only express several implementation manners of the present application, the description is more specific and detailed, but it should not be understood as the limitation of the patent application scope. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An RDMA-based request apparatus, comprising: The request device comprises a sending processor and a receiving processor; The sending processor is configured to receive a work queue element issued by a driver, and if the work queue element is an RDMA read operation, convert the RDMA read operation into a request-side write request; The sending processor is further configured to send the request-side write request to a response device; The receiving processor is configured to receive and analyze the response-side write request sent by the response device to obtain target data and a destination address corresponding to the RDMA read operation, and write the target data into a storage space corresponding to the destination address.

2. The RDMA-based request apparatus according to claim 1, wherein, The request-side write request comprises a Local Routing Header field, a Global Routing Header field, a Base Transport Header field, a Message payload field, an Invariant CRC field and a Variant CRC field; the OpCode in the Base Transport Header field of the request-side write request uses a self-defined preset encoding, so that the response device can identify the operation type of the request-side write request as a read request converted into a write request according to the self-defined preset encoding.

3. The requesting apparatus according to claim 1, wherein The device further comprises a counter, The counter is configured to record the number of received response-side write requests that are in order and have not been reported to the driver when the receiving processor receives the response-side write request sent by the response device; The receiving processor is further configured to, when receiving a response-side Ack signal sent by the response device, if the work queue element corresponding to the response-side Ack signal is an RDMA read operation, and the number of response-side write requests recorded by the counter is greater than or equal to the number of response-side write requests expected to be received by the RDMA read operation, generate a completion queue element corresponding to the RDMA read operation, and send the completion queue element to the driver.

4. The requesting apparatus according to claim 3, wherein The sending processor is further configured to store the request-side write request to a request-side sending linked list; wherein the request-side sending linked list stores each request-side write request in sending order; The receiving processor is further configured to update the request-side sending linked list according to the response-side write request, and generate a request queue doorbell; The sending processor is further configured to generate a request-side Ack signal according to the request queue doorbell, and send the request-side Ack signal to the response device to notify the response device that the response-side write request sent by the response device has been received by the request device.

5. The requesting apparatus according to claim 4, wherein The request-side write request, the response-side write request, the request-side Ack signal and the response-side Ack signal all carry sequence numbers, the sequence number carried by the request-side write request and the sequence number carried by the response-side write request are independent of each other, the sequence number carried by the request-side Ack signal is the same as the sequence number carried by the response-side write request corresponding to the request-side Ack signal, and the sequence number carried by the response-side Ack signal is the same as the sequence number carried by the request-side write request corresponding to the response-side Ack signal.

6. The request device of claim 1, wherein, The sending processor is further configured to read the request-side write request from the request-side sending chain table and resend it to the response device when the response-side Ack signal corresponding to the request-side write request is not successfully received by the receiving processor within a preset time.

7. An RDMA-based response apparatus, wherein, The response device comprises a receiving processor, a sending processor and a data storage structure. The receiving processor is configured to receive a request-side write request sent by a request device, wherein the request-side write request is a write request converted from a read operation request. The receiving processor is further configured to parse the request-side write request to obtain a source data address, a destination address and a data length, write the source data address, the destination address and the data length into the data storage structure, and generate a request queue doorbell. The sending processor is configured to read the source data address, the destination address and the data length from the data storage structure according to the request queue doorbell, read target data according to the source data address and the data length, generate a response-side write request according to the target data and the destination address, and send the response-side write request to the request device, so that the request device writes the target data into a storage space corresponding to the destination address according to the response-side write request.

8. The response device of claim 7, wherein, The response-side write request comprises a Local Routing Header field, a Global Routing Header field, a Base Transport Header field, an Extended Transport Header field, a Message payload field, an Invariant CRC field and a Variant CRC field; and a custom preset encoding is used for an OpCode in the Base Transport Header field of the response-side write request, so that the request device can identify an operation type of the response-side write request according to the custom preset encoding.

9. The response device of claim 7, wherein, The receiving processor is further configured to parse the request-side write request to obtain a sequence number and write the sequence number into the data storage structure. The sending processor is further configured to read the sequence number from the data storage structure according to the request queue doorbell, generate a response-side Ack signal according to the sequence number, and send the response-side Ack signal to the request device to notify the request device that the request-side write request has been received.

10. The response device of claim 9, wherein, The sending processor is further configured to store the response-side write request into a response-side sending chain table; and the response-side sending chain table stores each response-side write request in a sending order. The receiving processor is further configured to respond to a retransmitted request-side write request of the request device, retransmit a corresponding response-side write request stored in the response-side sending chain table to the request device according to the retransmitted request-side write request, and send a response-side Ack signal to the request device to notify the request device that the retransmitted request-side write request has been received.

11. The response device of claim 9, wherein, The receiving processor is further configured to update the response-side sending chain table according to a request-side Ack signal sent by the requesting device in response to the request-side Ack signal. The sending processor is further configured to retransmit the response-side write request to the requesting device when the receiving processor fails to successfully receive the request-side Ack signal within a preset time.

12. The responsive device of claim 7, wherein, The request-side write request, the response-side write request, the request-side Ack signal, and the response-side Ack signal each carries a sequence number, the sequence number carried by the request-side write request and the sequence number carried by the response-side write request are independent of each other, the sequence number carried by the request-side Ack signal is the same as the sequence number carried by the response-side write request corresponding to the request-side Ack signal, and the sequence number carried by the response-side Ack signal is the same as the sequence number carried by the request-side write request corresponding to the response-side Ack signal.

13. An RDMA-based system, wherein, The application comprises: The requesting device according to any one of claims 1-6 and the responding device according to any one of claims 7-12.

Images