RDMA-based request device, response device and system

Abstract

The invention relates to a request device and a response device based on RDMA (Remote Direct Memory Access). The request device comprises a sending processor and a receiving processor; the sending processor is used for receiving a work queue element issued by a driving program, and if the work queue element is an RDMA read operation, the RDMA read operation is converted to obtain a request side write request; the sending processor is also used for sending the request side write request to the response device; and the receiving processor is used for receiving and analyzing a response side write request sent by the response device to obtain target data and a destination address corresponding to the RDMA read operation, and writing the target data into a storage space corresponding to the destination address. Through the application, the QPC data structure domain segment and the design difficulty are reduced, the message disorder processing complexity is reduced, and the consumption of a large amount of host memory and cache resources is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of remote direct memory access (RDMA), and in particular to a request device, a response device and a system based on RDMA. Background Art

[0002] like Figure 1 This is a schematic diagram of the data reading method currently used based on RDMA. Figure 1 The method comprises the following steps:

[0003] (1) The driver sends a work queue element (WQE) to the send queue (SQ) of the requester. The WQE is an RDMA read operation, which is used to instruct the requester to request the responder to read data.

[0004] (2) The requester's transmit processor (TXP) converts the WQE into an RDMA read request (RDMA readrequest) and writes the relevant information of the received read response into the client-side RDMA read list (Client RDMA readlist, CRRL), such as the starting packet sequence number (PSN) and the SGE where the read response is placed;

[0005] (3) The requester's TXP sends the RDMA read request to the responder through the network;

[0006] (4) The receiver (RXP) of the responder receives the RDMA read request, parses the RDMA read request to obtain the relevant information for generating a read response, writes the relevant information for generating the read response into the SRRL (server-side RDMA read linked list), such as RETH (Read Encyclopedia Table Header, a data structure), and generates a request queue doorbell (Receive Queue Doorbell, RQDB) to the responder's TXP;

[0007] (5) RQDB is sent from responder's RXP to responder's TXP;

[0008] (6) The responder's TXP receives the RQDB, reads the relevant information for generating a read response from the server-side RDMA read list (Service RDMA readlist, SRRL), and generates a read response based on the relevant information;

[0009] (7) The responder's TXP sends the Read response to the requester's RXP through the network;

[0010] (8) The requester's RXP receives the Read response, reads the relevant information of the received Read response in the CRRL, writes the data of the Read response into the corresponding buffer according to the relevant information, and generates a CQE (CQE is a completion queue element) to the SQ;

[0011] (9) Return CQE to the driver and the RDMA read operation is completed.

[0012] The above method has the following technical problems:

[0013] Each queue pair (QP) of the requester needs to have a CRRL data structure, and each QP of the responder needs to have an SRRL data structure. For example, each entry of the CRRL data structure is 512 bits and has a depth of 128; each entry of the SRRL data structure is 256 bits and has a depth of 128. If 1M QPs are supported, 12GB of memory resources are required, so a large amount of memory and cache resources are consumed. In addition, since an RDMA read operation may need to generate multiple Read responses, multiple Read responses may arrive in disorder, and the ReadresponsePSNs of different messages are discontinuous, the Read response disorder processing is more complicated. Moreover, each QP has a Queue Pair Context (QPC). When the number of QPs is large, many related QPCs need to be maintained, resulting in a large QPC bit width and a corresponding increase in complexity. Summary of the invention

[0014] The purpose of this application is to propose a request device, a response device and a system based on RDMA to solve the above technical problems.

[0015] To achieve the above object, according to a first aspect of the present application, a request device based on RDMA is provided, the request device comprising a sending processor and a receiving processor;

[0016] The sending processor is used to receive a work queue element sent by the driver, and if the work queue element is an RDMA read operation, convert the RDMA read operation to obtain a request profile write request;

[0017] The sending processor is further used to send the request profiling request to the response device;

[0018] The receiving processor is used to receive and parse the response profile writing request sent by the response device to obtain the target data and destination address corresponding to the RDMA read operation, and write the target data into the storage space corresponding to the destination address.

[0019] According to a second aspect of the present application, there is also provided an RDMA-based response device, the response device comprising a receiving processor, a sending processor and a data storage structure;

[0020] The response device includes a receiving processor, a sending processor and a data storage structure;

[0021] The receiving processor is used to receive a request profile writing request sent by a requesting device; wherein the request profile writing request is obtained by the requesting device converting the RDMA read operation when the work queue element sent by the driver to the requesting device is an RDMA read operation;

[0022] The receiving processor is further used to parse the request profile writing request to obtain a source data address, a destination address and a data length, write the source data address, the destination address and the data length into the data storage structure, and generate a request queue doorbell;

[0023] The sending processor is used to read the source data address, destination address and data length from the data storage structure according to the request queue doorbell, read the target data according to the source data address and data length, generate a response profile request according to the target data and destination address, and send the response profile request to the requesting device, so that the requesting device writes the target data into the storage space corresponding to the destination address according to the response profile request.

[0024] According to a third aspect of the present application, there is also provided an RDMA-based system, comprising: a requesting device according to the first aspect and a responding device according to the second aspect.

[0025] The above RDMA-based request device, response device and system have the following beneficial effects:

[0026] The request device and the response device are used in conjunction with each other. The same device can have the functions of the above-mentioned request device and response device at the same time. The RDMA read operation sent by the driver to the request device is converted into a request side write request, so that there is no need to set up a data storage structure (CRRL) in the request device to store the relevant information of the received read response. Since the read target data is directly written by the response device into the storage space corresponding to the destination address in the request device, the request device does not need to perform complex disorder processing and re-arrangement logic. Moreover, after converting the read operation into a write operation, each write operation is independent, and there is no need to track multiple unfinished read operations, thereby reducing the maintenance of QPC. In summary, it can avoid consuming a large amount of memory and cache resources.

[0027] Other features and advantages of the above-mentioned RDMA-based request device, response device and system will be described in the subsequent description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 The figure is a flow chart of the interaction between a traditional RDMA-based requesting device and a responding device.

[0030] Figure 2 The present invention is a flowchart of an interaction between a request device and a response device based on RDMA in an embodiment of the present application.

[0031] Figure 3 This is a schematic diagram of the message structure of an RDMA write request based on the IB protocol in an embodiment of the present application.

[0032] Figure 4 A schematic diagram of the message structure of a profiling request in an embodiment of the present application.

[0033] Figure 5 Schematic diagram of the message structure of the RDMA read response based on the IB protocol in an embodiment of the present application.

[0034] Figure 6 A schematic diagram of the message structure for responding to a profiling request in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The detailed description of the accompanying drawings is intended as an illustration of the currently preferred embodiment of the present application, and is not intended to represent the only form in which the present application can be implemented. It should be understood that the same or equivalent functions can be accomplished by different embodiments intended to be included in the spirit and scope of the present application.

[0036] An embodiment of the present application provides a request device based on RDMA. Figure 2 The working principle of the request device is shown in FIG. Figure 2 The requesting device (requester) includes a transmitting processor (TXP) and a receiving processor (RXP).

[0037] The sending processor is used to receive a work queue element sent by the driver, and if the work queue element is an RDMA read operation, convert the RDMA read operation to obtain a request profile write request.

[0038] Specifically, in RDMA, a work queue element (WQE) is an instruction sent by the driver to the requesting device to perform a specific network operation. In this embodiment, an RDMA read operation is to be implemented. Because WQE contains all the information required to perform an RDMA read operation, both the requesting device and the responding device are provided with a network interface card (NIC) to implement RDMA communication.

[0039] Specifically, the sending processor (located in the NIC) of the requesting device receives the WQE sent by the driver. This WQE is used to perform an RDMA read operation, which contains information such as the source data address (i.e., the data location on the responding device), the destination address (i.e., the address in the local memory used to store the read data), and the data length. The sending processor converts this WQE into a request side write request (RDMA read convert write), so that the original RDMA read operation is changed into a write operation. The request side write request contains information such as the source data address, the destination address, and the data length, which indicates that the responding device should read the target data from the source data address and write the target data to the destination address of the requesting device.

[0040] The sending processor is also used to send the request profile writing request to the response device, so that the response device reads the target data according to the source data address and data length, and generates a response profile writing request according to the target data, destination address and data length.

[0041] Specifically, the sending processor of the requesting device sends the converted request profile writing request to the responding device through the network to notify the responding device that the requesting device hopes to read data of a specific length from the source data address of the responding device and write it to the destination address of the requesting device. When the network interface card (NIC) of the responding device receives the request profile writing request, the receiving processor of the responding device will parse the request and read the target data according to the source data address and data length information in the request. The read target data refers to the data that the requesting device hopes to obtain, and the data is stored in the memory of the responding device. Then, the sending processor of the responding device will generate a response profile writing request (read response convert write) according to the read target data and the destination address in the request profile writing request, as well as the data length information. The sending processor of the responding device returns the response profile writing request to the requesting device, so as to write the read target data to the destination address of the requesting device, thereby realizing the change of the RDMA read operation into the RDMA write operation.

[0042] The receiving processor is used to receive and parse the response profile writing request sent by the response device to obtain the target data and destination address corresponding to the RDMA read operation, and write the target data into the storage space corresponding to the destination address.

[0043] Specifically, when the receiving processor of the requesting device (located in the network interface card NIC of the requesting device) receives a response profiling request, the receiving processor parses the response profiling request, extracts the target data and destination address information therefrom, and after extracting the target data and destination address, writes the target data into the storage space corresponding to the destination address in the memory of the requesting device. This means that the receiving processor of the requesting device will directly place the data read from the responding device into the memory location specified by the requesting device to complete the data reception.

[0044] In summary, the request device and the response device of this embodiment are used in conjunction with each other, and the same device can have the functions of the above-mentioned request device and the response device at the same time. The RDMA read operation sent by the driver to the request device is converted into a request side write request, so that there is no need to set a data storage structure (CRRL) in the request device to store the relevant information of the received read response. Since the read target data is directly written by the response device into the storage space corresponding to the destination address in the request device, the request device does not need to perform complex disorder processing and re-arrangement logic. Moreover, after the read operation is converted into a write operation, each write operation is independent, and there is no need to track multiple unfinished read operations, thereby reducing the maintenance of QPC. In summary, it can avoid consuming a large amount of memory and cache resources.

[0045] In some embodiments, the request profile writing request includes a Local Routing Header field, a GlobalRouting Header field, a Base Transport Header field, a Message payload field, an InvariantCRC field, and a Variant CRC field; the OpCode in the Base Transport Header field of the request profile writing request uses a custom preset code so that the responding device can identify the operation type of the request profile writing request as a write request converted from a read request according to the custom preset code.

[0046] Specifically, if Figure 3 The figure shows the message structure of the traditional RDMA write request based on the IB protocol. Figure 4 FIG. 1 shows the message structure of the request profile writing request customized in this embodiment. In traditional RDMA communication, the sending processor of the requesting device converts the RDMA read operation sent by the driver to the SQ into a read request, but this will cause the technical problems mentioned in the background technology. Therefore, in this embodiment, the sending processor of the requesting device converts the RDMA read operation sent by the driver to the SQ into Figure 4 The request profile write request of the message structure shown in the figure is different from the message structure of the traditional RDMA write request based on the IB protocol. Figure 3 The OpCode (operation code) in the Base TransportHeader of the message structure uses the code defined by the IB protocol, and the Figure 4 The OpCode (operation code) in the Base Transport Header of the message structure shown does not use the code already used by the IB protocol, but uses a custom preset code; for example, for an RC type reset, the OpCode can be 8'b000_11000 to implement the conversion of the RDMA read operation into a write operation proposed in this embodiment. In addition, Figure 4 The message payload of the message structure shown is the information of the original RDMA read operation sent by the driver, including but not limited to the address of the source data, the length of the source data, the storage address after the data is returned, etc.; Figure 4 The remaining fields of the message structure are the same as Figure 3 The contents of the same fields in the message structure are the same, so they will not be repeated here.

[0047] In some embodiments, the apparatus further comprises a counter;

[0048] The counter is used to record the number of response profiling requests received that are sequence-preserving and not reported to the driver when the receiving processor receives the response profiling request sent by the response device;

[0049] The receiving processor is also used to, when receiving a response side Ack signal sent by the response device, generate a completion queue element corresponding to the RDMA read operation if the work queue element corresponding to the response side Ack signal is an RDMA read operation, and the number of response side write requests recorded by the counter is greater than or equal to the number of response side write requests expected to be received by the RDMA read operation, and send the completion queue element to the driver.

[0050] Specifically, the receiving processor will check the number of response side write requests recorded by the counter only when it receives the response side Ack signal. Under normal circumstances, when an RDMA read operation is completed, the number of response side write requests recorded by the counter should be equal to the number of response side write requests expected to be received by the RDMA read operation. However, in special circumstances, the number of response side write requests recorded by the counter will be greater than the number of response side write requests expected to be received by the RDMA read operation. For example, in the case of packet loss and retransmission, as long as the number of response side write requests recorded by the counter is greater than or equal to the number of response side write requests expected to be received by the RDMA read operation when the response side Ack signal is received for inspection, it can be determined that the read data has been collected and the RDMA read operation has been completed. At this time, a completion queue element corresponding to the RDMA read operation can be generated and sent to the driver to notify the driver. At the same time, the number of response side write requests recorded by the counter needs to be updated.

[0051] In some embodiments, the sending processor is further used to store the request profile writing request into a request side sending linked list; wherein the request side sending linked list stores each request profile writing request in a sending order;

[0052] The receiving processor is further used to update the request side sending linked list according to the response profile writing request and generate a request queue doorbell;

[0053] The sending processor is also used to generate a request side Ack signal according to the request queue doorbell, and send the request side Ack signal to the responding device to notify the responding device that the response side writing request sent by it has been received by the requesting device.

[0054] Specifically, the receiving processor of the requesting device responds to the response side Ack signal sent by the responding device, indicating that the receiving processor of the responding device has successfully received the write request of the requesting side, and the receiving processor of the requesting device deletes the request side write request corresponding to the response side Ack signal from the request side sending linked list (TXC). The request side sending linked list stores each request side write request in the order of sending, and is used to track and manage incomplete request side write requests.

[0055] The sending processor of the requesting device generates a requesting side Ack signal according to the request queue doorbell. This signal is used to confirm that the requesting device has received the response profiling request. The sending processor of the requesting device sends the requesting side Ack signal to the responding device, which can be understood as the requesting device's confirmation of receipt of the response profiling request sent by the responding device.

[0056] In some embodiments, the request side write request, the response side write request, the request side Ack signal and the request side Ack signal all carry a serial number (PSN), the serial number carried by the request side write request and the serial number carried by the response side write request are independent of each other, the serial number carried by the request side Ack signal is the same as the serial number carried by its corresponding response side write request, and the serial number carried by the response side Ack signal is the same as the serial number carried by its corresponding request side write request.

[0057] Specifically, in traditional RDMA, since the PSNs of request messages and returned response messages are not continuous, a two-dimensional data structure of (MSN, IPSN) is generally used to describe the reception status of messages, where IPSN is the PSN inside the message, and each message starts from 0; this will increase the difficulty of processing out-of-order messages. In addition, the returned response message has responses with data and Ack signals without data; Ack signals and responses can aggregate the previous Ack signals, but Ack signals cannot aggregate the previous responses, which makes the order-keeping judgment of Ack signals and messages complicated. Aggregation means that a message can represent itself and the aggregated messages before it. To this end, the requesting device and the responding device of this embodiment are respectively configured with a set of independent PSNs, which are numbered according to their own sequence and are not affected by each other. For example, the PSN of the requesting device is PSN_a1~PSN_a100, and the PSN of the responding device is PSN_b1~PSN_b100. Therefore, the PSNs of the multiple request side writing requests sent sequentially by the requesting device are continuous, and the PSNs of the multiple response side writing requests sent sequentially by the responding device are also continuous. Moreover, the sequence number carried by the request side Ack signal is the same as the sequence number carried by its corresponding response side writing request, and the sequence number carried by the response side Ack signal is the same as the sequence number carried by its corresponding request side writing request. Therefore, the requesting device can perform order preservation operations based on the received response side writing request and the PSN carried by the response side Ack signal.

[0058] In some embodiments, the sending processor is also used to read the request side write request from the request side sending linked list and resend it to the response device when the receiving processor fails to successfully receive the response side Ack signal corresponding to the request side write request within a preset time.

[0059] Specifically, the sending processor of the requesting device stores the request profile writing request in the request side sending linked list, which stores each request profile writing request in the sending order and is used to track and manage unfinished requests. When the receiving processor of the requesting device fails to successfully receive the response side Ack signal corresponding to the request profile writing request within the preset time, it indicates that some error may have occurred, such as network failure, data transmission failure, etc. In this case, the sending processor of the requesting device will read the request profile writing request that has not been successfully received from the request side sending linked list, and then the sending processor of the requesting device will resend the request profile writing request to the responding device. The purpose of doing so is to allow the responding device to resend the response profile writing request corresponding to the request profile writing request. Through this mechanism, even if data transmission fails, RDMA communication can automatically try to retransmit the unsuccessful request, thereby improving the reliability of data transmission and the robustness of the system. This error handling mechanism is another important component in RDMA communication to ensure the reliability of data transmission.

[0060] Another embodiment of the present application provides a response device based on RDMA, such as Figure 2 As shown, the responder includes a receiving processor (RXP), a transmitting processor (TXP) and a data storage structure (SRRL).

[0061] The receiving processor is used to receive a request profile write request sent by a requesting device; wherein the request profile write request is a write request converted from a read operation request;

[0062] Specifically, the receiving processor (located in the NIC) of the responding device receives the request side write request (RDMA read convert write) sent by the requesting device. The request side write request is obtained by the requesting device through converting the work queue element (WQE) of the original RDMA read operation sent by the driver. This conversion means that the original RDMA read operation is changed into a write operation. The request side write request contains information such as the source data address, the destination address and the data length. This information indicates how much data the responding device should read from the source data address and write this data to the destination address of the requesting device.

[0063] The receiving processor is also used to parse the request profile writing request to obtain the source data address, destination address and data length, write the source data address, destination address and data length into the data storage structure, and generate a request queue doorbell (Receive Queue Doorbell, RQDB).

[0064] Specifically, the receiving processor of the responding device parses the request profile writing request and extracts information such as the source data address, the destination address and the data length. These information are key parameters required to perform RDMA operations. They indicate where the responding device reads data from and where the data is written. After extracting these information, the receiving processor of the responding device writes these information into the data storage structure (SRRL) of the responding device. This data storage structure is a data structure specifically used to store and track the RDMA operation status. Then, the receiving processor of the responding device generates an RQDB to notify the sending processor of the responding device that there is new data to be processed in the request queue. Through the RQDB, the sending processor of the responding device can know which data should be read from the data storage structure in order to generate the response profile writing request.

[0065] The sending processor is used to read the source data address, destination address and data length from the data storage structure according to the request queue doorbell, read the target data according to the source data address and data length, generate a response profile request according to the target data and destination address, and send the response profile request to the requesting device, so that the requesting device writes the target data into the storage space corresponding to the destination address according to the response profile request.

[0066] Specifically, the sending processor of the responding device receives the RQDB and reads the source data address, destination address and data length from the data storage structure. This information tells the sending processor where to read the data and where to write the data. Based on this information, the sending processor of the responding device reads the target data from the source data address. These target data are the data that the responding device needs to send back to the requesting device. Next, the sending processor of the responding device generates a response profile write request (read response convert write) based on the target data and the destination address. This response profile write request contains the target data and the destination address information, indicating the memory address to which the requesting device should write the target data. Finally, the sending processor sends this response profile write request back to the requesting device through the network, so that the requesting device can write the target data into the storage space corresponding to the destination address according to this response profile write request.

[0067] In summary, the response device of this embodiment is used in conjunction with the request device of the above embodiment. The same device can have the functions of the request device of the above embodiment and the response device of this embodiment at the same time. The RDMA read operation sent by the driver to the request device is converted into a request side write request, so that there is no need to set a data storage structure (CRRL) in the request device to store the relevant information of receiving the read response. Since the read target data is directly written by the response device into the storage space corresponding to the destination address in the request device, the request device does not need to perform complex disorder processing and re-arrangement logic. Moreover, after the read operation is converted into a write operation, each write operation is independent, and there is no need to track multiple unfinished read operations, thereby reducing the maintenance of QPC. In summary, it can avoid consuming a large amount of memory and cache resources.

[0068] In some embodiments, the response profiling request includes a Local Routing Header field, a GlobalRouting Header field, a Base Transport Header field, an Extended Transport Header field, a Message payload field, an Invariant CRC field, and a Variant CRC field; the OpCode in the BaseTransport Header field of the response profiling request uses a custom preset code so that the requesting device can identify the operation type of the response profiling request as a response to the request profiling request according to the custom preset code.

[0069] Specifically, if Figure 5 The figure shows the message structure of the traditional RDMA read response based on the IB protocol. Figure 6 FIG. 1 shows the message structure of the response profile writing request customized in this embodiment. In the traditional RDMA communication, the sending processor of the responding device will generate a read response for the read data to be returned, but this will cause the technical problems mentioned in the background technology. Therefore, in this embodiment, the sending processor of the requesting device will convert the RDMA read operation sent by the driver to the SQ into Figure 4 The request profile writing request of the message structure shown in the figure, the response device reads the data according to the request profile writing request and generates a Figure 6 The message structure of the response profile writing request shown in the message structure of the embodiment is different from the message structure of the traditional RDMA read response based on the IB protocol. Figure 5 The OpCode (operation code) in the Base Transport Header of the message structure uses the code defined by the IB protocol, and the Figure 6The OpCode (operation code) in the Base TransportHeader of the message structure shown does not use the code already used by the IB protocol, but uses a custom preset code; for example, for an RC type reset, the OpCode may be 8'b000_11001, to implement the conversion of the RDMA read operation into a write operation proposed in this embodiment. In addition, Figure 6 The Extended Transport Header of the message structure shown is the same as RETH, where VA, R_KEY, and DMALen are obtained from SRRL. The source comes from the request profiling request sent by the requesting device. Figure 6 The remaining fields of the message structure are the same as Figure 5 The contents of the same fields in the message structure are the same, so they will not be repeated here.

[0070] In some embodiments, the receiving processor parses the request profile writing request to further obtain a sequence number (PSN), and writes the sequence number into the data storage structure;

[0071] The sending processor is also used to read the serial number (PSN) from the data storage structure according to the request queue doorbell, generate a response side Ack signal according to the serial number, and after sending the response side write request to the requesting device, send the response side Ack signal to the requesting device to notify the requesting device that the request side write request has been received.

[0072] Specifically, the receiving processor of the responding device parses the request side write request and extracts the PSN therefrom. The PSN is an identifier used to identify the request side write request. The receiving processor writes the extracted PSN into the data storage structure of the responding device. The sending processor of the responding device receives the RQDB and reads the PSN from the data storage structure. The sending processor of the responding device generates a response side Ack signal based on the PSN. The response side Ack signal is the response device's confirmation of receipt of the data sent by the requesting device, indicating that the request side write request corresponding to the PSN has been received. After sending the response side write request to the requesting device, the sending processor of the responding device sends the response side Ack signal to the requesting device to notify the requesting device that the responding device has received the request side write request. Based on the above method, both parties in RDMA communication can ensure reliable transmission and sequential processing of data.

[0073] In some embodiments, the sending processor is further used to store the response profiling request in a response side sending linked list; wherein the response side sending linked list stores each response profiling request in a sending order;

[0074] The receiving processor is also used to respond to the request side writing request retransmitted by the requesting device, and retransmit the corresponding response side writing request stored in the response side sending linked list to the requesting device according to the retransmitted request side writing request; after retransmitting the response side writing request, send a response side Ack signal to the requesting device to notify the requesting device that the retransmitted request side writing request has been received.

[0075] Specifically, the sending processor of the responding device stores the response profiling request in the response side sending linked list (TXC), which stores each response profiling request in the sending order and is used to track and manage unfinished responses. The responding device responds to the request profiling request of the requesting device, and after sending a response profiling request to the requesting device, it will send a response side Ack signal to the requesting device to notify the requesting device that the request profiling request sent by it has been received by the responding device. This response side Ack signal should be received by the requesting device within a preset time. If the requesting device does not receive this response side Ack signal within the preset time, it means that some kind of error may have occurred, such as network failure, data transmission failure, etc., and the requesting device will retransmit the request profiling request to the responding device. When the receiving processor of the responding device receives the request profiling request, it will generate an RQDB to instruct the sending processor to retransmit the request profiling request. The sending processor reads the corresponding response profiling request from the response side sending linked list according to this RQDB, and the sending processor of the responding device will resend this response profiling request to the requesting device. Through the above mechanism, even if data transmission fails, RDMA communication can automatically try to retransmit the unsuccessful response, thereby improving the reliability of data transmission and the robustness of the system. This specific error handling mechanism is another important component in RDMA communication to ensure data transmission reliability.

[0076] In some embodiments, the receiving processor is further used to respond to the request side Ack signal sent by the requesting device, and update the response side sending linked list according to the request side Ack signal.

[0077] Specifically, the receiving processor of the responding device responds to the request side Ack signal sent by the requesting device, which means that the requesting device has successfully received the response profile writing request. The receiving processor of the responding device deletes the response profile writing request corresponding to the request side Ack signal from the response side sending linked list based on the request side Ack signal. The response side sending linked list stores each response profile writing request in the order of sending, and is used to track and manage incomplete responses.

[0078] The sending processor is further configured to retransmit the response profile writing request to the requesting device when the receiving processor fails to successfully receive the requesting side Ack signal within a preset time.

[0079] Specifically, when the receiving processor of the responding device fails to successfully receive the request side Ack signal corresponding to the response profile writing request it has sent within the preset time, it indicates that some kind of error may have occurred, such as network failure, data transmission failure, etc. In this case, the receiving processor of the responding device will generate an RQDB to instruct the sending processor to retransmit the request profile writing request. The sending processor reads the corresponding response profile writing request from the response side sending linked list according to this RQDB, and the sending processor of the responding device will resend the response profile writing request to the requesting device. After resending the response profile writing request, if the requesting device successfully receives the retransmitted response profile writing request, the requesting device will send a request side Ack signal to notify the responding device that it has received the retransmitted request profile writing request from the responding device. Through the above mechanism, even if data transmission fails, RDMA communication can automatically try to retransmit the unsuccessful response, thereby improving the reliability of data transmission and the robustness of the system. This specific error handling mechanism is another important component in RDMA communication to ensure the reliability of data transmission. 。

[0080] In some embodiments, the request side write request, the response side write request, the request side Ack signal and the request side Ack signal all carry serial numbers, the serial number carried by the request side write request and the serial number carried by the response side write request are independent of each other, the serial number carried by the request side Ack signal is the same as the serial number carried by its corresponding response side write request, and the serial number carried by the response side Ack signal is the same as the serial number carried by its corresponding request side write request.

[0081] Specifically, in traditional RDMA, since the PSNs of request messages and returned response messages are not continuous, a two-dimensional data structure of (MSN, IPSN) is generally used to describe the reception status of messages, where IPSN is the PSN inside the message, and each message starts from 0; this will increase the difficulty of processing out-of-order messages. In addition, the returned response message has responses with data and Ack signals without data; Ack signals and responses can aggregate the previous Ack signals, but Ack signals cannot aggregate the previous responses, which makes the order-keeping judgment of Ack signals and messages complicated. Aggregation means that a message can represent itself and the aggregated messages before it. To this end, the requesting device and the responding device of this embodiment are respectively configured with a set of independent PSNs, which are numbered according to their own sequence and are not affected by each other. For example, the PSN of the requesting device is PSN_a1~PSN_a100, and the PSN of the responding device is PSN_b1~PSN_b100. Therefore, the PSNs of the multiple request side writing requests sent sequentially by the requesting device are continuous, and the PSNs of the multiple response side writing requests sent sequentially by the responding device are also continuous. Moreover, the sequence number carried by the request side Ack signal is the same as the sequence number carried by its corresponding response side writing request, and the sequence number carried by the response side Ack signal is the same as the sequence number carried by its corresponding request side writing request. Therefore, the requesting device can perform order preservation operations based on the received response side writing request and the PSN carried by the response side Ack signal.

[0082] Another embodiment of the present application provides a system based on RDMA, such as Figure 2 As shown, the system includes: a request device as described in the above embodiment and a response device as described in the above embodiment. The working principle of the system of this embodiment can be obtained by referring to the contents of the request device as described in the above embodiment and the response device as described in the above embodiment, so it will not be repeated here.

[0083] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many updates and modifications will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A request device based on RDMA, characterized in that: The requesting device includes a sending processor and a receiving processor; The sending processor is used to receive a work queue element sent by the driver, and if the work queue element is an RDMA read operation, convert the RDMA read operation to obtain a request profile write request; The sending processor is further used to send the request profiling request to the response device; The receiving processor is used to receive and parse the response profile writing request sent by the response device to obtain the target data and destination address corresponding to the RDMA read operation, and write the target data into the storage space corresponding to the destination address.

2. The RDMA-based request device according to claim 1, characterized in that: The request profile writing request includes a Local Routing Header field, a Global Routing Header field, a Base Transport Header field, a Message payload field, an Invariant CRC field and a Variant CRC field; the OpCode in the BaseTransport Header field of the request profile writing request uses a custom preset code so that the response device can identify the operation type of the request profile writing request as a write request converted from a read request according to the custom preset code.

3. The request device according to claim 1, characterized in that: The device also includes a counter, The counter is used to record the number of response profiling requests received that are sequence-preserving and not reported to the driver when the receiving processor receives the response profiling request sent by the response device; The receiving processor is also used to, when receiving a response side Ack signal sent by the response device, generate a completion queue element corresponding to the RDMA read operation if the work queue element corresponding to the response side Ack signal is an RDMA read operation, and the number of response side write requests recorded by the counter is greater than or equal to the number of response side write requests expected to be received by the RDMA read operation, and send the completion queue element to the driver.

4. The request device according to claim 3, characterized in that: The sending processor is further used to store the request profile writing request into a request side sending linked list; wherein the request side sending linked list stores each request profile writing request in a sending order; The receiving processor is further used to update the request side sending linked list according to the response profile writing request and generate a request queue doorbell; The sending processor is also used to generate a request side Ack signal according to the request queue doorbell, and send the request side Ack signal to the responding device to notify the responding device that the response side writing request sent by it has been received by the requesting device.

5. The request device according to claim 4, characterized in that: The request side write request, the response side write request, the request side Ack signal and the request side Ack signal all carry serial numbers. The serial number carried by the request side write request and the serial number carried by the response side write request are independent of each other. The serial number carried by the request side Ack signal is the same as the serial number carried by its corresponding response side write request, and the serial number carried by the response side Ack signal is the same as the serial number carried by its corresponding request side write request.

6. The request device according to claim 1, characterized in that: The sending processor is also used to read the request side write request from the request side sending linked list and resend it to the response device when the receiving processor fails to successfully receive the response side Ack signal corresponding to the request side write request within a preset time.

7. A response device based on RDMA, characterized in that: The response device includes a receiving processor, a sending processor and a data storage structure; The receiving processor is used to receive a request profile write request sent by a requesting device; wherein the request profile write request is a write request converted from a read operation request; The receiving processor is further used to parse the request profile writing request to obtain a source data address, a destination address and a data length, write the source data address, the destination address and the data length into the data storage structure, and generate a request queue doorbell; The sending processor is used to read the source data address, destination address and data length from the data storage structure according to the request queue doorbell, read the target data according to the source data address and data length, generate a response profile request according to the target data and destination address, and send the response profile request to the requesting device, so that the requesting device writes the target data into the storage space corresponding to the destination address according to the response profile request.

8. The response device according to claim 7, characterized in that: The response profiling request includes a LocalRouting Header field, a Global Routing Header field, a Base Transport Header field, an Extended Transport Header field, a Message payload field, an Invariant CRC field, and a Variant CRC field; the OpCode in the Base Transport Header field of the response profiling request uses a custom preset code so that the requesting device can identify the operation type of the response profiling request according to the custom preset code.

9. The response device according to claim 7, characterized in that: The receiving processor parses the request profile writing request and obtains a serial number, and writes the serial number into the data storage structure; The sending processor is also used to read the serial number from the data storage structure according to the request queue doorbell, generate a response side Ack signal according to the serial number, and send the response side Ack signal to the requesting device to notify the requesting device that the request side write request has been received.

10. The response device according to claim 9, characterized in that: The sending processor is further used to store the response profiling request into a response side sending linked list; wherein the response side sending linked list stores each response profiling request in a sending order; The receiving processor is also used to respond to the request side writing request retransmitted by the requesting device, and retransmit the corresponding response side writing request stored in the response side sending linked list to the requesting device according to the retransmitted request side writing request; and send a response side Ack signal to the requesting device to notify the requesting device that the retransmitted request side writing request has been received.

11. The response device according to claim 9, characterized in that: The receiving processor is further used to respond to the request side Ack signal sent by the requesting device, and update the response side sending linked list according to the request side Ack signal; The sending processor is further configured to retransmit the response profile writing request to the requesting device when the receiving processor fails to successfully receive the requesting side Ack signal within a preset time.

12. The request device according to claim 7, characterized in that: The request side write request, the response side write request, the request side Ack signal and the request side Ack signal all carry serial numbers. The serial number carried by the request side write request and the serial number carried by the response side write request are independent of each other. The serial number carried by the request side Ack signal is the same as the serial number carried by its corresponding response side write request, and the serial number carried by the response side Ack signal is the same as the serial number carried by its corresponding request side write request.

13. A system based on RDMA, characterized in that: include: The requesting device as claimed in claims 1-6 and the responding device as claimed in claims 7-12.

Images