RDMA-based requester, RDMA-based responder, and RDMA-based system
By converting RDMA read operations into write operations and using request and response devices in conjunction, the problems of high memory and cache resource consumption and complex out-of-order processing are solved, achieving efficient data transmission and simplified QPC maintenance.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- SHENZHEN JAGUAR MICROSYSTEMS CO LTD
- Filing Date
- 2025-09-25
- Publication Date
- 2026-06-04
AI Technical Summary
Existing RDMA technology requires a large amount of memory and cache resources in the request and response devices, and out-of-order processing is complex, resulting in high QPC maintenance complexity.
By converting RDMA read operations into request-write requests and response-write requests, the request and response devices are used together to directly write the target data to the destination address, avoiding the need to set up data storage structures and perform out-of-order processing in the request device.
It reduces the consumption of memory and cache resources, simplifies out-of-order processing and QPC maintenance, and improves system reliability and efficiency.
Smart Images
Figure CN2025123993_04062026_PF_FP_ABST
Abstract
Description
A request device, response device, and system based on RDMA
[0001] Related applications
[0002] This application claims priority to Chinese patent application filed on September 27, 2024, with application number 202411363358.0 and entitled "A Request Apparatus, Response Apparatus and System Based on RDMA", the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to the field of Remote Direct Memory Access (RDMA) technology, and more specifically to an RDMA-based request device, response device, and system. Background Technology
[0004] Figure 1 is a schematic diagram of the current RDMA-based data reading method. The method in Figure 1 includes the following steps:
[0005] (1) The driver sends a work queue element (WQE) to the send queue (SQ) of the requester; the WQE is specifically an RDMA read operation, which instructs the requester to request data from the responder.
[0006] (2) The requester’s sending processor (TXP) converts the WQE into an RDMA read request and writes the relevant information of the received read response into the client-side RDMA read list (CRRL), such as the packet sequence number (PSN) and the SGE where the read response is placed.
[0007] (3) The requester’s TXP sends the RDMA read request to the responder over the network;
[0008] (4) When the receiver (RXP) of the responder receives the RDMA read request, it parses the RDMA read request to obtain the relevant information for generating the read response, and writes the relevant information for generating the read response into the SRRL (server-side RDMA read list), such as RETH (Read Encyclopedia Table Header, a data structure), and generates the Receive Queue Doorbell (RQDB) of the TXP of the responder.
[0009] (5) RQDB is sent from the responder's RXP to the responder's TXP.
[0010] (6) When the responder’s TXP receives the RQDB, it reads the relevant information for generating a read response from the server-side RDMA read list (SRRL) and generates a read response based on the relevant information.
[0011] (7) The responder’s TXP sends the Read response over the network to the requester’s RXP.
[0012] (8) When the requester’s RXP receives the Read response, it reads the relevant information about receiving the read response in the CRRL, writes the data of the Read response into the corresponding buffer according to the relevant information, and generates a CQE to the SQ (CQE is the completion queue element).
[0013] (9) Return CQE to the driver, and the RDMA read operation is complete.
[0014] The above method has the following technical problems:
[0015] Each queue pair (QP) of the requester needs a CRRL data structure, and each QP of the responder needs an SRRL data structure. For example, each entry in the CRRL data structure is 512 bits with a depth of 128; each entry in the SRRL data structure is 256 bits with a depth of 128. If 1 million QPs are supported, 12GB of memory resources are required, thus consuming a large amount of memory and cache resources. In addition, since an RDMA read operation may generate multiple read responses, these read responses may arrive out of order, and the ReadResponsePSNs of different messages are not contiguous, making out-of-order read response handling quite complex. Moreover, each QP has a queue pair context (QPC). When the number of QPs is large, many related QPCs need to be maintained, resulting in a large QPC bit width and a corresponding increase in complexity. Summary of the Invention
[0016] The purpose of this application is to provide an RDMA-based request device, response device, and system to solve the aforementioned technical problems.
[0017] To achieve the above objectives, according to a first aspect of this application, an RDMA-based request apparatus is provided, the request apparatus including a transmitting processor and a receiving processor;
[0018] The sending processor is used to receive work queue elements issued by the driver. If the work queue element is an RDMA read operation, it converts the RDMA read operation to obtain a request for side-writing.
[0019] The sending processor is further configured to send the request profiler request to the response device;
[0020] The receiving processor is configured to receive and parse the response write request sent by the response device to obtain the target data and destination address corresponding to the RDMA read operation, and write the target data into the storage space corresponding to the destination address.
[0021] According to a second aspect of this application, an RDMA-based response device is also provided, the response device including a receiving processor, a transmitting processor, and a data storage structure;
[0022] The response device includes a receiving processor, a transmitting processor, and a data storage structure;
[0023] The receiving processor is configured to receive a request for sidetracking sent by the requesting device; wherein, the request for sidetracking is obtained by the requesting device converting an RDMA read operation when the work queue element issued by the driver to the requesting device is an RDMA read operation;
[0024] The receiving processor is further configured to parse the request sidetracking request to obtain the source data address, destination address and data length, write the source data address, destination address and data length into the data storage structure, and generate a request queue doorbell;
[0025] The sending processor is configured to read the source data address, destination address, and data length from the data storage structure according to the request queue doorbell, read the target data according to the source data address and data length, generate a response profiler request according to the target data and destination address, and send the response profiler request to the requesting device, so that the requesting device writes the target data into the storage space corresponding to the destination address according to the response profiler request.
[0026] According to a third aspect of this application, an RDMA-based system is also provided, comprising: a requesting device according to the first aspect and a response device according to the second aspect.
[0027] The above-described RDMA-based request device, response device, and system have the following advantages:
[0028] The request and response devices work together, and a single device can simultaneously perform the functions of both. The RDMA read operation sent by the driver to the request device is converted into a request-side write request. This eliminates the need for a data storage structure (CRRL) in the request device to store information related to the received read response. Since the target data is directly written to the memory space corresponding to the destination address in the request device by the response device, the request device does not need to perform complex out-of-order processing and reordering logic. Furthermore, after converting the read operation into a write operation, each write operation is independent, eliminating the need to track multiple incomplete read operations. This reduces QPC maintenance and, in summary, avoids consuming large amounts of memory and cache resources.
[0029] Other features and advantages of the RDMA-based request device, response device and system described above will be set forth in the following description. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.
[0031] Figure 1 is a flowchart of the interaction between the traditional RDMA-based request and response devices.
[0032] Figure 2 is an interaction flowchart of a request device and a response device based on RDMA in an embodiment of this application.
[0033] Figure 3 is a schematic diagram of the message structure of an RDMA write request based on the IB protocol in an embodiment of this application.
[0034] Figure 4 is a schematic diagram of the message structure of the request for profiler in an embodiment of this application.
[0035] Figure 5 is a schematic diagram of the message structure of the RDMA read response based on the IB protocol in an embodiment of this application.
[0036] Figure 6 is a schematic diagram of the message structure in response to a profile request in an embodiment of this application. Detailed Implementation
[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0038] One embodiment of this application provides an RDMA-based request device. Figure 2 shows the working principle of the request device. Referring to Figure 2, the request device (requester) includes a transmitting processor (TXP) and a receiving processor (RXP).
[0039] The sending processor is used to receive work queue elements issued by the driver. If the work queue element is an RDMA read operation, the RDMA read operation is converted to obtain a request for sidetracking.
[0040] Specifically, in RDMA, a Work Queue Element (WQE) is an instruction issued by the driver to the requesting device to perform a specific network operation. In this embodiment, an RDMA read operation is to be implemented because the WQE contains all the information required to perform an RDMA read operation. Both the requesting device and the responding device are equipped with Network Interface Cards (NICs) to enable RDMA communication.
[0041] Specifically, the sending processor of the requesting device (located in the NIC) receives a WQE from the driver. This WQE is used to perform an RDMA read operation and contains information such as the source data address (i.e., the data location on the responding device), the destination address (i.e., the address in local memory used to store the read data), and the data length. The sending processor converts this WQE into a request write (RDMAread convert write), changing the original RDMA read operation into a write operation. The request write contains information such as the source data address, the destination address, and the data length, instructing the responding device to read the target data from the source data address and write the target data to the destination address of the requesting device.
[0042] The sending processor is further configured to send the request for profiling to the response device, so that the response device reads the target data according to the source data address and data length, and generates a response for profiling according to the target data, the destination address and the data length.
[0043] Specifically, the sending processor of the requesting device sends the converted request for writing to the responding device via the network, notifying the responding device that it wishes to read data of a specific length from the source data address of the responding device and write it to the destination address of the requesting device. When the network interface card (NIC) of the responding device receives this request for writing, its receiving processor parses the request and reads the target data based on the source data address and data length information in the request. The read target data refers to the data that the requesting device wishes to obtain, and this data is stored in the memory of the responding device. Next, the sending processor of the responding device generates a response for writing (read response convert write) based on the read target data, the destination address in the request for writing, and the data length information. The sending processor of the responding device returns this response for writing to the requesting device, thus converting the RDMA read operation into an RDMA write operation.
[0044] The receiving processor is configured to receive and parse the response write request sent by the response device to obtain the target data and destination address corresponding to the RDMA read operation, and write the target data into the storage space corresponding to the destination address.
[0045] Specifically, when the receiving processor of the requesting device (located in the network interface card (NIC) of the requesting device) receives the response profiler request, the receiving processor parses the response profiler request, extracts the target data and destination address information from it, and writes the target data into the memory of the requesting device corresponding to the destination address. This means that the receiving processor of the requesting device will directly place the data read from the response device into the memory location specified by the requesting device to complete the data reception.
[0046] In summary, the request and response devices in this embodiment are used in conjunction, and the same device can simultaneously possess the functions of both the request and response devices. The RDMA read operation issued by the driver to the request device is converted into a request-side write request, eliminating the need to set up a data storage structure (CRRL) in the request device to store information related to receiving the read response. Since the target data to be read is directly written by the response device to the storage space corresponding to the destination address in the request device, the request device does not need to perform complex out-of-order processing and reordering logic. Moreover, after converting the read operation into a write operation, each write operation is independent, eliminating the need to track multiple incomplete read operations, thus reducing QPC maintenance. In summary, this avoids consuming large amounts of memory and cache resources.
[0047] In some embodiments, the request profiler includes a Local Routing Header field, a Global Routing Header field, a Base Transport Header field, a Message payload field, an Invariant CRC field, and a Variant CRC field; the OpCode in the Base Transport Header field of the request profiler uses a custom preset encoding so that the response device can identify the operation type of the request profiler as a write request converted from a read request based on the custom preset encoding.
[0048] Specifically, Figure 3 shows the message structure of a traditional RDMA write request based on the IB protocol, and Figure 4 shows the message structure of a custom write request in this embodiment. In traditional RDMA communication, the sending processor of the requesting device converts the RDMA read operation sent by the driver to the SQ into a read request. However, this causes the technical problems mentioned in the background. Therefore, in this embodiment, the sending processor of the requesting device converts the RDMA read operation sent by the driver to the SQ into a write request with the message structure shown in Figure 4. The message structure of the custom write request in this embodiment differs from the message structure of a traditional RDMA write request based on the IB protocol. The OpCode in the Base Transport Header of the message structure in Figure 3 uses the encoding defined by the IB protocol, while the OpCode in the Base Transport Header of the message structure shown in Figure 4 of this embodiment does not use the encoding already used by the IB protocol, but uses a custom preset encoding. For example, for RC type reset, the OpCode can be 8'b000_11000, to realize the conversion of RDMA read operation into write operation proposed in this embodiment. In addition, the message payload of the message structure shown in Figure 4 is the information of the original RDMA read operation issued by the driver, including but not limited to the address of the source data, the length of the source data, and the storage address after the data is returned; the remaining fields of the message structure in Figure 4 are the same as the content of the same fields in the message structure in Figure 3, so they will not be described again.
[0049] In some embodiments, the device further includes a counter;
[0050] The counter is used to record the number of ordered response side-writing requests that are not reported to the driver when the receiving processor receives a response side-writing request sent by the response device.
[0051] The receiving processor is further configured to, when receiving a response-side Ack signal sent by the response device, if the work queue element corresponding to the response-side Ack signal is an RDMA read operation and the number of response-side write requests recorded by the counter is greater than or equal to the number of response-side write requests expected to be received by the RDMA read operation, generate a completion queue element corresponding to the RDMA read operation and send the completion queue element to the driver.
[0052] Specifically, the receiving processor only checks the number of response write requests recorded by the counter when it receives the response side Ack signal. Under normal circumstances, when an RDMA read operation is completed, the number of response write requests recorded by the counter should be equal to the number of response write requests expected to be received by the RDMA read operation. However, in special cases, the number of response write requests recorded by the counter may be greater than the number of response write requests expected to be received by the RDMA read operation. For example, in the case of packet loss and retransmission, as long as the number of response write requests recorded by the counter is greater than or equal to the number of response write requests expected to be received by the RDMA read operation when the response side Ack signal is received, it can be determined that the read data reception has been completed and the RDMA read operation has been completed. At this time, a completion queue element corresponding to the RDMA read operation can be generated and sent to the driver to notify the driver. At the same time, the number of response write requests recorded by the counter also needs to be updated.
[0053] In some embodiments, the sending processor is further configured to store the request side-writing request in a request side-sending linked list; wherein the request side-sending linked list stores each request side-writing request in the order of sending.
[0054] The receiving processor is also configured to update the requesting side sending list according to the response sidewriting request, and generate a request queue doorbell;
[0055] The sending processor is further configured to generate a request-side Ack signal based on the request queue doorbell, and send the request-side Ack signal to the response device to notify the response device that the response side write request it sent has been received by the request device.
[0056] Specifically, the receiving processor of the requesting device responds to the response side Ack signal sent by the response device, indicating that the receiving processor of the response device has successfully received the write request from the requesting side. The receiving processor of the requesting device deletes the write request corresponding to the response side Ack signal from the request side sending list (TXC). The request side sending list stores each write request in the order of sending and is used to track and manage incomplete write requests.
[0057] The sending processor of the requesting device generates a request-side Ack signal based on the request queue doorbell. This signal is used to confirm that the requesting device has received the response profiler request. The sending processor of the requesting device sends the request-side Ack signal to the response device, which can be understood as the requesting device's confirmation of receipt of the response profiler request sent by the response device.
[0058] In some embodiments, the request profiling request, the response profiling request, the request-side Ack signal, and the request-side Ack signal all carry a sequence number (PSN). The sequence number carried by the request profiling request and the sequence number carried by the response profiling request are independent of each other. The sequence number carried by the request-side Ack signal is the same as the sequence number carried by its corresponding response profiling request, and the sequence number carried by the response-side Ack signal is the same as the sequence number carried by its corresponding request profiling request.
[0059] Specifically, in traditional RDMA, because the PSNs of request messages and response messages are not contiguous, a two-dimensional data structure of (MSN, IPSN) is generally used to describe the message reception status, where IPSN is the PSN within the message, and each message starts from 0; this increases the difficulty of out-of-order message processing. Furthermore, the returned response message contains both a response with data and an Ack signal without data; the Ack signal and response can be aggregated with preceding Ack signals, but the Ack signal cannot be aggregated with preceding responses, making the order-keeping determination of the Ack signal and message more complex. Aggregation means that a message can represent itself and the aggregated messages before it. To address this, the request device and the response device in this embodiment are each configured with an independent set of PSNs, which are numbered sequentially and are not affected by each other. For example, the PSNs of the request device are PSN_a1 to PSN_a100, and the PSNs of the response device are PSN_b1 to PSN_b100. Therefore, the PSNs of multiple request profilers sent sequentially by the request device are consecutive, and the PSNs of multiple response profilers sent sequentially by the response device are also consecutive. Moreover, the sequence number carried by the request-side Ack signal is the same as the sequence number carried by its corresponding response profiler, and the sequence number carried by the response-side Ack signal is the same as the sequence number carried by its corresponding request profiler. Therefore, the request device can perform order-preserving operations based on the PSNs carried by the received response profiler and the response-side Ack signal.
[0060] In some embodiments, the sending processor is further configured to read the request profile request from the request side sending list and resend it to the response device when the receiving processor fails to successfully receive the response side Ack signal corresponding to the request profile request within a preset time.
[0061] Specifically, the sending processor of the requesting device stores the request profiler requests in a request-side sending list. This list stores each request profiler request in the order of transmission and is used to track and manage incomplete requests. When the receiving processor of the requesting device fails to receive the response-side Ack signal corresponding to the request profiler request within a preset time, it indicates that some error may have occurred, such as a network failure or data transmission failure. In this case, the sending processor of the requesting device reads the request profiler request for which the response was not received from the request-side sending list and then retransmits this request profiler request to the responding device. The purpose of this is to allow the responding device to retransmit the response profiler request corresponding to this request profiler request. Through this mechanism, even in the event of data transmission failure, RDMA communication can automatically attempt to retransmit the unsuccessful request, thereby improving the reliability of data transmission and the robustness of the system. This error handling mechanism is another important component in RDMA communication that ensures the reliability of data transmission.
[0062] Another embodiment of this application provides an RDMA-based response device, as shown in FIG2. The response device includes a receive processor (RXP), a transmit processor (TXP), and a data storage structure (SRRL).
[0063] The receiving processor is configured to receive a write request sent by the requesting device; wherein the write request is a write request converted from a read operation request.
[0064] Specifically, the receiving processor of the responding device (located in the NIC) receives the request write request (RDMA read convert write) sent by the requesting device. This request write request is obtained by the requesting device by converting the work queue element (WQE) of the original RDMA read operation issued by the driver. This conversion means that the original RDMA read operation is changed into a write operation. The request write request contains information such as the source data address, destination address, and data length. This information indicates how much data the responding device should read from the source data address and write this data to the destination address of the requesting device.
[0065] The receiving processor is further configured to parse the request profiler request to obtain the source data address, destination address and data length, write the source data address, destination address and data length into the data storage structure, and generate a Receive Queue Doorbell (RQDB).
[0066] Specifically, the receiver processor of the response device parses the request profiler request and extracts information such as the source data address, destination address, and data length. This information is crucial for performing RDMA operations, instructing the response device where to read data and where to write it. After extracting this information, the receiver processor writes it into the response device's data storage structure (SRRL), a data storage structure specifically designed for storing and tracking the RDMA operation status. Next, the receiver processor generates an RQDB to notify the transmitter processor that there is new data to process in the request queue. Through the RQDB, the transmitter processor knows which data to read from the data storage structure to generate the response profiler request.
[0067] The sending processor is configured to read the source data address, destination address, and data length from the data storage structure according to the request queue doorbell, read the target data according to the source data address and data length, generate a response profiler request according to the target data and destination address, and send the response profiler request to the requesting device, so that the requesting device writes the target data into the storage space corresponding to the destination address according to the response profiler request.
[0068] Specifically, the sending processor of the response device receives the RQDB and reads the source data address, destination address, and data length from the data storage structure. This information tells the sending processor where to read the data and where to write it. Based on this information, the sending processor reads the target data from the source data address; this target data is the data that the response device needs to send back to the requesting device. Next, the sending processor generates a read response convert write request based on the target data and destination address. This read response convert write request contains the target data and destination address information, indicating the memory address where the requesting device should write the target data. Finally, the sending processor sends this read response convert write request back to the requesting device via the network, allowing the requesting device to write the target data to the storage space corresponding to the destination address according to the read response convert write request.
[0069] In summary, the response device of this embodiment is used in conjunction with the request device of the above embodiments. The same device can simultaneously possess the functions of both the request device of the above embodiments and the response device of this embodiment. The RDMA read operation sent by the driver to the request device is converted into a request-side write request, eliminating the need to set up a data storage structure (CRRL) in the request device to store information related to receiving the read response. Since the target data to be read is directly written by the response device to the storage space corresponding to the destination address in the request device, the request device does not need to perform complex out-of-order processing and reordering logic. Moreover, after converting the read operation into a write operation, each write operation is independent, eliminating the need to track multiple incomplete read operations, thus reducing QPC maintenance. In summary, this avoids consuming large amounts of memory and cache resources.
[0070] In some embodiments, the response profiler request includes a Local Routing Header field, a Global Routing Header field, a Base Transport Header field, an Extended Transport Header field, a Message payload field, an Invariant CRC field, and a Variant CRC field; the OpCode in the Base Transport Header field of the response profiler request uses a custom preset encoding so that the requesting device can identify the operation type of the response profiler request as a response to the request profiler request based on the custom preset encoding.
[0071] Specifically, Figure 5 shows the message structure of a traditional RDMA read response based on the IB protocol, and Figure 6 shows the message structure of a custom response profiler request in this embodiment. In traditional RDMA communication, the sending processor of the responding device generates a read response for the read data to be returned. However, this causes the technical problems mentioned in the background. Therefore, in this embodiment, the sending processor of the requesting device converts the RDMA read operation sent by the driver to SQ into a request profiler request with the message structure shown in Figure 4. The responding device reads the data according to the request profiler request and generates a response profiler request with the message structure shown in Figure 6. The message structure of the custom response profiler request in this embodiment differs from the message structure of a traditional RDMA read response based on the IB protocol. The OpCode in the Base Transport Header of the message structure in Figure 5 uses the encoding defined by the IB protocol, while the Base Transport Header of the message structure shown in Figure 6 of this embodiment... The OpCode (operation code) in the header does not use the encoding already used by the IB protocol, but instead uses a custom preset encoding; for example, for RC type reset, the OpCode can be 8'b000_11001, to realize the conversion of RDMA read operation to write operation proposed in this embodiment. In addition, the ExtendTransport header of the message structure shown in Figure 6 is the same as RETH, where VA, R_KEY, and DMALen are obtained from SRRL, originating from the request write request sent by the requesting device. The remaining fields of the message structure in Figure 6 are the same as the contents of the same fields in the message structure in Figure 5, so they will not be described again.
[0072] In some embodiments, the receiving processor parses the request sidetracking request to obtain a sequence number (PSN) and writes the sequence number into the data storage structure;
[0073] The sending processor is further configured to read the sequence number (PSN) from the data storage structure according to the request queue doorbell, generate a response side Ack signal according to the sequence number, and send the response side Ack signal to the request device after sending the response side request to the request device, so as to notify the request device that the request side request has been received.
[0074] Specifically, the receiving processor of the response device parses the request profiler request and extracts the PSN (Programmatical Notification Number), which is an identifier used to identify the request profiler request. The receiving processor writes the extracted PSN into the data storage structure of the response device. The sending processor of the response device receives the RQDB (Reference Quality Database) and reads the PSN from the data storage structure. The sending processor of the response device generates a response-side Ack signal based on the PSN. The response-side Ack signal is an acknowledgment of receipt of the data sent by the requesting device, indicating that the request profiler request corresponding to the PSN has been successfully received. After sending the response profiler request to the requesting device, the sending processor of the response device sends the response-side Ack signal to the requesting device, notifying the requesting device that the response device has received the request profiler request. Based on the above method, both parties in RDMA communication can ensure reliable data transmission and sequential processing.
[0075] In some embodiments, the sending processor is further configured to store the response profiler request in a response-side sending list; wherein the response-side sending list stores each response profiler request in the sending order.
[0076] The receiving processor is further configured to respond to the request profile request retransmitted by the requesting device, and retransmit the corresponding response profile request stored in the response side sending list to the requesting device according to the retransmitted request profile request; after retransmitting the response profile request, send a response side Ack signal to the requesting device to notify the requesting device that the retransmitted request profile request has been successfully received.
[0077] Specifically, the sending processor of the response device stores response profiler requests in a response-side sending list (TXC). This list stores response profiler requests in the order they were sent, and is used to track and manage incomplete responses. When the response device responds to a request profiler request from the requesting device, after sending a response profiler request to the requesting device, it sends a response-side Ack signal to the requesting device to notify it that its request profiler request has been received. This response-side Ack signal should be received by the requesting device within a preset time. If the requesting device does not receive this response-side Ack signal within the preset time, it indicates that some error may have occurred, such as a network failure or data transmission failure, and the requesting device will retransmit the request profiler request to the response device. When the receiving processor of the response device receives a request profiler request, it generates an RQDB instructing the sending processor to retransmit the request profiler request. Based on this RQDB, the sending processor reads the corresponding response profiler request from the response-side sending list, and the sending processor of the response device retransmits this response profiler request to the requesting device. Through the above mechanism, even if data transmission fails, RDMA communication can automatically attempt to retransmit the unsuccessful response, thereby improving the reliability of data transmission and the robustness of the system. This specific error handling mechanism is another important component in RDMA communication to ensure the reliability of data transmission.
[0078] In some embodiments, the receiving processor is further configured to respond to a request-side Ack signal sent by the requesting device and update the response-side sending list according to the request-side Ack signal.
[0079] Specifically, the receiving processor of the response device responds to the request-side Ack signal sent by the requesting device, which means that the requesting device has successfully received the response profile request. Then, the receiving processor of the response device deletes the response profile request corresponding to the request-side Ack signal from the response-side sending list according to the request-side Ack signal. This response-side sending list stores each response profile request in the order of sending and is used to track and manage incomplete responses.
[0080] The sending processor is further configured to retransmit the response sidewriting request to the requesting device when the receiving processor fails to successfully receive the requesting side Ack signal within a preset time.
[0081] Specifically, when the receiving processor of the response device fails to receive the request-side Ack signal corresponding to its sent response profiling request within a preset time, it indicates that an error may have occurred, such as a network failure or data transmission failure. In this case, the receiving processor of the response device generates an RQDB instructing the sending processor to retransmit the response profiling request. Based on this RQDB, the sending processor reads the corresponding response profiling request from the response-side sending list and retransmits the response profiling request to the requesting device. After retransmitting the response profiling request, if the requesting device successfully receives the retransmitted response profiling request, the requesting device sends a request-side Ack signal to notify the requesting device that it has received the retransmitted response profiling request. Through this mechanism, even in the event of data transmission failure, RDMA communication can automatically attempt to retransmit the unsuccessful response, thereby improving the reliability of data transmission and the robustness of the system. This specific error handling mechanism is another important component in RDMA communication that ensures the reliability of data transmission.
[0082] In some embodiments, the request profiling request, the response profiling request, the request-side Ack signal, and the request-side Ack signal all carry a sequence number. The sequence number carried by the request profiling request and the sequence number carried by the response profiling request are independent of each other. The sequence number carried by the request-side Ack signal is the same as the sequence number carried by its corresponding response profiling request, and the sequence number carried by the response-side Ack signal is the same as the sequence number carried by its corresponding request profiling request.
[0083] Specifically, in traditional RDMA, because the PSNs of request messages and response messages are not contiguous, a two-dimensional data structure of (MSN, IPSN) is generally used to describe the message reception status, where IPSN is the PSN within the message, and each message starts from 0; this increases the difficulty of out-of-order message processing. Furthermore, the returned response message contains both a response with data and an Ack signal without data; the Ack signal and response can be aggregated with preceding Ack signals, but the Ack signal cannot be aggregated with preceding responses, making the order-keeping determination of the Ack signal and message more complex. Aggregation means that a message can represent itself and the aggregated messages before it. To address this, the request device and the response device in this embodiment are each configured with an independent set of PSNs, which are numbered sequentially and are not affected by each other. For example, the PSNs of the request device are PSN_a1 to PSN_a100, and the PSNs of the response device are PSN_b1 to PSN_b100. Therefore, the PSNs of multiple request profilers sent sequentially by the request device are consecutive, and the PSNs of multiple response profilers sent sequentially by the response device are also consecutive. Moreover, the sequence number carried by the request-side Ack signal is the same as the sequence number carried by its corresponding response profiler, and the sequence number carried by the response-side Ack signal is the same as the sequence number carried by its corresponding request profiler. Therefore, the request device can perform order-preserving operations based on the PSNs carried by the received response profiler and the response-side Ack signal.
[0084] Another embodiment of this application provides an RDMA-based system, as shown in FIG2. The system includes a request device as described in the above embodiments and a response device as described in the above embodiments. The working principle of the system in this embodiment can be obtained by referring to the contents of the request device and the response device described in the above embodiments, so it will not be described in detail here.
[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0086] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A request device based on RDMA, wherein, The requesting device includes a sending processor and a receiving processor; The sending processor is used to receive work queue elements issued by the driver. If the work queue element is an RDMA read operation, it converts the RDMA read operation to obtain a request for side-writing. The sending processor is further configured to send the request profiler request to the response device; The receiving processor is configured to receive and parse the response write request sent by the response device to obtain the target data and destination address corresponding to the RDMA read operation, and write the target data into the storage space corresponding to the destination address.
2. The RDMA-based request device according to claim 1, wherein, The request profiler includes a Local Routing Header field, a Global Routing Header field, a Base Transport Header field, a Message payload field, an Invariant CRC field, and a Variant CRC field. The OpCode in the Base Transport Header field of the request profiler uses a custom preset encoding so that the response device can identify the operation type of the request profiler as a write request converted from a read request based on the custom preset encoding.
3. The requesting device according to claim 1, wherein, The device also includes a counter. The counter is used to record the number of ordered response side-writing requests that are not reported to the driver when the receiving processor receives a response side-writing request sent by the response device. The receiving processor is further configured to, when receiving a response-side Ack signal sent by the response device, if the work queue element corresponding to the response-side Ack signal is an RDMA read operation and the number of response-side write requests recorded by the counter is greater than or equal to the number of response-side write requests expected to be received by the RDMA read operation, generate a completion queue element corresponding to the RDMA read operation and send the completion queue element to the driver.
4. The requesting device according to claim 3, wherein, The sending processor is further configured to store the request side-writing request in a request side sending list; wherein the request side sending list stores each request side-writing request in the sending order. The receiving processor is also configured to update the requesting side sending list according to the response sidewriting request, and generate a request queue doorbell; The sending processor is further configured to generate a request-side Ack signal based on the request queue doorbell, and send the request-side Ack signal to the response device to notify the response device that the response side write request it sent has been received by the request device.
5. The requesting device according to claim 4, wherein, The request profiler request, the response profiler request, the request-side Ack signal, and the request-side Ack signal all carry a sequence number. The sequence number carried by the request profiler request and the sequence number carried by the response profiler request are independent of each other. The sequence number carried by the request-side Ack signal is the same as the sequence number carried by its corresponding response profiler request, and the sequence number carried by the response-side Ack signal is the same as the sequence number carried by its corresponding request profiler request.
6. The requesting device according to claim 1, wherein, The sending processor is further configured to, when the receiving processor fails to successfully receive the response-side Ack signal corresponding to the request profile request within a preset time, read the request profile request from the request-side sending list and resend it to the response device.
7. A response device based on RDMA, wherein, The response device includes a receiving processor, a transmitting processor, and a data storage structure; The receiving processor is configured to receive a write request sent by the requesting device; wherein the write request is a write request converted from a read operation request. The receiving processor is further configured to parse the request sidetracking request to obtain the source data address, destination address and data length, write the source data address, destination address and data length into the data storage structure, and generate a request queue doorbell; The sending processor is configured to read the source data address, destination address, and data length from the data storage structure according to the request queue doorbell, read the target data according to the source data address and data length, generate a response profiler request according to the target data and destination address, and send the response profiler request to the requesting device, so that the requesting device writes the target data into the storage space corresponding to the destination address according to the response profiler request.
8. The response device according to claim 7, wherein, The response profiler request includes a Local Routing Header field, a Global Routing Header field, a Base Transport Header field, an Extended Transport Header field, a Message payload field, an Invariant CRC field, and a Variant CRC field. The OpCode in the Base Transport Header field of the response profiler request uses a custom preset encoding so that the requesting device can identify the operation type of the response profiler request based on the custom preset encoding.
9. The response device according to claim 7, wherein, The receiving processor parses the request side-writing request and obtains a sequence number, and writes the sequence number into the data storage structure; The sending processor is further configured to read the sequence number from the data storage structure according to the request queue doorbell, generate a response-side Ack signal according to the sequence number, and send the response-side Ack signal to the requesting device to notify the requesting device that the request side write request has been received.
10. The response device according to claim 9, wherein, The sending processor is further configured to store the response sidewriting request to a response side sending list; wherein the response side sending list stores each response sidewriting request in the sending order; The receiving processor is further configured to respond to the request profiler request retransmitted by the requesting device, retransmit the corresponding response profiler request stored in the response side sending list to the requesting device according to the retransmitted request profiler request, and send a response side Ack signal to the requesting device to notify the requesting device that the retransmitted request profiler request has been successfully received.
11. The response device according to claim 9, wherein, The receiving processor is further configured to respond to the request-side Ack signal sent by the requesting device and update the response-side sending list according to the request-side Ack signal; The sending processor is further configured to retransmit the response sidewriting request to the requesting device when the receiving processor fails to successfully receive the requesting side Ack signal within a preset time.
12. The response device according to claim 7, wherein, The request profiler request, the response profiler request, the request-side Ack signal, and the request-side Ack signal all carry a sequence number. The sequence number carried by the request profiler request and the sequence number carried by the response profiler request are independent of each other. The sequence number carried by the request-side Ack signal is the same as the sequence number carried by its corresponding response profiler request, and the sequence number carried by the response-side Ack signal is the same as the sequence number carried by its corresponding request profiler request.
13. A system based on RDMA, wherein, include: The request device as described in any one of claims 1-6 and the response device as described in any one of claims 7-12.