Implementation method for optimizing RDMA congestion control

Abstract

The invention relates to the technical field of communication technology, in particular to an implementation method for optimizing RDMA (Remote Direct Memory Access) congestion control, which comprises the following steps: a request end network card sends a SEND / WRITE request message, and a response end network card replies a high-priority OAMACK response message when receiving a plurality of loaded SEND / WRITE requests; a request end network card sends a READ request message, a response end network card replies a READ Resp response message, the method uses two control windows to control the sent request, the control of the SEND / WRITE request adopts a mode of adding a high-priority OAMACK message to replace a service ACK message, the opposite end network card replies the high-priority OAMACK once receiving a plurality of loaded SEND / WRITE request messages, and the service ACK message is sent to the opposite end network card after receiving a plurality of loaded SEND / WRITE request messages. The high-priority OAMACK can reply immediately without being limited by a protocol, and the scheme adopts the high-priority OAMACK message to control the SEND / WRITE request message flow, so that the problem of algorithm misrecognition is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of communication technologies, and in particular to a method for optimizing RDMA congestion control. Background Art

[0002] Currently, computers in data center networks can exchange data using Remote Direct Memory Access (RDMA). This allows a computer's network interface card to read data from or write data to another computer's memory over the network without requiring the computer's operating system to intervene. RDMA over Ethernet is called RDMA over Converged Ethernet (RoCE).

[0003] To avoid performance losses in RoCE due to network congestion and packet loss, advanced network congestion control methods are required. RDMA operations primarily include SEND, WRITE, and READ. For SEND and WRITE operations, the requesting NIC sends a request, and the responding NIC replies with an ACK. The RDMA protocol engine uses the received ACK response message to complete the SEND and WRITE operations. At this point, the flow control algorithm can naturally control the rate of SEND and WRITE requests sent by the requesting NIC by utilizing the time and frequency of the ACK response. For RDMA READ operations, the requesting NIC sends a request, and the responding NIC replies with a READ Resp response. The RDMA protocol engine uses the received READ Resp response message to complete the READ operation. The READ operation traffic is primarily reflected in this response message, as there is no existing ACK response for the READ Resp response message.

[0004] An existing flow control algorithm uses the time and frequency of receiving service ACK responses to control the rate of SEND and WRITE requests sent in the sending direction of the network card when performing rate control for SEND and WRITE operations. When performing rate control for READ operations, the frequency of sending READ REQs is adjusted by the rate of receiving READ Resp messages to control the rate at which the responding network card replies to READ Resp.

[0005] The existing flow control scheme uses the time and frequency of receiving service ACK to control the rate of SEND and WRITE requests in the sending direction of the network card, which has the problem of algorithm misidentification. If a READ request that needs to read a large amount of data is sent before the requesting-side network card sends the SEND and WRITE requests, the responding-side network card will continue to reply with READ Resp response messages until the READ Resp response message is sent. It will then send the confirmation ACK message for the received SEND and WRITE request messages. At this time, the requesting-side network card seems to have not received the service ACK message for a long time, causing the algorithm to limit the rate of sending SEND and WRITE requests when the network path on its sending side is not congested, resulting in a significant loss of bandwidth utilization. At the same time, when controlling the RDMA READ operation, the frequency of sending READ REQs is adjusted by the rate at which the READ Resp message is received, thereby controlling the rate at which the responding NIC replies to the READ Resp. However, this solution only adjusts the frequency of READ REQs. When the READ WQE is slightly larger, for example, 1MB, each READ REQ request issued by the requesting NIC means that the responding NIC can continuously reply with 1MB of READ Resp responses in a short period of time. This solution cannot achieve fine-grained control of RDMA READ operations. Summary of the Invention

[0006] The object of the present invention is to provide an implementation method for optimizing RDMA congestion control, aiming to control the flow of SEND / WRITE request messages by using high-priority OAMACK messages without algorithm misidentification problems.

[0007] To achieve the above object, the present invention provides a method for optimizing RDMA congestion control, comprising the following steps:

[0008] The requesting end network card sends a SEND / WRITE request message, and the responding end network card replies with a high-priority OAMACK response message every time it receives several SEND / WRITE requests with load;

[0009] The requesting network card sends a READ request message, and the responding network card replies with a READ Resp response message.

[0010] The requesting end network card sends a SEND / WRITE request message, and the responding end network card receives several SEND / WRITE requests with load and replies with a high-priority OAMACK response message, which also includes:

[0011] The requesting end network card adjusts the SEND / WRITE request message window value that the requesting end can send according to the OAMACK response message time or rate.

[0012] The requesting end network card adjusting the SEND / WRITE request message window value that can be sent by the requesting end according to the OAMACK response message time or rate further includes:

[0013] The requesting end network card sends a SEND / WRITE request message according to the adjusted window value.

[0014] The requesting end network card sending a SEND / WRITE request message according to the adjusted window value further includes:

[0015] The requesting-side NIC reports a CQE based on the received service OAMACK response message, informing the host side that the SEND / WRITE operation is completed.

[0016] The requesting end network card sends a READ request message, and the responding end network card replies with a READ Resp response message, which further includes:

[0017] The requesting end network card determines the receivable READ Resp response packet message window value according to the time or rate of receiving the READ Resp response packet.

[0018] The present invention provides an implementation method for optimizing RDMA congestion control. The requesting end network card sends a SEND / WRITE request message, and the responding end network card replies with a high-priority OAMACK response message every time it receives several SEND / WRITE requests with load. The requesting end network card sends a READ request message, and the responding end network card replies with a READ Resp response message. The method uses two control windows to control the sent requests. The control of the SEND / WRITE request adopts a method of adding a high-priority OAMACK message to replace the service's own ACK message. The opposite end network card replies with a high-priority OAMACK every time it receives several SEND / WRITE request messages with load. The high-priority OAMACK is not restricted by the protocol and can be replied immediately. The local end network card adjusts the sending window of the SEND / WRITE request using the time and frequency of the received OAMACK message. The control of the READ request adopts a method of splitting the RDMA READ WQE to control the size of the READ request sent by the requesting end (i.e., the length and number of the READ Resp messages expected to be replied by the responding end) and the READ request frequency, so as to fine-grainedly adjust the response time of the responding end network card. Resp message rate. This solution uses high-priority OAMACK messages to control the SEND / WRITE request message traffic, eliminating algorithm misidentification issues and decoupling from the service itself, simplifying the congestion control solution and making it more scalable. It also uses READ REQ unpacking to achieve fine-grained congestion control of RDMA READ operations, controlling the READ Resp reception frequency on the requesting side and completely avoiding the READ Resp incast phenomenon on the network card. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is a diagram of the large traffic and overall congestion control ideas caused by the requesting end network card in the network.

[0021] Figure 2 This is a flow chart of an implementation method for optimizing RDMA congestion control provided by the present invention.

[0022] Figure 3 This is a flow chart of a method for a requesting-side network card to control the flow of SEND / WRITE request messages.

[0023] Figure 4This is a flow chart of a method for a requesting-side network card to control the READ Resp message flow caused by a READ request message.

[0024] Figure 5 This is the overall block diagram of the hardware implementation of the RDMA congestion control method. DETAILED DESCRIPTION

[0025] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0026] See also Figures 1 to 5 The present invention provides a method for optimizing RDMA congestion control, comprising the following steps:

[0027] S1: The requesting end network card sends a SEND / WRITE request message. The responding end network card replies with a high-priority OAMACK response message every time it receives several SEND / WRITE requests with load. The requesting end network card adjusts the SEND / WRITE request message window value that the requesting end can send based on the time or rate of the OAMACK response message. The requesting end network card sends the SEND / WRITE request message according to the adjusted window value. The requesting end network card reports the CQE based on the received service OAMACK response message to inform the host side that the SEND / WRITE operation is completed.

[0028] In this embodiment of the present invention, the requesting-side network card A (host A) sends SEND / WRITE request messages according to the initial window. The responding-side network card B (host B) replies with a high-priority OAMACK response message every time it receives several loaded SEND / WRITE requests. The requesting-side network card A adjusts the window value of the SEND / WRITE request messages that the requesting-side can send based on the time or rate of receiving the OAMACK response messages. The requesting-side network card A sends SEND / WRITE request messages according to the set window value. The requesting-side network card A reports the CQE based on the received service ACK response message to inform the host side that the SEND / WRITE operation is complete.

[0029] S2 The requesting end network card sends a READ request message, and the responding end network card replies with a READ Resp response message. The requesting end network card determines the receivable READ Resp response packet message window value based on the time or rate of receiving the READ Resp response packet.

[0030] In an embodiment of the present invention, the requesting end network card A sends a READ request message according to the initial window; the responding end network card B replies with a READ Resp response message as required; the requesting end network card A determines a receivable READ Resp response packet message window value based on the time or rate of receiving the READ Resp response packet; the requesting end network card A converts the receivable READ Resp response packet message window value into a READ Resp response packet message window value that the requesting end network card can send and that requires a reply; the requesting end network card A splits and sends the READ request WQE issued by the software according to the READ Resp response packet message window value that requires a reply that the requesting end network card can send and that requires a reply; the requesting end network card A receives and aggregates the READ Resp response message; when the requesting end network card A receives a complete READ request WQE, it reports a CQE to inform the host side that a READ operation has been completed.

[0031] In this embodiment of the present invention, the software sends a WQE for a READ request. The Dma_length carried in the WQE is set by the user according to usage requirements. The congestion control algorithm module determines whether the receiving path is congested based on the rate and frequency of receiving RDMA RESP response packets, and provides the READ Resp message length expected to be replied by the peer end within a unit time (i.e., the DMALength carried in the RETH). If the Dma_length carried in the WQE of the received READ request is greater than the expected value, the READ request sent according to the expected Dma_length cannot completely send the READ request WQE issued by the software. It is necessary to send the WQE in multiple READ request packets.

[0032] During implementation, when unpacking and sending the WQE of the READ request, the requesting end records the request packet information every time it sends a READ request packet;

[0033] When sending the last READ request packet of the WQE, the last READ request packet flag of the WQE is set to 1 when recording each READ request packet information (to facilitate determining whether the WQE of the READ request is completed when receiving a READ Resp, and reporting the CQE completed by the WQE of the READ request).

[0034] Figure 5 This is the overall block diagram of the hardware implementation of the RDMA congestion control method. The splitting and aggregation of READ request WQEs are completed in the hardware. Note that when sending retransmissions, the hardware no longer splits the READ request.

[0035] Among them: the sending scheduling unit is used to schedule the sending opportunities of different QPs. For the DB of the OAMACK response message, the sending scheduling unit should take the highest priority and schedule it to the TX sending unit so that the OAMACK response message can be sent quickly; the READ request splitting unit splits the READ request WQE according to the READ Resp message length currently expected to be replied by the peer calculated by the congestion control algorithm unit; the TX sending unit sends the split READ request packet and sends the SEND / WRITE request packet according to the SEND / WRITE sending window value calculated by the congestion control algorithm unit, and sends a high-priority OAMACK message at the same time; the RX receiving unit receives the service ACK / READ Resp response and OAMACK message; the congestion control algorithm unit determines the receivable READ Resp response packet message window value based on the READ Resp response packet time or rate, and converts it into the READ Resp message length window value currently expected to be replied by the peer, and adjusts the sendable READ Resp message length window value; determine the sendable SEND / WRITE request packet window value based on the time or rate of receiving OAMACK response packets; at the same time, generate an OAMACK DB and send it to the sending scheduling unit every time several SEND / WRITE request messages with load are received; READ Resp aggregation unit aggregates multiple READ Resp messages and determines whether the WQE of a READ request is completed based on the recorded request packet information; CQE reporting unit reports the CQE of the SEND / WRITE / READ request WQE completed.

[0036] The above disclosure is merely a preferred embodiment of a method for optimizing RDMA congestion control according to the present invention and is not intended to limit the scope of the present invention. Persons skilled in the art will appreciate that any equivalent modifications made by implementing all or part of the above embodiment in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for optimizing RDMA congestion control, characterized in that: The following steps are involved: The requesting end network card sends a SEND / WRITE request message, and the responding end network card replies with a high priority OAMACK response message each time it receives several SEND / WRITE requests with load; The requesting network card sends a READ request message, and the responding network card replies with a READ Resp response message.

2. The method for optimizing RDMA congestion control as claimed in claim 1, characterized in that ； The requesting end network card sends a SEND / WRITE request message, and the responding end network card receives several SEND / WRITE requests with load and replies with a high priority OAMACK response message, which also includes: The requesting end network card adjusts the SEND / WRITE request message window value that the requesting end can send according to the OAMACK response message time or rate.

3. The method for optimizing RDMA congestion control as described in claim 2, characterized in that ； The requesting end network card adjusts the SEND / WRITE request message window value that can be sent by the requesting end according to the OAMACK response message time or rate, and further includes: The requesting end network card sends a SEND / WRITE request message according to the adjusted window value.

4. The method for optimizing RDMA congestion control as described in claim 3 is characterized in that ； The requesting end network card sends a SEND / WRITE request message according to the adjusted window value, and further includes: The requesting end network card reports CQE according to the received service OAMACK response message, informing the host side to complete the SEND / WRITE operation.

5. The method for optimizing RDMA congestion control as claimed in claim 1, characterized in that ； The requesting end network card sends a READ request message, and the responding end network card replies with a READ Resp response message, which also includes: The requesting end network card determines the receivable READ Resp response packet message window value according to the time or rate of receiving the READ Resp response packet.

Images