A system for transparent adaptation of RDMA protocol in high RTT environment

Abstract

The application discloses a kind of RDMA protocol transparent adaptation systems supporting high RTT environment, by introducing logical transport node, and the mode of inserting two logical transport nodes between two physical nodes, constitute a segmented logical transport channel, form a multistage pipeline transmission channel by near-end RDMA transmission channel and far-end RDMA transmission channel, realize the logical isolation of the transmission state of near-end transmission channel and the transmission state of far-end transmission channel.Simultaneously, by realizing link measurement, RTT-aware retransmission, RTT-aware sequencing, near-end answering mechanism for near-end physical node and fast congestion feedback mechanism for far-end logical transport node in logical transport node, the problems of unstable service traffic caused by flow control delay and low link utilization caused by data retransmission faced by traditional RDMA transmission under high RTT environment are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data transmission, and in particular to a RDMA protocol transparent adaptation system supporting high RTT environment. BACKGROUND

[0002] With the rapid development of cloud computing, supercomputing and intelligent computing, cross-data center data migration, especially the remote real-time migration of super-large-scale data, has become a basic requirement for key businesses such as financial transactions, scientific computing, AI training, disaster recovery synchronization, and has become a key bottleneck restricting system performance.

[0003] Data long-distance transmission not only makes the packet loss problem more obvious, but also makes the low utilization of transmission link caused by data retransmission more prominent. Moreover, it is accompanied by long Round-Trip Time (RTT) phenomenon, and the stability problem of business traffic caused by the hysteresis effect of flow control mechanism will also face great challenges.

[0004] Current cross-data center interconnection generally adopts TCP / IP mechanism to realize by renting operator link, and data transmission often needs to pass through the processing of operating system and multi-layer software protocol stack, which not only leads to low data transmission efficiency, but also causes large consumption of CPU resources. In order to solve these problems, the industry generally uses Remote Direct Memory Access (RDMA) mechanism to realize data migration. However, the existing RDMA technical solution still faces many technical bottlenecks and challenges when facing the demand of long-distance and large-scale real-time data transmission in wide area network environment:

[0005] 1) The increase of transmission link inherent delay and the long feedback period of congestion control algorithm such as Data Center Quantized Congestion Notification (DCQCN) and TIMELY (RTT-based data center congestion control algorithm) cause the dramatic increase of RTT round-trip delay, which further causes the dramatic increase of data buffer pressure at the sending end.

[0006] 2) The untimely transmission of congestion control information will cause the sending end to be unable to quickly perceive the network state change, resulting in congestion response hysteresis. The hysteresis congestion control will cause unstable flow adjustment, causing business traffic to fluctuate dramatically, and even possibly packet loss.

[0007] 3) The single data retransmission mechanism facing the changing RTT link will further reduce the link utilization, causing waste of transmission bandwidth resources.

[0008] Briefly, the existing RDMA technology architecture faces the challenge of sharp decline in transmission performance when facing long-distance, large-scale, real-time data movement requirements. To solve this problem, the present application proposes a transparent adaptation system of RDMA protocol supporting high RTT environment, aiming to improve the efficiency and stability of large bandwidth, long distance data movement, and provide a more efficient and reliable solution for long distance data movement application. SUMMARY

[0009] In view of the poor stability and low utilization rate of communication bandwidth resources of the existing RDMA technology in long-distance data transmission, the technical solution introduces a logical transmission node, divides the existing physical link of data transmission based on the logical transmission node, adopts a multi-stage pipeline segmented communication idea, and proposes a transparent adaptation system of RDMA protocol supporting high RTT environment.

[0010] To achieve the above purpose, the present application is realized by the following technical scheme:

[0011] The present application provides a transparent adaptation system of RDMA protocol supporting high RTT environment, comprising two end-side physical nodes and network-side physical links, by inserting a logical transmission node between each end-side physical node and network-side physical link, a multi-stage pipeline transmission channel comprising a near-end RDMA transmission channel and a far-end RDMA transmission channel is formed.

[0012] The logical transmission node comprises an ACK near-end answering function unit for realizing fast data reception confirmation for the near-end physical node, a transmission link adaptation function unit for realizing RDMA transmission adaptation for the near-end physical node, a link measurement function unit for measuring the delay of the network-side physical link between the two logical transmission nodes, and a congestion control function unit for realizing network-side physical link congestion early warning for the far-end logical transmission node.

[0013] The system realizes the decoupling of the standard RDMA transmission mechanism of the end-side physical node and the state of the network-side physical link through the logical transmission node, so that the end-side physical node is not aware of the delay change and link type of the network-side physical link.

[0014] Further, the multi-stage pipeline transmission channel comprises a three-stage transmission pipeline.

[0015] The first-stage transmission pipeline is composed of a source-side physical node and a first logical transmission node, and is a short-distance transmission channel, i.e., a near-end RDMA transmission channel; the second-stage transmission pipeline is composed of the first logical transmission node, a network-side physical link and a second logical transmission node, and is a long-distance transmission channel, i.e., a far-end RDMA transmission channel; and the third-stage transmission pipeline is composed of the second logical transmission node and a destination-side physical node, and is the same as the first-stage transmission pipeline mechanism, and belongs to the near-end RDMA transmission channel.

[0016] Further, the ACK near-end response function unit performs an operation of: implementing a flow control mechanism for a near-end physical node to achieve a fast response, completing a response processing mechanism for a protocol A of an end-side physical node, and generating a response confirmation message of data reception completion for the end-side physical node.

[0017] Further, the transmission link adaptation function unit performs an operation of: carrying received near-end physical node communication data to a protocol B of a network-side physical link facing a far-end logical transmission node, implementing an RDMA transmission adaptation for the near-end physical node; and the adaptation logic includes extracting DMA communication data of the near-end physical node, carrying communication data based on the protocol A to the protocol B through a queue sharing mechanism for transmission, and vice versa, i.e., adapting communication data based on the protocol B to the protocol A for transmission to the end-side physical node.

[0018] Further, the protocol A and the protocol B are the same or different DMA protocols, depending on the type of the network-side physical link.

[0019] Further, the link measurement function unit performs an operation of: periodically or on-demand measuring a delay for a network-side physical link between two logical transmission nodes; the two logical transmission nodes each independently initiate a measurement request and also each independently respond to a measurement request of a peer; a time stamp information is carried in the measurement request message, a measurement response end copies the time stamp information in the measurement request message to fill into a measurement response message and feeds back to the measurement request end, and the measurement request end calculates an RTT of the network-side physical link transmission according to the time stamp information in the response message.

[0020] Further, the congestion control function unit performs an operation of: performing flow control based on information of link measurement and buffer state; predicting congestion based on a periodic link measurement result and a change trend of a shared buffer queue state, and performing flow control before the congestion occurs to avoid the congestion.

[0021] Further, the logical transmission node further includes an RTT-aware retransmission function unit, an RTT-aware ordering function unit and a retransmission-aware adaptive function unit; wherein,

[0022] The RTT-aware retransmission function unit is configured to obtain the average RTT level of the network-side physical link based on link measurement, select a retransmission mechanism corresponding to different RTT states, and initialize the retransmission control logic.

[0023] The RTT-aware sequencing function unit is configured to set a sequencing buffer space based on the maximum RTT of link transmission, thereby optimizing the transmission delay.

[0024] The retransmission-aware adaptive function unit is configured to obtain the state of the transmission link in real time through traffic monitoring, judge whether the link quality needs a retransmission mechanism based on the retransmission ratio of the transmission link, and activate the retransmission mechanism if yes.

[0025] Further, each logical transmission node comprises two transmission interfaces: a front interface facing the network-side physical link and a back interface facing the end-side physical node.

[0026] Further, the network-side physical link is selected from an optical transport network private line, a metropolitan Ethernet private line, or a multiprotocol label switching virtual private network private line according to application requirements, and the selected private line network is equivalent to a section of physical link with a long RTT.

[0027] Compared with the prior art, the beneficial effects of the present application are as follows: the RDMA protocol transparent adaptation system supporting a high-RTT environment proposed by the present application aims to improve the efficiency and stability of data movement facing a large bandwidth and a long distance, and provides a more efficient and reliable solution for long-distance data movement applications. The present application can be used to modify and adapt the existing short-distance RDMA communication network or build a new long-distance efficient RDMA communication transmission system, and solve the problems of poor stability and low data transmission efficiency of the existing RDMA communication mechanism in long-distance data transmission. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor:

[0029] Figure 1 The figure is a three-stage pipeline adaptation logic partitioning diagram in the present application;

[0030] Figure 2 The figure is a physical node structure diagram in the present application;

[0031] Figure 3 The figure is a logical transmission node function structure diagram in the present application;

[0032] Figure 4 This is a schematic diagram of the physical link in this invention;

[0033] Figure 5 This is a schematic diagram of proximal substitution in the present invention;

[0034] Figure 6 This is a schematic diagram of link measurement in this invention;

[0035] Figure 7 This is a schematic diagram of the congestion control process in this invention;

[0036] Figure 8 This is a schematic diagram illustrating the selection of the retransmission mechanism in this invention;

[0037] Figure 9 This is a schematic diagram of the retransmission sensing adaptive mechanism in this invention. Detailed Implementation

[0038] This invention proposes a transparent adaptation system for RDMA protocols that supports high RTT environments. It can be used to modify and adapt existing short-distance RDMA communication networks or to build new long-distance, high-efficiency RDMA communication transmission systems, solving the problems of poor stability and low data transmission efficiency faced by existing RDMA communication mechanisms in long-distance data transmission.

[0039] This invention provides a transparent adaptation system for RDMA protocol that supports high RTT environments. The system includes two end-side physical nodes and a network-side physical link. By inserting logical transmission nodes between each end-side physical node and the network-side physical link, a multi-stage pipelined transmission channel containing a near-end RDMA transmission channel and a far-end RDMA transmission channel is formed.

[0040] The logical transmission node includes an ACK near-end acknowledgment function unit for fast data reception confirmation for near-end physical nodes, a transmission link adaptation function unit for RDMA transmission adaptation for near-end physical nodes, a link measurement function unit for delay measurement of the network-side physical link between two logical transmission nodes, and a congestion control function unit for network-side physical link congestion warning for far-end logical transmission nodes.

[0041] The system decouples the standard RDMA transmission mechanism of the end-side physical node from the physical link state of the network side through logical transmission nodes, so that the end-side physical node is unaware of the latency changes and link type of the physical link on the network side.

[0042] The entire end-to-end remote data transmission system consists of a three-stage transmission pipeline, such as... Figure 1As shown, the end-side physical node A1 and logical transmission node B1 constitute the first-level transmission pipeline. This first-level pipeline is a short-distance transmission channel, referred to here as the near-end RDMA transmission channel. Its data transmission mechanism is the same as the existing standard RDMA mechanism and will not be elaborated further. The second-level transmission pipeline consists of logical transmission node B1, logical transmission node B2, and a network-side physical link C. This is a long-distance transmission channel and the core of the entire adaptation system. The third-level transmission pipeline consists of logical transmission node B2 and end-side physical node A2. Its mechanism is the same as the first-level pipeline and it also belongs to the near-end RDMA transmission channel. Here, the communication model of the end-side physical nodes is abstracted into a simple structure composed of communication nodes and a Direct Memory Access (DMA) engine, as follows: Figure 2 As shown, the communication node is responsible for the transmission control of service data, including actively initiating and passively responding to data DMA transmission operations. The DMA engine built into the RNIC (Remote Network Interface Card) is responsible for the first-level transmission of service data, storing the service traffic into the buffer of the logical transmission node without blocking. The logical transmission node is then responsible for lossless transmission over the long physical network link. The functional structure diagram of the logical transmission node is as follows. Figure 3 As shown, it includes logical functional units such as ACK near-end proxy, transmission link adaptation, congestion control, link measurement, packet loss retransmission (RTT-aware retransmission), and out-of-order reordering (RTT-aware reordering), as well as a buffer queue management mechanism. The network-side physical link generally refers to the actual usable physical network. Depending on application requirements, it can be selected from Optical Transport Network (OTN) leased lines, metropolitan area Ethernet leased lines, or Multiprotocol Label Switching Virtual Private Network (MPLS VPN) leased lines, etc. Here, the leased network is equivalent to a physical link with a long RTT.

[0043] like Figure 5 As shown, the ACK near-end proxy function unit mainly aims to achieve rapid response through the flow control mechanism of the near-end RDMA physical node, complete the response processing mechanism of Protocol A for the end-side physical node, and generate an acknowledgment message for data reception completion for the end-side physical node A1. Since the ACK near-end proxy function unit and the near-end physical node are short-distance point-to-point communication, the network-side physical link error rate and packet loss rate are extremely low, and the impact of packet loss is no longer considered; the ACK near-end proxy function unit mainly aims to achieve rapid acknowledgment of data reception for the near-end RDMA transmission node.

[0044] like Figure 4As shown, the transmission link adaptation function unit receives communication data from the near-end RDMA physical node and carries the data in the communication protocol B of the network-side physical link facing the far-end logical transmission node, realizing RDMA transmission adaptation for the near-end physical node. This mainly involves adapting the DMA protocol for the near-end physical node based on the link type and data transmission mechanism of the leased long-distance physical network. The adaptation logic includes extracting the DMA communication data from the near-end physical node, carrying the communication data based on protocol A into protocol B for transmission through a queue sharing mechanism, and vice versa. Communication data based on protocol B is also adapted to protocol A and then transmitted to the end-side physical node. Protocol A and protocol B can be the same DMA protocol or different protocols, depending on the link type of the long-distance physical network.

[0045] like Figure 6 As shown, the link measurement function unit mainly completes the RTT delay measurement and delay monitoring of communication between two logical transmission nodes based on the network-side physical link C; logical transmission node B1 and logical transmission node B2 independently initiate measurement requests and independently respond to the measurement requests of the other end; the measurement request message carries timestamp information, and the measurement response end will copy the timestamp in the measurement request message and fill it into the measurement response message and feed it back to the measurement request end. The measurement request end can calculate the RTT of the network-side physical link transmission based on the timestamp information in the response message.

[0046] like Figure 7 As shown, the congestion control function unit mainly implements flow control based on information from two aspects: link measurement and buffer status. Congestion prediction is performed based on the changing trends of periodic link measurement results and shared buffer queue status. Flow control is then implemented before congestion actually occurs to avoid it. Taking congestion control from logical transmission node B2 to logical transmission node B1 as an example, node B1 receives communication data from node B2. Through the queue sharing mechanism and protocol conversion logic, the communication data is sent to node A1. When node A1 cannot receive all the data in time, the data accumulates in B1's shared buffer, leading to a gradual increase in buffer space occupancy. The congestion prediction mechanism generates a congestion warning message and a flow control message in a timely manner based on the changing trend of buffer space occupancy, notifying the data sender before congestion occurs. After receiving the flow control message, node B2 promptly adjusts the transmission flow to avoid actual congestion. Meanwhile, based on the periodic measurement mechanism, node B1 can also sense the changes in the network RTT status of the physical link between it and node B2. The congestion prediction logic of node B1 will detect the RTT measurement value in real time. When the RTT suddenly increases steadily, a congestion warning will be generated, and the flow control mechanism of node B2 will also be triggered.

[0047] To adapt to physical links with different RTT levels, an RTT-aware retransmission mechanism, an RTT-aware sorting mechanism, and a retransmission-aware adaptive mechanism are introduced into the logical transmission nodes.

[0048] like Figure 8 As shown, the RTT-aware retransmission functional unit's main idea is to select different retransmission mechanisms for physical links with different RTT levels. The primary goal is to achieve better link utilization while ensuring lossless transmission. Since different retransmission mechanisms have varying retransmission overhead, affecting link utilization, matching different retransmission mechanisms to different RTT physical networks can achieve the lowest link overhead and optimal performance. The RTT-aware retransmission functional unit obtains the average RTT level of the physical network based on link measurements, selects the optimal retransmission mechanism for different RTT states, and initializes the retransmission control logic.

[0049] The RTT-aware sorting unit primarily aims to optimize transmission latency. In high RTT transmission scenarios (RTT greater than 50 microseconds), the Go-Back-N (Go-Back-N) transmission mechanism transmits excessive amounts of useless data during retransmissions, severely impacting link utilization. To mitigate this impact, selective retransmission is employed in high RTT scenarios. However, this mechanism can lead to out-of-order data transmission. In applications requiring order preservation, a sorting mechanism is necessary to maintain the data flow in order. However, sorting operations are heavily influenced by RTT; the sorting buffer must be greater than or equal to the Bandwidth-DelayProduct (BDP) corresponding to RTT / 2 of the latency. An excessively large buffer increases transmission latency, while an excessively small buffer leads to sorting failure. The RTT-aware sorting mechanism precisely sets the sorting buffer space based on the maximum RTT of the link transmission, thereby achieving optimal transmission latency.

[0050] The retransmission-aware adaptive functional unit aims to optimize buffer usage and link transmission latency at the data sending end of logical transmission nodes. This mechanism automatically monitors the traffic status of the transmission link within the logical transmission node, primarily by calculating the cumulative effective data transmission volume sent from the local end. It also monitors the retransmission request status from the remote logical transmission node, calculating the amount of data to be retransmitted based on these requests. The ratio of these two values ​​measures the link quality from the near-end logical transmission node to the remote transmission node. When the link quality reaches a certain level (considered very good), the retransmission mechanism can be disabled, further reducing transmission latency. Simultaneously, the buffer space reserved for the retransmission mechanism is released and used for reverse data transmission, increasing the available buffer space for reverse DMA transmission. The working logic flowchart of this mechanism is shown below. Figure 9 As shown, the status of the transmission link is obtained in real time through traffic monitoring. Based on the retransmission ratio P of the transmission link, it is determined whether the link quality requires a retransmission mechanism. Through real-time monitoring of the link quality, the retransmission mechanism is adaptively activated to optimize buffering and output latency.

[0051] It should be noted that during RDMA transmission, packet loss due to physical layer link errors necessitates a data retransmission function to ensure data transmission reliability. However, data retransmission reduces link bandwidth utilization, thus degrading RDMA performance. Generally, the link error rate is less than 10%. -12 When the error rate is lower than 10, the link quality is considered very good, and the impact of the bit error rate is small. Although the transport layer cannot directly perceive the link quality, it can perceive retransmission events caused by bit errors. It can calculate the equivalent bit error rate by the frequency of retransmission events, and then judge the link quality by the bit error rate. For example, if the bit error rate is less than 10, the link quality is considered good. -12 If the link quality is considered good, the retransmission mechanism can be turned off.

[0052] See Figure 9 Traffic monitoring can statistically determine the total amount of data transmitted (X) and the amount of data retransmitted (Y) per unit time. The amount of retransmitted data can be equated to the retransmission overhead caused by bit errors. This retransmission overhead can be measured using the ratio P = Y / X. Therefore, the value of P can be used to measure the physical layer bit error rate. When the bit error rate is less than 10... -12 Even less than 10 -14 When the link quality is considered excellent, the retransmission mechanism can be disabled; when the bit error rate of the equivalent conversion is detected to be greater than 10... -12 If necessary, the retransmission mechanism can be activated again. The value of P is an empirical value set based on transmission characteristics.

[0053] For example, if the number of bits transmitted per unit time t is X and the number of retransmitted bits is Y, then Y should be equal to the number of retransmissions multiplied by the size of the retransmitted data block. The number of retransmissions is affected by the physical layer bit error rate and is related to the size of the data block in which the bit error occurs.

[0054] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A transparent adaptation system for RDMA protocol supporting high RTT environments, characterized in that, It includes two end-side physical nodes and a network-side physical link. By inserting logical transmission nodes between each end-side physical node and the network-side physical link, a multi-stage pipelined transmission channel containing a near-end RDMA transmission channel and a far-end RDMA transmission channel is formed. The logical transmission node includes an ACK near-end acknowledgment function unit for fast data reception confirmation for near-end physical nodes, a transmission link adaptation function unit for RDMA transmission adaptation for near-end physical nodes, a link measurement function unit for delay measurement of the network-side physical link between two logical transmission nodes, and a congestion control function unit for network-side physical link congestion warning for far-end logical transmission nodes. The system decouples the standard RDMA transmission mechanism of the end-side physical node from the physical link state of the network side through logical transmission nodes, so that the end-side physical node is unaware of the latency changes and link type of the physical link on the network side. The ACK near-end response function unit performs the following operations: it implements a fast response through the flow control mechanism for near-end physical nodes, completes the response processing mechanism for protocol A for end-side physical nodes, and generates a response confirmation message for data reception completion for end-side physical nodes. The transmission link adaptation function unit performs the following operations: it carries the received communication data from the near-end physical node into Protocol B of the network-side physical link facing the far-end logical transmission node, thereby realizing RDMA transmission adaptation for the near-end physical node; the adaptation logic includes extracting the DMA communication data of the near-end physical node, and through a queue sharing mechanism, carrying the communication data based on Protocol A into Protocol B for transmission, and conversely, adapting the communication data based on Protocol B into Protocol A, and then transmitting it to the end-side physical node; The link measurement function unit performs the following operations: periodic or on-demand latency measurements on the network-side physical link between two logical transmission nodes; each logical transmission node independently initiates a measurement request and independently responds to the other end's measurement request; the measurement request message carries timestamp information, and the measurement response end copies the timestamp from the measurement request message and fills it into the measurement response message, feeding it back to the measurement request end; the measurement request end calculates the RTT of the network-side physical link transmission based on the timestamp information in the response message.

2. The RDMA protocol transparent adaptation system supporting high RTT environments according to claim 1, characterized in that, The multi-stage pipeline transmission channel includes a three-stage transmission pipeline; The first-level transmission pipeline consists of a source-side physical node and a first logical transmission node, and is a short-distance transmission channel, namely a near-end RDMA transmission channel. The second-level transmission pipeline consists of the first logical transmission node, the network-side physical link, and the second logical transmission node. It is a long-distance transmission channel, namely the remote RDMA transmission channel. The third-level transmission pipeline consists of a second logical transmission node and a destination-side physical node. It operates on the same mechanism as the first-level transmission pipeline and belongs to the near-end RDMA transmission channel.

3. The RDMA protocol transparent adaptation system supporting high RTT environments according to claim 1, characterized in that, Protocol A and Protocol B may be the same or different DMA protocols, depending on the physical link type on the network side.

4. The RDMA protocol transparent adaptation system supporting high RTT environments according to claim 1, characterized in that, The congestion control function unit performs the following operations: flow control based on information from both link measurement and cache status; congestion prediction based on the changing trends of periodic link measurement results and shared cache queue status; and flow control before congestion occurs to avoid congestion.

5. The RDMA protocol transparent adaptation system supporting high RTT environments according to claim 1, characterized in that, The logical transmission node further includes an RTT-aware retransmission function unit, an RTT-aware sequencing function unit, and a retransmission-aware adaptive function unit; wherein... The RTT-aware retransmission function unit is used to obtain the average RTT level of the physical link on the network side based on link measurement, select the corresponding retransmission mechanism for different RTT states, and initialize the retransmission control logic. The RTT-aware sorting function unit is used to set a sorting buffer space based on the maximum RTT of the link transmission, thereby optimizing the transmission latency. The retransmission sensing and adaptive function unit is used to obtain the status of the transmission link in real time through traffic monitoring, and determine whether the link quality requires a retransmission mechanism based on the retransmission ratio of the transmission link. If so, the retransmission mechanism is activated.

6. The RDMA protocol transparent adaptation system supporting high RTT environments according to claim 1, characterized in that, Each logical transmission node contains two transmission interfaces: a forward interface facing the network-side physical link and a backward interface facing the end-side physical node.

7. The RDMA protocol transparent adaptation system supporting high RTT environments according to claim 1, characterized in that, The network-side physical link is selected from optical transport network leased lines, metropolitan area Ethernet leased lines, or multiprotocol label switching virtual private network leased lines according to application requirements, and the selected leased network is equivalent to a physical link with a long RTT.

Images