RDMA protocol transparent adaptation system supporting high RTT environment

Abstract

The invention discloses an RDMA (Remote Direct Memory Access) protocol transparent adaptation system supporting a high RTT (Room-to-Tilt) environment, which is characterized in that a segmented logic transmission channel is formed by introducing logic transmission nodes and inserting two logic transmission nodes between two physical nodes, and a multi-stage assembly line transmission channel consisting of a near-end RDMA transmission channel and a far-end RDMA transmission channel is formed; and logic isolation between the transmission state of the near-end transmission channel and the transmission state of the far-end transmission channel is realized. Meanwhile, link measurement, RTT perception retransmission, RTT perception sorting, a near-end pickup mechanism facing near-end physical nodes and a rapid congestion feedback mechanism facing far-end logic transmission nodes are realized in the logic transmission nodes; the problems of unstable service flow caused by flow control delay, low link utilization rate caused by data retransmission and the like in traditional RDMA transmission in a high RTT environment are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data transmission, and in particular to a RDMA protocol transparent adaptation system supporting high RTT environment. BACKGROUND

[0002] With the rapid development of cloud computing, supercomputing and intelligent computing, cross-data center data migration, especially the remote real-time migration of super-large-scale data, has become a basic requirement for key businesses such as financial transactions, scientific computing, AI training, disaster recovery synchronization, and has become a key bottleneck restricting system performance.

[0003] Data long-distance transmission not only makes the packet loss problem more obvious, but also makes the low utilization of transmission link caused by data retransmission more prominent. Moreover, it is accompanied by long Round-Trip Time (RTT) phenomenon, and the stability problem of business traffic caused by the hysteresis effect of flow control mechanism will also face great challenges.

[0004] Current cross-data center interconnection generally adopts TCP / IP mechanism to realize by renting operator link, and data transmission often needs to pass through the processing of operating system and multi-layer software protocol stack, which not only leads to low data transmission efficiency, but also causes large consumption of CPU resources. In order to solve these problems, the industry generally uses Remote Direct Memory Access (RDMA) mechanism to realize data migration. However, the existing RDMA technical scheme still faces many technical bottlenecks and challenges when facing the demand of long-distance and large-scale real-time data transmission in wide area network environment: 1) The increase of transmission link inherent delay and the long feedback period of congestion control algorithm such as Data Center Quantized Congestion Notification (DCQCN) and TIMELY (RTT-based data center congestion control algorithm) cause the dramatic increase of RTT round-trip delay, which further causes the dramatic increase of data buffer pressure at the sending end.

[0005] 2) The untimely transmission of congestion control information will cause the sending end to be unable to quickly perceive the network state change, resulting in congestion response hysteresis. The hysteresis congestion control will cause unstable flow adjustment, causing business traffic to fluctuate dramatically, and even possibly packet loss.

[0006] 3) The single data retransmission mechanism facing the changing RTT link will further reduce the link utilization, causing waste of transmission bandwidth resources.

[0007] Briefly, the existing RDMA technology architecture faces the challenge of sharp decline in transmission performance when facing long-distance, large-scale, real-time data movement requirements, in order to solve this problem, the present application provides a transparent adaptation system of RDMA protocol supporting high RTT environment, aiming at improving the efficiency and stability of large bandwidth, long distance data movement, and providing a more efficient and reliable solution for long distance data movement application. SUMMARY

[0008] In view of the problems of poor stability and low utilization rate of communication bandwidth resources of the existing RDMA technology in long-distance data transmission, the technical scheme introduces a logical transmission node, divides the existing physical link of data transmission based on the logical transmission node, adopts a multi-stage pipeline segmented communication idea, and provides a transparent adaptation system of RDMA protocol supporting high RTT environment.

[0009] In order to achieve the above purpose, the present application is realized by the following technical scheme: The present application provides a transparent adaptation system of RDMA protocol supporting high RTT environment, comprising two end-side physical nodes and network-side physical links, by inserting a logical transmission node between each end-side physical node and network-side physical link, a multi-stage pipeline transmission channel comprising a near-end RDMA transmission channel and a far-end RDMA transmission channel is formed. The logical transmission node comprises an ACK near-end answering function unit for realizing fast data reception confirmation for the near-end physical node, a transmission link adaptation function unit for realizing RDMA transmission adaptation for the near-end physical node, a link measurement function unit for measuring the delay of the network-side physical link between the two logical transmission nodes, and a congestion control function unit for realizing network-side physical link congestion early warning for the far-end logical transmission node. The system realizes the decoupling of the standard RDMA transmission mechanism of the end-side physical node and the state of the network-side physical link through the logical transmission node, so that the end-side physical node is not aware of the delay change and link type of the network-side physical link.

[0010] Further, the multi-stage pipeline transmission channel comprises a three-stage transmission pipeline. The first-stage transmission pipeline is composed of a source end-side physical node and a first logical transmission node, and is a short-distance transmission channel, that is, a near-end RDMA transmission channel; the second-stage transmission pipeline is composed of the first logical transmission node, a network-side physical link and a second logical transmission node, and is a long-distance transmission channel, that is, a far-end RDMA transmission channel; the third-stage transmission pipeline is composed of the second logical transmission node and a destination end-side physical node, and has the same mechanism as the first-stage transmission pipeline, and belongs to the near-end RDMA transmission channel.

[0011] Further, the ACK near-end answering function unit performs the operation of: implementing a fast response to a flow control mechanism of the near-end physical node, completing an answering processing mechanism of the protocol A of the end-side physical node, and generating an answering confirmation message of data receiving completion of the end-side physical node.

[0012] Further, the transmission link adaptation function unit performs the operation of: carrying the received near-end physical node communication data to the protocol B of the network-side physical link facing the far-end logical transmission node, and implementing the RDMA transmission adaptation facing the near-end physical node; the adaptation logic includes extracting the DMA communication data of the near-end physical node, carrying the communication data based on the protocol A to the protocol B for transmission through a queue sharing mechanism, and vice versa, adapting the communication data based on the protocol B to the protocol A, and then transmitting to the end-side physical node.

[0013] Further, the protocol A and the protocol B are the same or different DMA protocols, depending on the network-side physical link type.

[0014] Further, the link measurement function unit performs the operation of: periodically or on-demand measuring the delay of the network-side physical link between the two logical transmission nodes; the two logical transmission nodes each independently initiate a measurement request and also each independently respond to the measurement request of the opposite end; the measurement request message carries timestamp information, the measurement response end copies the timestamp in the measurement request message to fill into the measurement response message and feeds back to the measurement request end, and the measurement request end calculates the RTT of the network-side physical link transmission according to the timestamp information in the response message.

[0015] Further, the congestion control function unit performs the operation of: performing flow control based on the information of link measurement and buffer state; predicting congestion based on the periodic link measurement result and the change trend of the shared buffer queue state, and performing flow control before the congestion occurs to avoid the congestion.

[0016] Further, the logical transmission node further includes an RTT-aware retransmission function unit, an RTT-aware ordering function unit, and a retransmission-aware adaptive function unit; wherein, The RTT-aware retransmission function unit is configured to obtain the average RTT level of the network-side physical link based on the link measurement, select a corresponding retransmission mechanism for different RTT states, and initialize the retransmission control logic; The RTT-aware ordering function unit is configured to set an ordering buffer space based on the maximum RTT of the link transmission, so as to optimize the transmission delay; The retransmission-aware adaptive function unit is configured to acquire the state of the transmission link in real time through traffic monitoring, judge whether the retransmission mechanism is needed based on the retransmission ratio of the transmission link, and if so, activate the retransmission mechanism.

[0017] Further, each logical transport node contains two transport interfaces: a forward interface facing the network-side physical link and a backward interface facing the end-side physical node.

[0018] Further, the network-side physical link selects an optical transport network private line, a metropolitan Ethernet private line or a multiprotocol label switching virtual private network private line according to application requirements, and equivalent to a section of physical link with long RTT.

[0019] Compared with the prior art, the beneficial effects of the present application are: the RDMA protocol transparent adaptation system supporting high RTT environment provided by the present application aims to improve the efficiency and stability of long-distance data movement oriented to large bandwidth, and provides a more efficient and reliable solution for long-distance data movement application. The present application can be used to modify and adapt the existing short-distance RDMA communication network or build a new long-distance efficient RDMA communication transmission system, and solve the problems of poor stability and low data transmission efficiency of the existing RDMA communication mechanism in long-distance data transmission. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor: Figure 1 The figure is a schematic diagram of the three-stage flow adaptation logic in the present application; Figure 2 The figure is a schematic diagram of the physical node structure in the present application; Figure 3 The figure is a functional structure diagram of the logical transport node in the present application; Figure 4 The figure is a schematic diagram of the physical link in the present application; Figure 5 The figure is a schematic diagram of the near-end answering in the present application; Figure 6 The figure is a schematic diagram of the link measurement in the present application; Figure 7 The figure is a schematic diagram of the congestion control process in the present application; Figure 8 The figure is a schematic diagram of the retransmission mechanism selection in the present application; Figure 9 The figure is a schematic diagram of the retransmission awareness adaptive mechanism in the present application. DETAILED DESCRIPTION

[0021] The application provides a RDMA protocol transparent adaptation system supporting a high RTT environment, which can be used for adapting and reconstructing an existing short-distance RDMA communication network or constructing a new long-distance high-efficiency RDMA communication transmission system, and solves the problems of poor stability and low data transmission efficiency of the existing RDMA communication mechanism in long-distance data transmission.

[0022] The application provides a RDMA protocol transparent adaptation system supporting a high RTT environment, which comprises two end-side physical nodes and a network-side physical link, and a logical transmission node is inserted between each end-side physical node and the network-side physical link to form a multi-stage pipeline transmission channel comprising a near-end RDMA transmission channel and a far-end RDMA transmission channel. The logical transmission node comprises an ACK near-end answering function unit for realizing fast data reception confirmation for the near-end physical node, a transmission link adaptation function unit for realizing RDMA transmission adaptation for the near-end physical node, a link measurement function unit for measuring the delay of the network-side physical link between the two logical transmission nodes, and a congestion control function unit for realizing network-side physical link congestion early warning for the far-end logical transmission node. The system realizes the decoupling of the standard RDMA transmission mechanism of the end-side physical node and the state of the network-side physical link through the logical transmission node, so that the end-side physical node is not aware of the delay change and the link type of the network-side physical link.

[0023] The whole end-to-end data remote transmission system is composed of three-stage transmission pipelines, as shown in the figure, Figure 1 The end-side physical node A1 and the logical transmission node B1 form the first-stage transmission pipeline, which is a short-distance transmission channel, referred to as a near-end RDMA transmission channel here, and the data transmission mechanism is the same as the existing standard RDMA mechanism and will not be described herein. The second-stage transmission pipeline is composed of the logical transmission node B1, the logical transmission node B2 and a network-side physical link C, and is a long-distance transmission channel, which is also the core of the whole adaptation system. The third-stage transmission pipeline is composed of the logical transmission node B2 and the end-side physical node A2, and has the same mechanism as the first-stage transmission pipeline and also belongs to the near-end RDMA transmission channel. Here, the communication model of the end-side physical node is abstracted as a simple structure composed of a communication node and a direct memory access (DMA) engine, as shown in the figure, Figure 2 The communication node is responsible for the transmission control of the business data, including the active initiation and passive response of the data DMA transmission operation, and the DMA engine built in the RNIC (remote network interface card) is responsible for the first-stage transmission of the business data, and the business traffic is stored into the cache of the logical transmission node without blocking, and the logical transmission node is responsible for the lossless transmission based on the physical network super-long link. The functional structure diagram of the logical transmission node is as shown in the figure,Figure 3 As shown, the ACK near-end answering, transmission link adaptation, congestion control, link measurement, packet retransmission (RTT-aware retransmission), out-of-order rearrangement (RTT-aware rearrangement), and other logical functional units and buffer queue management mechanisms are included. The network side physical link refers to the actual available physical network, and according to application requirements, an optical transport network (OTN) private line, a metropolitan area Ethernet private line, or a multiprotocol label switching virtual private network (MPLS VPN) private line, etc. can be selected. Here, the private line network is equivalent to a physical link with a long RTT.

[0024] As shown in Figure 5 The ACK near-end answering functional unit mainly faces the flow control mechanism of the near-end RDMA physical node to achieve fast response, completes the response processing mechanism of the protocol A of the end-side physical node, and generates a data reception completion response confirmation message by the end-side physical node A1. Since the ACK near-end answering functional unit and the near-end physical node belong to short-distance point-to-point communication, the network side physical link error rate and packet loss rate are extremely low, and the influence of packet loss is no longer considered. The ACK near-end answering functional unit mainly faces the near-end RDMA transmission node to achieve fast confirmation of data reception.

[0025] As shown in Figure 4 The transmission link adaptation functional unit receives the communication data of the near-end RDMA physical node, and carries the data to the communication protocol B of the network side physical link facing the far-end logical transmission node, realizes the RDMA transmission adaptation of the near-end physical node, and mainly completes the adaptation processing of the DMA protocol of the near-end physical node based on the link type and data transmission mechanism of the leased long-distance physical network. The adaptation logic includes extracting the DMA communication data of the near-end physical node, carrying the communication data based on the protocol A to the protocol B for transmission through the queue sharing mechanism, and vice versa. The communication data based on the protocol B is adapted to the protocol A, and then transmitted to the end-side physical node. The protocols A and B can be the same DMA protocol or different protocols, which depends on the link type of the long-distance physical network.

[0026] As shown in Figure 6As shown, the link measurement function unit mainly completes the communication RTT delay measurement and delay monitoring between two logical transmission nodes based on network side physical link C; the logical transmission node B1 and the logical transmission node B2 each independently initiates a measurement request and each independently responds to the measurement request of the opposite end; the time stamp information is carried in the measurement request message, the measurement response end copies the time stamp in the measurement request message to fill in the measurement response message and feeds back to the measurement request end, and the measurement request end can calculate the RTT of the network side physical link transmission according to the time stamp information in the response message.

[0027] As shown in Figure 7 The congestion control function unit mainly realizes flow control based on the information of link measurement and buffer state. The congestion prediction is based on the periodic link measurement results and the change trend of the shared buffer queue state, and the flow control is performed before the congestion occurs to avoid congestion. Taking the congestion control of the logical transmission node B2 to the logical transmission node B1 direction as an example, the communication data from the B2 node is received by the B1 node, and the communication data is sent to the A1 node through the queue sharing mechanism and the protocol conversion logic. When the A1 node cannot timely receive all the data, the data will accumulate in the B1 shared buffer, and then the buffer space occupancy rate will gradually increase. The congestion prediction mechanism will timely generate a congestion warning message when the congestion has not occurred, generate a flow control message, and notify the sending end of the data. After receiving the flow control message, the B2 node timely controls the sending flow to avoid real congestion. At the same time, based on the periodic measurement mechanism, the B1 node can also perceive the network RTT state change of the physical link between the B2 node, and the congestion prediction logic of the B1 node will detect the RTT measurement value in real time. When the RTT suddenly increases stably, a congestion warning will be generated, and the flow control mechanism of the B2 node will also be triggered.

[0028] In order to adapt to different RTT levels of physical links, RTT-aware retransmission mechanism, RTT-aware ordering mechanism and retransmission-aware adaptive mechanism are introduced in the logical transmission node.

[0029] As shown in Figure 8 The RTT-aware retransmission function unit, the main idea of the RTT-aware retransmission mechanism is to select different retransmission mechanisms for different RTT level physical links, the main purpose is to pursue better link utilization rate on the basis of ensuring lossless transmission. Due to the difference in retransmission overhead accompanied by different retransmission mechanisms when retransmission occurs, the link utilization rate will be affected, and when facing different RTT physical networks, matching different retransmission mechanisms can achieve the lowest link overhead and achieve the best performance. The RTT-aware retransmission function unit can obtain the average RTT level of the physical network based on link measurement, select the optimal retransmission mechanism corresponding to different RTT states, and initialize the retransmission control logic.

[0030] RTT-aware ordering function unit, the main purpose of RTT-aware ordering is to optimize transmission delay. In the high RTT transmission scenario (RTT is greater than 50 microseconds), the transmission mechanism of Go-Back-N (back N frame protocol) will transmit too much useless data when retransmission occurs, resulting in serious damage to link utilization. In order to weaken the influence of this retransmission mechanism on link utilization, selective retransmission mechanism is used for retransmission in high RTT transmission scenario, but this mechanism will cause data stream disorder, and in some applications with ordering requirements, ordering mechanism must be used for ordering processing of data stream. However, the ordering operation is greatly affected by RTT, that is, the ordering cache must be greater than or equal to the bandwidth-delay product (Bandwidth-Delay Product, BDP) corresponding to the link delay under RTT / 2, and too large cache will cause additional transmission delay, and too small cache will cause ordering failure. RTT-aware ordering mechanism is based on the maximum RTT of link transmission, and the ordering cache space is accurately set, so as to realize the optimization of transmission delay.

[0031] Re-transmission-aware adaptive function unit, the main purpose of re-transmission-aware adaptive mechanism is to optimize the cache occupation of logical transmission node data sending end and the link transmission delay. The re-transmission-aware adaptive mechanism automatically monitors the traffic state of the transmission link in the logical transmission node, mainly to count the cumulative effective transmission data amount sent by the local end; and monitors the retransmission request state from the remote logical transmission node, mainly to calculate the retransmission data amount based on the retransmission request, and the ratio of the two can measure the link quality from the near-end logical transmission node to the remote transmission node. When the link quality reaches a certain level, it can be considered that the link quality is very good, and then the retransmission mechanism can be closed, further shortening the transmission delay of the link, at the same time releasing the cache space reserved for the retransmission mechanism, and using the released cache for the cache of reverse transmission data, improving the available cache space of reverse DMA transmission. The working logic flow chart of the mechanism is shown in Figure 9 The mechanism can realize the adaptive activation of the retransmission mechanism through real-time monitoring of the link quality, so as to optimize the cache and output delay.

[0032] It should be noted that, during RDMA transmission, due to the problem of packet loss caused by physical layer link error, data retransmission function is needed to ensure the reliability of data transmission, but data retransmission will affect the link bandwidth utilization and reduce the performance of RDMA. Generally, the link error rate is less than 10 -12Or lower, the link quality is considered very good, and the error rate has little effect; the transmission layer cannot directly perceive the link quality, but it can perceive the retransmission events caused by errors, and can calculate the equivalent error rate through the frequency of retransmission events, and then judge the link quality through the error rate, such as when the error rate is less than 10 -12 , the link quality is considered to be good, and the retransmission mechanism can be closed.

[0033] Referring to Figure 9 , the total amount of data transmitted per unit time X and the amount of retransmitted data Y can be counted through flow monitoring, the amount of retransmitted data is equivalent to the retransmission overhead caused by errors, and the retransmission overhead caused by errors can be measured by the ratio of P=Y / X, then the value of P can be used to measure the physical layer error rate, when the error rate is less than 10 -12 or even less than 10 -14 , the link quality is considered to be very good, and the retransmission mechanism can be closed; when the equivalent error rate is greater than 10 -12 , the retransmission mechanism can be started again. The value of P is an empirical value set according to the transmission characteristics.

[0034] Exemplarily, assuming that the number of bits transmitted per unit time t is X, and the number of bits retransmitted is Y, then Y should be equal to the number of retransmissions multiplied by the size of the retransmitted data block, and the number of retransmissions is related to the physical layer error rate and the size of the transmission data block where the error occurs.

[0035] The above embodiments are only used to illustrate the design idea and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and implement it, and the protection scope of the present application is not limited to the above embodiments. Therefore, any equivalent changes or modifications made according to the principles and design ideas disclosed by the present application are within the protection scope of the present application.

Claims

1. A transparent adaptation system for RDMA protocol supporting high RTT environments, characterized in that, The system comprises two end-side physical nodes and network-side physical links, and a multi-stage pipeline transmission channel is formed by inserting logical transmission nodes between each end-side physical node and network-side physical link, which comprises a near-end RDMA transmission channel and a far-end RDMA transmission channel. The logical transmission node comprises an ACK near-end answering function unit for realizing fast data reception confirmation for the near-end physical node, a transmission link adaptation function unit for realizing RDMA transmission adaptation for the near-end physical node, a link measurement function unit for measuring the delay of the network-side physical link between two logical transmission nodes, and a congestion control function unit for realizing network-side physical link congestion early warning for the far-end logical transmission node. The system decouples the standard RDMA transmission mechanism of the end-side physical node and the state of the network-side physical link through the logical transmission node, so that the end-side physical node is not aware of the delay change and link type of the network-side physical link.

2. The RDMA protocol transparent adaptation system for supporting high RTT environments of claim 1, wherein, The multi-stage pipeline transmission channel comprises three-stage transmission pipelines. The first-stage transmission pipeline is composed of a source end-side physical node and a first logical transmission node, and is a short-distance transmission channel, i.e., a near-end RDMA transmission channel. The second-stage transmission pipeline is composed of the first logical transmission node, a network-side physical link and a second logical transmission node, and is a long-distance transmission channel, i.e., a far-end RDMA transmission channel. The third-stage transmission pipeline is composed of the second logical transmission node and a destination end-side physical node, and has the same mechanism as the first-stage transmission pipeline, belonging to the near-end RDMA transmission channel.

3. The RDMA protocol transparent adaptation system for supporting high RTT environments of claim 1, wherein, The ACK near-end answering function unit realizes fast response for the flow control mechanism of the near-end physical node, completes the answering processing mechanism for the end-side physical node protocol A, and generates a data reception completion answering confirmation message for the end-side physical node.

4. The RDMA protocol transparent adaptation system for supporting high RTT environments of claim 3, wherein, The transmission link adaptation function unit carries the received near-end physical node communication data to the network-side physical link protocol B facing the far-end logical transmission node, realizes RDMA transmission adaptation for the near-end physical node, and the adaptation logic comprises extracting the DMA communication data of the near-end physical node, carrying the protocol A-based communication data to the protocol B through a queue sharing mechanism for transmission, and vice versa, i.e., adapting the protocol B-based communication data to the protocol A for transmission to the end-side physical node.

5. The RDMA protocol transparent adaptation system for supporting high RTT environments of claim 4, wherein, The protocol A and the protocol B are the same or different DMA protocols, depending on the network-side physical link type.

6. The RDMA protocol transparent adaptation system for high RTT environments of claim 1, wherein, The link measurement function unit periodically or on-demand measures the delay of the network-side physical link between two logical transmission nodes, and two logical transmission nodes independently initiate measurement requests and respond to measurement requests from the opposite end. The measurement request message carries timestamp information, the measurement response end copies the timestamp in the measurement request message into the measurement response message and feeds back to the measurement request end, and the measurement request end calculates the RTT of the network-side physical link transmission according to the timestamp information in the response message.

7. The RDMA protocol transparent adaptation system for high RTT environments of claim 1, wherein, The execution operation of the congestion control function unit is to control flow based on information of link measurement and buffer state; to predict congestion based on periodic link measurement results and change trend of shared buffer queue state, and to control flow before congestion occurs to avoid congestion.

8. The RDMA protocol transparent adaptation system for high RTT environment support of claim 1, wherein, The logical transmission node further comprises an RTT-aware retransmission function unit, an RTT-aware sequencing function unit and a retransmission-aware adaptive function unit; wherein, The RTT-aware retransmission function unit is configured to obtain average RTT level of the network-side physical link based on link measurement, select a retransmission mechanism corresponding to different RTT states, and initialize retransmission control logic; The RTT-aware sequencing function unit is configured to set sequencing buffer space based on maximum RTT of link transmission, thereby optimizing transmission delay; The retransmission-aware adaptive function unit is configured to acquire state of the transmission link in real time through traffic monitoring, judge whether the link quality needs a retransmission mechanism based on retransmission ratio of the transmission link, and activate the retransmission mechanism if yes.

9. The RDMA protocol transparent adaptation system for high RTT environment according to claim 1, wherein, Each logical transmission node comprises two transmission interfaces: a forward interface facing the network-side physical link and a backward interface facing the end-side physical node.

10. The RDMA protocol transparent adaptation system for supporting high RTT environments of claim 1, wherein, The network-side physical link selects an optical transport network private line, a metropolitan Ethernet private line or a multiprotocol label switching virtual private network private line according to application requirements, and equivalently converts the selected private line network into a section of physical link with long RTT.

Images