RDMA (Remote Direct Memory Access) transmission method between data centers and Internet gateway equipment

By deploying Internet-restricted devices at the data center exit, rapid retransmission and congestion management of data packets are achieved, and the problem of low RDMA transmission efficiency in cross-data center networks is solved, and data transmission efficiency and bandwidth utilization are improved.

CN119030944APending Publication Date: 2024-11-26PENG CHENG LAB
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411176376.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-26

Smart Images

  • Figure CN119030944A_ABST
    Figure CN119030944A_ABST
Patent Text Reader

Abstract

The invention discloses an inter-data-center RDMA transmission method and an internet gateway device, the inter-data-center RDMA transmission method is applied to the internet gateway device deployed at an exit of a data center, and the method comprises the following steps: receiving a notification data packet fed back by a home terminal data center based on an inter-data-center data flow, determining expected retransmission data packet information according to the notification data packet; and carrying out data packet retransmission according to the expected retransmission data packet information and the cached data packet. And the Internet gateway equipment has the capabilities of packet loss sensing, congestion management, packet loss retransmission and the like, and is used for supporting the operation of the RDMA transmission method between the data centers. According to the invention, when the data packet is lost, the data packet is retransmitted according to the information of the data packet expected to be retransmitted and the data packet cached on the gateway equipment of the data center at the home terminal, so that the data packet is retransmitted compared with the existing mode of retransmitting the data packet by triggering a fallback N-step protocol retransmission mechanism; according to the method, the data retransmission efficiency during packet loss can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to an RDMA transmission method between data centers and an Internet gateway device. Background Art

[0002] In recent years, the scale of data center (DC) infrastructure has expanded significantly to support the needs of scientific research, big data processing, and artificial intelligence. Due to resource constraints such as land, energy, and connectivity, large cloud service providers usually deploy multiple data centers in different regions to balance user needs and operational economics. At the same time, applications that rely on cross-data center networks are also evolving. Many data-intensive applications, such as data analysis, machine learning, graph processing, and supercomputing, involve large amounts of data distributed across multiple data centers. In particular, with the rapid development of the field of artificial intelligence, the explosive growth of its model training parameters has also increased the demand for computing power cluster size. Therefore, providing stronger computing power to support ultra-large-scale intelligent computing needs by combining the computing power of data centers in multiple regions has also become a recent research hotspot.

[0003] Remote Direct Memory Access (RDMA) has gradually become a standard technology for data center networks with its advantages of ultra-low latency, high throughput, and low CPU resource consumption. However, RDMA technology requires the underlying network to provide lossless network support. The existing Priority Flow Control (PFC) is usually used to ensure the losslessness of switch ports. However, the use of PFC usually brings problems such as header congestion, PFC pause frame storms, and even network deadlocks. Therefore, academia and industry have introduced technologies such as the switch's explicit congestion notification (ECN) marking, end-to-end round-trip time (RTT) delay, and in-band network telemetry (INT), and designed and implemented a series of congestion control protocols (such as the DCQCN algorithm, TIMELY algorithm, HPCC algorithm, etc.). By applying these congestion control protocols, the sender can adjust the rate more agilely and accurately to reduce congestion in the network.

[0004] However, the response time of such end-to-end congestion control protocols, i.e., the delay of the control loop, is at least one round trip time (RTT). In a cross-data center network environment, since the delay of the feedback loop is usually tens of milliseconds or even more, the sender needs to experience a long delay before receiving the congestion signal. During this period, the sender continues to transmit data packets, resulting in the accumulation and overflow of the data queue, causing packet loss. In order to solve the problem of high end-to-end congestion control delay of RDMA in wide area networks, various methods have been proposed in related technologies to reduce the response time of the sender to congestion notifications. However, these methods still cannot avoid the lag of notifications between data centers caused by long-distance fiber transmission delays, so the risk of packet loss caused by microbursts still exists. The Go-Back-N (GBN) retransmission mechanism triggered after packet loss not only seriously affects the flow completion time (FCT), but also leads to extremely low bandwidth utilization. Therefore, how to improve the efficiency of data transmission between data centers has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide an inter-data center RDMA transmission method and an Internet gateway device, aiming to solve the technical problem of low efficiency of data transmission between data centers.

[0006] To achieve the above object, the present application proposes an inter-data center RDMA transmission method, which is applied to an Internet gateway device deployed at the data center egress, and the inter-data center RDMA transmission method includes the following steps:

[0007] Receive notification data packets from the local data center based on data flow feedback between data centers;

[0008] Determining expected retransmission data packet information according to the notification data packet;

[0009] The data packet is retransmitted according to the expected retransmission data packet information and the buffered data packet.

[0010] Optionally, the step of retransmitting the data packet according to the expected retransmission data packet information and the cached data packet includes:

[0011] Determining whether the retransmitted data packet corresponding to the expected retransmitted data packet information is overwritten based on the data packet cache information;

[0012] If the retransmitted data packet is not covered, retransmitting the data packet from the storage module according to the expected retransmitted data packet information and the cached data packet;

[0013] If the retransmitted data packet has been covered, an inter-data center packet loss notification is generated based on the expected retransmitted data packet information, and the inter-data center packet loss notification is sent to the opposite data center corresponding to the inter-data center data flow.

[0014] Optionally, after the step of receiving a notification data packet fed back by the local data center based on the data flow between data centers, the step further includes:

[0015] Classifying the notification data packet, and when the classification result is a congestion notification, determining a data packet transmission suspension time according to the notification data packet;

[0016] generating a congestion notification data packet based on the data packet transmission suspension time;

[0017] The congestion notification data packet is sent to a peer data center corresponding to the inter-data center data flow, and the peer data center suspends sending the inter-data center data flow based on the data packet sending suspension time in the congestion notification data packet.

[0018] Optionally, before the step of receiving a notification data packet fed back by the local data center based on the data flow between data centers, the step further includes:

[0019] When receiving an inter-data center data packet sent by a peer data center, determining whether an inter-data center data flow corresponding to the inter-data center data packet is marked as a congested state according to the flow state table;

[0020] If the inter-data center data flow is not marked as a congested state, obtaining an expected data packet sequence number of the inter-data center data flow;

[0021] The inter-data center data packets are cached and forwarded according to the expected data packet sequence number.

[0022] Optionally, the step of caching and forwarding the inter-data center data packet according to the expected data packet sequence number includes:

[0023] Comparing the expected data packet sequence number with an actual data packet sequence number corresponding to the inter-data center data packet;

[0024] When the expected data packet sequence number is equal to the actual data packet sequence number, determining whether the inter-data center data flow is in a retransmission state;

[0025] When the data flow between the data centers is not in a retransmission state, buffering the data packets between the data centers, and forwarding the data packets between the data centers to the local data center;

[0026] When the data flow between the data centers is in a retransmission state, the data packets between the data centers are cached and not forwarded to the local data center.

[0027] Optionally, the inter-data center RDMA transmission method further includes:

[0028] When receiving a data packet to be sent sent by the local data center, judging whether the data flow to be sent corresponding to the data packet to be sent is in a congested state according to the sending flow state table;

[0029] When the data stream to be sent is not in a congested state, determining a sequence number of a data packet to be sent corresponding to the data stream to be sent;

[0030] When the sequence number of the data packet to be sent is equal to the sequence number of the data packet corresponding to the data packet to be sent, the data packet to be sent is sent to the opposite-end data center corresponding to the data packet to be sent.

[0031] Optionally, the flow state table of the Internet gateway device maintains the flow state of the data flow between data centers, and the maintenance step includes:

[0032] After receiving the to-be-sent data packet sent by the local data center and the inter-data center data packet sent by the opposite data center, judging whether the actual data packet number of the received data packet is consistent with the corresponding expected data packet number according to the flow state table, if not consistent, marking the flow state corresponding to the received data packet as a congested state;

[0033] After receiving the to-be-sent data packet sent by the local data center and the inter-data center data packet sent by the opposite data center, it is determined whether the actual data packet number of the received data packet is consistent with the corresponding expected data packet number according to the flow state table. If they are consistent, the flow state corresponding to the received data packet is marked as a non-congested state.

[0034] Optionally, the retransmission state is maintained by a flow state table of the Internet gateway device, and the maintenance step includes:

[0035] After the data packet retransmission process is started, the corresponding flow state is marked as the retransmission state until the retransmission process ends and the retransmission state in the corresponding flow state is cleared.

[0036] Optionally, when the flow state corresponding to the received data packet is a congested state, the received data packet will be discarded to reduce the transmission of redundant data.

[0037] In addition, to achieve the above purpose, the present application also proposes an Internet gateway device, which is applied to the above inter-data center RDMA transmission method, and the Internet gateway device includes:

[0038] A receiving module, used to receive a notification data packet from the local data center based on the data flow feedback between data centers;

[0039] A determination module, configured to determine information of a desired retransmission data packet according to the notification data packet;

[0040] The data packet retransmission module is used to retransmit the data packet according to the expected retransmission data packet information and the cached data packet.

[0041] The present application receives a notification data packet from the local data center based on the data flow feedback between data centers, determines the expected retransmission data packet information according to the notification data packet, and retransmits the data packet according to the expected retransmission data packet information and the cached data packets. The Internet gateway device has the capabilities of packet loss perception, congestion management, and packet loss retransmission, which are used to support the operation of the inter-data center RDMA transmission method. Since the present application retransmits data packets according to the expected retransmission data packet information and the data packets cached on the Internet gateway device of the local data center when data packets are lost, compared with the existing method of triggering the back-off N-step protocol retransmission mechanism to retransmit data packets, the above method of the present application can improve the efficiency of data retransmission when packets are lost. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart of the first embodiment of the RDMA transmission method between data centers of the present application is provided;

[0045] Figure 2 A schematic diagram of the Internet gateway device structure provided for the first embodiment of the RDMA transmission method between data centers of the present application;

[0046] Figure 3 A schematic diagram of a data transmission process provided for the first embodiment of the RDMA transmission method between data centers of the present application;

[0047] Figure 4 A flow chart of the second embodiment of the RDMA transmission method between data centers of the present application;

[0048] Figure 5 This is a schematic diagram of the module structure of the Internet gateway device according to an embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: receiving a notification data packet from the local data center based on the data flow feedback between data centers, determining the expected retransmission data packet information according to the notification data packet; and retransmitting the data packet according to the expected retransmission data packet information and the cached data packets. The Internet gateway device has the capabilities of packet loss perception, congestion management, and packet loss retransmission, which are used to support the operation of the inter-data center RDMA transmission method. Since the present application retransmits data packets according to the expected retransmission data packet information and the data packets cached on the Internet gateway device of the local data center when data packets are lost, compared with the existing method of triggering the fallback N-step protocol retransmission mechanism to retransmit data packets, the above method of the present application can improve the efficiency of data retransmission when packets are lost.

[0053] Since the present application retransmits data packets based on the expected retransmission data packet information and the data packets cached on the Internet gateway device when data packets are lost, compared with the existing method of triggering the back-off N-step protocol retransmission mechanism to retransmit data packets, the above method of the present application can improve the data retransmission efficiency when packets are lost.

[0054] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an Internet gateway device, etc. The following takes the Internet gateway device as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the embodiment of the present application provides an inter-data center RDMA transmission method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the inter-data center RDMA transmission method of the present application.

[0056] In this embodiment, the inter-data center RDMA transmission method is applied to an Internet gateway device installed in a data center, and the inter-data center RDMA transmission method includes steps S10 to S30:

[0057] Step S10, receiving a notification data packet from the local data center based on the data flow feedback between data centers;

[0058] It should be noted that in this embodiment, an Internet gateway (DCI-GW) device is deployed at the exit of each data center (DC). The Internet gateway device is designed with flow state table, congestion management, retransmission perception, storage, data packet post-processing, notification generation and other modules to achieve more accurate congestion notification and timely packet loss retransmission. For details, please refer to Figure 2 , Figure 2 A schematic diagram of the Internet gateway device structure provided for the first embodiment of the RDMA transmission method between data centers of this application; wherein, the classifier: This module is used to identify various types of data packets input into the DCI-GW and assign them to different processing logics according to the type of data packets. In particular, the retransmission notification from the local DC will be diverted to the retransmission logic; the congestion status from the local DC will be diverted to the congestion management logic; the INTER-DC data flow data packets from the local and remote DCs will be assigned to the flow state table for further processing.

[0059] Retransmission perception: Receive the packet loss retransmission notification feedback from the local DC, and parse the retransmission notification data packet, including the flow information required to be retransmitted (generally identified by IP address and QPN), the starting data packet sequence number (Packet Sequence Number, PSN) required to be retransmitted, etc.; based on the parsed information, encapsulate it into a retransmission request and send it to the storage module.

[0060] Flow state table: maintains all current input and output INTER-DC data flow information, records the state information of each flow, and should at least include the most recent packet sequence number (PSN), whether it is marked as congested, and whether it is marked as retransmission state. The most recent PSN is used to detect whether the INTER-DC data flow has packet loss. The congestion state is used to identify whether the current INTER-DC data flow has packet loss and is in the process of obtaining retransmission packets from the opposite DC. The retransmission flag is used to indicate whether the flow is in the state of obtaining packets from the cache for retransmission. This module needs to change the state of the INTER-DC data flow (congestion, retransmission) according to the packet loss and congestion conditions obtained by looking up the table for the data packet and mark the data packet for cache or discard. If necessary, a packet loss notification request needs to be generated to trigger the subsequent packet loss notification generation logic.

[0061] Congestion management: Receive congestion status information from the local DC, infer the congestion status of the INTER-DC data flow based on the feedback information, and send corresponding congestion notifications to the remote DC as a basis for flow control. In particular, the congestion status information from the local DC may vary according to the specific implementation method, and may be a packet loss notification fed back by the local DC, or a congestion notification message CNP in the DCQCN protocol, etc.

[0062] Storage module: stores Inter-DC data packets (i.e., PSN continuous data packets) from the opposite DC that have passed the congestion status check; retrieves and sends retransmission packets according to retransmission requests.

[0063] Packet post-processing: This module mainly performs three functions. The first is to filter out the packets marked as discarded by the flow state table. These packets belong to the INTER-DC data flow that is already in the packet loss retransmission state. They are discarded to save bandwidth resources and avoid the generation of unnecessary packet loss retransmission notifications. The second is to copy the packets marked as cached and send them to the storage module. Finally, the packet loss notification request carried in the metadata of a specific packet is separated and sent to the notification generation module.

[0064] Notification generation: used to generate packet loss notification (generally refers to NACK) and congestion notification. Parse the packet loss notification request from the data packet post-processing module and the congestion notification request from the congestion management module, and encapsulate the parsed information into the corresponding notification message and send it out.

[0065] The general process of this embodiment can be referred to Figure 3 , Figure 3 The data transmission process diagram provided in the first embodiment of the RDMA transmission method between data centers of the present application is shown in FIG. Figure 3 DC1 in the middle, the peer DC is Figure 3 DC2; local DCI-GW Figure 3 DCI-GW1 in the middle, the peer DCI-GW is Figure 3 In DCI-GW2, INTER-DC data flow is the data flow transmitted across DCs. The DCI-GW device can cache the data packets from the opposite DC in the storage module in units of flows. For the packet loss of the INTER-DC data flow (i.e., the data flow from the opposite DC) in the local DC, it can respond quickly through feedback link 1 and retransmit immediately, avoiding the extremely high delay caused by the packet loss notification and retransmission through the long-distance optical fiber feedback; the DCI-GW device can monitor the packet loss of the INTER-DC data flow, and promptly notify the opposite DC through feedback link 2, and retransmit the packet loss in advance; the DCI-GW device can determine the current congestion status of each INTER-DC data flow through the congestion status information received from the local DC, and generate an INTER-DC congestion notification to feedback to the opposite DC. The opposite DC receives the INTER-DC congestion notification and performs corresponding flow control operations.

[0066] In a specific implementation, the INTER-DC data stream from DC2 to DC1 will perform packet loss detection but not cache when passing through DCI-GW2, so as to promptly detect packet loss and request retransmission, thereby avoiding the long delay and subsequent bandwidth waste caused by cross-DC retransmission. After packet loss detection by DCI-GW2, it is transmitted to the DCI-GW device of DC1 (i.e., DCI-GW1) via a long-distance, high-capacity optical fiber. In DCI-GW1, packet loss detection will be performed on the INTER-DC data stream and continuous data packets will be cached, and then forwarded to DC1. Therefore, the notification data packet is a data packet fed back by the local data center, i.e., DC1, based on the currently transmitted data flow between data centers, which may include information such as the sequence number of the currently received data packet.

[0067] In a specific implementation, after receiving the notification data packet from the local data center based on the data flow feedback between data centers, the Internet gateway device classifies the notification data packet and determines the classification result. Specifically, it can be determined whether the notification data packet is a packet loss notification, a congestion notification, or a normal flow status notification. The normal flow status notification means that there is no packet loss or congestion. The notification data packet is assigned to the corresponding processing logic for the next step of processing.

[0068] Step S20, determining information of a desired retransmission data packet according to the notification data packet;

[0069] It should be noted that the determination of the expected retransmission data packet information based on the notification data packet may be that the Internet gateway device parses the notification data packet, and obtains the expected retransmission data packet information when the notification data packet is a packet loss notification. The expected retransmission data packet information may include the IP address, QPN, PSN and other information of the expected retransmission data packet.

[0070] Step S30: retransmitting the data packet according to the expected retransmission data packet information and the cached data packet.

[0071] It should be noted that the data packet retransmission based on the expected retransmission data packet information and the cached data packet can be to determine the cached retransmission data packet from the storage module of the Internet gateway device according to the PSN of the expected retransmission data packet, and forward the retransmission data packet to the local data center (local DC).

[0072] Further, in order to improve the retransmission efficiency of the data packet, the step S30 may include: determining whether the retransmission data packet corresponding to the expected retransmission data packet information is overwritten based on the data packet buffer information;

[0073] If the retransmitted data packet is not covered, retransmitting the data packet from the storage module according to the expected retransmitted data packet information and the cached data packet;

[0074] If the retransmitted data packet has been covered, an inter-data center packet loss notification is generated based on the expected retransmitted data packet information, and the inter-data center packet loss notification is sent to the opposite data center corresponding to the inter-data center data flow.

[0075] It should be noted that the data packet cache information is the data packet information cached in the Internet gateway device, and may include the PSN information of the most recently cached data packet. The judgment based on the data packet cache information whether the retransmission data packet corresponding to the expected retransmission data packet information is overwritten may be based on the PSN information of the most recently cached data packet in the data packet cache information. If the retransmission data packet is not overwritten, the address of the retransmission data packet stored in the storage module is calculated according to the expected retransmission data packet information, and the data packet is retransmitted from the specified position according to the calculated address, and the retransmission rate is consistent with the original INTER-DC data flow rate until the PSN of the retransmission data packet is consistent with the PSN most recently written in the cache module, and the retransmission stops, and the state is synchronized to the flow state table. If the retransmission data packet has been overwritten, it means that there is no retransmission data packet corresponding to the expected retransmission data packet information in the storage module. At this time, an inter-data center packet loss notification can be generated based on the expected retransmission data packet information, so that the inter-data center packet loss notification is sent to the opposite data center, so that the opposite data center retransmits the data packet corresponding to the expected retransmission data packet information.

[0076] In the specific implementation, after the INTER-DC data flow from DC2 is forwarded to DC1, DC1 will feed back a notification data packet to the DCI-GW1 device. When the DCI-GW1 device identifies the notification data packet as a packet loss notification through the classifier, it will divert it to the retransmission perception module for processing. The retransmission perception module parses the packet loss notification, obtains the IP address, QPN, PSN and other information of the expected retransmission data packet, and combines this information into a retransmission request and sends it to the storage module. The data structure of the storage module is abstracted as a two-dimensional circular buffer structure. Each circular buffer is used to store a data packet of an INTER-DC data flow. Each circular buffer maintains a recently written and read PSN for overwrite judgment and retransmission. After receiving the retransmission request, the storage module determines whether the retransmission packet has been overwritten based on the information in the request and the recently written PSN information of the corresponding circular buffer. a) If it is not overwritten, the storage address corresponding to the expected retransmitted data packet is calculated, and the data packet retransmission starts from the specified position according to the calculated address. The retransmission rate is consistent with the original INTER-DC data flow rate until the PSN of the retransmitted data packet is consistent with the PSN recently written to the ring buffer. The retransmission stops and the state is synchronized to the flow state table. b) If it is overwritten, the retransmission request is sent to the notification generation module through the data packet post-processing module, a new Inter-DC packet loss notification is generated, and the data packet retransmission is initiated from the opposite DC through the slow retransmission path.

[0077] Furthermore, in order to improve the data transmission efficiency between data centers, after step S10, the method further includes: classifying the notification data packet, and when the classification result is a congestion notification, determining a data packet suspension sending time according to the notification data packet;

[0078] generating a congestion notification data packet based on the data packet transmission suspension time;

[0079] The congestion notification data packet is sent to a peer data center corresponding to the inter-data center data flow, and the peer data center suspends sending the inter-data center data flow based on the data packet sending suspension time in the congestion notification data packet.

[0080] It should be noted that the determining of the data packet transmission suspension time according to the notification data packet may be to calculate the time that the opposite end needs to suspend transmission according to the rate of the INTER-DC data flow and the number of data packets that need to be retransmitted.

[0081] In a specific implementation, the congestion notification related to the Inter-DC data flow in DC1 will be forwarded to the DCI-GW1 device. The DCI-GW1 device will generate an INTER-DC congestion notification based on the current number of data packets that need to be retransmitted, the INTER-DC data flow rate, and the queue queuing as a reference for the DC flow control of the opposite end. In particular, the form of the congestion notification may vary according to different specific embodiments. This embodiment takes the packet loss notification as an example of the congestion notification. Specifically: When the packet loss notification is used as the congestion notification, the packet loss notification related to the INTER-DC data flow in DC1 will be copied and sent to the congestion management module for processing. The congestion management module will calculate the time that the opposite end needs to suspend sending based on the current rate of the INTER-DC data flow and the number of data packets that need to be retransmitted, and generate a corresponding congestion notification request and send it to the notification generation module to generate an INTER-DC congestion notification data packet. After retransmission is triggered, the data packet sending logic of the INTER-DC data flow changes from the original direct forwarding to indirect forwarding through cache. Therefore, the other end needs to suspend sending for a period of time to adjust the forwarding mode to normal mode. The suspension time only needs to meet the sending time of all retransmission packets to ensure that the bandwidth loss caused by retransmission is minimized.

[0082] This embodiment receives a notification data packet from the local data center based on the data flow feedback between data centers, determines the expected retransmission data packet information according to the notification data packet, and retransmits the data packet according to the expected retransmission data packet information and the cached data packets. The Internet gateway device has the capabilities of packet loss perception, congestion management, and packet loss retransmission, which are used to support the operation of the inter-data center RDMA transmission method. Since this embodiment retransmits data packets according to the expected retransmission data packet information and the data packets cached on the Internet gateway device of the local data center when data packets are lost, compared with the existing method of triggering the back-off N-step protocol retransmission mechanism to retransmit data packets, the above method of this embodiment can improve the efficiency of data retransmission when packets are lost.

[0083] In this embodiment, the INTER-DC data flow from the opposite DC first passes through the packet loss detection of the opposite DCI-GW device, and performs instant packet loss feedback and subsequent filtering of redundant data packets for the packet lost data flow; the local DCI-GW device detects whether the INTER-DC data flow from the opposite DC has packet loss due to congestion. When packet loss does not occur, the DCI-GW will cache these INTER-DC data flows; when packet loss occurs, the caching stops and sends a packet loss notification to the opposite DC until the retransmitted packet arrives at the DCI-GW device, and the caching starts again. The packet loss caused by congestion in the local DC is fed back to the corresponding DCI-GW device in the form of a packet loss notification. The DCI-GW device starts retransmission from the specified retransmission data packet in the cache according to the specific information in the packet loss notification. At the same time, the DCI-GW device can infer the congestion status of each flow through the congestion status fed back from the local DC (which can be based on the statistics of the packet loss notification or the notification message generated by other congestion control protocols), and generate an accurate INTER-DC congestion notification to feed back to the opposite DC. The opposite DC receives the INTER-DC congestion notification and performs corresponding flow control operations. This embodiment can also be used in combination with various advanced congestion control algorithms, by improving the packet retransmission response speed and average bandwidth utilization of these algorithms in micro-burst situations, so as to significantly enhance the performance when used in cross-data center scenarios.

[0084] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , Figure 4 The flowchart provided in the second embodiment of the RDMA transmission method between data centers of the present application includes the following steps before step S10:

[0085] Step S001: upon receiving an inter-data center data packet sent by a peer data center, determining whether an inter-data center data flow corresponding to the inter-data center data packet is marked as being in a congested state according to a flow state table;

[0086] It should be noted that the flow state table is used to maintain all current input and output INTER-DC data flow information. The determination of whether the inter-data center data flow corresponding to the inter-data center data packet is marked as a congested state according to the flow state table can be to first determine whether the inter-data center data packet hits the flow state table. If not, it means that the state information of the data flow corresponding to the inter-data center data packet has not been recorded in the flow state table. At this time, a table entry is added to the INTER-DC data flow to which the inter-data center data packet belongs, which is used to record the state information of the data flow, including data flow information such as packet loss or congestion. If the inter-data center data packet hits the flow state table, it is determined according to the flow state table whether the inter-data center data flow corresponding to the inter-data center data packet is marked as a congested state. The determination of whether the inter-data center data packet hits the flow state table can be determined by the triplet of the inter-data center data packet.

[0087] Step S002: if the data flow between data centers is not marked as being in a congested state, obtaining an expected data packet sequence number of the data flow between data centers;

[0088] It should be noted that the acquisition of the expected data packet sequence number of the data flow between data centers may be to determine the expected data packet sequence number based on the flow state information of the data flow between data centers, and the expected data packet sequence number may be the sequence number of the data packet that the local data center currently expects to receive.

[0089] Step S003: caching and forwarding the inter-data center data packets according to the expected data packet sequence number.

[0090] It should be noted that the caching and forwarding of the inter-data center data packets according to the expected data packet sequence number can be that when the expected data packet sequence number is equal to the sequence number corresponding to the inter-data center data packet, the inter-data center data packet is cached, and when the inter-data center data flow is not in a retransmission state, the inter-data center data packet is forwarded to the local data center.

[0091] Furthermore, in order to improve the RDMA transmission performance across data centers, the step S003 may include: comparing the expected data packet sequence number with the actual data packet sequence number corresponding to the data packet between data centers;

[0092] When the expected data packet sequence number is equal to the actual data packet sequence number, determining whether the inter-data center data flow is in a retransmission state;

[0093] When the data flow between the data centers is not in a retransmission state, buffering the data packets between the data centers, and forwarding the data packets between the data centers to the local data center;

[0094] When the data flow between the data centers is in a retransmission state, the data packets between the data centers are cached and not forwarded to the local data center.

[0095] It should be understood that even if the expected data packet sequence number is equal to the actual data packet sequence number, it does not mean that the data packet between data centers needs to be cached and forwarded at the same time, because if the data flow between data centers is in a retransmission state, it will cause repeated transmission. Therefore, when the expected data packet sequence number is equal to the actual data packet sequence number, it is determined whether the data flow between data centers is in a retransmission state. When the data flow between data centers is not in a retransmission state, the data packet between data centers is cached and forwarded to the local data center. When the data flow between data centers is in a retransmission state, the data packet state is modified to cache only, that is, cached but discarded, to avoid repeated transmission.

[0096] In the specific implementation, the INTER-DC data flow from DC2 to DC1 is transmitted to the DC I-GW device (i.e. DCI-GW1) of DC1 through a long-distance, high-capacity optical fiber after packet loss detection by its local DCI-GW (i.e. DCI-GW2). In DCI-GW1, packet loss detection is performed on the INTER-DC data flow and continuous data packets are cached. Specifically:

[0097] 1) The data stream from DCI-GW 2 enters DCI-GW 1 through long-distance optical fiber transmission, and first queries the flow state table.

[0098] 2) If the data packet does not hit any input flow table entry, add an entry for the INTER-DC data flow to which the data packet belongs; if the data packet hits an output flow table entry, check whether the flow is marked as congested.

[0099] 3) If it is marked as congested, it means that the flow has lost packets and is in the packet loss retransmission state. Compare whether the expected PSN in the flow state information of the flow is equal to the PSN of the data packet.

[0100] a) If they are equal, it means that the data packet is a retransmission data packet requested earlier, the congestion mark of the flow is removed, the expected PSN is modified, the state of the data packet is changed to cache, and the data packet is sent out normally;

[0101] b) If the PSNs are not equal, it means that the data packet requested for retransmission earlier has not arrived yet. The data packet is a redundant data packet and is marked as discarded.

[0102] 4) If it is not marked as congested, compare whether the expected PSN in the flow state information of the flow is equal to the PSN of the data packet.

[0103] a) If they are equal, modify the expected PSN in the flow state information of the flow, and further check whether the flow is marked as retransmission state.

[0104] i. If yes, it means that the current stream is in the state of retransmitting packets from the cache, and the packet status is modified to cache only, that is, cache but discard, to avoid repeated transmission;

[0105] ii. If not, it means that the current flow has no packet loss, and the data packet status is changed to cache, that is, cache and forward.

[0106] b) If they are not equal, it means that packet loss has occurred, the data packet status is marked as discarded, the flow is changed to congestion status, and a packet loss notification request is generated.

[0107] Furthermore, in order to improve cache utilization, decisions can be made based on the congestion status perception and prediction of the current DCI flow according to the currently used congestion control algorithm, that is, only Inter-DC data flows with high congestion risks are cached to improve the cache utilization of the DCI-GW device.

[0108] Further, in order to improve the data transmission efficiency, the present embodiment further includes: when receiving a data packet to be sent sent by the local data center, judging whether the data flow to be sent corresponding to the data packet to be sent is in a congested state according to the sending flow state table;

[0109] When the data stream to be sent is not in a congested state, determining a sequence number of a data packet to be sent corresponding to the data stream to be sent;

[0110] When the sequence number of the data packet to be sent is equal to the sequence number of the data packet corresponding to the data packet to be sent, the data packet to be sent is sent to the opposite-end data center corresponding to the data packet to be sent.

[0111] It should be noted that the data packet to be sent is a data packet to be sent to other data by the local data center. The sending flow state table may be a table for maintaining state information of the data flow corresponding to the data packet to be sent. The sequence number of the data packet to be sent corresponding to the data flow to be sent may be the sequence number of the data packet to be sent currently in the sending flow state table.

[0112] Furthermore, the flow state table of the Internet gateway device maintains the flow state of the data flow between data centers, and the maintenance steps include:

[0113] After receiving the to-be-sent data packet sent by the local data center and the inter-data center data packet sent by the opposite data center, judging whether the actual data packet number of the received data packet is consistent with the corresponding expected data packet number according to the flow state table, if not consistent, marking the flow state corresponding to the received data packet as a congested state;

[0114] After receiving the to-be-sent data packet sent by the local data center and the inter-data center data packet sent by the opposite data center, it is determined whether the actual data packet number of the received data packet is consistent with the corresponding expected data packet number according to the flow state table. If they are consistent, the flow state corresponding to the received data packet is marked as a non-congested state.

[0115] The retransmission state is maintained by the flow state table of the Internet gateway device, and the maintenance step includes:

[0116] After the data packet retransmission process is started, the corresponding flow state is marked as the retransmission state until the retransmission process ends and the retransmission state in the corresponding flow state is cleared.

[0117] Furthermore, in order to reduce the transmission of redundant data, this embodiment discards the received data packet when the flow state corresponding to the received data packet is a congested state, so as to reduce the transmission of redundant data.

[0118] In the specific implementation, please refer to the above Figure 3 , the INTER-DC data flow from DC2 to DC1 will perform packet loss detection but not cache when passing through its local DCI-GW (i.e. DCI-GW2), so as to timely discover packet loss and request retransmission, avoiding the long delay and subsequent bandwidth waste caused by cross-DC retransmission. Specifically:

[0119] 1) The INTER-DC data flow from DC 2 first passes through DCI-GW 2 and queries the flow state table.

[0120] 2) If the data packet does not hit any output flow table entry, add an entry for the INTER-DC data flow to which the data packet belongs; if the data packet hits an output flow table entry, check whether the flow is marked as congested.

[0121] 3) If it is marked as congested, it means that the flow has lost packets and is in the packet loss retransmission state. Compare whether the expected PSN in the flow state information of the flow is equal to the PSN of the data packet.

[0122] a) If they are equal, it means that the data packet is a retransmission data packet requested earlier. The congestion mark of the flow is removed, the expected PSN is modified, and the data packet is sent out normally.

[0123] b) If they are not equal, it means that the data packet requested for retransmission earlier has not arrived yet. The data packet is a redundant data packet and is marked as discarded.

[0124] 4) If it is not marked as congested, compare whether the expected PSN in the flow state information of the flow is equal to the PSN of the data packet.

[0125] a) If they are equal (i.e., no packet loss is detected), the expected PSN in the flow state information of the flow is sent normally and modified.

[0126] b) If they are not equal, it means that packet loss has occurred, the data packet status is marked as discarded, the flow is changed to congestion status, and a packet loss notification request is generated.

[0127] When receiving an inter-data center data packet sent by a peer data center, this embodiment determines whether the inter-data center data flow corresponding to the inter-data center data packet is marked as a congested state according to the flow state table; if the inter-data center data flow is not marked as a congested state, the expected data packet sequence number of the inter-data center data flow is obtained; and the inter-data center data packet is cached and forwarded according to the expected data packet sequence number. When receiving an inter-data center data packet sent by a peer data center, this embodiment determines whether the inter-data center data flow corresponding to the inter-data center data packet is marked as a congested state according to the flow state table, and can timely detect whether it is currently in a congested or packet loss state, thereby improving data transmission efficiency.

[0128] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the RDMA transmission method between data centers of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0129] The present application also provides an Internet gateway device, which is applied to the above-mentioned inter-data center RDMA transmission method. Figure 5 , the Internet gateway device comprises:

[0130] A receiving module 10, configured to receive a notification data packet from a local data center based on data flow feedback between data centers;

[0131] A determination module 20, configured to determine information of a desired retransmission data packet according to the notification data packet;

[0132] The data packet retransmission module 30 is used to retransmit the data packet according to the expected retransmission data packet information and the cached data packet.

[0133] This embodiment receives a notification data packet from the local data center based on the data flow feedback between data centers, determines the expected retransmission data packet information according to the notification data packet, and retransmits the data packet according to the expected retransmission data packet information and the cached data packets. The Internet gateway device has the capabilities of packet loss perception, congestion management, and packet loss retransmission, which are used to support the operation of the inter-data center RDMA transmission method. Since this embodiment retransmits data packets according to the expected retransmission data packet information and the data packets cached on the Internet gateway device of the local data center when data packets are lost, compared with the existing method of triggering the back-off N-step protocol retransmission mechanism to retransmit data packets, the above method of this embodiment can improve the efficiency of data retransmission when packets are lost.

[0134] The Internet gateway device provided by the present application adopts the inter-data center RDMA transmission method in the above embodiment, which can solve the technical problem of low efficiency of data transmission between data centers. Compared with the prior art, the beneficial effects of the Internet gateway device provided by the present application are the same as the beneficial effects of the inter-data center RDMA transmission method provided by the above embodiment, and other technical features in the Internet gateway device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0135] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0136] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0137] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0139] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for RDMA transmission between data centers, characterized in that: The inter-data center RDMA transmission method is applied to an Internet gateway device deployed at a data center exit, and the inter-data center RDMA transmission method comprises the following steps: Receive notification data packets from the local data center based on data flow feedback between data centers; Determining expected retransmission data packet information according to the notification data packet; The data packet is retransmitted according to the expected retransmission data packet information and the buffered data packet.

2. The inter-data center RDMA transmission method according to claim 1, characterized in that: The step of retransmitting the data packet according to the expected retransmission data packet information and the cached data packet comprises: Determining whether the retransmitted data packet corresponding to the expected retransmitted data packet information is overwritten based on the data packet cache information; If the retransmitted data packet is not covered, retransmitting the data packet from the storage module according to the expected retransmitted data packet information and the cached data packet; If the retransmitted data packet has been covered, an inter-data center packet loss notification is generated based on the expected retransmitted data packet information, and the inter-data center packet loss notification is sent to the opposite data center corresponding to the inter-data center data flow.

3. The inter-data center RDMA transmission method according to claim 1, characterized in that: After the step of receiving a notification data packet from the local data center based on the data flow feedback between data centers, the method further includes: Classifying the notification data packet, and when the classification result is a congestion notification, determining a data packet transmission suspension time according to the notification data packet; generating a congestion notification data packet based on the data packet transmission suspension time; The congestion notification data packet is sent to a peer data center corresponding to the inter-data center data flow, and the peer data center suspends sending the inter-data center data flow based on the data packet sending suspension time in the congestion notification data packet.

4. The inter-data center RDMA transmission method according to claim 1, characterized in that: Before the step of receiving a notification data packet from the local data center based on the data flow feedback between data centers, the step further includes: When receiving an inter-data center data packet sent by a peer data center, determining whether an inter-data center data flow corresponding to the inter-data center data packet is marked as a congested state according to the flow state table; If the inter-data center data flow is not marked as a congested state, obtaining an expected data packet sequence number of the inter-data center data flow; The inter-data center data packets are cached and forwarded according to the expected data packet sequence number.

5. The inter-data center RDMA transmission method according to claim 4, characterized in that: The step of caching and forwarding the data packets between data centers according to the expected data packet sequence number includes: Comparing the expected data packet sequence number with an actual data packet sequence number corresponding to the inter-data center data packet; When the expected data packet sequence number is equal to the actual data packet sequence number, determining whether the inter-data center data flow is in a retransmission state; When the data flow between the data centers is not in a retransmission state, buffering the data packets between the data centers, and forwarding the data packets between the data centers to the local data center; When the data flow between the data centers is in a retransmission state, the data packets between the data centers are cached and not forwarded to the local data center.

6. The inter-data center RDMA transmission method according to claim 1, characterized in that: The inter-data center RDMA transmission method further includes: When receiving a data packet to be sent sent by the local data center, judging whether the data flow to be sent corresponding to the data packet to be sent is in a congested state according to the sending flow state table; When the data stream to be sent is not in a congested state, determining a sequence number of a data packet to be sent corresponding to the data stream to be sent; When the sequence number of the data packet to be sent is equal to the sequence number of the data packet corresponding to the data packet to be sent, the data packet to be sent is sent to the opposite end data center corresponding to the data packet to be sent.

7. The inter-data center RDMA transmission method according to claim 6, characterized in that: The flow state table of the Internet gateway device maintains the flow state of the data flow between data centers, and the maintenance steps include: After receiving the to-be-sent data packet sent by the local data center and the inter-data center data packet sent by the opposite data center, judging whether the actual data packet number of the received data packet is consistent with the corresponding expected data packet number according to the flow state table, if not consistent, marking the flow state corresponding to the received data packet as a congested state; After receiving the to-be-sent data packet sent by the local data center and the inter-data center data packet sent by the opposite data center, it is determined whether the actual data packet number of the received data packet is consistent with the corresponding expected data packet number according to the flow state table. If they are consistent, the flow state corresponding to the received data packet is marked as a non-congested state.

8. The inter-data center RDMA transmission method according to claim 5, characterized in that: The retransmission state is maintained by the flow state table of the Internet gateway device, and the maintenance step includes: After the data packet retransmission process is started, the corresponding flow state is marked as the retransmission state until the retransmission process ends and the retransmission state in the corresponding flow state is cleared.

9. The inter-data center RDMA transmission method according to any one of claims 1 to 6, characterized in that: When the flow state corresponding to the received data packet is a congested state, the received data packet will be discarded to reduce the transmission of redundant data.

10. An Internet gateway device, characterized in that: The internet gateway device is applied to the inter-data center RDMA transmission method according to any one of claims 1 to 9, and the internet gateway device includes: A receiving module, used to receive a notification data packet from the local data center based on the data flow feedback between data centers; A determination module, configured to determine information of a desired retransmission data packet according to the notification data packet; The data packet retransmission module is used to retransmit the data packet according to the expected retransmission data packet information and the cached data packet.

Citation Information

Cited By

  • RDMA protocol transparent adaptation system supporting high RTT environment

    CN120915724A