Rdma communication load balancing method and system

Abstract

The application provides an RDMA communication load balancing method and system, the method comprising: receiving each sub-RDMA traffic corresponding to target RDMA traffic sent by a sending end respectively; wherein each sub-RDMA traffic is obtained by a preset segmentation module in an RNIC in the sending end segmenting the target RDMA traffic; determining a target transmission path of a head data packet in each sub-RDMA traffic based on a preset congestion path table locally; and sending each sub-RDMA traffic to a target switch based on the target transmission path. The application can effectively improve the universality of RDMA communication load balancing, effectively reduce the complexity of RDMA communication load balancing, and effectively improve communication efficiency.

Description

Technical Field

[0001] This application relates to the field of communications, and in particular to an RDMA communication load balancing method and system. Background Technology

[0002] The growing demands of data center applications are driving the development of high-speed networks, requiring high throughput, low latency, and low CPU overhead. For example, distributed machine learning training requires bandwidth of 100Gbps or higher; WebSearch has high latency requirements, demanding minimal query response time. Remote Direct Memory Access (RDMA) allows hosts to exchange data directly in memory via kernel bypass and offloads network I / O functions to dedicated network interface cards (RNICs) to reduce CPU overhead. RDMA provides high throughput (100 / 400Gbps) and ultra-low latency (~1µs) with low CPU overhead. Because of these advantages, modern data center networks widely use RoCEv2 (RDMA on Converged Ethernet Version 2) to achieve high performance in data centers. Existing RDMA is connection-based single-path transmission, meaning that once an RDMA connection is established, the flow is transmitted through only one fixed path. However, modern data center topologies are typically designed to be scalable, meaning there are multiple end-to-end paths between any two server racks. Existing RDMA, a single-path transmission method, is not only sensitive to path failures but also struggles to fully utilize the multiple paths available in modern data centers. For example, in distributed deep learning applications, RDMA transmission cannot fully utilize network link resources. Therefore, to maximize the utilization of network link resources, a load balancing strategy for RDMA is needed.

[0003] Existing load balancing methods for RDMA are either based on custom network interface cards (NICs), which are incompatible with existing NICs in data centers and are difficult to deploy on a large scale, i.e., they have poor versatility; or they utilize SDN controllers, which require pre-configuration of NIC virtualization on the end side. When the path changes, all NICs need to be reconfigured, which is a complicated process and leads to poor communication efficiency. Summary of the Invention

[0004] In view of this, embodiments of this application provide an RDMA communication load balancing method and system to eliminate or improve one or more defects existing in the prior art.

[0005] The first aspect of this application provides an RDMA communication load balancing method performed by a source switch, the method comprising:

[0006] Each sub-RDMA traffic corresponding to the target RDMA traffic sent by the transmitting end is received; wherein each sub-RDMA traffic is obtained in advance by the segmentation module preset in the RNIC in the transmitting end after segmenting the target RDMA traffic;

[0007] The target transmission path of the header data packet in each of the sub-RDMA traffic is determined based on a pre-set congestion path table in the local area.

[0008] Each of the sub-RDMA traffic is sent to the target switch based on the target transmission path.

[0009] In some embodiments of this application, the sub-RDMA traffic is obtained by the segmentation module dividing the target work queue element corresponding to the target RDMA traffic into multiple sub-work queue elements, and transmitting each sub-work queue element to the RNIC through its corresponding QP.

[0010] In some embodiments of this application, determining the target transmission path of the header data packets in each of the sub-RDMA traffic based on a pre-set local congestion path table includes:

[0011] The first hash value of the header data packet is calculated based on the hash algorithm, and the current transmission path is determined based on the first hash value. The current transmission path is determined to be congested based on the congestion path table. If it is congested, the second hash value of the first hash value is calculated based on the hash algorithm, and the current transmission path is updated based on the second hash value until the current transmission path is not congested. Then the current transmission path is used as the target transmission path.

[0012] In some embodiments of this application, it also includes:

[0013] Receive the first congestion data packet sent by the target switch;

[0014] Mark all transmission paths associated with the first congested data packet as congested, and store each marked transmission path in the congested path table.

[0015] In some embodiments of this application, it also includes:

[0016] Set the preset duration for each congested path in the congestion path table;

[0017] If, within the preset duration of the congestion path, a second congestion data packet is received again from the target switch and the second congestion data packet is associated with the congestion path, the preset duration corresponding to the congestion path is reset.

[0018] A second aspect of this application provides an RDMA communication load balancing method performed by a target switch, the method comprising:

[0019] The source switch receives each sub-RDMA traffic sent by the source switch based on a target transmission path; wherein the target transmission path is obtained by the source switch based on a pre-set congestion path table in its local location, which determines the header data packets in each sub-RDMA traffic; each sub-RDMA traffic is obtained by pre-segmenting the target RDMA traffic by a segmentation module preset in the RNIC in the sending end, and then sent by the sending end to the source switch.

[0020] In some embodiments of this application, if a data packet in the sub-RDMA traffic carries a path congestion marker, then a congestion data packet is sent to the source switch.

[0021] A third aspect of this application provides an RDMA communication load balancing system, which includes: a source switch and a target switch connected by a wire.

[0022] The source switch is used to execute the RDMA communication load balancing method described in the first aspect.

[0023] The target switch is used to execute the RDMA communication load balancing method described in the second aspect.

[0024] A fourth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the RDMA communication load balancing method described in the first aspect or the RDMA communication load balancing method described in the second aspect.

[0025] The fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the RDMA communication load balancing method described in the first aspect or the RDMA communication load balancing method described in the second aspect.

[0026] This application provides an RDMA communication load balancing method and system. The method includes: receiving each sub-RDMA traffic corresponding to a target RDMA traffic sent by a transmitter; wherein each sub-RDMA traffic is pre-segmented by a segmentation module preset in the RNIC of the transmitter to obtain the target RDMA traffic; determining the target transmission path of the header data packets in each sub-RDMA traffic based on a pre-set congestion path table in the local system; and sending each sub-RDMA traffic to a target switch based on the target transmission path. This application can effectively improve the universality of RDMA communication load balancing and effectively reduce the complexity of RDMA communication load balancing, thereby effectively improving communication efficiency.

[0027] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0028] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings:

[0030] Figure 1 This is a flowchart illustrating a first RDMA communication load balancing method in one embodiment of this application.

[0031] Figure 2 This is a schematic diagram of the structure of an RDMA communication load balancing system in another embodiment of this application.

[0032] Figure 3 This is a schematic diagram of the target RDMA traffic segmentation process in another embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0034] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0035] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0036] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0037] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0038] The following examples will provide a detailed description.

[0039] This application provides a first RDMA communication load balancing method that can be executed by the source switch. See also Figure 1 The RDMA communication load balancing method specifically includes the following:

[0040] Step 110: Receive each sub-RDMA traffic corresponding to the target RDMA traffic sent by the transmitting end; wherein each sub-RDMA traffic is obtained in advance by the segmentation module preset in the RNIC in the transmitting end after segmenting the target RDMA traffic.

[0041] Step 120: Determine the target transmission path of the header data packets in each of the sub-RDMA traffic based on the pre-set congestion path table in the local system.

[0042] It's important to note that the source switch only hashes the first data packet in the sub-stream; subsequent data packets are transmitted along the same path. This is because RDMA employs a go-back-N mechanism. If the transmission path changes within the sub-stream and a subsequent data packet arrives first, retransmission will be initiated, significantly impacting communication efficiency.

[0043] Step 130: Send each of the sub-RDMA traffic to the target switch based on the target transmission path.

[0044] Specifically, the source switch receives each sub-RDMA traffic corresponding to the target RDMA (Remote Direct Memory Access) traffic sent by the sender; then, based on a pre-set congestion path table, it determines the target transmission path for the header data packets in each sub-RDMA traffic. Finally, based on the target transmission path, it sends each sub-RDMA traffic to the target switch, thereby effectively improving the universality of RDMA communication load balancing and reducing its complexity, thus effectively improving communication efficiency.

[0045] Each sub-RDMA traffic is pre-segmented by the Cervine Shaper in the RNIC (RNMANetwork Interface Card) of the transmitting end, which segments the target RDMA traffic. For details, see... Figure 3 The sub-RDMA traffic is segmented by the segmentation module to obtain the target work queue element corresponding to the target RDMA traffic (i.e., Figure 3 The WQE (Work Queue Element) in the code is divided into multiple sub-work queue elements, and each sub-work queue element is processed through its corresponding QP (such as...). Figure 3 The QP1, QP2 and QP3 in the RNIC are transmitted to obtain the data.

[0046] in addition, Figure 3 In RDMA, QP (Queue Pair) is a fundamental concept, representing a combination of a transmit work queue and a receive work queue. Bitmap is a data structure called a bitmap. CQE stands for Completion Queue Element. The Poll_cq function returns the CQE to the application. Userspace Driver represents the user-space driver. The Post_send function sends the WQE task to the RNIC.

[0047] It's important to note that a WQE is considered a "task description," containing detailed information about the task (e.g., sending 10 bits of data at address 0x12345678 to the target node). Therefore, all packets associated with a WQE are transmitted as a single RDMA stream. Our idea is to segment the stream by dividing the WQE. We divide a WQE into N sub-WQEs, thus dividing the stream into N sub-streams. However, since RDMA's order-preserving transmission is achieved through PSN bits, the same PSN is used within the same QP connection. This means that even if a WQE is divided into N sub-WQEs and then transmitted through the same QP, out-of-order delivery can still occur when the sub-streams pass through the source ToR switch and hash to different paths. This out-of-order delivery is due to the order-preserving nature of transmission within the same QP. Therefore, we address this out-of-order transmission problem by transmitting sub-WQEs through different QPs.

[0048] To further reduce the complexity of RDMA communication load balancing, step 120 includes:

[0049] The first hash value of the header data packet is calculated based on the hash algorithm, and the current transmission path is determined based on the first hash value. The current transmission path is determined to be congested based on the congestion path table. If it is congested, the second hash value of the first hash value is calculated based on the hash algorithm, and the current transmission path is updated based on the second hash value until the current transmission path is not congested. Then the current transmission path is used as the target transmission path.

[0050] Specifically, the source switch calculates the corresponding hash value based on the five-tuple of the header data packet and determines the current transmission path based on the first hash value. Then, it determines whether the current transmission path is congested based on the congestion path table. If so, it updates the hash value by performing a second hash and updates the current transmission path based on the updated hash value until the current transmission path is no longer congested. Then, the current transmission path is used as the target transmission path, thereby further reducing the complexity of RDMA communication load balancing.

[0051] To further reduce the complexity of RDMA communication load balancing, the aforementioned first RDMA communication load balancing method also includes:

[0052] Step 140: Receive the first congestion data packet sent by the target switch.

[0053] Step 150: Mark all transmission paths associated with the first congested data packet as congested, and store each marked transmission path in the congested path table.

[0054] Specifically, the source switch receives the first congestion data packet sent by the target switch; then it marks multiple transmission paths associated with the first congestion data packet as congested and stores each marked transmission path in the congestion path table, thereby further reducing the complexity of RDMA communication load balancing.

[0055] The marked transmission path will only transmit RDMA substreams that have not yet been completed.

[0056] To further improve the accuracy of path congestion control, after step 150, the following steps are also included:

[0057] Set the preset duration for each congested path in the congestion path table;

[0058] If, within the preset duration of the congestion path, a second congestion data packet is received again from the target switch and the second congestion data packet is associated with the congestion path, the preset duration corresponding to the congestion path is reset.

[0059] Specifically, the source switch sets a preset duration for each congestion path in the congestion path table; within the preset duration of the congestion path, if a second congestion data packet sent by the target switch is received again and the second congestion data packet is associated with the congestion path, the preset duration corresponding to the congestion path is reset, thereby further improving the accuracy of path congestion control.

[0060] This application also provides a second RDMA communication load balancing method that can be executed by the target switch. See [link to relevant documentation] Figure 1 The RDMA communication load balancing method specifically includes the following:

[0061] The source switch receives each sub-RDMA traffic sent by the source switch based on a target transmission path; wherein the target transmission path is obtained by the source switch based on a pre-set congestion path table in its local location, which determines the header data packets in each sub-RDMA traffic; each sub-RDMA traffic is obtained by pre-segmenting the target RDMA traffic by a segmentation module preset in the RNIC in the sending end, and then sent by the sending end to the source switch.

[0062] Specifically, the target switch receives each sub-RDMA traffic sent by the source switch based on the target transmission path. The target transmission path is obtained by the source switch based on a pre-set congestion path table in its local system, which determines the header data packets in each sub-RDMA traffic. Each sub-RDMA traffic is pre-segmented by a segmentation module in the RNIC at the sending end, and then sent by the sending end to the source switch. This effectively improves the universality of RDMA communication load balancing and reduces its complexity, thereby improving communication efficiency.

[0063] To further improve the accuracy of path congestion control, the aforementioned second RDMA communication load balancing method also includes:

[0064] If the data packets in the sub-RDMA traffic carry path congestion markers, then congestion data packets are sent to the source switch.

[0065] Specifically, if the data packets in the sub-RDMA traffic carry path congestion markers (ECN markers, Explicit Congestion Notification), that is, the data packets are marked as congested by the Spine switch during transmission, the target switch sends congestion data packets to the source switch, thereby further improving the accuracy of path congestion control.

[0066] From a software perspective, this application also provides an RDMA communication load balancing system for performing all or part of the described RDMA communication load balancing method, see [link to relevant documentation]. Figure 2 The RDMA communication load balancing system specifically includes the following components:

[0067] The source switch 10 and the target switch 20 are connected by a wire.

[0068] The source switch 10 is used to execute the aforementioned first RDMA communication load balancing method.

[0069] The target switch 20 is used to execute the aforementioned second RDMA communication load balancing method.

[0070] This application provides an RDMA communication load balancing method and system. The method includes: receiving each sub-RDMA traffic corresponding to a target RDMA traffic sent by a transmitter; wherein each sub-RDMA traffic is pre-segmented by a segmentation module preset in the RNIC of the transmitter to obtain the target RDMA traffic; determining the target transmission path of the header data packets in each sub-RDMA traffic based on a pre-set congestion path table in the local system; and sending each sub-RDMA traffic to a target switch based on the target transmission path. This application can effectively improve the universality of RDMA communication load balancing and effectively reduce the complexity of RDMA communication load balancing, thereby effectively improving communication efficiency.

[0071] This application also provides an electronic device, such as a central server, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the first RDMA communication load balancing method or the second RDMA communication load balancing method mentioned in the above embodiments. The processor and the memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and the memory via wired or wireless means.

[0072] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0073] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the first RDMA communication load balancing method or the second RDMA communication load balancing method in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the first RDMA communication load balancing method or the second RDMA communication load balancing method in the above method embodiments.

[0074] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0075] The one or more modules are stored in the memory, and when executed by the processor, they execute the first RDMA communication load balancing method or the second RDMA communication load balancing method in the embodiment.

[0076] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0077] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0078] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0079] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned first RDMA communication load balancing method or second RDMA communication load balancing method. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0080] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0081] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0082] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0083] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for RDMA communication load balancing, the method comprising: The application comprises: receiving each sub-RDMA traffic corresponding to the target RDMA traffic sent by the sending end; wherein each sub-RDMA traffic is obtained by pre-segmenting the target RDMA traffic by a segmentation module preset in the RNIC in the sending end; each sub-RDMA traffic contains a header packet and subsequent data packets; determining the target transmission path of the header packet in each sub-RDMA traffic based on a congestion path table preset locally; sending the header packet and subsequent data packets corresponding to each sub-RDMA traffic to a target switch based on the target transmission path; The method for determining the target transmission path of the header packet in each sub-RDMA traffic based on the congestion path table preset locally comprises: calculating a first hash value of the header packet based on a hash algorithm, and determining the current transmission path based on the first hash value; determining whether the current transmission path is congested based on the congestion path table, if yes, calculating a second hash value of the first hash value based on the hash algorithm, and updating the current transmission path based on the second hash value until the current transmission path is not congested, then taking the current transmission path as the target transmission path.

2. The RDMA communication load balancing method of claim 1, wherein, The sub-RDMA traffic is obtained by cutting the target work queue element corresponding to the target RDMA traffic into multiple sub-work queue elements by the segmentation module, and transmitting each sub-work queue element to the RNIC through a corresponding QP.

3. The RDMA communication load balancing method of claim 1, wherein, The method further comprises: receiving a first congestion data packet sent by a target switch; marking multiple transmission paths associated with the first congestion data packet as congested, and storing each marked transmission path in the congestion path table.

4. The RDMA communication load balancing method of claim 3, wherein, The method further comprises: setting the preset duration of each congestion path in the congestion path table; if a second congestion data packet sent by the target switch is received again within the preset duration of the congestion path and the second congestion data packet is associated with the congestion path, resetting the preset duration corresponding to the congestion path.

5. A method of RDMA communication load balancing, the method comprising: The method comprises: receiving the header packet and subsequent data packets corresponding to each sub-RDMA traffic sent by a source switch based on a target transmission path; wherein the target transmission path is determined by the source switch based on a congestion path table preset locally for the header packet in each sub-RDMA traffic; each sub-RDMA traffic contains a header packet and subsequent data packets; each sub-RDMA traffic is obtained by pre-segmenting the target RDMA traffic by a segmentation module preset in the RNIC in the sending end, and sent to the source switch by the sending end; The method for determining the target transmission path of the header packet in each sub-RDMA traffic based on the congestion path table preset locally comprises: calculating a first hash value of the head packet based on a hash algorithm, and determining a current transmission path based on the first hash value; judging whether the current transmission path is congested based on the congestion path table, if yes, calculating a second hash value of the first hash value based on the hash algorithm, and updating the current transmission path based on the second hash value until the current transmission path is not congested, then taking the current transmission path as the target transmission path.

6. The RDMA communication load balancing method of claim 5, wherein, If the data packet in the sub-RDMA traffic carries a path congestion mark, a congestion data packet is sent to the source switch.

7. An RDMA communication load balancing system, comprising: The RDMA communication load balancing method comprises the following steps: a source switch and a target switch connected by a wire; the source switch is configured to perform the RDMA communication load balancing method in any one of claims 1-4; the target switch is configured to perform the RDMA communication load balancing method in claim 5 or 6.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the RDMA communication load balancing method in any one of claims 1-4, or the RDMA communication load balancing method in any one of claims 5 or 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the RDMA communication load balancing method in any one of claims 1-4, or the RDMA communication load balancing method in any one of claims 5 or 6.

Images