A message transmission system and method

Abstract

The application provides a message transmission system and method. The system comprises a source device, a switch and a destination device. The source device is configured to add path selection information to a message to be sent, and send the message with the added path selection information to the destination device through the switch. The switch is configured to select a forwarding path according to the analyzed path selection information after receiving the message, and send the message through the selected forwarding path. The destination device is configured to receive the message sent by the source device. The technical solution of the application can control the source device to uniformly hash multiple messages sent to the destination device on different transmission paths, thereby effectively improving the total path bandwidth utilization.

Description

Technical Field

[0001] This application belongs to the field of message transmission technology, and specifically relates to a message transmission system and method. Background Technology

[0002] In distributed training scenarios within AI computing clusters, hosts establish long-lived QP (Queue Pair) connections based on the RDMA protocol as communication links. Since there are numerous interconnection paths between nodes via several hops of switches depending on the transmission distance, the switch group employs a hash algorithm based on a five-tuple (source IP, destination IP, source port, destination port, protocol type) to perform equivalent multi-path routing. This ensures that packets with the same QP identifier are always forwarded along a fixed physical path during transmission. This mechanism effectively avoids out-of-order packet issues by maintaining flow state consistency, guaranteeing reliable transmission of the RoCEv2 protocol stack. Summary of the Invention

[0003] The purpose of this application is to provide a message transmission system and method, which aims to solve the technical problem in related technologies where different messages on the same communication link are always forwarded along a fixed physical path, which can easily cause traffic congestion in some scenarios.

[0004] According to a first aspect of this application, a message transmission system is provided, comprising: a source device, a switch, and a destination device;

[0005] The source device is used to add path selection information to the message to be sent, and send the message with the added path selection information to the destination device through the switch;

[0006] The switch is configured to select a forwarding path based on the parsed path selection information after receiving the packet, and forward the packet out through the selected forwarding path.

[0007] The destination device is used to receive the message sent by the source device.

[0008] In some optional embodiments, the path selection information includes a forwarding hop value and multiple path identifiers; the multiple path identifiers correspond to different forwarding hop values.

[0009] After parsing the path selection information, the switch updates the forwarding hop value and selects a forwarding path based on the path identifier corresponding to the updated forwarding hop value.

[0010] In some optional embodiments, a QP communication connection is pre-established between the source device and the destination device. For multiple packets with the same QP identifier, the source device forwards the multiple packets to the destination device via multiple different physical paths by adding the corresponding path selection information.

[0011] In some optional embodiments, the message is an RDMA message, and the source device adds the path selection information to the source port field of the RDMA message.

[0012] In some alternative embodiments, the switch includes other switches besides the access layer switch that is directly connected to the destination device.

[0013] According to a second aspect of this application, a message transmission method is provided, comprising:

[0014] The source device adds path selection information to the message to be sent, and then sends the message with the added path selection information to the destination device through the switch;

[0015] After receiving the packet, the switch selects a forwarding path based on the parsed path selection information;

[0016] The switch forwards the packet through the selected forwarding path;

[0017] The destination device receives the message sent by the source device.

[0018] In this embodiment, under the transmission mechanism of the aforementioned message transmission system, the source device can specify the forwarding path of the message by adding path selection information to the message, thereby allowing the switch to forward the message to the destination device along the specified path. This method allows the source device to control the even distribution of multiple messages destined for the destination device across different transmission paths, effectively improving the overall path bandwidth utilization.

[0019] In some optional embodiments, the path selection information includes a forwarding hop value and multiple path identifiers; the multiple path identifiers correspond to different forwarding hop values; the step of selecting a forwarding path based on the parsed path selection information includes:

[0020] After the switch parses the path selection information, it updates the forwarding hop value.

[0021] The switch selects a forwarding path based on the path identifier corresponding to the updated forwarding hop value.

[0022] This approach allows each switch to select the next-hop switch to which a packet should be forwarded based on the corresponding path identifier in the path selection information. This enables different packets to be transmitted through different physical paths, effectively improving the overall path bandwidth utilization.

[0023] In some optional embodiments, a QP communication connection is pre-established between the source device and the destination device. For multiple packets with the same QP identifier, the source device forwards the multiple packets to the destination device via multiple different physical paths by adding the corresponding path selection information.

[0024] In this way, since the path selection information can be flexibly changed by the source device, the source device can control the physical path through which the packet is forwarded, making the forwarding path flexible and changeable. Ultimately, this achieves the goal of sending packets with the same QP identifier through multiple paths, avoiding the phenomenon that all packets with the same QP identifier will be canceled due to congestion on a certain physical path in some scenarios.

[0025] In some optional embodiments, the message is an RDMA message, and the source device adds the path selection information to the source port field of the RDMA message.

[0026] This approach allows packets with the same QP identifier based on the RDMA protocol to be transmitted via multiple paths during transmission. Therefore, if a part of the physical path fails, packets on other physical paths will not be abnormally destroyed.

[0027] In some alternative embodiments, the switch includes other switches besides the access layer switch that is directly connected to the destination device.

[0028] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures and processes shown in the description and the accompanying drawings. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a structural block diagram of a message transmission system according to an exemplary embodiment of this application.

[0031] Figure 2 This is a schematic diagram of a network topology according to an exemplary embodiment of this application.

[0032] Figure 3 This is a schematic diagram of a message format according to an exemplary embodiment of this application.

[0033] Figure 4 This is a flowchart illustrating a message transmission method according to an exemplary embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] The routing methods used in the related technologies in the background section have the following limitations:

[0036] 1. Hash Polarization: During large model training, periodic elephant flows (such as gradient synchronization traffic) with fixed 5-tuples (source IP, destination IP, source port, destination port, protocol type) can trigger ECMP (Equal Cost Multipath) hash collisions, leading to persistent congestion hotspots on specific links of multi-hop switches. Real-world data shows that this polarization phenomenon can reduce network effective bandwidth utilization by 40%-60%.

[0037] 2. Lack of dynamic path optimization: This routing mechanism lacks the ability to perceive the real-time status of links. Even if there are low-load alternative paths, it still mechanically executes the initial hash decision. This static routing strategy is difficult to adapt to dynamically changing network loads and is prone to causing a surge in tail latency in Incast multi-to-one scenarios.

[0038] Based on the above analysis, see Figure 1 As shown, this application exemplarily proposes a message transmission system, including: a source device, a switch, and a destination device;

[0039] The source device is used to add path selection information to the message to be sent, and then send the message with the added path selection information to the destination device through the switch.

[0040] A switch is used to select a forwarding path based on the parsed path selection information after receiving a packet, and then forward the packet out through the selected forwarding path.

[0041] The destination device is used to receive messages sent by the source device.

[0042] For example, the message transmission system can transmit messages based on a CLOS topology, which includes a server layer, access layer, aggregation layer, and core layer. The source and destination devices can reside in the server layer, such as network interface cards (NICs). Multiple switches can be located in the access, aggregation, and core layers respectively. The source and destination devices can communicate based on the RDMA (Remote Direct Memory Access) protocol; for instance, the source device, switch, and destination device can communicate under the RoCE (RDMA over Converged Ethernet) protocol framework.

[0043] For example, the source device and the destination device can establish a communication connection in advance. When the source device sends a message to the destination device, it adds path selection information to the message and transmits the message with the added path selection information to the access layer switch directly connected to the source device. After parsing the path selection information in the message, the access layer switch selects a forwarding path based on the path selection information. For example, after determining the next-hop switch based on the path selection information, it forwards the message to the determined next-hop switch. This next-hop switch can be an aggregation layer switch. The aggregation layer switch performs the same processing mechanism, and after being forwarded through multiple switches, the message is forwarded to the access layer switch directly connected to the destination device. The access layer switch then directly transmits the message to the destination device, where it is processed accordingly. Therefore, the forwarding path selected by the switch can be understood as the next-hop switch to which the message is to be forwarded. On the message forwarding path, all switches except the access layer switch directly connected to the destination device perform the above path selection logic.

[0044] In this embodiment, under the transmission mechanism of the aforementioned message transmission system, the source device can specify the forwarding path of the message by adding path selection information to the message, thereby allowing the switch to forward the message to the destination device along the specified path. This method allows the source device to control the even distribution of multiple messages destined for the destination device across different transmission paths, effectively improving the overall path bandwidth utilization.

[0045] In some optional implementations, the path selection information includes a forwarding hop value and multiple path identifiers; each path identifier corresponds to a different forwarding hop value.

[0046] After the switch parses the path selection information, it updates the forwarding hop value and selects a forwarding path based on the path identifier corresponding to the updated forwarding hop value.

[0047] For example, the path selection information added to a packet by the source device may include a forwarding hop value and a path identifier. The forwarding hop value is set to an initial value by the source device, such as 0. Each time the packet is forwarded to a switch, the switch updates the forwarding hop value, for example, by incrementing it by 1. Multiple path identifiers can be set. After the forwarding hop value is updated, the updated forwarding hop value can correspond to one of the path identifiers, so that the current switch selects the path identifier corresponding to the updated forwarding hop value to select the forwarding path.

[0048] The following example illustrates this.

[0049] like Figure 2 As shown, the network topology of the message transmission system includes a server layer, access layer, aggregation layer, and core layer. A communication connection is established between host H0 in logical domain Pod0 and host H0 in logical domain Pod1. When host H0 in Pod0, as the source device, sends a message to host H0 in Pod1, as the destination device, the path selection information added to the message is represented as: {hop, P1, P2, P3, P4}, where hop is the forwarding hop value, initially 0, and P1-P4 are the path identifiers corresponding to hop values ​​of 1-4. When the switch receives the message, it first increments the hop value by 1, that is, the hop count by 1, and then selects the corresponding path identifier among P1-P4 based on the current hop value. For example, if hop is 1, P1 is selected; if hop is 2, P2 is selected, and so on. Then, the forwarding path is calculated based on the selected path identifier. For example, if switch L0, which is directly connected to host H0 in Pod0, receives a packet sent by host H0 in Pod0, it can increment the hop value in the packet by 1, thus updating its value to 1. Then, based on this value of 1, it selects the value of path identifier P1, and then selects a forwarding path based on the value of P1. If the forwarding path selected based on the value of P1 is switch S2 in the aggregation layer, then switch L0 forwards the packet to switch S2. After receiving the packet, switch S2 performs the same operation, thus forwarding the packet to a switch in the core layer, such as switch C0. Similarly, switch C0 then forwards the packet to the aggregation layer switch in the Pod1 domain, until the packet is forwarded to host H0 in the Pod1 domain.

[0050] This approach allows each switch to select the next-hop switch to forward a packet based on the corresponding path identifier in the path selection information. This enables control over the transmission of different packets using different physical paths, effectively improving overall path bandwidth utilization. Since the path selection information can be dynamically changed at the endpoint, when a path fails, the source device can modify the path identifier of subsequent packets to guide the switch to select an alternative path, preventing QP connection interruption due to a single path failure.

[0051] In some optional implementations, a QP communication connection is pre-established between the source device and the destination device. For multiple packets with the same QP identifier, the source device adds corresponding path selection information to make the multiple packets forward to the destination device via multiple different physical paths.

[0052] For example, the network topology of the message transmission system is a CLOS architecture, which follows the RoCE protocol framework. A long-lived QP connection is pre-established between the source and destination devices, and message transmission is performed using this QP connection as the communication link. Messages sent from the source device to the destination device on the same QP link all carry the same QP identifier in the message header. The switch typically performs equal-cost multi-path routing for received messages using a hash algorithm based on a five-tuple (source IP, destination IP, source port, destination port, protocol type), ensuring that messages with the same QP identifier always follow a fixed physical path during transmission. In this embodiment, to avoid packets with the same QP identifier being transmitted on a fixed physical path, the source device adds path selection information to the packets. When the switch performs equal-cost multi-path routing, it also uses the path identifier corresponding to the forwarding hop value in the path selection information as a factor in the hash calculation. For multiple packets with the same QP identifier, the source device can add corresponding path selection information, so that the forwarding paths obtained by the switch from hashing different packets with the same QP identifier are no longer fixed physical paths, but different physical paths. For example, multiple packets with the same QP identifier can be evenly distributed on different physical paths in the network topology. This application adds path selection information to the UDP source port domain segment, so that when the switch uses the equal-cost multi-path routing algorithm to perform a hash calculation on the five-tuple including the UDP source port to obtain the final forwarding path, the path selection information added to the UDP source port domain segment causes multiple packets with the same QP identifier to be forwarded on different physical paths.

[0053] In this way, since the path selection information can be flexibly changed by the source device, the source device can control the physical path through which the packet is forwarded, making the forwarding path flexible and changeable. Ultimately, this achieves the goal of sending packets with the same QP identifier through multiple paths, avoiding the phenomenon that all packets with the same QP identifier will be canceled due to congestion on a certain physical path in some scenarios.

[0054] In some alternative implementations, the message is an RDMA message, and the source device adds path selection information to the source port field of the RDMA message.

[0055] Still with Figure 3For example, under the RoCE protocol framework, the UDP source port number in the RDMA message format allows users to customize this field according to their needs. It does not involve protocol validity checks, therefore the source device and the switch can cooperate to use this field, allowing the source device to control the forwarding path selected by the switch. The RDMA message format is as follows: Figure 3 As shown, the UDP header includes the source port number and the destination port number. In this embodiment, the source device adds path selection information to the domain segment where the source port number is located.

[0056] This approach allows packets with the same QP identifier based on the RDMA protocol to be transmitted via multiple paths during transmission. Therefore, if a part of the physical path fails, packets on other physical paths will not be abnormally destroyed.

[0057] In some alternative implementations, the switch includes switches other than the access layer switch that is directly connected to the destination device.

[0058] For example, since the destination device is directly connected to the access layer switch, after the access layer switch corresponding to the destination device receives a packet destined for the destination device, it can forward the packet to the destination device on the directly connected path without needing to perform path selection. Therefore, the access layer switch corresponding to the destination device does not need to parse path selection information, nor does it need to perform path selection based on path selection information.

[0059] See Figure 4 The flowchart illustrates that this application also provides an exemplary message transmission method, including:

[0060] Step S401: The source device adds path selection information to the message to be sent, and sends the message with the added path selection information to the destination device through the switch;

[0061] Step S402: After receiving the packet, the switch selects a forwarding path based on the parsed path selection information;

[0062] Step S403: The switch forwards the packet through the selected forwarding path;

[0063] Step S404: The destination device receives the message sent by the source device.

[0064] In some optional implementations, the path selection information includes a forwarding hop value and multiple path identifiers; each path identifier corresponds to a different forwarding hop value; the forwarding path is selected based on the parsed path selection information, including:

[0065] After the switch parses the path selection information, it updates the forwarding hop value;

[0066] The switch selects a forwarding path based on the path identifier corresponding to the updated forwarding hop value.

[0067] In some optional implementations, a QP communication connection is pre-established between the source device and the destination device. For multiple packets with the same QP identifier, the source device adds corresponding path selection information to make the multiple packets forward to the destination device via multiple different physical paths.

[0068] In some alternative implementations, the message is an RDMA message, and the source device adds path selection information to the source port field of the RDMA message.

[0069] In some alternative implementations, the switch includes switches other than the access layer switch that is directly connected to the destination device.

[0070] The above method can be implemented using the message transmission system provided in the above embodiments. For specific implementation details, please refer to the description of the message transmission system in the above embodiments, which will not be repeated here.

[0071] It is understood that the circuit structures, names, and parameters described in the above embodiments are merely examples. Those skilled in the art can also make readily conceived combinations and adjustments to the structural features of the above embodiments according to their needs, and the concept of this application should not be limited to the specific details of the above examples.

[0072] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A message transmission system, characterized in that, include: Source device, switch, and destination device; The source device is used to add path selection information to the message to be sent, and send the message with the added path selection information to the destination device through the switch; A QP communication connection is pre-established between the source device and the destination device. For multiple packets with the same QP identifier, the source device adds corresponding path selection information so that the multiple packets are forwarded to the destination device via multiple different physical paths. The switch is configured to, upon receiving the packet, select a forwarding path based on the parsed path selection information, and forward the packet through the selected path. The source device specifies the forwarding path by adding path selection information to the packet, thereby allowing the switch to forward the packet to the destination device along the specified path. The path selection information includes a forwarding hop value and multiple path identifiers. Each path identifier corresponds to a different forwarding hop value. After parsing the path selection information, the switch updates the forwarding hop value and selects a forwarding path based on the path identifier corresponding to the updated forwarding hop value. The destination device is used to receive the message sent by the source device.

2. The message transmission system according to claim 1, characterized in that, The message is an RDMA message, and the source device adds the path selection information to the source port field of the RDMA message.

3. The message transmission system according to claim 1, characterized in that, The switches include all switches other than the access layer switches that are directly connected to the destination device.

4. A message transmission method, characterized in that, include: The source device adds path selection information to the message to be sent, and then sends the message with the added path selection information to the destination device through the switch; A QP communication connection is pre-established between the source device and the destination device. For multiple packets with the same QP identifier, the source device adds corresponding path selection information so that the multiple packets are forwarded to the destination device via multiple different physical paths. After receiving the packet, the switch selects a forwarding path based on the parsed path selection information; The switch forwards the packet through the selected forwarding path; wherein, the source device specifies the forwarding path of the packet by adding path selection information to the packet, so that the switch forwards the packet to the destination device along the specified forwarding path; the path selection information includes a forwarding hop value and multiple path identifiers; the multiple path identifiers correspond to different forwarding hop values; after parsing the path selection information, the switch updates the forwarding hop value and selects a forwarding path according to the path identifier corresponding to the updated forwarding hop value; The destination device receives the message sent by the source device.

5. The message transmission method according to claim 4, characterized in that, The message is an RDMA message, and the source device adds the path selection information to the source port field of the RDMA message.

6. The message transmission method according to claim 4, characterized in that, The switches include all switches other than the access layer switches that are directly connected to the destination device.

Images