A method, apparatus, device and medium for congestion recovery based on fast reroute

Abstract

The application discloses a congestion recovery method and device based on fast rerouting, equipment and medium. The method comprises the following steps: acquiring multiple transmission paths between a first end device and a second end device through at least one layer of network equipment; determining a path of to-be-transmitted traffic in the multiple transmission paths by using a five-tuple monitoring technology, and determining state information corresponding to the path of the to-be-transmitted traffic; detecting whether congestion occurs on the path of the to-be-transmitted traffic according to the state information; if the congestion occurs, finding a target path in at least one candidate path by using a color block matching algorithm, and updating the path where the congestion occurs to the target path. The method can detect the path in advance when the congestion occurs, quickly find an optimal path for rerouting operation, and thus reduce the influence of the congestion on application performance.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a congestion recovery method, apparatus, device, and medium based on fast rerouting. Background Technology

[0002] The rapid development of high-performance storage and computing applications has placed higher demands on data center network performance. Remote Direct Memory Access (RDMA), due to its kernel bypass feature, allows direct data copying between memory locations, significantly reducing CPU overhead, and is therefore increasingly used in data centers. Currently, RDMA networks are widely deployed in data centers both domestically and internationally, and some internet companies have also begun using RDMA to deploy accelerated storage, machine learning, and heterogeneous computing services.

[0003] The popular architecture in data centers is the Fat-Tree and Spine-Leaf architecture based on CLOS networks, which leverages the performance of RDMA network cards through a combination of multi-layer switches. This architecture primarily achieves load balancing using a hop-by-hop, flow-based equal-cost multi-path (ECMP) routing algorithm based on five-tuple hashes. However, during RDMA network communication, the source port number is randomly hashed from the source-destination queue pair (qp) numbers, further leading to randomness in routing. When concurrent RDMA communication occurs on a multi-layer switch topology, uneven load distribution can easily cause congestion, degrading application performance.

[0004] Specifically, when a single switch becomes congested, multiple paths connected to that switch will be affected, thus impacting the communication performance of multiple RDMAs. Furthermore, after congestion occurs, route reselection is typically performed using a brute-force search method to determine a new path; however, this route reselection method is cumbersome, inefficient, and negatively impacts application performance. Summary of the Invention

[0005] The purpose of this invention is to solve the problem of accurately locating the congestion point and quickly resolving congestion when network node congestion occurs. To solve this technical problem, embodiments of this application provide a congestion recovery method, apparatus, and device based on fast rerouting, specifically disclosing the following technical solutions:

[0006] In a first aspect, embodiments of the present invention disclose a congestion recovery method based on fast rerouting, the method comprising:

[0007] Obtain multiple transmission paths between the first-end device and the second-end device, which pass through at least one layer of network devices;

[0008] The five-tuple monitoring technology is used to determine the path of the traffic to be transmitted among the multiple transmission paths, and to determine the status information corresponding to the path of the traffic to be transmitted.

[0009] Based on the status information, detect whether congestion has occurred on the path of the traffic to be transmitted;

[0010] If congestion occurs, the target path is found in at least one candidate path using a color block matching algorithm, and the congested path is updated to the target path.

[0011] Secondly, embodiments of the present invention also disclose a congestion recovery device based on fast rerouting, the device comprising:

[0012] The acquisition unit is used to acquire multiple transmission paths between the first end device and the second end device that pass through at least one layer of network devices.

[0013] The determining unit is used to determine the path of the traffic to be transmitted among the multiple transmission paths using the five-tuple monitoring technology, and to determine the status information corresponding to the path of the traffic to be transmitted.

[0014] The detection unit is used to detect whether congestion has occurred on the path of the traffic to be transmitted based on the status information.

[0015] The search unit is used to find the target path from at least one candidate path by using a color block matching algorithm when the detection unit detects that congestion has occurred.

[0016] An update unit is used to update the congested path to the target path.

[0017] Thirdly, embodiments of the present invention also disclose an electronic device, including a processor and a memory, wherein the memory is coupled to the processor; wherein the memory stores computer-readable program instructions, and when the instructions are executed by the processor, the congestion recovery method based on fast rerouting as described in the first aspect above is implemented.

[0018] In addition, embodiments of the present invention also disclose a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the congestion recovery method based on fast rerouting described in the first aspect above.

[0019] The congestion recovery method, apparatus, and device based on fast rerouting provided by this invention can detect paths in advance when network congestion occurs and quickly find the optimal path (target path) for rerouting, thereby reducing the impact of congestion on application performance. Specifically, in terms of congestion perception, five-tuple monitoring is used to quickly perceive the original application path and network conditions, thereby determining which paths in the transmission path are congested. In terms of congestion recovery, a color-block matching algorithm is used to find the target path in linear time complexity, and then fast rerouting is performed by modifying the source port number, thereby quickly resolving the congestion problem.

[0020] Furthermore, this method improves overall performance by over 10% in terms of bandwidth through the use of congestion awareness and recovery mechanisms, enabling faster training tasks. Simultaneously, compared to previous random hashing routing, proactive routing using rerouting exhibits lower variance, more stable overall bandwidth, and less fluctuation in task training. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an RDMA communication scenario provided in an embodiment of the present invention;

[0023] Figure 2 This is a flowchart of a congestion recovery method based on fast rerouting provided in an embodiment of the present invention;

[0024] Figure 3 A flowchart of another congestion recovery method based on fast rerouting provided in an embodiment of the present invention;

[0025] Figure 4 This is a flowchart of an update rerouting method provided in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of a color block matching algorithm provided in an embodiment of the present invention;

[0027] Figure 6 This is an experimental topology diagram containing three topological structures provided in this embodiment of the invention;

[0028] Figure 7 This is a structural block diagram of a congestion recovery device provided in an embodiment of the present invention;

[0029] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Currently, several solutions exist to address data center network congestion. Traditional algorithms such as Data Center Quantized Congestion Notification (DCQCN) and Transport Informed by Measurement of Latency (TIMELY) utilize Explicit Congestion Notification (ECN) and round-trip time (RTT) to assess network quality, respectively, and combine them with sliding window algorithms for flow control, effectively handling common congestion issues. However, these traditional algorithms cannot differentiate between the congestion states of switches at different layers, relying solely on ECN and RTT to identify congestion along an entire path. When a single switch at a certain layer becomes congested, multiple paths connected to that switch are simultaneously affected, resulting in multi-path congestion.

[0032] Furthermore, when considering load issues on switches, while High Precision Congestion Control (HPCC) can utilize in-band network telemetry (INT) to obtain switch load information, its adjustments are only made when congestion actually occurs. It also lacks efficient path reselection mechanisms and suffers from inherent latency discrepancies, thus impacting application performance. Additionally, this method requires significant modifications to existing switches, making incremental deployment on existing network systems impractical.

[0033] This invention provides a congestion recovery method based on fast rerouting. This method can be applied to a data center system, such as the CLOS network system, which supports RDMA. Figure 1The diagram shows a schematic of an RDMA communication scenario, which includes two end devices, such as a first end device and a second end device. Between the first end device and the second end device, there is a three-layer network architecture, namely the 3.0 network architecture, which is a CLOS architecture composed of three-layer switches. The host (first end device) is directly connected to the first-layer switch below, and then communicates with the second end device through the second-layer and third-layer switches.

[0034] The Layer 1 and Layer 3 switches are symmetrical to each other. If there are 3 layers in total, the first, second and third layers can be identified as S0, S1 and S0 respectively; if there are 5 layers in total, the first to fifth layers can be identified as S0, S1, S2, S1 and S0 respectively.

[0035] It should be understood that the two end devices mentioned above may include more or fewer layers of switches, such as one layer, two layers, four layers, etc., but this embodiment limits this. Of course, the system may also include more end devices, not limited to the first end device and the second end device.

[0036] Furthermore, the first or second terminal device can be a network device, such as a server, server cluster, or data center. The devices in the first, second, and third layers can be network devices, such as switches, but this embodiment does not impose any limitations on this.

[0037] The technical solutions provided in the embodiments of this application will be described in detail below.

[0038] This embodiment designs a fast rerouting algorithm for data center networks based on color block matching for the CLOS network architecture. It utilizes five-tuple monitoring and in-band network telemetry technology to enable end-network collaborative information perception and rapid congestion localization. Simultaneously, the color block matching algorithm finds a new transmission path within linear time complexity, updating the original congested path and thus accelerating the congestion recovery process.

[0039] See Figure 2 This application provides a congestion recovery method based on fast rerouting, which can be applied to the aforementioned methods. Figure 1 The system shown, and the method can be executed by a first end device or a second end device, specifically, the method includes:

[0040] Step 101: Obtain multiple transmission paths between the first end device and the second end device, passing through at least one layer of network devices.

[0041] Each transmission path is a route from the first end device through a switch in each network layer to the second end device, and all transmission paths from the first end device to the second end device can be planned and obtained in advance when constructing the network. Optionally, in this embodiment, there are M transmission paths between the first end device and the second end device, where M ≥ 2.

[0042] Step 102: Use the five-tuple monitoring technology to determine the path of the traffic to be transmitted among the multiple transmission paths, and determine the status information corresponding to the path of the traffic to be transmitted.

[0043] A quintuple typically refers to the source IP address (src_IP), source port (src_port), destination IP address (dst_IP), destination port (dst_port), and transport layer protocol.

[0044] When task traffic arrives, RDMA calls ib_modify_qp to create source and destination queues for communication with qp. Five-tuple monitoring based on extended Berkeley Packet Filter (eBPF) technology is used to track this function call and obtain the five-tuple information for this flow.<protocol,src_ip,dst_ip,src_port,dst_port> Then, based on the equal cost multipath (ECMP) algorithm, after obtaining the quintuple information, the path that the traffic will take can be determined, i.e., the path of the traffic to be transmitted.

[0045] The status information corresponding to the traffic path to be transmitted includes the status information of each switch on the path. The status of the switches includes three states: no traffic transmission, traffic transmission but saturation, and transmission saturation, i.e., the traffic has reached a threshold. These states can be obtained by measuring parameters such as latency and packet loss rate. In this embodiment, the status information corresponding to the traffic path to be transmitted refers to the states of whether each traffic path is congested, has traffic transmission but is not congested, or has no traffic transmission.

[0046] Step 103: Detect whether congestion has occurred on the path of the traffic to be transmitted based on the status information.

[0047] Since the transmission path includes at least one switch, it is possible to determine whether the transmission path containing that switch is congested by detecting the status of the switch. Furthermore, the status of each switch can be determined by parameters such as packet loss rate and latency. For example, on the first path, if a switch on the first path becomes congested, then the first path is determined to be congested; conversely, if all switches on the first path are not congested, then the first path is determined to be uncongested.

[0048] Step 104: If congestion occurs, the target path is found in at least one candidate path using the color block matching algorithm, and the congested path is updated to the target path.

[0049] The color-block matching algorithm is a method for determining available and congested switches based on the status of different switches along various paths. This method can be used to find target paths, i.e., paths that are not congested. Color blocks refer to a pre-calculated, color-coded network diagram of the entire network, where each switch is marked with a color from dark to light (e.g., red, pink, white) to represent three different switch states. Darker colors indicate higher traffic volume, potentially even congestion; lighter colors indicate lower traffic volume, or even no traffic.

[0050] In this step, the path of the switch with a color block matching algorithm is found to be white or pink, the target path is determined, and then the port number of the original path is updated to the new port number in the target path, thereby quickly alleviating congestion.

[0051] The method provided in this embodiment can detect paths in advance when congestion occurs and quickly find the optimal path (target path) for rerouting, thereby reducing the impact of congestion on application performance. Specifically, in terms of congestion awareness, five-tuple monitoring is used to quickly perceive the original application path and network conditions, thereby determining which paths in the transmission path are congested. In terms of congestion recovery, a color-block matching algorithm is used to find the target path in linear time complexity, and then fast rerouting is performed by modifying the source port number, thereby quickly resolving the congestion problem.

[0052] Furthermore, this method improves overall performance by over 10% in terms of bandwidth through the use of congestion awareness and recovery mechanisms, enabling faster training tasks. Simultaneously, compared to previous random hashing routing, proactive routing using rerouting exhibits lower variance, more stable overall bandwidth, and less fluctuation in task training.

[0053] Optionally, in one possible implementation of this embodiment, step 102 specifically includes: determining the path of the traffic to be transmitted among multiple transmission paths using probe messages sent by in-band network telemetry based on the 5-tuple monitoring traffic information.

[0054] By utilizing 5-tuple-based monitoring and INT-based fast probes, the path traversed by current traffic and its congestion status are simulated and assessed. For example, using INT, a full path probe is performed by traversing ports to obtain the correspondence between source ports and port paths, where the source port and path have a one-to-one relationship. Simultaneously, since the relationship between paths and switches is many-to-many, to save space, the switch status (Switch_info) is stored separately, and the congestion level of the transmission path is judged based on latency and packet loss rate.

[0055] Inband Network Telemetry (INT) is a hybrid measurement technology that fundamentally uses data plane traffic to collect, carry, organize, and report network status information without using separate control plane management traffic. INT technology has two key aspects: First, it's "Inband," meaning it utilizes data plane traffic to avoid using dedicated protocol messages to achieve the desired protocol objectives; second, it's "Telemetry," characterized by measuring network data and reporting it remotely.

[0056] Based on the src_port in the 5-tuple information, INT can be used to simulate business traffic. Since the 5-tuple is the same, the path probed by INT is the same as the path that the task traffic will take. Based on this, the path to be transmitted can be obtained, such as the original path old_path.

[0057] Further, in this embodiment, the status information corresponding to the path of the traffic to be transmitted includes: status information of at least one switch on the path of the traffic to be transmitted. Step 102 above, determining the status information corresponding to the path of the traffic to be transmitted, includes: using the probe message to determine the status information of at least one switch on the path of the traffic to be transmitted.

[0058] In this embodiment, the fast probe method can quickly predict the path of the traffic to be transmitted by using 5-tuple monitoring and in-band network telemetry, and obtain the path to be transmitted and the network status (switch status information) corresponding to these paths, so as to prepare for the subsequent judgment and recovery of congested networks.

[0059] In one possible implementation of the above embodiments, the status information of the switch is represented by transmission delay. Step 103 above, detecting whether congestion occurs on the path of the traffic to be transmitted based on the status information, includes: traversing whether the transmission delay of each switch on the path of the traffic to be transmitted exceeds a threshold; if the transmission delay of at least one switch exceeds the threshold, then it is determined that congestion has occurred on the entire transmission path where the switch is located.

[0060] Specifically, first, according to step 102 above, the paths of the traffic to be transmitted and the status information of these paths are determined.

[0061] Taking an n-layer switch as an example, it is possible to obtain the switch for each hop on the transmission path and the congestion status of that switch.

[0062] For example, the status information corresponding to path 1 to be transmitted includes:

[0063]

[0064] Here, Switch_1~n represents switches 1~n, delay_1~n and loss_1~n correspond to the latency and packet loss rate of switches 1~n, respectively. By comparing whether the latency and packet loss rate exceed the threshold, the degree of congestion of the switches can be judged. Taking latency as an example, delay_limit is the latency threshold (this value is obtained through experience).

[0065] Currently, the initial state of a switch in path 1 is represented by a color, i.e., white (switch_state = 0). If the delay of this switch exceeds the threshold (delay_i > delay_limit) at a certain detection time, the corresponding color of the switch is marked as red (switch_state = 2); if it does not exceed the threshold (delay_limit), it is marked as pink (switch_state = 1). This detection process will continue until the quintuple is destroyed.

[0066] According to this method, latency is detected at each switch along the transmission path. If the switch is marked in red, the entire path containing that switch is considered to be congested, and step 104 is executed. If the switch is marked in pink, it means that there is traffic transmission on the switch but no congestion has occurred. In this case, rerouting is not required, and the process ends after updating the path load information.

[0067] The switch status information (represented by the character "S") in the network of the entire topology and the load information in the path (represented by the character "P") can be used as the axis and matrix of the color block matching algorithm, respectively.

[0068]

[0069]

[0070] In this context, switch_1 represents the first-layer switch, switch_3 represents the third-layer switch; switch_1_1 represents the first switch in the first layer, switch_1_a represents the a-th switch in the first layer, switch_3_1 represents the first switch in the third layer, and switch_3_c represents the c-th switch in the third layer.

[0071] All available paths from the first end device to the second end device, where path_1_1_1 represents the path consisting of the first switch of the first layer, the first switch of the second layer, and the first switch of the third layer; path_a_b_c represents the path consisting of the a-th switch of the first layer, the b-th switch of the second layer, and the c-th switch of the third layer, and so on for other paths.

[0072] Optionally, in one possible implementation, before step 103 above, which detects whether congestion has occurred on the path of the traffic to be transmitted, such as... Figure 3 As shown, it also includes:

[0073] Step 103-1: Mark each switch along the multiple paths with a color beforehand, where different colors represent different levels of congestion at the switches.

[0074] The different colors include: a first color, a second color, and a third color. The first color indicates no traffic transmission on the switch; the second color indicates that the traffic on the switch has reached the load threshold, indicating congestion; and the third color indicates that there is traffic transmission on the switch, but the load threshold has not been reached, indicating no congestion. Step 104 above includes:

[0075] Step 104-1: Based on the color markings on all switches, determine one or more target switches using a matching algorithm of color blocks on the axis and color blocks in the matrix.

[0076] Step 104-2: Determine the target path through the target switches based on the congestion status of the one or more target switches.

[0077] Specifically, step 104-1 above includes: determining the color of the switch on each layer based on the color markings on all switches; if the color is the first color, determining the current switch as the target switch; if the color is the second color, determining that the current switch cannot be selected as the target switch; if the color is the third color, determining the target switch by combining the application information on the path where the current switch is located.

[0078] In a specific example, for each layer of switches, the optimal switch (target switch) is selected sequentially. Taking the i-th layer switch switch_i as an example: First, the color of the switch on the axis is determined. If the color is white (i.e., the first color), it means there is no traffic on the switch, and it can be selected directly. If it is red (i.e., the second color), it means the current switch has reached its load threshold and cannot be selected. If it is pink (i.e., the third color), it means the current switch has traffic but is not congested, which is a common situation in data centers, and the colors in the matrix need to be compared further. The colors in the matrix represent application information on the path, such as the number of flows and the duration of traffic on the path. Taking the number of flows as an example, the lighter the color, the more flows on the switch, so the lightest color block min{path_i_j_k} is selected. By matching the color blocks on the axis (switch information) and the color blocks in the matrix (path information), the selection of the target switch can be achieved in linear time.

[0079] In the above embodiments, after determining that the path of the traffic to be transmitted is congested, congestion recovery is required. In this embodiment, step 104 utilizes a color-block matching algorithm, which can quickly find the target path within linear time complexity and achieve rerouting by modifying the source port number, thus resolving the data center congestion problem.

[0080] Specifically, in one possible implementation, determining the target path containing the target switch based on the one or more target switches includes: obtaining load information on the path where the target switch is located; and determining the target path based on the load information and the congestion status of the one or more target switches.

[0081] In this embodiment, the load information on the path where the target switch is located can be determined by the source port src_port in the five-tuple information and the simulated service traffic of the in-band network telemetry. The load information on each transmission path is obtained, and then the currently available target path is determined by the load information and the congestion status of the switch (such as by the latency or packet loss rate mentioned above).

[0082] For example, if there are three paths: path1, path2, and path3, and the original path is path1, and it is determined that the switch on path1 is congested, while the switches on path2 and path3 are transmitting traffic but not congested, then the path with the lower load can be selected as the target path based on the load conditions on path2 and path3. For example, if path2 has a lower load, then path2 will be selected as the target path.

[0083] After selecting the target switch and target path for each layer of the topology in the above steps, it is necessary to combine the switches of each layer into a complete path. Since the topologies commonly found in current data centers are fully connected, the new path `new_path` can be obtained simply by combining the switches of each layer.

[0084] In this embodiment, the selection of the target path not only considers network-side information, such as switch congestion, but also end-side load information, achieving end-network convergence-assisted routing. Network-side information can quickly narrow down the search space and improve search efficiency; end-side load information can further refine the route selection based on the network-side routing results, resulting in a lower load and higher transmission efficiency for the selected target path.

[0085] In another possible implementation of the above embodiments, such as Figure 4 As shown, step 105 above, which updates the congested path to the target path, specifically includes:

[0086] Step 105-1: Obtain the correspondence between the probed paths and ports in the multiple transmission paths.

[0087] Step 105-2: Determine the target port number corresponding to the target path based on the correspondence.

[0088] Step 105-3: Update the port number of the congested path to the target port number.

[0089] Specifically, based on the path-port correspondence Port_Path obtained from the full probe in the above steps, the corresponding port new_port, i.e., the target port number, can be obtained. The target path is then obtained, such as the new path (new_path) represented as:

[0090] new_path =<switch_1,switch_2,,...,switch_n>

[0091] The new port corresponding to this new path is: Port_Path = {...,<new_port,new_path> ...}

[0092] Step 105-3 modifies the source port number from `ori_port` to `new_port`, thus changing the path from the original `old_path` to the new `new_path`. Dynamic port selection can be easily achieved by adding the task's dynamic dependency library. When the port needs to be modified, simply write the 5-tuple to a fixed location; the dynamic dependency library will automatically read it and perform rerouting. If the dynamic dependency library cannot be automatically read, routing will proceed according to the default method.

[0093] Optionally, after step 104, the method further includes: destroying the quintuple information and re-monitoring the congestion status of multiple transmission paths between the first end device and the second end device.

[0094] Specifically, at the end of the task, 5-tuple monitoring is used to track calls to the ib_destory_qp function and destroy the 5-tuple information. For streams already in the data center, their state is maintained throughout their lifecycle.

[0095] In this embodiment, after rerouting a congested path, the path information needs to be updated so that subsequent traffic can be congested and recovered based on the path's status. One implementation is to increment the path load by 1, expressed as: path_i_j_k = path_i_j_k + 1

[0096] Correspondingly, the colors of the color blocks in the matrix are updated according to the load level; at the same time, the color block information on the axis is updated, for example, the color is changed to the color corresponding to the current detection step. Also, since a new port `new_port` is used in step 105-3, probing needs to continue to complete the other ports corresponding to the path `new_path`.

[0097] In one specific embodiment, such as Figure 5 As shown, a single flow path reconstruction process is disclosed.

[0098] by Figure 5 Taking the first two layers (S0 and S1) of the network topology as an example, assuming the current network status is as shown in Table 1. It can be seen that 15 traffic streams (1-15) are unevenly distributed in the current topology (reflected by latency), and after probing, 12 ports (numbered 50001-50012) are available for user use. The congestion status of the switches is judged using a latency threshold; if the latency is greater than 1000ns, congestion is considered to have occurred. Additionally, the switch status column represents the switch's color: "0" is white, indicating no traffic on the current switch; "1" is pink, indicating traffic on the current switch but not congested; and "2" is red, indicating traffic on the current switch and congestion has occurred.

[0099] Table 1. Real-time Network Status

[0100]

[0101] In a machine learning training task, there are multiple streaming paths. In this example, we take a new stream, such as a distance of 16. First, we detect whether the path is congested, i.e., congestion location, which corresponds to steps 101-103 above.

[0102] Specifically, this includes: Before the task begins, a full probe using INT is performed to obtain the correspondence between ports (50001-50012) and paths, as shown in the table below (e.g., port 50001 corresponds to path S0_1-S1_1). When traffic arrives, the source port number is monitored using a 5-tuple, and the path obtained using INT is S0_3-S1_1, where S0_3-S1_1 indicates that a path is formed between switches S0_3 and S1_1. Simultaneously, the switch latency can be obtained. As shown in Table 1, the latency of switch S0_3 is 1600ns, which is greater than the threshold of 1000ns, so it is marked in red; similarly, the latency of switch S1_1 is 1500μs, which is also greater than the threshold, so it is also marked in red. Based on the color markings, congestion was detected on the path from switch S0_3 to S1_1 of the traffic to be transmitted. Therefore, it is necessary to reroute the congested path, for example, by using a color block matching algorithm.

[0103] The congestion recovery process, corresponding to step 104 above, specifically includes:

[0104] For switch S0, the current switch is red, and all candidate switches are pink (meaning there is traffic on the candidate switches). Therefore, the lightest color block (meaning there are the fewest flows on the current path) is selected from the matrix drawn by the pre-traffic detection. For example, switch S0_1 is selected. For switch S1, the current switch is red, and there are white candidate switches (meaning there is no traffic on this switch). Therefore, S1_4 is selected directly.

[0105] Additionally, the currently selected path is S0_1-S1_4, and according to the table, the available port on this path is 50017. Finally, by modifying the source port to 50017 and replacing the source port number 51472, a new route is rebuilt, thus avoiding the congested path. Simultaneously, the path load is updated, changing the load status of path S0_1-S1_4 from 0 to 1.

[0106] Furthermore, when using the color block matching algorithm for performance evaluation, this embodiment provides three sets of experimental data for three different topologies (Topology 1, Topology 2, and Topology 3) of rerouting. For example, the method steps of this implementation are performed under Layer 1, Layer 2, and Layer 3 topologies, respectively. Figure 6 As shown, the color block matching algorithm can find the optimal path in linear time complexity. The evaluation results of the color block matching algorithm are shown in Table 2.

[0107] Table 2. Evaluation of Color Block Matching Algorithm

[0108] Topology Brute-force search (μs) Color block matching (μs) 1 45.5 7.4 2 267.4 9.1 3 2698.5 11.8

[0109] As shown in Table 2, for the general brute-force search method, the time to select the optimal path increases exponentially with the linear increase in the number of topology layers, reaching the millisecond level at four layers. In this embodiment, by using color-block matching, the routing time increases linearly with the number of topology layers, and the time to select the optimal path also remains linear, consistently at the microsecond level. Moreover, the more topology layers there are, the more obvious the advantages of the color-block matching algorithm become, saving a significant amount of time and improving the efficiency of rerouting compared to the brute-force search method.

[0110] In addition, regarding the overall application performance evaluation, the single-flow rerouting scheme of this embodiment was used for an overall evaluation during a single machine learning training task. Taking all-reduce as an example, before using the rerouting scheme, the overall bandwidth was low and unstable; after using the fast rerouting scheme, the average bandwidth was improved, and it could approach the theoretical maximum bandwidth (100GB / s) even with background flows, as shown in Table 3.

[0111] Table 3. Average bandwidth of the all-reduce algorithm

[0112] 1 2 3 4 5 Avg Before using rerouting (GB / s) 84 85 88 73 68 79 After using rerouting (GB / s) 90 90 86 90 86 88

[0113] As shown in Table 3, before using the congestion awareness and recovery mechanism, the bandwidth of the all-reduce algorithm was very unstable, and the average bandwidth was low, less than 80GB / s. After using the congestion awareness and recovery mechanism provided in this embodiment, the bandwidth can be improved by more than 10%, which is closer to the theoretical maximum value, allowing the training task to be executed faster. At the same time, compared with the previous random hash routing, the active routing using rerouting has a smaller variance, more stable overall bandwidth, and less fluctuation in task training, making it more stable.

[0114] The method provided in this embodiment, in terms of congestion awareness, uses the ECMP routing algorithm to obtain the path from the hash of the five-tuple information. In RDMA scenarios, the commonly used traffic transmission protocol is RoCEv2 (based on UDP, protocol value 17). The source and destination IPs are specified by the service, and the destination port is fixed at 4791. Therefore, determining the source port uniquely determines the path. Using a congestion awareness method based on fast probes, the path traversed by service traffic is pre-positioned during the RDMA connection establishment phase through five-tuple monitoring. Then, an INT probe message is sent to pre-probe indicators such as latency and packet loss rate of the original path, thereby quickly determining whether congestion has occurred on the path of the traffic to be transmitted.

[0115] For congestion recovery, the path of the traffic to be transmitted, obtained through 5-tuple monitoring, and the global state information obtained through switch information probing are used as inputs. The next step is to use a color block matching algorithm to obtain a new path for rerouting within linear time complexity. For multi-layer switches, the optimal state of the switch is searched sequentially at each layer and combined into a complete path. At the same time, the corresponding port number can be found for rerouting through full INT probing.

[0116] This invention also discloses a congestion recovery device based on fast rerouting, such as... Figure 7 As shown, the device is used to implement the congestion recovery method in the foregoing embodiments. The device includes: an acquisition unit 710, a determination unit 720, a detection unit 730, a search unit 740, and an update unit 750. In addition, the device may include other more or fewer units and structures, which are not limited in this embodiment.

[0117] The acquisition unit 710 is used to acquire multiple transmission paths between the first end device and the second end device that pass through at least one layer of network devices.

[0118] The determining unit 720 is used to determine the path of the traffic to be transmitted among the multiple transmission paths using the five-tuple monitoring technology, and to determine the status information corresponding to the path of the traffic to be transmitted.

[0119] The detection unit 730 is used to detect whether congestion occurs on the path of the traffic to be transmitted based on the status information.

[0120] The search unit 740 is used to find the target path in at least one candidate path using a color block matching algorithm when the detection unit detects that congestion has occurred.

[0121] The update unit 750 is used to update the congested path to the target path.

[0122] Optionally, in one specific implementation of this embodiment, the determining unit 720 is specifically used to determine the path of the traffic to be transmitted among multiple transmission paths based on the five-tuple information of the five-tuple monitoring traffic and the probe message sent by in-band network telemetry.

[0123] Optionally, in one specific implementation of this embodiment, the status information includes: status information of at least one switch on the path of the traffic to be transmitted. The determining unit 720 is further configured to determine the status information of at least one switch on the path of the traffic to be transmitted using the probe message.

[0124] Optionally, in another specific implementation of this embodiment, the switch status information is represented by transmission delay; the detection unit 730 is specifically used to traverse whether the transmission delay of each switch on the path of the traffic to be transmitted exceeds a threshold; if so, it is determined that congestion has occurred on the entire transmission path where the switch is located.

[0125] Optionally, in another specific embodiment of this invention, the above-mentioned device further includes a configuration unit, wherein the configuration unit... Figure 7 Not shown, this configuration unit is used to pre-mark each switch along the multiple paths with a color before detecting whether congestion occurs on the path of the traffic to be transmitted, wherein different colors represent different congestion conditions of the switches.

[0126] The lookup unit 740 is further configured to determine one or more target switches based on the color markings on all switches, using a matching algorithm of color blocks on axes and color blocks in a matrix; and to determine the target path passing through the target switches based on the congestion status of the one or more target switches.

[0127] Optionally, in another specific embodiment of this example, the lookup unit 740 is further configured to obtain load information on the path where the target switch is located; and determine the target path based on the load information and the congestion status of the one or more target switches.

[0128] Optionally, in another specific embodiment of this example, the different colors include: a first color, a second color, and a third color, wherein the first color represents no traffic transmission on the switch, the second color represents that the traffic on the switch has reached the load threshold, and the third color represents that there is traffic transmission on the switch, but the load threshold has not been reached.

[0129] The lookup unit 740 is also used to determine the color of the switch on each layer based on the color markings on all switches;

[0130] If the color is the first color, then the current switch is determined to be the target switch;

[0131] If the color is the second color, then it is determined that the current switch cannot be selected as the target switch;

[0132] If the color is the third color, then the target switch is determined by combining the application information on the path where the current switch is located.

[0133] Optionally, in another specific embodiment of this example, the update unit 750 is specifically used to obtain the correspondence between the probed paths and ports in the multiple transmission paths; determine the target port number corresponding to the target path according to the correspondence; and update the port number of the congested path to the target port number.

[0134] In addition, the above-mentioned device also includes a destruction unit, which is used to destroy the five-tuple information after updating the congested path to the target path, and to re-monitor the congestion status of multiple transmission paths between the first end device and the second end device through the detection unit 730.

[0135] The device provided in this embodiment can detect paths in advance when congestion occurs and quickly find the optimal path for rerouting, thereby reducing the impact of congestion on application performance. Specifically, in terms of congestion perception, five-tuple monitoring is used to quickly perceive the original application path and network conditions, thereby determining which paths in the transmission path are congested. In terms of congestion recovery, a color block matching algorithm is used to find the target path in linear time complexity, and then fast rerouting is performed by modifying the source port number, thereby quickly resolving the congestion problem.

[0136] In addition, embodiments of the present invention also provide an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 110 and a memory 120, wherein the processor 110 and the memory 120 may be connected via a bus or other means. Figure 8 For example, the connection is via a bus. Furthermore, the electronic device also includes at least one interface 130, which can be a communication interface or other interface; this embodiment does not impose any limitations on this.

[0137] The processor 110 can be a central processing unit (CPU). The processor 110 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0138] The memory 120, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the video synthesis method in this embodiment of the invention. The processor 110 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory 120, thereby implementing the congestion recovery method of fast rerouting in the above method embodiment.

[0139] The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor 110, etc. Furthermore, the memory 120 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the processor 110 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0140] In addition, at least one interface 130 is used for communication between the electronic device and external devices, such as communication with a server. Optionally, at least one interface 130 can also be used to connect peripheral input / output devices, such as a keyboard or display screen.

[0141] The one or more modules are stored in the memory 120, and when executed by the processor 110, they perform actions such as... Figures 2 to 4 The embodiment shown illustrates a congestion recovery method based on fast rerouting.

[0142] Optionally, the aforementioned electronic device is an end device, such as a first end device or a second end device; or it can be a network device, such as a switch in each layer.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0144] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A congestion recovery method based on fast rerouting, characterized in that, The method includes: Obtain multiple transmission paths between the first-end device and the second-end device, which pass through at least one layer of network devices; The five-tuple monitoring technology is used to determine the path of the traffic to be transmitted among the multiple transmission paths, and to determine the status information corresponding to the path of the traffic to be transmitted. Based on the status information, detect whether congestion has occurred on the path of the traffic to be transmitted; If congestion occurs, the color block matching algorithm is used to find the target path in at least one candidate path, and the congested path is updated to the target path. The step of finding the target path in at least one candidate path using a color block matching algorithm includes: Based on the color markings on all switches, one or more target switches are determined using a matching algorithm of color blocks on the axis and color blocks in the matrix; The target path through the target switches is determined based on the congestion status of the one or more target switches.

2. The method according to claim 1, characterized in that, The method of using 5-tuple monitoring technology to determine the path of the traffic to be transmitted among the multiple transmission paths includes: Based on the quintuple information of the monitored traffic, the path of the traffic to be transmitted is determined among multiple transmission paths using probe messages sent by in-band network telemetry.

3. The method according to claim 2, characterized in that, The status information includes: status information of at least one switch on the path of the traffic to be transmitted; Determining the status information corresponding to the traffic path to be transmitted includes: The probe message is used to determine the status information of at least one switch on the path of the traffic to be transmitted.

4. The method according to claim 3, characterized in that, The switch status information is represented by transmission delay; Detecting whether congestion occurs on the path of the traffic to be transmitted based on the status information includes: Check whether the transmission delay of each switch on the path of the traffic to be transmitted exceeds the threshold. If the transmission delay of at least one switch exceeds the threshold, then congestion is determined to have occurred on the entire transmission path where the switch is located.

5. The method according to any one of claims 1-4, characterized in that, Before detecting whether congestion has occurred on the path of the traffic to be transmitted, the method further includes: Each switch along the multiple paths is pre-marked with a color, where different colors represent different levels of congestion at the switches.

6. The method according to claim 5, characterized in that, Determining the target path containing the target switches based on the one or more target switches includes: Obtain the load information on the path where the target switch is located; The target path is determined based on the load information and the congestion status of the one or more target switches.

7. The method according to claim 5, characterized in that, The different colors include: a first color, a second color, and a third color, wherein the first color indicates that there is no traffic transmission on the switch, the second color indicates that the traffic on the switch has reached the load threshold, and the third color indicates that there is traffic transmission on the switch, but the load threshold has not been reached. Based on the color markings on all switches, one or more target switches are identified using a matching algorithm of color blocks on the axis and color blocks in the matrix, including: Determine the color of the switches on each layer based on the color markings on all switches; If the color is the first color, then the current switch is determined to be the target switch; If the color is the second color, then it is determined that the current switch cannot be selected as the target switch; If the color is the third color, then the target switch is determined by combining the application information on the path where the current switch is located.

8. The method according to any one of claims 1-4, characterized in that, Updating the congested path to the target path includes: Obtain the correspondence between the probed paths and ports in the multiple transmission paths; Based on the correspondence, determine the target port number corresponding to the target path; Update the port number of the congested path to the target port number.

9. The method according to any one of claims 2-4, characterized in that, After updating the congested path to the target path, the method further includes: The quintuple information is destroyed, and the congestion status of multiple transmission paths between the first end device and the second end device is re-monitored.

10. A congestion recovery device based on fast rerouting, characterized in that, The device includes: The acquisition unit is used to acquire multiple transmission paths between the first end device and the second end device that pass through at least one layer of network devices. The determining unit is used to determine the path of the traffic to be transmitted among the multiple transmission paths using the five-tuple monitoring technology, and to determine the status information corresponding to the path of the traffic to be transmitted. The detection unit is used to detect whether congestion has occurred on the path of the traffic to be transmitted based on the status information; The search unit is used to find the target path from at least one candidate path by using a color block matching algorithm when the detection unit detects that congestion has occurred. An update unit is used to update the congested path to the target path; The step of finding the target path in at least one candidate path using a color block matching algorithm includes: Based on the color markings on all switches, one or more target switches are determined using a matching algorithm of color blocks on the axis and color blocks in the matrix; The target path through the target switches is determined based on the congestion status of the one or more target switches.

11. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory is coupled to the processor; The memory stores computer-readable program instructions, which, when executed by the processor, implement the congestion recovery method based on fast rerouting as described in any one of claims 1 to 9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, the congestion recovery method based on fast rerouting as described in any one of claims 1 to 9 is implemented.

Images