Evaluating available bandwidth between leaf switches in fat tree networks

Abstract

The invention discloses evaluating available bandwidth between leaf switches in a fat tree network. A system for bandwidth estimation includes an interface and a processor. The interface communicates with a fat tree (FT) network that includes a leaf switch belonging to an underlying layer, a backbone switch belonging to a top layer, and an intermediate switch belonging to one or more intermediate layers. The processor: estimates an available bandwidth between a first leaf switch and a second leaf switch by identifying a first bottom-to-top subtree to which the first leaf switch belongs and a second bottom-to-top subtree to which the second leaf switch belongs; determining a path count for at least some switches in the intermediate layer, wherein the path count for a switch indicates a number of paths through the switch, via the first bottom-to-top sub-tree to the first leaf switch, and via the second bottom-to-top sub-tree to the second leaf switch; and estimating the available bandwidth based on the path count.

Description

Technical Field

[0001] The present invention relates generally to data communication networks, and more particularly to a method and system for estimating available bandwidth in a Fat-Tree (FT) network. Background Art

[0002] Fat-Tree (FT) is a network topology widely used in data centers and other data communication networks. The FT topology was first described by Leiserson in "Fat-Trees: Universal Networks for Hardware-Efficient Supercomputing", IEEE Transactions on Computers, Volume C-34, Issue 10, October 1985.

[0003] A generalized form of the FT topology was described by Ohring et al. in "On Generalized Fat Trees," Proceedings of the 9th International Symposium on Parallel Processing, Santa Barbara, California, April 1995. Another class of FT topologies, called Quasi Fat Trees (QFT), was described by Zahavi et al. in "Quasi Fat Trees for HPC Clouds and Their Fault-Resilient Closed-Form Routing," Proceedings of the 22nd Annual IEEE Symposium on High Performance Interconnects, Mountain View, California, August 2014. Summary of the invention

[0004] One embodiment of the present invention described herein provides a system for bandwidth estimation, which includes an interface and a processor. The interface is used to communicate with a fat tree (FT) network, which includes multiple switches. These switches include (i) multiple leaf switches belonging to the bottom layer, (ii) multiple trunk switches belonging to the top layer, and (iii) multiple intermediate switches belonging to one or more intermediate layers between the top layer and the bottom layer; multiple links connect selected switches among these switches. The processor is used to: receive topology information indicating the current topology of the FT network from the FT network via the interface, and estimate the available bandwidth between the first leaf switch and the second leaf switch based on the topology information. The processor is used to estimate the available bandwidth in the following manner: identifying, from a plurality of bottom-to-top subtrees of the FT network (each bottom-to-top subtree extending from a leaf switch to a top layer), a first bottom-to-top subtree to which a first leaf switch belongs and a second bottom-to-top subtree to which a second leaf switch belongs; determining path counts of at least some switches in the middle layer, wherein the path counts of the switches indicate the number of paths that (i) pass through the switch, (ii) reach the first leaf switch via the first bottom-to-top subtree, and (iii) reach the second leaf switch via the second bottom-to-top subtree; and estimating the available bandwidth between the first leaf switch and the second leaf switch based on the path counts.

[0005] In some embodiments, the processor is configured to: identify a lowest common layer, the lowest common layer being defined as a lowest intermediate layer of the FT network including a joint parent switch of a first leaf switch and a second leaf switch; and determine path counts of switches in the lowest common layer.

[0006] In the disclosed embodiment, a processor is used to: assign indexes to switches in an FT network by scanning leaf switches in sequence, and for each leaf switch in the sequence, traverse a bottom-to-top subtree to which the leaf switch belongs from the leaf switch to the top layer, and assign an index to each traversed switch that has not yet been assigned any index, the index being unique to the leaf switch at least within the layer to which the switch belongs; define (i) a first group including indexes assigned to a parent of a first leaf switch in a lowest common layer, and (ii) a second group including indexes assigned to a parent of a second leaf switch in the lowest common layer; and for each switch in the lowest common layer, determine a path count based on an intersection between the first group of indexes and the second group of indexes.

[0007] In an example embodiment, after completing the sequence, the processor is configured to identify switches that are not assigned any index or are assigned conflicting indexes, and assign non-conflicting consolidated indexes to the identified switches.

[0008] In one embodiment, a processor is configured to determine a total path count between a first leaf switch and a second leaf switch by: identifying a set of switches in a given layer that are common parents of the first leaf switch and the second leaf switch; for each common parent in the set, finding a minimum between (i) a first number of ports connecting the common parent to the first leaf switch and (ii) a second number of ports connecting the common parent to the second leaf switch; and summing the minimum values ​​over the set of common parents.

[0009] In some embodiments, the topology information indicates a failure in one or more links.In an example embodiment, when estimating the available bandwidth, the processor only considers failures that occur in links connecting the leaf switches to the middle tier immediately above the bottom tier.

[0010] According to the embodiments described herein, a method for bandwidth estimation is also provided. The method includes: receiving topology information indicating a current topology of a fat tree (FT) network, the FT network including a plurality of switches and a plurality of links connecting selected switches among the plurality of switches, the plurality of switches including: (i) a plurality of leaf switches belonging to a bottom layer, (ii) a plurality of spine switches belonging to a top layer, and (iii) a plurality of intermediate switches belonging to one or more intermediate layers between the top layer and the bottom layer. Based on the topology information, an available bandwidth between a first leaf switch and a second leaf switch is estimated in the following manner: from a plurality of bottom-to-top subtrees of the FT network (each bottom-to-top subtree extending from a leaf switch to a top layer), a first bottom-to-top subtree to which the first leaf switch belongs and a second bottom-to-top subtree to which the second leaf switch belongs are identified; path counts of at least some switches in the intermediate layer are determined, wherein the path counts of the switches indicate the number of paths that (i) pass through the switch, (ii) reach the first leaf switch via the first bottom-to-top subtree, and (iii) reach the second leaf switch via the second bottom-to-top subtree; and the available bandwidth between the first leaf switch and the second leaf switch is estimated based on the path counts.

[0011] According to the embodiments described herein, there is also provided a computer software product comprising a tangible, non-transitory computer-readable medium storing program instructions. These instructions, when read by a processor, cause the processor to: receive topology information indicating a current topology of a fat tree (FT) network, the FT network comprising a plurality of switches and a plurality of links connecting selected switches of the plurality of switches, the plurality of switches comprising (i) a plurality of leaf switches belonging to a bottom layer, (ii) a plurality of spine switches belonging to a top layer, and (iii) a plurality of intermediate switches belonging to one or more intermediate layers between the top layer and the bottom layer; and estimate, based on the topology information, an available bandwidth between a first leaf switch and a second leaf switch by: identifying, from a plurality of bottom-to-top subtrees of the FT network, each bottom-to-top subtree extending from a leaf switch to a top layer, a first bottom-to-top subtree to which the first leaf switch belongs and a second bottom-to-top subtree to which the second leaf switch belongs; determining path counts of at least some of the switches in the intermediate layer, wherein the path counts of the switches indicate the number of paths that (i) pass through the switch, (ii) reach the first leaf switch via the first bottom-to-top subtree, and (iii) reach the second leaf switch via the second bottom-to-top subtree; and estimate the available bandwidth between the first leaf switch and the second leaf switch based on the path counts.

[0012] The present invention will be more fully understood through the following detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram schematically illustrating a network management system (NMS) for managing a fat tree (FT) network according to an embodiment of the present invention;

[0014] Figure 2A and Figure 2B is a diagram schematically illustrating an FT network with a failed link according to an embodiment of the present invention;

[0015] Figure 3 According to an embodiment of the present invention Figure 2A and Figure 2B A table of the maximum available bandwidth between leaf switch pairs in the FT network;

[0016] Figure 4 is a flowchart schematically illustrating a method for estimating a maximum available bandwidth between leaf switch pairs using coloring according to an embodiment of the present invention;

[0017] Figure 5 is a diagram schematically illustrating a process of assigning an index ("coloring") to a parent switch according to an embodiment of the present invention;

[0018] Figure 6 and Figure 7 is a diagram illustrating an example pseudo code for identifying a parent switch of each leaf switch in an FT network according to an embodiment of the present invention;

[0019] Figure 8-Figure 11 is a diagram illustrating example pseudo code for assigning indices ("coloring") to switches according to an embodiment of the present invention;

[0020] Fig.12 is a diagram showing an example pseudo code for counting the number of paths between leaf switch pairs according to an embodiment of the present invention; and

[0021] Fig.13 is a diagram illustrating an example pseudo code for calculating the maximum available bandwidth between leaf switch pairs according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] Overview

[0023] A fat tree (FT) network includes multiple switches and multiple communication links connecting the switches to each other. The switches are arranged in layers, including a leaf layer (also called the bottom layer), a spine layer (also called the top layer), and optionally one or more middle layers. The switches in the leaf layer, the spine layer, and the middle layer are referred to as leaf switches, spine switches, and middle switches in this article, respectively. The hosts served by the FT network are usually connected to the switches in the leaf layer.

[0024] Various flavors of classic FT topologies have been developed and used, including the generalized FT and quasi-FT topologies cited above. In the context of this disclosure and claims, the term "FT network" refers broadly to classic FT and any variants thereof. A FT network can be viewed as comprising multiple "bottom-to-top subtrees". Each bottom-to-top subtree extends from a corresponding leaf switch to the backbone layer.

[0025] Links in an FT network may fail during the life of the network. In this context, the term "failed link" refers to a link that is unable to transmit data at the specified bandwidth and quality of the link. Links may fail for various reasons, for example, due to defects in the cable or switch port. The embodiments described herein refer primarily to complete failures, i.e., failures that reduce the link bandwidth to zero. However, the disclosed techniques can also be used for failures that reduce the link performance but still retain some available bandwidth.

[0026] It is worth noting that the number of failed links alone does not fully describe the actual impact of failed links on network performance. In other words, whether a certain number of failed links has a tolerable or severe impact on the network depends on where the failure occurs in the network topology. Understanding the impact of failed links on the main quality factors of the network enables improved network management. For example, maintenance operations, such as replacing failed links, can be correctly prioritized. For another example, processing tasks can be assigned to hosts while taking into account the actual impact of failed links. For another example, alarms triggered by failed links can be filtered and / or assigned different severities based on the actual severity of the failure.

[0027] The embodiments described herein provide methods and systems for evaluating the actual impact of a failed link on the performance of an FT network. More specifically, the disclosed technology evaluates the maximum available bandwidth between leaf switch pairs.

[0028] Knowing the available bandwidth between pairs of leaf switches (referred to herein as "leaf-to-leaf bandwidth") is of considerable value for "job placement," i.e., for assigning processing tasks to hosts. By taking into account the available leaf-to-leaf bandwidth, tasks that are closely related (and therefore communicate extensively with each other) can be assigned to hosts served by leaf switches that have large available bandwidth between them. By the same token, tasks that are independent of each other can be assigned to hosts served by leaf switches that have smaller leaf-to-leaf bandwidth.

[0029] The disclosed techniques are typically performed by a network management system (NMS) coupled to a FT network. In some embodiments, the NMS includes an interface for communicating with a switch of the FT network and a processor that performs the methods described herein. The processor receives topology information from the switch, which indicates which links are functional (operable) and which links are faulty. Based on the topology information, the processor estimates the maximum available bandwidth for each pair of leaf switches. The processor can estimate the corresponding maximum available bandwidth for each possible pair of leaf switches or only selected pairs of leaf switches.

[0030] In an example embodiment, the processor estimates the maximum available bandwidth between a pair of leaf switches (referred to as a first leaf switch and a second leaf switch) by:

[0031] ■Identify the first bottom-to-top subtree to which the first leaf switch belongs and the second bottom-to-top subtree to which the second leaf switch belongs. The first bottom-to-top subtree includes all parent switches of the first leaf switch. Similarly, the second bottom-to-top subtree includes all parent switches of the second leaf switch. Therefore, identifying the two bottom-to-top subtrees is equivalent to identifying the parent switch of each leaf switch.

[0032] ■ Using two bottom-to-top subtrees, identify the lowest common layer of the two leaf switches. The lowest common layer is defined as the lowest layer of the FT network that contains at least one switch that is a parent switch of the two leaf switches.

[0033] ■ Determine the "path count" on the switch at the lowest common layer. The path count of a switch indicates the number of paths that (i) pass through the switch, (ii) reach a first leaf switch via a first bottom-to-top subtree, and (iii) reach a second leaf switch via a second bottom-to-top subtree. Path counts are typically counted at a port (link) granularity, i.e., multiple links connected in parallel between two switches are counted as multiple paths.

[0034] ■ The available bandwidth between the first leaf switch and the second leaf switch is derived from the path count, for example, in units of link bandwidth.

[0035] In practice, the main contributor to the computational complexity of the above scheme is the task of finding a common parent switch for each pair of leaf switches. In some embodiments described herein, the processor efficiently performs this task in a preparatory stage called "coloring".

[0036] In a typical coloring process, a processor scans the FT network and assigns an index ("color") to each switch. Typically, the processor starts at a leaf switch and scans the subtree of leaf switches from bottom to top layer by layer toward the backbone layer. In a given layer, the processor assigns a unique index (also referred to herein as a "color") to the parent switch of the leaf switch. The index is unique within the layer and not necessarily unique across the entire network.

[0037] After completing the index assignment up to and including the top (trunk) layer, the processor proceeds to the next leaf switch and repeats the process. If the processor encounters a switch that has already been assigned an index (because it belongs to the bottom-to-top subtree of another leaf switch that has been scanned), the switch is skipped. When the bottom-to-top subtrees of all leaf switches have been scanned and "colored", the process ends. In some embodiments, the processor can then perform a consolidation process that assigns indexes to any switches that were missed due to failed (missed) links and / or were assigned conflicting indexes.

[0038] The "coloring" process is performed for the entire FT network, not for any specific leaf switch pair. The processor can then use the assigned index to find a set of common parents for any desired leaf switch pair (possibly but not necessarily all leaf switch pairs). The processor then derives the leaf-to-leaf bandwidth for any desired leaf switch pair by summing the path counts connecting the leaf switches at the common parents in the lowest common layer.

[0039] The leaf-to-leaf bandwidth calculation scheme disclosed herein is very efficient and therefore particularly attractive for use in large FT networks. A computational complexity comparison between the disclosed technique and the fastest known "max flow" algorithm (called "Dinic algorithm") is given below.

[0040] System Description

[0041] Figure 1 2 is a block diagram schematically illustrating a network management system (NMS) 20 used by an operator 24 to manage a fat tree (FT) network 28 according to an embodiment of the present invention. The network 28 may include, for example, a data center, a high performance computing (HPC) cluster, or any other suitable type of network. The network 28 may operate according to any suitable network protocol, such as Ethernet or InfiniBand. TM (IB).

[0042] exist Figure 1 In the example of FIG. 3 , NMS 20 includes an interface 32 and a processor 36. Interface 32 communicates with network 28. Processor 36 performs various tasks of NMS 20. Specifically, processor 36 estimates the available bandwidth between switch pairs in network 28 using the techniques described herein.

[0043] The network 28 includes a plurality of network switches 40 and a plurality of network links 44. The switches 40 may include, for example, a layer 2 Ethernet or IB switch. The network links 44 may include, for example, electrical or optical cables. Each link 44 connects a port of one switch 40 to a port of another switch 40. A given pair of switches may be connected in parallel via more than a single link 44. The network 28 serves a plurality of hosts 48, for example, servers or other computers.

[0044] According to the FT topology, the switches 40 of the network 28 are arranged in multiple layers (levels). This example shows a three-layer network, which includes a leaf layer 52 (also called the bottom layer), a spine layer 56 (also called the top layer), and a middle layer 60. The hosts 48 are connected to the switches in the leaf layers. In alternative embodiments, the network 28 may include multiple middle layers located between the leaf layers and the spine layer.

[0045] like Figure 1The configuration of NMS 20 and network 28 shown is an example configuration selected purely for conceptual clarity. In alternative embodiments, any other suitable configuration may be used. For the sake of clarity, elements that are not necessary for understanding the principles of the present invention are omitted from the figure.

[0046] The various elements of NMS 20 and network 28 may be implemented in hardware, for example, using one or more application specific integrated circuits (ASICs) or FPGAs, in software, or using a combination of hardware and software elements. In some embodiments, processor 36 may be implemented in part or in whole using one or more general purpose processors that are programmed with software to perform the functions described herein. For example, the software may be downloaded to any processor in electronic form over a network, or alternatively or additionally, provided and / or stored on a non-transitory tangible medium, such as a magnetic, optical or electronic memory.

[0047] Estimation of the maximum available bandwidth between leaf switches

[0048] Figure 2A is a diagram schematically illustrating an FT network 64 according to an embodiment of the present invention. In this example, the network 64 includes a total of 48 switches 40, which are arranged in three layers - leaf layer 52, spine layer 56 and middle layer 60. Each switch is numbered using a three-digit index, where:

[0049] ■The first digit indicates the tier to which the switch belongs (1-leaf, 2-middle, 3-spine).

[0050] ■The second digit indicates the subset to which the switch belongs within this layer (Subset 0 to Subset 3).

[0051] ■ The third number is the index of the individual switch within the subset (switch 0 to switch 3).

[0052] Any FT network (where Figure 2AThe network 64 of FIG. 6 is a non-limiting and illustrative example) can be viewed as including multiple bottom-to-top subtrees. Each bottom-to-top subtree starts from one of the leaf switches (i.e., originates from) and extends upward toward the backbone layer 56 via the middle layer 60. For example, the bottom-to-top subtree starting from leaf switch “110” includes switches “210”-“213” in the middle layer 60, and all backbone switches in the backbone layer 56. Similarly, the bottom-to-top subtree starting from leaf switch “123” includes switches “220”-“223” in the middle layer 60, and all backbone switches in the backbone layer 56. The path from one leaf switch (“source switch”) to another leaf switch (“destination switch”) first traverses upward in the bottom-to-top subtree of the source switch, and then traverses downward in the bottom-to-top subtree of the destination switch. Some paths reach the backbone layer 56 (e.g., the path from switch “100” to switch “122”). Other paths (eg, the path from switch “ 100 ” to switch “ 103 ”) only reach the middle layer 60 .

[0053] like Figure 2A As shown in the example of , the FT network has considerable redundancy because host pairs 48 can communicate via multiple alternative paths (each path traversing multiple links 44 and one or more switches 40). In addition to redundancy, the multiple alternative paths also increase the available bandwidth because traffic can be distributed in parallel among them. A failed link in the network reduces the level of redundancy and also reduces the available bandwidth of at least part of the network.

[0054] Over time, some links 44 may fail. As mentioned above, the actual impact of failed links on network performance depends not only on their number, but also on how the failed links are distributed in the FT topology. For example, failed links that are evenly distributed across the network will generally have less impact than failed links that are concentrated in a particular area. Figure 2A In the example of FIG. 6 , network 64 includes three failed links (marked in bold and numbered 72A-72C in the figure).

[0055] In some embodiments, the processor 36 of the NMS 20 receives topology information indicating the current topology of the network (e.g., which links 44 are operational and which links 44 are faulty) from the network 28. Based on the topology information, the processor 36 estimates and reports the maximum available bandwidth between each pair of leaf switches (the switches 40 in the leaf layer 52, which are denoted as "1XX") using a method described in detail below.

[0056] Figure 2B It is schematically shown Figure 2A Figure 64 of different presentations of the FT network. Figure 2B The network topology and Figure 2A The only difference between the two figures is the order in which the switches of the backbone layer 56 (denoted as "3XX" switches) are drawn. Figure 2B The presentation of shows more clearly which leaf switch pairs are primarily affected by the three failed links 72A-72C.

[0057] like Figure 2B As shown, the faulty link 72A (between switch "100" and switch "200") and the faulty link 72B (between switch "110" and switch "210") disconnect the leaf switch "100" and the leaf switch "110" from the entire bipartite graph, which connects switch "200" and switch "110" through the backbone layer 56. Switch "100" and switch "110" remain connected to each other through three other complete bipartite graphs.

[0058] On the other hand, leaf switch "120" is disconnected from the additional bipartite graph between layer 60 and layer 56. This is due to failed link 72C. Therefore, leaf switch "100" and leaf switch "120" remain connected to each other only through two bipartite graphs.

[0059] Figure 3 According to an embodiment of the present invention Figure 2A and Figure 2B 4. A table of the maximum available bandwidth between leaf switch pairs in the FT network 64 of FIG. The maximum available bandwidth is given in units of the full bandwidth of a single link 44, assuming, as a non-limiting example, that each link 44 has equal bandwidth.

[0060] As shown in the figure, the leaf switches that are not affected by the failed link are connected to each other through a total of four parallel links, so the maximum available bandwidth is 4. Some leaf switch pairs (e.g., the "100" and "110" pair discussed above) have a maximum available bandwidth of 3. Other leaf switch pairs (e.g., the "100" and "120" pair discussed above) have a maximum available bandwidth of only 2.

[0061] In various embodiments, processor 36 may use various methods to estimate the maximum available bandwidth between leaf switches in the FT network. The following simplified description outlines one possible solution at a high level for a given leaf switch pair. Figure 4 The detailed flow of an efficient method is described, which includes (i) a preparation phase performed in advance for all leaf switch pairs, and (ii) an estimation phase performed separately for each pair of leaf switches.

[0062] For a given pair of leaf switches (referred to as a "first leaf switch" and a "second leaf switch"), the processor 36 identifies a first bottom-to-top subtree and a second bottom-to-top subtree to which the first leaf switch and the second leaf switch belong, respectively. Using the two bottom-to-top subtrees (which are actually corresponding sets of parents of the two leaf switches), the processor 36 identifies the lowest common layer of the two leaf switches. As described above, the lowest common layer is the lowest layer of the FT network that contains at least one switch 40 that is a parent of the two leaf switches.

[0063] The processor 36 then scans the switches in the lowest common layer that are common parents of the two leaf switches. For each common parent, the processor 36 determines a corresponding "path count". The path count for a switch indicates the number of paths that (i) pass through the switch, (ii) reach the first leaf switch via a first bottom-to-top subtree, and (iii) reach the second leaf switch via a second bottom-to-top subtree. Typically, the processor 36 counts the path count at a port (link) granularity so that multiple links connected in parallel between two switches are counted as multiple paths.

[0064] The processor 36 then derives the available bandwidth between the first leaf switch and the second leaf switch from the path count. The path count of a switch is equal to the maximum available bandwidth between the first leaf switch and the second leaf switch via the switch. The sum of the path counts on all common parents of the two leaf switches in the lowest common layer indicates the total maximum available bandwidth between the two leaf switches. In one embodiment, the processor 36 calculates the sum and reports it as the maximum available bandwidth (in units of the bandwidth of the individual links 44).

[0065] In some embodiments, processor 36 calculates the leaf-to-leaf bandwidth (e.g., in Figure 3 The leaf-to-leaf bandwidth is reported to operator 24 in the form of a table or any other suitable form. Operator 24 may take any suitable action based on the calculated leaf-to-leaf bandwidth, such as allocating computing tasks to hosts 48 in a manner that best utilizes the available leaf-to-leaf bandwidth.

[0066] The above method flow is a highly simplified flow, depicted purely for conceptual clarity. In alternative embodiments, the processor 36 may use any other suitable flow to calculate the maximum available bandwidth between leaf switch pairs. For example, some calculations (e.g., identification of the parent switch) may be performed jointly for all leaf switches in advance in a computationally efficient manner. This type of process is described below.

[0067] Efficient leaf-to-leaf bandwidth estimation using "coloring"

[0068] The fastest known algorithm for finding the maximum flow between pairs of nodes in a graph is the well-known "Dinic algorithm." For example, Dinic described the Dinic algorithm in "An Algorithm for Solving the Maximum Flow Problem in Networks with Power Estimation," Doklady Akademii Nauk, SSSR, 1970, Volume 11, Pages 1277-1280.

[0069] The asymptotic computational complexity of Dinic's algorithm is O(V 2 E), where E and V represent the number of edges and vertices in the graph, respectively. Applying Dinic’s algorithm to all leaf switch pairs in the FT network 28 has O(V 4 E 2 ), the number of leaf switches is approximately ~V. This computational complexity is prohibitive in large networks.

[0070] In some embodiments of the present invention, the processor 36 significantly reduces the computational complexity of leaf-to-leaf bandwidth estimation by using an efficient process that identifies the set of common parent switches for each pair of leaf switches. In these embodiments, the processor 36 first performs a preliminary process (referred to herein as "coloring") that assigns indices ("colors") to the various switches of the FT network. The "coloring" process is performed for the entire FT network, not for any particular pair of leaf switches. The processor 36 then uses the assigned indices to find the set of common parents for any desired pair of leaf switches, and derives the leaf-to-leaf bandwidth therefrom.

[0071] Compared with Dinic's algorithm (whose complexity is O(V 4 E 2 ) level), the computational complexity of the disclosed coloring-based scheme is O(log(V)V 2 ) magnitude.

[0072] Figure 4 FIG. 4 is a flowchart schematically illustrating a method for estimating a maximum available bandwidth between leaf switch pairs using coloring according to an embodiment of the present invention.

[0073] The method begins with the processor 36 finding and storing, for each leaf switch, a list of parent switches of the leaf switches toward a single selected spine switch. For a given leaf switch, the processor 36 typically finds the parent switch by proceeding from the spine switch down to the leaf switch. Then by comparing the lists of parents of two leaf switches, the processor 36 can determine (i) the set of common parents of the two leaf switches and (ii) the lowest common layer of the two leaf switches. Figure 6 and Figure 7 Example pseudo code for collecting a list of parents of a leaf switch is given.

[0074] In the shading stage 94, the processor 36 starts at the level immediately above the leaf level (the lowest intermediate level) and distributes indexes upward to the switches in the FT network 28. Figure 5 An illustrative example of this process is depicted. These indexes then assist in establishing a unique bottom-to-top subtree to which each leaf switch belongs. In one embodiment, the processor 36 assigns indexes as follows:

[0075] 1. Select a leaf switch.

[0076] 2. Set the current layer to the lowest middle layer (the layer immediately above the leaf layer).

[0077] 3. Scan the parents of the leaf switches in the current layer and assign an index to any switch that has not yet been assigned an index. The index may include, for example, an integer. The assigned index is unique at least within the current layer. If an index has already been assigned to a switch, skip the switch.

[0078] 4. After scanning the entire current layer, increment the current layer to the next higher layer and repeat step 3 above until the backbone layer (including the backbone layer) is reached.

[0079] 5. After completing steps 2-4 above for a given leaf switch, continue with another leaf switch until indexes are assigned to the parents of all leaf switches.

[0080] Example pseudocode for assigning indices is below Figure 8-Figure 10 In some cases, one or more switches may still not be assigned an index and / or may be assigned conflicting indexes, for example, due to a missing (failed) link. In one embodiment, after completing steps 1-5 above, processor 36 performs a consolidation process that resolves such missing and / or conflicting assignments. The following Fig.11 A sample pseudo code is given to implement the integration process.

[0081] In link counting phase 98, for each leaf switch, for each parent level, processor 36 sums the total number of links connecting the leaf switch to parent switches belonging to the same bottom-to-top subtree (identified by their assigned indexes). Processor 36 performs this counting process for each parent level. Fig.12 Example pseudo code for executing stage 98 is given in .

[0082] Stages 90-98 of the above method constitute the preparation process. In the bandwidth calculation stage 102, the processor 36 uses the information collected in the preparation process to calculate the available bandwidth between the leaf switch pairs. For a given leaf switch pair (denoted as "first leaf switch" and "second leaf switch"), the processor 36 performs the following operations:

[0083] 1. Identify the lowest common layer of the two leaf switches (by comparing the lists of parents of the two leaf switches collected in stage 90).

[0084] 2. Select the common parent of the two leaf switches in the lowest common layer.

[0085] 3. Determine (i) the number of links via which the first leaf switch is connected to the common parent switch, and (ii) the number of links via which the second leaf switch is connected to the common parent switch. Take the minimum of the two link numbers. This minimum value (also called the "path count") gives the maximum available bandwidth (in units of the bandwidth of a single link) between the two leaf switches via the common parent switch being checked. The path count can be determined from the intersection of the two sets of indices ("colors") of the two leaf switches.

[0086] 4. Repeat steps 2-3 for all common parents of the two leaf switches in the lowest common layer. Sum the maximum available bandwidth of the common parents (calculated in step 3) on all common parents of the two leaf switches in the lowest common layer. This sum gives the total available bandwidth between the pair of leaf switches.

[0087] under Fig.13 Example pseudocode for finding the maximum available bandwidth between a pair of leaf switches is given.

[0088] Figure 5 is a diagram showing the processor 36 in accordance with an embodiment of the present invention. Figure 4 FIG. 9 is a diagram of the index assignment ("coloring") process performed at stage 94 of FIG. 1 . In this example, the network is a four-layer FT network, which includes a leaf layer 52, two intermediate layers shown as 60A and 60B, and a spine layer 56. Each switch 40 in layer 60A and above is assigned a corresponding index ("color"). The index is depicted in a circle next to its corresponding switch.

[0089] As explained above, processor 36 begins assigning indexes to switches in the layer immediately above the leaf layer. In this layer, processor 36 assigns a different index to each switch. In this example, the layer immediately above the leaf layer is layer 60A, and the switches in this layer are assigned indexes 1 to 16.

[0090] Then, processor 36 starts the assignment of the bottom-to-top subtree from switch "2000". This bottom-to-top subtree includes switch "3000" and switch "3010" in layer 60B, and switch "4000" and switch "4010" in layer 56. Processor 36 assigns index "1" to these switches, which is unique within the corresponding layer.

[0091] Next, processor 36 continues to the bottom-to-top subtree of switch "2001". The bottom-to-top subtree includes switch "3020" and switch "3030" in layer 60B, and switch "4020" and switch "4030" in layer 56. These switches have not yet been assigned an index, so processor 36 assigns them an index of "2", which is unique within the corresponding layer.

[0092] Processor 36 repeats the process in a similar manner for the bottom-to-top subtree of switch “2010” (which results in unique index “3” being assigned to switches “3000”, “3011”, “4001”, and “4011”), and for the bottom-to-top subtree of switch “2011” (which results in unique index “4” being assigned to switches “3021”, “3031”, “4021”, and “4031”).

[0093] When processing the bottom-to-top subtree of switch "2020", processor 36 finds that all switches in the subtree ("3000", "3010", "4000", and "4010") have already been assigned indexes. Therefore, processor 36 skips these switches. For the same reason, the same skipping occurs when processing the bottom-to-top subtree of switches "2021", "2030", and "2031". None of the parents of these switches warrants the assignment of a new index.

[0094] When processing the bottom-to-top subtree of switch "2100", processor 36 again encounters switches without an index (switches "3100", "3110") and assigns them a unique index of "5". In layer 56, the bottom-to-top subtree of switch "2100" has only switches that already have an index.

[0095] The process continues in the same manner until all switches in layer 60A and above are assigned the indexes shown in the figure.

[0096] Example pseudocode

[0097] Figure 6-Figure 13 Example pseudocode for performing various stages of the disclosed leaf-to-leaf bandwidth calculation process according to an embodiment of the present invention is provided. For example, code based on the pseudocode may be provided by Figure 1The pseudo-code is provided as a non-limiting example; any other suitable code flow may be used in alternative embodiments.

[0098] Figure 6 An example pseudo code of a routine for returning the parent switch of each leaf switch in the FT network according to an embodiment of the present invention is shown. This task corresponds to Figure 4 Stage 90.

[0099] Figure 7 Shows Figure 6 Example pseudocode for the depth-first search (DFS) routine used in the pseudocode of FIG. The DFS routine scans the FT network starting from the specified backbone (top-level) switch in DFS order.

[0100] Figure 8-Figure 11 is a diagram illustrating example pseudo code for assigning indices ("coloring") to switches according to an embodiment of the present invention.

[0101] The pseudo-code initially collects the pre-assigned indexes (colors) from all parents of each leaf switch and records them for each layer of the FT network. The pseudo-code then consolidates the collected set of indices for each layer. If a conflict is found, i.e., if different index values ​​are assigned to the parents of a given switch in a layer, the pseudo-code selects the lowest index value among the different index values. Upon finding any different or missing assignments to switches in a layer, the pseudo-code reassigns the index to a single (e.g., minimum or new) value.

[0102] The main routine of the coloring pseudocode is as follows Figure 8 Depicted. Fig. 9 A DFS routine is shown for collecting the pre-assigned color index for each layer - DFS accumulates the assigned colors for each visited switch of a given switch and above into a data structure shown as "Acc", which holds the assigned color set for each layer. The operator "Acc[l] = |" means adding the obtained color to the accumulated color set for layer l. Fig.10 A routine is shown that consolidates the acquired pre-existing parent colors, assigns dn-up colors when needed, and determines if the color needs to be set (or reset). The routine returns the final single color required for each parent of the leaf switch, as well as an indication of whether a color setting is required. Fig.11 The integration process is shown.

[0103] Fig.12 is a diagram illustrating an example pseudo code for counting the number of paths between leaf switch pairs according to an embodiment of the present invention. Fig.12The pseudo code of counts the number of leaf switch ports connected to each dn-up color value. The pseudo code loops over all ports of the leaf switch and creates a per-layer dictionary "T" (also called a map) containing color values ​​and their counts.

[0104] Fig.13 is a diagram showing an example pseudocode for calculating the maximum available bandwidth between leaf switch pairs according to an embodiment of the present invention. The pseudocode calculates the number of paths ("flows") between leaf switch pairs represented as t1 and t2. The pseudocode first uses "parents[t1]" and "parents[t2]" to obtain the common parent layer lc=h-|parents[t1]&parents[t2]|+1. The pseudocode then looks for a mapping that gives the number of ports of each color ("T1=T[t1][lc]", "T2=T[t2][lc]"). The total number of flows between t1 and t2 is given by the sum of the minimum values ​​of each of the same colors in t1 and t2.

[0105] Although the embodiments described herein mainly address the maximum flow calculation of all leaf switch pairs in a fat tree, the methods and systems described herein may also be used for other applications, such as load balancing in fat tree routing.

[0106] Therefore, it should be understood that the above embodiments are cited only as examples, and the present invention is not limited to the contents specifically shown and described above. On the contrary, the scope of the present invention includes the combination and sub-combination of the various features described above, as well as changes and modifications that those skilled in the art would think of after reading the above description and are not disclosed in the prior art. The documents incorporated by reference into this patent application should be considered as an integral part of this application, unless the definition of any term in these incorporated documents conflicts with the definition explicitly or implicitly made in this specification, otherwise only the definition in this specification should be considered.

Claims

1. A system for bandwidth estimation, the system comprising: An interface for communicating with a fat-tree FT network, the FT network comprising: a plurality of switches including (i) a plurality of leaf switches belonging to a bottom layer, (ii) a plurality of spine switches belonging to a top layer, and (iii) a plurality of intermediate switches belonging to one or more intermediate layers between the top layer and the bottom layer; and a plurality of links connected between selected switches among the plurality of switches; and a processor configured to: Receiving topology information indicating a current topology of the FT network from the FT network via the interface, and estimating an available bandwidth between the first leaf switch and the second leaf switch based on the topology information in the following manner: From a plurality of bottom-to-top subtrees of the FT network, identifying a first bottom-to-top subtree to which the first leaf switch belongs and a second bottom-to-top subtree to which the second leaf switch belongs, each bottom-to-top subtree extending from a leaf switch to the top layer; determining a path count for at least some switches in the middle layer, wherein the path count for a switch indicates a number of paths that (i) pass through the switch, (ii) reach the first leaf switch via the first bottom-to-top subtree, and (iii) reach the second leaf switch via the second bottom-to-top subtree; and The available bandwidth between the first leaf switch and the second leaf switch is estimated based on the path count.

2. The system of claim 1 , wherein the processor is configured to identify a lowest common layer and determine the path counts of switches in the lowest common layer, the lowest common layer being defined as a lowest intermediate layer of the FT network including a joint parent switch of the first leaf switch and the second leaf switch.

3. The system of claim 2, wherein the processor is configured to: Assigning indexes to the switches in the FT network by scanning the plurality of leaf switches in sequence, and for each leaf switch in the sequence, traversing the bottom-to-top subtree to which the leaf switch belongs from the leaf switch to the top layer, and assigning an index to each traversed switch that has not been assigned any index, the index being specific to the leaf switch at least within the layer to which the switch belongs; defining (i) a first group comprising an index assigned to a parent of the first leaf switch in the lowest common layer, and (ii) a second group comprising an index assigned to a parent of the second leaf switch in the lowest common layer; and For each switch in the lowest common layer, the path count is determined based on an intersection between the first set of indices and the second set of indices.

4. The system of claim 3, wherein after completing the sequence, the processor is configured to identify switches that are not assigned any index or are assigned conflicting indexes, and to assign a non-conflicting consolidated index to the identified switches.

5. The system of claim 1, wherein the processor is configured to determine a total path count between the first leaf switch and the second leaf switch by: identifying a set of switches in a given layer that are common parents of the first leaf switch and the second leaf switch; For each common parent in the set, finding a minimum between (i) a first number of ports connecting the common parent to the first leaf switch and (ii) a second number of ports connecting the common parent to the second leaf switch; and The minimum values ​​over the set of common parents are summed.

6. The system of claim 1, wherein the topology information indicates a failure in one or more of the plurality of links.

7. The system of claim 6, wherein in estimating the available bandwidth, the processor only considers failures occurring in links connecting the leaf switch to an intermediate layer immediately above the bottom layer.

8. A method for bandwidth estimation, the method comprising: receiving topology information indicating a current topology of a fat-tree FT network, the FT network comprising a plurality of switches and a plurality of links connected between selected switches among the plurality of switches, the plurality of switches comprising (i) a plurality of leaf switches belonging to a bottom layer, (ii) a plurality of spine switches belonging to a top layer, and (iii) a plurality of intermediate switches belonging to one or more intermediate layers between the top layer and the bottom layer; and Based on the topology information, the available bandwidth between the first leaf switch and the second leaf switch is estimated in the following manner: From a plurality of bottom-to-top subtrees of the FT network, identifying a first bottom-to-top subtree to which the first leaf switch belongs and a second bottom-to-top subtree to which the second leaf switch belongs, each bottom-to-top subtree extending from a leaf switch to the top layer; determining a path count for at least some switches in the middle layer, wherein the path count for a switch indicates a number of paths that (i) pass through the switch, (ii) reach the first leaf switch via the first bottom-to-top subtree, and (iii) reach the second leaf switch via the second bottom-to-top subtree; and The available bandwidth between the first leaf switch and the second leaf switch is estimated based on the path count.

9. The method of claim 8, wherein determining the path count comprises: A lowest common layer is identified and the path counts of switches in the lowest common layer are determined, the lowest common layer being defined as a lowest intermediate layer of the FT network including a joint parent switch of the first leaf switch and the second leaf switch.

10. The method of claim 9, wherein determining the path count comprises: Assigning indexes to the switches in the FT network by scanning the leaf switches in sequence, and for each leaf switch in the sequence, traversing the bottom-to-top subtree to which the leaf switch belongs from the leaf switch to the top layer, and assigning an index to each traversed switch that has not been assigned any index, the index being specific to the leaf switch at least within the layer to which the switch belongs; defining (i) a first group comprising an index assigned to a parent of the first leaf switch in the lowest common layer, and (ii) a second group comprising an index assigned to a parent of the second leaf switch in the lowest common layer; as well as For each switch in the lowest common layer, the path count is determined based on an intersection between the first set of indices and the second set of indices.

11. The method according to claim 10, further comprising: After the sequence is completed, switches that are not assigned any index or are assigned conflicting indexes are identified, and the identified switches are assigned a non-conflicting consolidated index.

12. The method of claim 8, wherein determining the path count comprises determining a total path count between the first leaf switch and the second leaf switch by: identifying a set of switches in a given layer that are common parents of the first leaf switch and the second leaf switch; For each common parent in the set, finding a minimum between (i) a first number of ports connecting the common parent to the first leaf switch and (ii) a second number of ports connecting the common parent to the second leaf switch; and The minimum values ​​over the set of common parents are summed.

13. The method of claim 8, wherein the topology information indicates a failure in one or more of the plurality of links.

14. The method of claim 13, wherein estimating the available bandwidth comprises: Only failures occurring in the links connecting the leaf switch to the middle tier immediately above the bottom tier are considered.

15. A computer software product, the product comprising a tangible, non-transitory computer-readable medium having program instructions stored therein, the instructions, when read by a processor, causing the processor to: receiving topology information indicating a current topology of a fat-tree FT network, the FT network comprising a plurality of switches and a plurality of links connected between selected switches among the plurality of switches, the plurality of switches comprising (i) a plurality of leaf switches belonging to a bottom layer, (ii) a plurality of spine switches belonging to a top layer, and (iii) a plurality of intermediate switches belonging to one or more intermediate layers between the top layer and the bottom layer; and Based on the topology information, the available bandwidth between the first leaf switch and the second leaf switch is estimated in the following manner: From a plurality of bottom-to-top subtrees of the FT network, identifying a first bottom-to-top subtree to which the first leaf switch belongs and a second bottom-to-top subtree to which the second leaf switch belongs, each bottom-to-top subtree extending from a leaf switch to the top layer; determining a path count for at least some switches in the middle layer, wherein the path count for a switch indicates a number of paths that (i) pass through the switch, (ii) reach the first leaf switch via the first bottom-to-top subtree, and (iii) reach the second leaf switch via the second bottom-to-top subtree; and The available bandwidth between the first leaf switch and the second leaf switch is estimated based on the path count.

16. The product of claim 15, wherein the instructions cause the processor to identify a lowest common layer and determine the path counts of switches in the lowest common layer, the lowest common layer being defined as a lowest intermediate layer of the FT network including a joint parent switch of the first leaf switch and the second leaf switch.

17. The product of claim 16, wherein the instructions cause the processor to: Assigning indexes to the switches in the FT network by scanning the leaf switches in sequence, and for each leaf switch in the sequence, traversing the bottom-to-top subtree to which the leaf switch belongs from the leaf switch to the top layer, and assigning an index to each traversed switch that has not been assigned any index, the index being specific to the leaf switch at least within the layer to which the switch belongs; defining (i) a first group comprising an index assigned to a parent of the first leaf switch in the lowest common layer, and (ii) a second group comprising an index assigned to a parent of the second leaf switch in the lowest common layer; and For each switch in the lowest common layer, the path count is determined based on an intersection between the first set of indices and the second set of indices.

18. The article of claim 17, wherein the instructions cause the processor to: after completing the sequence, identify switches that are not assigned any index or are assigned conflicting indexes, and assign non-conflicting consolidated indexes to the identified switches.

19. The product of claim 15, wherein the instructions cause the processor to determine a total path count between the first leaf switch and the second leaf switch by: identifying a set of switches in a given layer that are common parents of the first leaf switch and the second leaf switch; For each common parent in the set, finding a minimum between (i) a first number of ports connecting the common parent to the first leaf switch and (ii) a second number of ports connecting the common parent to the second leaf switch; and The minimum values ​​over the set of common parents are summed.

20. The product of claim 15, wherein the topology information indicates a failure in one or more of the plurality of links.

Images