Large-scale networks with high port utilization

Abstract

This disclosure relates to large-scale networks with high port utilization. A data communication system includes multiple disjoint sets of switches, each set comprising multiple disjoint subsets of switches within that set. Local links interconnect the switches in each subset in a fully connected topology, such that no switch in any given subset is connected in a single hop to any switch in any other subset within the same set. Global links interconnect the sets of switches, each global link connecting a switch in one set to another switch in another set, such that each subset of switches in any given set is connected in a single hop to at least one subset in every other set of switches via at least one global link.

Description

Technical Field

[0001] This invention generally relates to data communication, and more specifically to high-speed, large-scale computer networks. Background Technology

[0002] High-speed computer networks, such as data center networks and high-performance computing (HPC) compute node clusters, include switches linked together in selected interconnect topologies. Such topologies include, for example, mesh, fat-tree (FT), and dragonfly (DF) topologies. The term "switch" is used broadly in the context of this specification and the claims to refer to various network switching nodes, including but not limited to bridges and routers.

[0003] For example, Kim et al. described a dragonfly topology in "Technology-Driven, High-Scalable Dragonfly Topology," Proceedings of the 2008 International Symposium on Computer Architecture (2008), pp. 77-88. U.S. Patent Application Publication 2010 / 0049942 describes a dragonfly processor interconnect network comprising: multiple processor nodes; multiple routers, each directly coupled to multiple end nodes, the routers coupled to each other and arranged in a group; and multiple groups of routers such that each group is connected to every other group via at least one direct connection.

[0004] As another example, U.S. Patent 9,699,067 describes a topology called "Dragonfly Plus," the disclosure of which is incorporated herein by reference. In this topology, the communication network comprises multiple nodes arranged in groups such that the nodes in each group are interconnected in a bipartite topology, and the groups are interconnected in a mesh topology. The nodes are configured to transmit traffic between the source host and their respective destination host by routing packets between nodes on a path that does not traverse any intermediate hosts other than the source and destination hosts.

[0005] Based on graph theory principles, "extender" network topologies have been proposed as a more optimized alternative to traditional data center networks. For example, Valadarsky et al. described such topologies in their paper "Xpander: Towards Optimal-Performance Datacenters," published at CoNEXT'16 (December 2016, Irvine, California). Through theoretical calculations and simulations, the authors demonstrated that Xpander achieves "near-optimal performance" in terms of throughput, bandwidth guarantees, robustness to traffic variations, resilience to failures, incremental scalability, and path length. Summary of the Invention

[0006] The embodiments of the present invention described below provide improved systems and methods for data communication.

[0007] Therefore, according to one embodiment of the present invention, a data communication system is provided, comprising a plurality of non-overlapping sets of switches, each set comprising a plurality of non-overlapping subsets of switches within that set. Local links interconnect the switches in each subset in a fully connected topology, such that all switches in any given subset of any given set of switches are connected via local links to all other switches in that given subset, while no switch in any given subset is connected in a single hop to any switch in any other subset within the given set via a global link. Global links interconnect the sets of switches. Each global link connects a corresponding first switch in one set of switches to a corresponding second switch in another set, such that each subset in any given set of switches is connected in a single hop to at least one subset of a subset of every other set of switches within the set via at least one global link.

[0008] In one embodiment, in each subset of switches, a first switch in the subset is connected to a first switch in a first set of switches in another set via a first global link, and a second switch in the subset is connected to a second switch in a second set of switches in another set via a second global link.

[0009] Additionally or alternatively, different subsets of any given set are connected via corresponding global links in a single hop to different corresponding subsets of other sets of switches.

[0010] Further, or alternatively, at least some subsets in any given set of switches are not connected in a single hop to all subsets in every other set of switches via a global link. In one embodiment, each subset in each set of switches is connected in a single hop to a single corresponding subset in every other set of switches via a corresponding global link.

[0011] In another embodiment, all subsets in each switch set are connected in a single hop to all subsets in every other switch set within the switch set via corresponding global links.

[0012] In some embodiments, the system includes a routing manager configured to define paths for transmitting packets between switches in the system via local and global links, wherein these paths are limited to a predefined maximum number of hops. In the disclosed embodiments, the switches are configured to transmit packets on paths using a number of virtual channels equal to the predefined maximum number of hops, while transitioning at each hop of each path via the virtual channels in a predefined sequence.

[0013] According to one embodiment of the present invention, a communication method is also provided, comprising partitioning switches in a network into multiple sets of non-overlapping switches, each set comprising multiple non-overlapping subsets of switches within the set. Switches in each subset are interconnected using local links in a fully connected topology, such that all switches in any given subset of any given set of switches are connected via local links to all other switches in the given subset, while none of the switches in the given subset are connected in a single hop to any switch in any other subset within the given set. The sets of switches are interconnected using global links, such that each global link connects a corresponding first switch in one set of the set to a corresponding second switch in another set of the set, and such that each subset of switches in any given set of switches in the set is connected in a single hop to at least one subset of every other set of switches in the set via at least one global link.

[0014] According to one embodiment of the present invention, a data communication system is further provided, comprising a plurality of non-overlapping sets of switches, each set comprising a plurality of non-overlapping subsets of switches within that set. Local links interconnect the switches in each subset in a fully connected topology, wherein no switch in any given subset is connected in a single hop to any switch in any other subset within the same set via a global link. Global links interconnect the sets of switches. Each global link connects a switch in one set to another switch in another set, such that each subset in any given set of switches is connected in a single hop to at least one subset in every other set of switches via at least one global link.

[0015] The invention will be more fully understood through the following detailed description of embodiments, taken in conjunction with the accompanying drawings. Attached Figure Description

[0016] Figure 1 This is a block diagram schematically illustrating a data communication system according to an embodiment of the present invention;

[0017] Figure 2 It is shown schematically. Figure 1 A detailed block diagram of the system's components; and

[0018] Figure 3 This is a flowchart illustrating a method for routing packets in a communication network according to an embodiment of the present invention. Detailed Implementation

[0019] The cost-effectiveness of a network topology can be measured by its "port utilization," which is defined as follows: In other words, port utilization specifies the amount of cable required to connect a given number of hosts. Since cabling is a major cost factor in high-speed networks, it is desirable to maximize port utilization while still meeting performance requirements for low latency and high reliability. Dragonfly topologies are popular in data center networks and HPC clusters because they offer high connectivity and ease of routing between network nodes. However, compared to newer topologies, such as the "Xpander" topology mentioned above, Dragonfly networks have lower port utilization.

[0020] The embodiments of the invention described herein provide a novel network topology that offers low communication latency and ease of routing comparable to Dragonfly networks, while achieving higher port utilization. This embodiment is based on a fully connected local subset of switches in the network, as in Dragonfly networks, but makes the global links between these subsets more sparse and optimized.

[0021] In the disclosed embodiments, the switches in the data communication system are divided into multiple disjoint sets, and the switches in each such set are further divided into multiple disjoint subsets. Within each subset, local links interconnect the switches in a fully connected topology, meaning that all switches in any given subset are connected in a single-hop manner to all other switches in that subset. However, within any given set, no switch in any given subset is connected in a single hop to any switch in any other subset within that set.

[0022] Different sets of switches are interconnected via global links, meaning each global link connects a switch in one set to a switch in another set. Global links are arranged such that each subset of any given set of switches is connected in a single hop to at least one subset of every other set of switches via at least one global link. The number and connectivity of global links are chosen to optimize port utilization while meeting performance objectives such as low latency and high fault tolerance.

[0023] As used in the context of this specification and the claims, the term "hop" refers to a single link between a pair of network devices, through which a packet traverses its path across the network. In other words, if a packet traverses N nodes (including the source and destination nodes) on its path across the network, it will cover N-1 hops. In some embodiments, to achieve high port utilization, at least some subsets of a given set of switches are connected in a single hop to only certain subsets within other sets of switches via a global link, rather than connecting to all subsets. However, to maintain low latency and ease of routing, a given subset of a given set of switches can be connected in a single hop to multiple subsets of each other set via a global link. In one embodiment, each subset of switches is connected in a single hop to one of the subsets in each other set of switches via a global link.

[0024] Now for reference Figure 1 and Figure 2 The diagram schematically illustrates a data communication system 20 according to an embodiment of the present invention. Figure 1 It is a block diagram that represents the system overview. Figure 2 It is a block diagram that schematically represents the details of the system components.

[0025] System 20 includes a network of switches 30, which are divided into multiple disjoint sets 22, 24, 26, and 28. Each of these sets is further divided into multiple disjoint subsets 32, 34, and 36. While network 20 in this example includes four sets of switches, each set containing three subsets of three switches, the principles of this embodiment can be similarly applied to larger or smaller networks, where each subset has more or fewer switches (but each subset has at least two switches), and each set contains more or fewer subsets. Furthermore, despite symmetrical configurations, for example… Figure 1 As shown, where all sets and subsets are of equal size, it is easier to implement, but in alternative implementations, some sets and / or subsets can have different sizes.

[0026] like Figure 2 As shown, each switch 30 includes multiple ports 38 interconnected via switching logic 50. Each port 38 typically includes hardware circuitry, including physical layer (PHY) and media access control (MAC) interface circuitry. Each port 38 connects to either a local link 40 or a global link 44. Links 40 and 44 typically include electronic or optical communication cables and / or circuit traces, although they may alternatively include wireless links. Some local links 40 connect switch 30 to network end nodes, such as host 42 (in... Figure 1(Illustrated as H1, H2, H3, and H4 in the illustration). Switching logic 50 receives incoming data packets through port 38 and forwards the packets through other ports 38 according to logical instructions, such as those stored in forwarding table 52. (When switch 30 is configured as a router, forwarding table 52 includes a routing table and may contain multiple alternative forwarding paths for adaptive routing.) Although switching logic 50... Figure 2 While shown as a simple cross-connect, switch 30 can actually include more complex switching and routing logic as well as buffer memory.

[0027] In each subset 32, 34, 36 of any one of sets 22, 24, 26, and 28, local links 40 interconnect switches 30 in a fully connected topology. This means that all switches in any given subset of any given set of switches are connected in a single hop to all other switches in the same subset via local links 40. On the other hand, no switch in any given subset 32, 34, or 36 is connected in a single hop to any switch in any other subset within the same set. Thus, for example, in set 24, all switches 30 in subset 32 ​​are connected to other switches in subset 32 ​​via local links 40, but none of these switches are connected in a single hop to any switch in subsets 34 and 36 of set 24. Therefore, to route packets from a source in subset 32 ​​of set 24 to a destination in subset 34 of set 24, it is necessary to transfer packets from subset 32 ​​to a switch in a different set (22, 26, or 28), which then forwards the packets to the destination via one or more additional hops.

[0028] Global links 44 interconnect different sets 22, 24, 26, and 28 of switches 30. Each global link connects a corresponding switch in one set to a switch in another set. The global links 44 are arranged such that each of the subsets 32, 34, and 36 in any given set of switches is connected in a single hop to at least one subset of every other set of switches in the set via at least one global link 44. Using this lowest level of connectivity, each set 22, 24, 26, and 28 will be connected to every other set of switches via multiple global links 44, each global link 44 connecting to a different subset within the set. As defined above, this topology enables packets to be routed throughout system 20 with high port utilization.

[0029] In addition, Figure 1In the illustrated topology, in each subset 32, 34, 36 of any given set, one switch 40 in the subset is connected to a switch in one of the other sets via a first global link 44, and another switch in the same subset is connected to a switch in a different set of the other sets via a second global link 44. For example, the leftmost switch 30 in subset 32 ​​of set 22 can be connected to a switch in set 28, while the middle switch in that subset is connected to a switch in set 26, and the rightmost switch is connected to a switch in set 24. Additionally or alternatively, different subsets 32, 34, 36 of any given set 22, 24, 26, 28 of switch 30 are connected in a single hop to different corresponding subsets of each other set of switches via corresponding global links 44. These conditions provide a relatively uniform distribution of global links 44 among subsets 32, 34, 36 of all sets 22, 24, 26, 28, which approximates the nondeterministic distribution of links in Xpander networks.

[0030] For similar reasons, in some embodiments, at least some subsets of 32, 34, 36 of any given set 22, 24, 26, 28 of switch 30 are not connected in a single hop via a global link to all subsets in each other set of switches. In one embodiment, each subset in a given set of switches is connected in a single hop via a global link to a single corresponding subset in each other set of switches. Therefore, two or more hops may be required to transmit packets from a given subset in one set of switches to one or more subsets in another set of switches. However, the global link layout ensures that there are connections (including multi-hop connections) between all subsets of switches. As previously described, in each subset, switch 30 is fully connected by local link 40.

[0031] Or, such as Figure 1 As shown, a denser global link 44 layout can be used to provide tighter connections between switches 30, although still achieving higher port utilization than the Dragonfly network. In this configuration, each switch 30 in each subset 32, 34, 36 of each set 22, 24, 26, 28 is connected to a single switch in each of the other sets via global links 44. Furthermore, each switch in each subset of any given set is connected to a different subset of each set in the other sets. Thus, for example, in subset 32 ​​of set 22, the rightmost switch is connected to the switch in subset 32 ​​of set 24; the middle switch is connected to the switch in subset 34 of set 24; and the leftmost switch is connected to the switch in subset 36 of set 24. Other subsets are interconnected similarly. This topology helps balance network traffic load while supporting simple routing rules and relatively short path lengths.

[0032] Once the network topology is defined, the routing manager 54 ( Figure 2 This defines the packet transmission path between each source and destination in system 20 and loads the corresponding path information into the forwarding table 52. The route manager 54 typically includes a programmable processor 56, with memory 58 and an interface 60 for receiving routing information from and sending routing information to nodes in system 20. The functionality of the route manager 54 can be executed, for example, by a general-purpose computer under the control of suitable software. This software can be downloaded to the processor 56 electronically, for example, via a network. Additionally or alternatively, the software can be stored on a tangible, non-transitory computer-readable medium, such as an electronic, optical, or magnetic storage medium.

[0033] Figure 3 This is a flowchart illustrating a method for routing packets in a communication network according to an embodiment of the present invention. For the sake of specificity and clarity, the method is described with reference to the elements of system 20 and is performed by routing manager 54, as follows: Figure 1 and Figure 2 As shown. Alternatively, the principle of this method can be advantageously applied to other networks where switches are divided into sets and subsets as described above, such that within each subset, switches are fully interconnected via local links, without single-hop connections between different subsets within the same set, and global links interconnect subsets of different sets. This method aims to distribute traffic evenly across the links in the network while avoiding potential deadlocks, such as when buffers fill up due to a "credit loop".

[0034] In length setting step 70, route manager 54 sets the maximum path length for all flows to be routed in system 20. This maximum path length specifies the maximum number of hops allowed on any path between the source and destination nodes in system 20 and is set to a sufficiently large value to ensure that all nodes in the system can communicate with each other. Increasing the maximum path length allows traffic to be distributed more evenly across the network, but at the cost of greater latency and potentially increased memory requirements in switch 30 to accommodate a larger number of virtual channels (described below).

[0035] In the selection step, route manager 54 selects source and destination node pairs in system 20. For each pair, route manager 54 applies a routing algorithm in path identification step 74 to identify all paths with a path length less than or equal to the maximum allowed path length between the source and destination nodes. In path classification step 76, paths are classified by length, from the shortest path connecting the selected source and destination node pairs to the maximum allowed path length. In the coverage step 78, this routing process continues until all source and destination node pairs have been covered.

[0036] Based on the set of paths defined in step 76, route manager 54 builds routing tables for all switches 30 in routing step 80. The routing tables typically include multiple alternative paths to at least some destinations, enabling switches 30 to adaptively select a next hop for each packet, for example, based on network congestion reports. In this way, switches 30 will evenly distribute traffic among links 40 and 44 in system 20. When building the routing tables, route manager 54 prioritizes shorter paths while avoiding overloading any link. Routing information is stored by switches 30 in the corresponding forwarding table 52. Figure 2 )middle.

[0037] To avoid potential deadlocks, the route manager 54 instructs the switch 30 to use a different VC for each hop of each path in the routing table during the Virtual Channel (VC) allocation step 82. For this purpose, for example, packets are transmitted using a default VC on the first hop of each path, denoted as VC0. On each subsequent hop, the VC increments, meaning packets are transmitted using VC1 on the second hop, VC2 on the third, and so on. Therefore, on each hop along any given path, each VC deterministically transitions to the next VC in the sequence, and no VC transitions back to VC0. The number of virtual channels required in this scheme (including VC0) is equal to the maximum path length set in step 70. This deterministic unidirectional progression of virtual channels along each path ensures that credit loops do not occur on any path.

[0038] It should be understood that the above embodiments are cited as examples, and the present invention is not limited to the specific contents shown and described above. Rather, the scope of the present invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications that would occur to those skilled in the art after reading the above description and that are not disclosed in the prior art.

Claims

1. A data communication system, comprising: Multiple sets of non-overlapping switches, each set comprising multiple non-overlapping subsets of the switches in the set; Local links, which interconnect the switches in each of the subsets in a fully connected topology, such that all switches in any given subset of any given set of switches are connected to all other switches in the given subset via the local links, while none of the switches in the given subset are connected in a single hop to any switch in any other subset within the given set of switches; and Global links interconnecting the sets of switches, each global link connecting a corresponding first switch of one set in the set to a corresponding second switch of another set in the set, such that each subset of any given set of switches in the set is connected in a single hop to at least one subset of every other set of switches in the set via at least one of the global links.

2. The system of claim 1, wherein in each subset of the switches, a first switch in the subset is connected to a first switch in a first set of switches in another set via a first global link, and a second switch in the subset is connected to a second switch in a second set of switches in another set via a second global link.

3. The system of claim 1, wherein different subsets of any given set in the set are connected in a single hop to different corresponding subsets of the subsets of another set of switches via corresponding global links in the global links.

4. The system of claim 1, wherein at least some subsets of any given set of switches in the set of switches are not connected in a single hop to all subsets of every other set of switches in the set of switches via the global link.

5. The system of claim 4, wherein each subset in each set of switches is connected in a single hop to a single corresponding subset in each other set of switches via a corresponding global link in the global link.

6. The system of claim 1, wherein all subsets in each set of switches are connected in a single hop to all subsets in each other set of switches in the set of switches via a corresponding global link in the global link.

7. The system of claim 1, further comprising a routing manager configured to define paths for transmitting packets between switches in the system via the local links and the global links, wherein the paths are limited to a predefined maximum number of hops.

8. The system of claim 7, wherein the switch is configured to transmit the packets on the path using a plurality of virtual channels equal to the predefined maximum number of hops, while switching through the virtual channels in a predefined sequence at each hop of each path.

9. A communication method, comprising: The network switches are partitioned into multiple sets of non-overlapping switches, each set comprising multiple non-overlapping subsets of the switches in the set; The switches in each of the given subsets are interconnected using local links in a fully connected topology, such that all switches in any given subset of any given set of switches are connected to all other switches in the given subset via the local links, while none of the switches in the given subset are connected in a single hop to any switch in any other subset within the given set of switches; and The sets of switches are interconnected using global links such that each global link connects a corresponding first switch in one set of the set to a corresponding second switch in another set of the set, and such that each subset of any given set of switches in the set is connected in a single hop to at least one subset of that subset in every other set of switches in the set via at least one of the global links.

10. The method of claim 9, wherein interconnecting the set of switches comprises: The first switch in each subset of the switches is connected to the first switch in the first set of other sets via a first global link, and the second switch in the subset is connected to the second switch in the second set of other sets via a second global link.

11. The method of claim 9, wherein interconnecting the set of switches comprises: In a single hop, different subsets of any given set in the set are connected to different corresponding subsets of the subsets of other sets of switches via the corresponding global links in the global links.

12. The method of claim 9, wherein at least some subsets of any given set of switches in the switch set are not connected in a single hop to all subsets of every other set of switches in the switch set via the global link.

13. The method of claim 12, wherein interconnecting the set of switches comprises: Each subset of each set of switches is connected in a single hop to a single corresponding subset of the subset of each other set of switches in the set of switches via the corresponding global link in the global link.

14. The method of claim 9, wherein interconnecting the set of switches comprises: In a single hop, all subsets of each set of switches are connected to all subsets of every other set of switches in the set of switches via the corresponding global links in the global links.

15. The method of claim 9, further comprising defining paths for transmitting packets between switches in the network via the local link and the global link, wherein the paths are limited to a predefined maximum number of hops.

16. The method of claim 15, further comprising transmitting the packets between the switches on the path using a plurality of virtual channels equal to a predefined maximum number of hops, while transitioning through the virtual channels in a predefined sequence at each hop of each path.

17. A data communication system, comprising: Multiple sets of non-overlapping switches, each set comprising multiple non-overlapping subsets of the switches in the set; Local links, which interconnect the switches in each of the subsets in a fully connected topology, wherein no switch in any given subset is connected in a single hop to any switch in any other subset within the same set; and Global links interconnecting the sets of switches, each global link connecting a switch in one set of the set to another switch in another set of the set, such that each subset of any given set of switches is connected in a single hop to at least one subset of every other set of switches via at least one global link.

18. The system of claim 17, wherein at least some subsets of any given set of switches in the set of switches are not connected in a single hop to all subsets of every other set of switches in the set of switches via the global link.

19. The system of claim 17, wherein all subsets in each of the switch sets are connected in a single hop to all subsets in each other switch set in the switch set via a corresponding global link in the global link.

20. The system of claim 17, further comprising a routing manager configured to define paths for transmitting packets between switches in the system via the local link and the global link, wherein the paths are limited to a predefined maximum number of hops.

Images