Systems and methods for supporting heterogeneous and asymmetric dual-rail fabric configurations in a high performance computing environment

Abstract

To provide a system and method for supporting heterogeneous and asymmetric dual-rail fabric configurations in a high performance computing environment.SOLUTION: In the system 2000, one or more computers each including one or more microprocessors include a plurality of hosts, each of the plurality of hosts including at least one dual port adapter, and further including a private fabric including two or more switches and a public fabric including a cloud fabric. At a host of the plurality of hosts, a workload can be provisioned, a placement policy can be assigned to the provisioned workload, and network traffic between peer nodes of the provisioned workload can be assigned to one or more of the private fabric and the public fabric according to the placement policy.SELECTED DRAWING: Figure 20

Description

[Technical Field]

[0001] Copyright Notice A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of this patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyrights whatsoever.

[0002] Priority claim This application is a continuation of application Ser. No. 16 / 547,329, filed on August 21, 2019, entitled "SYSTEM AND METHOD FOR SUPPORTING HETEROGENEOUS AND ASYMMETRIC DUAL RAIL FABRIC CONFIGURATIONS IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT." This application claims the benefit of priority to U.S. patent application Ser. No. 16 / 267,072, filed February 4, 2019, entitled "SYSTEM AND METHOD FOR USING INFINIBAND ROUTING ALGORITHMS FOR ETHERNET FABRICS IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT," which also claims the benefit of priority to U.S. patent application Ser. No. 16 / 261,362, filed January 29, 2019, entitled "SYSTEM AND METHOD FOR A MULTICAST SEND DUPLICATION INSTEAD OF REPLICATION IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT." This application claims the benefit of priority to U.S. patent application Ser. No. 16 / 547,332, filed August 21, 2019, entitled "SYSTEM AND METHOD FOR ON-DEMAND UNICAST FORWARDING IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT," which also claims the benefit of priority to U.S. patent application Ser. No. 16 / 547,335, filed August 21, 2019, entitled "SYSTEM AND METHOD FOR A SINGLE LOGICAL IP SUBNET ACROSS MULTIPLE INDEPENDENT LAYER 2 (L2) SUBNETS IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT." This application claims the benefit of priority to U.S. patent applications entitled "System and Method for a Single Logical IP Subnet Across Multiple Networks," each of which is incorporated herein by reference in its entirety.

[0003] This application is related to the following applications, each of which is incorporated by reference in its entirety: U.S. Provisional Patent Application No. 62 / 552,818, filed August 31, 2017; U.S. Provisional Patent Application No. 62 / 679,465, filed June 1, 2018; U.S. Provisional Patent Application No. 16 / 115,138, filed August 28, 2018; U.S. Provisional Patent Application No. 62 / 679,469, filed June 1, 2018; U.S. Provisional Patent Application No. 62 / 679,478, filed June 1, 2018; and U.S. Provisional Patent Application No. 62 / 679,491, filed June 1, 2018. [Background technology]

[0004] background As larger cloud computing architectures are deployed, the performance and management bottlenecks associated with traditional networking and storage become significant issues. There has been growing interest in using high performance interconnects such as InfiniBand (IB) and RoCE (RDMA (Remote Direct Memory Access) over Converged Ethernet) technologies as the basis for cloud computing fabrics. This is the general area that embodiments of the present teachings are intended to address. Summary of the Invention [Means for solving the problem]

[0005] overview: A system and method for supporting heterogeneous and asymmetric dual-rail fabric configurations in a high-performance computing environment may include a plurality of hosts in one or more computers, each host including one or more microprocessors, each host including at least one dual-port adapter, and a private fabric including two or more switches, and a public fabric including a cloud fabric. A workload may be provisioned on a host of the plurality of hosts. A placement policy may be assigned to the provisioned workload. Network traffic between peer nodes of the provisioned workload may then be allocated to one or more of the private fabric and the public fabric according to the placement policy.

[0006] A system and method for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high-performance computing environment. A method can provide a logical device in a computer including one or more microprocessors, the logical device being addressed by a Layer 3 address, the logical device including multiple network adapters, each of the network adapters including a physical port. The method can further provide a plurality of switches in the computer including one or more microprocessors. The method can arrange the multiple switches into multiple separate Layer 2 subnets. The method can provide a mapping table in the logical device.

[0007] Particular aspects of the invention are set out in the independent claims, with various optional embodiments set out in the dependent claims. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of an InfiniBand environment according to one embodiment. [Figure 2] FIG. 1 illustrates an example of a split cluster environment according to one embodiment. [Figure 3] FIG. 1 illustrates an example of a tree topology in a network environment according to one embodiment. [Figure 4] FIG. 1 illustrates an exemplary shared port architecture according to one embodiment. [Figure 5] FIG. 1 illustrates an exemplary vSwitch architecture according to one embodiment. [Figure 6] FIG. 2 illustrates an exemplary vPort architecture according to one embodiment. [Figure 7] FIG. 2 illustrates an exemplary vSwitch architecture with pre-populated LIDs according to one embodiment. [Figure 8] FIG. 2 illustrates an exemplary vSwitch architecture with dynamic LID allocation according to one embodiment. [Figure 9] FIG. 2 illustrates an exemplary vSwitch architecture with dynamic LID assignment and pre-populated LIDs to the vSwitch, according to one embodiment. [Figure 10] FIG. 1 illustrates an exemplary multi-subnet InfiniBand fabric according to one embodiment. [Figure 11] FIG. 2 illustrates an interconnection between two subnets in a high-performance computing environment, according to one embodiment. [Figure 12] FIG. 1 illustrates an interconnection between two subnets via a dual-port virtual router configuration in a high-performance computing environment, according to one embodiment. [Figure 13] FIG. 1 illustrates a flowchart of a method for supporting a dual-port virtual router in a high-performance computing environment, according to one embodiment. [Figure 14] 1 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment. [Figure 15] 1 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment. [Figure 16] 1 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment. [Figure 17] 1 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment. [Figure 18] 1 is a flowchart of a method for redundant independent networks in a high performance computing environment. [Figure 19] 1 illustrates a system for supporting rail number association for proper connectivity in a high performance computing environment, according to one embodiment. [Figure 20] 1 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment. [Figure 21] 1 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment. [Figure 22] 1 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment. [Figure 23] 1 is a flowchart of a method for determining a placement policy for a provisioned or provisioning workload in a system for supporting heterogeneous and asymmetric dual-rail fabric configurations. [Figure 24] 1 is a flowchart of a method for determining a fabric type selection for a provisioned or provisioning workload, according to one embodiment. [Figure 25] 1 is a flowchart of a method for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment. [Figure 26] 1 illustrates a system for multicast transmission replication instead of replication in a high performance computing environment, according to one embodiment. [Figure 27] 1 is a flowchart of a method for multicast transmission replication instead of replication in a high performance computing environment. [Figure 28] 1 illustrates a system for supporting link aggregation (LAG) according to one embodiment. [Figure 29] 1 illustrates a system for supporting multi-chassis link aggregation (MLAG), according to one embodiment. [Figure 30] 1 illustrates a system for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, according to one embodiment. [Figure 31] 1 is a flowchart of a method for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Detailed Description: The present teachings are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference numerals refer to like elements. References to "some" or "some" embodiments are not necessarily to the same embodiments, and such references mean at least one. While specific implementations are described, it is understood that these specific implementations are provided for illustrative purposes only. A person skilled in the art will recognize that other components and configurations can be used without departing from the scope and spirit of the claimed subject matter.

[0010] Common reference numbers may be used to denote like elements throughout the drawings and detailed description, and thus a reference number used in one drawing may or may not be referenced in the detailed description specific to that drawing if the element is described elsewhere.

[0011] According to one embodiment, a system and method for supporting heterogeneous and asymmetric dual-rail fabric configurations in a high performance computing environment is described herein.

[0012] According to one embodiment, a system and method for supporting a single logical IP subnet across multiple independent layer 2 subnets in a high performance computing environment is described herein.

[0013] In some embodiments, the following description of the present teachings will be directed to InfiniBand as an example of a high performance network. TM (IB) network. Throughout the following discussion, we will refer to InfiniBand TMThe following description may refer to the InfiniBand Trade Association Architecture Specification (variously referred to as the InfiniBand Specification, IB Specification, or Legacy IB Specification). Such references will be understood to refer to the InfiniBand Trade Association Architecture Specification, Volume 1, Version 1.3, published in March 2015 and available at http: / / www.inifinibanda.org, which is incorporated herein by reference in its entirety. It will be apparent to those skilled in the art that other types of high-performance networks may be used without limitation. The following description also uses a fat-tree topology as an example of a fabric topology. It will be apparent to those skilled in the art that other types of fabric topologies may be used without limitation.

[0014] In some other embodiments, the following description applies to RoCE (RDMA (Remote Direct Memory Access) RDMA over Converged Ethernet ( RoCE is a standard protocol that enables efficient RDMA data transfer over Ethernet networks, enabling transport offload and superior performance with hardware RDMA engine implementation. RoCE is a standard protocol specified in the InfiniBand Trade Association (IBTA) standard. RoCE is a UD It utilizes RDMA (User Datagram Protocol) encapsulation, allowing it to cross Layer 3 networks. RDMA is a key capability used natively by InfiniBand interconnect technology. Both InfiniBand and Ethernet RoCE share a common user API, but have different physical and link layers.

[0015] According to one embodiment, various portions of this specification include references to InfiniBand fabrics when describing various implementations, however, those skilled in the art will readily understand that the various embodiments described herein may also be implemented in RoCE fabrics.

[0016] To meet the demands of modern cloud computing (e.g., the exascale era), virtual machines must support Remote Direct Memory Access (RDMA). The ability to utilize low-overhead network communication paradigms such as RDMA Because RDMA bypasses the OS stack and communicates directly with the hardware, pass-through technologies such as single-root I / O virtualization (SR-IOV) network adapters can be used. In accordance with one embodiment, a virtual switch (vSwitch) SR-IOV architecture can be applied to a high-performance lossless interconnect network. Because network reconfiguration time is critical to making live migration a viable option, a scalable, topology-independent dynamic reconfiguration mechanism can be provided in addition to the network architecture.

[0017] According to one embodiment, a routing strategy for a virtualized environment using a vSwitch can be further provided, and an efficient routing algorithm can be provided for a network topology (e.g., a fat-tree topology). The dynamic reconfiguration mechanism can be further tuned to minimize the overhead imposed on the fat-tree.

[0018] According to one embodiment of the present teachings, virtualization can be beneficial for efficient resource utilization and flexible resource allocation in cloud computing. Live migration enables optimizing resource utilization by moving virtual machines (VMs) between physical servers in a manner that is transparent to applications. Thus, virtualization enables consolidation, on-demand provisioning of resources, and flexibility through live migration.

[0019] InfiniBand TM InfiniBand TM (IB) stands for InfiniBand TM Trade Association (InfiniBand TM It is an open-standard, lossless network technology developed by the IEEE 802.11b Trade Association. The technology is based on a serial point-to-point full-duplex interconnect that provides high-throughput and low-latency communications, especially targeted at high-performance computing (HPC) applications and data centers.

[0020] InfiniBand TM ·Architecture (InfiniBand Architecture: IBA) is 2 Supports layer topology division. At the lower layers, an IB network is called a subnet, and a subnet may contain a set of hosts interconnected using switches and point-to-point links. At higher levels, an IB fabric consists of one or more subnets that may be interconnected using routers.

[0021] Within a subnet, hosts may be connected using switches and point-to-point links. In addition, there may be one master management entity, a subnet manager (SM), that resides on a designated device in the subnet. The subnet manager is responsible for configuring, starting, and maintaining the IB subnet. In addition, the subnet manager (SM) may be responsible for performing routing table calculations in the IB fabric. Here, for example, routing in an IB network aims to provide fair load balancing between all source-destination pairs in the local subnet.

[0022] Through the subnet management interface, the subnet manager exchanges control packets called subnet management packets (SMP) with the subnet management agent (SMA). An agent exists on every IB subnet device. Using SMP, the subnet manager discovers the fabric and manages the end nodes and switches. You can configure and receive notifications from the SMA.

[0023] According to one embodiment, routing within a subnet in an IB network is performed using a linear forwarding table (LF) stored in the switch. The LFT can be based on the Host Channel Adapter (Host Channel Adapter) on the end nodes. The LFT is calculated by the SM according to the routing mechanism in use. HCA (Hardware Adapter) ports and switches are addressed using local identifiers (LIDs). Each entry in the LFT contains a destination LID (DLI). D) and an output port. Only one entry per LID in the table is supported. When a packet arrives at a switch, its output port is determined by looking up the DLID in the switch's forwarding table. Routing is deterministic because packets follow the same path in the network between a given source-destination pair (LID pair).

[0024] Generally, all other subnet managers except the master subnet manager operate in standby mode for fault tolerance. However, in the situation where the master subnet manager fails, a new master subnet manager is elected by the standby subnet managers. The master subnet manager also performs periodic sweeps of the subnet to detect any topology changes and updates the network accordingly. Reconfigure the network.

[0025] Additionally, hosts and switches within a subnet may be addressed using a local identifier (LID), and a single subnet may be limited to 49151 unicast LIDs. In addition to the LID, which is a local address valid within the subnet, each IB device may have a 64-bit global unique identifier (GUID). The GUID may be used to form a global identifier (GID), which is an IB Layer 3 (L3) address.

[0026] The SM may calculate routing tables (i.e., connections / routes between each pair of nodes in a subnet) at network initialization time. Additionally, whenever the topology changes, the routing tables may be updated to ensure connectivity and optimal performance. During normal operation, the SM may perform periodic light sweeps of the network to check for topology changes. If a change is discovered during a light sweep, Alternatively, if the SM receives a message (trap) signaling a network change, the SM may reconfigure the network according to the discovered change.

[0027] For example, the SM may reconfigure the network when the network topology changes, such as when a link goes down, a device is added, or a link is removed. The reconfiguration step may include a step performed during network initialization. Furthermore, the reconfiguration may have a local scope that is limited to the subnet where the network change occurred. Also, segmentation of a large fabric using routers may limit the reconfiguration scope.

[0028] An example of an InfiniBand fabric is shown in Figure 1, which illustrates an example InfiniBand environment 100 according to one embodiment. In the example shown in Figure 1, nodes A101-E105 communicate using an InfiniBand fabric 120 via respective host channel adapters 111-115. According to one embodiment, various nodes (e.g., nodes A101-E105) may be represented by various physical devices. According to one embodiment, various nodes (e.g., nodes A101-E105) may be represented by various virtual devices, such as virtual machines.

[0029] Partitioning in InfiniBand According to one embodiment, an IB network may support partitioning as a security mechanism for isolating logical groups of systems that share a network fabric. Each HCA port on a node in the fabric may be a member of one or more partitions. Partition membership is managed by a centralized partition manager, which may be part of the SM. The SM may organize the partition membership information for each port as a table of 16-bit partition keys (P_Key). The SM may also manage these Switch and router ports can be configured with a partition enforcement table that contains P_Key information associated with end nodes that send or receive data traffic through the port. Additionally, in the general case, the partition membership of a switch port may represent the collection of all memberships indirectly associated with LIDs routed through the port in the egress direction (towards the link).

[0030] According to one embodiment, a partition is a logical group of ports, and members of a group can only communicate with other members of the same logical group. Isolation can be enforced in host channel adapters (HCAs) and switches by filtering packets using partition membership information. Packets with invalid partitioning information can be dropped as soon as they reach the ingress port. In a partitioned IB system, partitions can be used to create tenant clusters. With partitions in place, nodes cannot communicate with other nodes that belong to different tenant clusters. In this way, the security of the system can be guaranteed even in the presence of faulty or malicious tenant nodes.

[0031] According to one embodiment, for communication between nodes, queue pairs (QP) and end-to-end contexts (EEC), with the exception of the management queue pair (QP0 and QP1), can be assigned to specific partitions. P_Key information can then be added to all transmitted IB transport packets. When a packet arrives at an HCA port or switch, its P_Key value can be checked against a table configured by the SM. If an invalid P_Key value is found, the packet is immediately discarded. In this way, communication is only allowed between ports that share a partition.

[0032] An example of an IB partition is shown in FIG. 2, which illustrates an example of a partitioned cluster environment, according to one embodiment. In the example shown in FIG. 2, nodes A 101-E 105 communicate via their respective host channel adapters 111-115 using an InfiniBand fabric 120. Nodes A-E are arranged into partitions: partition 1 130, partition 2 140, and partition 3 150. Partition 1 includes node A 101 and node D 104. Partition 2 includes node A 101, node B 102, and node C 103. Partition 3 includes node C 103 and node E 105. Due to this arrangement of partitions, node D 104 and node E 105 cannot communicate because they do not share a partition. However, for example, node A 101 and node C 103 can communicate because they are both members of partition 2 140.

[0033] Virtual Machines on InfiniBand Over the past decade, hardware virtualization support has substantially reduced CPU overhead. Virtualizing the memory management unit significantly reduces memory overhead; utilizing high-speed SAN storage or distributed network file systems reduces storage overhead; and device pass-through technologies such as Single Root Input / Output Virtualization (SR-IOV) reduce device overhead. The prospects for virtualized High Performance Computing (HPC) environments have improved significantly as network I / O overhead has been reduced through the use of high-performance interconnect solutions. Clouds now support virtual HPC (vHPC) clusters with high-performance interconnect solutions, providing the necessary performance. It is possible to provide

[0034] However, when coupled with lossless networks such as InfiniBand (IB), some cloud features such as live migration of virtual machines (VMs) remain problematic due to the complex addressing and routing schemes used in these solutions.IB is an interconnect network technology that offers high bandwidth and low latency, making it well suited for HPC and other communication-intensive workloads.

[0035] The traditional approach to connecting IB devices to VMs is by using direct-assigned SR-IOV. However, achieving live migration of VMs assigned to IB host channel adapters (HCAs) using SR-IOV has proven challenging. Each IB-attached node has three different addresses (i.e., LID, GUID, and GID). When a live migration occurs, one or more of these addresses change. Other nodes communicating with the migrating VM (VM-in-migration) may lose connectivity. This When this occurs, the IB subnet manager (SM) is notified that it should reconnect by sending a Subnet Administration (SA) record route query. An attempt can be made to restore the lost connection by finding out the new address of the virtual machine.

[0036] IB uses three different types of addresses. The first type of address is a 16-bit local identifier (LID). At least one unique LID is assigned by the SM to each HCA port and each switch. The LID is used to route traffic within a subnet. Because the LID is 16 bits long, 65,536 unique address combinations can be configured, of which only 49,151 (0x0001-0xBFFF) can be used as unicast addresses. As a result, the number of available unicast addresses defines the maximum size of an IB subnet. The second type of address is a 64-bit globally unique identifier (GUID) assigned by the manufacturer to each device (e.g., HCA and switch) and each HCA port. The SM may assign an additional subnet-specific GUID to an HCA port, which is useful when SR-IOV is used. The third type of address is a 128-bit global identifier (GID). A GID is a valid IPv6 unicast address, at least one of which is assigned to each HCA port. The GID is formed by combining a globally unique 64-bit prefix assigned by the fabric administrator with the GUID address of each HCA port.

[0037] Fat Tree (FTree) Topology and Routing According to one embodiment, some IB-based HPC systems employ a fat-tree topology to take advantage of the useful properties that fat-trees offer, such as full bisection bandwidth and high throughput due to the availability of multiple paths between each source-destination pair. The original concept behind fat trees was to employ thicker links between nodes as the tree approached the root of the topology, using more available bandwidth. The thicker links could help avoid congestion in higher-level switches, preserving bisectional bandwidth.

[0038] 3 illustrates an example of a tree topology in a network environment, according to one embodiment. As shown in FIG. 3, one or more end nodes 201-204 may be connected in a network fabric 200. The network fabric 200 may be based on a fat-tree topology including multiple leaf switches 211-214 and multiple spine or root switches 231-234. In addition, the network fabric 200 may include one or more intermediate switches, such as switches 221-224.

[0039] 3, each of end nodes 201-204 may be a multi-homed node, i.e., a single node that is connected to two or more portions of network fabric 200 via multiple ports. For example, node 201 may include ports H1 and H2, node 202 may include ports H3 and H4, node 203 may include ports H5 and H6, and node 204 may include ports H7 and H8.

[0040] Additionally, each switch may have multiple switch ports. For example, root switch 231 may have switch ports 1-2, root switch 232 may have switch ports 3-4, root switch 233 may have switch ports 5-6, and root switch 234 may have switch ports 7-8.

[0041] According to an embodiment, the fat-tree routing mechanism is one of the most popular routing algorithms for IB-based fat-tree topologies. The fat-tree routing mechanism is also integrated into OFED (Open Fabric Enterprise Distribution: a standard software for building and deploying IB-based applications). This is implemented in the OpenSM (hardware stack) subnet manager.

[0042] The goal of a fat-tree routing mechanism is to generate an LFT that uniformly spreads shortest-path routes across links in the network fabric. The mechanism traverses the fabric in indexing order and assigns target LIDs for end nodes, and therefore corresponding routes, to each switch port. For end nodes connected to the same leaf switch, the indexing order may depend on the switch ports to which the end nodes are connected (i.e., the port numbering sequence). For each port, the mechanism may maintain a port usage counter, and each time a new route is added, the port usage counter may be used to select the least frequently used port.

[0043] According to one embodiment, in a partitioned subnet, nodes that are not members of a common partition are not allowed to communicate. In practice, this means that some of the routes assigned by the Fat Tree routing algorithm will not be used for user traffic. A problem arises if the Fat Tree routing mechanism generates LFTs for those routes in the same way as other functional paths. This behavior can degrade balancing on links because nodes are routed in indexing order. Because routing is done without awareness of partitions, Fat Tree routed subnets generally provide poor isolation between partitions.

[0044] According to one embodiment, a Fat Tree is a hierarchical network topology that can scale with available network resources. It is easily constructed using commodity switches arranged at various levels of hierarchy. Additionally, various variants of fat-trees are publicly available, including k-ary-n-tree, Extended Generalized Fat-Tree (XGFT), Parallel Ports Generalized Fat-Tree (PGFT), and Real Life Fat-Tree (RLFT).

[0045] Also, a k-ary-n-tree is an n-level fat tree with k n end nodes and n·k n-1 The XGFT fat tree comprises a k-ary-n-tree and a k-ary-n-tree, each with 2k ports. Each switch has the same number of connections up and down the tree. The XGFT fat tree extends the k-ary-n-tree by allowing both a different number of up and down connections for the switches and a different number of connections at each level in the tree. The PGFT definition further extends the XGFT topology to allow multiple connections between switches. A wide variety of topologies can be defined using XGFT and PGFT. However, for practical purposes, a restricted version of PGFT, RLFT, is introduced to define fat trees commonly found in modern HPC clusters. RLFT uses the same port count switches for all levels in the fat tree.

[0046] Input / Output (I / O) Virtualization According to one embodiment, I / O Virtualization (IOV) can make I / O available by allowing virtual machines (VMs) access to the underlying physical resources. The combination of storage traffic and inter-server communication can place an unbearable strain on a single server's I / O resources, resulting in backlogs and idle processors waiting for data. As the number of I / O requests increases, IOV can provide availability and improve the performance, scalability, and elasticity of (virtualized) I / O resources to rival performance levels seen in modern CPU virtualization.

[0047] According to one embodiment, IOV is desired to enable sharing of I / O resources and to allow protected access to resources from VMs. IOV separates the logical device exposed to a VM from its physical implementation. Currently, emulation, paravirtualization, direct assignment (DA), and single-root I / O There can be various types of IOV technologies, such as virtualization (SR-IOV).

[0048] According to one embodiment, one type of IOV technology is software emulation. Software emulation can enable a separated front-end / back-end software architecture. The front-end can be a device driver located in a VM and communicate with a back-end implemented by a hypervisor to provide I / O access. Physical device sharing ratios are high, and live migration of VMs can be achieved with only milliseconds of network downtime. However, software emulation introduces additional, undesirable computational overhead.

[0049] According to one embodiment, another type of IOV technology is direct device assignment. Direct device assignment requires that an I / O device be attached to a VM, but the device is not shared between VMs. Direct assignment, or device passthrough, offers near-unique performance with minimal overhead. The physical device bypasses the hypervisor and is directly attached to the VM. However, a drawback of such direct device assignment is that there is no sharing between virtual machines, which limits scalability, such as one physical network card being attached to one VM. do.

[0050] According to one embodiment, Single Root IOV (SR-IOV) is Hardware virtualization can allow a physical device to appear as multiple independent, lightweight instances of the same device. These instances can be assigned to VMs as pass-through devices and accessed as Virtual Functions (VFs). The hypervisor accesses the device through a unique, fully functional Physical Function (PF) (per device). SR-I OV mitigates the scalability issues of purely direct allocation. However, a problem presented by SR-IOV is that it can impair VM migration. Among these IOV technologies, SR-IOV extends the PCI Express (PCIe) standard with a means to allow multiple VMs to directly access a single physical device while maintaining near-inherent performance. This allows SR-IOV to offer superior performance and scalability.

[0051] SR-IOV allows a PCIe device to expose multiple virtual devices that can be shared among multiple guests by assigning one virtual device to each guest. Each SR-IOV device has at least one physical function (PF) and one or more associated virtual functions (VFs). A PF is a communication function controlled by a virtual machine monitor (VMM) or hypervisor. VFs are lightweight PCIe functions, whereas VFs are regular PCIe functions. Each VF has its own base address (BAR) and is assigned a unique requestor ID. The unique requestor ID is managed by the I / O memory management unit (I / O memory management unit). The IOMMU also applies memory and interrupt translation between PFs and VFs.

[0052] Unfortunately, direct device allocation techniques present a barrier to cloud providers in situations where transparent live migration of virtual machines is desired for data center optimization. The essence of live migration is that the memory contents of a VM are copied to a remote hypervisor. Furthermore, the VM is suspended in the source hypervisor, and the VM's operation is resumed in the destination. When using software emulation methods, network interfaces are virtual so that their internal states are stored in memory and then copied. Therefore, downtime can be reduced to a few milliseconds.

[0053] However, migration becomes more difficult when direct device assignment techniques such as SR-IOV are used. In this situation, the entire internal state of the network interface cannot be copied because it is tied to the hardware. Instead, the SR-IOV VF assigned to the VM is detached, a live migration is performed, and a new VF is attached at the destination. For InfiniBand and SR-IOV, this process can result in downtime on the order of several seconds. Furthermore, in the SR-IOV shared port model, the VM's address changes after migration, which adds overhead to the SM and negatively impacts the performance of the underlying network fabric.

[0054] InfiniBand SR-IOV Architecture - Shared Port There can be various types of SR-IOV models (eg, a shared port model, a virtual switch model, and a virtual port model).

[0055] 4 illustrates an exemplary shared port architecture according to one embodiment. As shown, a host 300 (e.g., a host channel adapter) may interact with a hypervisor 310. The hypervisor 310 may assign various virtual functions 330, 340, and 350 to several virtual machines. Similarly, physical functions may be handled by the hypervisor 310.

[0056] 4, a host (e.g., an HCA) appears as a single port to the network with a single shared LID and shared Queue Pair (QP) space between the physical function 320 and the virtual functions 330, 350, 350. However, each function (i.e., the physical function and the virtual function) may have its own GID.

[0057] As shown in FIG. 4, according to one embodiment, various GIDs can be assigned to virtual and physical functions, and a special queue pair, QP0 and QP1 (i.e., InfiniBand TM A dedicated queue pair used for management packets is owned by the physical function. These QPs are exposed to the VFs as well, but the VFs are not allowed to use QP0 (all SMPs coming from the VF towards QP0 are discarded), and QP1 can act as a proxy for the actual QP1 owned by the PF.

[0058] According to one embodiment, the shared port architecture may enable highly scalable data centers that are not limited by the number of VMs (attached to the network by being assigned to virtual functions) because LID space is only consumed by the physical machines and switches in the network.

[0059] However, a drawback of the shared port architecture is that it cannot provide transparent live migration, thereby hindering the potential for flexible VM placement. Because each LID is associated with a specific hypervisor and shared among all VMs residing on that hypervisor, a migrating VM (i.e., a virtual machine migrating to a destination hypervisor) must change its LID to the LID of the destination hypervisor. Furthermore, as a result of the restricted QP0 access, a subnet manager cannot be run inside a VM.

[0060] InfiniBand SR-IOV Architecture Model - Virtual Switch (vSwitch) 5 illustrates an exemplary vSwitch architecture according to one embodiment. As shown, a host 400 (e.g., a host channel adapter) can interact with a hypervisor 410, which can assign various virtual functions 430, 440, and 450 to several virtual machines. Similarly, physical functions can be handled by hypervisor 410. A virtual switch 415 can also be handled by hypervisor 401.

[0061] According to one embodiment, in the vSwitch architecture, each virtual function 430, 440, 450 is a full virtual Host Channel Adapter (vHCA), which means that in hardware, the VM assigned to the VF is assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. To the rest of the network and SM, the HCA 400 appears as a switch with additional nodes connected via virtual switch 415. The hypervisor 410 can use the PF 420 (assigned to the virtual function) to )VM uses VF.

[0062] According to one embodiment, the vSwitch architecture provides transparent virtualization. However, because each virtual function is assigned a unique LID, the available number of LIDs is quickly consumed. Similarly, if many LID addresses are used (i.e., one for each physical function and each virtual function), more communication paths must be computed by the SM and more subnet management packets (SMPs) must be sent to the switch to update their LFTs. For example, computing communication paths can take several minutes in a large network. Because the LID space is limited to 49,151 unicast LIDs and each VM (through a VF) occupies one LID per physical node and switch, the number of active VMs is limited by the number of physical nodes and switches in the network, and vice versa.

[0063] InfiniBand SR-IOV Architecture Model - Virtual Port (vPort) 6 illustrates an exemplary vPort concept according to one embodiment. As shown, a host 300 (e.g., a host channel adapter) can interact with a hypervisor 410 that can allocate various virtual functions 330, 340, and 350 to several virtual machines. Similarly, physical functions can be handled by the hypervisor 310.

[0064] According to one embodiment, the vPort concept is loosely defined to allow vendors implementation freedom (e.g., the definition does not stipulate that implementations should be SRIOV-only), and the purpose of vPort is to standardize how VMs are handled in a subnet. The vPort concept allows for the definition of both an SR-IOV shared port-like architecture and a vSwitch-like architecture, or a combination of these architectures, which may be more scalable in both the spatial and performance domains. Also, vPorts support optional LIDs, and unlike shared ports, the SM is aware of all vPorts available in a subnet, even if the vPorts do not use dedicated LIDs.

[0065] InfiniBand SR-IOV Architecture Model - LID Pre-Populated vSwitch According to one embodiment, the present disclosure provides a system and method for providing a vSwitch architecture with pre-populated LIDs.

[0066] 7 illustrates an exemplary vSwitch architecture with pre-populated LIDs, according to one embodiment. As shown, several switches 501-504 are configured to support InfiniBand TMCommunications can be established between members of a fabric, such as a fabric. The fabric can include several hardware devices, such as host channel adapters 510, 520, and 530. Furthermore, host channel adapters 510, 520, and 530 can interact with hypervisors 511, 521, and 531, respectively. Each hypervisor, along with the host channel adapters, can further interact with, configure, and assign to several virtual machines several virtual functions 514, 515, 516, 524, 525, 526, 534, 535, and 536. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 can additionally assign virtual machine 2 551 to virtual function 2 515 and virtual machine 3 552 to virtual function 3 516. The hypervisor 531 can further assign a virtual machine 4 553 to a virtual function 1 534. The user can access the host channel adapters through fully functional physical functions 513, 523 and 533 on each of the host channel adapters.

[0067] According to one embodiment, each of switches 501-504 may include several ports (not shown) that are used to configure linear forwarding tables to direct traffic within network switching environment 600.

[0068] According to one embodiment, virtual switches 512, 522, and 532 can be handled by respective hypervisors 511, 521, 531. In such a vSwitch architecture, each virtual function is a full virtual host channel adapter (vHCA), which means that in hardware, a VM assigned to a VF is assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. To the rest of the network and SM (not shown), HCAs 510, 520, and 530 appear as switches with additional nodes connected via virtual switches.

[0069] According to one embodiment, the present disclosure provides a system and method for providing a vSwitch architecture with pre-populated LIDs. Referring to FIG. 7, LIDs are pre-populated for various physical functions 513, 523, and 533, as well as for virtual functions 514-516, 524-526, and 534-536 (even virtual functions not currently associated with active virtual machines). For example, physical function 513 is pre-populated with LID 1, and virtual function 1 534 is pre-populated with LID 10. When a network is booted, LIDs are pre-populated in the SR-IOV vSwitch-enabled subnet. Populated VFs are assigned LIDs as shown in FIG. 7, even if not all of the VFs are occupied by VMs in the network.

[0070] According to one embodiment, many similar physical host channel adapters can have two or more ports (with two ports shared for redundancy), and a virtual HCA can also be represented by two ports and connected to an external IB subnet via one or more virtual switches.

[0071] According to one embodiment, in a vSwitch architecture with pre-populated LIDs, each hypervisor consumes one LID for itself via the PF and can consume one or more LIDs for each additional VF. The sum of all VFs available across all hypervisors in an IB subnet gives the maximum amount of VMs that can run in the subnet. For example, in an IB subnet with 16 virtual functions per hypervisor in the subnet, each hypervisor consumes 17 LIDs in the subnet (one LID for each of the 16 virtual functions and one LID for the physical function). In such an IB subnet, the theoretical hypervisor limit for a single subnet is defined by the number of available unicast LIDs: 2891 (49151 available LIDs divided by 17 LIDs per hypervisor), and the total number of VMs (i.e., limit) is 46256 (2891 hypervisors multiplied by 16 VFs per hypervisor). (In practice, these numbers will be smaller, as each switch, router, or dedicated SM node in the IB subnet will consume LIDs as well.) Note that vSwitches do not need to occupy additional LIDs, as they can share LIDs with PFs.

[0072] According to one embodiment, a LID pre-populated vSwitch architecture In ,communication paths are calculated for all LIDs once the network is booted.,When a new VM needs to be started, the system does not need to add a new,LID in the subnet.,Otherwise, the operation that can completely reconfigure the network,,including recalculating the paths, is the most time-consuming,factor. Instead, available ports for VMs are located in one of the,hypervisors (i.e., available virtual functions), and virtual machines are,assigned to available virtual functions.

[0073] According to one embodiment, the LID pre-populated vSwitch architecture also enables the ability to compute and use different routes to reach different VMs hosted by the same hypervisor. Essentially, this allows such subnets and networks to use LID-Mask-Control-like (LMC-like) features to provide alternative routes towards one physical machine without being bound by the LMC constraint that requires LIDs to be contiguous. The freedom to use non-contiguous LIDs is particularly useful when a VM needs to migrate and its associated LID needs to be delivered to the destination.

[0074] In accordance with one embodiment, several considerations can be taken into account along with the above-described advantages of a LID pre-populated vSwitch architecture. For example, because LIDs are pre-populated in an SR-IOV vSwitch-enabled subnet when the network is booted, the initial route computation (e.g., at startup) may take longer than if the LIDs were not pre-populated.

[0075] InfiniBand SR-IOV Architecture Model - vSwitch with Dynamic LID Allocation According to one embodiment, the present disclosure provides a system and method for providing a vSwitch architecture with dynamic LID allocation.

[0076] 8 illustrates an exemplary vSwitch architecture with dynamic LID assignment, according to one embodiment. As shown, several switches 501-504 are configured to support InfiniBand TMCommunications can be established between members of a fabric, such as a fabric. The fabric can include several hardware devices, such as host channel adapters 510, 520, and 530. Host channel adapters 510, 520, and 530 can further interact with hypervisors 511, 521, and 531, respectively. Each hypervisor, along with the host channel adapters, can further interact with, configure, and assign to several virtual machines several virtual functions 514, 515, 516, 524, 525, 526, 534, 535, and 536. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 can additionally assign virtual machine 2 551 to virtual function 2 515 and virtual machine 3 552 to virtual function 3 516. Hypervisor 531 can further assign virtual machine 4 553 to virtual function 1 534. The hypervisor can access the host channel adapters through fully functional physical functions 513, 523 and 533 on each of the host channel adapters.

[0077] According to one embodiment, each of switches 501-504 may include several ports (not shown) that are used to configure linear forwarding tables to direct traffic within network switching environment 700.

[0078] According to one embodiment, the virtual switches 512, 522 and 532 are The VFs can be handled by the HCAs 510, 520, and 531. In such a vSwitch architecture, each virtual function is a full virtual host channel adapter (vHCA), which means that in hardware, the VM assigned to the VF is assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. To the rest of the network and SM (not shown), the HCAs 510, 520, and 530 appear as switches with additional nodes connected via virtual switches.

[0079] According to one embodiment, the present disclosure provides a system and method for providing a vSwitch architecture with dynamic LID assignment. Referring to FIG. 8 , various physical functions 513, 523, and 533 are dynamically assigned LIDs, with physical function 513 receiving LID 1, physical function 523 receiving LID 2, and physical function 533 receiving LID 3. Those virtual functions associated with active virtual machines may also receive dynamically assigned LIDs. For example, because virtual machine 1 550 is active and associated with virtual function 1 514, virtual function 514 may be assigned LID 5. Similarly, virtual function 2 515, virtual function 3 516, and virtual function 1 534 are each associated with an active virtual function. As such, these virtual functions are assigned LIDs: LID 7 is assigned to virtual function 2 515, LID 11 is assigned to virtual function 3 516, and LID 9 is assigned to virtual function 1 534. Unlike a vSwitch, which has pre-populated LIDs, virtual functions that are not currently associated with an active virtual machine do not receive an LID assignment.

[0080] According to one embodiment, dynamic LID assignment can substantially reduce initial path computation: When a network is booting for the first time and no VMs are present, a relatively small number of LIDs can be used for initial path computation and LFT distribution.

[0081] According to one embodiment, many similar physical host channel adapters can have two or more ports (with two ports shared for redundancy), and a virtual HCA can also be represented by two ports and connected to an external IB subnet via one or more virtual switches.

[0082] According to one embodiment, when a new VM is created in a system utilizing a vSwitch with dynamic LID allocation, a free VM slot is discovered and a unique, unused unicast LID is discovered as well to determine on which hypervisor the newly added VM should boot. However, there is no known route in the switch's LFT and network to handle the newly added LID. Computing a new set of routes to handle the newly added VM is undesirable in a dynamic environment where several VMs may be booted every minute. In a large IB subnet, computing a new set of routes could take several minutes, and this procedure would have to be repeated each time a new VM is booted.

[0083] Advantageously, according to one embodiment, since all VFs in a hypervisor share the same uplink with the PF, there is no need to compute a new set of routes. All that is required is to iterate through the LFTs of all physical switches in the network, copy the forwarding ports from the LID entries belonging to the PF of the hypervisor (on which the VM is created) to the newly added LID, and send a single SMP to update the corresponding LFT block of the particular switch. This eliminates the need for the system and method to compute a new set of routes.

[0084] According to one embodiment, in a vSwitch with a dynamic LID allocation architecture, The LIDs assigned to VMs on each hypervisor do not need to be contiguous. Comparing the LIDs assigned to VMs on each hypervisor between a pre-populated vSwitch and a dynamic LID allocation vSwitch, one can see that the assigned LIDs are discontinuous in the dynamic LID allocation architecture, whereas the pre-populated LIDs are essentially contiguous. Furthermore, in the vSwitch dynamic LID allocation architecture, when a new VM is created, the next available LID is used for the lifetime of the VM. Conversely, in a pre-populated vSwitch, each VM inherits the LID already assigned to its corresponding VF, and in a network without live migration, VMs assigned consecutively to a given VF get the same LID.

[0085] According to one embodiment, a vSwitch with a dynamic LID allocation architecture can address the shortcomings of a vSwitch with a pre-populated LID architecture model at the expense of some additional network and runtime SM overhead. Each time a VM is created, the LFT of the physical switch in the subnet is updated with the newly added LID associated with the created VM. This operation requires one subnet management packet (SMP) to be sent per switch. Because each VM uses the same route as its host hypervisor, features such as LMC are also unavailable. However, there is no limit on the total number of VFs present on all hypervisors, and the number of VFs may exceed the unicast LID limit. In such a case, of course, not all VFs can be simultaneously granted on active VMs. Having more spare hypervisors and VFs adds flexibility for recovering from and optimizing fragmented network failures when operating near the unicast LID limit.

[0086] InfiniBand SR-IOV Architecture Model - Dynamic LID Allocation and Pre-Populated LID vSwitch 9 illustrates an exemplary vSwitch architecture with dynamic LID assignment and pre-populated LIDs for the vSwitch, according to one embodiment. As shown, several switches 501-504 are configured to support InfiniBand TM Communication can be established between members of a fabric, such as a fabric. The fabric can include several hardware devices, such as host channel adapters 510, 520, and 530. The host channel adapters 510, 520, and 530 can further interact with hypervisors 511, 521, and 531, respectively. Each hypervisor, along with the host channel adapters, can further interact with, configure, and assign to several virtual machines several virtual functions 514, 515, 516, 524, 525, 526, 534, 535, and 536. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 can additionally assign virtual machine 2 551 to virtual function 2 515. Hypervisor 521 can assign virtual machine 3 552 to virtual function 3 526. Hypervisor 531 can further assign virtual machine 4 553 to virtual function 2 535. The hypervisor can access the host channel adapters through fully functional physical functions 513, 523, and 533 on each of the host channel adapters.

[0087] According to one embodiment, each of switches 501-504 may include several ports (not shown) that are used to configure linear forwarding tables to direct traffic within network switching environment 800.

[0088] According to one embodiment, virtual switches 512, 522, and 532 can be handled by respective hypervisors 511, 521, 531. In such a vSwitch architecture, each virtual function is a full virtual host channel adapter (vHCA), which means that in hardware, a VM assigned to a VF is assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. To the rest of the network and SM (not shown), HCAs 510, 520, and 530 appear as switches with additional nodes connected via virtual switches.

[0089] According to one embodiment, the present disclosure provides a system and method for providing a hybrid vSwitch architecture with dynamic LID assignment and pre-populated LIDs. Referring to FIG. 9 , hypervisor 511 may be deployed with a vSwitch with a pre-populated LID architecture, and hypervisor 521 may be deployed with a vSwitch with pre-populated LIDs and dynamic LID assignment. Hypervisor 531 may be deployed with a vSwitch with dynamic LID assignment. Thus, physical function 513 and virtual functions 514-516 have their LIDs pre-populated (i.e., even virtual functions that are not assigned to active virtual machines are assigned LIDs). Physical function 523 and virtual function 1 524 may have their LIDs pre-populated, while virtual function 2 525 and virtual function 3 526 have their LIDs dynamically assigned (i.e., virtual function 2 525 is available for dynamic LID assignment, and virtual function 3 526 has been dynamically assigned an LID of 11 because it is attached to virtual machine 3 552). Finally, the functions (physical and virtual functions) associated with hypervisor 3 531 can have their LIDs dynamically assigned. This results in virtual function 1 534 and virtual function 3 536 being available for dynamic LID assignment, while virtual function 2 535 has been dynamically assigned an LID of 9 because it is attached to virtual machine 4 553.

[0090] 9, in which both LID pre-populated vSwitches and dynamic LID allocation vSwitches are utilized (independently or combined within any given hypervisor), the number of pre-populated LIDs per host channel adapter can be defined by a fabric administrator and can be in the range 0<=pre-populated VFs<=total VFs (per host channel adapter). The VFs available for dynamic LID allocation can be found by subtracting the number of pre-populated VFs from the total number of VFs (per host channel adapter).

[0091] According to one embodiment, many similar physical host channel adapters can have two or more ports (with two ports shared for redundancy), and a virtual HCA can also be represented by two ports and connected to an external IB subnet via one or more virtual switches.

[0092] InfiniBand - Inter-subnet communication (Fabric Manager) According to one embodiment, in addition to providing an InfiniBand fabric within one subnet, embodiments of the present disclosure may also provide an InfiniBand fabric that spans two or more subnets.

[0093] 10 illustrates an exemplary multi-subnet InfiniBand fabric according to one embodiment. As shown in this figure, multiple switches 1001-1004 within subnet A 1000 provide the infrastructure within subnet A 1000 (e.g., the IB subnet). A number of switches 1021-1024 may provide communication between members of a fabric, such as an InfiniBand fabric, within subnet B 1040 (e.g., an IB subnet). The fabric may include a number of hardware devices, such as, for example, a channel adapter 1010. The host channel adapter 1010 may interact with a hypervisor 1011. The hypervisor may set up a number of virtual functions 1014 with the host channel adapters it interacts with. In addition, the hypervisor may assign a virtual machine to each virtual function. For example, virtual machine 1 1015 is assigned to virtual function 1 1014. The hypervisor may access its associated host channel adapter through a fully functional physical function, such as physical function 1013, on each host channel adapter. A number of switches 1021-1024 may provide communication between members of a fabric, such as an InfiniBand fabric, within subnet B 1040 (e.g., an IB subnet). The fabric may include a number of hardware devices, such as, for example, a host channel adapter 1030. The host channel adapter 1030 may interact with a hypervisor 1031. The hypervisor can set up multiple virtual functions 1034 along with the host channel adapters they interact with. In addition, the hypervisor can assign a virtual machine to each virtual function. For example, virtual machine 2 1035 is assigned to virtual function 2 1034. The hypervisor can access its associated host channel adapter through a fully functional physical function, such as physical function 1033, on each host channel adapter. Note that although only one host channel adapter is shown in each subnet (i.e., subnet A and subnet B), it should be understood that each subnet may contain multiple host channel adapters and their corresponding components.

[0094] According to one embodiment, each host channel adapter may further be associated with a virtual switch, such as virtual switch 1012 and virtual switch 1032, and as noted above, each HCA may be set up with a different architectural model. Although both subnets in Figure 10 are shown as using the architectural model of a vSwitch with pre-populated LIDs, this is not intended to suggest that all such subnet configurations may follow a similar architectural model.

[0095] According to one embodiment, at least one switch in each subnet may be associated with a router. For example, switch 1002 in subnet A 1000 is associated with router 1005, and switch 1021 in subnet B 1040 is associated with router 1006.

[0096] According to one embodiment, at least one device (e.g., a switch, a node, etc.) can be associated with a fabric manager (not shown). The fabric manager can be used, for example, to discover inter-subnet fabric topologies, create fabric profiles (e.g., virtual machine fabric profiles), and build virtual machine-related database objects that form the basis for building the virtual machine fabric profiles. Additionally, the fabric manager can define legal inter-subnet connectivity regarding which subnets are allowed to communicate over which router ports and using which partition numbers.

[0097] According to one embodiment, when traffic from an originating source, such as virtual machine 1 in subnet A, is destined for a different subnet, such as virtual machine 2 in subnet B, the traffic can be directed to a router in subnet A, i.e., router 1005, which can then send the traffic to subnet B via a link with router 1006.

[0098] Virtual Dual Port Router According to one embodiment, a dual port router abstraction is implemented to route from a global route header (GRH) to an LR. It is possible to provide a simple method that allows defining an inter-subnet router function based on a switch hardware implementation that has the capability to perform conversion to a local route header (LRH) in addition to performing normal LRH-based switching.

[0099] According to one embodiment, a Virtual Dual Port Router can be logically connected outside of a corresponding switch port and can present an InfiniBand compliant view to a standard management entity, such as a subnet manager.

[0100] According to one embodiment, the dual port router model connects different subnets, with each subnet handling packet forwarding and address mapping on the ingress path to the subnet. This shows that it is possible to connect subnets in a way that gives complete control over the subnetworks and does not affect the routing and logical connectivity within any of the misconnected subnets.

[0101] According to one embodiment, in situations involving misconnected fabrics, the virtual dual port router abstraction can also be used to allow management entities such as subnet managers and IB diagnostic software to function correctly in the presence of unintended physical connections to remote subnets.

[0102] 11 illustrates an interconnection between two subnets in a high performance computing environment, according to one embodiment. Before being configured with a virtual Dual Port Router, Subnet A Switch 1120 in subnet B 1101 may be connected through switch port 1121 of switch 1120 via physical connection 1110 to switch 1130 in subnet B 1102 through switch port 1131 of switch 1130. In such an embodiment, each of switch ports 1121 and 1131 may function as both a switch port and a router port.

[0103] According to one embodiment, the problem with this configuration is that a management entity, such as a subnet manager, in an InfiniBand subnet cannot distinguish between physical ports that are both switch ports and router ports. In this situation, the SM can treat a switch port as having a router port connected to it. However, if the switch port is connected to another subnet with a different subnet manager, for example, via a physical link, the subnet manager can send discovery messages to the physical link. However, such discovery messages are not allowed in the other subnet.

[0104] FIG. 12 illustrates an interconnection between two subnets via a dual-port virtual router configuration in a high-performance computing environment, according to one embodiment.

[0105] According to one embodiment, after configuration, the dual port virtual router configuration can be provided to the subnet manager such that the appropriate end nodes that indicate the edges of the subnets for which the subnet manager is responsible are known.

[0106] According to one embodiment, a switch port in a switch 1220 in subnet A 1201 can be connected (i.e., logically connected) to a router port 1211 in a virtual router 1210 via a virtual link 1223. A physical link 1203, which may have two ends, connects subnet A 1201 to subnet B 1202 via a first end of the physical link, via router port II 1212, and via a second end of the physical link, via router port II 1212, and via a virtual router 1230 in subnet B 1202. 1232. Virtual router 1230 may further include a router port 1231 that may be connected (i.e., logically connected) to a switch port 1241 on switch 1240 via a virtual link 1233.

[0107] According to one embodiment, a subnet manager (not shown) on subnet A can discover router port 1211 on virtual router 1210 as the endpoint of the subnet that the subnet manager controls. The dual-port virtual router abstraction allows the subnet manager on subnet A to treat subnet A in the usual way (e.g., as specified in the InfiniBand standard). At the subnet management agent level, the dual-port virtual router abstraction A fabric abstraction can be provided so that a normal switch port appears to the SM, and then at the SMA level, the abstraction can be provided so that there is another port connected to this switch port, and this port becomes a router port on a dual-port virtual router. The local SM can continue to use the traditional fabric topology (where the SM sees the ports as standard switch ports), and therefore the SM sees the router ports as end ports. A physical connection can be made between two switch ports that are also configured as router ports in two different subnets.

[0108] According to one embodiment, a dual port virtual router can also solve the problem that a physical link may be mistakenly connected to any other switch port in the same subnet, or to a switch port that is not intended to provide connectivity to another subnet. Thus, the methods and systems described herein also represent what is outside of a subnet.

[0109] According to one embodiment, a local SM in a subnet, such as subnet A, determines a switch port and then determines the router port connected to this switch port (e.g., router port 1211 connected to switch port 1221 via virtual link 1223). The SM considers router port 1211 to be the edge of the subnet that it manages, so the SM cannot send discovery and / or management messages further than this point (e.g., to router port II 1212).

[0110] According to one embodiment, the dual port virtual router provides the advantage that the dual port virtual router abstraction is managed entirely by a management entity (e.g., SM or SMA) within the subnet to which the dual port virtual router belongs. By keeping management local, the system does not need to provide an external, independent management entity; that is, each side of the inter-subnet connection is responsible for configuring its own dual port virtual router.

[0111] According to one embodiment, when a packet such as an SMP destined for a remote destination (i.e., outside the local subnet) arrives at a local target port that is not configured through the dual-port virtual router, the local port can return a message indicating that it is not a router port.

[0112] Many features of the present teachings can be implemented in, using, or with the aid of hardware, software, firmware, or a combination thereof. Thus, features of the present teachings may be implemented using a processing system (e.g., including one or more processors).

[0113] 13 illustrates a method for supporting a dual-port virtual router in a high-performance computing environment according to one embodiment. In step 1310, the method may provide a first subnet in one or more computers including one or more microprocessors. The first subnet includes a plurality of switches, the plurality of switches including at least leaf switches, each of the plurality of switches including a plurality of switch ports. The first subnet further includes a plurality of host channel adapters, each of the host channel adapters including at least one host channel adapter port, a plurality of end nodes, each of the end nodes associated with at least one host channel adapter of the plurality of host channel adapters, and a subnet manager, the subnet manager executing in one of the plurality of switches and the plurality of host channel adapters.

[0114] In step 1320, the method may configure a switch port of the plurality of switch ports on a switch of the plurality of switches as a router port.

[0115] In the switch 1330, the method can logically connect switch ports configured as router ports to a virtual router, the virtual router including at least two virtual router ports.

[0116] Redundant fully independent and semi-independent networks According to one embodiment, a mission-critical system should operate correctly and always provide the relevant services to the correct clients with the correct data, within the expected response times and overall performance constraints specified for the system.

[0117] According to one embodiment, for a system implemented as a distributed cluster of computers, this also implies that communication between the required set of computers in the cluster must always be operational.

[0118] According to one embodiment, a desirable premise for a network communication system to be operational is that all physical connectivity between components such as host adapters and switches is correct in that the correct components and connectors are connected via the correct cables, and that all components involved are configured in the correct manner.

[0119] However, according to one embodiment, communications should not typically rely on only one point of failure because hardware components can fail and operators can make mistakes. When a failure or mistake occurs that prevents communication over one component and communication path, it is of utmost importance to detect this and have communications taken over to an alternative, redundant communication path immediately, or at least before the inoperable communication path has any significant impact on system operation or response times.

[0120] Also, according to one embodiment, whenever there is a failure that leaves some or all current communications vulnerable if a second failure occurs, it is important that relevant remedial action can occur as soon as possible and / or additional backup measures can be employed to prevent total loss of service.

[0121] According to one embodiment, another aspect is that because system and application software are rarely error-free and there is often a need to improve system features over time, it is important to be able to upgrade various software components within the system without causing any outages. In distributed cluster systems, this typically means a "rolling upgrade" model where redundant components are upgraded in a strict order, and the system is always fully operational with the required set of components and the required communications.

[0122] Also, according to one embodiment, physical modifications may be required in terms of adding additional HW components and / or replacing existing HW components with more capable HW components to improve the system's capabilities and / or expand its capacity. In this case, such upgrades may also mean new software and / or firmware in the system.

[0123] However, new software and firmware versions, as well as new HW versions, introduce the risk of introducing new or early undetected errors and the risk of interoperability problems between different software / firmware components on the same node or between different nodes in the cluster. Therefore, in the ideal case, the ability to operate the system should not depend on only a single type of software being used throughout the system and on successful upgrades from one software version to another. To achieve this, one approach is to ensure that the required services can be realized in a redundant manner by two sets of different types of hardware components, with different sets of software and firmware, and that there are no dependencies, or at a minimum, very well-defined and controlled dependencies, between the different sets of redundant components.

[0124] According to one embodiment, an example of such a system is the use of geographically separate systems implemented with different computer types and using different software, but backup data can be transferred between those systems using a neutral format with minimal dependency on either system. A less strict version of this is where the primary and backup sites use the same type of equipment but are not upgraded at the same time. Thus, typically, the backup site continues to use version N until sufficient confidence has been established with version N+1 at the primary site.

[0125] According to one embodiment, another approach is to have several independent implementations of the same functionality running in parallel. This approach has been used in space missions where multiple versions of critical components were developed independently by different teams. A less extreme version of this approach applied to cluster communication infrastructure is to have two independent networks, each implemented by gear (HW and SW / FW) from different vendors, but communication between pairs of computers can fail over between the two network infrastructures. This approach to network redundancy can then be applied orthogonally to whether different types of computers and host software are used in the cluster to provide similar types of independence and redundancy at this level as well.

[0126] Still, from a practical standpoint, cost and complexity are also important factors when designing mission-critical, highly available systems, according to one embodiment. Thus, different deployments may use different levels of redundancy and different levels of multiple infrastructure types (i.e., more than one) to suit the budget and risk scenarios for the associated system deployment.

[0127] According to one embodiment, a fully redundant private fabric is still subject to management and congestion problem / bug propagation, as well as "disturbances" caused by link failures and reconfigurations from one redundant portion to the other when implemented as a single subnet.

[0128] According to one embodiment, to provide two (or more) fully independent or semi-independent fabrics, the hosts may include redundant connectivity to each fabric to prevent loss of connectivity if pairs of hosts each lose their single connection to a different independent fabric.

[0129] According to one embodiment, the problem is to provide redundant connectivity between two fabrics that is independent of the normal management of each fabric and that is used for data communication when no other options exist.

[0130] FIG. 14 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment.

[0131] According to one embodiment, two or more rails may be provided, such as rail A 1401 and rail B 1421. Although not shown, each independent rail may include one or more interconnected switches and multicast proxies (MC proxies), such as MC proxy 1402 and MC proxy 1422. Additionally, each rail may include a highly available path service (HAPS), such as HAPS 1403 and HAPS 1423. A rail may be connected to several hosts, such as host 1 1430 through host N 1440. Although not shown, each host may include one or more end nodes connected to the rail via one or more host channel adapters. Additionally, the end nodes may include one or more virtual machines (e.g., utilizing a virtual switch, virtual port, or other similar architecture described above or similar in nature), as described above with respect to a virtualized environment. According to one embodiment, each host may include a multipath selection component, such as MP selection 1431 and MP selection 1441.

[0132] According to one embodiment, the term "rail" can be used to identify both two or more independent fabrics / subnets and redundant connectivity / links from hosts. Each rail can provide redundant, isolated, point-to-point (for unicast) or point-to-multipoint (for multicast) traffic between end nodes.

[0133] According to one embodiment, Rail A and Rail B may be connected via one or more inter-rail links (IRLs).

[0134] According to an embodiment, the term "IRL" (Inter Rail Link) refers to: It can be thought of as similar to an Inter Switch Link (ISL). However, the IRL can be used in a limited manner by not being part of any rail.

[0135] According to one embodiment, the term "MC Proxy" refers to a multicast proxy. The MC proxy can include a high availability component that forwards selected multicast packets from one rail to the other (e.g., ARP (Address Resolution Protocol) requests).

[0136] According to one embodiment, the term "HAPS" may refer to a High Availability Path Service (HA). HAPS operates within the context of one rail but may include a high availability component that communicates with peers in other rails to enable unicast data traffic forwarding over an IRL whenever two hosts need / require communication.

[0137] According to one embodiment, the term "MP selection" can refer to host-based HA / multipath logic that selects which rails to use for different connections to different peer hosts.

[0138] According to one embodiment, in a single rack topology, two leaf switches in the rack can represent two rails (i.e., a minimal topology in which the two rails are separated by hardware). There can be at least two IRLs between the leaf switches. In the case of an InfiniBand topology, each leaf switch can be a single subnet with an embedded subnet manager that is the master subnet manager for each respective rail.

[0139] According to one embodiment, in a multi-rack topology, two leaf switches in each rack can represent two rails. Each leaf switch in a single rack represents a different rail. At least one spine switch in each rack. The set of spines is divided into two groups, one in each rail. There may be special handling for dual and three-rack configurations to avoid a single spine being a SPOF (single point of failure) in any rail. A leaf switch in one rail is connected to all spines in the same rail (but not to other rails). When M>1, the IRL is between two or more pairs of spines from each rail. For InfiniBand topologies, redundant SMs can be provided and located on two or more switches (or possibly dedicated hosts) in each rail.

[0140] According to one embodiment, in both single-rack and multi-rack topologies, fat-tree routing can ignore inter-rail links. In the case of InfiniBand topologies, different rails are configured with different M_Keys (Management Keys) to ensure there is no subnet manager interaction / interference across IRLs.

[0141] According to one embodiment, the "HA Path Service" (HAPS) can track the complete HCA node and port population within each subnet (which may also include system image GUIDs for handling hosts with multiple HCA configurations).

[0142] According to one embodiment, HAPS can use GID in-service / GID out-of-service event notifications from SA or a special protocol. If HAPS is implemented as a host-based service (which may be co-located with MC proxy instances), no special protocol is required by default to track node population, but HAPS will have a scope limited by the partitions that its associated HCA ports can be members of.

[0143] According to one embodiment, a HAPS implementation co-located with a master SM may The scalability of the partitions may be such that they can have more direct interaction and would not be limited to representing only specific partitions.

[0144] According to one embodiment, HAPS can keep track of switches with "cross-link" ports in each L2 subnet and ensure proper connectivity, similar to how an "inter-subnet manager" can ensure proper connectivity between peer router ports.

[0145] According to one embodiment, HAPS can establish unicast LID forwarding for "remote LIDs" that require unicast forwarding over crosslinks. This can, in principle, be done "independently" from the master SM in the associated L2 subnet. The requirement would be that the SM could be instructed (by configuration policy to use a specific LID, but still set a "LinearFDBTop" value for each switch based on another configuration parameter). In this way, the master SM in each L2 subnet would operate with non-overlapping LID ranges, but the switches in each L2 subnet would still be able to forward unicast packets with DLID values ​​in the ranges belonging to other (redundant) L2 subnets.

[0146] According to one embodiment, the LID range boundaries are determined by the Linear Forwarding Table (LFT). As long as the LFT blocks are aligned on LFT block boundaries, it is possible for the HAPS to update the LFT blocks to manage unicast cross-connectivity independently of (and simultaneously with) the master SM in the local subnet. The actual updates may be performed directly via SMP operations or via a special agent on the switch.

[0147] According to one embodiment, an alternative scheme is for the HAPS to request the local SM to consider the associated cross-link port to represent all remote LIDs currently required for remote connectivity via this cross-link port. (This is similar to the processing of a router port, but this would be an entirely new SM operation, whereas a router port only needs a single LID processed during the normal subnet discovery / initialization process.) In this embodiment, the local master SM / SA can be provided with a path record for the remote port GID that reflects the associated DLID (similar to how an "inter-subnet manager" provides the local master SM with a path record for a remote port in a router-based context). Without the router port, the SA can look up the path record based on the information provided, but understand that the cross-link port is a "local destination" in the local subnet.

[0148] According to one embodiment, when this unicast cross-link processing is combined with a scheme that does not require a path record query (e.g., as described in "SYSTEM AND METHOD FOR SUPPORTING NODE ROLE ATTRIBUTES IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT" in application Ser. No. 15,416,899, filed Jan. 26, 2017), No. 7,991,006, entitled "Filtering Redundant Packets in Computer Network Equipments," which are incorporated herein by reference in their entireties, the local SM / SA does not need to know anything about the route to the remote port GUID.

[0149] FIG. 15 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment.

[0150] According to one embodiment, this diagram illustrates a single rack implementation for supporting redundant independent networks in a high performance computing environment.

[0151] According to one embodiment, within a single rack topology, a rack may include two or more leaf switches 1512-13. The rack may additionally and optionally include a number of other switches 1511. These additional switches are optional, as two leaf switches are the minimum base configuration for a single rack topology.

[0152] According to one embodiment, within a single rack topology in which two leaf switches are interconnected by a set of inter-switch links (ISLs) or intra-rail links (IRLs), two or more rails can be defined through leaf switch partitioning by assigning each leaf switch to a separate rail.

[0153] According to one embodiment, in the absence of leaf switch partitioning, there may generally be restrictions on which port numbers can be used for ISLs, depending on switch hardware specific constraints on packet buffer allocation and switch HW resource organization.

[0154] According to one embodiment, in this situation, each leaf switch 1512 and 1513 represents a separate rail, with the ISL representing the IRL. In the specific case of an InfiniBand fabric, each leaf switch represents a single subnet with an embedded subnet manager that is the master subnet manager.

[0155] According to one embodiment, in a situation where the ports on each leaf switch are partitioned to provide two rails within each leaf switch, the total number of ports in each leaf switch is divided into two partitions, with each partition representing an independent or semi-independent rail. Within a RoCE fabric partition, there may also be restrictions on which port numbers can be used for ISLs. (By default, the same set of ports will be used for ISLs in both partitioned and non-partitioned leaf switches.) According to one embodiment, to reduce the number of switches needed to implement both the RoCE fabric and access to client networks from private fabric-based systems in smaller (e.g., quarter rack) configurations, leaf switches can be partitioned (e.g., ports on the leaf switches) and used to implement the private RoCE fabric into one physical partition (i.e., a set of physical ports / connectors) representing the private RoCE fabric and another non-overlapping partition representing access to client networks (e.g., an on-premises data center network). Thus, each such physically partitioned switch can have two non-overlapping sets of ports, where only the set of ports dedicated to the private RoCE fabric can represent connectivity within the RoCE fabric (and vice versa).

[0156] FIG. 16 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment.

[0157] According to one embodiment, in a multi-rack topology, there can be X racks, each rack containing a number of switches, including at least some leaf switches. In the illustrated embodiment, system 1600 includes X racks, including rack 1 1610, rack 2 1620, rack X-1 1630, and rack X 1640. Each rack contains multiple Rack 1 includes leaf switches 1612-13 and spine switch 1611, rack 2 includes leaf switches 1622-23 and spine switch 1621, rack X-1 includes leaf switches 1632-33 and spine switch 1631, and rack X includes leaf switches 1642-43 and spine switch 1641. Each spine switch is also connected to two inter-rail links as shown in the figure.

[0158] According to one embodiment, in a multi-rack topology, each rack contains at least two leaf switches, and the leaf switches belong to separate rails. In the diagram, the rail to which each switch belongs is indicated by either a "1" or a "2" marked on each switch. Similarly, in a multi-rack topology, each rack has at least one spine switch. The set of spine switches is divided into two groups, one for each rail. The leaf switches in one rail are connected to all spine switches in the same rail, but not to spine switches in different rails. There can be more than one IRL, M, between two or more pairs of spine switches from each rail. In an InfiniBand topology, redundant subnet managers reside on two or more switches (or dedicated hosts) in each rail.

[0159] According to one embodiment, each leaf switch in each rack has a set of uplinks distributed among all spine switches in the fabric. Generally, there may be restrictions on which port numbers can be used for uplinks, depending on switch HW-specific constraints on packet buffer allocation and switch HW resource organization. Each spine switch has a certain number of downlinks distributed among all leaf switches in the fabric.

[0160] According to one embodiment, as long as the spine switch is only supposed to provide downlink connectivity, there need not be any difference in characteristics between different ports. However, in a particular configuration, there may still be reasons to limit the port numbers that can be used for downlinks, since not all spine ports may be utilized (connected).

[0161] According to one embodiment, a multi-rack configuration typically does not use partitioned leaf switches. On the other hand, as noted above, a single-rack configuration may or may not use partitioned leaf switches. Therefore, a fabric definition for a single-rack configuration may also include information about whether the leaf switch configuration used is partitioned. However, a smaller multi-rack configuration created based on expanding an existing single-rack configuration may also use partitioned leaf switches within one or more racks.

[0162] According to one embodiment, both the single rack and multi-rack topologies can support two different embodiments: fully independent rails and semi-independent rails.

[0163] According to one embodiment, in a fully independent rail topology, each rail consists of an independent set of switches and there is no connectivity between switch ports belonging to different rails. A typical use case for this would be having two or more dual-port adapters per server.

[0164] According to one embodiment, in such a case, each server (e.g., host) may have redundant connectivity to each rail. Thus, the lack of a single point of failure for a single adapter or single adapter port for any server means that the corresponding server can This can lead to an inability to send and receive data traffic on individual fabric rails.

[0165] According to one embodiment, if two servers do not both have connectivity to at least one common rail, then a pair of servers (or any pair of VMs - one for each pair of servers) cannot belong to the same logical cluster when fabric-based communication between cluster nodes is required.

[0166] According to one embodiment, when non-overlapping subsets of adapters in each server are connected to non-overlapping sets of rails (i.e., no adapter has ports that connect to more than one of the non-overlapping sets of rails), the different rails are also independent with respect to communication protocols and software and firmware versions, including both switch firmware, fabric management software, adapter firmware, and adapter driver software.

[0167] According to one embodiment, the systems and methods described herein can further support semi-independent rails.

[0168] According to one embodiment, in a semi-independent rail topology, each rail consists of an independent set of switches, and there is no connectivity between switch ports belonging to different rails that would normally be used for data traffic. However, there may be "dormant" physical connectivity between switches in different rails that is used to provide connectivity between pairs of servers that would otherwise be unable to communicate because they do not both have operational connectivity to the same rail. Such connectivity may be achieved by IRLs or by other means.

[0169] According to one embodiment, a typical use case for this configuration is when you typically have only a single dual-port adapter where each adapter port is connected to a leaf switch on a different rail, in which case any single port / link failure for any server means that it cannot send or receive data traffic on the corresponding fabric rail.

[0170] According to one embodiment, if two servers do not both have connectivity to at least one common rail, some of the "dormant" connectivity between switches in different rails may be utilized to re-establish connectivity between this particular pair of servers, or alternatively, the pair of servers (or any pair of VMs - one for each server pair) cannot belong to the same logical cluster when fabric-based communication between cluster nodes is required.

[0171] FIG. 17 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment.

[0172] More specifically, this diagram shows a dual-rail topology with an IRL gateway.

[0173] According to one embodiment, there may be two or more rails, such as rail A 1701 and rail B 1721. Although not shown, each independent rail may include one or more interconnected switches and multicast proxies (MC proxies), such as MC proxy 1702 and MC proxy 1722. Additionally, each rail may include a highly available path service (HAPS), such as HAPS 1703 and HAPS 1723. The rail may be connected to several hosts, such as host 1 1730 through host N 1740. Although not shown, each host may include one or more end nodes connected to the rail via one or more host channel adapters. Additionally, the end nodes may include one or more virtual machines (e.g., utilizing virtual switches, virtual ports, or other similar architectures described above or similar in nature), as described above with respect to the virtualized environment. According to one embodiment, each host may include multipath selection components, such as MP selection 1731 and MP selection 1741.

[0174] According to one embodiment, instead of direct switching for switching connectivity to inter-rail links 1705-1708, several gateway instances 1750 and 1752 can be provided, each gateway instance providing a packet processing engine 1751 and 1753.

[0175] According to one embodiment, a packet processing engine (PPS) may be provided at a node in the topology, such as a gateway node.

[0176] According to one embodiment, to increase the level of independence between redundant fabrics, a dual-port high-performance packet processing engine (PPS) can be used in place of direct switch-to-switch links for both control and data traffic.

[0177] According to one embodiment, these types of packet processing engines can be used in a scalable manner for additional purposes. These PPSs can be used to provide firewalls between different system instances. PPSs can be used to provide gateways to cloud / data center networks to connect different private fabrics. PPSs can be used to provide gateways between IB-based private fabrics and Enet (Ethernet)-based private fabrics. PPSs can be used to provide gateways between private fabrics and client networks.

[0178] According to one embodiment, the system and method can track physical and logical connectivity. This can be achieved by utilizing leaf switch monitoring of connected end ports and inter-switch connectivity. Furthermore, the system and method can utilize a hierarchical query and reporting scheme to distribute all relevant connectivity and power-on information for all relevant end nodes and ports to all relevant peer nodes. Such reporting can also include nodes and ports that have full connectivity to their local leaf switches but limited connectivity in intermediate fabrics.

[0179] Additionally, the system and method can utilize powered-on adapter / NIC firmware / driver and checking schemes to detect and report node power-off (in addition to link failures) to reduce / avoid the need for additional peer-peer checks in addition to facilitating route rebalancing and failover.

[0180] According to one embodiment, the system and method can support multicast, address resolution, and routing. Idempotent multicast operations, such as ARP, can occur in parallel on multiple routes. To ensure "at least once" semantics for multicast, selective replication is allowed or multicast traffic is routed. Traffic can use an extended protocol that allows a receiver to process a single MC message only once. Multiple simultaneous address resolution requests to the same node can be answered by multiple interfaces on different rails, and the requester can then select which rail to use for further communication.

[0181] FIG. 18 is a flowchart of a method for redundant independent networks in a high performance computing environment, according to one embodiment.

[0182] In step 1810, the method may provide, in a computer including one or more microprocessors, one or more switches and one or more racks, each of the one or more racks including a set of one or more switches, each set of one or more switches including at least a leaf switch, and may further provide a plurality of host channel adapters, at least one of the plurality of host channel adapters including firmware and a processor, and further provide a plurality of hosts.

[0183] In step 1820, the method can provision two or more rails, which provide redundant connectivity between multiple hosts.

[0184] In step 1830, the method may separate data traffic between the multiple hosts onto one of two or more rails.

[0185] According to one embodiment, when implementing a highly available cluster network / fabric, it is important that redundancy be implemented in a manner that minimizes the risk of a problem in one area of ​​the fabric propagating to other redundant areas of the fabric.

[0186] Also, according to one embodiment, when recovery or failover actions are required within the fabric, it is important that such actions do not impose a load on either the control plane or data plane infrastructure that could cause significant performance or forward progress problems.

[0187] According to one embodiment, to scale system size and be compatible with legacy high-availability host communication runtime systems, each host can have redundant interface connectivity to the fabric, and each such redundant interface can reach any other interface in the redundant fabric. Notably, this implies that if two hosts each have problems with one interface, they should still be able to communicate using the remaining operational interface. Thus, interface redundancy can be applied to each individual host and is completely independent of which interfaces are available on other hosts.

[0188] According to one embodiment, whenever a host interface or switch fails, it is possible to re-establish such communication independently of which interface was used to initiate the relevant communication. This implies that network-level redundancy cannot be based on two completely independent networks, where packets injected into one of the networks cannot be forwarded to the other network. Therefore, a "semi-independent rails" model can be used to support legacy HA communication schemes while maximizing the independence between redundant networking components.

[0189] Goals for a "semi-independent" HA fabric: According to one embodiment, each redundant host interface should connect to a separate L2 subnet (also known as a "rail") in the HA fabric.

[0190] According to one embodiment, there can be a single broadcast domain that spans two L2 subnets, allowing ARP requests from a single interface to reach all other operational host interfaces, regardless of which L2 subnet each such interface is directly connected to.

[0191] According to one embodiment, data traffic (e.g., RDMA) between hosts should not cross between L2 subnets as long as at least one of the L2 subnets has at least one connected and operational interface for each of the hosts.

[0192] According to one embodiment, whenever any pair of hosts that need to communicate cannot both establish data traffic between interfaces on a single L2 subnet, a route between L2 subnets should be established for the necessary data traffic between the involved hosts.

[0193] According to one embodiment, the host stack on each host can easily determine which interface to use for communication with particular other hosts, even if the default decisions on each host involve different "rails."

[0194] According to one embodiment, in the case of InfiniBand, it should not be possible for a host to initiate an SA request that crosses an L2 subnet boundary.

[0195] According to one embodiment, it should not be possible for a host to cause congestion that spreads from one L2 subnet to another.

[0196] Specific examples of InfiniBand Fabric implementations: HA Path Service (HAPS) According to one embodiment, the "HA Path Service" (HAPS) can track the complete HCA node and port population within each subnet (which may also include system image GUIDs for handling hosts with multiple HCA configurations).

[0197] According to one embodiment, HAPS can use GID in-service / GID out-of-service event notifications from SA or a special protocol. When HAPS is implemented as a host-based service (which may be co-located with MC proxy instances), no special protocol is required by default to track node population, but HAPS has a scope limited by the partitions that its associated HCA ports can be members of.

[0198] According to one embodiment, a HAPS implementation co-located with the master SM may have more direct interaction with the SM and would not be limited to representing only a specific partition.

[0199] According to one embodiment, HAPS can keep track of switches with "cross-link" ports in each L2 subnet and ensure proper connectivity, similar to how an "inter-subnet manager" can ensure proper connectivity between peer router ports.

[0200] According to one embodiment, HAPS can establish unicast LID forwarding for "remote LIDs" that require unicast forwarding over crosslinks. This can, in principle, be done "independently" from the master SM in the associated L2 subnet. The requirement would be that the SM could be instructed (by configuration policy) to use a specific LID, but still set a "LinearFDBTop" value for each switch based on another configuration parameter. In this way, the master SM in each L2 subnet would operate with non-overlapping LID ranges, but the switches in each L2 subnet would still be able to forward unicast packets with DLID values ​​in the ranges belonging to other (redundant) L2 subnets.

[0201] According to one embodiment, the LID range boundaries are determined by the Linear Forwarding Table (LFT). As long as the LFT blocks are aligned on LFT block boundaries, it is possible for the HAPS to update the LFT blocks to manage unicast cross-connectivity independently of (and simultaneously with) the master SM in the local subnet. The actual updates may be performed directly via SMP operations or via a special agent on the switch.

[0202] According to one embodiment, an alternative scheme is for the HAPS to request the local SM to consider the associated cross-link port to represent all remote LIDs currently required for remote connectivity via this cross-link port. (This is similar to the handling of router ports, but this would be an entirely new SM operation, whereas router ports only need a single LID to be handled during the normal subnet discovery / initialization process.) In this embodiment, the local master SM / SA can be provided with a path record for the remote port GID that reflects the associated DLID (similar to how an "inter-subnet manager" provides the local master SM with a path record for a remote port in a router-based context). Without the router port, the SA can look up the path record based on the information provided, but understand that the cross-link port is a "local destination" in the local subnet.

[0203] According to one embodiment, if this unicast cross-link processing is combined with a scheme that does not require a path record query, the local SM / SA does not need to know anything about the route to the remote port GUID.

[0204] Identifying and Handling "Cross-Link" Ports According to one embodiment, by default, redundant InfiniBand L2 subnets may be configured with non-overlapping M_Key ranges before any cross-link connectivity is provided and / or before any SMs are active in either L2 subnet. In this way, the SMs in each redundant subnet will not attempt to discover or configure anything beyond the switch ports in their local subnets that connect to neighboring redundant subnets.

[0205] According to one embodiment, it is not expected that existing operational single subnet-based systems can be converted to a dual L2 subnet configuration without significant interruption to normal operations, and therefore, this type of reconfiguration is expected to occur during maintenance windows when system services are not expected to be operational.

[0206] According to one embodiment, vendor-specific SMA attributes can be used to establish a protocol that allows explicit configuration of switches as belonging to a particular redundant L2 subnet, and also which switch port numbers are assumed to represent cross-link connectivity to peer redundant L2 subnets.

[0207] According to one embodiment, the use of vendor-specific SMA attributes would be similar to how an "inter-subnet manager" handles virtual router ports, however, the implementation would be quite different since there are no router ports or port virtualization in this context.

[0208] According to one embodiment, it is possible to identify and handle cross-link ports without using special vendor-specific SMA attributes, based on detailed configuration information and possibly reliance on node description substrings, but this would be more complex and more subject to configuration errors.

[0209] Specific realizations of Ethernet (Private) Fabric: According to one embodiment, redundancy within the interconnection fabric should ensure as much as possible that failures / problems in one redundant portion do not propagate to other portions. Ultimately, this implies physically and logically independent infrastructures. However, the cost of this is that each node may have redundant connectivity to each such independent fabric, or else the ability to restore connectivity between two or more servers each having a single-link problem is greatly reduced. By utilizing a new method for providing Ethernet link connectivity between two independent subnets without impacting management or fault containment for each individual subnet, it is possible to address both goals simultaneously.

[0210] According to one embodiment, an Ethernet private fabric may be implemented using a traditional Ethernet unicast forwarding scheme with a combination of spanning tree and link aggregation variants, or may be implemented using explicit forwarding of individual unicast destination addresses in the same manner that forwarding of IB packets within a single IB subnet is implemented.

[0211] According to one embodiment, when explicitly forwarding individual unicast addresses, the high level scheme is as follows: According to one embodiment, a complete topology is collected from a set of relevant switches in terms of port-to-port connectivity between switches and ports and between switch ports and end ports. (Optimization may involve only collecting topology deltas after an initial complete discovery.) According to one embodiment, the topology information is converted into a format that can be processed by the same routing logic (also known as a routing engine) used for the corresponding InfiniBand fabric.

[0212] According to one embodiment, the routing engine uses the topology along with the policies over which physical end ports are allowed to communicate (e.g., VLANs) (as well as other related optimization and distribution criteria as in the IB fabric) to generate a list of tuples mapping end port addresses to ports for each switch instance in the topology.

[0213] According to one embodiment, the resulting (delta) forwarding entry list is distributed to switches (ie, those requiring updates).

[0214] According to one embodiment, multicasting may be implemented using legacy Ethernet methods in both cases for end port membership processing and for forwarding of MC packets.

[0215] Multicast Proxy Service According to one embodiment, the same / similar considerations as in the InfiniBand case apply: if the L2 source address in the proxied MC packet is not the same as the corresponding ARP request "sender hardware address," the ARP request may be discarded, or the generation of a unicast ARP reply may result in an incorrect L2 destination address. As in the InfiniBand case, to avoid reliance on special host stack processing, the best approach is for the proxy to be able to send the proxied MC packet with a source L2 address corresponding to the original sender in another L2 subnet.

[0216] HA Path Service (HAPS) According to one embodiment, the "HA Path Service" - HAPS, can track the complete end-port population in each subnet.

[0217] According to one embodiment, ports belonging to the same NIC or the same host can be correlated between two subnets.

[0218] According to one embodiment, it is possible to identify situations where a pair of hosts can communicate only if the associated unicast traffic is forwarded over a cross-link between the two subnets, as in the case of IB.

[0219] According to one embodiment, if explicit end port address forwarding is used for unicast traffic, the processing will be very similar to the IB case. However, for legacy Ethernet unicast forwarding, and in the general case, this may also require a proxy-style gateway function for unicast traffic.

[0220] According to one embodiment, unicast forwarding in the Ethernet fabric is MAC-based, so there are no restrictions on the MAC ranges used for the various L2 subnets - the only constraint is that each host port should have at least a unique MAC within the associated site / domain. In the case of a private fabric, the associated domains are two redundant L2 subnets.

[0221] According to one embodiment, if the switch is capable of performing forwarding based on either L2 and / or L3 addresses, it is also possible to use forwarding based on L3 (IP) addresses instead of L2 MAC.

[0222] Identifying and Handling "Cross-Link" Ports According to one embodiment, based on individual switches being configured to belong to different "rails" combined with neighbor connectivity information from each switch, it is possible to identify both switch-switch connectivity and intended (or accidental...) cross-link connectivity that belongs to the same rail / subnet (fat tree) topology.

[0223] According to one embodiment, cross-link connectivity will never be considered for normal forwarding, either unicast or multicast connectivity, in that case, but the intended cross-connectivity will be used to back up inter-rail paths in the case of explicit end-port address forwarding. As noted above, the use of cross-links when legacy Ethernet unicast forwarding schemes are used would generally require proxy / gateway functionality for both multicast and unicast forwarding between different rails.

[0224] According to one embodiment, when configuring Ethernet switches and NICs for RoCE-based RDMA traffic, the associated links typically have available packet reception Whenever the buffer capacity falls below a certain threshold, the downstream switch or NIC port is configured in a "lossless" mode that generates a "frame stop" to the upstream transmitting switch or NIC port. The transmitting port then stops (pauses) from sending more packets at the associated priority (if any) until the downstream port again has buffer capacity above a certain threshold.

[0225] According to one embodiment, similar to other networking technologies where packet flow control can lead to network-wide backpressure, lossless RoCE fabric topologies must be routed in a deadlock-free manner, and flows that are intended to have forward progress independent of each other must use different priorities.

[0226] According to one embodiment, for dual-rail topologies, it makes sense to ensure that only the internal topology of a single rail must be considered for deadlock-freeness. By configuring the cross-links as "lossy" (i.e., downstream switch ports do not send frame pauses), it is possible to provide deadlock-free topology routing in each rail without considering the topology in the rails connected by the cross-link. Instead, the cross-link ports are considered end ports within the local rail from a deadlock-free routing perspective and therefore cannot be part of cycle dependencies that could lead to deadlock.

[0227] According to one embodiment, the lossless cross-link configuration also means that congestion on one rail does not spread to the other rail, thereby satisfying the key requirement of independence between redundant rails.

[0228] Rail number association According to one embodiment, a private InfiniBand or RoCE physical fabric configuration (as used herein, the terms "fabric" or "private fabric" may refer to a private InfiniBand or RoCE-based fabric) is defined in terms of a particular number of racks, each rack with a set of associated switches, and each such associated switch having a particular role (leaf or spine).

[0229] According to one embodiment, similar to access to client networks from private fabric-based systems in smaller (e.g., quarter rack) configurations, one scheme is to "partition" the leaf switches used to implement the private fabric into one physical partition (i.e., a set of physical ports / connectors) that represents the private fabric and another non-overlapping partition that represents access to the client network (e.g., an on-premises data center network). Each such physically partitioned switch would thus have two non-overlapping sets of ports, where only the set of ports dedicated to the private fabric can represent connectivity within the private fabric (and vice versa).

[0230] According to one embodiment, a multi-rack configuration typically does not use partitioned leaf switches. On the other hand, as noted above, a single-rack configuration may or may not use partitioned leaf switches. Therefore, the fabric definition for a single-rack configuration will also include information regarding whether the leaf switch configuration used is partitioned. However, a smaller multi-rack configuration created based on extending an existing single-rack configuration may use partitioned leaf switches within one or more racks. It can also be used.

[0231] According to one embodiment, to support a fabric configuration with two or more independent networks or "fabric rails," or simply "rails," each switch will also be associated with a rail number. Each such "rail" will thus represent a topology of one or more switches with connectivity and traffic patterns that are independent of other rails. (That is, there will be no links typically used for data traffic between a switch in one rail and another switch in another rail.) According to one embodiment, the private fabric definition facilitates the following high-level features: ● The existence and availability of all defined switches can be automatically checked at runtime and exceptions can be reported immediately. The configuration of each switch (and local switch ports) can be automatically defined and validated according to its role and connectivity within the complete fabric. • The physical connectivity of the complete fabric can be verified according to predefined rules for each type of system. Incorrect and / or degraded connectivity between switches in the fabric (i.e., misplacement of cables and / or lack of a sufficient number of links) can be automatically detected and reported, while preventing incorrect connectivity (e.g., to prevent potential packet forwarding loops or deadlocks) ensures that no additional negative effects are caused to data traffic.

[0232] A special case is the scenario where a link is disabled due to a high error rate. Any connectivity between a defined switch in a private fabric and another switch or other network device that represents an unexpected device in the defined fabric can be reported and not affect the operation of the fabric.

[0233] According to one embodiment, the basic fabric definition outlined above focuses on switch topology and is not related to the number and / or type of servers that are (or may be) part of the fabric. Thus, by default, any number of physical servers may be present in the fabric, up to the maximum number represented by the various fabric configuration types. Furthermore, such servers may be dynamically added and removed without affecting the basic fabric definition. However, such private fabric-based systems are unlikely to be highly dynamic in the sense that physical servers are added, removed, or replaced daily—the physical configuration is typically expected to remain the same over many weeks and months.

[0234] According to one embodiment, based on the expectation that the server configuration will remain stable over a significant period of time, the fabric definition may optionally be extended with parameters that define the expected number and type of servers that will be present in the system and / or in each rack.

[0235] According to one embodiment, this extended fabric definition facilitates the following additional high-level features: ● The existence and availability (in terms of connectivity to the private fabric) of all defined servers can be checked automatically at runtime and exceptions can be reported immediately. Servers (types and / or The presence (in terms of connectivity to the private fabric) of one or more (or multiple) devices can be immediately detected and reported. - Additional policies can determine whether unexpected server types or excess numbers should be accepted for data traffic, but the default would be "report and accept." In a single rack configuration with partitioned leaf switches, the connectivity observed between a server and the private fabric can be correlated with the connectivity observed between the corresponding server and the client network (i.e., as long as a policy exists regarding the connection of servers to the client network).

[0236] o It can identify any server that is connected to a client network but not to a private fabric (and vice versa for servers that are expected to have client network connectivity).

[0237] o In addition to the connectivity between the servers in the local rack and the client network, it is also possible to monitor the uplink connectivity for the client network switch partition.

[0238] Supported fabric topologies: According to one embodiment, for a "single rail" system, the supported topologies can be divided into two main classes: Single rack topology with two leaf switches interconnected by a set of Inter-Switch Links (ISLs): If there is no leaf switch partition: ◆ In general, there may be restrictions on which port numbers can be used for ISLs, depending on switch HW specific constraints on packet buffer allocation and switch HW resource organization.

[0239] With leaf switch division: ◆The total set of switch ports is divided into two partitions.

[0240] Within a RoCE fabric partition, there may again be restrictions on which port numbers can be used for ISLs (by default, the same set of ports is used for ISLs in both partitioned and non-partitioned leaf switches). ◆ In the case of a client network partition, there may be restrictions or defaults on which ports are used for local server connectivity and which ports are used for the uplink. • Multi-rack topology (max N racks) where each rack has at least two leaf switches and two or more spine switches are located in separate racks.

[0241] Each leaf switch in each rack has a set of uplinks that are distributed among all the spine switches in the fabric.

[0242] ◆ Generally, depending on switch HW specific constraints on packet buffer allocation and switch HW resource organization, there may be constraints on which port numbers can be used for uplinks.

[0243] Each spine switch has a certain number of downlinks that are distributed among all the leaf switches in the fabric.

[0244] As long as a spine switch is only supposed to provide downlink connectivity, there is no difference in characteristics between different ports. However, in a particular configuration, not all spine ports may be utilized (connected), so there is still a reason to limit the port numbers that can be used for downlinks.

[0245] According to one embodiment, in the case of a multi-rail topology, each individual fabric rail can correspond to either a single rack or a multi-rack topology, as outlined above. However, within each rack, different leaf switches can belong to different rails, and the full set of spine switches can be divided among the defined rails. To increase the level of redundancy and bisection bandwidth within each rail, the total number of switches across all rails in a multi-rail configuration can be higher than in a corresponding single-rail configuration.

[0246] According to one embodiment, there are two main models for fabric rail independence: fully independent rail and semi-independent rail. ●Fully independent rails: Each rail consists of an independent set of switches, and there is no connectivity between switch ports belonging to different rails.

[0247] A typical use case for this would be having two or more dual port adapters per server.

[0248] In this case, each server can have redundant connectivity to each rail. Therefore, there is no single point of failure for a single adapter or single adapter port for any server that could result in the corresponding server being unable to send and receive data traffic on any fabric rail.

[0249] If the two servers do not both have connectivity to at least one common rail, then a pair of servers (or any pair of VMs - one for each of the pair of servers) cannot belong to the same logical cluster, which requires fabric-based communication between the cluster nodes.

[0250] o If non-overlapping subsets of adapters in each server are connected to non-overlapping sets of rails (i.e., no adapter has ports that connect to more than one of the non-overlapping sets of rails), the different rails are also independent with respect to communication protocols and software and firmware versions, including both switch firmware, fabric management software, adapter firmware, and adapter driver software. Semi-independent rail: Each rail consists of a set of independent switches, and there is no connectivity between switch ports belonging to different rails that is normally used for data traffic. However, there may be "dormant" physical connectivity between switches in different rails that is used to provide connectivity between pairs of servers that would otherwise not be able to communicate because they do not both have operational connectivity to the same rail.

[0251] o A typical use case for this configuration is when each adapter port has only a single dual-port adapter connected to a leaf switch, typically on a different rail.

[0252] ◆ In this case, any single port / link failure for any server means that it cannot send or receive data traffic on the corresponding fabric rail.

[0253] If two servers do not both have connectivity to at least one common rail, some of the "dormant" connectivity between switches in different rails can be utilized to re-establish connectivity between this particular pair of servers, or the pair of servers (or any pair of VMs - one for each pair of servers) cannot belong to the same logical cluster, which requires fabric-based communication between cluster nodes.

[0254] According to one embodiment, each switch may have persistent knowledge of the type of fabric it belongs to and its own role and rail association in that fabric.

[0255] According to one embodiment, each switch may have persistent knowledge of the identity, role, and rail association of every other switch within a defined fabric.

[0256] According to one embodiment, each switch can reflect its fabric-related configuration information to its directly connected peer ports via exported LLDP type information.

[0257] According to one embodiment, each leaf switch in a single rack fabric may have prior knowledge of the required ISL connectivity (if any) to other leaf switches.

[0258] According to one embodiment, each partitioned leaf switch in a single rack fabric tracks which ports belong to which partition and can reflect that partition type as part of its exported LLDP type information.

[0259] According to one embodiment, each leaf switch in a multi-rack fabric may have prior knowledge of the required uplink connectivity to the spine switches in that fabric.

[0260] According to one embodiment, each spine switch in a multi-rack fabric may have prior knowledge of the required downlink connectivity to the leaf switches in that fabric.

[0261] According to one embodiment, each switch can determine the neighbor connectivity of each operational switch port without enabling data traffic through that port.

[0262] According to one embodiment, each switch can dynamically determine whether the discovered neighbor connectivity represents legitimate connectivity for that switch, and if so, how to configure its local ports.

[0263] According to one embodiment, each switch can dynamically configure its switch ports according to pre-defined rules for relevant legal connectivity and synchronize this configuration with its associated peer ports before allowing data traffic through that port.

[0264] According to one embodiment, each switch may share its physical neighborhood connectivity with other switches in the fabric. It can be dynamically shared with all switches.

[0265] According to one embodiment, each switch can monitor its ability to communicate with any other defined switches on the management network.

[0266] According to one embodiment, each switch can dynamically share with all other switches in the fabric a list of other switches with which it can communicate.

[0267] According to one embodiment, each switch should provide log information reflecting changes in physical connectivity, along with the status of such connectivity as valid, incorrect, or degraded, and changes in its ability to communicate with other switches in the fabric over the management network.

[0268] According to one embodiment, at the fabric level there should be coordinated logging and event reporting of fabric-level connectivity issues, including lack of management connectivity between pairs / sets of switches and complete loss of reachability to a particular switch from the rest of the fabric.

[0269] According to one embodiment, critical events such as an unexpected (ie, for longer than the expected reboot period) loss of fabric-level contact with a particular switch should be supported as an Automatic Service Request (ASR).

[0270] According to one embodiment, the leaf switches can monitor physical server connectivity according to policies (if any) regarding expected and required server presence and connectivity in the fabric.

[0271] According to one embodiment, the partitioned leaf switches must be able to monitor client network connectivity to servers according to policies (if any) regarding the expected and required presence of servers in the fabric.

[0272] According to one embodiment, in a multi-rail fabric configuration, each switch can ensure that all normal connectivity is between switches that belong to the same rail.

[0273] According to one embodiment, in the case of a semi-independent multi-rail fabric configuration, any physical connectivity between switches in different rails must be verified as complying with the required inter-rail connectivity, and this connectivity should not then be used for data traffic in the normal case.

[0274] Fabric connectivity check for different switch roles and topologies: According to one embodiment, each individual switch can verify that it has the correct connectivity according to its role and the type of fabric configuration / topology it belongs to: Partitioned leaf switches in a single-rack, single-rail configuration: Within a switch partition, there should be at least N1-1 ports representing ISLs.

[0275] If there are restrictions on which port numbers can be used as ISLs, then all ports that represent ISLs must be in the set of associated ports.

[0276] ◆There should be a maximum of N1-2 ISLs. Within a switch partition, there can be at most N2 ports connecting to host adapter ports.

[0277] Each ISL must connect to a single remote switch with a partitioned leaf switch role, with the peer ports configured as ISLs and the remote switch belonging to the same single fabric instance. (Non-partitioned) leaf switches in a single-rack, single-rail configuration: There should be at least N3-1 ports representing ISLs.

[0278] If there are restrictions on which port numbers can be used as ISLs, then all ports that represent ISLs must be in the set of associated ports.

[0279] ●There should be a maximum of N3-2 ISLs. ◆There can be up to N4 ports connected to a host adapter port.

[0280] ◆Each ISL must connect to a single remote switch that is not partitioned and has the leaf switch role, and the peer port is configured as an ISL, and the remote switch belongs to the same single fabric instance. Leaf switches in a multi-rack single-rail configuration: o There should be at least N5-1 ports representing uplinks.

[0281] ◆ If there are constraints on which port numbers can be used as uplinks, then all ports that represent uplinks must be in the set of associated ports.

[0282] ◆There should be a maximum of N5-2 uplinks. There can be up to N6 ports connected to a host adapter port.

[0283] Each uplink must connect to a switch that has the spine switch role and belongs to the same single fabric instance. There must be uplink connectivity to each spine switch defined for the fabric instance.

[0284] The set of uplinks must be divided evenly among all spine switches defined for the fabric instance. Spine switches in multi-rack single-rail configurations: All connected ports should represent downlinks to leaf switches that are defined to be part of the same fabric instance.

[0285] ◆ If there are constraints on which port numbers can be used as downlinks, then all ports that represent downlinks must be in the set of associated ports.

[0286] There should be no ports connected to host ports. o There can be at least one downlink connecting each leaf switch defined for a fabric instance.

[0287] The set of downlinks must be divided evenly among all leaf switches defined for the fabric instance.

[0288] According to one embodiment, for a multi-rail fabric configuration, the connectivity rules are the same as for the single-rail configuration described above, with the additional rule that all such connectivity must be on the same level. Thus, in the case of a single rack with only two leaf switches, the requirement may be that no ISLs should be connected at all (fully independent rails), or that such ISLs should represent "dormant connectivity" (semi-independent rails).

[0289] According to one embodiment, with respect to server connectivity, connectivity checks can ensure that individual servers have redundant connectivity to the fabric according to a defined single-rail or multi-rail policy.

[0290] According to one embodiment, connectivity that does not comply with the above rules will be logged, reported, and not enabled for data traffic.

[0291] According to one embodiment, lost connectivity is recorded and reported, and the involved switches (or servers) are considered to have degraded (or incomplete) connectivity.

[0292] According to one embodiment, the available connectivity is used to direct data traffic within the fabric according to defined configurations and routing / forwarding policies for the associated topology within a single or multiple fabric rails.

[0293] Example implementation: According to one embodiment, the physical fabric connection control implementation has the following components: ● A switch-based local link monitoring (LLM) daemon tracks local link status, remote connectivity (if any) and local traffic and error counters. ● The switch-based Physical Connectivity Control (PCC) daemon receives local in-band fabric connectivity information from the LLM and, based on this, determines whether the local connectivity complies with the requirements for this switch instance (relative to its role in the overall fabric configuration) and, if so, what high-level configuration type to apply to the associated switch ports. ● The Hardware (HW) Configuration Daemon (HWCD) configures the switch and switch ports (using the switch HW platform specific interface) according to the high level configuration types provided by the PCC for the switch and each connected port with enabled remote connectivity. The Fabric Availability and Synchronization Daemon (FAS) synchronizes fabric configuration information and associated policies among all defined switches in the fabric. The FAS also tracks reachability on the management network for all defined switches in the fabric and coordinates the distribution of this information among the PCC instances on each switch.

[0294] According to one embodiment, based on the persistent fabric configuration information and role information currently defined for the local switch, the PCC constantly evaluates the current local connectivity by correlating the neighbor connectivity information received from the LLM with the expected connectivity defined by the configuration and role information and pre-connectivity rules.

[0295] According to one embodiment, by default, all recently trained links will not be enabled for data traffic, so any connectivity that does not comply with the relevant rules will be left in this state. However, whenever connectivity complies with the relevant rules, the PCC will instruct the HW to configure and enable the port according to the determined legal link type (i.e., hostlink, ISL, uplink, or downlink). Give commands to the CD.

[0296] According to one embodiment, once a newly trained link is evaluated by a PCC, the associated connectivity information is distributed via the FAS to all other PCCs on all other switches in the fabric.

[0297] According to one embodiment, the FASs on all switches in the fabric will constantly monitor whether all other switches are reachable on the management network and will report the current status to the current master FAS.

[0298] According to one embodiment, a PCC co-located with a master FAS evaluates the complete set of in-band and management network connectivity from both its fellow PCC instances and the FASs at any given time. Based on this evaluation, and based on a defined timeout period for lack of connectivity or reachability, the master PCC will generate an alarm event and associated ASR.

[0299] FIG. 19 illustrates a system for supporting rail number association for proper connectivity in a high performance computing environment, according to one embodiment.

[0300] According to one embodiment, system 1900 supports a number of racks, including rack 1 1910, rack 2 1920, rack 3 1930, and rack 4 1940. Each of the racks can support a number of switches, both leaf switches (i.e., switches directly connected to HCAs (not shown)) and spine switches (each switch that is not a leaf switch). These include leaf switches 1912-13, 1922-23, 1932-33, and 1942-43, and spine switches 1911, 1921, 1931, and 1941.

[0301] According to one embodiment, system 1900 is a multi-rail topology where each rail represents a different fabric. In the system shown in Figure 19, there are two independent rails: Rail 1 and Rail 2 (shown in the figure).

[0302] According to one embodiment, to support a fabric configuration having two or more independent networks or "fabric rails," or simply "rails," each switch is also associated with a rail number (e.g., 1 and 2 as shown). Each such "rail" represents a topology of one or more switches with connectivity and traffic patterns that are independent of other rails (i.e., there are no data links typically used for data traffic between a switch in one rail and another switch in the other rail). (However, there may be inter-rail links for management traffic.)

[0303] According to one embodiment, the rails shown are completely independent rails, with each rail consisting of an independent set of switches, and there is no connectivity between switch ports belonging to different rails. A typical use case for this would be having two or more dual-port adapters per server. In this case, each server (not shown) can have redundant connectivity to each rail. Therefore, there is no single point of failure with respect to a single adapter or single adapter port for any server, which could result in the corresponding server being unable to send and receive data traffic on any fabric rail.

[0304] According to one embodiment, if two servers do not both have connectivity to at least one common rail, then the pair of servers (or any pair of VMs - each of the pair of servers) two nodes) cannot belong to the same logical cluster if fabric-based communication between cluster nodes is required.

[0305] According to one embodiment, when non-overlapping subsets of adapters in each server are connected to non-overlapping sets of rails (i.e., no adapter has ports that connect to more than one of the non-overlapping sets of rails), the different rails are also independent with respect to communication protocols and software and firmware versions, including both switch firmware, fabric management software, adapter firmware, and adapter driver software.

[0306] Heterogeneous and asymmetric dual-ray configuration According to one embodiment, systems that use a private RDMA fabric for various types of high performance computing (HPC) related communications, as well as for high performance, high availability databases and other data services, often also have connectivity to different networks that are used, for example, for client access or access to resources outside the system—including resources on the Internet at large.

[0307] According to one embodiment, for example, a special case of such a system is a cloud-based system where the entire cloud may include hundreds of thousands of servers (i.e., hundreds to thousands of server racks) connected via a global cloud network. Within the entire cloud, servers may be organized into smaller islands or points of delivery (PODs) (e.g., 8-32 server racks in close proximity) that are also interconnected by a high-performance RDMA fabric based either on RoCE, InfiniBand, or other implementations of high-performance RDMA. There may be a set of PODs (e.g., a set of datacenters). According to one embodiment, a POD may refer to a module of network, compute, storage, and application components that work together to deliver a network service. A POD may include, for example, a number of racks, servers, switches, networks, and infrastructure. A POD can be designed to be rolled in, plugged in, and quickly up and running. A POD may include, for example, a complete datacenter, or may represent a granular expansion unit in a larger installation.

[0308] According to one embodiment, traditionally, the performance of global cloud networks has been inferior to that achievable via high-performance, dedicated RDMA fabrics. This is partly because the underlying link speeds in cloud networks can be slower than those provided in dedicated RDMA fabrics (e.g., 25 Gb / s vs. 100 Gb / s), but also because, in particular, strict rules for access control and firewall rule enforcement within cloud networks imply schemes involving stateful firewall processing of each individual network packet, resulting in lower overall packet rates for each node and significantly increasing message latency between communicating nodes. However, as “prior art” link speeds increase and the level of integration of on-chip logic functions increases with every new hardware generation, it is increasingly likely that workloads that typically relied on communication over dedicated, private RDMA fabrics will also be able to utilize cloud networks for RDMA-based communication with “reasonable” performance.

[0309] According to one embodiment, another aspect of such a network is scalability, with respect to the number of nodes that can communicate effectively and how flexible nodes can be allocated for a particular distributed workload. A key advantage of a global cloud network is that it allows any node to communicate with any other node, thus providing ultimate node allocation flexibility. However, such a network may require a large number of nodes to communicate with each other, thus providing ultimate node allocation flexibility. Such global connectivity also implies long distances in terms of the number of switch hops, which typically increases message latency and may also increase the risk of performance degradation due to congestion between different flows in the network. Thus, to reduce message latency and the risk of congestion between independent workloads, node allocation algorithms for distributed workloads may typically attempt to allocate nodes in contiguous parts of the network (i.e., adjacent in terms of the number of switch hops).

[0310] However, according to one embodiment, as a result of the ever-increasing level of integration of logic on a single chip, a single switch chip can now provide over 100 100 Gb / s ports interconnected by a non-blocking full crossbar. Thus, in a fully non-blocking fat-tree topology, the number of nodes that can communicate within a distance of only three switch hops is on the order of thousands. As a result, very large PODs within the cloud can be built, with private RDMA fabric connectivity between the cloud network and all servers within the POD, and the maximum distance between any such servers is just three switch hops.

[0311] According to one embodiment, the cloud network infrastructure may be available for workloads that use high-performance RDMA communications, but a private, dedicated RDMA fabric may still have non-trivial performance advantages. Thus, the use of a private RDMA fabric within individual PODs within the cloud can provide a supplementary, higher-performance communications infrastructure in addition to the cloud network, which may make sense for many workloads.

[0312] According to one embodiment, a POD in which servers are connected by both a cloud network and a private RDMA fabric can, in principle, be considered an example of a "fully independent dual-rail" configuration, where the cloud network and private RDMA fabric each represent either a single rail or, potentially, a semi- or even fully independent dual-rail configuration. However, a traditional high-availability configuration in which each server has dual ports connecting to a single switched network infrastructure with dual leaf switches and a shared spine switch infrastructure may be a typical scenario. Furthermore, due to cost constraints and complexity related to the number of switches and cables to provide further redundant server connectivity within each such rail, each such rail can also be implemented in a non-redundant manner, with the server having one adapter and single port connecting to the cloud network and another adapter and single port connecting to the private RDMA fabric.

[0313] According to one embodiment, the cloud network and the private RDMA fabric represent a heterogeneous and asymmetric dual-rail configuration: they both represent full connectivity (i.e., non-degraded) to all servers in the POD, and they both can achieve associated RDMA-based communication between any pair of servers in the POD, although best-case communication performance may not be identical between the two rails. In a typical case, the private RDMA fabric may provide the best (expected) performance and therefore will be the primary choice for RDMA-based communication within the POD.

[0314] However, according to one embodiment, a cloud network may also be selected for RDMA-based communication in the following potential scenarios: a) a pair of nodes requiring RDMA-based communication cannot communicate on the private RDMA fabric; b) RDMA-based communication is required between nodes belonging to different PODs; c) the current relevant load on the private RDMA fabric is (sufficiently) high and the cloud network may not be able to communicate on the private RDMA fabric; d) the current relevant load on the cloud network is (sufficiently) low and the effective communication performance between pairs of nodes within a single POD is better over the cloud network than over the private RDMA fabric; or d) the SLA (Service Level Agreement) for the relevant workload indicates that the use of the premium communication services offered by the private RDMA fabric is not required or permitted.

[0315] According to one embodiment, an implementation of an asymmetric dual-rail system includes various management modules. These may include, for example, a distributed workload placement orchestrator (DWPO) and a connectivity and rail selection manager (CRSM). The DWPO is typically part of an overall cloud workload management infrastructure and is responsible for assigning individual logical nodes within a distributed workload to physical nodes within a cloud / datacenter. Logical nodes may be implemented as containers within the physical node OS, virtual machines controlled by a hypervisor on the physical node, or bare-metal OS / system images on the physical node.

[0316] According to one embodiment, the DWPO can use placement policy information associated with a workload to determine whether placement within a single POD is required (or desired), and if so, what kind of adjacency is required (or desired) between various pairs of logical nodes within the workload. Adjacency parameters and requirements may include latency targets and expected traffic patterns and capacities.

[0317] According to one embodiment, based on correlating the placement policy information for a workload with the current utilization levels and SLAs associated with existing workloads, the DWPO can determine either a best effort or placement with a stringent SLA for the workload being orchestrated. In some cases, this can suggest that the existing workload be reorganized. The placement of the new workload and the potential reorganization of the existing workload may involve a single POD or multiple PODs and PODs with or without a private RDMA fabric, depending on the available resources and associated SLA requirements.

[0318] According to one embodiment, the CRSM has one centralized component that is an integral part of the overall cloud workload management infrastructure and another component that is integrated with the communications runtime system on all logical nodes.

[0319] According to one embodiment, the CRSM, which may be centralized, can cooperate with the DWPO to determine a connection policy for the various logical nodes in the distributed workload regarding primary rail selection and a connection parameter policy for the peer logical nodes with which each logical node is to communicate. In the simplest case, this policy can use default parameters and use the preferred rail (i.e., typically a private RDMA fabric, if present) as long as it provides connectivity for the associated peer nodes. In more complex scenarios, individual or groups of logical node pairs can have explicit rail selection and connection parameter policies for each direction.

[0320] According to one embodiment, each logical node CRSM instance will use policy information from the central CRSM to establish and maintain relevant connections with associated peer nodes. Node-local CRSM instances can also adapt connection setups for individual peer nodes according to currently available connectivity and associated rail load, as well as according to SLA constraints that exist for overall workload and communication with associated individual peer nodes.

[0321] FIG. 20 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment.

[0322] According to one embodiment, several host nodes, such as host node 1 2005 and host node 2 2010, may be supported within system 2000. Each of the host nodes may include several dual port adapters (e.g., dual port network adapters), such as dual port adapters 2006, 2007, and 2008 on host node 1 and dual port adapters 2011, 2012, and 2013 on host node 2. Each dual port adapter may provide two ports, by which the host node may be connected to one or more networks, such as a private fabric 2040 and a cloud fabric 2030, such as a public cloud. As shown, private fabric 2040 includes, for example, two sets of duplicate switches, switch 1 2020 and switch 2 2021, and switch 3 2022 and switch 4 2023, with such pairs of switches representing completely independent rails within the private fabric.

[0323] According to one embodiment, each host node may include multiple dual-port adapters. For example, in the illustrated embodiment, each host node supports three dual-port adapters. Of these three dual-port adapters, two of the dual-port adapters on each host node may be connected to the private fabric 2040, leaving one adapter connected to the cloud fabric 2030. In this way, each host node may be connected to two completely independent rails (i.e., Switch 1 and Switch 2, and Switch 3 and Switch 4, with each pair of duplicate switches representing a completely independent rail) via the private fabric and also via the cloud fabric.

[0324] If necessary, modified versions of this drawing may be reproduced multiple times (e.g., "medium" and "small" below) to show that the private fabric itself can be both semi-independent and fully independent, and that host connectivity can vary from a single adapter with one port connected to the cloud and one port connected to the private fabric, to multiple dual-port adapters with different adapters connecting different, completely independent rails within the private fabric and multiple adapters connecting the cloud.

[0325] According to one embodiment, a determination can be made that any given packet should utilize a private fabric depending on the type of traffic flow within the network. Within the private fabric, two completely independent rails are provided, and each rail may be connected to a separate dual-port adapter at the host node. In some embodiments, a private fabric offers higher and more secure performance than a cloud fabric, but generally has fewer resources, thus making flows within the private fabric more expensive.

[0326] According to one embodiment, the private fabric is fully controlled and redundant. On the other hand, due to the nature of the cloud fabric, it is generally considered not to be fully controlled, but may provide redundant connectivity. According to one embodiment, having at least two fabrics (private and public) provides an asymmetric scheme in which two adapters in each host represent fully redundant connectivity to two rails in the private fabric, and a third adapter provides some (potentially redundant) connectivity in the public cloud. By having redundant connectivity to the cloud, this provides alternative options for traffic flow between hosts.

[0327] According to one embodiment, a workload (e.g., including multiple virtual machines implemented on, for example, the two host nodes shown in FIG. 20 and deployed and configured such that the VMs can communicate with each other) can be deployed within the environment shown in FIG. 20. For a given workload, a policy can be associated with the deployed workload (e.g., a “gold card” workload policy dictates that packets of such workload should utilize a high-level private RDMA fabric, or, for example, an “economy” workload policy dictates that packets of such workload should utilize whatever bandwidth is available (e.g., private if available, public cloud otherwise)). Thus, these different workload policies can be taken into account when determining what type of connectivity should be used for any given workload in the system. For example, if a first workload includes a high “gold card” workload policy, packets in that workload could be strictly allocated and provisioned within the private fabric. Alternatively, if another workload includes a lower "economy" workload policy, packets associated with such workload may attempt to utilize the private fabric but may also utilize the public cloud for connectivity.

[0328] FIG. 21 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment.

[0329] According to one embodiment, several host nodes, such as host node 1 2105 and host node 2 2110, may be supported within system 2100. Each of the host nodes may include several dual port adapters (e.g., dual port network adapters), such as dual port adapters 2106 and 2108 on host node 1 and dual port adapters 2111 and 2113 on host node 2. Each dual port adapter may provide two ports, by which the host node may be connected to one or more networks, such as a private fabric 2140 and a cloud fabric 2130, such as a public cloud. As shown, private fabric 2140 includes, for example, a set of duplicate switches, switch 1 2120 and switch 2 2121, where each switch represents a semi-independent rail within private fabric 2140.

[0330] According to one embodiment, each of the host nodes may include multiple dual-port adapters. For example, in the illustrated embodiment, each host node supports two dual-port adapters. Of these two dual-port adapters, one of the dual-port adapters on each host node may be connected to the private fabric 2140, while the other adapter may remain connected to the cloud fabric 2130. In this manner, each host node may be connected to two semi-independent rails (i.e., Switch 1 and Switch 2) via the private fabric and also via the cloud fabric.

[0331] FIG. 22 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations according to one embodiment.

[0332] According to one embodiment, several host nodes may be supported within system 2200, such as host node 1 2205 and host node 2 2210. Each of the host nodes may include a number of dual port adapters (e.g., dual port network adapters), such as dual port adapter 2206 on host node 1 and dual port adapter 2211 on host node 2. Each dual port adapter The adapter may provide two ports by which the host node may be connected to one or more networks, such as a private fabric 2240 and a cloud fabric 2230, such as a public cloud. As shown, the private fabric 2240 includes, for example, a switch 2220, which represents a single rail within the private fabric 2240.

[0333] According to one embodiment, each host node may include multiple dual-port adapters. For example, in the illustrated embodiment, each host node supports one dual-port adapter. Within this dual-port adapter, one port may be connected to the private fabric and represent one rail, and the other port may be connected via the cloud fabric and represent another rail.

[0334] 23 is a flowchart of a method for determining a placement policy for a provisioned or provisioned workload in a system for supporting heterogeneous and asymmetric dual-rail fabric configurations. According to one embodiment, a workload may include a number of hosts, and the hosts may include a number of physical hosts and / or a number of virtual machines.

[0335] According to one embodiment, in step 2300, a provisioned workload (either a workload that has already been provisioned or a workload that is being provisioned) may be provided.

[0336] According to one embodiment, in step 2310, the workload may be examined to determine the placement policy associated with the provisioned workload.

[0337] According to one embodiment, in step 2315, the method may determine whether the workload's placement policy specifies strict connectivity within a single POD.

[0338] According to one embodiment, in step 2320, if the placement policy dictates strict placement within a single pod, then compute nodes within that single pod may be assigned to the provisioned workload.

[0339] According to one embodiment, in step 23250, if the placement policy does not dictate strict placement within a single POD, compute nodes may be assigned to workloads without POD considerations.

[0340] FIG. 24 is a flowchart of a method for determining a fabric type selection for a particular communication within a provisioned or being provisioned workload, according to one embodiment.

[0341] In step 2400, according to one embodiment, the method can determine whether any of a set of conditions are met, the set of conditions including: 1) is the workload prohibited from using the private fabric?; 2) does the peer (i.e., target) node belong to a different POD?; 3) is the peer node not currently available on or through the private fabric?; 4) is the current load on the private fabric higher than a set threshold and the load on the cloud fabric lower than another set threshold?

[0342] According to one embodiment, at 2410, the method may determine whether any of the conditions of step 2400 are true.

[0343] According to one embodiment, if one or more of the conditions of step 2400 are true, the method may, in step 2420, set up communication to the target node (peer node) via the cloud fabric.

[0344] According to one embodiment, if none of the conditions of step 2400 are true, the method may proceed to step 2430 to set up communication to the target node (peer node) over the private fabric.

[0345] FIG. 25 is a flowchart of a method for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment.

[0346] In step 2510, the method may provide, in a computer including one or more microprocessors, a plurality of hosts, each of the plurality of hosts including at least one dual-port adapter, and further provide a private fabric including two or more switches and a public fabric including a cloud fabric.

[0347] In step 2520, the method may provision a workload on a host of the plurality of hosts.

[0348] In step 2530, the method can assign placement policies to the provisioned workloads.

[0349] In step 2540, the method may allocate network traffic between peer nodes of the provisioned workload to one or more of the private and public fabrics according to a placement policy.

[0350] Thus, from one aspect, a system and method for supporting heterogeneous and asymmetric dual-rail fabric configurations in a high-performance computing environment have been described. A method may provide for a system including a plurality of hosts in one or more computers, each including one or more microprocessors, each of the plurality of hosts including at least one dual-port adapter, and further including a private fabric including two or more switches and a public fabric including a cloud fabric. A workload may be provisioned on a host of the plurality of hosts. A placement policy may be assigned to the provisioned workload. Network traffic between peer nodes of the provisioned workload may then be allocated to one or more of the private fabric and the public fabric according to the placement policy.

[0351] Multicast Send Replication According to one embodiment, in telecommunications systems, the issue of high availability has been a critical issue for decades, and the concept of a "carrier-grade system" suggests a system with very low failover times before service is restored. One technique used in such systems is to replicate all network traffic on dual independent networks based on protocol extensions that allow the receiver to filter incoming traffic so that only one logical packet is forwarded up the network stack even if two copies are received. In the context of very high-bandwidth lossless interconnection fabrics, such replication is impractical in the general case due to excessive usage of both network bandwidth in the fabric and IO bandwidth in the involved servers. However, selective use of such techniques for multicast traffic can eliminate explicit multicasting between independent subnets. It is conceivable that this would significantly reduce the need for multicast replication or encountering timeout situations due to individual multicast messages being lost in the network. Importantly, this can be achieved while preserving the important semantics that a single multicast message is received "at most once" by each potential receiver.

[0352] According to one embodiment, a sender of a multicast message classifies a multicast sending behavior based on whether the multicast message represents idempotent behavior, which has no negative effect if a receiving node should process more than a single copy of the multicast message being sent. In this case, two or more independent multicast messages may be sent over one or more local interfaces and target one or more individual multicast addresses. However, if the message does not represent idempotent behavior, multiple copies of the same message can only be sent as long as the message is encapsulated in a network packet with an encapsulation header that uniquely identifies the particular logical message instance. The identifier in the encapsulation header represents a monotonically increasing number unique to the sender, so that the receiver can ensure that subsequently received messages with the same identification number are not forwarded to higher levels of the protocol stack and / or any receiving applications.

[0353] According to one embodiment, one aspect of this scheme is that in the case of an application-level protocol with multicast-based requests and unicast-based responses, the same encapsulation scheme can also be used for unicast responses, allowing the responder to send multiple copies in parallel without the risk of confusing the requester with multiple copies. Alternatively, the responder can send only a single regular unicast response, in which case it is up to the requester to send a new multicast request with a new instance number in the encapsulation header. In this case, it is up to the client / application-specific protocol to take care of duplicate messages due to retries at the client / application layer.

[0354] According to one embodiment, for multicast traffic that is not classified as either idempotent or not, the default policy may be to use an encapsulation scheme (i.e., the default logical network interface provides this functionality by default). In this case, different logical interfaces may be provided for clients / applications that are aware of the alternatives and are able to perform the correct classification.

[0355] According to one embodiment, alternatively, the default interface may provide legacy functionality, and encapsulated traffic would then only be provided to clients / applications that explicitly request this functionality via a special logical interface.

[0356] Example implementation: According to one embodiment, a sending node may have a persistent "system image generation number" that is incremented and persistently stored each time the associated node boots. In addition, each sending node will have a "current message sequence number" that is incremented for every message sent over a logical interface, representing an encapsulation and duplication. That is, the current message sequence number is the same for each duplicated message in a set of duplicated messages. The message identification in the encapsulation header is then the combination of the "system image generation number" and the "current message sequence number." Then, if a receiving node receives at least two copies of the same message (both duplicates), it can ignore the later-arriving message and discard it.

[0357] According to one embodiment, only one sequence number is generated for a replicated message, regardless of how many replicates are used. However, it is possible to extend this by leaving some bits (e.g., only one bit) to encode the rail number on which the message was sent. This can have some value in evaluating traffic and also facilitates more information regarding the potential use of the inter-rail link between the two rails. However, to prevent multiple versions from being delivered to a receiving application, the receiver needs to recognize two (or more) replicated versions of the same message as replicates.

[0358] According to one embodiment, a sending node may replicate a multicast message and transmit each copy of the multicast message on each of two or more independent or semi-independent rails to which the sending node has access.

[0359] According to one embodiment, if the persistent "System Image Generation Number" is lost for an associated sending node, special procedures are performed to ensure that all potential receivers are notified and can reset any expected current identification numbers for the associated sending node. In the context of a high-availability cluster configuration, handling of the "System Image Generation Number" may typically be part of the procedure for including a node in a logical cluster. Thus, unless a node is included in the cluster, other nodes will not accept any incoming multicast packets from that node, regardless of the "System Image Generation Number" to which it is associated. The cluster membership processing protocol may decide to remove a node from the cluster for various reasons, particularly if the node can no longer be communicated with (e.g., because it has crashed for some reason or is being rebooted). In this case, the cluster membership decision may be communicated to the remaining cluster members, suggesting that communication with the removed node, including multicast communication, be immediately stopped. Each time an excluded node reboots, or attempts to (re)join the cluster for any reason, it can negotiate its "system image generation number" as part of the associated cluster membership joining protocol. In this way, it can present its current next generation number based on having retrieved it from persistent storage, or it can obtain a new generation number from the cluster membership control system if a new generation number is not readily available locally to it. In either case, the cluster membership control system can ensure that conflicting generation numbers are not available.

[0360] According to one embodiment, each receiver can maintain dynamic state information for each sender, where the sending node's ID is maintained along with the currently expected message identifier. Arriving messages with encapsulation headers representing higher numbers will be accepted and forwarded, while messages with identifiers already received (i.e., values ​​lower than the currently expected value) will be discarded.

[0361] According to one embodiment, the protocol may use an alternative protocol identifier at the basic packet layer (e.g., using a special packet / protocol type field at the data link layer) to more easily separate packets representing encapsulated and replicated messages from standard network traffic. Alternatively, a set of dedicated multicast addresses may be allocated for the purpose of implementing this replication protocol, and then the original multicast addresses may be used. The multicast address can be conveyed to be included in the encapsulation header.

[0362] 26 illustrates a system for multicast transmission replication instead of replication in a high performance computing environment, according to one embodiment. Notably, this figure illustrates such an implementation on one rail, with the other rail not shown.

[0363] According to one embodiment, system 2600 may include several switches, such as spine switches 2611, 2621, 2631, and 2641, and leaf switches 2612-13, 2622-23, 2632-33, and 2642-43, which may interconnect a number of nodes 2650-2653.

[0364] According to one embodiment, the sending node 2650 may include a persistent "system image generation number" 2654 that is incremented and persistently stored each time the node boots. Additionally, the sending node 2650 may include a "current message sequence number" 2655 that is incremented for each set of multicast messages sent over the logical interface, representing encapsulation and duplication. Multicast packets 2655 sent and addressed to a multicast address (e.g., MGID) may include an encapsulation header that represents the combination of the "system image generation number" and the "current message sequence number."

[0365] According to one embodiment, each receiver may maintain dynamic state information about the sending node 2650, where the sending node's ID is maintained along with the currently expected message identifier. An arriving message with an encapsulation header representing a higher number will be accepted and forwarded (since it represents a new message), while a message with an identifier already received or a value lower than the expected message identifier will be discarded because the encapsulation header represents a message that has already been received.

[0366] FIG. 27 is a flowchart of a method for multicast transmission replication instead of replication in a high performance computing environment.

[0367] According to one embodiment, in step 2710, a method may provide a plurality of switches and a plurality of hosts, the plurality of hosts being interconnected via a plurality of switches, and a host among the plurality of hosts including a multicast sender node, the sender node including a system image generation module and a current message sequence module.

[0368] According to one embodiment, in step 2720, the method may organize multiple switches into two rails, where the two or more rails provide redundant connectivity between multiple hosts.

[0369] According to one embodiment, in step 2730, a method can include sending, by a multicast sender node, two duplicate multicast packets addressed to a multicast address, each of the two or more duplicate multicast packets being sent on a different one of two rails. A receiving node can receive two versions of the same multicast packet, but can deliver only one to a communication stack / client above the layer that handles the encapsulation header.

[0370] According to one embodiment, in step 2740, when a host of the plurality of hosts receives two or more of the two or more duplicate multicast packets, the method further comprises: You can drop everything except the

[0371] A single logical IP subnet across multiple disjoint Layer 2 (L2) subnets According to one embodiment, the system and method can provide a virtual single IP subnet across physically separate L2 subnets by allowing multiple end ports to simultaneously represent the same IP address.

[0372] According to one embodiment, a highly available network interface model is based on link aggregation (LAG). In this model, the host stack sees a single logical port, and the underlying driver software and adapter and switch firmware ensure that different flows are load-balanced across available links, and that flows currently mapped to a failed link are taken over by the remaining operational links. By introducing switches that support multi-chassis LAG (MLAG), this HA model can be applied across multiple switches, implying no single point of failure whatsoever. However, this model also implies tight integration between redundant switches / networks, thus violating the goal of independent networks. An alternative model is to not use LAG protocols at all, and instead allow host-side interfaces to realize a single IP address over multiple local network ports. In this way, outgoing traffic can use any port in the associated logical group, and incoming traffic for a particular destination IP address will be accepted on any port in the group. If the networks involved are completely independent, host-side operations may have to select a different source port (a.k.a., a different rail) to retry a message.

[0373] According to one embodiment, the interface model presented to the generic host stack may still be the same as in the LAG / MLAG case (ie, there is a single local network port with a single source IP address).

[0374] According to one embodiment, a single local IP (L3) address is then associated with the hardware address of each local end port.

[0375] According to one embodiment, the interface driver tracks whether each local end port is connected to the same L2 subnet or separate L2 subnets and can handle both outgoing and incoming Address Resolution Protocol (ARP) requests accordingly. For outgoing ARP requests, the interface driver can send one multicast request through each local port. For incoming ARP requests, the interface driver will send a single unicast ARP reply through the local port on which the request arrived. The hardware address of the ARP reply sent out on the network will always be the hardware address of the corresponding local end port.

[0376] According to one embodiment, the hardware address provided to the local host stack along with the remote IP address is a special value that the local interface driver can use to look up the "real" HW address used for the remote IP address via one or more of the local end ports.

[0377] According to one embodiment, the local end port and remote hardware address used for the remote IP address are dynamic entities that the local interface driver can dynamically adjust over time to achieve both optimal availability and optimal load balancing.

[0378] FIG. 28 illustrates a system for supporting link aggregation (LAG) according to one embodiment.

[0379] According to one embodiment, a network packet such as packet 1 2801 may be destined for a logical device such as logical device 1 2810 (e.g., a link aggregation device). Before passing through the logical device, the packet may include or be associated with (e.g., in the packet header) a local source IP address (SIP1), a destination IP address (DIP1), and a destination MAC (Media Access Control) address (DMAC1).

[0380] According to one embodiment, a logical device may be associated with its own address, such as an IP address, IP1, which may include, for example, a layer 3 address.

[0381] According to one embodiment, the logical device may further include one or more physical network adapter devices (not shown), each of which may have one or more physical ports (AP1, AP2...APn), each of which may have its own MAC (L2) address (not shown), but may have a single IP address (IP1) of the logical device 2810.

[0382] According to one embodiment, packet 2 2802, after passing through the logical device, may include, for example in the header, a local source IP address (SIP2) and a destination IP address (DIP2) in addition to a destination MAC address (DMAC2). In an embodiment, in a LAG situation, these addresses may be the same as the packet before passing through the logical device, i.e., SIP1 is the same as SIP2, DIP1 is the same as DIP2, and DMAC1 is the same as DMAC2.

[0383] According to one embodiment, the logic device determines which port / rail on switch 1 2820 (eg, SP1-1 through SP1-6) to send packet 2 to its destination port based on, for example, load balancing.

[0384] FIG. 29 illustrates a system for supporting multi-chassis link aggregation (MLAG), according to one embodiment.

[0385] According to one embodiment, a network packet such as packet 1 2901 may be destined for a logical device such as logical device 1 2910 (e.g., a link aggregation device). Before passing through the logical device, the packet may include or be associated with (e.g., in the packet header) a local source IP address (SIP1), a destination IP address (DIP1), and a destination MAC (Media Access Control) address (DMAC1).

[0386] According to one embodiment, a logical device may be associated with its own address, such as an IP address, IP1, which may include a layer 3 address.

[0387] According to one embodiment, the logical device may further include one or more physical network adapter devices (not shown), each having one or more physical ports (AP1, AP2...APn), each of which may have its own MAC (L2) address (not shown), but may share a single IP address (IP1) of the logical device 2910.

[0388] According to one embodiment, packet 2 2902, after passing through the logical device, may include, for example in the header, a local source IP address (SIP2) and a destination IP address (DIP2) in addition to the destination MAC address (DMAC2). In an embodiment, in a LAG situation, these addresses may be the same as the packet before passing through the logical device, i.e., SIP1 is the same as SIP2, DIP1 is the same as DIP2, and DMAC1 is the same as DMAC2.

[0389] According to one embodiment, the logic device determines, for example, based on load balancing, which port / rail to send packet 2 to its destination port among ports on switch 1 2920 (e.g., SP1-1 to SP1-6) or ports on switch 2 2930 (e.g., SP2-1 to SP2-6).

[0390] According to one embodiment, in an MLAG scenario, the switches, Switch 1 and Switch 2, may further include one or more control links between them, which allow the multiple switches to appear as a single entity, yet still provide more redundancy than the LAG embodiment in which only one switch is provided.

[0391] FIG. 30 illustrates a system for supporting a single logical IP subnet across multiple independent layer 2 subnets in a high performance computing environment, according to one embodiment.

[0392] According to one embodiment, a network packet such as packet 1 3001 may be destined for a logical device such as logical device 1 3010 (e.g., a link aggregation device). Before passing through the logical device, the packet may include or be associated with (e.g., in the packet header) a local source IP address (SIP1), a destination IP address (DIP1), and a destination MAC (Media Access Control) address (DMAC1).

[0393] According to one embodiment, a logical device may be associated with its own address, such as an IP address, IP1, which may include, for example, a layer 3 address.

[0394] According to one embodiment, the logical device may further include one or more physical network adapter devices (not shown), each having one or more physical ports (AP1, AP2...APn), each of which may have their own MAC (L2) address (not shown), and each of which may have their own network-specific IP address (not shown) and rail number association (not shown).

[0395] According to one embodiment, packet 2 3002, after passing through the logical device, may include, for example, in the header, a local source IP address (SIP2) and a destination IP address (DIP2) in addition to a destination MAC address (DMAC2).

[0396] According to one embodiment, based on the mapping function 3011, the destination IP (DIP2) and destination MAC address (DMAC2) may contain new addresses based on a lookup of the input destination IP (DIP1) and input destination MAC address (DMAC1) in the mapping function. The source IP address may remain the same, i.e., SIP1 may be the same as SIP2.

[0397] According to one embodiment, based on the mapping function, the IP addresses of switches 1 and 3 are then mapped from a single Layer 3 address (IP1 of the logical device) to the IP addresses of switches 1 and 3, which can be seen as an MLAG scenario. The packet can be routed to multiple layer 2 subnets, such as the subnets represented by Switch 3 3020 and Switch 2 3030, as well as Switch 3 3040, which can be viewed as a LAG scenario.

[0398] According to one embodiment, in an MLAG scenario, the switches, Switch 1 and Switch 2, may further include one or more control links between them, which allow the multiple switches to appear as a single entity, yet still provide more redundancy than the LAG embodiment in which only one switch is provided.

[0399] According to one embodiment, the system shown shows a single logical device with an L3 address supporting two L2 subnets, one via MLAG and one via LAG, however, those skilled in the art will readily appreciate a variety of different scenarios in which a single logical device supports multiple L2 subnets, such as n MLAG subnets and m LAG subnets, where the sum of n and m is greater than 2.

[0400] According to one embodiment, the logic device determines, for example, based on load balancing, which rail / port to send packet 2 to its destination port among ports on switch 1 3020 (e.g., SP1-1 to SP1-6) or ports on switch 2 2300 (e.g., SP2-1 to SP2-6).

[0401] According to one embodiment, a system such as that described in Figure 30 can support an outbound mapping function. In such a situation, the system and method can look up a destination description based on the logical destination MAC. The system and method can then obtain the current physical device and port and the remote physical IP and physical MAC from the destination description. The packet header can then be updated with the local and remote physical addresses before the packet is sent.

[0402] According to one embodiment, a system such as that described in Figure 30 can support an incoming mapping function. In such a situation, the method and system can look up a logical device description associated with the physical device and port on which the incoming packet is received. The method and system can then verify that the correct physical destination address for the receiving physical port is included in the packet header. The packet header can then be updated to reflect the logical IPP of the local device before the packet is forwarded to the generic IP level in the host stack.

[0403] According to one embodiment, a system such as that described in Figure 30 can support outgoing ARP (Address Resolution Protocol) requests. In such a situation, the system and method can generate an ARP request multicast packet for each physical port associated with the logical device. The system and method can then fill in the associated source address of the corresponding physical port in each ARP request multicast packet. The system and method can then transmit each ARP request multicast packet from the corresponding physical port.

[0404] According to one embodiment, a system such as that described in Figure 30 can support incoming ARP (Address Resolution Protocol) requests. The system and method can verify that the requested IP of an incoming ARP request on a physical port corresponds to the logical IP of the logical device with which the physical port is associated. The system and method can then send an ARP response on the physical port on which the ARP request was received, where the ARP response reflects the logical device IP address as the sender IP address and the sender hardware address includes both the physical port MAC address and the physical port IP address.

[0405] According to one embodiment, a system such as that described in FIG. 30 can support incoming ARP (Address Resolution Protocol) responses. The system and method can verify that the target IP and HW addresses correspond to the receiving physical port's physical IP and physical MAC addresses. The system and method can record the physical IP and MAC addresses from the sender hardware address of the ARP response and associate them with the destination logical IP address identified by the sender IP address in the ARP response. The system and method can then generate a logical MAC address to represent the mapping of the logical destination IP address to the associated physical port and destination physical IP and MAC addresses. The ARP table can be updated with the logical IP address and the associated logical MAC address.

[0406] FIG. 31 is a flowchart of a method for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, according to one embodiment.

[0407] In step 3110, the method may provide, in a computer including one or more microprocessors, a logical device, the logical device being addressed by a Layer 3 address, the logical device including a plurality of network adapters, each of the network adapters including a physical port, and the method may further provide a plurality of switches.

[0408] In step 3120, the method can arrange the switches into multiple separate layer 2 subnets.

[0409] In step 3130, the method may provide a mapping table in the logical device.

[0410] Thus, from one aspect, a system and method for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high-performance computing environment are described. A method can provide a logical device in a computer including one or more microprocessors, the logical device being addressed by a Layer 3 address, the logical device including multiple network adapters, each of the network adapters including a physical port, and the method can further provide multiple switches. The method can arrange the multiple switches into multiple separate Layer 2 subnets. The method can provide a mapping table in the logical device.

[0411] While various embodiments of the present invention have been described, it should be understood that the above embodiments are presented by way of example and not limitation. The above embodiments have been selected and described in order to explain the principles of the taught techniques and their practical application. The above embodiments illustrate systems and methods in which the techniques taught herein are utilized to enhance the performance of the systems and methods by providing new and / or improved features and / or by providing benefits such as reduced resource utilization, increased capacity, improved efficiency, and reduced latency.

[0412] In some embodiments, features of the present teachings are implemented, in whole or in part, in a computer that includes a processor, a storage medium such as memory, and a network card for communicating with other computers. In some embodiments, features of the present teachings are implemented in a computer system where one or more clusters of computers are connected over a local area network (LAN). Networks such as LANs, switched fabric networks (e.g., InfiniBand), or wide area networks (WANs) provide The distributed computing environment may have all the computers in one location, or may have a cluster of computers in various remote geographic locations connected by a WAN.

[0413] In some embodiments, features of the present teachings are implemented in whole or in part in a cloud as part of a cloud computing system or as a service, based on shared, elastic resources delivered to users in a self-service, coordinated manner using web technologies. There are five characteristics of the cloud (as defined by the National Institute of Standards and Technology): on-demand self-service, wide-area network access, resource pooling, rapid elasticity, and measured services. Cloud deployment models include public, private, and hybrid. Cloud service models include Software as a Service (SaaS), Platform as a Service (PaaS), Database as a Service (DBaaS), and Infrastructure as a Service (IaaS). As used herein, cloud is a combination of hardware, software, network, and web technologies that delivers shared, elastic resources to users in a self-service, coordinated manner. Unless otherwise specified, cloud, as used herein, encompasses public cloud, private cloud, and hybrid cloud embodiments, and all cloud deployment models, including, but not limited to, cloud SaaS, cloud DBaaS, cloud PaaS, and cloud IaaS.

[0414] In some embodiments, features of the present teachings are implemented using or with the aid of hardware, software, firmware, or a combination thereof. In some embodiments, features of the present teachings are implemented using a processor configured or programmed to perform one or more functions of the methods taught herein. The processor, in some embodiments, may be a single or multi-chip processor, a digital signal processor (DSP), a system on a chip (SOC), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a DMA controller (DMAC ... Field programmable gate array (FPGA) or other programmable A logic device, a state machine, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. In some implementations, features of the present teachings may be implemented by circuitry specialized for a particular function. In other implementations, the features may be implemented in a processor configured to perform a particular function using instructions stored on a computer-readable storage medium, for example.

[0415] In some embodiments, features of the present teachings are incorporated into software and / or firmware to control the hardware of a processing system and / or networking system, and to enable the processor and / or network to interact with other systems that utilize features of the strategies taught herein. Such software or firmware may include, but is not limited to, application code, device drivers, operating systems, virtual machines, hypervisors, application programming interfaces, programming languages, and execution environments / containers. Appropriate software coding can be performed by those skilled in the art of software. As will be apparent to those skilled in the art, skilled programmers can readily prepare such programs based on the teachings of the present disclosure.

[0416] In some embodiments, the present teachings include a computer program product, which is a storage medium or computer-readable medium having instructions stored thereon. These instructions can be used to program or otherwise configure a system, such as a computer, to perform any of the processes or functions of the methods taught herein. The storage medium or computer-readable medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, and magneto-optical disks, ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), and any type of medium or device suitable for storing instructions and / or data. In certain embodiments, the storage medium is a non-transitory storage medium or a non-transitory computer-readable medium. In some embodiments, the computer-readable medium includes a transitory medium, such as a transmission signal or carrier wave. Such transitory medium may occur as part of a transmission between components of a single computer system and / or may occur as part of a transmission between multiple computer systems.

[0417] The above description is not intended to be exhaustive or to limit the present teachings to the precise form disclosed. Furthermore, while embodiments of the present teachings have been described using a particular series of transactions and steps, it will be apparent to those skilled in the art that the scope is not limited to the series of transactions and steps described above. Furthermore, while embodiments of the present teachings have been described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are within the scope. Furthermore, while specific combinations of features of the present teachings have been described in various embodiments, it should be understood that different combinations of these features are within the scope of the present disclosure, such as features from one embodiment being incorporated into another embodiment. Furthermore, it will be apparent to those skilled in the art that various additions, deductions, deletions, modifications, and other changes in form, details, implementation, and application can be made without departing from the spirit and scope of the claimed subject matter. The broader spirit and scope of the present invention is intended to be defined by the appended claims and their equivalents.

[0418] Aspects of the present disclosure are described in the following numbered clauses: 1. A system for supporting heterogeneous and asymmetric dual-rail configurations in a high performance computing environment, comprising: a computer including one or more microprocessors; a plurality of hosts, each of the plurality of hosts including a dual port adapter, the system further comprising: a first rail including a private fabric, the private fabric including a plurality of switches disposed in two or more racks, each of the two or more racks including a set of the plurality of switches; and the system further comprising: a second rail including a public fabric, the public fabric including a cloud fabric, the system further comprising: a distributed workload placement orchestrator; a host of the plurality of hosts is connected to the first rail via a first port of the dual port adapter of the host; the one of the plurality of hosts is connected to the second rail via a second port of the dual port adapter of the one of the hosts; a workload is provisioned on the one of the plurality of hosts; A placement policy is assigned to the provisioned workload; The system, wherein the distributed workload placement orchestrator utilizes the placement policy assigned to the provisioned workload to allocate network traffic of the provisioned workload to one or more of the first rail including the private fabric and the second rail including the public fabric in accordance with the placement policy.

[0419] 2. A system according to clause 1, The private fabric provides two or more rails, which provide redundant connectivity for the provisioned workloads within the private fabric.

[0420] 3. A system according to clause 2, The placement policy assigned to the provisioned workload is a high priority placement policy.

[0421] 4. A system according to clause 3, The network traffic of the provisioned workload is strictly allocated within the private fabric.

[0422] 5. A system according to any one of the preceding clauses, The public fabric provides two or more rails, which provide redundant connectivity for the provisioned workloads within the public fabric.

[0423] 6. A system according to clause 5, The placement policy assigned to the provisioned workload is a low priority placement policy.

[0424] 7. A system according to clause 6, Network traffic for the provisioned workload is allocated within the public fabric and the private fabric.

[0425] 8. A method for supporting redundant independent networks in a high performance computing environment, comprising: In a computer including one or more microprocessors, providing a plurality of hosts, each of the plurality of hosts including a dual port adapter; the method further comprising, in a computer including the one or more microprocessors: providing a first rail including a private fabric, the private fabric including a plurality of switches arranged in two or more racks, each of the two or more racks including a set of the plurality of switches; and the method further comprising: providing a second rail including a public fabric, the public fabric including a cloud fabric; and the method further comprising: providing a distributed workload placement orchestrator, the method further comprising: connecting a host of the plurality of hosts to the first rail via a first port of the dual port adapter of the host; connecting the one of the plurality of hosts to the second rail via a second port of the dual port adapter of the one host; provisioning a workload on the one of the plurality of hosts; assigning a placement policy to the provisioned workload; and based on the placement policy assigned to the provisioned workload, the distributed workload placement orchestrator assigns network traffic of the provisioned workload to one or more of the first rail including the private fabric and the second rail including the public fabric in accordance with the placement policy.

[0426] 9. The method of clause 8, The private fabric provides two or more rails, which provide redundant connectivity for the provisioned workload.

[0427] 10. The method of clause 9, The placement policy assigned to the provisioned workload is a high priority placement policy.

[0428] 11. The method of clause 10, Network traffic between peer nodes of the provisioned workload is strictly provisioned within the private fabric.

[0429] 12. Any one of the methods set forth in clauses 8 to 11, The public fabric provides two or more rails, which provide redundant connectivity for the provisioned workloads.

[0430] 13. A method according to clause 12, The placement policy assigned to the provisioned workload is a low priority placement policy.

[0431] 14. A method according to clause 13, Network traffic between peer nodes of the provisioned workload is allocated within the public fabric.

[0432] 15. A computer-readable medium bearing instructions for supporting heterogeneous and asymmetric dual-rail configurations in a high performance computing environment, the instructions, when read and executed by a computer, causing the computer to perform steps including: In a computer including one or more microprocessors, providing a plurality of hosts, each of the plurality of hosts including a dual port adapter; and the step further comprising: in the computer including the one or more microprocessors: providing a first rail including a private fabric, the private fabric including a plurality of switches arranged in two or more racks, each of the two or more racks including a set of the plurality of switches; and the step further includes, in the computer including the one or more microprocessors: providing a second rail including a public fabric, the public fabric including a cloud fabric, the step further comprising: in the computer including the one or more microprocessors: providing a distributed workload placement orchestrator, said steps further comprising: A certain host among the above multiple hosts is connected to the above dual port adapter of the above certain host. connecting the first rail through a first port of a power supply; connecting the one of the plurality of hosts to the second rail via a second port of the dual port adapter of the one host; provisioning a workload on the one of the plurality of hosts; assigning a placement policy to the provisioned workload; and based on the placement policy assigned to the provisioned workload, the distributed workload placement orchestrator assigns network traffic of the provisioned workload to one or more of the first rail including the private fabric and the second rail including the public fabric in accordance with the placement policy.

[0433] 16. The computer-readable medium of clause 15, The private fabric provides two or more rails, which provide redundant connectivity for the provisioned workload.

[0434] 17. The computer-readable medium of clause 9, The placement policy assigned to the provisioned workload is a high priority placement policy.

[0435] 18. The computer-readable medium of clause 10, Network traffic between peer nodes of the provisioned workload is strictly provisioned within the private fabric.

[0436] 19. A computer-readable medium according to any one of clauses 15 to 18, The public fabric provides two or more rails, which provide redundant connectivity for the provisioned workloads.

[0437] 20. The computer-readable medium of clause 19, the placement policy assigned to the provisioned workload is a low priority placement policy; Network traffic between peer nodes of the provisioned workload is allocated within the public fabric.

Claims

1. 1. A system for supporting heterogeneous and asymmetric dual-rail configurations in a high performance computing environment, comprising: a computer including one or more microprocessors; a plurality of hosts, each of the plurality of hosts including a dual port adapter, the system further comprising: a first rail including a private fabric, the private fabric including a plurality of switches disposed in two or more racks, each of the two or more racks including a set of the plurality of switches; and the system further comprising: a second rail including a public fabric, the public fabric including a cloud fabric, the system further comprising: a distributed workload placement orchestrator; a host of the plurality of hosts is connected to the first rail via a first port of the dual port adapter of the host; the one of the plurality of hosts is connected to the second rail via a second port of the dual port adapter of the one of the hosts; a workload is provisioned on the one of the plurality of hosts; a placement policy is assigned to the provisioned workload; The system, wherein the distributed workload placement orchestrator utilizes the placement policy assigned to the provisioned workload to allocate network traffic of the provisioned workload to one or more of the first rail including the private fabric and the second rail including the public fabric in accordance with the placement policy.

2. The system of claim 1 , wherein the private fabric provides two or more rails, the two or more rails providing redundant connectivity for the provisioned workload within the private fabric.

3. The system of claim 2 , wherein the placement policy assigned to the provisioned workload is a high priority placement policy.

4. The system of claim 3 , wherein the network traffic of the provisioned workload is strictly allocated within the private fabric.

5. The system of claim 1 , wherein the public fabric provides two or more rails, the two or more rails providing redundant connectivity for the provisioned workloads within the public fabric.

6. The system of claim 5 , wherein the placement policy assigned to the provisioned workload is a low priority placement policy.

7. The system of claim 6 , wherein network traffic of the provisioned workload is allocated within the public fabric and the private fabric.

8. 1. A method for supporting redundant independent networks in a high performance computing environment, comprising: In a computer including one or more microprocessors, providing a plurality of hosts, each of the plurality of hosts including a dual port adapter; the method further comprising, in the computer including the one or more microprocessors, providing a first rail including a private fabric, the private fabric including a plurality of switches disposed in two or more racks, each of the two or more racks including a set of the plurality of switches; the method further comprising: providing a second rail including a public fabric, the public fabric including a cloud fabric, the method further comprising: in the computer including the one or more microprocessors: providing a distributed workload placement orchestrator, the method further comprising: connecting a host of the plurality of hosts to the first rail via a first port of the dual port adapter of the host; connecting the one of the plurality of hosts to the second rail via a second port of the dual port adapter of the one host; provisioning a workload on the one of the plurality of hosts; assigning a placement policy to the provisioned workload; and based on the placement policy assigned to the provisioned workload, the distributed workload placement orchestrator assigns network traffic of the provisioned workload to one or more of the first rail including the private fabric and the second rail including the public fabric in accordance with the placement policy.

9. The method of claim 8 , wherein the private fabric provides two or more rails, the two or more rails providing redundant connectivity for the provisioned workload.

10. The method of claim 9 , wherein the placement policy assigned to the provisioned workload is a high priority placement policy.

11. The method of claim 10 , wherein network traffic between peer nodes of the provisioned workload is strictly provisioned within the private fabric.

12. The method of any one of claims 8 to 11, wherein the public fabric provides two or more rails, the two or more rails providing redundant connectivity for the provisioned workload.

13. The method of claim 12 , wherein the placement policy assigned to the provisioned workload is a low priority placement policy.

14. The method of claim 13 , wherein network traffic between peer nodes of the provisioned workload is allocated within the public fabric.

15. 1. A computer-readable medium bearing instructions for supporting heterogeneous and asymmetric dual-rail configurations in a high performance computing environment, the instructions, when read and executed by a computer, causing the computer to perform steps including: In a computer including one or more microprocessors, providing a plurality of hosts, each of the plurality of hosts including a dual port adapter; the step further comprises, in the computer including the one or more microprocessors: providing a first rail including a private fabric, the private fabric including a plurality of switches disposed in two or more racks, each of the two or more racks including a set of the plurality of switches; the step further comprising: providing a second rail including a public fabric, the public fabric including a cloud fabric, the step further comprising: in the computer including the one or more microprocessors: providing a distributed workload placement orchestrator, said step further comprising: connecting a host of the plurality of hosts to the first rail via a first port of the dual port adapter of the host; connecting the one of the plurality of hosts to the second rail via a second port of the dual port adapter of the one host; provisioning a workload on the one of the plurality of hosts; assigning a placement policy to the provisioned workload; and based on the placement policy assigned to the provisioned workload, the distributed workload placement orchestrator assigns network traffic of the provisioned workload to one or more of the first rail including the private fabric and the second rail including the public fabric in accordance with the placement policy.

16. 16. The computer-readable medium of claim 15, wherein the private fabric provides two or more rails, the two or more rails providing redundant connectivity for the provisioned workload.

17. The computer-readable medium of claim 9 , wherein the placement policy assigned to the provisioned workload is a high priority placement policy.

18. The computer-readable medium of claim 10 , wherein network traffic between peer nodes of the provisioned workload is strictly provisioned within the private fabric.

19. 19. The computer-readable medium of claim 15, wherein the public fabric provides two or more rails, the two or more rails providing redundant connectivity for the provisioned workloads.

20. the placement policy assigned to the provisioned workload is a low priority placement policy; 20. The computer-readable medium of claim 19, wherein network traffic between peer nodes of the provisioned workload is allocated within the public fabric.

Images