Subnet system and method for single logical IP subnet across multiple independent layer 2 (L2) subnets in high performance computing environment

Abstract

To provide methods and systems for supporting a single logical IP subnet across multiple independent layer 2 subnets in a high performance computing environment.SOLUTION: A method provides, as a computer including one or more microprocessors, a logical device, the logical device being addressed by a layer 3 address, wherein the logical device comprises a plurality of network adapters, each of the network adapters comprising a physical port. The method further provides a plurality of switches and arranges the plurality of switches into a plurality of discrete layer 2 subnets to provide a mapping table at the logical device.SELECTED DRAWING: Figure 31

Description

[Technical field]

[0001] Copyright Notice A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0002] Priority claims and cross-references to related applications This application is a continuation of application Ser. No. 16 / 547,335, filed on Aug. 21, 2019, entitled “SYSTEM AND METHOD FOR A SINGLE LOGICAL IP SUBNET ACROSS MULTIPLE INDEPENDENT LAYER 2 (L2) SUBNETS IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT.” This application claims the benefit of priority to U.S. patent application Ser. No. 16 / 261,362, filed Jan. 29, 2019, entitled “SYSTEM AND METHOD FOR A MULTICAST SEND DUPLICATION INSTEAD OF REPLICATION IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT,” which is also a joint application of U.S. patent application Ser. No. 16 / 261,362, filed Jan. 29, 2019, entitled “SYSTEM AND METHOD FOR A MULTICAST SEND DUPLICATION INSTEAD OF REPLICATION IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT.” This application claims the benefit of priority to U.S. patent application Ser. No. 16 / 547,332, filed on August 21, 2019, entitled "SYSTEM AND METHOD FOR ON-DEMAND UNICAST FORWARDING IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT," which also claims the benefit of priority to U.S. patent application Ser. No. 16 / 267,072, filed on February 4, 2019, entitled "SYSTEM AND METHOD FOR USING INFINIBAND ROUTING ALGORITHMS FOR ETHERNET FABRICS IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT," which also claims the benefit of priority to U.S. patent application Ser. No. 16 / 267,072, filed on February 4, 2019, entitled "SYSTEM AND METHOD FOR USING INFINIBAND ROUTING ALGORITHMS FOR ETHERNET FABRICS IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT," which also claims the benefit of priority to U.S. patent application Ser. This application claims the benefit of priority to U.S. patent application Ser. No. 16 / 547,329, filed Aug. 21, 2019, entitled “SYSTEM AND METHOD FOR SUPPORTING HETEROGENEOUS AND ASYMMETRIC DUAL RAIL FABRIC CONFIGURATIONS IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT,” which application is also a continuation of U.S. patent application Ser. No. 16 / 547,329, filed Aug. 21, 2019, entitled “SYSTEM AND METHOD FOR SUPPORTING HETEROGENEOUS AND ASYMMETRIC DUAL RAIL FABRIC CONFIGURATIONS IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT.” This application claims the benefit of priority to U.S. patent applications entitled "System and Method for Supporting a Dual Rail Fabric Configuration," each of which is incorporated herein by reference in its entirety.

[0003] This application is related to the following applications, each of which is incorporated by reference in its entirety: U.S. Provisional Patent Application No. 62 / 552,818, filed August 31, 2017; U.S. Provisional Patent Application No. 62 / 679,465, filed June 1, 2018; U.S. Provisional Patent Application No. 16 / 115,138, filed August 28, 2018; U.S. Provisional Patent Application No. 62 / 679,469, filed June 1, 2018; U.S. Provisional Patent Application No. 62 / 679,478, filed June 1, 2018; and U.S. Provisional Patent Application No. 62 / 679,491, filed June 1, 2018. [Background technology]

[0004] background As larger cloud computing architectures are deployed, the performance and management bottlenecks associated with traditional networks and storage become significant issues. There has been growing interest in using high performance interconnects such as InfiniBand (IB) and RoCE (RDMA (Remote Direct Memory Access) over Converged Ethernet) technologies as the basis for cloud computing fabrics. This is the general area that embodiments of the present teachings are intended to address. Summary of the Invention [Means for solving the problem]

[0005] overview: A system and method for supporting heterogeneous and asymmetric dual-rail fabric configurations in a high performance computing environment. A method may provide for a system including a plurality of hosts in one or more computers, each including one or more microprocessors, each of the plurality of hosts including at least one dual port adapter, and further including a private fabric including two or more switches, and a public fabric including a cloud fabric. A workload may be provisioned on a host of the plurality of hosts. A placement policy may be assigned to the provisioned workload. Network traffic between peer nodes of the provisioned workload may then be assigned to one or more of the private fabric and the public fabric according to the placement policy.

[0006] A system and method for supporting a single logical IP subnet across multiple independent layer 2 subnets in a high performance computing environment. A method can provide a logical device in a computer including one or more microprocessors, the logical device being addressed by a layer 3 address, the logical device including a plurality of network adapters, each of the network adapters including a physical port, and the method can further provide a plurality of switches in the computer including one or more microprocessors. The method can arrange the plurality of switches into multiple separate layer 2 subnets. The method can provide a mapping table in the logical device.

[0007] Particular aspects of the invention are set out in the independent claims and various optional embodiments are set out in the dependent claims. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of an InfiniBand environment according to one embodiment. [Diagram 2] FIG. 1 illustrates an example of a partitioned cluster environment according to one embodiment. [Diagram 3] FIG. 2 illustrates an example of a tree topology in a network environment according to one embodiment. [Figure 4] FIG. 2 illustrates an exemplary shared port architecture according to one embodiment. [Diagram 5] FIG. 2 illustrates an exemplary vSwitch architecture according to one embodiment. [Figure 6] FIG. 2 illustrates an exemplary vPort architecture according to one embodiment. [Figure 7] FIG. 2 illustrates an exemplary vSwitch architecture with pre-populated LIDs according to one embodiment. [Figure 8] FIG. 2 illustrates an exemplary vSwitch architecture with dynamic LID allocation according to one embodiment. [Figure 9] FIG. 2 illustrates an exemplary vSwitch architecture with dynamic LID allocation and pre-populated LIDs to the vSwitch, according to one embodiment. [Figure 10] FIG. 2 illustrates an exemplary multi-subnet InfiniBand fabric according to one embodiment. [Figure 11] FIG. 2 illustrates an interconnection between two subnets in a high performance computing environment, according to one embodiment. [Figure 12] FIG. 2 illustrates an interconnection between two subnets in a high performance computing environment via a dual port virtual router configuration according to one embodiment. [Figure 13] FIG. 2 illustrates a flowchart of a method for supporting a dual port virtual router in a high performance computing environment, according to one embodiment. [Figure 14] 1 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment. [Figure 15]1 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment. [Figure 16] 1 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment. [Figure 17] 1 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment. [Figure 18] 1 is a flowchart of a method for redundant independent networks in a high performance computing environment. [Figure 19] 1 illustrates a system for supporting rail number association for proper connectivity in a high performance computing environment, according to one embodiment. [Figure 20] 1 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment. [Figure 21] 1 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment. [Figure 22] 1 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment. [Figure 23] 1 is a flowchart of a method for determining a placement policy for a provisioned or provisioned workload in a system for supporting heterogeneous and asymmetric dual-rail fabric configurations. [Figure 24] 1 is a flowchart of a method for determining a fabric type selection for a provisioned or provisioning workload, according to one embodiment. [Diagram 25] 1 is a flowchart of a method for supporting heterogeneous and asymmetric dual-rail fabric configurations according to one embodiment. [Figure 26]1 illustrates a system for multicast transmission replication instead of replication in a high performance computing environment, according to one embodiment. [Figure 27] 1 is a flowchart of a method for multicast transmission replication instead of replication in a high performance computing environment. [Figure 28] 1 illustrates a system for supporting link aggregation (LAG) according to one embodiment. [Figure 29] 1 illustrates a system for supporting multi-chassis link aggregation (MLAG), according to one embodiment. [Diagram 30] 1 illustrates a system for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, according to one embodiment. [Diagram 31] 1 is a flowchart of a method for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Detailed Description: The present teachings are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference numerals refer to like elements. References to "a" or "some" embodiments are not necessarily to the same embodiments, and such references mean at least one. While specific implementations are described, it is understood that these specific implementations are provided for illustrative purposes only. A person skilled in the art will recognize that other components and configurations can be used without departing from the scope and spirit of the claimed subject matter.

[0010] Common reference numbers may be used to denote similar elements throughout the figures and detailed description, and thus a reference number used in one figure may or may not be referenced in the detailed description specific to that figure if the element is described elsewhere.

[0011] According to one embodiment, a system and method for supporting heterogeneous and asymmetric dual-rail fabric configurations in a high performance computing environment is described herein.

[0012] According to one embodiment, a system and method are described herein that supports a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment.

[0013] In some embodiments, the following description of the present teachings will be directed to InfiniBand as an example of a high performance network. TM Uses InfiniBand (IB) network. Throughout the following description, InfiniBand is used TM In some cases, references may be made to the InfiniBand Trade Association Architecture Specification, Volume 1, Version 1.3 (variously referred to as the InfiniBand specification, the IB specification, or the legacy IB specification). Such references are understood to be references to the InfiniBand Trade Association Architecture Specification, Volume 1, Version 1.3, published in March 2015 and available at http: / / www.inifinibandta.org, which is incorporated herein by reference in its entirety. It will be apparent to those skilled in the art that other types of high performance networks may be used without any limitation. The following description also uses a fat tree topology as an example for a fabric topology. It will be apparent to those skilled in the art that other types of fabric topologies may be used without any limitation.

[0014] In some other embodiments, the following description applies to RoCE (RDMA (Remote Direct Memory Access) RDMA over Converged Ethernet is used. RoCE (Redirection over Ethernet) is a standard protocol that enables efficient data transfer of RDMA over Ethernet networks, enabling transport offload and superior performance with hardware RDMA engine implementation. RoCE is a standard protocol defined in the InfiniBand Trade Association (IBTA) standard. RoCE is a standard protocol defined in the InfiniBand Trade Association (IBTA) standard. It utilizes RDMA (User Datagram Protocol) encapsulation, allowing it to cross Layer 3 networks. RDMA is a key capability used natively by InfiniBand interconnect technology. Both InfiniBand and Ethernet RoCE share a common user API, but have different physical and link layers.

[0015] According to one embodiment, various portions of this specification include references to InfiniBand fabrics when describing various implementations, however, those skilled in the art will readily appreciate that the various embodiments described herein may also be implemented in a RoCE fabric.

[0016] To meet the demands of modern clouds (e.g., the exascale era), virtual machines must support Remote Direct Memory Access (RDMA). The ability to use low-overhead network communication paradigms such as RDMA (Remote-Distributed Multimedia Access) Since RDMA bypasses the OS stack and communicates directly with the hardware, a pass-through technology such as a single-root I / O virtualization (SR-IOV) network adapter can be used. In accordance with one embodiment, a virtual switch (vSwitch) SR-IOV architecture can be applied to a high-performance lossless interconnect network. Since network reconfiguration time is critical to make live migration a viable option, a scalable, topology-independent dynamic reconfiguration mechanism can be provided in addition to the network architecture.

[0017] According to an embodiment, a routing strategy for a virtualized environment using a vSwitch can be further provided, and an efficient routing algorithm can be provided for a network topology (e.g., a fat-tree topology). The dynamic reconfiguration mechanism can be further tuned to minimize the overhead imposed on the fat-tree.

[0018] According to an embodiment of the present teachings, virtualization can be beneficial for efficient resource utilization and flexible resource allocation in cloud computing. Live migration allows for optimizing resource utilization by moving virtual machines (VMs) between physical servers in a manner that is transparent to applications. Thus, virtualization allows for consolidation, on-demand provisioning of resources, and flexibility through live migration.

[0019] InfiniBand TM InfiniBand TM (IB) stands for InfiniBand TM Trade Association (InfiniBand TMSerial Point-to-Point Full-Duplex Interconnect is an open standard lossless network technology developed by the IEEE 802.11b Trade Association. The technology is based on serial point-to-point full-duplex interconnects that provide high throughput and low latency communications, especially targeted at high-performance computing (HPC) applications and data centers.

[0020] InfiniBand TM The architecture (InfiniBand Architecture: IBA) is Supports layer topology division. At the lower layers, an IB network is called a subnet, and a subnet may contain a set of hosts that are interconnected using switches and point-to-point links. At higher levels, an IB fabric consists of one or more subnets that may be interconnected using routers.

[0021] Within a subnet, hosts may be connected using switches and point-to-point links. In addition, there may be one master management entity, i.e., a subnet manager (SM), that resides on a designated device in the subnet. The subnet manager is responsible for configuring, starting, and maintaining the IB subnet. In addition, the subnet manager (SM) may be responsible for performing routing table calculations in the IB fabric, where, for example, routing in the IB network aims at fair load balancing between all source-destination pairs in the local subnet.

[0022] Through the subnet management interface, the subnet manager exchanges control packets, called subnet management packets (SMP), with the subnet management agent (SMA). An agent exists on every IB subnet device. Using SMP, the subnet manager discovers the fabric and associates the end nodes and switches. You can configure and receive notifications from the SMA.

[0023] According to one embodiment, routing within a subnet in an IB network is performed using a linear forwarding table (LF) stored in the switch. The LFT may be based on the Host Channel Adapters (Host Channel Adapters) on the end nodes. The LFT is calculated by the SM according to the routing mechanism in use. HCA (HCA) ports and switches are addressed using a local identifier (LID). Each entry in the LFT is a destination LID (DLI). D) and an output port. Only one entry per LID in the table is supported. When a packet arrives at a switch, its output port is determined by looking up the DLID in the switch's forwarding table. Routing is deterministic because packets follow the same path in the network between a given source-destination pair (LID pair).

[0024] Generally, all other subnet managers except the master subnet manager operate in standby mode for fault tolerance. However, in the situation where the master subnet manager fails, a new master subnet manager is elected by the standby subnet manager. The master subnet manager also performs periodic sweeps of the subnet to detect any topology changes and updates the network accordingly. Reconfigure the network.

[0025] Additionally, hosts and switches within a subnet may be addressed using a local identifier (LID), and a single subnet may be limited to 49151 unicast LIDs. In addition to the LID, which is a local address valid within the subnet, each IB device may have a 64-bit global unique identifier (GUID). The GUID may be used to form a global identifier (GID), which is the IB Layer 3 (L3) address.

[0026] The SM may compute the routing table (i.e., the connections / routes between each pair of nodes in a subnet) at network initialization time. Additionally, whenever the topology changes, the routing table may be updated to ensure connectivity and optimal performance. During normal operation, the SM may perform periodic light sweeps of the network to check for topology changes. If a change is found during a light sweep, Alternatively, if the SM receives a message (trap) signaling a network change, the SM may reconfigure the network according to the discovered change.

[0027] For example, the SM may reconfigure the network when the network topology changes, such as when a link goes down, a device is added, or a link is removed. The reconfiguration step may include a step performed during network initialization. Furthermore, the reconfiguration may have a local scope that is limited to the subnet where the network change occurred. Also, segmentation of a large fabric using routers may limit the reconfiguration scope.

[0028] An example of an InfiniBand fabric is shown in FIG 1, which illustrates an example of an InfiniBand environment 100 according to one embodiment. In the example shown in FIG 1, nodes A 101-E 105 communicate using an InfiniBand fabric 120 via respective host channel adapters 111-115. According to one embodiment, the various nodes (e.g., nodes A 101-E 105) may be represented by different physical devices. According to one embodiment, the various nodes (e.g., nodes A 101-E 105) may be represented by different virtual devices, such as virtual machines.

[0029] Partitioning in InfiniBand According to one embodiment, an IB network may support partitioning as a security mechanism to isolate logical groups of systems that share a network fabric. Each HCA port on a node in the fabric may be a member of one or more partitions. Partition membership is managed by a centralized partition manager, which may be part of the SM. The SM may organize the partition membership information for each port as a table of 16-bit partition keys (P_Keys). The SM may also manage these partitions. Switch and router ports can be configured with a partition enforcement table that contains P_Key information associated with end nodes that send or receive data traffic through the port. In addition, in the general case, the partition membership of a switch port may represent the collection of all memberships indirectly associated with LIDs routed through the port in the egress direction (towards the link).

[0030] According to one embodiment, a partition is a logical group of ports, and members of a group can only communicate with other members of the same logical group. Isolation can be enforced in host channel adapters (HCAs) and switches by filtering packets using partition membership information. Packets with invalid partitioning information can be dropped as soon as they reach the ingress port. In a partitioned IB system, partitions can be used to create tenant clusters. With partitions in place, nodes cannot communicate with other nodes that belong to different tenant clusters. In this way, the security of the system can be guaranteed even in the presence of faulty or malicious tenant nodes.

[0031] According to one embodiment, for communication between nodes, Queue Pairs (QP) and End-to-End Contexts (EEC), except for the management queue pair (QP0 and QP1), can be assigned to a specific partition. P_Key information can then be added to all IB transport packets sent. When a packet arrives at an HCA port or switch, its P_Key value can be checked against a table configured by the SM. If an invalid P_Key value is found, the packet is immediately discarded. In this way, communication is only allowed between ports that share a partition.

[0032] An example of an IB partition is shown in FIG. 2, which illustrates an example of a partitioned cluster environment, according to one embodiment. In the example shown in FIG. 2, nodes A 101-E 105 communicate through their respective host channel adapters 111-115 using an InfiniBand fabric 120. Nodes A-E are arranged in partitions: partition 1 130, partition 2 140, and partition 3 150. Partition 1 includes node A 101 and node D 104. Partition 2 includes node A 101, node B 102, and node C 103. Partition 3 includes node C 103 and node E 105. Due to this arrangement of partitions, nodes D 104 and E 105 cannot communicate since they do not share a partition. However, for example, nodes A 101 and C 103 can communicate since they are both members of partition 2 140.

[0033] Virtual Machines on InfiniBand Over the past decade, hardware virtualization support has substantially reduced CPU overhead. memory overhead is significantly reduced by virtualizing the memory management unit; storage overhead is reduced by utilizing high-speed SAN storage or distributed network file systems; and device pass-through technologies such as Single Root Input / Output Virtualization (SR-IOV) are used to reduce memory overhead. The prospects for virtualized High Performance Computing (HPC) environments have improved significantly as network I / O overhead has been reduced through the use of high performance interconnect solutions. Clouds are now able to serve virtual HPC (vHPC) clusters with the necessary performance. It is possible to provide

[0034] However, when coupled with lossless networks such as InfiniBand (IB), some cloud features such as live migration of virtual machines (VMs) remain problematic due to the complex addressing and routing schemes used in these solutions. IB is an interconnect network technology that offers high bandwidth and low latency, making it well suited for HPC and other communication-intensive workloads.

[0035] The traditional approach to attaching an IB device to a VM is by using direct-assigned SR-IOV. However, achieving live migration of a VM assigned to an IB Host Channel Adapter (HCA) using SR-IOV has proven challenging. Each IB-attached node has three different addresses (i.e. LID, GUID, and GID). When a live migration occurs, one or more of these addresses change. Other nodes communicating with the VM-in-migration may lose connectivity. This When this occurs, the IB subnet manager (SM) is notified that it should reconnect by sending a Subnet Administration (SA) record route query. An attempt can be made to restore the lost connection by finding out the new address of the virtual machine.

[0036] IB uses three different types of addresses. The first type of address is a 16-bit local identifier (LID). At least one unique LID is assigned by the SM to each HCA port and each switch. The LID is used to route traffic within a subnet. Since the LID is 16 bits long, 65536 unique address combinations can be configured, of which only 49151 (0x0001-0xBFFF) can be used as unicast addresses. As a result, the number of available unicast addresses defines the maximum size of an IB subnet. The second type of address is a 64-bit globally unique identifier (GUID) assigned by the manufacturer to each device (e.g., HCA and switch) and each HCA port. The SM may assign additional subnet-specific GUIDs to HCA ports, which is useful when SR-IOV is used. The third type of address is a 128-bit global identifier (GID). A GID is a valid IPv6 unicast address, at least one of which is assigned to each HCA port. A GID is formed by combining a globally unique 64-bit prefix assigned by the fabric administrator with the GUID address of each HCA port.

[0037] Fat-Tree (FTree) Topology and Routing According to one embodiment, some IB-based HPC systems employ a fat-tree topology to take advantage of the useful properties that fat-trees offer, such as full bisection bandwidth and high throughput due to the availability of multiple paths between each source-destination pair. The initial concept behind fat trees was to employ thicker links between nodes with more available bandwidth as the tree approached the root of the topology. The thicker links could help avoid congestion in higher level switches, and bisectional bandwidth would be preserved.

[0038] 3 illustrates an example of a tree topology in a network environment, according to one embodiment. As illustrated in FIG. 3, one or more end nodes 201-204 may be connected in a network fabric 200. The network fabric 200 may be based on a fat tree topology including multiple leaf switches 211-214 and multiple spine or root switches 231-234. In addition, the network fabric 200 may include one or more intermediate switches, such as switches 221-224.

[0039] 3, each of end nodes 201-204 may be a multi-homed node, i.e., a single node that is connected to two or more portions of network fabric 200 via multiple ports. For example, node 201 may include ports H1 and H2, node 202 may include ports H3 and H4, node 203 may include ports H5 and H6, and node 204 may include ports H7 and H8.

[0040] In addition, each switch may have multiple switch ports. For example, root switch 231 may have switch ports 1-2, root switch 232 may have switch ports 3-4, root switch 233 may have switch ports 5-6, and root switch 234 may have switch ports 7-8.

[0041] According to an embodiment, the fat-tree routing mechanism is one of the most popular routing algorithms for IB-based fat-tree topologies. The fat-tree routing mechanism is also integrated into OFED (Open Fabric Enterprise Distribution: a standard software for building and deploying IB-based applications). This is implemented in the OpenSM (Software Stack) Subnet Manager.

[0042] The objective of the fat-tree routing mechanism is to generate an LFT that uniformly spreads shortest path routes across the links in the network fabric. The mechanism traverses the fabric in indexing order and assigns the target LIDs of the end nodes, and therefore the corresponding routes, to each switch port. For end nodes connected to the same leaf switch, the indexing order may depend on the switch ports to which the end nodes are connected (i.e., the port numbering sequence). For each port, the mechanism may maintain a port usage counter, and whenever a new route is added, the port usage counter may be used to select the least frequently used port.

[0043] According to one embodiment, in partitioned subnets, nodes that are not members of a common partition are not allowed to communicate. In practice, this means that some of the routes assigned by the Fat Tree routing algorithm are not used for user traffic. Problems arise if the Fat Tree routing mechanism generates LFTs for those routes in the same way as other functional paths. This behavior can degrade balancing on the links, since nodes are routed in indexing order. Fat Tree routed subnets generally provide poor isolation between partitions, since routing is done without awareness of partitions.

[0044] According to one embodiment, a Fat Tree is a hierarchical network topology that can scale with available network resources. It is easily constructed using commodity switches arranged at various levels of hierarchy. Furthermore, various variants of fat-trees are publicly available, including k-ary-n-tree, Extended Generalized Fat-Tree (XGFT), Parallel Ports Generalized Fat-Tree (PGFT) and Real Life Fat-Tree (RLFT).

[0045] A k-ary-n-tree is a fat tree with n levels, n end nodes and n k n-1 The XGFT fat tree comprises a k-ary-n-tree with 100 k ports and 100 Gbps switches, each with 2k ports. Each switch has the same number of connections up and down in the tree. The XGFT fat tree extends the k-ary-n-tree by allowing both a different number of up and down connections for the switches and a different number of connections at each level in the tree. The PGFT definition further extends the XGFT topology to allow multiple connections between switches. A wide variety of topologies can be defined using XGFT and PGFT. However, for practical purposes, a restricted version of PGFT, RLFT, is introduced to define fat trees that are commonly found in modern HPC clusters. RLFT uses the same port count switches for all levels in the fat tree.

[0046] Input / Output (I / O) Virtualization According to one embodiment, I / O Virtualization (IOV) can make I / O available by allowing Virtual Machines (VMs) to access the underlying physical resources. Combining storage traffic with inter-server communication can place an overwhelming burden on a single server's I / O resources, resulting in backlogs and idle processors waiting for data. As the number of I / O requests increases, IOV can provide availability and improve the performance, scalability and elasticity of (virtualized) I / O resources to rival performance levels seen in modern CPU virtualization.

[0047] According to one embodiment, IOVs are desired to enable sharing of I / O resources and to allow access to resources from VMs to be protected. IOVs separate the logical devices exposed to a VM from their physical implementation. Currently, emulation, paravirtualization, direct assignment (DA), and single-root I / O There can be various types of IOV technologies, such as SR-IOV.

[0048] According to one embodiment, one type of IOV technology is software emulation. Software emulation can enable a separated front-end / back-end software architecture. The front-end can be a device driver located in the VM and communicate with the back-end realized by the hypervisor to provide I / O access. The physical device sharing ratio is high and live migration of VMs can be achieved with only milliseconds of network downtime. However, software emulation introduces additional undesirable computational overhead.

[0049] According to one embodiment, another type of IOV technique is direct device assignment. Direct device assignment requires that an I / O device be attached to a VM, but the device is not shared between VMs. Direct assignment, or device pass-through, offers nearly inherent performance with minimal overhead. The physical device is attached directly to the VM, bypassing the hypervisor. However, a drawback of such direct device assignment is that there is no sharing between virtual machines, limiting scalability, such as one physical network card being attached to one VM. do.

[0050] According to one embodiment, Single Root IOV (SR-IOV) is Hardware virtualization may allow a physical device to appear as multiple independent, lightweight instances of the same device. These instances can be assigned to VMs as pass-through devices and accessed as Virtual Functions (VFs). The hypervisor accesses the device through a unique (per device) fully functional Physical Function (PF). SR-I OV mitigates the scalability issues of purely direct allocation. However, a problem presented by SR-IOV is that it can impair VM migration. Among these IOV technologies, SR-IOV extends the PCI Express (PCIe) standard with a means to allow multiple VMs to directly access a single physical device while still maintaining near-inherent performance. This allows SR-IOV to offer superior performance and scalability.

[0051] SR-IOV allows a PCIe device to expose multiple virtual devices that can be shared among multiple guests by assigning one virtual device to each guest. Each SR-IOV device has at least one physical function (PF) and one or more associated virtual functions (VFs). A PF is a communication function controlled by a virtual machine monitor (VMM) or hypervisor. VFs are lightweight PCIe functions, whereas VFs are regular PCIe functions. Each VF has its own base address (BAR) and is assigned a unique requester ID. The unique requester ID is managed by the I / O memory management unit (I / O memory management unit). The IOMMU is responsible for memory and interrupt translations between PFs and VFs.

[0052] Unfortunately, direct device allocation techniques present a barrier for cloud providers in situations where transparent live migration of virtual machines is desired for data center optimization. The essence of live migration is that the memory contents of the VM are copied to a remote hypervisor. Furthermore, the VM is suspended in the source hypervisor and the VM's operation is resumed in the destination. When using software emulation methods, the network interfaces are virtual so that their internal states are stored in memory and then copied. Thus, the downtime can be reduced to a few milliseconds.

[0053] However, migration becomes more difficult when direct device assignment techniques such as SR-IOV are used. In this situation, the entire internal state of the network interface cannot be copied since it is tied to the hardware. Instead, the SR-IOV VF assigned to the VM is detached and a live migration is performed, and a new VF is attached at the destination. In the case of InfiniBand and SR-IOV, this process can result in downtime on the order of a few seconds. Furthermore, in the SR-IOV shared port model, the address of the VM changes after migration, which adds overhead to the SM and negatively impacts the performance of the underlying network fabric.

[0054] InfiniBand SR-IOV Architecture - Shared Port There can be various types of SR-IOV models, such as a shared port model, a virtual switch model, and a virtual port model.

[0055] 4 illustrates an exemplary shared port architecture according to one embodiment. As shown, a host 300 (e.g., a host channel adapter) may interact with a hypervisor 310. The hypervisor 310 may assign various virtual functions 330, 340, and 350 to a number of virtual machines. Similarly, physical functions may be handled by the hypervisor 310.

[0056] 4, a host (e.g., an HCA) appears as a single port in the network with a single shared LID and shared Queue Pair (QP) space between the physical function 320 and the virtual functions 330, 350, 350. However, each function (i.e., physical function and virtual function) may have their own GID.

[0057] As shown in FIG. 4, according to one embodiment, various GIDs can be assigned to virtual and physical functions, and a special queue pair, QP0 and QP1 (i.e., InfiniBand TM A dedicated queue pair used for management packets is owned by the Physical Function. These QPs are exposed to the VFs as well, but the VFs are not allowed to use QP0 (all SMP coming from the VF towards QP0 are dropped), and QP1 can act as a proxy for the actual QP1 owned by the PF.

[0058] According to one embodiment, the shared port architecture may enable highly scalable data centers that are not limited by the number of VMs (attached to the network by being assigned to virtual functions) because LID space is only consumed by the physical machines and switches in the network.

[0059] However, a drawback of the shared port architecture is that it cannot provide transparent live migration, which hampers the possibilities for flexible VM placement. Because each LID is associated with a particular hypervisor and is shared among all VMs residing on the hypervisor, a migrating VM (i.e., a virtual machine migrating to a destination hypervisor) must change its LID to the LID of the destination hypervisor. Furthermore, as a result of the restricted QP0 access, a subnet manager cannot be run inside a VM.

[0060] InfiniBand SR-IOV Architecture Model - Virtual Switch (vSwitch) 5 illustrates an exemplary vSwitch architecture according to one embodiment. As shown, a host 400 (e.g., a host channel adapter) can interact with a hypervisor 410, which can assign various virtual functions 430, 440, and 450 to a number of virtual machines. Similarly, physical functions can be handled by the hypervisor 410. A virtual switch 415 can also be handled by the hypervisor 401.

[0061] According to one embodiment, in the vSwitch architecture, each virtual function 430, 440, 450 is a full virtual Host Channel Adapter (vHCA), which means that in hardware, the VM assigned to the VF is assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. To the rest of the network and SM, the HCA 400 appears as a switch with additional nodes connected via the virtual switch 415. The hypervisor 410 can use the PF 420 (assigned to the virtual function) to )VM uses VF.

[0062] According to one embodiment, the vSwitch architecture provides transparent virtualization. However, because each virtual function is assigned a unique LID, the number of available LIDs is quickly consumed. Similarly, if many LID addresses are used (i.e., one for each physical function and each virtual function), more communication paths must be computed by the SM and more subnet management packets (SMPs) must be sent to the switch to update their LFTs. For example, the computation of communication paths can take several minutes in a large network. Since the LID space is limited to 49151 unicast LIDs and each VM (through a VF) occupies one LID for each physical node and switch, the number of active VMs is limited by the number of physical nodes and switches in the network and vice versa.

[0063] InfiniBand SR-IOV Architecture Model - Virtual Port (vPort) 6 illustrates an exemplary vPort concept according to one embodiment. As shown, a host 300 (e.g., a host channel adapter) can interact with a hypervisor 410 that can assign various virtual functions 330, 340, and 350 to a number of virtual machines. Similarly, physical functions can be handled by the hypervisor 310.

[0064] According to one embodiment, the vPort concept is loosely defined to give vendors implementation freedom (e.g., the definition does not specify that implementations should be SRIOV-only), and the purpose of the vPort is to standardize how VMs are handled in a subnet. With the vPort concept, both SR-IOV shared port-like architectures and vSwitch-like architectures, or combinations of these architectures, can be defined, which may be more scalable in both spatial and performance domains. Also, vPorts support optional LIDs, and unlike shared ports, the SM is aware of all vPorts available in a subnet, even if the vPorts do not use dedicated LIDs.

[0065] InfiniBand SR-IOV Architecture Model - vSwitch with pre-populated LID According to one embodiment, the present disclosure provides a system and method for providing a vSwitch architecture with pre-populated LIDs.

[0066] 7 illustrates an exemplary vSwitch architecture with pre-populated LIDs, according to one embodiment. As shown in the figure, several switches 501-504 are configured to support InfiniBand TMCommunication can be established between members of a fabric, such as a fabric. The fabric can include several hardware devices, such as host channel adapters 510, 520, 530. Additionally, host channel adapters 510, 520, and 530 can interact with hypervisors 511, 521, and 531, respectively. Each hypervisor, along with the host channel adapters, can further interact with, configure, and assign to several virtual machines several virtual functions 514, 515, 516, 524, 525, 526, 534, 535, and 536. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 can additionally assign virtual machine 2 551 to virtual function 2 515 and virtual machine 3 552 to virtual function 3 516. The hypervisor 531 can further assign a virtual machine 4 553 to a virtual function 1 534. The clients can access the host channel adapters through fully functional physical functions 513, 523 and 533 on each of the host channel adapters.

[0067] According to one embodiment, each of the switches 501-504 may include several ports (not shown) that are used to configure linear forwarding tables to direct traffic within the network switching environment 600.

[0068] According to one embodiment, virtual switches 512, 522 and 532 can be handled by respective hypervisors 511, 521, 531. In such a vSwitch architecture, each virtual function is a full virtual host channel adapter (vHCA), which means that in hardware, a VM assigned to a VF is assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. To the rest of the network and SM (not shown), HCAs 510, 520 and 530 appear as switches with additional nodes connected through virtual switches.

[0069] According to one embodiment, the present disclosure provides a system and method for providing a vSwitch architecture with pre-populated LIDs. With reference to FIG. 7, LIDs are pre-populated for various physical functions 513, 523, and 533, as well as virtual functions 514-516, 524-526, 534-536 (even virtual functions not currently associated with an active virtual machine). For example, physical function 513 is pre-populated with LID 1, and virtual function 1 534 is pre-populated with LID 10. When a network is booted, LIDs are pre-populated in SR-IOV vSwitch-enabled subnets. Populated VFs are assigned LIDs as shown in FIG. 7, even if all of the VFs are not occupied by VMs in the network.

[0070] According to one embodiment, many similar physical host channel adapters can have two or more ports (with two ports shared for redundancy), and a virtual HCA can also be represented by two ports and connected to an external IB subnet via one or more virtual switches.

[0071] According to one embodiment, in a LID pre-populated vSwitch architecture, each hypervisor consumes one LID for itself via the PF and can consume one or more LIDs for each additional VF. The sum of all VFs available in all hypervisors in an IB subnet gives the maximum amount of VMs that can be run in the subnet. For example, in an IB subnet with 16 virtual functions per hypervisor in the subnet, each hypervisor consumes 17 LIDs in the subnet (1 LID for each 16 virtual functions and 1 LID for the physical function). In such an IB subnet, the theoretical hypervisor limit for a single subnet is defined by the number of available unicast LIDs, which is 2891 (49151 available LIDs divided by 17 LIDs per hypervisor), and the total number of VMs (i.e., limit) is 46256 (2891 hypervisors multiplied by 16 VFs per hypervisor). (In practice, these numbers are smaller, since each switch, router, or dedicated SM node in the IB subnet consumes LIDs as well.) Note that vSwitches do not need to occupy additional LIDs, since they can share LIDs with PFs.

[0072] According to one embodiment, a LID pre-populated vSwitch architecture In, once the network is booted, communication paths are calculated for all LIDs. If a new VM needs to be started, the system does not need to add a new LID in the subnet. Otherwise, the operation that may completely reconfigure the network, including recalculating paths, is the most time-consuming element. Instead, available ports for VMs are located in one of the hypervisors (i.e., available virtual functions), and virtual machines are attached to the available virtual functions.

[0073] According to one embodiment, the LID pre-populated vSwitch architecture also enables the ability to compute and use different routes to reach different VMs hosted by the same hypervisor. Essentially, this allows such subnets and networks to use LID-Mask-Control-like (LMC-like) features to provide alternative routes towards one physical machine without being bound by the constraints of LMC requiring LIDs to be contiguous. The freedom to use non-contiguous LIDs is particularly useful when a VM needs to migrate and its associated LID is delivered to the destination.

[0074] In accordance with one embodiment, several considerations may be taken into account along with the above-mentioned advantages of a LID pre-populated vSwitch architecture: For example, because LIDs are pre-populated in an SR-IOV vSwitch-enabled subnet when the network is booted, the initial route computation (e.g., at startup) may take longer than if the LIDs were not pre-populated.

[0075] InfiniBand SR-IOV Architecture Model - vSwitch with Dynamic LID Allocation According to one embodiment, the present disclosure provides a system and method for providing a vSwitch architecture with dynamic LID allocation.

[0076] 8 illustrates an exemplary vSwitch architecture with dynamic LID allocation, according to one embodiment. As shown in the figure, several switches 501-504 are configured to support InfiniBand TMCommunications can be established between members of a fabric, such as a fabric. The fabric can include several hardware devices, such as host channel adapters 510, 520, 530. The host channel adapters 510, 520, and 530 can further interact with hypervisors 511, 521, and 531, respectively. Each hypervisor, along with the host channel adapters, can further interact with, configure, and assign to several virtual machines several virtual functions 514, 515, 516, 524, 525, 526, 534, 535, and 536. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 can additionally assign virtual machine 2 551 to virtual function 2 515 and virtual machine 3 552 to virtual function 3 516. Hypervisor 531 can further assign virtual machine 4 553 to virtual function 1 534. The hypervisor can access the host channel adapters through fully functional physical functions 513, 523 and 533 on each of the host channel adapters.

[0077] According to one embodiment, each of the switches 501-504 may include several ports (not shown) that are used to configure linear forwarding tables to direct traffic within the network switching environment 700.

[0078] According to one embodiment, the virtual switches 512, 522 and 532 are The VFs can be handled by the VF-visors 511, 521 and 531. In such a vSwitch architecture, each virtual function is a full virtual host channel adapter (vHCA), which means that in hardware, the VMs assigned to the VFs are assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. To the rest of the network and SM (not shown), the HCAs 510, 520 and 530 appear as switches with additional nodes connected through virtual switches.

[0079] According to one embodiment, the present disclosure provides a system and method for providing a vSwitch architecture with dynamic LID assignment. With reference to FIG. 8, various physical functions 513, 523, and 533 are dynamically assigned LIDs, with physical function 513 receiving LID1, physical function 523 receiving LID2, and physical function 533 receiving LID3. Those virtual functions associated with active virtual machines may also receive dynamically assigned LIDs. For example, virtual machine 1 550 is active and associated with virtual function 1 514, so virtual function 514 may be assigned LID5. Similarly, virtual function 2 515, virtual function 3 516, and virtual function 1 534 are each associated with an active virtual function. As such, these virtual functions are assigned LIDs: LID 7 is assigned to virtual function 2 515, LID 11 is assigned to virtual function 3 516, and LID 9 is assigned to virtual function 1 534. Unlike a vSwitch, which has pre-populated LIDs, virtual functions that are not currently associated with an active virtual machine do not receive an LID assignment.

[0080] According to one embodiment, with dynamic LID allocation, initial path computation can be substantially reduced: when a network is booting for the first time and no VMs are present, a relatively small number of LIDs can be used for initial path computation and LFT distribution.

[0081] According to one embodiment, many similar physical host channel adapters can have two or more ports (with two ports shared for redundancy), and a virtual HCA can also be represented by two ports and connected to an external IB subnet via one or more virtual switches.

[0082] According to one embodiment, when a new VM is created in a system utilizing a vSwitch with dynamic LID allocation, a free VM slot is found and a unique unused unicast LID is found as well to determine on which hypervisor the newly added VM should be booted. However, there is no known route in the switch's LFT and network to handle the newly added LID. Computing a new set of routes to handle the newly added VM is undesirable in a dynamic environment where several VMs may be booted every minute. In a large IB subnet, computing a new set of routes can take several minutes, and this procedure would have to be repeated every time a new VM is booted.

[0083] Advantageously, according to one embodiment, since all VFs in a hypervisor share the same uplink with the PF, there is no need to compute a new set of routes. It is only necessary to iterate through the LFTs of all physical switches in the network, copy forwarding ports from the LID entries belonging to the PF of the hypervisor (where the VM is created) to the newly added LID, and send a single SMP to update the corresponding LFT block of the particular switch. This eliminates the need for the system and method to compute a new set of routes.

[0084] According to one embodiment, in a vSwitch with a dynamic LID allocation architecture, The LIDs assigned to VMs on each hypervisor do not need to be contiguous. If one compares the LIDs assigned on each hypervisor VM in a pre-populated vSwitch with a dynamic LID assignment, one can see that the LIDs assigned in the dynamic LID assignment architecture are discontinuous, whereas the pre-populated LIDs are essentially contiguous. Furthermore, in the vSwitch dynamic LID assignment architecture, when a new VM is created, the next available LID is used for the lifetime of the VM. Conversely, in a pre-populated vSwitch, each VM inherits the LID already assigned to its corresponding VF, and in a network without live migration, VMs assigned contiguously to a given VF get the same LID.

[0085] According to one embodiment, a vSwitch with dynamic LID allocation architecture can solve the shortcomings of a vSwitch with pre-populated LID architecture model at the expense of some additional network and runtime SM overhead. Every time a VM is created, the LFT of the physical switch in the subnet is updated with the newly added LID associated with the created VM. This operation requires one Subnet Management Packet (SMP) to be sent per switch. Since each VM uses the same route as its host hypervisor, features such as LMC are also unavailable. However, there is no limit on the total number of VFs present on all hypervisors, and the number of VFs may exceed the unicast LID limit. In such a case, of course, not all VFs on active VMs can be granted at the same time, and having more spare hypervisors and VFs adds flexibility to recover and optimize failures in a fragmented network when operating near the unicast LID limit.

[0086] InfiniBand SR-IOV Architecture Model - Dynamic LID Allocation and LID Pre-Populated vSwitch 9 illustrates an exemplary vSwitch architecture with dynamic LID allocation and pre-populated LIDs for the vSwitch, according to one embodiment. As shown in the figure, several switches 501-504 are configured to support InfiniBand TM Communication can be established between members of a fabric, such as a fabric. The fabric can include several hardware devices, such as host channel adapters 510, 520, 530. The host channel adapters 510, 520, and 530 can further interact with hypervisors 511, 521, and 531, respectively. Each hypervisor, along with the host channel adapters, can further interact with, configure, and assign to several virtual machines several virtual functions 514, 515, 516, 524, 525, 526, 534, 535, and 536. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 can additionally assign virtual machine 2 551 to virtual function 2 515. Hypervisor 521 can assign virtual machine 3 552 to virtual function 3 526. Hypervisor 531 may further assign virtual machine 4 553 to virtual function 2 535. The hypervisor may access the host channel adapters through fully functional physical functions 513, 523 and 533 on each of the host channel adapters.

[0087] According to one embodiment, each of switches 501-504 may include several ports (not shown) that are used to configure linear forwarding tables to direct traffic within network switching environment 800.

[0088] According to one embodiment, virtual switches 512, 522 and 532 can be handled by respective hypervisors 511, 521, 531. In such a vSwitch architecture, each virtual function is a full virtual host channel adapter (vHCA), which means that in hardware, the VM assigned to the VF is assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. To the rest of the network and SM (not shown), HCAs 510, 520 and 530 appear as switches with additional nodes connected through the virtual switches.

[0089] According to one embodiment, the present disclosure provides a system and method for providing a hybrid vSwitch architecture with dynamic LID allocation and pre-populated LIDs. With reference to FIG. 9, a vSwitch with a pre-populated LID architecture may be deployed in hypervisor 511, and a vSwitch with pre-populated LIDs and dynamic LID allocation may be deployed in hypervisor 521. A vSwitch with dynamic LID allocation may be deployed in hypervisor 531. Thus, physical function 513 and virtual functions 514-516 have their LIDs pre-populated (i.e., even virtual functions that are not attached to an active virtual machine are assigned LIDs). Physical function 523 and virtual function 1 524 may have their LIDs pre-populated, while virtual function 2 525 and virtual function 3 526 have their LIDs dynamically assigned (i.e., virtual function 2 525 is available for dynamic LID assignment, and virtual function 3 526 has been dynamically assigned a LID of 11 because it is attached to virtual machine 3 552). Finally, the functions (physical and virtual functions) associated with hypervisor 3 531 may have their LIDs dynamically assigned. This results in virtual function 1 534 and virtual function 3 536 being available for dynamic LID assignment, while virtual function 2 535 has been dynamically assigned a LID of 9 because it is attached to virtual machine 4 553.

[0090] 9, where both LID pre-populated vSwitches and dynamic LID allocation vSwitches are utilized (independently or in combination within any given hypervisor), the number of pre-populated LIDs per host channel adapter can be defined by a fabric administrator and can be in the range of 0<=pre-populated VFs<=total VFs (per host channel adapter). The VFs available for dynamic LID allocation can be found by subtracting the number of pre-populated VFs from the total number of VFs (per host channel adapter).

[0091] According to one embodiment, many similar physical host channel adapters can have two or more ports (with two ports shared for redundancy), and a virtual HCA can also be represented by two ports and connected to an external IB subnet via one or more virtual switches.

[0092] InfiniBand - Inter-Subnet Communication (Fabric Manager) According to one embodiment, in addition to providing an InfiniBand fabric within one subnet, embodiments of the present disclosure may also provide an InfiniBand fabric that spans two or more subnets.

[0093] FIG. 10 illustrates an exemplary multi-subnet InfiniBand fabric according to one embodiment. As shown in this figure, multiple switches 1001-1004 within a subnet A 1000 provide interconnects within the subnet A 1000 (e.g., an IB subnet). A fabric may provide communication between members of a fabric, such as a FiniBand fabric. The fabric may include a number of hardware devices, such as, for example, a channel adapter 1010. The host channel adapter 1010 may interact with a hypervisor 1011. The hypervisor may set up a number of virtual functions 1014 with the host channel adapters it interacts with. In addition, the hypervisor may assign a virtual machine to each virtual function. For example, virtual machine 1 1015 is assigned to virtual function 1 1014. The hypervisor may access its associated host channel adapter through a fully functional physical function, such as physical function 1013, on each host channel adapter. A number of switches 1021-1024 may provide communication between members of a fabric, such as, for example, an InfiniBand fabric, within a subnetwork B 1040 (e.g., an IB subnet). The fabric may include a number of hardware devices, such as, for example, a host channel adapter 1030. The host channel adapter 1030 may interact with a hypervisor 1031. The hypervisor can set up multiple virtual functions 1034 along with the host channel adapters they interact with. Additionally, the hypervisor can assign a virtual machine to each of the virtual functions. For example, virtual machine 2 1035 is assigned to virtual function 2 1034. The hypervisor can access its associated host channel adapter through a fully functional physical function, such as physical function 1033, on each host channel adapter. Note that although only one host channel adapter is shown in each subnet (i.e., subnet A and subnet B), it should be understood that multiple host channel adapters and their corresponding components may be included within each subnet.

[0094] According to one embodiment, each host channel adapter may further be associated with a virtual switch, such as virtual switch 1012 and virtual switch 1032, and as noted above, each HCA may be set up with a different architectural model. Although both subnets in Figure 10 are shown using the architectural model of a vSwitch with pre-populated LIDs, this is not intended to suggest that all such subnet configurations may follow a similar architectural model.

[0095] According to one embodiment, at least one switch in each subnet may be associated with a router. For example, switch 1002 in subnet A 1000 is associated with router 1005, and switch 1021 in subnet B 1040 is associated with router 1006.

[0096] According to one embodiment, at least one device (e.g., a switch, a node, etc.) can be associated with a fabric manager (not shown). The fabric manager can be used, for example, to discover inter-subnet fabric topologies, create fabric profiles (e.g., virtual machine fabric profiles), and build virtual machine related database objects that form the basis for building the virtual machine fabric profiles. In addition, the fabric manager can define legal inter-subnet connectivity as to which subnets are allowed to communicate over which router ports and with which partition numbers.

[0097] According to one embodiment, when traffic from an originating source, such as virtual machine 1 in subnet A, is destined for a different subnet, such as virtual machine 2 in subnet B, the traffic can be directed to a router in subnet A, i.e., router 1005, which can then send the traffic to subnet B via a link with router 1006.

[0098] Virtual Dual Port Router According to one embodiment, a dual port router abstraction is a router that receives a route from a global route header (GRH) to a LR. It is possible to provide a simple method that allows defining an inter-subnet router function based on a switch hardware implementation that has the capability to perform conversion to a local route header (LRH) in addition to performing normal LRH-based switching.

[0099] According to one embodiment, a Virtual Dual Port Router can be logically connected outside of a corresponding switch port and can present an InfiniBand compliant view to a standard management entity, such as a subnet manager.

[0100] According to one embodiment, the dual port router model is configured to route different subnets, with each subnet handling packet forwarding and address mapping on the ingress path to the subnet. This shows that it is possible to connect subnets with complete control over the subnetworks and without affecting the routing and logical connectivity within any of the incorrectly connected subnets.

[0101] According to one embodiment, in situations involving incorrectly connected fabrics, the Virtual Dual Port Router abstraction can also be used to allow management entities such as subnet managers and IB diagnostic software to function correctly in the presence of unintended physical connections to remote subnets.

[0102] 11 illustrates an interconnection between two subnets in a high performance computing environment, according to one embodiment. Prior to being configured with a Virtual Dual Port Router, Subnet A A switch 1120 in subnet B 1101 may be connected through a switch port 1121 of switch 1120, via physical connection 1110, to a switch 1130 in subnet B 1102 through a switch port 1131 of switch 1130. In such an embodiment, each of switch ports 1121 and 1131 may function both as a switch port and as a router port.

[0103] According to one embodiment, the problem with this configuration is that a management entity, such as a subnet manager in an InfiniBand subnet, cannot distinguish between a physical port that is both a switch port and a router port. In this situation, the SM can treat the switch port as having a router port connected to it. However, if the switch port is connected to another subnet with another subnet manager, for example, via a physical link, the subnet manager can send discovery messages to the physical link. However, such discovery messages are not allowed in the other subnet.

[0104] FIG. 12 illustrates an interconnection between two subnets via a dual-port virtual router configuration in a high performance computing environment according to one embodiment.

[0105] According to one embodiment, after configuration, the dual port virtual router configuration can be provided to the subnet manager such that the appropriate end nodes that indicate the edges of the subnets for which the subnet manager is responsible are known to the subnet manager.

[0106] According to one embodiment, a switch port in a switch 1220 in subnet A 1201 may be connected (i.e., logically connected) to a router port 1211 in a virtual router 1210 via a virtual link 1223. A physical link 1203, which may have two ends, connects subnet A 1201 to subnet B 1202 via a first end of the physical link, via a second end of the physical link, via router port II 1212, and via a virtual router 1230 in subnet B 1202, via router port II 1212. 1232. The virtual router 1230 may further include a router port 1231 that may be connected (i.e., logically connected) to a switch port 1241 on the switch 1240 via a virtual link 1233.

[0107] According to one embodiment, a subnet manager (not shown) on subnet A can discover router port 1211 on virtual router 1210 as the endpoint of a subnet that the subnet manager controls. The dual port virtual router abstraction allows the subnet manager on subnet A to treat subnet A in the normal manner (e.g., as specified in the InfiniBand standard). At the subnet management agent level, the dual port virtual router A fabric abstraction can be provided so that a normal switch port appears to the SM, and then at the SMA level, the abstraction can be provided so that there is another port connected to this switch port, which becomes a router port on a dual-port virtual router. In the local SM, the traditional fabric topology can still be used (where the SM sees the ports as normal switch ports), and therefore the SM sees the router ports as end ports. The physical connection can be between two switch ports that are also configured as router ports in two different subnets.

[0108] In accordance with one embodiment, the dual port virtual router can also solve the problem that a physical link may be mistakenly connected to any other switch port in the same subnet, or to a switch port that is not intended to provide connectivity to another subnet, and thus the methods and systems described herein also represent what is outside the subnet.

[0109] According to one embodiment, a local SM in a subnet, such as subnet A, determines a switch port and then determines the router port that is connected to this switch port (e.g., router port 1211, which is connected to switch port 1221 via virtual link 1223). The SM considers router port 1211 to be the edge of the subnet that it manages, so the SM cannot send discovery and / or management messages further than this point (e.g., to router port II 1212).

[0110] According to one embodiment, the dual port virtual router provides the advantage that the dual port virtual router abstraction is managed entirely by a management entity (e.g., SM or SMA) within the subnet to which the dual port virtual router belongs. By making the management local only, the system does not need to provide an external, separate management entity. That is, each side of the inter-subnet connection is responsible for configuring its own dual port virtual router.

[0111] According to one embodiment, when a packet such as an SMP destined for a remote destination (i.e., outside the local subnet) arrives at a local target port that is not configured through the dual-port virtual router, the local port can return a message indicating that it is not a router port.

[0112] Many features of the present teachings may be implemented in, using, or with the aid of hardware, software, firmware, or a combination thereof. Thus, features of the present teachings may be implemented using a processing system (e.g., including one or more processors).

[0113] 13 illustrates a method for supporting a dual port virtual router in a high performance computing environment according to one embodiment. In step 1310, the method can provide a first subnet on one or more computers including one or more microprocessors. The first subnet includes a plurality of switches, the plurality of switches including at least a leaf switch, each of the plurality of switches including a plurality of switch ports. The first subnet further includes a plurality of host channel adapters, each of the host channel adapters including at least one host channel adapter port, a plurality of end nodes, each of the end nodes associated with at least one host channel adapter of the plurality of host channel adapters, and a subnet manager, the subnet manager executing in one of the plurality of switches and the plurality of host channel adapters.

[0114] In step 1320, the method may configure a switch port of the plurality of switch ports on a switch of the plurality of switches as a router port.

[0115] In the switch 1330, the method can logically connect switch ports configured as router ports to a virtual router, which includes at least two virtual router ports.

[0116] Redundant fully-independent and semi-independent networks According to one embodiment, a mission-critical system should operate correctly and always provide relevant services to the right clients with the right data, within the expected response times and overall performance constraints defined for the system.

[0117] According to one embodiment, for a system implemented as a distributed cluster of computers, this also implies that communication between the required set of computers in the cluster must always be operational.

[0118] According to one embodiment, a desirable premise for a network communications system to be operable is that all physical connectivity between components such as host adapters and switches is correct, in that the correct components and connectors are connected via the correct cables, and that all involved components are configured in the correct manner.

[0119] However, according to one embodiment, communications should not normally rely on only one point of failure, since hardware components can fail and operators can make mistakes. If a failure or mistake occurs that prevents communication over one component and communication path, it is paramount that this be detected and communications be handed over to an alternative, redundant communication path immediately, or at least before the inoperative communication path has any significant impact on system operation or response times.

[0120] Also, according to one embodiment, whenever a failure exists that leaves some or all current communications vulnerable if a second failure were to occur, it is important that relevant remedial action can occur as soon as possible and / or additional backup measures can be employed to prevent total loss of service.

[0121] According to one embodiment, another aspect is that since system and application software is very rarely error-free and there is often a need to improve the features of a system over time, it is important to be able to upgrade various software components in the system without causing any outages. In a distributed cluster system, this typically means a "rolling upgrade" model where redundant components are upgraded in a strict order and the system is always fully operational with the required set of components and the required communications.

[0122] Also, according to one embodiment, to improve the capabilities and / or expand the capacity of the system, physical modifications may be required in terms of additional HW components and / or replacing existing HW components with more capable HW components, in which case such upgrades may also mean new software and / or firmware in the system.

[0123] However, new software and firmware versions as well as new HW versions introduce the risk of introducing new or early undetected errors and the risk of interoperability problems between different software / firmware components on the same node or between different nodes in the cluster. Therefore, in the ideal case, the ability to operate the system should not depend on only a single type of software being used throughout the system and on the successful upgrade from one software version to another. To achieve this, one approach is to ensure that the required services can be realized in a redundant manner by two sets of different types of hardware components, with different sets of software and firmware, and that there are no dependencies at all, or at a minimum very well-defined and controlled dependencies, between the sets of different sets of redundant components.

[0124] According to one embodiment, an example of such a system is the use of geographically separate systems implemented with different computer types and using different software, but backup data can be transferred between those systems using a neutral format with minimal dependency on any of the systems. A less strict version of this is where the primary and backup sites use the same type of equipment, but are not upgraded at the same time. Thus, typically the backup site continues to use version N until sufficient trust has been established with version N+1 at the primary site.

[0125] According to one embodiment, another approach is to have several independent implementations of the same functionality running in parallel. This approach has been used in space missions where multiple versions of critical components were developed independently of each other by different teams. A less extreme version of this approach applied to cluster communication infrastructure is to have two independent networks, each implemented by gear (HW and SW / FW) from different vendors, but communication between pairs of computers can fail over between the two network infrastructures. This approach to network redundancy can then be applied orthogonally to whether different types of computers and host software are used in the cluster, to provide a similar type of independence and redundancy at this level as well.

[0126] Still, from a practical standpoint, cost and complexity are also important factors when designing mission-critical, highly available systems, according to one embodiment. Thus, different deployments may use different levels of redundancy and different levels of multiple infrastructure types (i.e., more than one) to suit the budget and risk scenarios for the associated system deployment.

[0127] According to one embodiment, a fully redundant private fabric is still subject to the propagation of management and congestion issues / bugs, as well as "disturbances" caused by link failures and reconfigurations from one redundant part to the other when implemented as a single subnet.

[0128] According to one embodiment, to provide two (or more) fully independent or semi-independent fabrics, the hosts may include redundant connectivity to each fabric to prevent loss of connectivity if pairs of hosts each lose a single connection to a different independent fabric.

[0129] According to one embodiment, the problem is to provide redundant connectivity between two fabrics that is independent of the normal management of each fabric and that is used for data communication when no other options exist.

[0130] FIG. 14 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment.

[0131] According to one embodiment, there may be two or more rails, such as rail A 1401 and rail B 1421. Although not shown, each independent rail may include one or more interconnected switches and multicast proxies (MC proxies), such as MC proxy 1402 and MC proxy 1422. Additionally, each rail may include a highly available path service (HAPS), such as HAPS 1403 and HAPS 1423. A rail may be connected to several hosts, such as host 1 1430 through host N 1440. Although not shown, each host may include one or more end nodes connected to the rail via one or more host channel adapters. Additionally, the end nodes may include one or more virtual machines (e.g., utilizing a virtual switch, virtual port, or other similar architecture described above or similar in nature), as described above with respect to a virtualized environment. According to one embodiment, each host may include a multipath selection component, such as MP selection 1431 and MP selection 1441.

[0132] According to one embodiment, the term "rail" may be used to identify both two or more independent fabrics / subnets and redundant connectivity / links from hosts. Each rail may provide redundant, isolated, point-to-point (for unicast) or point-to-multipoint (for multicast) traffic between end nodes.

[0133] According to one embodiment, Rail A and Rail B may be connected via one or more Inter-Rail Links (IRLs).

[0134] According to an embodiment, the term "IRL" (Inter Rail Link) refers to It can be thought of as similar to an Inter Switch Link (ISL). However, the IRL can be used in a limited manner by not being part of any rail.

[0135] According to one embodiment, the term "MC Proxy" refers to a multicast proxy. The MC proxy may include a high availability component that forwards selected multicast packets from one rail to the other (e.g., Address Resolution Protocol (ARP) requests).

[0136] According to one embodiment, the term "HAPS" may refer to a High Availability Path Service. HAPS may include a high availability component that operates within the context of one rail, but communicates with peers in other rails to enable unicast data traffic forwarding over the IRL whenever two hosts need / require to communicate.

[0137] According to one embodiment, the term "MP selection" may refer to host-based HA / multipath logic that selects which rails to use for different connections to different peer hosts.

[0138] According to one embodiment, in a single rack topology, two leaf switches in a rack can represent two rails (i.e., a minimal topology in which the two rails are separated by hardware). There can be at least two IRLs between the leaf switches. In the case of an InfiniBand topology, each leaf switch can be a single subnet with an embedded subnet manager that is the master subnet manager for each respective rail.

[0139] According to one embodiment, in a multi-rack topology, two leaf switches in each rack can represent two rails. Each leaf switch in a single rack represents a different rail. At least one spine switch in each rack. The set of spines is divided into two groups, one in each rail. There may be special handling for dual and three-rack configurations to avoid a single spine being a SPOF (single point of failure) in any rail. A leaf switch in one rail is connected to all spines in the same rail (but not to other rails). For M>1, the IRL is between two or more pairs of spines from each rail. For InfiniBand topologies, redundant SMs can be provided and located on two or more switches (or possibly dedicated hosts) in each rail.

[0140] According to one embodiment, in both single-rack and multi-rack topologies, fat-tree routing can ignore inter-rail links. In the case of InfiniBand topologies, different rails are configured with different M_Keys (Management Keys) to ensure there is no subnet manager interaction / interference across IRLs.

[0141] According to one embodiment, the "HA Path Service" (HAPS) can track the complete HCA node and port population within each subnet (which may also include system image GUIDs for handling hosts with multiple HCA configurations).

[0142] According to one embodiment, HAPS can use GID in-service / GID out-of-service event notifications from SA or a special protocol. If HAPS is implemented as a host-based service (which may be co-located with the MC proxy instance), then by default no special protocols are required to track node population, but HAPS will have a scope limited by the partitions that its associated HCA ports can be members of.

[0143] According to one embodiment, a HAPS implementation co-located with the master SM may A partition can have more direct interaction with a particular object and would not be limited to representing only a particular partition.

[0144] According to one embodiment, HAPS can keep track of switches in each L2 subnet that have "cross-link" ports and ensure proper connectivity, similar to how an "inter-subnet manager" can ensure proper connectivity between peer router ports.

[0145] According to one embodiment, HAPS can establish unicast LID forwarding for "remote LIDs" that require unicast forwarding over cross-links. This can in principle be done "independently" from the master SM in the relevant L2 subnet. The requirement would be that the SM could be instructed (configuration policy to use a specific LID, but still set the "LinearFDBTop" value for each switch based on another configuration parameter). In this way, the master SM of each L2 subnet would operate with non-overlapping LID ranges, but the switches of each L2 subnet would still be able to forward unicast packets with DLID values ​​in the ranges belonging to other (redundant) L2 subnets.

[0146] According to one embodiment, the LID range boundaries are determined by the Linear Forwarding Table (LFT). As long as the LFT blocks are aligned on LFT block boundaries, it is possible for the HAPS to update the LFT blocks to manage unicast cross-connectivity independently of (and simultaneously with) the master SM in the local subnet. The actual updates may be performed directly via SMP operations or via a special agent on the switch.

[0147] According to one embodiment, an alternative scheme is for the HAPS to request the local SM to consider the associated cross-link port to represent all remote LIDs currently required for remote connectivity via this cross-link port. (This is similar to the way router ports are handled, but this would be an entirely new SM operation, whereas router ports only require a single LID to be handled during the normal subnet discovery / initialization process.) In this embodiment, the local master SM / SA can be provided with a path record for the remote port GID that reflects the associated DLID (similar to how an "inter-subnet manager" provides the local master SM with path records for remote ports in a router-based context). Without the router port, the SA can look up the path record based on the information provided, but understand that the cross-link port is a "local destination" in the local subnet.

[0148] According to one embodiment, when this unicast cross-link processing is combined with a scheme that does not require a path record query (e.g., as described in “SYSTEM AND METHOD FOR SUPPORTING NODE ROLE ATTRIBUTES IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT” in application Ser. No. 15,416,899, filed Jan. 26, 2017), No. 7,991,006, entitled "Filtering Redundant Packets in Computer Network Equipments," which are incorporated herein by reference in their entireties, the local SM / SA does not need to know anything about the route to the remote port GUID.

[0149] FIG. 15 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment.

[0150] According to one embodiment, this diagram illustrates a single rack implementation for supporting redundant independent networks in a high performance computing environment.

[0151] According to one embodiment, in a single rack topology, a rack may include two or more leaf switches 1512-13. A rack may additionally and optionally include a number of other switches 1511. These additional switches are optional, as two leaf switches are the minimum base configuration for a single rack topology.

[0152] According to one embodiment, in a single rack topology where two leaf switches are interconnected by a set of inter-switch links (ISLs) or intra-rail links (IRLs), two or more rails can be defined through leaf switch partitioning by assigning each leaf switch to a separate rail.

[0153] According to one embodiment, in the absence of leaf switch partitioning, there may generally be constraints on which port numbers can be used for ISLs, depending on switch hardware specific constraints on packet buffer allocation and switch HW resource organization.

[0154] According to one embodiment, in such a situation, each leaf switch 1512 and 1513 represents a separate rail, where an ISL represents an IRL. In the particular case of an InfiniBand fabric, each leaf switch represents a single subnet with an embedded subnet manager that is the master subnet manager.

[0155] According to one embodiment, in a situation where the ports on each leaf switch are partitioned to provide two rails within each leaf switch, the total number of ports in each leaf switch is divided into two partitions, with each partition representing an independent or semi-independent rail. Within a RoCE fabric partition, there may also be restrictions on which port numbers can be used for ISLs. (By default, the same set of ports will be used for ISLs in both partitioned and non-partitioned leaf switches.) According to one embodiment, to reduce the number of switches required to realize both the RoCE fabric and access to client networks from private fabric-based systems in smaller (e.g., quarter rack) configurations, leaf switches may be partitioned (e.g., ports on the leaf switches) and used to realize the private RoCE fabric into one physical partition (i.e., set of physical ports / connectors) representing the private RoCE fabric and another non-overlapping partition representing access to client networks (e.g., on-premise data center networks). Thus, each such physically partitioned switch may have two non-overlapping sets of ports, where only the set of ports dedicated to the private RoCE fabric can represent connectivity within the RoCE fabric (and vice versa).

[0156] FIG. 16 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment.

[0157] According to one embodiment, in a multi-rack topology, there may be X racks, with each rack including a number of switches, including at least some leaf switches. In the illustrated embodiment, the system 1600 includes X racks, including rack 1 1610, rack 2 1620, rack X-1 1630, and rack X 1640. Each rack includes multiple Rack 1 includes leaf switches 1612-13 and spine switch 1611, rack 2 includes leaf switches 1622-23 and spine switch 1621, rack X-1 includes leaf switches 1632-33 and spine switch 1631, and rack X includes leaf switches 1642-43 and spine switch 1641. Each spine switch is also connected to two inter-rail links as shown in the figure.

[0158] According to one embodiment, in a multi-rack topology, each rack contains at least two leaf switches, which belong to separate rails. In the figure, the rail to which each switch belongs is indicated by either a "1" or a "2" shown on each switch. Similarly, in a multi-rack topology, there is at least one spine switch in each rack. The set of spine switches is divided into two groups, one for each rail. Leaf switches in one rail are connected to all spine switches in the same rail, but not to spine switches in different rails. There can be more than one M IRL between two or more pairs of spine switches from each rail. In an InfiniBand topology, redundant subnet managers are on two or more switches (or dedicated hosts) in each rail.

[0159] According to one embodiment, each leaf switch in each rack has a set of uplinks distributed among all spine switches in the fabric. In general, depending on switch HW specific constraints on packet buffer allocation and switch HW resource organization, there may be constraints on which port numbers can be used for uplinks. Each spine switch has a certain number of downlinks distributed among all leaf switches in the fabric.

[0160] According to one embodiment, as long as the spine switch is only supposed to provide downlink connectivity, there need not be any difference in characteristics between different ports. However, in a particular configuration, there may still be reasons to limit the port numbers that can be used for downlinks, since not all spine ports may be utilized (connected).

[0161] According to one embodiment, in the case of a multi-rack configuration, there is typically no use of partitioned leaf switches. On the other hand, as described above, a single-rack configuration may or may not use partitioned leaf switches. Thus, the fabric definition for a single-rack configuration may also include information regarding whether the leaf switch configuration used is partitioned or not. However, a smaller multi-rack configuration created based on expanding an existing single-rack configuration may also use partitioned leaf switches in one or more racks.

[0162] According to one embodiment, both the single rack and multi-rack topologies can support two different embodiments: fully independent rails and semi-independent rails.

[0163] According to one embodiment, in the case of a fully independent rail topology, each rail consists of an independent set of switches and there is no connectivity between switch ports belonging to different rails. A typical use case for this would be having two or more dual-port adapters per server.

[0164] According to one embodiment, in such a case, each server (e.g., host) may have redundant connectivity to each rail. Thus, the lack of a single point of failure for any single adapter or single adapter port for any server means that the corresponding server can This can lead to the inability to send and receive data traffic on individual fabric rails.

[0165] According to one embodiment, if two servers do not both have connectivity to at least one common rail, then a pair of servers (or any pair of VMs - one for each of the server pairs) cannot belong to the same logical cluster when fabric-based communication between cluster nodes is required.

[0166] According to one embodiment, when non-overlapping subsets of adapters in each server are connected to non-overlapping sets of rails (i.e., no adapter has a port that connects to more than one of the non-overlapping sets of rails), the different rails are also independent with respect to communication protocols and software and firmware versions, including both switch firmware, fabric management software, adapter firmware and adapter driver software.

[0167] According to one embodiment, the systems and methods described herein can further support semi-independent rails.

[0168] According to one embodiment, in a semi-independent rail topology, each rail consists of an independent set of switches, and there is no connectivity between switch ports belonging to different rails that is normally used for data traffic. However, there may be "dormant" physical connectivity between switches in different rails that is used to provide connectivity between pairs of servers that would not otherwise be able to communicate because they do not both have operational connectivity to the same rail. Such connectivity may be achieved by IRLs or by other means.

[0169] According to one embodiment, a typical use case for this configuration is when you typically have only a single dual-port adapter where each adapter port is connected to a leaf switch on a different rail, in which case any single port / link failure for any server means that it cannot send or receive data traffic on the corresponding fabric rail.

[0170] According to one embodiment, if two servers do not both have connectivity to at least one common rail, some of the "dormant" connectivity between switches in different rails may be utilized to re-establish connectivity between this particular pair of servers, or alternatively, the pair of servers (or one for each of any pair of VMs-server pairs) cannot belong to the same logical cluster when fabric-based communication between cluster nodes is required.

[0171] FIG. 17 illustrates a system for supporting redundant independent networks in a high performance computing environment, according to one embodiment.

[0172] More specifically, the figure shows a dual-rail topology with an IRL gateway.

[0173] According to one embodiment, there may be two or more rails, such as rail A 1701 and rail B 1721. Although not shown, each independent rail may include one or more interconnected switches and a multicast proxy (MC proxy), such as MC proxy 1702 and MC proxy 1722. Additionally, each rail may include a highly available path service (HAPS), such as HAPS 1703 and HAPS 1723. The rail may be connected to several hosts, such as host 1 1730 through host N 1740. Although not shown, each host may include one or more end nodes connected to the rail via one or more host channel adapters. Additionally, the end nodes may include one or more virtual machines (e.g., utilizing virtual switches, virtual ports, or other similar architectures described above or similar in nature), as discussed above with respect to the virtualized environment. According to one embodiment, each host may include a multipath selection component, such as MP selection 1731 and MP selection 1741.

[0174] According to one embodiment, instead of direct switching for switching connectivity to the inter-rail links 1705-1708, several gateway instances 1750 and 1752 can be provided, each gateway instance providing a packet processing engine 1751 and 1753.

[0175] According to one embodiment, a packet processing engine (PPS) may be provided at a node in the topology, such as a gateway node.

[0176] According to one embodiment, to increase the level of independence between the redundant fabrics, a dual-port high performance packet processing engine (PPS) can be used in place of direct switch-to-switch links for both control and data traffic.

[0177] According to an embodiment, these types of packet processing engines can be used in a scalable manner for additional purposes. These PPSs can be used to provide a firewall between different system instances. The PPSs can be used to provide a gateway to a cloud / data center network to connect different private fabrics. The PPSs can be used to provide a gateway between an IB-based private fabric and an Enet (Ethernet)-based private fabric. The PPSs can be used to provide a gateway between a private fabric and a client network.

[0178] According to one embodiment, the system and method can track physical and logical connectivity. This can be accomplished by utilizing leaf switch monitoring of connected end ports and inter-switch connectivity. Additionally, the system and method can utilize a hierarchical query and reporting scheme to distribute all relevant connectivity and power-on information for all relevant end nodes and ports to all relevant peer nodes. Such reporting can also include nodes and ports that have full connectivity to their local leaf switches but limited connectivity in intermediate fabrics.

[0179] Additionally, the system and method can utilize powered on adapter / NIC firmware / driver and check schemes to detect and report node power off (in addition to link failure) to reduce / avoid the need for additional peer-peer checks in addition to facilitating route rebalancing and failover.

[0180] According to one embodiment, the system and method can support multicast, address resolution, and routing. Idempotent multicast operations such as ARP can occur in parallel on multiple rails. Selective duplication is allowed or multicast traffic can be forwarded to ensure "at least once" semantics for multicast. Traffic can use an extended protocol that allows a receiver to process a single MC message only once. Multiple simultaneous address resolution requests to the same node can be answered by multiple interfaces on different rails, and the requester can then select which rail to use for further communication.

[0181] FIG. 18 is a flowchart of a method for redundant independent networks in a high performance computing environment, according to one embodiment.

[0182] At step 1810, the method may include providing, in a computer including one or more microprocessors, one or more switches and one or more racks, each of the one or more racks including a set of one or more switches, each set of one or more switches including at least a leaf switch, and further providing a plurality of host channel adapters, at least one of the plurality of host channel adapters including firmware and a processor, and further providing a plurality of hosts.

[0183] In step 1820, the method can provision two or more rails, which provide redundant connectivity between multiple hosts.

[0184] In step 1830, the method can separate data traffic between the multiple hosts onto one of two or more rails.

[0185] According to one embodiment, when implementing a highly available cluster network / fabric, it is important that redundancy is implemented in a manner that minimizes the risk that a problem in one area of ​​the fabric will propagate to other redundant areas of the fabric.

[0186] Also according to one embodiment, when recovery or failover actions are required within the fabric, it is important that such actions do not impose a load on either the control plane or data plane infrastructure that could cause significant performance or forward progress problems.

[0187] According to one embodiment, to scale system size and be compatible with legacy high availability host communication runtime systems, each host can have redundant interface connectivity to the fabric, and each such redundant interface can reach any other interface in the redundant fabric. Notably, this implies that if two hosts each have a problem with one interface, they should still be able to communicate using the remaining operational interface. Thus, interface redundancy can be applied to each individual host and is completely independent of which interfaces are available on the other host.

[0188] According to one embodiment, whenever a host interface or switch fails, it is possible to re-establish such communication, independent of which interface is used to initiate the relevant communication. This implies that network-level redundancy cannot be based on two completely independent networks, where packets injected into one of the networks cannot be forwarded to the other network. Therefore, to support legacy HA communication schemes while maximizing the independence between redundant networking components, a "semi-independent rails" model can be used.

[0189] Goals for a "semi-independent" HA fabric: According to one embodiment, each redundant host interface should connect to a separate L2 subnet (aka "rail") in the HA fabric.

[0190] According to one embodiment, there can be a single broadcast domain that spans two L2 subnets, allowing ARP requests from a single interface to reach all other operational host interfaces, regardless of which L2 subnet each such interface is directly connected to.

[0191] According to one embodiment, data traffic (e.g., RDMA) between hosts should not cross between L2 subnets as long as at least one of the L2 subnets has at least one connected and operational interface for each of the hosts.

[0192] According to one embodiment, whenever any pair of hosts that need to communicate are not both able to establish data traffic between interfaces on a single L2 subnet, a route between L2 subnets should be established for the necessary data traffic between the involved hosts.

[0193] According to one embodiment, the host stack on each host can easily determine which interface to use for communication with particular other hosts, even if the default decisions on each host involve different "rails."

[0194] According to one embodiment, in the case of InfiniBand, it should not be possible for a host to initiate an SA request that crosses an L2 subnet boundary.

[0195] According to one embodiment, it should not be possible for a host to cause congestion that spreads from one L2 subnet to another.

[0196] Specific examples of InfiniBand Fabric implementations: HA Path Service (HAPS) According to one embodiment, the "HA Path Service" (HAPS) can track the complete HCA node and port population within each subnet (which may also include system image GUIDs for handling hosts with multiple HCA configurations).

[0197] According to one embodiment, HAPS can use GID in-service / GID out-of-service event notifications from SA or a special protocol. When HAPS is implemented as a host-based service (which may be co-located with MC proxy instances), no special protocols are required by default to track node population, but HAPS has a scope limited by the partitions that its associated HCA ports can be members of.

[0198] According to one embodiment, a HAPS implementation co-located with the master SM may have more direct interaction with the SM and would not be limited to only representing a specific partition.

[0199] According to one embodiment, HAPS can keep track of switches in each L2 subnet that have "cross-link" ports and ensure proper connectivity, similar to how an "inter-subnet manager" can ensure proper connectivity between peer router ports.

[0200] According to one embodiment, HAPS can establish unicast LID forwarding for "remote LIDs" that require unicast forwarding over cross-links. This can in principle be done "independently" from the master SM in the relevant L2 subnet. The requirement would be that the SM could be instructed (configuration policy to use a specific LID but still set the "LinearFDBTop" value for each switch based on another configuration parameter). In this way, the master SM in each L2 subnet would work with non-overlapping LID ranges, but the switches in each L2 subnet would still be able to forward unicast packets with DLID values ​​in the ranges belonging to other (redundant) L2 subnets.

[0201] According to one embodiment, the LID range boundaries are determined by the Linear Forwarding Table (LFT). As long as the LFT blocks are aligned on LFT block boundaries, it is possible for the HAPS to update the LFT blocks to manage unicast cross-connectivity independently of (and simultaneously with) the master SM in the local subnet. The actual updates may be performed directly via SMP operations or via a special agent on the switch.

[0202] According to one embodiment, an alternative scheme is for the HAPS to request the local SM to consider the associated cross-link port to represent all remote LIDs currently required for remote connectivity via this cross-link port. (This is similar to the way router ports are handled, but this would be an entirely new SM operation, whereas router ports only require a single LID to be handled during the normal subnet discovery / initialization process.) In this embodiment, the local master SM / SA can be provided with a path record for the remote port GID that reflects the associated DLID (similar to how an "inter-subnet manager" provides the local master SM with path records for remote ports in a router-based context). Without the router port, the SA can look up the path record based on the information provided, but understand that the cross-link port is a "local destination" in the local subnet.

[0203] According to one embodiment, if this unicast cross-link processing is combined with a scheme that does not require a path record query, the local SM / SA does not need to know anything about the route to the remote port GUID.

[0204] Identifying and handling "cross-link" ports According to one embodiment, by default, redundant InfiniBand L2 subnets may be configured with non-overlapping M_Key ranges before any cross-link connectivity is provided and / or before any SMs are active in either L2 subnet. In this way, the SMs in each redundant subnet will not attempt to discover or configure anything beyond the switch ports in their local subnets that connect to neighboring redundant subnets.

[0205] According to one embodiment, it is not expected that existing operational single subnet-based systems can be converted to a dual L2 subnet configuration without significant interruption to normal operations, and therefore, such reconfiguration is expected to occur in maintenance windows where system services are not expected to be operational.

[0206] According to one embodiment, vendor-specific SMA attributes can be used to establish a protocol that allows explicit configuration of switches as belonging to a particular redundant L2 subnetwork, and also which switch port numbers are assumed to represent cross-link connectivity to peer redundant L2 subnetworks.

[0207] According to one embodiment, the use of vendor-specific SMA attributes would be similar to how an "Inter-Subnet Manager" handles virtual router ports, however the implementation would be quite different since there are no router ports or port virtualization in this context.

[0208] According to one embodiment, based on detailed configuration information and possibly reliance on node description substrings, it is possible to identify and handle cross-link ports without using special vendor-specific SMA attributes, but this would be more complex and more subject to configuration errors.

[0209] Specific realizations of Ethernet (Private) Fabric: According to one embodiment, redundancy in the interconnection fabric should ensure as much as possible that failures / problems in one redundant part do not propagate to the other part. Ultimately, this means physically and logically independent infrastructures. However, the price of this is that each node may have redundant connectivity to each such independent fabric, or the ability to restore connectivity between two or more servers, each having a single link problem, is greatly reduced. By utilizing a new method for providing Ethernet link connectivity between two independent subnets, without affecting management or fault containment for each individual subnet, both goals can be addressed simultaneously.

[0210] According to one embodiment, an Ethernet Private Fabric may be realized using traditional Ethernet unicast forwarding schemes with a combination of spanning tree and link aggregation variants, or may be realized using explicit forwarding of individual unicast destination addresses in the same manner that forwarding of IB packets within a single IB subnet is realized.

[0211] According to one embodiment, when explicitly forwarding individual unicast addresses, the high level scheme is as follows: According to one embodiment, a complete topology is collected from a set of relevant switches in terms of port-to-port connectivity between switches and ports, and between switch ports and end ports. (Optimization may only involve collecting topology deltas after an initial complete discovery.) According to one embodiment, the topology information is converted into a format that can be processed by the same routing logic (aka routing engine) used for the corresponding InfiniBand fabric.

[0212] According to one embodiment, the routing engine uses the topology along with the policies over which physical end ports are allowed to communicate (e.g., VLANs) (as well as other associated optimization and distribution criteria as in the IB fabric) and generates a list of tuples mapping end port addresses to ports for each switch instance in the topology.

[0213] According to one embodiment, the resulting (delta) forwarding entry list is distributed to the switches (ie, those that require updating).

[0214] According to one embodiment, multicast may be implemented in either case using legacy Ethernet methods for both end port membership processing and forwarding of MC packets.

[0215] Multicast Proxy Service According to one embodiment, the same / similar considerations apply as in the InfiniBand case: if the L2 source address in the proxied MC packet is not the same as the corresponding ARP request "sender hardware address", the ARP request may be dropped or the generation of a unicast ARP reply may result in an incorrect L2 destination address. As in the InfiniBand case, to avoid reliance on special host stack processing, the best approach is for the proxy to send the proxied MC packet with a source L2 address corresponding to the original sender in the other L2 subnet.

[0216] HA Path Service (HAPS) According to one embodiment, the "HA Path Service"-HAPS can track the complete end port population in each subnet.

[0217] According to one embodiment, ports that belong to the same NIC or the same host can be correlated between two subnets.

[0218] According to one embodiment, it is possible to identify situations where a pair of hosts can communicate only if the associated unicast traffic is forwarded over a cross-link between the two subnets, such as in the IB case.

[0219] According to one embodiment, if explicit end port address forwarding is used for unicast traffic, the process will be very similar to the IB case, however, for legacy Ethernet unicast forwarding, and in the general case, this may also require a proxy-style gateway function for unicast traffic.

[0220] According to one embodiment, since unicast forwarding in the Ethernet fabric is MAC based, there are no restrictions on the MAC ranges used for the various L2 subnets - the only constraint is that each host port should have at least a unique MAC within the associated site / domain. In the case of a private fabric, the associated domains are two redundant L2 subnets.

[0221] According to one embodiment, if a switch is capable of performing forwarding based on either L2 and / or L3 addresses, it is also possible to use forwarding based on L3 (IP) addresses instead of L2 MAC.

[0222] Identifying and handling "cross-link" ports According to one embodiment, based on individual switches being configured to belong to different "rails" combined with neighbor connectivity information from each switch, it is possible to identify both switch-switch connectivity and intended (or accidental...) cross-link connectivity that belongs to the same rail / subnet (fat tree) topology.

[0223] According to one embodiment, cross-link connectivity will never be considered for normal forwarding of either unicast or multicast connectivity in that case, but the intended cross-connectivity will be used to back up inter-rail paths in the case of explicit end port address forwarding. As pointed out above, the use of cross-links when legacy Ethernet unicast forwarding schemes are used would generally require a proxy / gateway function for both multicast and unicast forwarding between different rails.

[0224] According to one embodiment, when configuring Ethernet switches and NICs for RoCE-based RDMA traffic, the associated links typically have available packet reception Whenever the buffer capacity falls below a certain threshold, the downstream switch or NIC port is configured in a "lossless" mode that generates a "frame stop" to the upstream transmitting switch or NIC port. The transmitting port then stops (pauses) from sending more packets with the associated priority (if any) until the downstream port again has buffer capacity above a certain threshold.

[0225] According to one embodiment, like other networking technologies where packet flow control can lead to network-wide backpressure, a lossless RoCE fabric topology must be routed in a deadlock-free manner, and flows that are intended to have forward progress independent of each other must use different priorities.

[0226] According to one embodiment, in the case of a dual-rail topology, it makes sense to ensure that only the internal topology of a single rail must be considered for deadlock-freeness. By configuring the cross-links as "lossy" (i.e., downstream switch ports do not send frame pauses), it is possible to provide routing of the deadlock-free topology in each rail without considering the topology in the rails connected by the cross-link. Instead, the cross-link ports are considered end ports in the local rail from a deadlock-free routing perspective, and therefore cannot be part of cycle dependencies that can lead to deadlocks.

[0227] According to one embodiment, the lossless cross-link configuration also means that congestion on one rail does not spread to the other rail, thus satisfying the key requirement of isolation between redundant rails.

[0228] Rail Number Association According to one embodiment, a private InfiniBand or RoCE physical fabric configuration (as used herein, the terms "fabric" or "private fabric" may refer to a private InfiniBand or RoCE-based fabric) is defined in terms of a particular number of racks, each rack with an associated set of switches, and each such associated switch having a particular role (leaf or spine).

[0229] According to one embodiment, similar to access to client networks from private fabric-based systems in smaller (e.g., quarter rack) configurations, one scheme is to "partition" the leaf switches used to realize the private fabric into one physical partition (i.e., set of physical ports / connectors) that represents the private fabric and another non-overlapping partition that represents access to the client networks (e.g., on-premise data center networks). Each such physically partitioned switch would thus have two non-overlapping sets of ports, where only the set of ports dedicated to the private fabric can represent connectivity within the private fabric (and vice versa).

[0230] According to one embodiment, in the case of a multi-rack configuration, there is typically no use of partitioned leaf switches. On the other hand, as described above, a single-rack configuration may or may not use partitioned leaf switches. Thus, the fabric definition for a single-rack configuration would also include information regarding whether the leaf switch configuration used is partitioned or not. However, a smaller multi-rack configuration created based on expanding an existing single-rack configuration may use partitioned leaf switches in one or more racks. It can also be used.

[0231] According to one embodiment, to support a fabric configuration having two or more independent networks or "fabric rails" or simply "rails," each switch will also be associated with a rail number. Each such "rail" will thus represent a topology of one or more switches with connectivity and traffic patterns that are independent of the other rails. (i.e., there will be no links between a switch in one rail and another switch in another rail that are typically used for data traffic.) According to one embodiment, the private fabric definition facilitates the following high-level features: ● The existence and availability of all defined switches can be automatically checked at run-time and exceptions can be reported immediately. ● The configuration of each switch (and local switch ports) can be automatically defined and verified according to its role and connectivity within the complete fabric. • The physical connectivity of the complete fabric can be verified according to predefined rules for each type of system. ● Incorrect and / or degraded connectivity between switches in the fabric (i.e., misplacement of cables and / or lack of a sufficient number of links) can be automatically detected and reported, while preventing incorrect connectivity (e.g., to prevent potential packet forwarding loops or deadlocks) ensures that it does not have additional negative effects on data traffic.

[0232] A special case is the scenario where a link is disabled due to a high error rate. Any connectivity between a defined switch in a private fabric and another switch or other network device that represents an unexpected device in the defined fabric can be reported and not impact the operation of the fabric.

[0233] According to one embodiment, the basic fabric definition outlined above focuses on switch topology and is not related to the number and / or type of servers that are (or may be) part of the fabric. Thus, by default, any number of physical servers may be present in the fabric, up to the maximum number represented by the various fabric configuration types. Also, such servers may be dynamically added and removed without affecting the basic fabric definition. However, there are few scenarios in which such private fabric-based systems are highly dynamic in the sense that physical servers are added, removed or replaced on a daily basis - the physical configuration is typically expected to remain the same over many weeks and months.

[0234] According to one embodiment, based on the expectation that the server configuration will remain stable over a significant period of time, the fabric definition may be optionally extended with parameters that define the expected number and type of servers that will be present in the system and / or in each rack.

[0235] According to one embodiment, this extended fabric definition facilitates the following additional high-level features: ● The presence and availability (with respect to connectivity to the private fabric) of all defined servers can be automatically checked at runtime and exceptions can be reported immediately. Servers (types and / or The presence (with respect to connectivity to the private fabric) of a single or multiple VLANs can be immediately detected and reported. ● Additional policies can determine whether unexpected server types or excess numbers should be accepted for data traffic, but the default would be "report and accept." In a single rack configuration with partitioned leaf switches, the connectivity observed between a server and the private fabric can be correlated with the connectivity observed between the corresponding server and the client network (i.e., as long as a policy exists regarding the connection of servers to the client network).

[0236] o It is possible to identify any server that is connected to a client network but not to a private fabric (and vice versa for servers that are expected to have client network connectivity).

[0237] o In addition to the connectivity between the servers in the local rack and the client network, it is also possible to monitor the uplink connectivity for the client network switch partitions.

[0238] Supported fabric topologies: According to one embodiment, for a "single rail" system, the supported topologies can be divided into two main classes: A single rack topology with two leaf switches interconnected by a set of Inter-Switch Links (ISLs): If there is no leaf switch partition: ◆ In general, depending on switch HW specific constraints on packet buffer allocation and switch HW resource organization, there may be constraints on which port numbers can be used for ISLs.

[0239] With leaf switch division: ◆The total set of switch ports is divided into two partitions.

[0240] Within a RoCE fabric partition, there may again be restrictions on which port numbers can be used for ISLs (by default, the same set of ports is used for ISLs in both partitioned and non-partitioned leaf switches). ◆ In case of a client network partition, there may be restrictions or defaults on which ports are used for local server connectivity and which ports are used for the uplink. • Multi-rack topology (max N racks) where each rack has at least two leaf switches and two or more spine switches are located in separate racks.

[0241] o Each leaf switch in each rack has a set of uplinks that are distributed among all the spine switches in the fabric.

[0242] ◆ In general, depending on switch HW specific constraints on packet buffer allocation and switch HW resource organization, there may be constraints on which port numbers can be used for the uplink.

[0243] o Each spine switch has a certain number of downlinks that are distributed among all the leaf switches in the fabric.

[0244] ◆ As long as spine switches are only supposed to provide downlink connectivity, there is no difference in characteristics between different ports. However, in a particular configuration, not all spine ports may be utilized (connected), so there is still a reason to restrict the port numbers that can be used for downlinks.

[0245] According to one embodiment, in the case of a multi-rail topology, each individual fabric rail can correspond to either a single rack or a multi-rack topology, as outlined above. However, within each rack, different leaf switches can belong to different rails, and the full set of spine switches can be divided among the defined rails. To increase the level of redundancy and bisection bandwidth within each rail, the total number of switches across all rails in a multi-rail configuration can be higher than in a corresponding single-rail configuration.

[0246] According to one embodiment, there are two main models for fabric rail independence: fully independent rail and semi-independent rail. ●Fully independent rails: Each rail consists of an independent set of switches and there is no connectivity between switch ports belonging to different rails.

[0247] o A typical use case for this would be having two or more dual port adapters per server.

[0248] In this case, each server can have redundant connectivity to each rail, so there is no single point of failure for a single adapter or single adapter port for any server that could result in the corresponding server being unable to send and receive data traffic on any fabric rail.

[0249] o If the two servers do not both have connectivity to at least one common rail, then a pair of servers (or any pair of VMs - one for each of the pair of servers) cannot belong to the same logical cluster, which requires fabric-based communication between the cluster nodes.

[0250] o If a non-overlapping subset of adapters in each server is connected to a non-overlapping set of rails (i.e., no adapter has a port that connects to more than one of the non-overlapping sets of rails), the different rails are also independent with respect to communication protocols and software and firmware versions, including both switch firmware, fabric management software, adapter firmware and adapter driver software. Semi-independent rail: o Each rail consists of an independent set of switches, and there is no connectivity between switch ports belonging to different rails that is normally used for data traffic. However, there may be "dormant" physical connectivity between switches in different rails that is used to provide connectivity between pairs of servers that would not otherwise be able to communicate because they do not both have operational connectivity to the same rail.

[0251] o A typical use case for this configuration is when each adapter port has only a single dual port adapter, typically connected to a leaf switch on a different rail.

[0252] ◆In this case, any single port / link failure for any server means that it cannot send or receive data traffic on the corresponding fabric rail.

[0253] o If two servers do not both have connectivity to at least one common rail, some of the "dormant" connectivity between switches in different rails may be utilized to re-establish connectivity between this particular pair of servers, or the pair of servers (or any pair of VMs - one for each of the server pairs) cannot belong to the same logical cluster, which requires fabric-based communication between cluster nodes.

[0254] According to one embodiment, each switch may have persistent knowledge of the type of fabric it belongs to and its own role and rail association in that fabric.

[0255] According to one embodiment, each switch may have persistent knowledge of the identity, role, and rail association of every other switch in a defined fabric.

[0256] According to one embodiment, each switch can reflect its fabric-related configuration information to its directly connected peer ports via exported LLDP type information.

[0257] According to one embodiment, each leaf switch in a single rack fabric may have prior knowledge of the required ISL connectivity (if any) to other leaf switches.

[0258] According to one embodiment, each partitioned leaf switch in a single rack fabric can track which ports belong to which partition and reflect that partition type as part of the exported LLDP type information.

[0259] According to one embodiment, each leaf switch in a multi-rack fabric may have prior knowledge of the required uplink connectivity to the spine switches in that fabric.

[0260] According to one embodiment, each spine switch in a multi-rack fabric may have prior knowledge of the required downlink connectivity to the leaf switches in that fabric.

[0261] According to one embodiment, each switch can determine the neighbor connectivity of each operational switch port without enabling data traffic through that port.

[0262] According to one embodiment, each switch can dynamically determine whether the discovered neighbor connectivity represents legitimate connectivity for that switch, and if so, how to configure its local ports.

[0263] According to one embodiment, each switch can dynamically configure a switch port according to pre-defined rules for associated legal connectivity and can synchronize this configuration with associated peer ports before allowing data traffic through that port.

[0264] According to one embodiment, each switch communicates its physical neighborhood connectivity with other switches in the fabric. It can be dynamically shared with all switches.

[0265] According to one embodiment, each switch can monitor its ability to communicate with any other defined switches on the management network.

[0266] According to one embodiment, each switch can dynamically share with all other switches in the fabric a list of other switches with which it can communicate.

[0267] According to one embodiment, each switch should provide log information reflecting changes in physical connectivity, along with the status of such connectivity as valid, incorrect, or degraded, and changes in its ability to communicate with other switches in the fabric over a management network.

[0268] According to one embodiment, at the fabric level there should be coordinated logging and event reporting of fabric-level connectivity issues, including lack of management connectivity between pairs / sets of switches and complete loss of reachability to a particular switch from the rest of the fabric.

[0269] According to one embodiment, critical events such as an unexpected (ie, for longer than the expected reboot period) loss of fabric-level contact with a particular switch should be supported as an Automatic Service Request (ASR).

[0270] According to one embodiment, the leaf switches can monitor physical server connectivity according to policies (if any) regarding expected and required server presence and connectivity in the fabric.

[0271] According to one embodiment, the partitioned leaf switches must be able to monitor client network connectivity to servers according to policies (if any) regarding the expected and required presence of servers in the fabric.

[0272] According to one embodiment, in the case of a multi-rail fabric configuration, each switch can ensure that all regular connectivity is between switches that belong to the same rail.

[0273] According to one embodiment, in the case of a semi-independent multi-rail fabric configuration, any physical connectivity between switches in different rails must be verified as complying with the required inter-rail connectivity, and this connectivity should not then be used for data traffic in the normal case.

[0274] Fabric connectivity checks for different switch roles and topologies: According to one embodiment, each individual switch can verify that it has the correct connectivity according to its role and the type of fabric configuration / topology it belongs to: Partitioned leaf switches in a single rack, single rail configuration: o Within a switch partition, there should be at least N1-1 ports representing ISLs.

[0275] If there are constraints on which port numbers can be used as ISLs, then all ports that represent ISLs must be in the set of associated ports.

[0276] ◆There should be a maximum of N1-2 ISLs. Within a switch partition, there can be at most N2 ports that connect to host adapter ports.

[0277] Each ISL must connect to a single remote switch with a partitioned leaf switch role, with the peer ports configured as ISLs, and the remote switch belongs to the same single fabric instance. (Non-partitioned) leaf switches in a single-rack, single-rail configuration: o There should be at least N3-1 ports representing ISLs.

[0278] If there are constraints on which port numbers can be used as ISLs, then all ports that represent ISLs must be in the set of associated ports.

[0279] ●There should be a maximum of N3-2 ISLs. ◆There can be up to N4 ports connected to a host adapter port.

[0280] ◆Each ISL must connect to a single remote switch that is not partitioned and has the leaf switch role, and the peer ports are configured as ISLs and the remote switches belong to the same single fabric instance. Leaf switches in a multi-rack single rail configuration: o There should be at least N5-1 ports representing uplinks.

[0281] ◆ If there are constraints on which port numbers can be used as uplinks, then all ports that represent uplinks must be in the set of associated ports.

[0282] ◆There should be a maximum of N5-2 uplinks. There can be up to N6 ports connected to a host adapter port.

[0283] Each uplink must connect to a switch that has the spine switch role and belongs to the same single fabric instance There must be uplink connectivity to each spine switch defined for the fabric instance.

[0284] The set of uplinks must be divided evenly among all spine switches defined for a fabric instance. Spine switches in a multi-rack single rail configuration: All connected ports should represent downlinks to leaf switches that are defined to be part of the same fabric instance.

[0285] ◆ If there are constraints on which port numbers can be used as downlinks, then all ports that represent downlinks must be in the set of associated ports.

[0286] There should be no ports connecting to host ports. o There may be at least one downlink connecting each leaf switch defined for a fabric instance.

[0287] The set of downlinks must be divided evenly among all leaf switches defined for the fabric instance.

[0288] According to one embodiment, for a multi-rail fabric configuration, the connectivity rules are the same as for the single-rail configuration described above, with the additional rule that such connectivity must all be on the same level. Thus, in the case of a single rack with only two leaf switches, the requirement may be that no ISLs should be connected at all (fully independent rails), or that such ISLs should represent "dormant connectivity" (semi-independent rails).

[0289] According to one embodiment, with respect to server connectivity, connectivity checks can ensure that individual servers have redundant connectivity to the fabric according to a defined single-rail or multi-rail policy.

[0290] According to one embodiment, connectivity that does not comply with the above rules will be logged, reported and not enabled for data traffic.

[0291] According to one embodiment, lost connectivity is recorded and reported, and the involved switches (or servers) are considered to have degraded (or incomplete) connectivity.

[0292] According to one embodiment, the available connectivity is used to direct data traffic within the fabric according to defined configurations and routing / forwarding policies for associated topologies within a single or multiple fabric rails.

[0293] Example of implementation: According to one embodiment, the physical fabric connection control implementation has the following components: ● A switch-based local link monitoring (LLM) daemon tracks local link status, remote connectivity (if any) and local traffic and error counters. ● The switch-based Physical Connectivity Control (PCC) daemon receives local in-band fabric connectivity information from the LLM and, based on this, determines whether the local connectivity complies with the requirements for this switch instance (for its role in the overall fabric configuration) and, if so, what high-level configuration type to apply to the associated switch ports. ● The Hardware (HW) Configuration Daemon (HWCD) configures the switch and switch ports (using switch HW platform specific interfaces) according to the high-level configuration types provided by the PCC for the switch and each connected port with enabled remote connectivity. The Fabric Availability and Synchronization Daemon (FAS) synchronizes fabric configuration information and associated policies among all defined switches in the fabric. The FAS also tracks reachability on the management network for all defined switches in the fabric and coordinates the distribution of this information among the PCC instances on each switch.

[0294] According to one embodiment, based on the persistent fabric configuration information and role information currently defined for the local switch, the PCC constantly evaluates the current local connectivity by correlating the neighbor connectivity information received from the LLM with the expected connectivity defined by the configuration and role information and pre-connectivity rules.

[0295] According to one embodiment, by default all recently trained links will not be enabled for data traffic, so any connectivity that does not comply with the relevant rules will be left in this state. However, whenever connectivity complies with the relevant rules, the PCC will instruct the HW to configure and enable the port according to the determined legal link type (i.e., hostlink, ISL, uplink or downlink). Give commands to the CD.

[0296] According to one embodiment, once a newly trained link is evaluated by a PCC, the associated connectivity information is distributed via the FAS to all other PCCs on all other switches in the fabric.

[0297] According to one embodiment, the FAS on every switch in the fabric will constantly monitor whether every other switch is reachable on the management network and will report the current status to the current master FAS.

[0298] According to one embodiment, a PCC co-located with a master FAS evaluates the complete set of in-band and management network connectivity from both its fellow PCC instances and the FASs at any given time. Based on this evaluation, and based on defined timeout periods for lack of connectivity or reachability, the master PCC will generate an alarm event and associated ASR.

[0299] FIG. 19 illustrates a system for supporting rail number association for proper connectivity in a high performance computing environment, according to one embodiment.

[0300] According to one embodiment, a number of racks are supported in system 1900, including rack 1 1910, rack 2 1920, rack 3 1930, and rack 4 1940. Each of the racks can support a number of switches, both leaf switches (i.e., switches that connect directly to HCAs (not shown)) and a number of spine switches (each switch that is not a leaf switch). These include leaf switches 1912-13, 1922-23, 1932-33, and 1942-43, and spine switches 1911, 1921, 1931, and 1941.

[0301] According to one embodiment, system 1900 is a multi-rail topology where each rail represents a different fabric. In the system shown in Figure 19, there are two independent rails: rail 1 and rail 2 (shown in the figure).

[0302] According to one embodiment, to support a fabric configuration having two or more independent networks or "fabric rails" or simply "rails," each switch is also associated with a rail number (e.g., 1 and 2 as shown). Each such "rail" represents a topology of one or more switches with connectivity and traffic patterns that are independent of the other rails (i.e., there are no data links between a switch in one rail and another switch in the other rail that are typically used for data traffic) (however, there may be inter-rail links for management traffic).

[0303] According to one embodiment, the rails shown are completely independent rails, with each rail consisting of an independent set of switches, and there is no connectivity between switch ports belonging to different rails. A typical use case for this would be having two or more dual-port adapters per server. In this case, each server (not shown) can have redundant connectivity to each rail. Thus, there is no single point of failure with respect to a single adapter or a single adapter port for any server, which could lead to the corresponding server being unable to send and receive data traffic on any fabric rail.

[0304] According to one embodiment, if two servers do not both have connectivity to at least one common rail, then each pair of servers (or any pair of VMs-servers) one at a time) cannot belong to the same logical cluster if fabric-based communication between cluster nodes is required.

[0305] According to one embodiment, when non-overlapping subsets of adapters in each server are connected to non-overlapping sets of rails (i.e., no adapter has a port that connects to more than one of the non-overlapping sets of rails), the different rails are also independent with respect to communication protocols and software and firmware versions, including both switch firmware, fabric management software, adapter firmware and adapter driver software.

[0306] Heterogeneous and asymmetric dual-ray configuration According to one embodiment, systems that use a private RDMA fabric for various types of high performance computing (HPC) related communications, as well as for high performance, high availability databases and other data services, often also have connectivity to different networks that are used, for example, for client access or for access to resources outside the system - including resources across the Internet.

[0307] According to one embodiment, for example, a special case of such a system is a cloud-based system where the entire cloud may contain hundreds of thousands of servers (i.e., hundreds to thousands of server racks) connected via a global cloud network. Within the entire cloud, the servers may be separated into smaller islands or points of delivery (PODs) (e.g., 8-32 server racks in close proximity) that are also interconnected by a high-performance RDMA fabric based either on RoCE, InfiniBand, or other implementations of high-performance RDMA. There may be a set of PODs (a set of datacenters, a ...

[0308] According to one embodiment, traditionally, the performance of global cloud networks has been inferior to that which can be achieved through high-performance, dedicated RDMA fabrics. This is partly because the basic link speeds in cloud networks can be slower than those provided in dedicated RDMA fabrics (e.g., 25 Gb / s vs. 100 Gb / s), but also in particular because the stringent rules for access control and firewall rule enforcement in cloud networks imply schemes with stateful firewall processing of each individual network packet, resulting in lower overall packet rates for each node and significantly increasing message latency between communicating nodes. However, as "prior art" link speeds increase and the level of integration of on-chip logic functions increases with every new hardware generation, it becomes increasingly likely that workloads that typically relied on communication over dedicated, private RDMA fabrics will also be able to utilize cloud networks for RDMA-based communication with "reasonable" performance.

[0309] According to one embodiment, another aspect of such a network is scalability in terms of the number of nodes that can communicate effectively and how flexible nodes can be allocated for a particular distributed workload. A key advantage of a global cloud network is that it allows any node to communicate with any other node, thus providing ultimate node allocation flexibility. However, such a network can be very scalable. Such global connectivity also implies large distances in terms of the number of switch hops, which in the normal case increases message latency and may also increase the risk of performance degradation due to congestion between different flows in the network. Thus, to reduce message latency and the risk of congestion between independent workloads, node allocation algorithms for distributed workloads may typically attempt to allocate nodes in contiguous parts of the network (i.e., adjacent in terms of the number of switch hops).

[0310] However, according to one embodiment, as a result of the ever-increasing level of integration of logic on a single chip, a single switch chip can now provide 100 or more 100 Gb / s ports interconnected by a non-blocking full crossbar. Thus, in a fully non-blocking fat-tree topology, the number of nodes that can communicate within a distance of only three switch hops is on the order of thousands. As a result, very large PODs in the cloud can be built, with private RDMA fabric connectivity between the cloud network and all servers in the POD, with the maximum distance between any such servers being just three switch hops.

[0311] According to one embodiment, the cloud network infrastructure may be available for workloads that use high performance RDMA communications, but a private, dedicated RDMA fabric may still have non-trivial performance advantages. Thus, the use of a private RDMA fabric within individual PODs within the cloud can provide a supplementary, higher performance communications infrastructure in addition to the cloud network, which may make sense for many workloads.

[0312] According to one embodiment, a POD in which a server is connected by both a cloud network and a private RDMA fabric can be regarded as an example of a "fully independent dual rail" configuration in principle, where the cloud network and the private RDMA fabric each may represent either a single rail, or potentially a semi- or even fully independent dual rail configuration. However, a traditional high availability configuration in which each server has dual ports connecting to a single switched network infrastructure with dual leaf switches and a shared spine switch infrastructure may be a typical scenario. Furthermore, due to cost constraints and complexity regarding the number of switches and cables to provide further redundant server connectivity within each such rail, each such rail can also be realized in a non-redundant manner, where the server has one adapter and single port connecting to the cloud network and another adapter and single port connecting to the private RDMA fabric.

[0313] According to one embodiment, the cloud network and the private RDMA fabric represent heterogeneous and asymmetric dual-rail configurations: they both represent full connectivity (i.e., non-degraded case) to all servers in the POD, and they both can achieve associated RDMA-based communication between any pair of servers in the POD, but the best-case communication performance may not be identical between the two rails. In a typical case, the private RDMA fabric may provide the best (expected) performance and therefore will be the primary choice for RDMA-based communication within the POD.

[0314] However, according to one embodiment, a cloud network may be selected for RDMA-based communication even in the following potential scenarios: a) a pair of nodes requiring RDMA-based communication cannot communicate on the private RDMA fabric; b) RDMA-based communication is required between nodes belonging to different PODs; c) the current relevant load on the private RDMA fabric is (sufficiently) high and the cloud network may not be able to communicate on the private RDMA fabric; d) if the current relevant load on the cloud network is (sufficiently) low and the effective communication performance between a pair of nodes in a single POD is better over the cloud network than over the private RDMA fabric; or d) if the SLA (Service Level Agreement) for the relevant workload indicates that the use of the premium communication services offered by the private RDMA fabric is not required or permitted.

[0315] According to one embodiment, an implementation of an asymmetric dual-rail system includes various management modules. These may include, for example, a distributed workload placement orchestrator (DWPO) and a connectivity and rail selection manager (CRSM). The DWPO is typically part of an overall cloud workload management infrastructure and is responsible for allocating individual logical nodes in a distributed workload to physical nodes in the cloud / datacenter. Logical nodes may be implemented as containers in the physical node OS, virtual machines controlled by a hypervisor on the physical node, or bare metal OS / system images on the physical node.

[0316] According to one embodiment, the DWPO can use placement policy information associated with a workload to determine whether placement within a single POD is required (or desired), and if so, what kind of adjacency is required (or desired) between various pairs of logical nodes within the workload. Adjacency parameters and requirements may include latency targets and expected traffic patterns and capacities.

[0317] According to one embodiment, based on correlating the placement policy information for a workload with the current utilization levels and SLAs associated with existing workloads, the DWPO can determine either a best effort or placement with a stringent SLA for the workload being orchestrated. In some cases, this can suggest that the existing workload be reorganized. The placement of new workloads and potential reorganization of existing workloads can involve a single POD or multiple PODs and PODs with or without a private RDMA fabric, depending on the available resources and associated SLA requirements.

[0318] According to one embodiment, the CRSM has one centralized component that is an integral part of the overall cloud workload management infrastructure, and another component that is integrated with the communication runtime system on all logical nodes.

[0319] According to one embodiment, the CRSM, which may be centralized, can work with the DWPO to determine a connection policy for the various logical nodes in the distributed workload regarding primary rail selection and a connection parameter policy for the peer logical nodes with which each logical node is to communicate. In the simplest case, this policy can use default parameters and use the preferred rail (i.e., typically a private RDMA fabric, if present) as long as it provides connectivity for the associated peer nodes. In more complex scenarios, individual or groups of logical node pairs can have explicit rail selection and connection parameter policies for each direction.

[0320] According to one embodiment, each logical node CRSM instance will use policy information from the central CRSM to establish and maintain relevant connections with associated peer nodes. The node local CRSM instances can also adapt connection setups for individual peer nodes according to the currently available connectivity and the load on the associated rails, as well as according to the SLA constraints that exist for the overall workload and communication with the associated individual peer nodes.

[0321] FIG. 20 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations, according to one embodiment.

[0322] According to one embodiment, several host nodes may be supported within system 2000, such as host node 1 2005 and host node 2 2010. Each of the host nodes may include several dual port adapters (e.g., dual port network adapters), such as dual port adapters 2006, 2007, and 2008 on host node 1, and dual port adapters 2011, 2012, and 2013 on host node 2. Each dual port adapter may provide two ports, by which the host nodes may be connected to one or more networks, such as a private fabric 2040 and a cloud fabric 2030, such as a public cloud. As shown, private fabric 2040 includes, for example, two sets of duplicate switches, switch 1 2020 and switch 2 2021, and switch 3 2022 and switch 4 2023, where such pairs of switches represent completely independent rails within the private fabric.

[0323] According to one embodiment, each of the host nodes may include multiple dual port adapters. For example, in the illustrated embodiment, each host node supports three dual port adapters. Of these three dual port adapters, two of the dual port adapters on each host node may be connected to the private fabric 2040, leaving one adapter to connect to the cloud fabric 2030. In this manner, each host node may be connected with two completely independent rails (i.e., switch 1 and switch 2, and switch 3 and switch 4, where each pair of duplicate switches represents a completely independent rail) via the private fabric, and also via the cloud fabric.

[0324] If necessary, modified versions of this drawing can be replicated multiple times (e.g., "Medium" and "Small" below) to show that the private fabric itself can be both semi-independent and fully independent, and that host connectivity can vary from a single adapter with one port connected to the cloud and one port connected to the private fabric, to multiple dual-port adapters with different adapters connecting different completely independent rails within the private fabric, and multiple adapters connecting the cloud.

[0325] According to one embodiment, a determination can be made that any given packet should utilize a private fabric depending on the type of traffic flow within the network. Within the private fabric, two completely independent rails are provided, and each rail may be connected to a separate dual port adapter at the host node. In some embodiments, the private fabric offers higher and more secure performance than the cloud fabric, but generally has fewer resources, thus making flows within the private fabric more expensive.

[0326] According to one embodiment, the private fabric is fully controlled and redundant. On the other hand, due to the nature of the cloud fabric, it is generally considered not to be fully controlled, but can provide redundant connectivity. According to one embodiment, having at least two fabrics (private and public) provides an asymmetric scheme where two adapters in each host represent fully redundant connectivity to two rails in the private fabric, and a third adapter provides some (potentially redundant) connectivity in the public cloud. By having redundant connectivity to the cloud, this provides alternative options for traffic flows between hosts.

[0327] According to one embodiment, a workload (e.g., including multiple virtual machines implemented on two host nodes as shown in FIG. 20 and deployed and configured such that the VMs can communicate with each other) may be deployed within the environment shown in FIG. 20. For a given workload, a policy may be associated with the deployed workload (e.g., a "gold card" workload policy dictates that packets of such workload should utilize a high level private RDMA fabric, or, for example, an "economy" workload policy dictates that packets of such workload should utilize whatever bandwidth is available (e.g., private if available, public cloud otherwise)). Thus, these different workload policies may be taken into account in determining what type of connectivity should be used for any given workload in the system. For example, if a first workload includes a high "gold card" workload policy, packets in that workload may be allocated and provisioned strictly within the private fabric. Alternatively, if another workload includes a lower "economy" workload policy, packets associated with such workload may attempt to utilize the private fabric but may also utilize the public cloud for connectivity.

[0328] FIG. 21 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations according to one embodiment.

[0329] According to one embodiment, several host nodes, such as host node 1 2105 and host node 2 2110, may be supported within system 2100. Each of the host nodes may include several dual port adapters (e.g., dual port network adapters), such as dual port adapters 2106 and 2108 on host node 1, and dual port adapters 2111 and 2113 on host node 2. Each dual port adapter may provide two ports, by which the host nodes may be connected to one or more networks, such as a private fabric 2140 and a cloud fabric 2130, such as a public cloud. As shown, private fabric 2140 includes, for example, a set of overlapping switches, switch 1 2120 and switch 2 2121, where each switch represents a semi-independent rail within private fabric 2140.

[0330] According to one embodiment, each of the host nodes may include multiple dual port adapters. For example, in the illustrated embodiment, each host node supports two dual port adapters. Of these two dual port adapters, one of the dual port adapters on each host node may be connected to the private fabric 2140, and the other adapter may remain connected to the cloud fabric 2130. In this manner, each host node may be connected with two semi-independent rails (i.e., switch 1 and switch 2) via the private fabric and also via the cloud fabric.

[0331] FIG. 22 illustrates a system for supporting heterogeneous and asymmetric dual-rail fabric configurations according to one embodiment.

[0332] According to one embodiment, several host nodes may be supported within system 2200, such as host node 1 2205 and host node 2 2210. Each of the host nodes may include a number of dual port adapters (e.g., dual port network adapters), such as dual port adapter 2206 on host node 1 and dual port adapter 2211 on host node 2. Each dual port adapter may include a number of dual port adapters (e.g., dual port network adapters). An adapter may provide two ports by which a host node may be connected to one or more networks, such as a private fabric 2240 and a cloud fabric 2230, such as a public cloud. As shown, private fabric 2240 includes, for example, switch 2220, which represents a single rail in private fabric 2240.

[0333] According to one embodiment, each of the host nodes may include multiple dual port adapters. For example, in the illustrated embodiment, each host node supports one dual port adapter, where one port is connected to the private fabric and represents one rail, and the other port is connected through the cloud fabric and represents another rail.

[0334] 23 is a flow chart of a method for determining a placement policy for a workload that has been provisioned or is being provisioned in a system for supporting heterogeneous and asymmetric dual-rail fabric configurations. According to one embodiment, a workload may include a number of hosts, where the hosts may include a number of physical hosts and / or a number of virtual machines.

[0335] According to one embodiment, in step 2300, a provisioned workload (either a workload that has already been provisioned or a workload that is being provisioned) may be provided.

[0336] According to one embodiment, in step 2310, the workload may be examined to determine a placement policy associated with the provisioned workload.

[0337] According to one embodiment, in step 2315, the method may determine whether the workload's placement policy specifies strict connectivity within a single POD.

[0338] According to one embodiment, in step 2320, if the placement policy dictates strict placement within a single pod, then compute nodes within that single pod may be assigned to the provisioned workload.

[0339] According to one embodiment, in step 23250, if the placement policy does not dictate strict placement within a single POD, compute nodes may be assigned to workloads without POD considerations.

[0340] FIG. 24 is a flow chart of a method for determining a fabric type selection for a particular communication within a workload that has been provisioned or is being provisioned, according to one embodiment.

[0341] In step 2400, according to one embodiment, the method can determine whether any of a set of conditions are met, including: 1) is the workload prohibited from using the private fabric?; 2) does the peer (i.e., target) node belong to a different POD?; 3) is the peer node not currently available on or through the private fabric?; 4) is the current load on the private fabric higher than a set threshold and the load on the cloud fabric lower than another set threshold?

[0342] According to one embodiment, at 2410, the method may determine whether any of the conditions of step 2400 are true.

[0343] According to one embodiment, if one or more of the conditions of step 2400 are true, the method may, in step 2420, set up communication to the target node (peer node) via the cloud fabric.

[0344] According to one embodiment, if none of the conditions of step 2400 are true, the method may proceed to step 2430 to set up communication over the private fabric to the target node (peer node).

[0345] FIG. 25 is a flowchart of a method for supporting heterogeneous and asymmetric dual-rail fabric configurations according to one embodiment.

[0346] In step 2510, the method may provide, in a computer including one or more microprocessors, a plurality of hosts, each of the plurality of hosts including at least one dual port adapter, and further provide a private fabric including two or more switches, and a public fabric including a cloud fabric.

[0347] In step 2520, the method may provision a workload on a host of the plurality of hosts.

[0348] In step 2530, the method may assign placement policies to the provisioned workloads.

[0349] In step 2540, the method may allocate network traffic between peer nodes of the provisioned workload to one or more of the private and public fabrics according to a placement policy.

[0350] Thus, from one perspective, a system and method for supporting heterogeneous and asymmetric dual-rail fabric configurations in a high-performance computing environment has been described. A method may provide for a system including a plurality of hosts in one or more computers, each including one or more microprocessors, each of the plurality of hosts including at least one dual-port adapter, and further including a private fabric including two or more switches, and a public fabric including a cloud fabric. A workload may be provisioned in a host of the plurality of hosts. A placement policy may be assigned to the provisioned workload. Network traffic between peer nodes of the provisioned workload may then be assigned to one or more of the private fabric and the public fabric according to the placement policy.

[0351] Multicast transmission replication According to one embodiment, in telecommunication systems, the issue of high availability has been a key issue for decades, and the concept of a "carrier grade system" suggests a system with very low failover time before service is restored. One of the techniques used in such systems is to replicate all network traffic on dual independent networks, based on protocol extensions that allow the receiver to filter the incoming traffic so that only one logical packet is forwarded up the network stack even if two copies are received. In the context of very high bandwidth lossless interconnection fabrics, such replication is not practical in the general case due to excessive usage of both network bandwidth in the fabric and IO bandwidth in the servers involved. However, by selectively using such techniques for multicast traffic, explicit multicast between independent subnets can be achieved. It is conceivable that this could significantly reduce the need for multicast replication or encountering timeout situations due to individual multicast messages being lost in the network. Importantly, this can be accomplished while preserving the important semantics that a single multicast message is received "at most once" by each potential receiver.

[0352] According to one embodiment, the sender of a multicast message classifies the multicast sending behavior based on whether the multicast message represents an idempotent behavior that would have no negative effects if a receiving node were to process more than a single copy of the multicast message being sent. In this case, two or more independent multicast messages may be sent over one or more local interfaces and target one or more individual multicast addresses. However, if the message does not represent an idempotent behavior, multiple copies of the same message can only be sent as long as the message is encapsulated in a network packet with an encapsulation header that uniquely identifies the particular logical message instance. The identifier in the encapsulation header represents a monotonically increasing number that is unique to the sender, so that the receiver can be assured that subsequently received messages with the same identification number will not be forwarded to higher levels of the protocol stack and / or any receiving applications.

[0353] According to one embodiment, an aspect of this scheme is that in the case of an application level protocol with multicast-based requests and unicast-based responses, the same encapsulation scheme can be used for unicast responses as well, to allow the responder to send multiple copies in parallel without the risk of confusing the requester with multiple copies. Alternatively, the responder can only send a single regular unicast response, in which case it is up to the requester to send a new multicast request with a new instance number in the encapsulation header. In this case, it is up to the client / application specific protocol to take care of duplicate messages due to retries at the client / application layer.

[0354] According to one embodiment, for multicast traffic that is not classified as either idempotent or not, the default policy may be to use an encapsulation scheme (i.e., the default logical network interface provides this functionality by default). In this case, different logical interfaces may be provided for clients / applications that are aware of the alternatives and can perform the correct classification.

[0355] According to one embodiment, alternatively, the default interface may provide legacy functionality, and encapsulated traffic would then be provided only to clients / applications that explicitly request this functionality via a special logical interface.

[0356] Example of implementation: According to one embodiment, a sending node may have a persistent "system image generation number" that is incremented and persistently stored each time the associated node boots. In addition, each sending node will have a "current message sequence number" that is incremented for every message sent over a logical interface, representing an encapsulation and duplication. That is, the current message sequence number is the same for each duplicated message in a set of duplicated messages. The message identification in the encapsulation header is then a combination of the "system image generation number" and the "current message sequence number." Then, when a receiving node receives at least two copies of the same message (both being duplicates), it can ignore the later arriving message and discard it.

[0357] According to one embodiment, in one embodiment, only one sequence number is generated for a replicated message, regardless of how many replicates are used. However, it is possible to extend this by leaving some bits (e.g., only one bit) to encode the rail number on which the message was sent. This can have some value in evaluating traffic and also facilitates more information on the potential use of the inter-rail link between the two rails. However, to prevent multiple versions being delivered to a receiving application, the receiving side needs to recognize two (or more) replicated versions of the same message as replicates.

[0358] According to one embodiment, a sending node may duplicate a multicast message and transmit each copy of the multicast message on each of two or more independent or semi-independent rails to which the sending node has access.

[0359] According to one embodiment, if the persistent "System Image Generation Number" is lost for the associated sender node, a special procedure is performed to ensure that all possible receivers are notified and can reset any expected current identification number for the associated sender node. In the context of a high-availability cluster configuration, handling of the "System Image Generation Number" may typically be part of the procedure for including a node in a logical cluster. Thus, unless a node is included in the cluster, other nodes will not accept any incoming received multicast packets from that node, regardless of which "System Image Generation Number" it is associated with. The cluster membership processing protocol may decide to remove a node from the cluster for various reasons, especially if the node can no longer be communicated with (e.g., because it has crashed for some reason or is rebooting). In this case, the cluster membership decision may be communicated to the remaining cluster members, suggesting that communication with the removed node, including multicast communication, be immediately stopped. Each time an excluded node reboots, or attempts to (re)join the cluster for any reason, it can negotiate its "system image generation number" as part of the associated cluster membership joining protocol. In this way, it can present its current next generation number based on having retrieved it from persistent storage, or it can obtain a new generation number from the cluster membership control system if a new generation number is not readily available locally to it. In either case, the cluster membership control system can ensure that conflicting generation numbers cannot be used.

[0360] According to one embodiment, each receiver can maintain dynamic state information for each sender, where the identity of the sending node is maintained along with the currently expected message identifier. An arriving message with an encapsulation header representing a higher number will be accepted and forwarded, while a message with an identifier already received (i.e., a value lower than the currently expected value) will be discarded.

[0361] According to one embodiment, the protocol may use an alternative protocol identifier at the basic packet layer (e.g., using a special packet / protocol type field at the data link layer) to more easily separate packets representing encapsulated and duplicated messages from standard network traffic. Alternatively, a set of dedicated multicast addresses may be allocated for the purpose of implementing this duplication protocol, and then the original multicast addresses may be forwarded to the data link layer. The multicast address can be conveyed for inclusion in the encapsulation header.

[0362] 26 illustrates a system for multicast transmission replication instead of replication in a high performance computing environment, according to one embodiment. Notably, the figure illustrates such an implementation on one rail, with the other rail not shown.

[0363] According to one embodiment, the system 2600 may include several switches, such as spine switches 2611, 2621, 2631, and 2641, and leaf switches 2612-13, 2622-23, 2632-33, and 2642-43, which may interconnect a number of nodes 2650-2653.

[0364] According to one embodiment, the sending node 2650 may include a persistent "system image generation number" 2654 that is incremented and persistently stored each time the node boots. Additionally, the sending node 2650 may include a "current message sequence number" 2655 that is incremented for each set of multicast messages sent over the logical interface, representing encapsulation and duplication. Multicast packets 2655 sent and addressed to a multicast address (e.g., MGID) may include an encapsulation header that represents a combination of the "system image generation number" and the "current message sequence number."

[0365] According to one embodiment, each receiver may maintain dynamic state information about the sending node 2650, where the sending node's identity is maintained along with the currently expected message identifier. An arriving message with an encapsulation header representing a higher number will be accepted and forwarded (since it represents a new message), while a message with an identifier already received or a value lower than the expected message identifier will be discarded since the encapsulation header represents a message already received.

[0366] FIG. 27 is a flowchart of a method for multicast transmission replication instead of replication in a high performance computing environment.

[0367] According to one embodiment, in step 2710, a method can provide a plurality of switches and a plurality of hosts, the plurality of hosts interconnected via a plurality of switches, a host among the plurality of hosts including a multicast sender node, the sender node including a system image generation module and a current message sequence module.

[0368] According to one embodiment, in step 2720, the method can include organizing multiple switches into two rails, where the two or more rails provide redundant connectivity between multiple hosts.

[0369] According to one embodiment, in step 2730, a method can include sending, by a multicast sender node, two duplicate multicast packets addressed to a multicast address, where each of the two or more duplicate multicast packets is sent on a different one of the two rails. A receiving node can receive two versions of the same multicast packet, but deliver only one to a communications stack / client above the layer that handles the encapsulation header.

[0370] According to one embodiment, in step 2740, the method further comprises, when a host of the plurality of hosts receives two or more of the two or more duplicate multicast packets, replicating a first of the two or more received multicast packets before delivering them to the communications stack. You can drop everything except the

[0371] A single logical IP subnet across multiple independent Layer 2 (L2) subnets According to one embodiment, the system and method can provide a virtual single IP subnet across physically separate L2 subnets by allowing multiple end ports to simultaneously represent the same IP address.

[0372] According to one embodiment, the highly available network interface model is based on link aggregation (LAG). In this model, the host stack sees a single logical port, and the underlying driver software as well as adapter and switch firmware ensure that different flows are load-balanced among the available links, and flows currently mapped to a failed link are taken over on the remaining operational links. By introducing switches that support multi-chassis LAG (MLAG), this HA model can be applied across multiple switches, implying no single point of failure at all. However, this model also implies tight integration between redundant switches / networks, thus violating the goal of independent networks. An alternative model is to not use the LAG protocol at all, and instead allow the host-side interface to realize a single IP address over multiple local network ports. In this way, outgoing traffic can use any port in the associated logical group, and incoming traffic for a particular destination IP address will be accepted on any port in the group. If the networks involved are completely independent, the host-side operation may have to choose a different source port (aka different rails) to retry a message.

[0373] According to one embodiment, the interface model presented to the generic host stack may still be the same as in the LAG / MLAG case (ie, there is a single local network port with a single source IP address).

[0374] According to one embodiment, a single local IP (L3) address is then associated with the hardware address of each local end port.

[0375] According to one embodiment, the interface driver can track whether each local end port is connected to the same or separate L2 subnetworks and handle both outgoing and incoming ARP (Address Resolution Protocol) requests accordingly. For outgoing ARP requests, the interface driver can send one multicast request through each local port. For incoming ARP requests, the interface driver will send a single unicast ARP response through the local port on which the request arrived. The hardware address of the ARP response sent out on the network will always be the hardware address of the corresponding local end port.

[0376] According to one embodiment, the hardware address provided to the local host stack along with the remote IP address is a special value that the local interface driver can use to look up the "real" HW address used for the remote IP address via one or more of the local end ports.

[0377] According to one embodiment, the local end ports and remote hardware addresses used for the remote IP addresses are dynamic entities that the local interface driver can dynamically adjust over time to achieve both optimal availability and optimal load balancing.

[0378] FIG. 28 illustrates a system for supporting link aggregation (LAG) according to one embodiment.

[0379] According to one embodiment, a network packet, such as packet 1 2801, may be destined for a logical device, such as logical device 1 2810 (e.g., a link aggregation device). Before passing through the logical device, the packet may include or be associated with (e.g., in the packet header) a local source IP address (SIP1), a destination IP address (DIP1), and a destination MAC (media access control) address (DMAC1).

[0380] According to one embodiment, a logical device may be associated with its own address, such as an IP address, IP1, which may include a layer 3 address, for example.

[0381] According to one embodiment, the logical device may further include one or more physical network adapter devices (not shown), each of which may have one or more physical ports (AP1, AP2...APn), each of which may have its own MAC (L2) address (not shown), but may have a single IP address (IP1) of the logical device 2810.

[0382] According to one embodiment, packet 2 2802, after passing through the logical device, may contain, for example in the header, a local source IP address (SIP2) and a destination IP address (DIP2) in addition to the destination MAC address (DMAC2). In an embodiment, in a LAG situation, these addresses may be the same as the packet before passing through the logical device, i.e., SIP1 is the same as SIP2, DIP1 is the same as DIP2, and DMAC1 is the same as DMAC2.

[0383] According to one embodiment, the logic device determines which port / rail on Switch1 2820 (eg, SP1-1 through SP1-6) to send Packet2 to its destination port based on, for example, load balancing.

[0384] FIG. 29 illustrates a system for supporting multi-chassis link aggregation (MLAG) according to one embodiment.

[0385] According to one embodiment, a network packet, such as packet 1 2901, may be destined for a logical device, such as logical device 1 2910 (e.g., a link aggregation device). Before passing through the logical device, the packet may include or be associated with (e.g., in the packet header) a local source IP address (SIP1), a destination IP address (DIP1), and a destination MAC (media access control) address (DMAC1).

[0386] According to one embodiment, a logical device may be associated with its own address, such as an IP address, IP1, which may include a layer 3 address.

[0387] According to one embodiment, the logical device may further include one or more physical network adapter devices (not shown), each having one or more physical ports (AP1, AP2...APn) each having its own MAC (L2) address (not shown) but sharing a single IP address (IP1) of the logical device 2910.

[0388] According to one embodiment, packet 2 2902, after passing through the logical device, may include, for example in the header, a local source IP address (SIP2) and a destination IP address (DIP2) in addition to the destination MAC address (DMAC2). In an embodiment, in a LAG situation, these addresses may be the same as the packet before passing through the logical device, i.e., SIP1 is the same as SIP2, DIP1 is the same as DIP2, and DMAC1 is the same as DMAC2.

[0389] According to one embodiment, the logic device determines, for example, based on load balancing, the rail / port on which to send packet 2 to its destination port among the ports on switch 1 2920 (e.g., SP1-1 to SP1-6) or the ports on switch 2 2930 (e.g., SP2-1 to SP2-6).

[0390] According to one embodiment, in an MLAG scenario, the switches, switch 1 and switch 2, may further include one or more control links between the switches, such that the multiple switches appear as a single entity, yet provide greater redundancy than the LAG embodiment in which only one switch is provided.

[0391] FIG. 30 illustrates a system for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, according to one embodiment.

[0392] According to one embodiment, a network packet, such as packet 1 3001, may be destined for a logical device, such as logical device 1 3010 (e.g., a link aggregation device). Before passing through the logical device, the packet may include or be associated with (e.g., in the packet header) a local source IP address (SIP1), a destination IP address (DIP1), and a destination MAC (media access control) address (DMAC1).

[0393] According to one embodiment, a logical device may be associated with its own address, such as an IP address, IP1, which may include a layer 3 address, for example.

[0394] According to one embodiment, the logical device may further include one or more physical network adapter devices (not shown), each having one or more physical ports (AP1, AP2...APn), each of which may have their own MAC (L2) address (not shown), and each of which may have their own network-specific IP address (not shown) and rail number association (not shown).

[0395] According to one embodiment, packet 2 3002, after passing through a logical device, may include, for example, in the header, a local source IP address (SIP2) and a destination IP address (DIP2) in addition to a destination MAC address (DMAC2).

[0396] According to one embodiment, based on the mapping function 3011, the destination IP (DIP2) and destination MAC address (DMAC2) may contain new addresses based on a lookup of the input destination IP (DIP1) and input destination MAC address (DMAC1) in the mapping function. The source IP address may remain the same, i.e., SIP1 may be the same as SIP2.

[0397] According to one embodiment, based on the mapping function, the IP addresses of switches 1 and 3 are then mapped from a single Layer 3 address (IP1 of the logical device) to what can be seen as an MLAG scenario. The packet can be routed to multiple layer 2 subnets, such as those represented by Switch 3 3020 and Switch 2 3030, as well as Switch 3 3040, which can be viewed as a LAG scenario.

[0398] According to one embodiment, in an MLAG scenario, the switches, switch 1 and switch 2, may further include one or more control links between the switches, such that the multiple switches appear as a single entity, yet provide greater redundancy than the LAG embodiment in which only one switch is provided.

[0399] According to one embodiment, the system shown shows a single logical device having an L3 address supporting two L2 subnets, one via MLAG and one via LAG, however, one skilled in the art will readily appreciate a variety of different scenarios in which a single logical device supports multiple L2 subnets, such as n MLAG subnets and m LAG subnets, where the sum of n and m is greater than two.

[0400] According to one embodiment, the logic device determines, for example, based on load balancing, the rail / port on which to send packet 2 to its destination port among the ports on switch 1 3020 (e.g., SP1-1 to SP1-6) or the ports on switch 2 2300 (e.g., SP2-1 to SP2-6).

[0401] According to one embodiment, a system such as that described in Figure 30 can support an outbound mapping function. In such a situation, the system and method can look up the destination description based on the logical destination MAC. The system and method can then obtain the current physical device and port and the remote physical IP and physical MAC from the destination description. The packet header can then be updated with the local and remote physical addresses before the packet is sent.

[0402] According to one embodiment, a system such as that described in Figure 30 can support an incoming mapping function. In such a situation, the method and system can look up a logical device description associated with the physical device and port on which the incoming packet is received. The method and system can then verify that the correct physical destination address for the receiving physical port is included in the packet header. The packet header can then be updated to reflect the logical IPP of the local device before the packet is forwarded to the generic IP level in the host stack.

[0403] According to one embodiment, a system such as that described in FIG. 30 can support outgoing ARP (Address Resolution Protocol) requests. In such a situation, the system and method can generate an ARP request multicast packet for each physical port associated with the logical device. Then, the system and method can fill in the associated source address of the corresponding physical port in each ARP request multicast packet. The system and method can then transmit each ARP request multicast packet from the corresponding physical port.

[0404] According to one embodiment, a system such as that described in Figure 30 can support incoming ARP (Address Resolution Protocol) requests. The system and method can verify that the requested IP of an incoming ARP request on a physical port corresponds to the logical IP of the logical device with which the physical port is associated. The system and method can then send an ARP response on the physical port on which the ARP request was received, where the ARP response reflects the logical device IP address as the sender IP address, and the sender hardware address includes both the physical port MAC address and the physical port IP address.

[0405] According to one embodiment, a system such as that described in FIG. 30 can support incoming ARP (Address Resolution Protocol) responses. The system and method can verify that the target IP and HW addresses correspond to the receiving physical port physical IP and physical MAC addresses. The system and method can record the physical IP and MAC addresses from the sender hardware address of the ARP response and associate them with the destination logical IP address identified by the sender IP address in the ARP response. The system and method can then generate a logical MAC address to represent the mapping of the logical destination IP address to the associated physical port and the destination physical IP and MAC addresses. The ARP table can be updated with the logical IP address and the associated logical MAC address.

[0406] FIG. 31 is a flowchart of a method for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, according to one embodiment.

[0407] In step 3110, the method may provide, in a computer including one or more microprocessors, a logical device, the logical device being addressed by a layer 3 address, the logical device including a plurality of network adapters, each of the network adapters including a physical port, and the method may further provide a plurality of switches.

[0408] In step 3120, the method can arrange the switches into multiple separate layer 2 subnets.

[0409] In step 3130, the method may provide a mapping table in the logical device.

[0410] Thus, from one aspect, a system and method for supporting a single logical IP subnet across multiple independent layer 2 subnets in a high performance computing environment has been described. A method can provide a logical device in a computer including one or more microprocessors, the logical device being addressed by a layer 3 address, the logical device including a plurality of network adapters, each of the network adapters including a physical port, and the method can further provide a plurality of switches. The method can arrange the plurality of switches into a plurality of separate layer 2 subnets. The method can provide a mapping table in the logical device.

[0411] While various embodiments of the present teachings have been described, it should be understood that the above embodiments are presented by way of example and not limitation. The above embodiments have been selected and described to explain the principles of the claimed invention and its practical application. The above embodiments illustrate systems and methods that can utilize the present technology to improve the performance of the systems and methods by providing new and / or improved features and / or by providing advantages such as reduced resource utilization, increased capacity, improved efficiency and reduced latency.

[0412] In some embodiments, features of the present teachings are implemented, in whole or in part, in a computer that includes a processor, a storage medium such as memory, and a network card for communicating with other computers. In some embodiments, features of the present teachings are implemented in a computer system in which one or more clusters of computers are connected to a local area network (LON). A network such as a LAN (Local Area Network), a switched fabric network (e.g., InfiniBand), or a Wide Area Network (WAN) can provide The present invention is implemented in a distributed computing environment connected by a WAN. A distributed computing environment may have all the computers in one location or may have a cluster of computers in various remote geographic locations connected by a WAN.

[0413] In some embodiments, features of the present teachings are implemented in the cloud as part of a cloud computing system or as a service, based in whole or in part on shared and elastic resources delivered to users in a self-service coordinated manner using web technologies. There are five characteristics of the cloud (as defined by the National Institute of Standards and Technology): on-demand self-service, wide area network access, resource pooling, fast elasticity, and measured services. Cloud deployment models include public, private, and hybrid. Cloud service models include Software as a Service (SaaS), Platform as a Service (PaaS), Database as a Service (DBaaS), and Infrastructure as a Service (IaaS). As used herein, cloud is a combination of hardware, software, network and web technologies that delivers shared, elastic resources to users in a self-service, coordinated manner. Unless otherwise specified, cloud as used herein encompasses public cloud, private cloud and hybrid cloud embodiments, all cloud deployment models including, but not limited to, cloud SaaS, cloud DBaaS, cloud PaaS and cloud IaaS.

[0414] In some embodiments, features of the present teachings are implemented using or with the aid of hardware, software, firmware, or a combination thereof. In some embodiments, features of the present teachings are implemented using a processor configured or programmed to perform one or more functions of the subject matter taught herein. The processor, in some embodiments, may be a single or multi-chip processor, a digital signal processor (DSP), a system on a chip (SOC), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a memory controller, ... Field programmable gate array (FPGA) or other programmable A logic device, a state machine, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. In some implementations, features of the present teachings may be implemented by circuitry specialized for a particular function. In other implementations, the features may be implemented in a processor configured to perform a particular function using instructions stored on a computer-readable storage medium, for example.

[0415] In some embodiments, features of the present teachings are incorporated into software and / or firmware to control the hardware of a processing system and / or networking system, and to enable the processor and / or network to interact with other systems that utilize features of the subject matter taught herein. Such software or firmware may include, but is not limited to, application code, device drivers, operating systems, virtual machines, hypervisors, application programming interfaces, programming languages, and execution environments / containers. Appropriate software coding can be accomplished by those skilled in the art of software. As will be apparent to those skilled in the art, a skilled programmer can readily prepare such an embodiment based on the teachings of the present disclosure.

[0416] In some embodiments, the present teachings include a computer program product, which is a storage medium or computer readable medium having instructions stored thereon. These instructions can be used to program or otherwise configure a system, such as a computer, to perform any of the processes or functions of the present invention. The storage medium or computer readable medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), and any type of medium or device suitable for storing instructions and / or data. In certain embodiments, the storage medium is a non-transitory storage medium or a non-transitory computer readable medium. In some embodiments, the computer readable medium includes a transitory medium, such as a transmission signal or carrier wave. Such transitory media may occur as part of a transmission between components of a single computer system and / or may occur as part of a transmission between multiple computer systems.

[0417] The above description is not intended to be exhaustive or to limit the scope of the invention to the precise form disclosed. Also, while the embodiments of the present teachings have been described using a particular sequence of transactions and steps, it will be apparent to those skilled in the art that they are not limited to the sequence of transactions and steps described above. Furthermore, while the embodiments of the present teachings have been described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the present teachings. Furthermore, although specific combinations of features of the present teachings have been described in various embodiments, it should be understood that different combinations of these features are within the scope of the present teachings, such as features of one embodiment that can be incorporated into another embodiment. Furthermore, it will be apparent to those skilled in the art that various additions, omissions, deletions, modifications and other changes in form, details, implementation and application can be made without departing from the spirit and scope of the claimed subject matter. It is intended that the broader spirit and scope of the present invention be defined by the appended claims and their equivalents.

[0418] Aspects of the present disclosure are described in the following numbered clauses: 1. A system for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, comprising: a computer including one or more microprocessors; a logical device, the logical device being addressed by a layer 3 address, the logical device including a plurality of network adapters, the system further comprising: a plurality of switches, each of the plurality of switches including a plurality of ports; The switches are arranged in separate layer 2 subnets; A mapping table is provided in the logical device; A packet is received at the logical device, the packet including a source address and multiple destination addresses; The logic device updates at least one of the plurality of destination addresses; The logic device forwards the received packet, including the updated destination address, to one of the plurality of distinct layer 2 subnets.

[0419] 2. A system as claimed in clause 1, The source address of the packet is a Layer 2 address. The destination addresses of the received packet are each a layer 2 address.

[0420] 3. A system according to clause 2, each of the plurality of network adapters includes an adapter port of a plurality of adapter ports; Each of the multiple adapter ports is addressed via a unique layer 2 address.

[0421] 4. The system of clause 3, wherein each of the physical ports on the logical device is further addressed by the layer 3 address of the logical device.

[0422] 5. A system according to clause 4, Two or more switches of the plurality of switches are arranged in a first Layer 2 subnetwork that includes a multi-chassis link aggregation scheme.

[0423] 6. A system according to clause 5, Another switch of the plurality of switches is disposed in a second Layer 2 subnetwork that includes a link aggregation scheme.

[0424] 7. A system according to clause 6, A packet is received at the logical device, the packet including a source Layer 2 address and a destination Layer 2 address; After receiving the packet, the logic device updates the destination Layer 2 address of the received packet based on the mapping table; The logical device forwards the updated packet on a physical port of the logical device; The updated destination layer 2 address is associated with one of the first or second layer 2 subnets.

[0425] 8. A method for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, comprising: Providing a computer including one or more microprocessors; providing a logical device, the logical device being addressed by a Layer 3 address, the logical device including a plurality of network adapters, the method further comprising: providing a plurality of switches, each of the plurality of switches including a plurality of ports, the method further comprising: arranging said plurality of switches into a plurality of separate layer 2 subnets; providing a mapping table in the logical device; receiving a packet at the logic device, the packet including a source address and a plurality of destination addresses, the method further comprising: updating at least one of the plurality of destination addresses by the logic device; The logic device forwards the received packet, including the updated destination address, to one of the plurality of distinct layer 2 subnets.

[0426] 9. The method according to clause 8, The source address of the packet is a Layer 2 address. The destination addresses of the received packet are each a layer 2 address.

[0427] 10. The method according to clause 9, each of the plurality of network adapters includes an adapter port of a plurality of adapter ports; Each of the multiple adapter ports is addressed via a unique layer 2 address.

[0428] 11. The method of clause 10, wherein each of the physical ports on the logical device is further addressed by the layer 3 address of the logical device.

[0429] 12. A method according to clause 11, Two or more switches of the plurality of switches are arranged in a first Layer 2 subnetwork that includes a multi-chassis link aggregation scheme.

[0430] 13. The method of clause 12, Another switch of the plurality of switches is disposed in a second Layer 2 subnetwork that includes a link aggregation scheme.

[0431] 14. A method according to clause 13, The logic device updates the destination Layer 2 address of the received packet based on the mapping table; The logical device forwards the updated packet on a physical port of the logical device; The updated destination layer 2 address is associated with one of the first or second layer 2 subnets.

[0432] 15. A computer-readable medium carrying instructions for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, the instructions, when read and executed by a computer, causing the computer to perform any of the steps of clauses 8 to 14.

Claims

1. 1. A system for supporting a single logical IP subnet across multiple independent layer 2 subnets in a high performance computing environment, comprising: a computer including one or more microprocessors; a logical device, the logical device being addressed by a layer 3 address, the logical device including a plurality of network adapters, the system further comprising: a plurality of switches, each of the plurality of switches including a plurality of ports; the plurality of switches are arranged in a plurality of distinct layer 2 subnets; A mapping table is provided in the logic device; A packet is received at the logical device, the packet including a source address and multiple destination addresses; The logic device updates at least one of the plurality of destination addresses; The logic device forwards the received packet, including the updated destination address, to one of the plurality of distinct layer 2 subnets.

2. the source address of the packet is a Layer 2 address; 2. The system of claim 1, wherein the destination addresses of the received packet are each a Layer 2 address.

3. each of the plurality of network adapters includes an adapter port of a plurality of adapter ports; 3. The system of claim 2, wherein each of the plurality of adapter ports is addressed via a unique Layer 2 address.

4. 4. The system of claim 3, wherein each of the physical ports on the logical device is further addressed by the layer 3 address of the logical device.

5. The system of claim 4 , wherein two or more of the plurality of switches are disposed in a first Layer 2 subnetwork that includes a multi-chassis link aggregation scheme.

6. The system of claim 5 , wherein another switch of the plurality of switches is disposed in a second Layer 2 subnetwork that includes a link aggregation scheme.

7. A packet is received at the logic device, the packet including a source Layer 2 address and a destination Layer 2 address; After receiving the packet, the logic device updates the destination Layer 2 address of the received packet based on the mapping table; The logical device forwards the updated packet on a physical port of the logical device; The system of claim 6 , wherein the updated destination Layer 2 address is associated with one of the first or second Layer 2 subnets.

8. 1. A method for supporting a single logical IP subnet across multiple independent Layer 2 subnets in a high performance computing environment, comprising: Providing a computer including one or more microprocessors; providing a logical device, the logical device being addressed by a layer 3 address, the logical device including a plurality of network adapters, the method further comprising: providing a plurality of switches, each of the plurality of switches including a plurality of ports; the method further comprising: arranging the plurality of switches into a plurality of separate layer 2 subnets; providing a mapping table in the logical device; receiving a packet at the logic device, the packet including a source address and a plurality of destination addresses, the method further comprising: the logic device updating at least one of the plurality of destination addresses; the logic device forwarding the received packet including the updated destination address to one of the plurality of distinct layer 2 subnets.

9. the source address of the packet is a Layer 2 address; 9. The method of claim 8, wherein the destination addresses of the received packet are each a Layer 2 address.

10. each of the plurality of network adapters includes an adapter port of a plurality of adapter ports; 10. The method of claim 9, wherein each of the plurality of adapter ports is addressed via a unique layer 2 address.

11. 11. The method of claim 10, wherein each of the physical ports on the logical device is further addressed by the layer 3 address of the logical device.

12. The method of claim 11 , wherein two or more switches of the plurality of switches are disposed in a first Layer 2 subnetwork that includes a multi-chassis link aggregation scheme.

13. The method of claim 12 , wherein another switch of the plurality of switches is disposed in a second Layer 2 subnetwork that includes a link aggregation scheme.

14. The logic device updates the destination Layer 2 address of the received packet based on the mapping table; The logical device forwards the updated packet on a physical port of the logical device; The method of claim 13 , wherein the updated destination Layer 2 address is associated with one of the first or second Layer 2 subnets.

15. A computer-readable medium carrying instructions for supporting a single logical IP subnet across multiple independent layer 2 subnets in a high performance computing environment, the instructions, when read and executed by a computer, causing the computer to perform any of the steps of claims 8 to 14.

Images