System and method for supporting RDMA bandwidth limitations in a proprietary architecture in a high performance computing environment
By implementing selective RDMA limiting and explicit bandwidth control on host channel adapters in a high-performance computing environment, the bandwidth congestion caused by RDMA read requests is solved, ensuring stable network performance and QOS assumptions among multiple tenants.
Patent Information
- Application Number
- CN202080080811.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-05-11
- Filing Date
- 2020-08-06
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-08-06
AI Technical Summary
In high-performance computing environments, it is difficult for the prior art to effectively manage and limit bandwidth congestion caused by RDMA read requests, affecting network performance, especially in the case of multi-tenant sharing of subnets.
By implementing selective RDMA limits in the host channel adapter, only trusted nodes are allowed to generate RDMA read requests and control inlet and exit bandwidths through explicit RDMA read bandwidth limits to prevent bandwidth overload.
It effectively limits the bandwidth overload caused by RDMA read requests, ensures stability of network performance, reduces interference between multiple tenants, and implements the QOS assumption of different communication types.
Smart Images

Figure CN114762304B_ABST
Abstract
Description
[0001] Copyright Notice
[0002] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.
[0003] Priority Statement and Cross-Reference to Related Applications:
[0004] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 937,594, filed on November 19, 2019, entitled “SYSTEM AND METHOD FOR PROVIDING QUALITY-OF-SERVICE AND SERVICE-LEVEL AGREEMENTS IN A PRIVATE FABRICIN A HIGH PERFORMANCE COMPUTING ENVIRONMENT,” which is incorporated herein by reference in its entirety.
[0005] This application also claims the benefit of priority to the following applications, each of which is incorporated by reference in its entirety: U.S. Patent Application No. 16 / 872,035, filed May 11, 2020, entitled “SYSTEM AND METHOD FOR SUPPORTING RDMA BANDWIDTHRESTRICTIONS IN A PRIVATE FABRIC IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT”; U.S. Patent Application No. 16 / 872,038, filed May 11, 2020, entitled “SYSTEM AND METHOD FOR PROVIDING BANDWIDTH CONGESTION CONTROL IN A PRIVATE FABRIC IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT”; U.S. Patent Application No. 16 / 872,038, filed May 11, 2020, entitled “SYSTEM AND METHOD FOR SUPPORTING Target Groups for Consumption Control in a PRIVATE FABRIC IN A HIGH and U.S. patent application Ser. No. 16 / 872,043, filed May 11, 2020, entitled “SYSTEM AND METHOD FOR SUPPORTING USE OF FORWARD AND BACKWARD CONCESTION NOTIFICATIONS IN A PRIVATE FABRIC IN A HIGH PERFORMANCE COMPUTING ENVIRONMENT.” Technical Field
[0006] The present teachings relate to systems and methods for enforcing Quality of Service (QOS) and Service Level Agreements (SLAs) in proprietary high-performance interconnect architectures such as InfiniBand (IB) and RoCE (RDMA (Remote Direct Memory Access) over Converged Ethernet). Background Art
[0007] With the introduction of larger cloud computing architectures, performance and management bottlenecks associated with traditional network and storage devices have become significant issues. There is increasing interest in using high-performance lossless interconnects such as InfiniBand (IB) technology as the foundation of cloud computing architectures. This is the general area that embodiments of the present invention are intended to address. Summary of the invention
[0008] Particular aspects are set out in the attached independent claims. Various alternative embodiments are set out in the dependent claims.
[0009] Systems and methods for providing RDMA (remote direct memory access) read requests as a restricted feature in a high performance computing environment are described herein. An example method may provide a first subnet at one or more microprocessors, the first subnet comprising: a plurality of switches, a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via a plurality of switches, and a plurality of end nodes, the plurality of end nodes comprising a plurality of virtual machines. The method may associate a host channel adapter with a selective RDMA restriction. The method may host a virtual machine in a plurality of virtual machines at a host channel adapter comprising a selective RDMA restriction. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 A diagram of an InfiniBand environment is shown in accordance with an embodiment.
[0011] Figure 2 A diagram of a partitioned cluster environment is shown in accordance with an embodiment.
[0012] Figure 3 A diagram showing a tree topology in a network environment according to an embodiment is shown.
[0013] Figure 4 An example shared port architecture is shown according to an embodiment.
[0014] Figure 5 An example vSwitch architecture is shown in accordance with an embodiment.
[0015] Figure 6 An example vPort architecture is shown in accordance with an embodiment.
[0016] Figure 7 An example vSwitch architecture with pre-populated LIDs is shown according to an embodiment.
[0017] Figure 8 An example vSwitch architecture with dynamic LID allocation is shown in accordance with an embodiment.
[0018] Fig. 9 An example vSwitch architecture with a vSwitch with dynamic LID allocation and pre-populated LIDs according to an embodiment is shown.
[0019] Fig.10 An example multi-subnet InfiniBand architecture is shown in accordance with an embodiment.
[0020] Fig.11 An interconnection between two subnets in a high performance computing environment is shown according to an embodiment.
[0021] Fig.12 An interconnection between two subnets via a dual-port virtual router configuration in a high performance computing environment is shown according to an embodiment.
[0022] Fig.13 A flow chart of a method for supporting a dual-port virtual router in a high performance computing environment according to an embodiment is shown.
[0023] Fig.14 A system for providing RDMA read requests as a restricted feature in a high performance computing environment according to an embodiment is shown.
[0024] Fig.15 A system for providing RDMA read requests as a restricted feature in a high performance computing environment according to an embodiment is shown.
[0025] Fig.16 A system for providing RDMA read requests as a restricted feature in a high performance computing environment according to an embodiment is shown.
[0026] Fig.17 A system for providing explicit RDMA read bandwidth limiting in a high performance computing environment according to an embodiment is shown.
[0027] Fig.18 A system for providing explicit RDMA read bandwidth limiting in a high performance computing environment according to an embodiment is shown.
[0028] Fig.19 A system for providing explicit RDMA read bandwidth limiting in a high performance computing environment according to an embodiment is shown.
[0029] Fig. 20 is a flow chart of a method for providing RDMA (Remote Direct Memory Access) read requests as a restricted feature in a high performance computing environment.
[0030] Fig.21 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0031] Fig. 22 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0032] Fig.23 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0033] Fig.24 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0034] Fig.25 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0035] Fig.26 is a flow chart of a method for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment.
[0036] Fig. 27 A system for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment is shown in accordance with an embodiment.
[0037] Fig.28 A system for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment is shown in accordance with an embodiment.
[0038] Fig.29 A system for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment is shown in accordance with an embodiment.
[0039] Fig.30 A system for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment is shown in accordance with an embodiment.
[0040] Fig.31 is a flow chart of a method for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment, according to an embodiment.
[0041] Fig.32 A system for combining ingress bandwidth arbitration and congestion feedback in a high performance computing environment according to an embodiment is shown.
[0042] Fig.33 A system for combining ingress bandwidth arbitration and congestion feedback in a high performance computing environment according to an embodiment is shown.
[0043] Fig.34A system for combining ingress bandwidth arbitration and congestion feedback in a high performance computing environment according to an embodiment is shown.
[0044] Fig.35 is a flow chart of a method of combining ingress bandwidth arbitration and congestion feedback in a high performance computing environment according to an embodiment.
[0045] Fig.36 A system for using multiple CE markings in both FECN and BECN in a high performance computing environment according to an embodiment is shown.
[0046] Fig.37 A system for using multiple CE markings in both FECN and BECN in a high performance computing environment according to an embodiment is shown.
[0047] Fig.38 A system for using multiple CE markings in both FECN and BECN in a high performance computing environment according to an embodiment is shown.
[0048] Fig.39 is a flow chart of a method for using multiple CE markings in both FECN and BECN in a high performance computing environment, according to an embodiment. DETAILED DESCRIPTION
[0049] The present teachings are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like reference numerals indicate similar elements. It should be noted that references to "one," "an," or "some" embodiments in the present disclosure do not necessarily refer to the same embodiments, and such references mean at least one. Although specific implementations are discussed, it should be understood that the specific implementations are provided for illustrative purposes only. Those skilled in the relevant art will recognize that other components and configurations may be used without departing from the spirit and scope.
[0050] Common reference numerals may be used throughout the Figures and the Detailed Description to refer to the same elements; thus, if the element is described elsewhere, a reference numeral used in a Figure may or may not be referenced in the detailed description specific to that Figure.
[0051] Described herein are systems and methods for providing quality of service (QOS) and service level agreements (SLAs) in a private architecture in a high performance computing environment.
[0052] According to an embodiment, the following description of the present teaching uses InfiniBand TM (IB) network as an example of a high performance network. Throughout the following description, reference may be made to InfiniBand TMSpecification (also variously referred to as the InfiniBand Specification, the IB Specification or the Legacy IB Specification). Such references are understood to be references to the InfiniBand IEEE 802.11bn specification published in March 2015, available at http: / / www.inifinibandta.org. Trade Association Architecture Specification( The entire contents of the IEEE 802.11 Trade Association Architecture Specification, Volume 1, Version 1.3, are incorporated herein by reference in their entirety. It will be apparent to one skilled in the art that other types of high performance networks may be used without limitation. The following description also uses a fat tree topology as an example of an architecture topology. It will be apparent to one skilled in the art that other types of architecture topologies may be used without limitation.
[0053] According to an embodiment, the following description uses RoCE (RDMA (Remote Direct Memory Access) over Converged Ethernet). RDMA over Converged Ethernet (RoCE) is a standard protocol that enables RDMA for efficient data transfer over Ethernet, allowing transport offload to be implemented with a hardware RDMA engine with excellent performance. RoCE is a standard protocol defined in the InfiniBand Trade Association (IBTA) standard. RoCE utilizes UDP (User Datagram Protocol) encapsulation, allowing it to go beyond Layer 3 networks. RDMA is a key capability used by the InfiniBand interconnect technology itself. Both InfiniBand and Ethernet RoCE share a common user API, but have different physical and link layers.
[0054] According to an embodiment, although various parts of the specification include references to the InfiniBand architecture when describing various implementations, a person of ordinary skill in the art will readily understand that the various embodiments described herein may also be implemented in a RoCE architecture.
[0055] To meet the needs of the cloud in the current era (e.g., the Exascale era), it is expected that virtual machines can take advantage of low-overhead network communication paradigms such as remote direct memory access (RDMA). RDMA bypasses the OS stack and communicates directly with the hardware, so pass-through technologies such as single root I / O virtualization (SR-IOV) network adapters can be used. According to an embodiment, a virtual switch (vSwitch) SR-IOV architecture can be provided for applicability in high-performance lossless interconnect networks. Since network reconfiguration time is critical to making live migration a practical option, a scalable and topology-independent dynamic reconfiguration mechanism can be provided in addition to the network architecture.
[0056] According to an embodiment, and in addition, a routing strategy for a virtualized environment using a vSwitch can be provided, and an efficient routing algorithm for a network topology (eg, a fat tree topology) can be provided. The dynamic reconfiguration mechanism can be further adjusted to minimize the overhead introduced in the fat tree.
[0057] According to embodiments of the present teachings, virtualization can be beneficial for efficient resource utilization and elastic resource allocation in cloud computing. Live migration makes it possible to optimize resource usage by moving virtual machines (VMs) between physical servers in an application-transparent manner. Therefore, virtualization can achieve integration, on-demand provisioning of resources, and elasticity through live migration.
[0058] InfiniBand TM
[0059] InfiniBand TM (IB) is InfiniBand TM An open standard lossless networking technology developed by a trade association. The technology is based on a serial point-to-point full-duplex interconnect that provides high throughput and low latency communications, specifically targeting high performance computing (HPC) applications and data centers.
[0060] InfiniBand TM The IBA supports two-tier topology partitioning. At the lower level, an IB network is called a subnet, where a subnet can include a collection of hosts interconnected using switches and point-to-point links. At the upper level, the IB architecture consists of one or more subnets that can be interconnected using routers.
[0061] Within a subnet, hosts may be connected using switches and point-to-point links. In addition, there may be a main management entity, the subnet manager (SM), which resides on a designated device in the subnet. The subnet manager is responsible for configuring, activating, and maintaining the IB subnet. Additionally, the subnet manager (SM) may be responsible for performing routing table calculations in the IB architecture. Here, for example, routing for an IB network is intended to properly load balance between all source and destination pairs in the local subnet.
[0062] Through the subnet management interface, the subnet manager exchanges control packets with the subnet management agent (SMA), which are called subnet management packets (SMP). The subnet management agent resides on each IB subnet device. By using the SMP, the subnet manager can discover the fabric, configure end nodes and switches, and receive notifications from the SMA.
[0063] According to an embodiment, intra-subnet routing in an IB network can be based on a linear forwarding table (LFT) stored in a switch. The LFT is calculated by the SM based on the routing mechanism in use. In the subnet, the host channel adapter (HCA) ports on the end nodes and switches are addressed using local identifiers (LIDs). Each entry in the linear forwarding table (LFT) includes a destination LID (DLID) and an output port. Only one entry per LID is supported in the table. When a packet arrives at the switch, its output port is determined by looking up the DLID in the switch's forwarding table. Routing is deterministic because packets take the same path in the network between a given source-destination pair (LID pair).
[0064] Generally speaking, all subnet managers except the primary subnet manager function in standby mode for fault tolerance. However, in case of failure of the primary subnet manager, the standby subnet manager negotiates the new primary subnet manager. The primary subnet manager also performs periodic scans of the subnet to detect any topology changes and reconfigure the network accordingly.
[0065] In addition, local identifiers (LIDs) can be used to address hosts and switches within a subnet, and a single subnet can be limited to 49,151 unicast LIDs. In addition to the LID, which is a local address valid within the subnet, each IB device can also have a 64-bit globally unique identifier (GUID). The GUID can be used to form a global identifier (GID) that is an IB layer three (L3) address.
[0066] At network initialization, the SM can calculate the routing table (i.e., the connections / routes between each pair of nodes within the subnet). In addition, the routing table can be updated whenever the topology changes to ensure connectivity and optimal performance. During normal operation, the SM can perform periodic light scans of the network to check for topology changes. If changes are found during the light scan, or if the SM receives information (traps) that signal a change in the network, the SM can reconfigure the network based on the changes found.
[0067] For example, when the network topology changes, such as when a link is disconnected, when a device is added, or when a link is removed, the SM can reconfigure the network. The reconfiguration steps can include steps performed during network initialization. In addition, the reconfiguration can have a local scope limited to the subnet in which the network change occurs. In addition, segmenting a large architecture with routers can limit the scope of the reconfiguration.
[0068] Figure 1 An example InfiniBand architecture is shown in , which shows a diagram of an InfiniBand environment 100 according to an embodiment. Figure 1In the example shown, nodes AE 101-105 communicate via corresponding host channel adapters 111-115 using InfiniBand architecture 120. According to an embodiment, various nodes (e.g., nodes AE 101-105) can be represented by various physical devices. According to an embodiment, various nodes (e.g., nodes AE 101-105) can be represented by various virtual devices such as virtual machines.
[0069] Partitioning in InfiniBand
[0070] According to an embodiment, an IB network may support partitioning as a security mechanism to provide isolation of logical groups of systems that share a network architecture. Each HCA port on a node in the architecture may be a member of one or more partitions. Partition membership is managed by a centralized partition manager, which may be part of the SM. The SM may configure the partition membership information on each port as a table of 16-bit partition keys (P_Keys). The SM may also configure switch and router ports with a partition implementation table containing P_Key information associated with end nodes that send or receive data traffic through these ports. In addition, in general, the partition membership of a switch port may represent the union of all memberships indirectly associated with LIDs that are routed via the port in the egress (towards the link) direction.
[0071] According to an embodiment, a partition is a logical group of ports so that members of the group can only communicate with other members of the same logical group. At the host channel adapter (HCA) and the switch, the partition membership information can be used to filter the packet to implement isolation. Once the packet arrives at the incoming port, the packet with invalid partition information can be discarded. In a partitioned IB system, partitions can be used to create tenant clusters. When the partition is implemented in place, the node cannot communicate with other nodes belonging to different tenant clusters. In this way, the security of the system can be guaranteed even when there are damaged or malicious tenant nodes.
[0072] According to an embodiment, for communication between nodes, in addition to managing queue pairs (QP0 and QP1), queue pairs (QP) and end-to-end contexts (EEC) can be assigned to specific partitions. Then, P_Key information can be added to each IB transmission packet sent. When a packet arrives at an HCA port or switch, its P_Key value can be verified against a table configured by the SM. If an invalid P_Key value is found, the packet is immediately discarded. In this way, communication is only allowed between ports of a shared partition.
[0073] Figure 2 An example of IB partitioning is shown in FIG, which shows a diagram of a partitioned cluster environment according to an embodiment. Figure 2In the example shown, nodes AE 101-105 communicate via corresponding host channel adapters 111-115 using InfiniBand architecture 120. Nodes AE are arranged into partitions, namely partition 1 130, partition 2 140, and partition 3 150. Partition 1 includes node A 101 and node D 104. Partition 2 includes node A 101, node B 102, and node C 103. Partition 3 includes node C 103 and node E 105. Due to the arrangement of the partitions, node D 104 and node E 105 are not allowed to communicate because these nodes do not share a partition. At the same time, for example, node A 101 and node C 103 are allowed to communicate because these nodes are members of partition 2 140.
[0074] Virtual Machines in InfiniBand
[0075] Over the past decade, the promise of virtualized high performance computing (HPC) environments has improved considerably, as CPU overhead has been virtually eliminated through hardware virtualization support; storage overhead has been significantly reduced by virtualizing the memory management unit; storage device overhead has been reduced by using fast SAN storage devices or distributed network file systems; and network I / O overhead has been reduced by using device pass-through technologies such as single root input / output virtualization (SR-IOV). Now, it is possible for clouds to accommodate virtual HPC (vHPC) clusters using high performance interconnect solutions and provide the necessary performance.
[0076] However, when coupled with lossless networks such as InfiniBand (IB), certain cloud functions, such as live migration of virtual machines (VMs), remain problematic due to the complex addressing and routing schemes used in these solutions. IB is an interconnect network technology that provides high bandwidth and low latency, making it well suited for HPC and other communication-intensive workloads.
[0077] The traditional method for connecting IB devices to VMs is by utilizing SR-IOV with direct assignment. However, implementing live migration of VMs assigned with IB host channel adapters (HCAs) using SR-IOV has proven to be challenging. Each IB-connected node has three different addresses: LID, GUID, and GID. When live migration occurs, one or more of these addresses change. Other nodes communicating with the migrating VM may lose connectivity. When this happens, an attempt to update the lost connection can be made by locating the new address of the virtual machine to be reconnected to by sending a subnet management (SA) path record query to the IB subnet manager (SM).
[0078] IB uses three different types of addresses. The first type of address is a 16-bit local identifier (LID). The SM assigns at least one unique LID to each HCA port and each switch. LID is used to route traffic within a subnet. Since the LID is 16 bits long, 65536 unique address combinations can be made, of which only 49151 (0x0001-0xBFFF) can be used as unicast addresses. Therefore, the number of available unicast addresses limits the maximum size of the IB subnet. The second type of address is a 64-bit globally unique identifier (GUID) assigned by the manufacturer to each device (e.g., HCA and switch) and each HCA port. The SM can assign an additional subnet-unique GUID to the HCA port, which is useful when using SR-IOV. The third type of address is a 128-bit global identifier (GID). GID is a valid IPv6 unicast address, and at least one GID is assigned to each HCA port. GID is formed by combining a globally unique 64-bit prefix assigned by the architecture administrator and the GUID address of each HCA port.
[0079] Fat Tree (FTree) Topology and Routing
[0080] According to an embodiment, some of the IB-based HPC systems employ a fat tree topology to take advantage of the useful properties provided by fat trees. These properties include full bisection bandwidth and inherent fault tolerance since multiple paths are available between each source destination pair. The original idea behind the fat tree is to employ fatter links with more available bandwidth between nodes as the tree moves toward the root of the topology. Fatter links can help avoid congestion in upper layer switches and maintain bisection bandwidth.
[0081] Figure 3 FIG. 2 shows a diagram of a tree topology in a network environment according to an embodiment. Figure 3 As shown, one or more end nodes 201-204 may be connected in a network architecture 200. The network architecture 200 may be based on a fat tree topology including a plurality of leaf switches 211-214 and a plurality of spine switches or root switches 231-234. In addition, the network architecture 200 may include one or more intermediate switches, such as switches 221-224.
[0082] Likewise Figure 3 As shown, each of the end nodes 201-204 may be a multi-homed node, i.e., a single node connected through multiple ports to two or more portions of the network architecture 200. For example, node 201 may include ports H1 and H2, node 202 may include ports H3 and H4, node 203 may include ports H5 and H6, and node 204 may include ports H7 and H8.
[0083] In addition, each switch may have multiple switch ports. For example, root switch 231 may have switch ports 1-2, root switch 232 may have switch ports 3-4, root switch 233 may have switch ports 5-6, and root switch 234 may have switch ports 7-8.
[0084] According to an embodiment, the fat tree routing mechanism is one of the most popular routing algorithms for IB-based fat tree topology. The fat tree routing mechanism is also implemented in OFED (Open Architecture Enterprise Distribution - a standard software stack for building and deploying IB-based applications) subnet manager OpenSM.
[0085] The fat tree routing mechanism aims to generate LFTs that evenly distribute shortest path routes across links in the network fabric. The mechanism traverses the fabric in index order and assigns target LIDs of end nodes (and therefore corresponding routes) to each switch port. For end nodes connected to the same leaf switch, the index order may depend on the switch ports to which the end nodes are connected (i.e., the port numbering order). For each port, the mechanism may maintain a port usage counter and may use this port usage counter to select the least used port each time a new route is added.
[0086] According to an embodiment, in a partitioned subnet, nodes that are not members of a common partition are not allowed to communicate. In practice, this means that some routes assigned by the fat-tree routing algorithm are not used for user traffic. This problem occurs when the fat-tree routing mechanism generates LFTs for those routes in the same way as it does for other functional paths. Since nodes are routed in order of index, this behavior may result in a degradation of balance on the links. Since routing can be performed regardless of the partition, in general, fat-tree routed subnets provide poorer isolation between partitions.
[0087] According to an embodiment, a fat tree is a hierarchical network topology that can be scaled using available network resources. Moreover, a fat tree is easily constructed using commercial switches placed at different levels of the hierarchy. Different variants of fat trees are generally available, including k-fork-n-trees, extended generalized fat trees (XGFTs), parallel port generalized fat trees (PGFTs), and real life fat trees (RLFTs).
[0088] A k-by-n-tree is a tree with k n terminal nodes and n·k n-1An n-level fat tree of 100 switches, each with 2k ports. Each switch has the same number of upstream and downstream connections in the tree. The XGFT fat tree extends the k-by-n-tree by allowing switches to have different numbers of upstream and downstream connections and different numbers of connections at each level in the tree. The PGFT definition further broadens the XGFT topology and allows multiple connections between switches. A wide variety of topologies can be defined using XGFT and PGFT. However, for practical purposes, RLFT, a restricted version of PGFT, was introduced to define the fat trees commonly seen in today's HPC clusters. RLFT uses switches with the same port count at all levels of the fat tree.
[0089] Input / Output (I / O) Virtualization
[0090] According to an embodiment, I / O virtualization (IOV) can provide I / O availability by allowing virtual machines (VMs) to access underlying physical resources. The combination of storage traffic and inter-server communication creates an increased load that can overwhelm the I / O resources of a single server, resulting in backlogs and idle processors as processors wait for data. IOV can provide availability as the number of I / O requests increases; and can improve the performance, scalability, and flexibility of (virtualized) I / O resources to match the performance levels seen in modern CPU virtualization.
[0091] According to an embodiment, IOV is desirable because it can allow sharing of I / O resources and provide protected access to resources by VMs. IOV decouples the logical devices exposed to the VM from the physical implementation of the logical devices. Currently, there may be different types of IOV technologies, such as emulation, paravirtualization, direct assignment (DA), and single root I / O virtualization (SR-IOV).
[0092] According to an embodiment, one type of IOV technology is software emulation. Software emulation can allow a decoupled front-end / back-end software architecture. The front end can be a device driver placed in a VM that communicates with a back end implemented by a hypervisor to provide I / O access. The physical device sharing ratio is high, and the real-time migration of the VM may only require a few milliseconds of network downtime. However, software emulation introduces additional, undesirable computational overhead.
[0093] According to an embodiment, another type of IOV technology is direct device assignment. Direct device assignment involves coupling an I / O device to a VM, where there is no device sharing between VMs. Direct assignment or device pass-through provides near-native performance with minimal overhead. The physical device bypasses the hypervisor and is directly attached to the VM. However, a disadvantage of this direct device assignment is limited scalability because there is no sharing between virtual machines—one physical network card is coupled to one VM.
[0094] According to an embodiment, single root IOV (SR-IOV) can allow a physical device to appear as multiple independent lightweight instances of the same device through hardware virtualization. These instances can be assigned to VMs as pass-through devices and accessed as virtual functions (VFs). The hypervisor accesses the device through a unique, fully featured physical function (PF) (per device). SR-IOV makes the scalability problem of pure direct allocation easy. However, the problem presented by SR-IOV is that it may affect VM migration. Among these IOV technologies, SR-IOV can extend the PCI Express (PCIe) specification by allowing direct access to a single physical device from multiple VMs while maintaining near-local performance. Therefore, SR-IOV can provide good performance and scalability.
[0095] SR-IOV allows PCIe devices to expose multiple virtual devices that can be shared among multiple guests by assigning a virtual device to each guest. Each SR-IOV device has at least one physical function (PF) and one or more associated virtual functions (VFs). PF is a normal PCIe function controlled by a virtual machine monitor (VMM) or hypervisor, while VF is a lightweight PCIe function. Each VF has its own base address (BAR) and is assigned a unique requester ID, which enables the I / O memory management unit (IOMMU) to distinguish between business flows from / to different VFs. The IOMMU also applies memory and interrupt conversion between PFs and VFs.
[0096] Unfortunately, however, direct device allocation techniques create obstacles for cloud providers in situations where transparent live migration of virtual machines is desired for data center optimization. The essence of live migration is to copy the memory contents of the VM to a remote hypervisor. The VM is then suspended at the source hypervisor and the operation of the VM is resumed at the destination. When a software emulation approach is used, the network interface is virtualized, so its internal state is stored in memory and also copied. Thus, downtime can be reduced to a few milliseconds.
[0097] However, migration becomes more difficult when direct device assignment techniques such as SR-IOV are used. In this case, the complete internal state of the network interface cannot be copied because it is tied to the hardware. Instead, the SR-IOV VF assigned to the VM is detached, the live migration will be run, and the new VF will be attached at the destination. In the case of InfiniBand and SR-IOV, this process can introduce downtime on the order of seconds. Furthermore, in the SR-IOV shared port model, after the migration, the address of the VM will change, resulting in additional overhead in the SM and negatively affecting the performance of the underlying network architecture.
[0098] InfiniBand SR-IOV Architecture - Shared Ports
[0099] There may be different types of SR-IOV models, for example, a shared port model, a virtual switch model, and a virtual port model.
[0100] Figure 4 An example shared port architecture according to an embodiment is shown. As shown, a host 300 (e.g., a host channel adapter) can interact with a hypervisor 310, which can assign various virtual functions 330, 340, 350 to multiple virtual machines. Similarly, physical functions can be handled by the hypervisor 310.
[0101] According to an embodiment, when using a Figure 4 When using a shared port architecture such as the one shown, the host (e.g., HCA) appears to the network as a single port with a single shared LID and shared queue pair (QP) space between the physical function 320 and the virtual functions 330, 350, 350. However, each function (i.e., physical function and virtual function) can have its own GID.
[0102] like Figure 4 As shown, according to an embodiment, different GIDs may be assigned to virtual functions and physical functions, and a special queue pair QP0 and QP1 (i.e., a dedicated queue pair for InfiniBand management packets) is owned by the physical function. These QPs are also exposed to the VF, but the VF is not allowed to use QP0 (all SMPs from the VF toward QP0 are dropped), and QP1 may act as a proxy for the actual QP1 owned by the PF.
[0103] According to an embodiment, a shared port architecture may allow for a highly scalable data center that is not limited by the number of VMs attached to the network (by being assigned to virtual functions) because the LID space is consumed only by the physical machines and switches in the network.
[0104] However, the disadvantage of the shared port architecture is that it cannot provide transparent live migration, thus hindering the possibility of flexible VM placement. Since each LID is associated with a specific hypervisor and is shared among all VMs residing on the hypervisor, the migrated VM (i.e., the virtual machine migrated to the destination hypervisor) must change its LID to the LID of the destination hypervisor. In addition, due to the restricted QP0 access, the subnet manager cannot run inside the VM.
[0105] InfiniBand SR-IOV Architecture Model - Virtual Switch (vSwitch)
[0106] Figure 5An example vSwitch architecture according to an embodiment is shown. As shown, a host 400 (e.g., a host channel adapter) can interact with a hypervisor 410, which can assign various virtual functions 430, 440, 450 to multiple virtual machines. Similarly, physical functions can be handled by the hypervisor 410. A virtual switch 415 can also be handled by the hypervisor 401.
[0107] According to an embodiment, in the vSwitch architecture, each virtual function 430, 440, 450 is a full virtual host channel adapter (vHCA), which means that the VM assigned to the VF is assigned a full set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space in hardware. To the rest of the network and the SM, the HCA 400 looks like a switch with additional nodes connected to it via the virtual switch 415. The hypervisor 410 can use the PF 420, and the VMs (attached to the virtual function) use the VF.
[0108] According to an embodiment, the vSwitch architecture provides transparent virtualization. However, because each virtual function is assigned a unique LID, the number of available LIDs is quickly consumed. Similarly, where many LID addresses are used (i.e., each physical function and each virtual function has a LID address), more communication paths must be calculated by the SM and more subnet management packets (SMPs) must be sent to the switch in order to update its LFT. For example, the calculation of communication paths may take several minutes in a large network. Because the LID space is limited to 49151 unicast LIDs, and because each VM (via VF), physical node, and switch each occupies one LID, the number of physical nodes and switches in the network limits the number of active VMs, and vice versa.
[0109] InfiniBand SR-IOV Architecture Model - Virtual Port (vPort)
[0110] Figure 6 An example vPort concept according to an embodiment is shown. As shown, a host 300 (e.g., a host channel adapter) can interact with a hypervisor 410, which can assign various virtual functions 330, 340, 350 to multiple virtual machines. Similarly, physical functions can be handled by the hypervisor 310.
[0111] According to an embodiment, the vPort concept is loosely defined to give vendors freedom to implement (e.g., the definition does not dictate that the implementation must be SRIOV-specific), and the goal of vPort is to standardize the way VMs are handled in a subnet. Using the vPort concept, SR-IOV-like shared ports and vSwitch-like architectures or a combination of both can be defined that are more scalable in both space and performance domains. vPorts support optional LIDs, and unlike shared ports, the SM is aware of all vPorts available in the subnet even if the vPort does not use a dedicated LID.
[0112] InfiniBand SR-IOV Architecture Model – vSwitch with pre-populated LIDs
[0113] According to an embodiment, the present disclosure provides systems and methods for providing a vSwitch architecture with pre-populated LIDs.
[0114] Figure 7 An example vSwitch architecture with a pre-filled LID according to an embodiment is shown. As shown, multiple switches 501-504 can provide communication between members of an architecture (such as an InfiniBand architecture) within a network switching environment 600 (e.g., an IB subnet). The architecture can include multiple hardware devices, such as host channel adapters 510, 520, 530. Each of the host channel adapters 510, 520, 530 can interact with hypervisors 511, 521, and 531, respectively. Each hypervisor can establish multiple virtual functions 514, 515, 516, 524, 525, 526, 534, 535, 536 in conjunction with the host channel adapter it interacts with and assign the multiple virtual functions to multiple virtual machines. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 may additionally assign virtual machine 2 551 to virtual function 2 515 and virtual machine 3 552 to virtual function 3 516. Hypervisor 531 may in turn assign virtual machine 4 553 to virtual function 1 534. The hypervisor may access the host channel adapters through the full featured physical functions 513, 523, 533 on each of the host channel adapters.
[0115] According to an embodiment, each of the switches 501 - 504 may include a plurality of ports (not shown) used to set up a linear forwarding table to direct traffic within the network switching environment 600 .
[0116] According to an embodiment, the virtual switches 512, 522, and 532 may be handled by their respective hypervisors 511, 521, 531. In such a vSwitch architecture, each virtual function is a complete virtual host channel adapter (vHCA), which means that the VM assigned to the VF is assigned a complete set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space in hardware. To the rest of the network and the SMs (not shown), the HCAs 510, 520, and 530 appear as switches with attached nodes connected to them via the virtual switches.
[0117] According to an embodiment, the present disclosure provides systems and methods for providing a vSwitch architecture with pre-populated LIDs. Figure 7 , LIDs are pre-populated to various physical functions 513, 523, 533 and virtual functions 514-516, 524-526, 534-536 (even those virtual functions that are not currently associated with an active virtual machine). For example, physical function 513 is pre-populated with LID 1, while virtual function 1 534 is pre-populated with LID 10. When the network is started, LIDs are pre-populated in the subnets that enable the SR-IOV vSwitch. Even if not all VFs are occupied by VMs in the network, the populated VFs are assigned LIDs, such as Figure 7 shown.
[0118] According to an embodiment, much like a physical Host Channel Adapter may have more than one port (two ports are common for redundancy), a virtual HCA may also be represented by two ports and connected to an external IB subnet via one, two, or more virtual switches.
[0119] According to an embodiment, in a vSwitch architecture with pre-filled LIDs, each hypervisor can consume one LID for itself through PF, and consume one more LID for each additional VF. The sum of all VFs available in all hypervisors in the IB subnet gives the maximum number of VMs allowed to run in the subnet. For example, in an IB subnet with 16 virtual functions per hypervisor in the subnet, each hypervisor consumes 17 LIDs in the subnet (one LID for each virtual function in the 16 virtual functions plus one LID for the physical function). In such an IB subnet, the theoretical hypervisor limit for a single subnet is determined by the number of available unicast LIDs, and is: 2891 (49151 available LIDs divided by 17 LIDs per hypervisor), and the total number of VMs (i.e., the limit) is 46256 (2891 hypervisors multiplied by 16 VFs per hypervisor). (In fact, these numbers are actually smaller, because each switch, router, or dedicated SM node in the IB subnet also consumes LIDs). Note that the vSwitch does not need to occupy an additional LID since it can share the LID with the PF.
[0120] According to an embodiment, in a vSwitch architecture with pre-populated LIDs, communication paths are calculated for all LIDs when the network is first started. When a new VM needs to be started, the system does not have to add a new LID in the subnet, an action that would otherwise result in a complete reconfiguration of the network, including path recalculation, which is the most time-consuming part. Instead, an available port (i.e., an available virtual function) for the VM is located in one of the hypervisors, and the virtual machine is attached to the available virtual function.
[0121] According to an embodiment, a vSwitch architecture with pre-populated LIDs also allows the ability to compute and use different paths to reach different VMs hosted by the same hypervisor. Essentially, this allows such subnets and networks to use features like LID Mask Control (LMC) to provide alternative paths toward a physical machine without being constrained by the limitations of LMC that require LIDs to be contiguous. Being able to freely use non-contiguous LIDs is particularly useful when a VM needs to be migrated and its associated LIDs are carried to a destination.
[0122] According to the embodiments, and the advantages of the vSwitch architecture with pre-populated LIDs shown above, certain considerations may be taken into account. For example, because the LIDs are pre-populated in the SR-IOV vSwitch-enabled subnet when the network boots up, the initial path calculation (e.g., at boot time) may take longer than if the LIDs were not pre-populated.
[0123] InfiniBand SR-IOV Architecture Model - vSwitch with Dynamic LID Allocation
[0124] According to an embodiment, the present disclosure provides systems and methods for providing a vSwitch architecture with dynamic LID allocation.
[0125] Figure 8 An example vSwitch architecture with dynamic LID allocation according to an embodiment is shown. As shown, multiple switches 501-504 can provide communication between members of an architecture (such as an InfiniBand architecture) within a network switching environment 700 (e.g., an IB subnet). The architecture can include multiple hardware devices, such as host channel adapters 510, 520, 530. Each of the host channel adapters 510, 520, 530 can interact with a hypervisor 511, 521, 531, respectively. Each hypervisor can also establish multiple virtual functions 514, 515, 516, 524, 525, 526, 534, 535, 536 in conjunction with the host channel adapter interacting with it and assign the multiple virtual functions to multiple virtual machines. For example, virtual machine 1550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 can additionally assign virtual machine 2 551 to virtual function 2 515, and assign virtual machine 3 552 to virtual function 3 516. The hypervisor 531 may in turn assign virtual machine 4 553 to virtual function 1 534. The hypervisor may access the host channel adapters through the full featured physical functions 513, 523, 533 on each of the host channel adapters.
[0126] According to an embodiment, each of the switches 501 - 504 may include a plurality of ports (not shown) used to set up a linear forwarding table to direct traffic within the network switching environment 700 .
[0127] According to an embodiment, the virtual switches 512, 522, and 532 may be handled by their respective hypervisors 511, 521, 531. In such a vSwitch architecture, each virtual function is a complete virtual host channel adapter (vHCA), which means that the VM assigned to the VF is assigned a complete set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space in hardware. To the rest of the network and the SMs (not shown), the HCAs 510, 520, and 530 appear as switches with attached nodes connected to them via the virtual switches.
[0128] According to an embodiment, the present disclosure provides systems and methods for providing a vSwitch architecture with dynamic LID allocation. Figure 8, LIDs are dynamically assigned to various physical functions 513, 523, 533, where physical function 513 receives LID1, physical function 523 receives LID 2, and physical function 533 receives LID 3. Those virtual functions associated with active virtual machines can also receive dynamically assigned LIDs. For example, since virtual machine 1 550 is active and associated with virtual function 1 514, virtual function 514 can be assigned LID 5. Similarly, virtual function 2 515, virtual function 3 516, and virtual function 1 534 are each associated with an active virtual function. As a result, these virtual functions are assigned LIDs, where LID 7 is assigned to virtual function 2 515, LID 11 is assigned to virtual function 3 516, and LID 9 is assigned to virtual function 1 534. Unlike a vSwitch with pre-populated LIDs, those virtual functions that are not currently associated with an active virtual machine do not receive LID assignments.
[0129] According to an embodiment, using dynamic LID allocation, initial path calculation can be significantly reduced. When the network is first started and no VMs exist, a relatively small number of LIDs can be used for initial path calculation and LFT distribution.
[0130] According to an embodiment, much like a physical Host Channel Adapter may have more than one port (two ports are common for redundancy), a virtual HCA may also be represented by two ports and connected to an external IB subnet via one, two, or more virtual switches.
[0131] According to an embodiment, when a new VM is created in a system utilizing a vSwitch with dynamic LID allocation, an idle VM timeslot is found in order to decide on which hypervisor to start the newly added VM, and a unique unused unicast LID is also found. However, there are no known paths in the network and the LFT of the switch is used to handle the newly added LID. Calculating a new set of paths to handle the newly added VM is undesirable in a dynamic environment where several VMs may be started every minute. In a large IB subnet, calculating a new set of routes may take several minutes, and this process will have to be repeated each time a new VM is started.
[0132] Advantageously, according to an embodiment, since all VFs in a hypervisor share the same uplinks with the PF, there is no need to compute a new set of routes. All that is required is to traverse the LFTs of all physical switches in the network, copy the LID entries of the forwarding ports from the PFs belonging to the hypervisor (where the VM is created) to the newly added LID, and send a single SMP to update the corresponding LFT block for the particular switch. Thus, the system and method avoids the need to compute a new set of routes.
[0133] According to an embodiment, the LIDs allocated in a vSwitch with a dynamic LID allocation architecture are not necessarily contiguous. When comparing the LIDs allocated on VMs on each hypervisor in a vSwitch with pre-populated LIDs to those in a vSwitch with dynamic LID allocation, it should be noted that the LIDs allocated in the dynamic LID allocation architecture are non-contiguous, while those that are pre-populated are contiguous in nature. In the vSwitch dynamic LID allocation architecture, when a new VM is created, the next available LID is used throughout the life cycle of the VM. In contrast, in a vSwitch with pre-populated LIDs, each VM inherits the LID that has been assigned to the corresponding VF, and in a network without live migration, VMs that are continuously attached to a given VF obtain the same LID.
[0134] According to an embodiment, a vSwitch with a dynamic LID allocation architecture can address the shortcomings of the vSwitch by utilizing a pre-populated LID architecture model at the expense of some additional network and runtime SM overhead. Each time a VM is created, the LFT of the physical switch in the subnet is updated with the newly added LID associated with the created VM. For this operation, a subnet management packet (SMP) per switch needs to be sent. Because each VM is using the same path as its host hypervisor, LMC-like functions are also not available. However, there is no limit to the total number of VFs present in all hypervisors, and the number of VFs can exceed the number of unicast LID limits. Of course, if this is the case, not all VFs are allowed to be attached to active VMs at the same time, but having more idle hypervisors and VFs increases the flexibility of disaster recovery and segmented network optimization when operations approach the unicast LID limit.
[0135] InfiniBand SR-IOV Architecture Model - vSwitch with Dynamic LID Allocation and Pre-populated LIDs
[0136] Fig. 9An example vSwitch architecture with a vSwitch having dynamic LID allocation and pre-filled LID according to an embodiment is shown. As shown, multiple switches 501-504 can provide communication between members of an architecture (such as an InfiniBand architecture) within a network switching environment 800 (e.g., an IB subnet). The architecture can include multiple hardware devices, such as host channel adapters 510, 520, 530. Each of the host channel adapters 510, 520, 530 can interact with hypervisors 511, 521, and 531, respectively. Each hypervisor can also establish multiple virtual functions 514, 515, 516, 524, 525, 526, 534, 535, 536 in conjunction with the host channel adapters it interacts with and assign the multiple virtual functions to multiple virtual machines. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 can additionally assign virtual machine 2 551 to virtual function 2 515. Hypervisor 521 may assign virtual machine 3 552 to virtual function 3 526. Hypervisor 531 may in turn assign virtual machine 4 553 to virtual function 2 535. The hypervisor may access the host channel adapters through the full featured physical functions 513, 523, 533 on each of the host channel adapters.
[0137] According to an embodiment, each of the switches 501 - 504 may include a plurality of ports (not shown) used to set up a linear forwarding table to direct traffic within the network switching environment 800 .
[0138] According to an embodiment, the virtual switches 512, 522, and 532 may be handled by their respective hypervisors 511, 521, 531. In such a vSwitch architecture, each virtual function is a complete virtual host channel adapter (vHCA), which means that the VM assigned to the VF is assigned a complete set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space in hardware. To the rest of the network and the SMs (not shown), the HCAs 510, 520, and 530 appear as switches with attached nodes connected to them via the virtual switches.
[0139] According to an embodiment, the present disclosure provides systems and methods for providing a hybrid vSwitch architecture with dynamic LID allocation and pre-populated LIDs. Fig. 9, hypervisor 511 can be arranged with a vSwitch having a pre-populated LID architecture, while hypervisor 521 can be arranged with a vSwitch having a pre-populated LID and dynamic LID allocation. Hypervisor 531 can be arranged with a vSwitch having a dynamic LID allocation. Therefore, physical function 513 and virtual functions 514-516 have their LIDs pre-populated (i.e., even those virtual functions that are not attached to active virtual machines are assigned LIDs). Physical function 523 and virtual function 1 524 can have their LIDs pre-populated, while virtual functions 2 and 3, 525 and 526 have their LIDs dynamically allocated (i.e., virtual function 2 525 can be used for dynamic LID allocation, and since virtual machine 3 552 is attached, virtual function 3 526 has a dynamically allocated LID 11). Finally, the functions (physical functions and virtual functions) associated with hypervisor 3 531 can have their LIDs dynamically allocated. This makes virtual functions 1 and 3, 534 and 536 available for dynamic LID allocation, while virtual function 2 535 has a dynamically allocated LID 9 since virtual machine 4 553 is attached thereto.
[0140] According to such Fig. 9 In the illustrated embodiments in which both vSwitches with pre-populated LIDs and vSwitches with dynamic LID allocation are utilized (either independently or in combination within any given hypervisor), the number of pre-populated LIDs per host channel adapter can be defined by the fabric administrator and can be in the range of 0<=pre-populated VFs<=total VFs (per host channel adapter), and the VFs available for dynamic LID allocation can be found by subtracting the number of pre-populated VFs from the total number of VFs (per host channel adapter).
[0141] According to an embodiment, much like a physical Host Channel Adapter may have more than one port (two ports are common for redundancy), a virtual HCA may also be represented by two ports and connected to an external IB subnet via one, two, or more virtual switches.
[0142] InfiniBand - Inter-subnet communication (Fabric Manager)
[0143] According to an embodiment, in addition to providing an InfiniBand architecture within a single subnet, embodiments of the present disclosure may also provide an InfiniBand architecture spanning two or more subnets.
[0144] Fig.10An example multi-subnet InfiniBand architecture according to an embodiment is shown. As shown, within subnet A 1000, multiple switches 1001-1004 can provide communication between members of an architecture (such as an InfiniBand architecture) within subnet A 1000 (e.g., an IB subnet). The architecture can include multiple hardware devices, such as, for example, a channel adapter 1010. The host channel adapter 1010 can interact with a hypervisor 1011. The hypervisor can also establish multiple virtual functions 1014 in conjunction with the host channel adapter interacting with it. The hypervisor can additionally assign a virtual machine to each virtual function, such as virtual machine 1 1015 being assigned to virtual function 1 1014. The hypervisor can access its associated host channel adapter through a full-featured physical function (such as physical function 1013) on each of the host channel adapters. Within subnet B 1040, multiple switches 1021-1024 can provide communication between members of an architecture (such as an InfiniBand architecture) within subnet B 1040 (e.g., an IB subnet). The architecture may include multiple hardware devices, such as, for example, channel adapters 1030. Host channel adapters 1030 may in turn interact with hypervisors 1031. The hypervisor may in turn establish multiple virtual functions 1034 with the host channel adapters with which it interacts. The hypervisor may additionally assign virtual machines to each of the virtual functions, such as virtual machine 2 1035 being assigned to virtual function 2 1034. The hypervisor may access its associated host channel adapter through a fully featured physical function on each of the host channel adapters, such as physical function 1033. It should be noted that while only one host channel adapter is shown within each subnet (i.e., subnet A and subnet B), it should be understood that multiple host channel adapters and their corresponding components may be included within each subnet.
[0145] According to an embodiment, each of the host channel adapters may be additionally associated with a virtual switch (such as virtual switch 1012 and virtual switch 1032), and each HCA may be established with a different architecture model, as described above. Fig.10 Both subnets in are shown as using a vSwitch with a pre-populated LID architecture model, but this is not meant to imply that all such subnet configurations can follow a similar architecture model.
[0146] According to an embodiment, at least one switch within each subnet may be associated with a router, such as switch 1002 within subnet A 1000 is associated with router 1005 , and switch 1021 within subnet B 1040 is associated with router 1006 .
[0147] According to an embodiment, at least one device (e.g., a switch, a node, etc.) may be associated with a fabric manager (not shown). The fabric manager may be used, for example, to discover inter-subnet fabric topology, create fabric profiles (e.g., virtual machine fabric profiles), and build virtual machine-related database objects that form the basis for building virtual machine fabric profiles. In addition, the fabric manager may define legal inter-subnet connectivity based on which subnets are allowed to communicate via which router ports using which partition numbers.
[0148] According to an embodiment, when traffic at an originating source (such as virtual machine 1 within subnet A) is addressed to a destination in a different subnet (such as virtual machine 2 within subnet B), the traffic can be addressed to a router within subnet A, i.e., router 1005, which can then pass the traffic to subnet B via its link with router 1006.
[0149] Virtual Dual Port Router
[0150] According to an embodiment, a dual-port router abstraction may provide a simple way to enable definition of subnet-to-subnet router functionality based on a switch hardware implementation that has the ability to perform GRH (Global Routing Header) to LRH (Local Routing Header) conversion in addition to performing normal LRH-based switching.
[0151] According to an embodiment, a virtual dual-port router can be logically connected to the outside of the corresponding switch port. Such a virtual dual-port router can provide a view that complies with the InfiniBand specification to a standard management entity (such as a subnet manager).
[0152] According to an embodiment, the dual-port router model means that different subnets can be connected in a manner where each subnet has full control over the forwarding of packets and address mapping in the ingress path to the subnet and does not affect routing and logical connectivity within any improperly connected subnet.
[0153] According to an embodiment, in situations involving improperly connected architectures, use of a virtual dual-port router abstraction may also allow management entities such as subnet managers and IB diagnostic software to behave correctly in the presence of unexpected physical connectivity to remote subnets.
[0154] Fig.11The interconnection between two subnets in a high performance computing environment according to an embodiment is shown. Before being configured with a virtual dual-port router, a switch 1120 in subnet A 1101 can be connected to a switch 1130 in subnet B 1102 via a switch port 1131 of switch 1130 via a physical connection 1110 through a switch port 1121 of switch 1120. In such an embodiment, each switch port 1121 and 1131 can function as both a switch port and a router port.
[0155] According to an embodiment, a problem with this configuration is that a management entity such as a subnet manager in an InfiniBand subnet cannot distinguish between a physical port that is both a switch port and a router port. In this case, the SM may treat the switch port as having a router port connected to the switch port. However, if the switch port is connected to another subnet via, for example, a physical link with another subnet manager, the subnet manager may be able to send a discovery message on the physical link. However, such a discovery message may not be allowed at the other subnet.
[0156] Fig.12 An interconnection between two subnets via a dual-port virtual router configuration in a high performance computing environment is shown according to an embodiment.
[0157] According to an embodiment, after configuration, a dual-port virtual router configuration may be provided so that the subnet manager sees the correct end node indicating the end of the subnet for which the subnet manager is responsible.
[0158] According to an embodiment, at a switch 1220 in subnet A 1201, the switch port may be connected (i.e., logically connected) to a router port 1211 in a virtual router 1210 via a virtual link 1223. The virtual router 1210 (e.g., a dual-port virtual router) (which may be logically contained within the switch 1220 while shown in the embodiment as being outside the switch 1220) may also include a second router port: a router port II 1212. According to an embodiment, a physical link 1203, which may have two ends, may connect subnet A 1201 to subnet B 1202 via a first end of the physical link via a router port II 1212 and a router port II 1232 in a virtual router 1230 contained in subnet B 1202 via a second end of the physical link. The virtual router 1230 may additionally include a router port 1231, which may be connected (i.e., logically connected) to a switch port 1241 on a switch 1240 via a virtual link 1233.
[0159] According to an embodiment, a subnet manager (not shown) on subnet A can detect a router port 1211 on a virtual router 1210 as an endpoint of a subnet controlled by the subnet manager. The dual-port virtual router abstraction can allow the subnet manager on subnet A to handle subnet A in a usual manner (e.g., as defined in the InifiniBand specification). At the subnet management agent level, a dual-port virtual router abstraction can be provided so that the SM sees a normal switch port, and then at the SMA level, sees that there is another port connected to the switch port and that this port is an abstraction of a router port on the dual-port virtual router. In the local SM, the conventional architecture topology can continue to be used (the SM sees the port as a standard switch port in the topology), and therefore the SM sees the router port as an end port. A physical connection can be made between two switch ports, which are also configured as router ports in two different subnets.
[0160] According to an embodiment, the dual-port virtual router can also solve the problem that the physical link may be mistakenly connected to some other switch port in the same subnet or connected to a switch port that is not intended to provide a connection to another subnet. Therefore, the methods and systems described herein also provide a representation of content outside the subnet.
[0161] According to an embodiment, within a subnet (such as subnet A), the local SM determines a switch port and then determines a router port connected to the switch port (e.g., router port 1211 connected to switch port 1221 via virtual link 1223). Since the SM considers router port 1211 to be the end of the subnet managed by the SM, the SM cannot send discovery messages and / or management messages beyond this point (e.g., to router port II 1212).
[0162] According to an embodiment, the benefit provided by the above-mentioned two-port virtual router is that the two-port virtual router abstraction is completely managed by a management entity (e.g., SM or SMA) within the subnet to which the two-port virtual router belongs. By allowing management only on the local side, the system does not have to provide an external independent management entity. That is, each side of the subnet-to-subnet connection can be responsible for configuring its own two-port virtual router.
[0163] According to an embodiment, in the event where a packet such as an SMP addressed to a remote destination (ie, outside the local subnet) arrives at a local target port that is not configured by a dual-port virtual router as described above, then the local port may return a message indicating that it is not a router port.
[0164] Many features of the present teachings can be implemented in, using, or with the help of hardware, software, firmware, or a combination thereof, in hardware, software, firmware, or a combination thereof. Therefore, the features of the present teachings can be implemented using a processing system (e.g., including one or more processors).
[0165] Fig.13 A method for supporting a dual-port virtual router in a high-performance computing environment according to an embodiment is shown. At step 1310, the method can provide a first subnet at one or more computers including one or more microprocessors, the first subnet comprising: a plurality of switches, the plurality of switches including at least a leaf switch, wherein each of the plurality of switches comprises a plurality of switch ports; a plurality of host channel adapters, each of the host channel adapters comprising at least one host channel adapter port; a plurality of end nodes, wherein each of the end nodes is associated with at least one host channel adapter in the plurality of host channel adapters; and a subnet manager, the subnet manager running on one of the plurality of switches and the plurality of host channel adapters.
[0166] At step 1320 , the method may configure a switch port of a plurality of switch ports on a switch of the plurality of switches as a router port.
[0167] At step 1330, the method may logically connect the switch port configured as a router port to a virtual router, the virtual router comprising at least two virtual router ports.
[0168] Quality of Service and Service Level Agreements in Private Architectures
[0169] According to an embodiment, a high performance computing environment such as a switched network running on InfiniBand or RoCE within a cloud and within a larger cloud installed at a customer and on-premise has the ability to deploy virtual machine (VM) based workloads, where an inherent requirement is that the quality of service (QOS) can be defined and controlled for different types of communication flows. In addition, workloads belonging to different tenants must execute within the boundaries of the relevant service level agreements (SLAs) while minimizing interference between such workloads and maintaining the QOS assumptions for different communication types.
[0170] RDMA read as restricted feature (ORA200246-US-NP-1)
[0171] According to an embodiment, when defining bandwidth limits in a system using a conventional network interface (NIC), it is usually sufficient to control the egress bandwidth that each node / VM is allowed to generate on the network.
[0172] However, according to an embodiment, for RDMA-based networking where different nodes are capable of generating RDMA read requests (i.e., egress bandwidth), this may represent a small amount of egress bandwidth. However, such RDMA read requests may potentially represent a very large amount of ingress RDMA traffic in response to such RDMA read requests. In this case, it is no longer sufficient to limit the egress bandwidth of all nodes / VMs in order to control the total traffic generation in the system.
[0173] According to an embodiment, by making RDMA read operations a restricted feature and only allowing such read requests from nodes / VMs that are trusted not to generate excessive RDMA read-based ingress bandwidth, the overall bandwidth utilization can be limited while only limiting the send (egress) bandwidth of untrusted nodes / VMs.
[0174] Fig.14 A system for providing RDMA read requests as a restricted feature in a high performance computing environment according to an embodiment is shown.
[0175] More specifically, according to an embodiment, Fig.14 A host channel adapter 1401 is shown including a hypervisor 1411. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 1414-1416, and a physical function (PF) 1413. The host channel adapter may additionally support / include multiple ports, such as ports 1402 and 1403, for connecting the host channel adapter to a network, such as network 1400. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect HCA 1401 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0176] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 1450 , VM2 1451 , and VM3 1452 .
[0177] According to an embodiment, the host channel adapter 1401 may additionally support a virtual switch 1412 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0178] According to an embodiment, the host channel adapter may implement trusted RDMA read restrictions 1460, whereby the read restrictions 1460 may be configured to prevent any virtual machine (e.g., VM1, VM2, and / or VM3) from issuing any RDMA read requests to the network (e.g., via ports 1402 or 1403).
[0179] According to an embodiment, trusted RDMA read restriction 1460 may implement a host channel adapter level block on the generation (i.e., issuance) of RDMA read request packets for certain types of packets from certain endpoints such as virtual machines or other physical nodes connected to the network using HCA 1401. The configurable restriction component 1460 may, for example, only allow trusted nodes (e.g., VMs or physical end nodes) to generate such packets.
[0180] According to an embodiment, the trusted RDMA read restriction component may be configured, for example, based on instructions received by a host channel adapter, or it may be directly configured, for example, by a subnet manager (not shown).
[0181] Fig.15 A system for providing RDMA read requests as a restricted feature in a high performance computing environment according to an embodiment is shown.
[0182] More specifically, according to an embodiment, Fig.15 A host channel adapter 1501 is shown including a hypervisor 1511. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 1514-1516, and a physical function (PF) 1513. The host channel adapter may additionally support / include multiple ports, such as ports 1502 and 1503, for connecting the host channel adapter to a network, such as network 1500. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect HCA 1501 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0183] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 1550 , VM2 1551 , and VM3 1552 .
[0184] According to an embodiment, the host channel adapter 1501 may additionally support a virtual switch 1512 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0185] According to an embodiment, the host channel adapter may implement trusted RDMA read restrictions 1560, whereby the read restrictions 1560 may be configured to prevent any virtual machine (e.g., VM1, VM2, and / or VM3) from issuing any RDMA read requests to the network (e.g., via ports 1502 or 1503).
[0186] According to an embodiment, trusted RDMA read restriction 1560 may implement a host channel adapter level block on the generation (i.e., issuance) of RDMA read request packets for certain types of packets from certain endpoints such as virtual machines or other physical nodes connected to the network using HCA 1501. The configurable restriction component 1560 may, for example, only allow trusted nodes (e.g., VMs or physical end nodes) to generate such packets.
[0187] According to an embodiment, the trusted RDMA read restriction component may be configured, for example, based on instructions received by a host channel adapter, or it may be directly configured, for example, by a subnet manager (not shown).
[0188] According to an embodiment, as an example, the trusted RDMA read restriction 1560 may be configured to trust VM1 1550 and distrust VM2 1551. Thus, an RDMA read request 1554 initiated from VM1 may be allowed, while an RDMA read request 1555 initiated from VM2 may be blocked before the read request leaves the host channel adapter 1501 (but not before the RDMA read request is received by the host channel adapter 1501). Fig.15 The inside is shown outside the HCA, this is for drawing convenience only).
[0189] Fig.16 A system for providing RDMA read requests as a restricted feature in a high performance computing environment according to an embodiment is shown.
[0190] According to an embodiment, within a high-performance computing environment such as a switching network or subnet 1600, multiple end nodes 1601 and 1602 can support multiple virtual machines VM1-VM4 1650-1653, which are interconnected via multiple switches, such as leaf switches 1611 and 1612, switches 1621 and 1622, and root switches 1631 and 1632.
[0191] Not shown in the figure are various host channel adapters that provide functionality for connecting nodes 1601 and 1602 and virtual machines to be connected to the subnet, according to an embodiment. Such embodiments are discussed above with respect to SR-IOV, where each virtual machine can be associated with a virtual function of a hypervisor on a host channel adapter.
[0192] According to an embodiment, in a general system, the RDMA egress bandwidth from any one virtual machine of an end node is limited to prevent any one virtual machine from monopolizing the bandwidth of any link connecting the end node to the subnet. However, such egress bandwidth limitation, while generally effective, does not prevent the virtual machine from issuing RDMA read requests, such as RDMA read requests 1654 and 1655. This is because such RDMA read requests are typically small packets and are of little use in utilizing the egress bandwidth.
[0193] However, according to an embodiment, such RDMA read requests may result in a large amount of return traffic being generated to issuing entities such as VM1 and VM3. In this case, for example, when read request 1654 results in a large amount of data traffic flowing back to VM1 due to the execution of the read request at the destination, then the RDMA read request may result in link congestion and network performance degradation.
[0194] According to an embodiment, this may result in performance loss of the subnet, especially where more than one tenant shares subnet 1600.
[0195] According to an embodiment, each node (or host channel adapter) may be configured with RDMA read restrictions 1660 and 1661 that will prevent any VM from issuing RDMA read requests when the VM is not trusted. Such RDMA read restrictions may vary from a permanent block on issuing RDMA read requests to setting a time range limit on when a virtual machine configured with the RDMA read request restrictions can issue RDMA read requests (e.g., during slow network traffic). In addition, the RDMA read restrictions 1660 and 1661 may additionally allow trusted VMs to issue RDMA read requests.
[0196] According to an embodiment, since there may be scenarios where multiple VMs / tenants are sharing a "new" HCA - i.e., an HCA that supports the relevant new features, but are performing RDMA requests to remote "old" HCAs that do not have such support, it makes sense to find a way to limit the ingress bandwidth that such VMs can generate in terms of RDMA read responses without relying on a static rate configuration on the "old" RDMA read responder HCA. There is no straightforward way to do this as long as the VMs are allowed to generate "any" RDMA read size. Furthermore, since multiple RDMA read requests generated over a period of time may in principle all receive response data at the same time, there is no way to ensure that the ingress bandwidth cannot exceed the maximum bandwidth for a very limited amount of time unless there is a limit on the size of the RDMA read that can be generated in a single request and a limit on the total number of outstanding RDMA read requests from the same vHCA port.
[0197] Therefore, according to an embodiment, assuming that a maximum read size is defined for the vHCA, the bandwidth control can be based on a quota for the sum of all outstanding read sizes, or a simpler solution can be to limit the maximum number of outstanding RDMA reads based only on the "worst case" read size. Therefore, in either case, there is no limit on the peak bandwidth within a short interval (except for the maximum link bandwidth of the HCA port), but the duration of this peak bandwidth "window" will be limited. However, in addition, the sending rate of RDMA read requests must be throttled so that the sending rate of requests does not exceed the maximum allowed ingress rate (assuming that responses with data are received at the same rate). In other words, the maximum outstanding request limit defines the worst-case short interval bandwidth, and the request sending rate limit will ensure that new requests cannot be generated immediately when a response is received, but only after a relevant delay representing an acceptable average ingress bandwidth for RDMA read responses. Therefore, in the worst case, the allowed number of requests have been sent without any response, and then all of these responses are received "at the same time". At this time, the next request can be sent immediately when the first response arrives, but subsequent requests will have to be delayed for a specified delay period. Therefore, over time, the average ingress bandwidth cannot exceed the bandwidth defined by the request rate. However, a smaller maximum number of outstanding requests will reduce the possible "variance".
[0198] Using explicit RDMA read bandwidth limits (ORA200246-US-NP-1)
[0199] According to an embodiment, when defining bandwidth limits in a system using a conventional network interface (NIC), it is usually sufficient to control the egress bandwidth that each node / VM is allowed to generate on the network.
[0200] However, according to an embodiment, for RDMA-based networking where different nodes can generate RDMA read requests representing a few request messages but potentially very many response messages, it is no longer sufficient to limit the egress bandwidth of all nodes / VMs in order to control the total traffic generation in the system.
[0201] According to an embodiment, by defining an explicit quota for how much RDMA read ingress bandwidth a node / VM is allowed to generate independent of any transmit / egress bandwidth limits, the overall traffic generation in the system can be controlled without relying on limiting the use of RDMA reads by untrusted nodes / VMs.
[0202] According to an embodiment, in addition to supporting average ingress bandwidth utilization due to locally generated RDMA read requests, the systems and methods may also support the worst-case duration / length of maximum link bandwidth bursts (i.e., as a result of RDMA read response "piling up").
[0203] Fig.17A system for providing explicit RDMA read bandwidth limiting in a high performance computing environment according to an embodiment is shown.
[0204] More specifically, according to an embodiment, Fig.17 A host channel adapter 1701 is shown including a hypervisor 1711. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 1714-1716, and a physical function (PF) 1713. The host channel adapter may additionally support / include multiple ports, such as ports 1702 and 1703, for connecting the host channel adapter to a network, such as network 1700. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 1701 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0205] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 1750 , VM2 1751 , and VM3 1752 .
[0206] According to an embodiment, the host channel adapter 1701 may additionally support a virtual switch 1712 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0207] According to an embodiment, the host channel adapter may implement RDMA read limits 1760, whereby read limits 1760 may be configured to set a quota on the amount of ingress bandwidth that any VM (of HCA 1701) may generate in response to RDMA read requests issued by a particular VM. Limiting such ingress bandwidth is performed locally on the host channel adapter.
[0208] According to an embodiment, the RDMA read limit component may be configured, for example, based on instructions received by a host channel adapter, or it may be directly configured, for example, by a subnet manager (not shown).
[0209] Fig.18 A system for providing explicit RDMA read bandwidth limiting in a high performance computing environment according to an embodiment is shown.
[0210] More specifically, according to an embodiment, Fig.18A host channel adapter 1801 is shown including a hypervisor 1811. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 1814-1816, and a physical function (PF) 1813. The host channel adapter may additionally support / include multiple ports, such as ports 1802 and 1803, which are used to connect the host channel adapter to a network, such as network 1800. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect HCA 1801 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0211] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 1850 , VM2 1851 , and VM3 1852 .
[0212] According to an embodiment, the host channel adapter 1801 may additionally support a virtual switch 1812 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0213] According to an embodiment, the host channel adapter may implement RDMA read limits 1860, whereby read limits 1860 may be configured to set a quota on the amount of ingress bandwidth that any VM (of HCA 1701) may generate in response to RDMA read requests issued by a particular VM. Limiting such ingress bandwidth is performed locally on the host channel adapter.
[0214] According to an embodiment, the RDMA read limit component may be configured, for example, based on instructions received by a host channel adapter, or it may be directly configured, for example, by a subnet manager (not shown).
[0215] According to an embodiment, for example, VM1 may have previously issued at least two RDMA read requests, requesting to perform a read operation on a connected node. In response, VM1 may be in the process of receiving multiple responses to the RDMA read requests (shown in the figure as RDMA read responses 1855 and 1854). Because these RDMA read responses may be very large, especially compared to the RDMA read request originally sent by VM1, these read responses 1854 and 1855 may be subject to RDMA read limits 1860, and the ingress bandwidth may be limited or throttled. This throttling may be based on an explicit ingress bandwidth limit, or it may be based on the QoS and / or SLA of VM1 set within the RDMA limit 1860.
[0216] Fig.19A system for providing explicit RDMA read bandwidth limiting in a high performance computing environment according to an embodiment is shown.
[0217] According to an embodiment, within a high-performance computing environment such as a switching network or subnet 1900, multiple end nodes 1901 and 1902 can support multiple virtual machines VM1-VM4 1950-1953, which are interconnected via multiple switches, such as leaf switches 1911 and 1912, switches 1921 and 1922, and root switches 1931 and 1932.
[0218] According to an embodiment, not shown in the figure are various host channel adapters that provide functionality for connecting nodes 1901 and 1902 and virtual machines to be connected to the subnet. Such embodiments are discussed above with respect to SR-IOV, where each virtual machine can be associated with a virtual function of a hypervisor on a host channel adapter.
[0219] According to an embodiment, in a general system, the RDMA egress bandwidth from any one virtual machine of an end node is limited to prevent any one virtual machine from monopolizing the bandwidth of any link connecting the end node to a subnet. However, such egress bandwidth limitation, while effective in general, does not prevent an influx of RDMA read responses from monopolizing the link between the requesting VM and the network.
[0220] In other words, if VM1 issues multiple RDMA read requests, VM1 has no control over when responses to these read requests are returned to VM1, according to an embodiment. This may result in a backup / pile-up of responses to RDMA read requests, each of which attempts to use the same link to return the requested information (via RDMA read responses 1954) to VM1. This may result in traffic congestion and backlogs in the network.
[0221] According to an embodiment, RDMA limits 1960 and 1961 may set quotas on the amount of ingress bandwidth a VM may generate in response to RDMA read requests issued by a particular VM. Limiting such ingress bandwidth is performed locally.
[0222] According to an embodiment, assuming that a maximum read size is defined for the vHCA, the bandwidth control can be based on a quota for the sum of all outstanding read sizes, or a simpler solution can be to limit the maximum number of outstanding RDMA reads based only on the "worst case" read size. Therefore, in either case, there is no limit on the peak bandwidth within a short interval (except for the HCA port maximum link bandwidth), but the duration of this peak bandwidth "window" will be limited. However, in addition, the sending rate of RDMA read requests must be throttled so that the sending rate of requests does not exceed the maximum allowed ingress rate (assuming that responses with data are received at the same rate). In other words, the maximum outstanding request limit defines the worst-case short interval bandwidth, and the request sending rate limit will ensure that new requests cannot be generated immediately when a response is received, but only after a relevant delay representing an acceptable average ingress bandwidth for RDMA read responses. Therefore, in the worst case, the allowed number of requests have been sent without any response, and then all of these responses are received "at the same time". At this point, the next request can be sent immediately when the first response arrives, but subsequent requests will have to be delayed by the specified delay time. Therefore, over time, the average ingress bandwidth cannot exceed the bandwidth defined by the request rate. However, a smaller maximum number of outstanding requests will reduce the possible "difference".
[0223] Fig. 20 is a flow chart of a method for providing RDMA (Remote Direct Memory Access) read requests as a restricted feature in a high performance computing environment, according to an embodiment.
[0224] According to an embodiment, at step 2010, the method may provide a first subnet at one or more microprocessors, the first subnet comprising a plurality of switches, a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via a plurality of switches.
[0225] According to an embodiment, at step 2020, the method may provide a plurality of end nodes, the plurality of end nodes comprising a plurality of virtual machines.
[0226] According to an embodiment, at step 2030, the method may associate a host channel adapter with a selective RDMA restriction.
[0227] According to an embodiment, at step 2040, the method may host a virtual machine among the plurality of virtual machines at a host channel adapter that includes a selective RDMA restriction.
[0228] Combining multiple shared bandwidth segments (ORA200246-US-NP-3)
[0229] According to an embodiment, conventional bandwidth / rate limiting schemes for network interfaces are typically limited to a combination of the total combined send rate and the maximum rate of each possible destination. However, in many cases, there is a shared bottleneck in the intermediate network / fabric topology, which means that the total bandwidth available to a set of destinations is limited by this shared bottleneck. Therefore, unless such shared bottlenecks are taken into account when deciding at what rates to send various data flows, the shared bottleneck is likely to be overloaded despite observing the per-destination rate limits.
[0230] According to an embodiment, the systems and methods herein may introduce an object "target group" to which multiple individual flows may be associated, and where the target group may represent the rate limits of individual (possibly shared) links or other bottlenecks within the network / fabric path that the flows are using. Additionally, the systems and methods may allow each flow to be associated with a hierarchy of such target groups, such that all link segments and any other (shared) bottlenecks in the path between the sender and the destination of the individual flows may be represented.
[0231] According to an embodiment, in order to limit the egress bandwidth, the system and method may establish a destination group that shares the bandwidth quota so as to reduce the possibility of congestion on the shared ISL (inter-switch link). This requires a destination / path-related lookup mechanism that can be managed based on which destinations / paths will be mapped to which groups at the logical level. This means that the super-privileged communication infrastructure must know the actual location of the peer nodes in the architecture topology and the relevant routing and capacity information that can be mapped to the "target group" (i.e., HCA-level object type) within the local HCA with the associated bandwidth quota. However, it is impractical for the hardware platform to directly look up the WQE (work queue entry) / packet address information to map to the associated (one or more) target groups. Alternatively, the HCA implementation can provide an association between an RC (reliably connected) QP (queue pair) and an address handle, which represents the sending context of the outgoing traffic and the associated target group. In this way, the association can be transparent at the verb level and can be set up by the super-privileged software level instead, and then implemented at the HCA hardware (and firmware) level. An important additional complexity associated with this scheme is that live VM migrations that maintain associated VM or vHCA port address information throughout the migration process may still mean that there are changes in the target groups for different communication peers. However, as long as the systems and methods tolerate some transient periods where the associated bandwidth quotas are not 100% correct, the target group associations do not have to be updated synchronously. Therefore, while the logical connections and communication capabilities may not change as a result of the VM migration, the (one or more) target groups associated with the RC connections and address handles in the migrating VM and its communication peer VMs may be "completely wrong" after the migration. This may mean that less bandwidth is utilized than is available (for example, when a VM is moved from a remote location to the same "leaf group" as its peers) and also mean that too much bandwidth is generated (for example, when a VM is moved from the same "leaf group" as its peers to a remote location, which means that the shared ISL has limited bandwidth).
[0232] According to an embodiment, the target group-specific bandwidth quota may in principle also be divided into quotas for specific priorities ("QOS classes") in order to reflect the expected bandwidth usage of various priorities within the associated paths in the architecture represented by the target group.
[0233] According to an embodiment, target groups decouple objects from specific destination addresses, and systems and methods gain the ability to represent intermediate, shared links or link groups that can represent bandwidth constraints in addition to the target and potentially more stringent than the target constraints.
[0234] According to an embodiment, the system and method may consider a hierarchy of target groups (bandwidth quotas) that reflect bandwidth / link sharing for different targets at different stages. In principle, this means that a particular flow should be associated with the target group (maximum rate) that represents the most limited rate in the hierarchy. That is, for example, if the target limit is 30Gb / s and the intermediate uplink limit is 50Gb / s, then the maximum rate to the target will never exceed 30Gb / s. On the other hand, if multiple 30Gb / s targets share the same 50Gb / s intermediate limit, then using the relevant target rate limit for flows toward these targets may mean exceeding the intermediate rate limit. Therefore, in order to ensure the best utilization and throughput within the relevant limits, all target groups in the relevant hierarchy can be considered in a relevant strict order. This means that packets can be sent toward the relevant destination only when and only when each target group in the hierarchy represents available bandwidth. Therefore, if a single flow is active for one of the targets in the above example, the flow will be allowed to operate at 30Gb / s. However, once another flow becomes active to the other destination (via the shared intermediate destination group), each flow will be limited to 25Gb / s. If in the next round, an additional flow to one of the two destinations becomes active, then the two flows to the same destination will each operate at 12.5Gb / s (i.e. on average, unless they have any additional bandwidth quotas / limits).
[0235] According to an embodiment, when multiple tenants share a server / HCA, in addition to sharing any intermediate ISL bandwidth, the initial egress bandwidth as well as the actual target bandwidth may also be shared. On the other hand, in a scenario where each tenant has a dedicated server / HCA, the intermediate ISL bandwidth represents the only possible "inter-tenant" bandwidth sharing.
[0236] According to an embodiment, the target group should generally be global for the HCA port, and the VF / tenant quota at the HCA level will represent the maximum local traffic that a tenant can generate for any combination of targets, either globally or for a specific priority. Nevertheless, it is still possible to use target groups specific to some tenants along with a "global" target group in the same hierarchy.
[0237] There are several possible ways to implement target groups and target group associations (hierarchies) that represent specific QPs or address handles, depending on the embodiment. However, a 16-bit target group ID space and support for up to 4 or 8 target group associations can be provided for each QP and address handle. Each target group ID value will then represent some hardware state that reflects an associated IPD (inter-packet delay) value for an associated rate and timer information that defines when the next packet associated with that target group can be sent.
[0238] According to an embodiment, since different flows / paths may use different "QOSIDs" (i.e., service levels, priorities, etc.) on the same shared link segment, different target groups may also be associated with the same link segment, such that different target groups represent bandwidth quotas for such different QOS IDs. However, it is also possible to represent target groups specific to a QOS ID as well as a single target group representing the physical link of the same link segment.
[0239] According to an embodiment, similarly, systems and methods may additionally distinguish between different flow types defined by explicit flow type packet header parameters and / or by taking into account the type of operation (e.g., RDMA read / write / send) in order to implement different "sub-quotas" to arbitrate between different such flow types. In particular, this may be useful to distinguish flows representing responder mode bandwidth (i.e., typically RDMA read response traffic) from requester mode traffic originally initiated by the local node itself.
[0240] According to an embodiment, "any" congestion can be avoided in principle if target groups are strictly used and the rate limits of all relevant sender HCAs add up to a total maximum rate that does not exceed the capacity of any target or shared ISL segment. However, this may mean severe limitations on the sustained bandwidth of different flows and low average utilization of available link bandwidth. Therefore, various rate limits can be set to allow different HCAs to use more optimistic maximum rates. In this case, the aggregate sum is higher than the sustainable maximum and congestion may therefore result.
[0241] Fig.21 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0242] More specifically, according to an embodiment, Fig.21 A host channel adapter 2101 is shown including a hypervisor 2111. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 2114-2116, and a physical function (PF) 2113. The host channel adapter may additionally support / include multiple ports, such as ports 2102 and 2103, which are used to connect the host channel adapter to a network, such as network 2100. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 2101 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0243] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 2150 , VM2 2151 , and VM3 2152 .
[0244] According to an embodiment, the host channel adapter 2101 may additionally support a virtual switch 2112 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0245] According to an embodiment, as shown, the network 2100 may include a plurality of switches, such as switches 2140 , 2141 , 2142 , and 2143 , which are interconnected and may be connected to the host channel adapter 2101 , for example, via leaf switches 2140 and 2141 .
[0246] According to an embodiment, switches 2140 - 2143 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0247] According to an embodiment, target groups, such as target groups 2170 and 2171, can be defined along inter-switch links (ISLs), such as the ISLs between leaf switch 2140 and switch 2142, and leaf switch 2141 and switch 2143. For example, these target groups 2170 and 2171 can represent bandwidth quotas as HCA objects stored at a target group repository 2161 associated with the HCA, which can be accessed by the rate limiting component 2160.
[0248] According to an embodiment, target groups 2170 and 2171 may represent specific (and different) bandwidth quotas. These bandwidth quotas may be divided into quotas for specific priorities ("QOS categories") to reflect the expected bandwidth usage of various priorities within the relevant paths in the architecture represented by the target group.
[0249] According to an embodiment, target groups 2170 and 2171 decouple objects from specific destination addresses, and systems and methods gain the ability to represent intermediate, shared links or link groups that can represent bandwidth limits in addition to the target and potentially more stringent than the target limit. That is, for example, if the default / original egress limit on VM2 2151 is set to a threshold, but the destination of a packet sent from VM2 is to pass through target group 2170, which sets a lower bandwidth limit, then the egress bandwidth from VM2 can be limited to a level lower than the level of the default / original egress limit set on VM2. For example, the HCA can be responsible for such throttling / egress bandwidth limit adjustments, depending on the target group involved in the routing of packets from VM2.
[0250] According to an embodiment, the target groups may also be hierarchical in nature, whereby the system and method may consider a hierarchy of target groups (bandwidth quotas) that reflect bandwidth / link sharing for different targets at different stages. In principle, this means that a particular flow should be associated with the target group (maximum rate) that represents the most limited rate in the hierarchy. That is, for example, if target group 2170 represents a bandwidth limit that is higher than the bandwidth limit of target group 2171, and the packet is addressed by two inter-switch links represented by the two target groups, then the bandwidth limit of target group 2171 is the controlling bandwidth limiting factor.
[0251] According to an embodiment, a target group may also be shared by multiple flows. For example, depending on the QoS and SLA associated with the respective flows, the bandwidth quota represented by the target group may be divided. As an example, if both VM1 and VM2 simultaneously send flows that would involve target group 2170, which, for example, represents a bandwidth quota of 10 Gb / s, and the respective flows have equal QoS and SLA associated therewith, then target group 2170 would represent a 5 Gb / s limit for each flow. This sharing or division of the target group bandwidth quota may vary based on the QoS and SLA associated with the respective flows.
[0252] Fig. 22 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0253] More specifically, according to an embodiment, Fig. 22 A host channel adapter 2201 is shown including a hypervisor 2211. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 2214-2216, and a physical function (PF) 2213. The host channel adapter may additionally support / include multiple ports, such as ports 2202 and 2203, which are used to connect the host channel adapter to a network, such as network 2200. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 2201 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0254] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 2250 , VM2 2251 , and VM3 2252 .
[0255] According to an embodiment, the host channel adapter 2201 may additionally support a virtual switch 2212 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0256] According to an embodiment, as shown, the network 2200 may include a plurality of switches, such as switches 2240 , 2241 , 2242 , and 2243 , which are interconnected and may be connected to the host channel adapter 2201 , for example, via leaf switches 2240 and 2241 .
[0257] According to an embodiment, switches 2240 - 2243 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0258] According to an embodiment, target groups may be defined at, for example, switch ports, such as target groups 2270 and 2271. As shown, target groups 2270 and 2271 are defined at switch ports of switches 2242 and 2243, respectively. For example, these target groups 2270 and 2271 may represent bandwidth quotas as HCA objects, which are stored at a target group repository 2261 associated with the HCA, which target group repository 2261 is accessible by the rate limiting component 2260.
[0259] According to an embodiment, target groups 2270 and 2271 may represent specific (and different) bandwidth quotas. These bandwidth quotas may be divided into quotas for specific priorities ("QOS categories") to reflect the expected bandwidth usage of various priorities within the relevant paths in the architecture represented by the target group.
[0260] According to an embodiment, target groups 2270 and 2271 decouple objects from specific destination addresses, and systems and methods gain the ability to represent intermediate, shared links or link groups that can represent bandwidth limits in addition to the target and potentially more stringent than the target limit. That is, for example, if the default / original egress limit on VM2 2251 is set to a threshold, but the destination of packets sent from VM2 will pass through target group 2270, which sets a lower bandwidth limit, then the egress bandwidth from VM2 can be limited to a level lower than the level of the default / original egress limit set on VM2. For example, the HCA can be responsible for such throttling / egress bandwidth limit adjustments, depending on the target group involved in the routing of packets from VM2.
[0261] According to an embodiment, the target groups may also be hierarchical in nature, whereby the system and method may consider a hierarchy of target groups (bandwidth quotas) that reflect bandwidth / link sharing for different targets at different stages. In principle, this means that a particular flow should be associated with the target group (maximum rate) that represents the most limited rate in the hierarchy. That is, for example, if target group 2270 represents a bandwidth limit that is higher than the bandwidth limit of target group 2271, and the packet is addressed by two inter-switch links represented by the two target groups, then the bandwidth limit of target group 2271 is the controlling bandwidth limiting factor.
[0262] According to an embodiment, a target group may also be shared by multiple flows. For example, depending on the QoS and SLA associated with the respective flows, the bandwidth quota represented by the target group may be divided. As an example, if both VM1 and VM2 simultaneously send flows that would involve target group 2270, which, for example, represents a bandwidth quota of 10 Gb / s, and the respective flows have equal QoS and SLA associated therewith, then target group 2270 would represent a 5 Gb / s limit for each flow. This sharing or division of the target group bandwidth quota may vary based on the QoS and SLA associated with the respective flows.
[0263] According to an embodiment, Fig.21 and Fig. 22 Target groups defined at inter-switch links and switch ports are shown, respectively. One of ordinary skill in the art will readily appreciate that target groups may be defined at different locations in a subnet, and that any given subnet is not limited to defining target groups only at ISLs and switch ports, but in general, such target groups may be defined at ISLs and switch ports within any given subnet.
[0264] Fig.23 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0265] More specifically, according to an embodiment, Fig.23 A host channel adapter 2301 is shown including a hypervisor 2311. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 2314-2316, and a physical function (PF) 2313. The host channel adapter may additionally support / include multiple ports, such as ports 2302 and 2303, which are used to connect the host channel adapter to a network, such as network 2300. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 2301 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0266] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 2350 , VM2 2351 , and VM3 2352 .
[0267] According to an embodiment, the host channel adapter 2301 may additionally support a virtual switch 2312 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0268] According to an embodiment, as shown, the network 2300 may include a plurality of switches, such as switches 2340 , 2341 , 2342 , and 2343 , which are interconnected and may be connected to the host channel adapter 2301 , for example, via leaf switches 2340 and 2341 .
[0269] According to an embodiment, switches 2340 - 2343 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0270] According to an embodiment, target groups, such as target groups 2370 and 2371, can be defined along inter-switch links (ISLs), such as the ISLs between leaf switch 2340 and switch 2342, and leaf switch 2341 and switch 2343. For example, these target groups 2370 and 2371 can represent bandwidth quotas as HCA objects stored at a target group repository 2361 associated with the HCA, which can be accessed by the rate limiting component 2360.
[0271] According to an embodiment, target groups 2370 and 2371 may represent specific (and different) bandwidth quotas. These bandwidth quotas may be divided into quotas for specific priorities ("QOS categories") to reflect the expected bandwidth usage of various priorities within the relevant paths in the architecture represented by the target group.
[0272] According to an embodiment, target groups 2370 and 2371 decouple objects from specific destination addresses, and systems and methods gain the ability to represent intermediate, shared links or link groups that can represent bandwidth limits in addition to the target and potentially more stringent than the target limit. That is, for example, if the default / original egress limit on VM2 2351 is set to a threshold, but the destination of a packet sent from VM2 is to pass through target group 2370, which sets a lower bandwidth limit, then the egress bandwidth from VM2 can be limited to a level lower than the level of the default / original egress limit set on VM2. For example, the HCA can be responsible for such throttling / egress bandwidth limit adjustments, depending on the target group involved in the routing of packets from VM2.
[0273] According to an embodiment, the target groups may also be hierarchical in nature, whereby the system and method may consider a hierarchy of target groups (bandwidth quotas) that reflect bandwidth / link sharing for different targets at different stages. In principle, this means that a particular flow should be associated with the target group (maximum rate) that represents the most limited rate in the hierarchy. That is, for example, if target group 2370 represents a bandwidth limit that is higher than the bandwidth limit of target group 2371, and the packet is addressed by two inter-switch links represented by the two target groups, then the bandwidth limit of target group 2371 is the controlling bandwidth limiting factor.
[0274] According to an embodiment, a target group may also be shared by multiple flows. For example, depending on the QoS and SLA associated with the respective flows, the bandwidth quota represented by the target group may be divided. As an example, if both VM1 and VM2 simultaneously send flows that will involve target group 2370, which, for example, represents a bandwidth quota of 10 Gb / s, and the respective flows have equal QoS and SLA associated therewith, then target group 2370 will represent a 5 Gb / s limit for each flow. This sharing or division of the target group bandwidth quota may vary based on the QoS and SLA associated with the respective flows.
[0275] According to an embodiment, the target group repository can query 2375 the target group 2370 to determine, for example, a bandwidth quota for the target group. After determining the bandwidth quota for the target group, the target group repository can store the quota value associated with the target group. The rate limiting component can then use the quota to a) determine whether the bandwidth quota for the target group is lower than the bandwidth quota for the VM based on QoS or SLA, and b) when making such a determination, update 2376 the bandwidth quota for the VM based on the path through the target group 2370.
[0276] Fig.24 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0277] More specifically, according to an embodiment, Fig.24 A host channel adapter 2401 is shown including a hypervisor 2411. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 2414-2416, and a physical function (PF) 2413. The host channel adapter may additionally support / include multiple ports, such as ports 2402 and 2403, which are used to connect the host channel adapter to a network, such as network 2400. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 2401 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0278] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 2450 , VM2 2451 , and VM3 2453 .
[0279] According to an embodiment, the host channel adapter 2401 may additionally support a virtual switch 2412 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0280] According to an embodiment, as shown, the network 2400 may include a plurality of switches, such as switches 2440 , 2441 , 2442 , and 2443 , which are interconnected and may be connected to the host channel adapter 2401 , for example, via leaf switches 2440 and 2441 .
[0281] According to an embodiment, switches 2440 - 2443 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0282] According to an embodiment, target groups, such as target groups 2470 and 2471, can be defined at, for example, switch ports. As shown, target groups 2470 and 2471 are defined at switch ports of switches 2442 and 2443, respectively. For example, these target groups 2470 and 2471 can represent bandwidth quotas as HCA objects, which are stored at a target group repository 2461 associated with the HCA, which can be accessed by the rate limiting component 2460.
[0283] According to an embodiment, target groups 2470 and 2471 may represent specific (and different) bandwidth quotas. These bandwidth quotas may be divided into quotas for specific priorities ("QOS categories") to reflect the expected bandwidth usage of various priorities within the relevant paths in the architecture represented by the target group.
[0284] According to an embodiment, target groups 2470 and 2471 decouple objects from specific destination addresses, and systems and methods gain the ability to represent intermediate, shared links or link groups that can represent bandwidth limits in addition to the target and potentially more stringent than the target limit. That is, for example, if the default / original egress limit on VM2 2451 is set to a threshold, but the destination of a packet sent from VM2 is to pass through target group 2470, which sets a lower bandwidth limit, then the egress bandwidth from VM2 can be limited to a level lower than the level of the default / original egress limit set on VM2. For example, the HCA can be responsible for such throttling / egress bandwidth limit adjustments, depending on the target group involved in the routing of packets from VM2.
[0285] According to an embodiment, the target groups may also be hierarchical in nature, whereby the systems and methods may consider a hierarchy of target groups (bandwidth quotas) that reflect bandwidth / link sharing for different targets at different stages. In principle, this means that a particular flow should be associated with the target group (maximum rate) that represents the most limited rate in the hierarchy. That is, for example, if target group 2470 represents a bandwidth limit that is higher than the bandwidth limit of target group 2471, and the packet is addressed by two inter-switch links represented by the two target groups, then the bandwidth limit of target group 2471 is the controlling bandwidth limiting factor.
[0286] According to an embodiment, a target group may also be shared by multiple flows. For example, depending on the QoS and SLA associated with the respective flows, the bandwidth quota represented by the target group may be divided. As an example, if both VM1 and VM2 simultaneously send flows that would involve target group 2470, which, for example, represents a bandwidth quota of 10 Gb / s, and the respective flows have equal QoS and SLA associated therewith, then target group 2470 would represent a 5 Gb / s limit for each flow. This sharing or division of the target group bandwidth quota may vary based on the QoS and SLA associated with the respective flows.
[0287] According to an embodiment, the target group repository can query 2475 the target group 2470 to determine, for example, a bandwidth quota for the target group. After determining the bandwidth quota for the target group, the target group repository can store the quota value associated with the target group. The rate limiting component can then use the quota to a) determine whether the bandwidth quota for the target group is lower than the bandwidth quota for the VM based on QoS or SLA, and b) when making such a determination, update 2476 the bandwidth quota for the VM based on the path through the target group 2470.
[0288] Fig.25 A system for combining multiple shared bandwidth segments in a high performance computing environment according to an embodiment is shown.
[0289] According to an embodiment, within a high-performance computing environment such as a switching network or subnet 2500, multiple end nodes 2501 and 2502 can support multiple virtual machines VM1-VM4 2550-2553, which are interconnected via multiple switches, such as leaf switches 2511 and 2512, switches 2521 and 2522, and root switches 2531 and 2532.
[0290] Not shown in the figure are various host channel adapters that provide functionality for connecting nodes 2501 and 2502 and virtual machines to be connected to the subnet, according to an embodiment. Such embodiments are discussed above with respect to SR-IOV, where each virtual machine can be associated with a virtual function of a hypervisor on a host channel adapter.
[0291] According to an embodiment, as discussed above, the concept inherent in such a switching architecture is that, while each end node or VM may have its own egress / ingress bandwidth limits that inflow and outflow traffic must comply with, there may also be links or ports within a subnet that represent bottlenecks for traffic flowing therein. Thus, when deciding at what rate traffic should flow into and out of such end nodes (such as VM1, VM2, VM3, or VM4), rate limiting components 2560 and 2561 can query various target groups, such as 2550 and 2551, to determine whether such target groups represent bottlenecks for traffic flows. Upon such a determination, rate limiting components 2560 and 2561 can then set different or new bandwidth limits on the endpoints that the rate limiting components can control.
[0292] In addition, according to an embodiment, target groups can be queried in a nested / hierarchical manner such that if traffic from VM1 to VM3 is to utilize target groups 2550 and 2551, then rate limit 2560 can consider the limits from both such target groups when determining the bandwidth limit from VM1 to VM3.
[0293] Fig.26 is a flow chart of a method for supporting target groups for congestion control in a private architecture in a high performance computing environment according to an embodiment.
[0294] According to an embodiment, at step 2610, the method may provide a first subnet at one or more microprocessors, the first subnet comprising: a plurality of switches, the plurality of switches comprising at least one leaf switch, wherein each of the plurality of switches comprises a plurality of switch ports; a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via a plurality of switches; and a plurality of end nodes, comprising a plurality of virtual machines.
[0295] According to an embodiment, at step 2620, the method may define a target group on an inter-switch link between two switches among the plurality of switches or on at least one of the ports of a switch among the plurality of switches, wherein the target group defines a bandwidth limit on the inter-switch link between two switches among the plurality of switches or on at least one of the ports of a switch among the plurality of switches.
[0296] According to an embodiment, at step 2630, the method may provide, at the host channel adapter, a target group repository stored in a memory of the host channel adapter.
[0297] According to an embodiment, at step 2640, the method may record the defined target group in a target group repository.
[0298] Combined target-specific send / RDMA write and RDMA read bandwidth limits (ORA200246-US-NP-2)
[0299] According to an embodiment, a node / VM may be the target of incoming data traffic that is the result of both Send and RDMA-Write operations initiated by peer nodes / VMs, and RDMA-Read operations initiated by the local node / VM itself. In this case, ensuring that the maximum or average ingress bandwidth of the local node / VM is within the required boundaries may become problematic unless all these flows are coordinated in terms of rate limiting.
[0300] According to an embodiment, the systems and methods described herein can implement target-specific egress rate control in a manner that allows all flows representing fetching data from local storage and sending data to associated remote destinations to be subject to the same shared rate limits and associated flow scheduling and arbitration. In addition, different flow types can be given different priorities and / or different shares of available bandwidth.
[0301] According to an embodiment, there is full control over all ingress bandwidth to a vHCA port as long as the target group association for flows from a "producer / sender" node implies bandwidth throttling for all outgoing data packets, including UD (Unreliable Datagram) sends, RDMA Writes, RDMA Sends, and RDMA Reads (i.e., RDMA Read Responses with Data). This is independent of whether the VM owning the target vHCA port is generating an "excessive" number of RDMA Read Requests to multiple peer nodes.
[0302] According to an embodiment, the coupling of target groups with flow-specific and "unslicited" BECN signaling means that each vHCA port ingress bandwidth can be dynamically throttled to any number of remote peers.
[0303] According to an embodiment, in addition to conveying pure CE marking / unmarking for different phase numbers, "active BECN" messages can also be used to convey specific rate values. In this way, it is possible to have a scenario where an initial incoming packet from a new peer (e.g., a communication management (CM) packet) can trigger the generation of one or more "active BECN" messages to the HCA (i.e., the relevant firmware / super-privileged software) from which the incoming packet came, as well as to the current communicating peer.
[0304] According to an embodiment, in the case where both ports on the HCA are used simultaneously (ie, active-active scenario), it may make sense to share target groups between local HCA ports if concurrent flows are likely to share some ISLs, or may even target the same destination ports.
[0305] According to an embodiment, another reason to share a target group between HCA ports is if the HCA local memory bandwidth cannot sustain the full link speed of both (all) HCA ports. In this case, the target group can be set so that the total aggregate link bandwidth never exceeds the local memory bandwidth, regardless of which port is involved on the source or destination HCA.
[0306] According to an embodiment, in the case of a fixed route towards a specific destination, any (one or more) intermediate target groups will typically represent only a single ISL at a specific stage in the path. However, when dynamic forwarding is active, both the target group and the ECN treatment must take this into account. In the case where dynamic forwarding decisions will only occur for balancing traffic between parallel ISLs between a pair of switches (e.g., uplinks from a single leaf switch to a single spine switch), then all treatments are in principle very similar to the treatment when only a single ISL is used. FECN notifications will occur based on the state of all ports in the relevant group, and signaling can be "aggressive" in the sense that it is signaled based on congestion indications from any port, or it can be more conservative and based on the size of the shared output queue of all ports in the group. The target group configuration typically represents the aggregate bandwidth of all links in the group, as long as forwarding allows any packet to select the best output port at that point in time. However, if each flow has a strict concept of packet order preservation, then the evaluation of bandwidth quotas will be more complicated because some flows may "have to" use the same ISL at a certain point in time. If such a flow ordering scheme is based on well-defined header fields, then it is best to represent each port in the group as an independent target group. In this case, the selection of the target group by the sender-side HCA must be able to perform the same evaluation of the header fields that will be associated with the RC QP connection or address handle as the switch will perform for each packet at runtime.
[0307] According to an embodiment, the initial target group rate for new remote targets may be conservatively set low by default. In this way, there is inherent throttling before the targets have a chance to update the associated rates. Thus, all such rate controls are independent of the VMs themselves involved, but the VMs will be able to request that the hypervisor update quotas for ingress and egress traffic for different remote peers, but this can only be granted within the total constraints defined for the local and remote vHCA ports.
[0308] Fig. 27 A system for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment is shown in accordance with an embodiment.
[0309] More specifically, according to an embodiment, Fig. 27A host channel adapter 2701 is shown including a hypervisor 2711. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 2714-2716, and a physical function (PF) 2713. The host channel adapter may additionally support / include multiple ports, such as ports 2702 and 2703, which are used to connect the host channel adapter to a network, such as network 2700. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 2701 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0310] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 2750 , VM2 2751 , and VM3 2752 .
[0311] According to an embodiment, the host channel adapter 2701 may additionally support a virtual switch 2712 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0312] According to an embodiment, as shown, the network 2700 may include a plurality of switches, such as switches 2740 , 2741 , 2742 , and 2743 , which are interconnected and may be connected to the host channel adapter 2701 , for example, via leaf switches 2740 and 2741 .
[0313] According to an embodiment, switches 2740 - 2743 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0314] According to an embodiment, target groups, such as target groups 2770 and 2771, may be defined at inter-switch links (ISLs), such as the ISLs between leaf switch 2740 and switch 2742, and leaf switch 2741 and switch 2743. For example, these target groups 2770 and 2771 may represent bandwidth quotas as HCA objects stored at a target group repository 2761 associated with the HCA, which may be accessed by the rate limiting component 2760.
[0315] According to an embodiment, target groups 2770 and 2771 may represent specific (and different) bandwidth quotas. These bandwidth quotas may be divided into quotas for specific priorities ("QOS categories") to reflect the expected bandwidth usage of various priorities within the relevant paths in the architecture represented by the target group.
[0316] According to an embodiment, target groups 2770 and 2771 decouple objects from specific destination addresses, and systems and methods gain the ability to represent intermediate, shared links or link groups that can represent bandwidth limits in addition to the target and potentially more stringent than the target limit. That is, for example, if the default / original egress limit on VM2 2751 is set to a threshold, but the destination of a packet sent from VM2 is to pass through target group 2770, which sets a lower bandwidth limit, then the egress bandwidth from VM2 can be limited to a level lower than the level of the default / original egress limit set on VM2. For example, the HCA can be responsible for such throttling / egress bandwidth limit adjustments, depending on the target group involved in the routing of packets from VM2.
[0317] According to an embodiment, the target groups may also be hierarchical in nature, whereby the system and method may consider a hierarchy of target groups (bandwidth quotas) that reflect bandwidth / link sharing for different targets at different stages. In principle, this means that a particular flow should be associated with the target group (maximum rate) that represents the most limited rate in the hierarchy. That is, for example, if target group 2770 represents a bandwidth limit that is higher than the bandwidth limit of target group 2771, and the packet is addressed by two inter-switch links represented by the two target groups, then the bandwidth limit of target group 2771 is the controlling bandwidth limiting factor.
[0318] According to an embodiment, a target group may also be shared by multiple flows. For example, depending on the QoS and SLA associated with the respective flows, the bandwidth quota represented by the target group may be divided. As an example, if both VM1 and VM2 simultaneously send flows that would involve target group 2770, which, for example, represents a bandwidth quota of 10 Gb / s, and the respective flows have equal QoS and SLA associated therewith, then target group 2770 would represent a 5 Gb / s limit for each flow. This sharing or division of the target group bandwidth quota may vary based on the QoS and SLA associated with the respective flows.
[0319] According to an embodiment, bandwidth quota and performance issues may arise when a VM (e.g., VM1 2750) is subjected to excessive ingress bandwidth 2790 from multiple sources. For example, this may occur in a situation where VM1 is subjected to one or more RDMA read responses while being subjected to one or more RDMA write operations, where the ingress bandwidth on VM1 comes from two or more sources (e.g., one RDMA read response is from a connected VM and one RDMA write request is from another connected VM). In this case, for example, a target group, such as a target group 2770 on an inter-switch link, may be updated to reflect a lower bandwidth quota than would normally be allowed via, for example, query 2775.
[0320] In addition, according to an embodiment, the rate limiting component 2760 of the HCA may additionally include a VM-specific rate limit 2762, which may negotiate with other peer HCAs to coordinate, for example, the ingress bandwidth limit of VM1, and the egress bandwidth limit of the node responsible for generating the ingress bandwidth on VM1. These other HCAs / nodes are not shown in this figure.
[0321] Fig.28 A system for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment is shown in accordance with an embodiment.
[0322] More specifically, according to an embodiment, Fig.28 A host channel adapter 2801 is shown including a hypervisor 2811. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 2814-2816, and a physical function (PF) 2813. The host channel adapter may additionally support / include multiple ports, such as ports 2802 and 2803, which are used to connect the host channel adapter to a network, such as network 2800. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 2801 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0323] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 2850 , VM2 2851 , and VM3 2852 .
[0324] According to an embodiment, the host channel adapter 2801 may additionally support a virtual switch 2812 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0325] According to an embodiment, as shown, the network 2800 may include a plurality of switches, such as switches 2840 , 2841 , 2842 , and 2843 , which are interconnected and may be connected to the host channel adapter 2801 , for example, via leaf switches 2840 and 2841 .
[0326] According to an embodiment, switches 2840 - 2843 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0327] According to an embodiment, target groups, such as target groups 2870 and 2871, can be defined at, for example, switch ports. As shown, target groups 2870 and 2871 are defined at switch ports of switches 2842 and 2843, respectively. For example, these target groups 2870 and 2871 can represent bandwidth quotas as HCA objects, which are stored at a target group repository 2861 associated with the HCA, which can be accessed by the rate limiting component 2860.
[0328] According to an embodiment, target groups 2870 and 2871 may represent specific (and different) bandwidth quotas. These bandwidth quotas may be divided into quotas for specific priorities ("QOS categories") to reflect the expected bandwidth usage of various priorities within the relevant paths in the architecture represented by the target group.
[0329] According to an embodiment, target groups 2870 and 2871 decouple objects from specific destination addresses, and systems and methods gain the ability to represent intermediate, shared links or link groups that can represent bandwidth limits in addition to the target and potentially more stringent than the target limit. That is, for example, if the default / original egress limit on VM2 2851 is set to a threshold, but the destination of a packet sent from VM2 is to pass through target group 2870, which sets a lower bandwidth limit, then the egress bandwidth from VM2 can be limited to a level lower than the level of the default / original egress limit set on VM2. For example, the HCA can be responsible for such throttling / egress bandwidth limit adjustments, depending on the target group involved in the routing of packets from VM2.
[0330] According to an embodiment, the target groups may also be hierarchical in nature, whereby the system and method may consider a hierarchy of target groups (bandwidth quotas) that reflect bandwidth / link sharing for different targets at different stages. In principle, this means that a particular flow should be associated with the target group (maximum rate) that represents the most limited rate in the hierarchy. That is, for example, if target group 2870 represents a bandwidth limit that is higher than the bandwidth limit of target group 2871, and the packet is addressed by two inter-switch links represented by the two target groups, then the bandwidth limit of target group 2871 is the controlling bandwidth limiting factor.
[0331] According to an embodiment, a target group may also be shared by multiple flows. For example, depending on the QoS and SLA associated with the respective flows, the bandwidth quota represented by the target group may be divided. As an example, if both VM1 and VM2 simultaneously send flows that will involve target group 2870, which, for example, represents a bandwidth quota of 10 Gb / s, and the respective flows have equal QoS and SLA associated therewith, then target group 2870 will represent a 5 Gb / s limit for each flow. This sharing or division of the target group bandwidth quota may vary based on the QoS and SLA associated with the respective flows.
[0332] According to an embodiment, bandwidth quota and performance issues may arise when a VM (e.g., VM1 2850) is subjected to excessive ingress bandwidth 2890 from multiple sources. For example, this may occur in a situation where VM1 is subjected to one or more RDMA read responses while being subjected to one or more RDMA write operations, where the ingress bandwidth on VM1 comes from two or more sources (e.g., one RDMA read response is from a connected VM and one RDMA write request is from another connected VM). In this case, for example, a target group, such as a target group 2870 on an inter-switch link, may be updated to reflect a lower bandwidth quota than would normally be allowed via, for example, query 2875.
[0333] In addition, according to an embodiment, the rate limiting component 2860 of the HCA may additionally include a VM-specific rate limit 2862, which may negotiate with other peer HCAs to coordinate, for example, the ingress bandwidth limit of VM1, and the egress bandwidth limit of the node responsible for generating the ingress bandwidth on VM1. These other HCAs / nodes are not shown in this figure.
[0334] Fig.29 A system for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment is shown in accordance with an embodiment.
[0335] More specifically, according to an embodiment, Fig.29 A host channel adapter 2901 is shown including a hypervisor 2911. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 2914-2916, and a physical function (PF) 2913. The host channel adapter may additionally support / include multiple ports, such as ports 2902 and 2903, which are used to connect the host channel adapter to a network, such as network 2900. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 2901 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0336] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 2950 , VM2 2951 , and VM3 2952 .
[0337] According to an embodiment, the host channel adapter 2901 may additionally support a virtual switch 2912 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0338] According to an embodiment, as shown, the network 2900 may include a plurality of switches, such as switches 2940 , 2941 , 2942 , and 2943 , which are interconnected and may be connected to the host channel adapter 2901 , for example, via leaf switches 2940 and 2941 .
[0339] According to an embodiment, switches 2940 - 2943 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0340] According to an embodiment, target groups, such as target group 2971, can be defined along an inter-switch link (ISL), such as an ISL between a leaf switch 2941 and a switch 2943. Other target groups can be defined, for example, at a switch port. As shown, target group 2970 is defined at a switch port of switch 2952. For example, these target groups 2970 and 2971 can represent bandwidth quotas as HCA objects, which are stored at a target group repository 2961 associated with the HCA, which target group repository 2961 can be accessed by the rate limiting component 2960.
[0341] According to an embodiment, target groups 2970 and 2971 may represent specific (and different) bandwidth quotas. These bandwidth quotas may be divided into quotas for specific priorities ("QOS categories") to reflect the expected bandwidth usage of various priorities within the relevant paths in the architecture represented by the target group.
[0342] According to an embodiment, target groups 2970 and 2971 decouple objects from specific destination addresses, and systems and methods gain the ability to represent intermediate, shared links or link groups that can represent bandwidth limits in addition to the target and potentially more stringent than the target limit. That is, for example, if the default / original egress limit on VM2 2951 is set to a threshold, but the destination of a packet sent from VM2 is to pass through target group 2970, which sets a lower bandwidth limit, then the egress bandwidth from VM2 can be limited to a level lower than the level of the default / original egress limit set on VM2. For example, the HCA can be responsible for such throttling / egress bandwidth limit adjustments, depending on the target group involved in the routing of packets from VM2.
[0343] According to an embodiment, the target groups may also be hierarchical in nature, whereby the system and method may consider a hierarchy of target groups (bandwidth quotas) that reflect bandwidth / link sharing for different targets at different stages. In principle, this means that a particular flow should be associated with the target group (maximum rate) that represents the most limited rate in the hierarchy. That is, for example, if target group 2970 represents a bandwidth limit that is higher than the bandwidth limit of target group 2971, and the packet is addressed by two inter-switch links represented by the two target groups, then the bandwidth limit of target group 2971 is the controlling bandwidth limiting factor.
[0344] According to an embodiment, a target group may also be shared by multiple flows. For example, depending on the QoS and SLA associated with the respective flows, the bandwidth quota represented by the target group may be divided. As an example, if both VM1 and VM2 simultaneously send flows that will involve target group 2970, which, for example, represents a bandwidth quota of 10 Gb / s, and the respective flows have equal QoS and SLA associated therewith, then target group 2970 will represent a 5 Gb / s limit for each flow. This sharing or division of the target group bandwidth quota may vary based on the QoS and SLA associated with the respective flows.
[0345] According to an embodiment, bandwidth quota and performance issues may arise when a VM (e.g., VM1 2950) is subjected to excessive ingress bandwidth 2990 from multiple sources. For example, this may occur in a situation where VM1 is subjected to one or more RDMA read responses while being subjected to one or more RDMA write operations, where the ingress bandwidth on VM1 comes from two or more sources (e.g., one RDMA read response is from a connected VM and one RDMA write request is from another connected VM). In this case, for example, a target group, such as a target group 2970 on an inter-switch link, may be updated to reflect a lower bandwidth quota than would normally be allowed via, for example, query 2975.
[0346] In addition, according to an embodiment, the rate limiting component 2960 of the HCA may additionally include a VM-specific rate limit 2962, which may negotiate with other peer HCAs to coordinate, for example, the ingress bandwidth limit of VM1, and the egress bandwidth limit of the node responsible for generating the ingress bandwidth on VM1. These other HCAs / nodes are not shown in this figure.
[0347] Fig.30 A system for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment is shown in accordance with an embodiment.
[0348] According to an embodiment, within a high-performance computing environment such as a switching network or subnet 3000, multiple end nodes 3001 and 3002 can support multiple virtual machines VM1-VM4 3050-3053, which are interconnected via multiple switches, such as leaf switches 3011 and 3012, switches 3021 and 3022, and root switches 3031 and 3032.
[0349] Not shown in the figure are various host channel adapters that provide functionality for connecting nodes 3001 and 3002 and virtual machines to be connected to the subnet, according to an embodiment. Such embodiments are discussed above with respect to SR-IOV, where each virtual machine can be associated with a virtual function of a hypervisor on a host channel adapter.
[0350] According to an embodiment, when a node (such as VM3 3052) processes RDMA read responses 3050 and RDMA write requests 3051 simultaneously (bandwidth on ingress), it may encounter bandwidth limitations (e.g., from rate limit 3061).
[0351] According to an embodiment, rate limits 3060 and 3061 may be configured to ensure that the ingress bandwidth quota is not violated, for example, by coordinating RDMA requests (i.e., a message requesting an RDMA read sent by VM3 to VM4, resulting in an RDMA read response 3050), and RDMA write operations (e.g., an RDMA write from VM2 to VM3).
[0352] For each individual node, systems and methods may have a target group chain such that a flow will always be coordinated with all other flows sharing link bandwidth in different parts of the fabric represented in the target group.
[0353] Fig.31 is a flow chart of a method for combining target-specific RDMA-write and RDMA-read bandwidth limits in a high performance computing environment, according to an embodiment.
[0354] According to an embodiment, at step 3110, the method may provide a first subnet at one or more microprocessors, the first subnet comprising: a plurality of switches, the plurality of switches comprising at least a leaf switch, wherein each of the plurality of switches comprises a plurality of switch ports; a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via a plurality of switches; and a plurality of end nodes, comprising a plurality of virtual machines.
[0355] According to an embodiment, at step 3120, the method may define a target group on an inter-switch link between two switches among the plurality of switches or on at least one of the ports of a switch among the plurality of switches, wherein the target group defines a bandwidth limit on the inter-switch link between two switches among the plurality of switches or on at least one of the ports of a switch among the plurality of switches.
[0356] According to an embodiment, at step 3130, the method may provide, at the host channel adapter, a target group repository stored in a memory of the host channel adapter.
[0357] According to an embodiment, at step 3140, the method may record the defined target group in a target group repository.
[0358] According to an embodiment, at step 3150, the method may receive, at an end node of a host channel adapter, ingress bandwidth from at least two remote sources, the ingress bandwidth exceeding an ingress bandwidth limit of the end node.
[0359] According to an embodiment, at 3160, in response to receiving ingress bandwidth from at least two sources, the method may update the bandwidth quota of the target group.
[0360] Combined ingress bandwidth arbitration and congestion feedback (ORA200246-US-NP-2)
[0361] According to an embodiment, when multiple sender nodes / VMs each and / or all send to a single receiver node / VM, it is not straightforward to achieve fairness between senders, avoid congestion, and at the same time limit the ingress bandwidth usage consumed by the receiver node / VM to (well) below the maximum limit of the maximum physical link bandwidth that the relevant network interface can provide for ingress traffic. In addition, when different bandwidth quotas should be allocated to different senders due to different SLA levels, then the equation becomes even more complicated.
[0362] According to embodiments, the systems and methods herein can extend traditional schemes for end-to-end congestion feedback to include initial negotiation of bandwidth quotas, dynamic adjustment of such bandwidth quotas (e.g., to accommodate changes in the number of sender nodes sharing the available bandwidth, or changes in SLAs), and dynamic congestion feedback to indicate that senders need to temporarily reduce the associated egress data rate, although the overall bandwidth quota remains unchanged. Relevant information is conveyed from a destination node to a sender node using explicit, proactive messages as well as "piggyback" information in data packets.
[0363] Fig.32 A system for combining ingress bandwidth arbitration and congestion feedback in a high performance computing environment according to an embodiment is shown.
[0364] More specifically, according to an embodiment, Fig.32 A host channel adapter 3201 is shown including a hypervisor 3211. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 3214-3216, and a physical function (PF) 3213. The host channel adapter may additionally support / include multiple ports, such as ports 3202 and 3203, which are used to connect the host channel adapter to a network, such as network 3200. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 3201 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0365] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 3250 , VM2 3251 , and VM3 3252 .
[0366] According to an embodiment, the host channel adapter 3201 may additionally support a virtual switch 3212 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0367] According to an embodiment, as shown, the network 3200 may include a plurality of switches, such as switches 3240 , 3241 , 3242 , and 3243 , which are interconnected and may be connected to the host channel adapter 3201 , for example, via leaf switches 3240 and 3241 .
[0368] According to an embodiment, switches 3240 - 3243 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0369] According to an embodiment, bandwidth quota and performance issues may arise when a VM (e.g., VM1 3250) is subjected to excessive ingress bandwidth 3290 from multiple sources. For example, this may occur when VM1 is subjected to one or more RDMA read responses while being subjected to one or more RDMA write operations, where the ingress bandwidth on VM1 comes from two or more sources (e.g., one RDMA read response from a connected VM and one RDMA write request from another connected VM).
[0370] According to an embodiment, the rate limiting component 3260 of the HCA may additionally include a VM-specific rate limit 3261, which may negotiate with other peer HCAs to coordinate, for example, the ingress bandwidth limit of VM1, and the egress bandwidth limit of the node responsible for generating the ingress bandwidth on VM1. For example, such initial negotiation may be performed to accommodate changes in the number of sender nodes sharing the available bandwidth, or changes in the SLA. These other HCAs / nodes are not shown in the figure.
[0371] According to an embodiment, the above negotiation may be updated based on, for example, explicit and unsolicited feedback messages 3291 generated as a result of the ingress bandwidth. For example, such feedback messages 3291 may be sent to multiple remote nodes responsible for generating the ingress bandwidth 3290 on VM1. Upon receiving such feedback messages, the sender nodes (bandwidth senders responsible for the ingress bandwidth on VM1) may update their associated egress bandwidth limits on the sender nodes so as not to overload, for example, the link connected to VM1, while still attempting to maintain QoS and SLA.
[0372] Fig.33 A system for combining ingress bandwidth arbitration and congestion feedback in a high performance computing environment according to an embodiment is shown.
[0373] More specifically, according to an embodiment, Fig.33 A host channel adapter 3301 is shown including a hypervisor 3311. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 3314-3316, and a physical function (PF) 3313. The host channel adapter may additionally support / include multiple ports, such as ports 3302 and 3303, which are used to connect the host channel adapter to a network, such as network 3300. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect HCA 3301 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0374] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 3350 , VM2 3351 , and VM3 3352 .
[0375] According to an embodiment, the host channel adapter 3301 may additionally support a virtual switch 3312 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0376] According to an embodiment, as shown, the network 3300 may include a plurality of switches, such as switches 3340 , 3341 , 3342 , and 3343 , which are interconnected and may be connected to the host channel adapter 3301 , for example, via leaf switches 3340 and 3341 .
[0377] According to an embodiment, switches 3340 - 3343 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0378] According to an embodiment, bandwidth quota and performance issues may arise when a VM (e.g., VM1 3350) is subjected to excessive ingress bandwidth 3390 from multiple sources. For example, this may occur when VM1 is subjected to one or more RDMA read responses while being subjected to one or more RDMA write operations, where the ingress bandwidth on VM1 comes from two or more sources (e.g., one RDMA read response from a connected VM and one RDMA write request from another connected VM).
[0379] According to an embodiment, the rate limiting component 3360 of the HCA may additionally include a VM-specific rate limit 3361, which may negotiate with other peer HCAs to coordinate, for example, the ingress bandwidth limit of VM1, and the egress bandwidth limit of the node responsible for generating the ingress bandwidth on VM1. For example, such initial negotiation may be performed to accommodate changes in the number of sender nodes sharing the available bandwidth, or changes in the SLA. These other HCAs / nodes are not shown in the figure.
[0380] According to an embodiment, the above negotiation may be updated based on, for example, piggybacked messages 3391 (messages residing on top of regular data or other communication packets sent between end nodes) generated as a result of the ingress bandwidth. For example, such piggybacked messages 3391 may be sent to multiple remote nodes responsible for generating the ingress bandwidth 3390 on VM1. Upon receiving such feedback messages, the sender nodes (bandwidth senders responsible for the ingress bandwidth on VM1) may update their associated egress bandwidth limits on the sender nodes so as not to overload, for example, the link connected to VM1, while still attempting to maintain QoS and SLA.
[0381] Fig.34A system for combining ingress bandwidth arbitration and congestion feedback in a high performance computing environment according to an embodiment is shown.
[0382] According to an embodiment, within a high-performance computing environment such as a switching network or subnet 3400, multiple end nodes 3401 and 3402 can support multiple virtual machines VM1-VM4 3450-3453, which are interconnected via multiple switches, such as leaf switches 3411 and 3412, switches 3421 and 3422, and root switches 3431 and 3432.
[0383] Not shown in the figure are various host channel adapters that provide functionality for connecting nodes 3401 and 3402 and virtual machines to be connected to the subnet, according to an embodiment. Such embodiments are discussed above with respect to SR-IOV, where each virtual machine can be associated with a virtual function of a hypervisor on a host channel adapter.
[0384] According to an embodiment, when a node (such as VM3 3452) receives, for example, multiple RDMA ingress bandwidth packets (e.g., multiple RDMA writes), such as 3451 and 3452, it may encounter ingress bandwidth limitations (e.g., from rate limit 3461). This situation may result, for example, when there is no communication between the various sender nodes to coordinate bandwidth limitations.
[0385] According to an embodiment, the systems and methods herein may extend the scheme of end-to-end congestion feedback to include initial negotiation of bandwidth quotas (i.e., negotiation of VM3, or bandwidth limits associated with VM3, with all sender nodes targeting VM3 within ingress traffic), dynamic adjustment of such bandwidth quotas (e.g., to accommodate changes in the number of sender nodes sharing available bandwidth, or changes in SLAs), and dynamic congestion feedback to indicate that senders need to temporarily reduce the associated egress data rate, although the overall bandwidth quota remains unchanged. For example, such dynamic congestion feedback may occur in a return message (e.g., feedback message 3470) to each sender node, indicating to each sender node the updated bandwidth limit to utilize when sending traffic to VM3. Such feedback messages 3460 may take the form of explicit, proactive messages as well as "piggyback" information in data packets to convey relevant information from the target node (i.e., VM3 in the depicted embodiment) to the sender node.
[0386] Fig.35 is a flow chart of a method of combining ingress bandwidth arbitration and congestion feedback in a high performance computing environment according to an embodiment.
[0387] According to an embodiment, at step 3510, the method may provide a first subnet at one or more microprocessors, the first subnet comprising: a plurality of switches, the plurality of switches comprising at least a leaf switch, wherein each of the plurality of switches comprises a plurality of switch ports; a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via a plurality of switches; and a plurality of end nodes, comprising a plurality of virtual machines.
[0388] According to an embodiment, at step 3520, the method may provide, at a host channel adapter, an end node ingress bandwidth quota associated with an end node attached to the host channel adapter.
[0389] According to an embodiment, at step 3530, the method may negotiate a bandwidth quota between an end node attached to a host channel adapter and a remote end node.
[0390] According to an embodiment, at step 3540, the method may receive, at an end node attached to a host channel adapter, an ingress bandwidth from a remote source that exceeds an ingress bandwidth limit of the end node.
[0391] According to an embodiment, at 3550, in response to receiving ingress bandwidth from at least two sources, the method may send a response message from the end node attached to the host channel adapter to the remote end node, the response message indicating that the ingress bandwidth quota of the end node attached to the host channel adapter is exceeded.
[0392] Use multiple CEs (experienced) in FECN (Forward Explicit Congestion Notification) and BECN (Backward Explicit Congestion Notification) signaling Congestion) flag (ORA200246-US-NP-4)
[0393] According to an embodiment, congestion notification is conventionally based on data packets encountering congestion at a certain point (e.g., a certain link segment between a certain node / switch pair along the path from a sender to a destination through a network / fabric topology) being marked with a "congested" status flag (also known as a CE flag), and this status is then reflected in a response packet sent back from the destination to the sender.
[0394] According to an embodiment, a problem with this scheme is that it does not allow a sender node to distinguish flows that are experiencing congestion at the same link segment, although they represent different destinations. Furthermore, when multiple paths are available between a sender and destination node pair, any information about the congestion of different alternative paths requires that a certain flow is active for the relevant destination via the relevant path.
[0395] According to an embodiment, the systems and methods described herein extend the congestion marking scheme to facilitate multiple CE markings in the same packet and configure the switch port to represent a phase number that defines what CE marking index it should update. Between a specific sender and a specific target, a specific path through an ordered sequence of switch ports will then represent a specific ordered list of unique phase numbers, thereby also representing a CE marking index number.
[0396] According to an embodiment, in this manner, a sender node receiving congestion feedback with multiple CE flags set can map the various CE flags to different "target group" contexts, which will then represent the relevant congestion condition status and the associated dynamic rate reduction. In addition, different flows for different targets will share congestion information and dynamic rate reduction status associated with the shared link segments represented by the shared "target group" in the sender node.
[0397] According to an embodiment, when congestion does occur, a key issue is that congestion feedback should ideally be associated with all relevant target groups in the hierarchy associated with the flow receiving the congestion feedback. The affected target groups should then dynamically adjust their maximum rates accordingly. Therefore, the hardware state of each target group must also include any current congestion state and related "throttling information".
[0398] According to an embodiment, the important aspect here is that the FECN signaling should have the ability to include multiple "congestion experienced" (CE) flags so that a switch that detects congestion can mark the flag corresponding to its stage in the topology. In a conventional fat tree, each switch has a unique (maximum) stage number in the upward direction and another unique (maximum) stage number in the downward direction. Therefore, a flow using a particular path will then be associated with a sequence of specific stage numbers that will include all or only a subset of the total set of stage numbers in the entire architecture. However, for a particular flow, the individual stage numbers associated with the path can then be mapped to one or more target groups associated with the flow. In this way, the BECN of a received flow can imply that the (one or more) target groups associated with each CE-marked stage in the BECN will be updated to indicate congestion, and the dynamic maximum rates of these target groups can then be adjusted accordingly.
[0399] According to an embodiment, while inherently applicable to fat-tree topologies, the concept of "phase numbers" for switches can be generalized to represent nearly any topology in which such numbers can be assigned to switches. However, in this general case, the phase numbers are not just a function of the output ports, but rather a function of each input / output port number tuple. The number of phase numbers required and the path-specific mappings to target groups are also more complex in the general case. Therefore, in this context, the reasoning assumes only a fat-tree topology.
[0400] According to an embodiment, multiple CE markers in a single packet is not currently a feature supported by the standard protocol header. Therefore, this can be supported based on an extension of the standard header, and / or it can be supported by inserting additional independent FECN packets in the flow - conceptually, the generation of additional packets in the flow is similar to using an encapsulation scheme within a switch, and the impact is that packets received at line rate cannot be forwarded at the same line rate because more "overhead bytes" must be transmitted downstream. Inserting additional packets will generally be more overhead than encapsulation, but as long as this overhead is amortized over multiple data packets (no need to send such additional notifications for each data packet), the overhead may be acceptable.
[0401] According to an embodiment, there may also be a scheme where the switch firmware can monitor congestion conditions within the switch and therefore send "active BECN" to the relevant sender nodes. However, this means that the switch firmware must have more state information about the relevant senders, as well as a mapping between ports and priorities and relevant senders, which may also include dynamic information about which addresses are involved in packets experiencing congestion.
[0402] According to an embodiment, for RC QP, the "CE flag to target group" mapping is typically part of the QP context, and any BECN information received in the ACK / response packet will thus be handled in a straightforward manner for the relevant QP context and the associated target group. However, in the case of "active BECNs" (e.g., due to datagram traffic having only application level responses / ACKs, or due to "congestion warnings" being broadcast to multiple potential senders), the reverse mapping is not straightforward - at least not in terms of automatic handling by hardware. Therefore, a better approach is to have a scheme where both FECNs can result in automatic hardware-generated BECNs in the case of connected (RC) flows, but both FECN events with hardware automatic BECN generation and FECN events without hardware-generated BECNs can be handled by firmware and / or super-privileged software associated with the HCA that receives the FECNs. In this way, there can be "active BECNs" generated by the FW / SW that are sent to one or more potential senders affected by the observed congestion. The FW / SW that receives these "active BECNs" can then perform a mapping to the relevant local target group based on the payload data in the received "BECN message", and can then trigger the local hardware to update the target group state, similar to what happens in the fully hardware-controlled handling of RC-related BECNs.
[0403] According to an embodiment, an RC ACK / response packet without any BECN notification or a subset of phase numbers with the CE flag set that is different (less than) the earlier recorded state may result in a corresponding update of the relevant target group within the local HCA. Similarly, the responder HCA (i.e., the relevant software / firmware) may send an "active BECN" to indicate that the congestion that was signaled earlier no longer exists.
[0404] According to an embodiment, as described above, the target group concept combined with dynamic congestion feedback at the hardware or firmware / software level provides flexible control of the egress bandwidth generated by the HCA and by individual vHCAs and tenants sharing a physical HCA.
[0405] According to an embodiment, since target groups are identified at the VM level completely independently of the associated remote address and path information, there is no dependency between the use of target groups and the extent to which communications from the VM are based on overlays or other virtual networking schemes. The only requirement is that the super-privileged software controlling the HCA resources is able to define the relevant mappings. In addition, a scheme with a "logical target group ID" at the VM / vHCA level can be used, which is then mapped to the actual target group by the HCA. However, it is unclear whether this is useful, except that it hides the actual target group ID from the tenant. In the case where it is necessary to change which target groups are associated with a particular destination because the underlying path has changed, then this may not involve other destinations. Therefore, in general, updating the target group must involve updating all involved QPs and address handles, rather than just updating the mapping of logical to physical target group IDs.
[0406] According to an embodiment, for a virtualized target HCA, individual vHCA ports rather than physical HCA ports may be represented as final destination target groups. In this manner, a target group hierarchy of a remote peer node may include a target group representing a destination physical HCA port and an additional target group representing a final destination based on a vHCA port. In this manner, the system and method have the ability to limit the ingress bandwidth of individual vHCA ports (VFs), while each physical HCA port bandwidth and associated sender target group means that the sum of each vHCA port ingress bandwidth quota will not have to remain less than the physical HCA port bandwidth (or associated bandwidth quota).
[0407] According to an embodiment, within a sender HCA, target groups may be used to represent sharing of physical HCA ports in the egress direction by assigning different target groups to different tenants. Furthermore, to facilitate tenant-level target groups for multiple VMs from the same tenant to share physical HCA ports, different target groups may be assigned to different such VMs. Such target groups are then set as the initial target groups for all egress communications from that VM.
[0408] Fig.36 A system for using multiple CE markings in both FECN and BECN in a high performance computing environment according to an embodiment is shown.
[0409] More specifically, according to an embodiment, Fig.36 A host channel adapter 3601 is shown including a hypervisor 3611. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 3614-3616, and a physical function (PF) 3613. The host channel adapter may additionally support / include multiple ports, such as ports 3602 and 3603, which are used to connect the host channel adapter to a network, such as network 3600. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect HCA 3601 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0410] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 3650 , VM2 3651 , and VM3 3652 .
[0411] According to an embodiment, the host channel adapter 3601 may additionally support a virtual switch 3612 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0412] According to an embodiment, as shown, the network 3600 may include a plurality of switches, such as switches 3640 , 3641 , 3642 , and 3643 , which are interconnected and may be connected to the host channel adapter 3601 , for example, via leaf switches 3640 and 3641 .
[0413] According to an embodiment, switches 3640 - 3643 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0414] According to an embodiment, the ingress packet 3690 may experience congestion at any stage of its path while traversing the network, and when such congestion is detected at any stage, the switch may mark the packet. In addition to marking the packet as having experienced congestion, the switch that performs the marking may additionally indicate the stage at which the packet experienced congestion. Upon arrival at the destination node, e.g., VM1 3650, VM1 may send a response packet (e.g., automatically) via an explicit feedback message 3691, which may indicate to the sending node that the packet experienced congestion, and at which stage(s) the packet experienced congestion.
[0415] According to an embodiment, the ingress packet may include a bit field that is updated to indicate where the packet experienced congestion, and the explicit feedback message may mirror / represent this bit field in notifying the sender node of such congestion.
[0416] According to an embodiment, each switch port represents a stage in the entire subnet. Thus, each packet sent in the subnet can pass through a maximum number of stages. In order to identify the location where congestion is detected (which may be in multiple locations), the congestion marker (e.g., CE flag) is extended from a simple binary flag (experienced congestion) to a bit field containing multiple bits. Each bit of the bit field can then be associated with a stage number, which can be assigned to each switch port. For example, in a three-stage fat tree, the maximum number of stages is three. When the system has a path from A to B and the route is known, then each end node can determine which switch port the packet passes through at any given stage of the path. By doing so, each end node can determine at which different switch ports the packet experienced congestion by correlating the route with the received congestion message.
[0417] According to an embodiment, the system can provide congestion feedback back that indicates which congestion was detected at which stage, and then if the end node has congestion due to a shared link segment, then congestion control is applied to that segment instead of to different end ports. This provides more granular information about congestion.
[0418] According to an embodiment, by providing this finer granularity, the end node can then use an alternate path when routing future packets. Or, for example, if the end node has multiple flows all going to different destinations, but congestion is detected at a common stage in the path, then a re-route can be triggered. The system and method provide an immediate reaction in terms of throttling associated with - rather than having 10 different congestion notifications. This can handle congestion notifications more efficiently.
[0419] Fig.37 A system for using multiple CE markings in both FECN and BECN in a high performance computing environment according to an embodiment is shown.
[0420] More specifically, according to an embodiment, Fig.37A host channel adapter 3701 is shown including a hypervisor 3711. The hypervisor may host / associate multiple virtual functions (VFs), such as VFs 3714-3716, and a physical function (PF) 3713. The host channel adapter may additionally support / include multiple ports, such as ports 3702 and 3703, which are used to connect the host channel adapter to a network, such as network 3700. The network may include, for example, a switched network, such as an InfiniBand network or a RoCE network, which may connect the HCA 3701 to multiple other nodes, such as switches, additional and separate HCAs, and the like.
[0421] According to an embodiment, as described above, each virtual function may host a virtual machine (VM), such as VM1 3750 , VM2 3751 , and VM3 3752 .
[0422] According to an embodiment, the host channel adapter 3701 may additionally support a virtual switch 3712 via a hypervisor. This is the case for implementing a vSwitch architecture. Although not shown, embodiments of the present disclosure may additionally support a virtual port (vPort) architecture, as described above.
[0423] According to an embodiment, as shown, the network 3700 may include a plurality of switches, such as switches 3740 , 3741 , 3742 , and 3743 , which are interconnected and may be connected to the host channel adapter 3701 , for example, via leaf switches 3740 and 3741 .
[0424] According to an embodiment, switches 3740-3743 may be interconnected and may additionally be connected to other switches and other end nodes (eg, other HCAs) not shown in the figure.
[0425] According to an embodiment, an ingress packet 3790 may experience congestion at any stage of its path while traversing the network, and when such congestion is detected at any stage, the switch may mark the packet. In addition to marking the packet as having experienced congestion, the switch that performs the marking may additionally indicate the stage at which the packet experienced congestion. Upon arrival at a destination node, such as VM1 3750, VM1 may send a response packet (e.g., automatically) via a piggyback message (a message residing on top of another message / packet sent from a receiving node to a sender node) 3791 that may indicate to the sending node that the packet experienced congestion, and at which stage(s) the packet experienced congestion.
[0426] According to an embodiment, the ingress packet may include a bit field that is updated to indicate where the packet experienced congestion, and the explicit feedback message may mirror / represent this bit field in notifying the sender node of such congestion.
[0427] According to an embodiment, each switch port represents a stage in the entire subnet. Thus, each packet sent in the subnet can pass through a maximum number of stages. In order to identify the location where congestion is detected (which may be in multiple locations), the congestion marker (e.g., CE flag) is extended from a simple binary flag (experienced congestion) to a bit field containing multiple bits. Each bit of the bit field can then be associated with a stage number, which can be assigned to each switch port. For example, in a three-stage fat tree, the maximum number of stages is three. When the system has a path from A to B and the route is known, then each end node can determine which switch port the packet passes through at any given stage of the path. By doing so, each end node can determine at which different switch ports the packet experienced congestion by correlating the route with the received congestion message.
[0428] According to an embodiment, the system can provide congestion feedback back that indicates which congestion was detected at which stage, and then if the end node has congestion due to a shared link segment, then congestion control is applied to that segment instead of to different end ports. This provides more granular information about congestion.
[0429] According to an embodiment, by providing this finer granularity, the end node can then use an alternate path when routing future packets. Or, for example, if the end node has multiple flows all going to different destinations, but congestion is detected at a common stage in the path, then a re-route can be triggered. The system and method provide an immediate reaction in terms of throttling associated with - rather than having 10 different congestion notifications. This can handle congestion notifications more efficiently.
[0430] Fig.38 A system for using multiple CE markings in both FECN and BECN in a high performance computing environment according to an embodiment is shown.
[0431] According to an embodiment, in a high-performance computing environment such as a switching network or subnet 3800, multiple end nodes 3801 and 3802 can support multiple virtual machines VM1-VM4 3850-3853, which are interconnected via multiple switches, such as leaf switches 3811 and 3812, switches 3821 and 3822, and root switches 3831 and 3832.
[0432] Not shown in the figure are various host channel adapters that provide functionality for connecting nodes 3801 and 3802 and virtual machines to be connected to the subnet, according to an embodiment. Such embodiments are discussed above with respect to SR-IOV, where each virtual machine can be associated with a virtual function of a hypervisor on a host channel adapter.
[0433] According to an embodiment, node, a packet 3851 sent from VM3 3852 to VM1 3850 may pass through the subnet 3800 via multiple links or stages (such as, stage 1 to stage 6), as shown in the figure. Packet 3851 may experience congestion at any of these stages when passing through the subnet, and when such congestion is detected at any stage, it can be marked by the switch. In addition to marking the packet as having experienced congestion, the switch that marks can also additionally indicate the stage at which the packet experienced congestion. Upon arriving at the destination node VM1, VM1 can send a response packet via a feedback message 3870 (e.g., automatically), which can indicate to VM3 3852 that the packet experienced congestion, and at which stage(s) the packet experienced congestion.
[0434] According to an embodiment, each switch port represents a stage in the entire subnet. Thus, each packet sent in the subnet can pass through a maximum number of stages. In order to identify the location where congestion is detected (possibly in multiple locations), the congestion marker (e.g., CE flag) is extended from a simple binary flag (experienced congestion) to a bit field containing multiple bits. Each bit of the bit field can then be associated with a stage number, which can be assigned to each switch port. For example, in a three-stage fat tree, the maximum number of stages is three. When the system has a path from A to B and the route is known, then each end node can determine which switch port the packet passes through at any given stage of the path. By doing so, each end node can determine at which different switch ports the packet experienced congestion by associating the route with the received congestion message.
[0435] According to an embodiment, the system can provide congestion feedback back that indicates at which stage congestion was detected, and then if an end node has congestion due to a shared link segment, congestion control is applied to that segment, rather than to a different end port. This provides more granular information about congestion.
[0436] According to an embodiment, by providing this finer granularity, the end node can then use an alternate path when routing future packets. Or, for example, if the end node has multiple flows all going to different destinations, but congestion is detected at a common stage in the path, then a re-route can be triggered. The system and method provide an immediate reaction in terms of throttling associated with - rather than having 10 different congestion notifications. This can handle congestion notifications more efficiently.
[0437] Fig.39 is a flow chart of a method for using multiple CE markings in both FECN and BECN in a high performance computing environment, according to an embodiment.
[0438] According to an embodiment, at step 3910, the method may provide a first subnet at one or more microprocessors, the first subnet comprising: a plurality of switches, the plurality of switches comprising at least a leaf switch, wherein each of the plurality of switches comprises a plurality of switch ports; a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via a plurality of switches; and a plurality of end nodes, comprising a plurality of virtual machines.
[0439] According to an embodiment, at step 3920, the method may receive an ingress packet from a remote end node at an end node attached to a host channel adapter, wherein the ingress packet traverses at least a portion of a first subnet before being received at the end node, wherein the ingress packet includes a marking indicating that the ingress packet experienced congestion during traversal of the at least a portion of the first subnet.
[0440] According to an embodiment, upon receiving the ingress packet, at step 3930, the method may, by the end node sending a response message from the end node attached to the host channel adapter to the remote end node, the response message indicating that the ingress packet experienced congestion during passage through at least a portion of the first subnet, wherein the response message includes a bit field.
[0441] QOS and SLA in switching architectures such as private architectures
[0442] According to an embodiment, private network architectures in the cloud and in larger clouds for customer and on-premises installations (e.g., private architectures, such as those used to build dedicated distributed devices or general-purpose high-performance computing resources) desire the ability to deploy VM-based workloads, where an essential requirement is that quality of service (QOS) can be defined and controlled for different types of communication flows. In addition, workloads belonging to different tenants must execute within the boundaries of related service level agreements (SLAs) while minimizing interference between such workloads and maintaining QOS assumptions for different communication types.
[0443] According to the embodiments, the following sections discuss relevant problem scenarios, goals, and potential solutions.
[0444] According to an embodiment, an initial approach for provisioning architecture resources to cloud customers (also referred to as "tenants") is that a tenant may be allocated a dedicated portion of a rack (e.g., a quarter rack), or one or more full racks. This granularity means that each tenant is guaranteed to have a communication SLA that is always met as long as the allocated resources are fully operational. The same is true when a single rack is divided into multiple portions, as the granularity is always a full physical server with an HCA. In principle, connections between different such servers in a single rack are always made through a single full crossbar switch. In this case, due to communication traffic between a set of servers belonging to the same tenant, there are no resources that are shared in a way that could cause contention or congestion between flows belonging to different tenants.
[0445] However, according to an embodiment, since the redundant switches are shared, it is critical that traffic generated by a workload on one server cannot target a server belonging to another tenant. Even if such traffic is not conducive to any communication or data leakage / observation between tenants, the result may be severe interference or even DOS-like (denial of service) effects on communication flows belonging to other tenants.
[0446] According to an embodiment, despite the fact that the presence of a fully crossbar leaf switch essentially means that all communications between local servers can occur via the local switch only, there are several situations where this may not be possible or is not implemented due to other practical issues:
[0447] According to an embodiment, for example, if the host bus (PCIe) generates bandwidth that can sustain only one fabric link at a time, it is critical that only one HCA port is used for data traffic at any point in time. Therefore, if not all servers agree on which local switch to use for data traffic, some traffic will have to go through the inter-switch links (ISLs) between local leaf switches.
[0448] According to an embodiment, if one or more servers have lost connectivity to one of the switches, then all communications must be conducted via the other switch. Thus, again, if not all pairs of servers agree to use the same single switch, then some data traffic will have to go through the ISL(s).
[0449] • According to an embodiment, if a server can use two HCA ports (and thus two leaf switches), but cannot enforce connections to only be made via HCA ports connected to the same switch, some data traffic may go over the ISL.
[0450] ο The reason this scheme was ultimately adopted is that the lack of socket / port numbers in the architecture host stack means that a process can only establish one socket to accept incoming connections. This socket can then only be associated with a single HCA port at a time. As long as the same single socket is also used when establishing outgoing connections, the system will end up with multiple connections requiring ISLs despite the large number of processes having their single sockets evenly distributed among the local HCA ports.
[0451] According to an embodiment, in addition to the special case single-rack scenario of implementing ISL usage / sharing outlined above, once the granularity of provisioning is extended to a multi-rack configuration where leaf switches in each rack are interconnected through spine switches, the communication SLAs of different tenants become highly dependent on which servers are assigned to which tenants, and how different communication flows are mapped to different switch-switch links through the fabric-level routing scheme. The key issue in this scenario is that two optimization aspects are somewhat contradictory:
[0452] • According to an embodiment, on the one hand, all concurrent flows to different destination ports should use as many different paths (ie, different switch-switch links - ISLs) through the fabric as possible in order to provide the best possible performance.
[0453] According to an embodiment, on the other hand, in order to provide predictable QOS and SLA for different tenants, it is important that flows belonging to different tenants do not compete for bandwidth on the same ISL at the same time. In general, this means that which paths different tenants can use must be restricted.
[0454] However, according to an embodiment, in some cases, depending on the size of the system, the number of tenants, and how servers are provisioned to different tenants, it may be unavoidable that flows belonging to different tenants will compete for bandwidth on the same ISL. In this case, from an architectural perspective, there are mainly two methods that can be used in order to solve the problem and reduce possible contention:
[0455] • According to an embodiment, limiting which switch buffer resources different tenants (or groups of tenants) can occupy ensures that flows from different tenants have independent forward progress from other tenant(s), despite competing for the same ISL bandwidth.
[0456] • According to an embodiment, an "admission control" mechanism is implemented which will limit the maximum bandwidth that one tenant can consume at the expense of other tenants.
[0457] According to an embodiment, for a physical architecture configuration, the problem is that the bisection bandwidth is as high as possible and ideally non-blocking or even over-provisioned. However, even with non-blocking bisection bandwidth, there may be scenarios where it is difficult to achieve the desired SLA for one or more tenants given the current allocation of servers to different tenants. In this case, the best approach is to re-provision at least some of the servers for different tenants to reduce the need for independent ISLs and, in general, bisection bandwidth.
[0458] According to an embodiment, some multi-rack systems have a blocked fat-tree topology, and the assumption behind this is that the workload will be provisioned so that related communication servers are largely located within the same rack, which means that a large portion of the bandwidth utilization is only between ports on the local leaf switch. In addition, in traditional workloads, most of the data traffic is from a fixed set of nodes to another fixed set of nodes. However, according to an embodiment, for next-generation servers with non-volatile memory and newer communication and storage middleware, the communication workload will be even more demanding and less predictable because different servers may provide multiple functions simultaneously.
[0459] According to an embodiment, the goal is to provide each tenant with provisioning granularity at the VM level rather than the physical server level. In addition, the goal is to be able to deploy up to dozens of VMs on the same physical server, where different sets of VMs on the same physical server may belong to different tenants, and each tenant may each represent multiple workloads with different characteristics.
[0460] Additionally, according to an embodiment, while current architecture deployments have used different Type of Service (TOS) associations to provide basic QOS (traffic separation) for different flow types (e.g., to prevent locked messages from becoming "stuck" after a large data transfer), it is also desirable to provide communication SLAs for different tenants. These SLAs should ensure that tenants experience workload throughput and response times that meet expectations, even if the workload has been provisioned on a physical infrastructure shared by other tenants. The tenant's associated SLA should be implemented independently of concurrent activities of workloads belonging to other tenants.
[0461] According to an embodiment, while a fixed (minimum) set of CPU cores / threads and physical memory can be provisioned for a workload on a fixed (minimum and / or maximum) set of physical servers, provisioning fixed / guaranteed networking resources is generally less straightforward as long as the deployment implies sharing of HCAs / NICs at the servers. Sharing of HCAs also essentially means that at least the ingress and egress links to / from the architecture are shared by different tenants. Therefore, while different CPU cores / threads can truly operate in parallel, there is no way to divide the capacity of a single architecture link except for some kind of bandwidth multiplexing or "time sharing". This basic bandwidth sharing may or may not be combined with the use of different "QOS IDs" (e.g., service level, priority, DSCP, traffic class, etc.) that will be considered when implementing buffer selection / allocation and bandwidth arbitration within the architecture.
[0462] According to an embodiment, the overall server memory bandwidth should be very high relative to the typical memory bandwidth requirements of any individual CPU core in order to prevent memory-intensive workloads on some cores from causing delays to other cores. Similarly, ideally, the available architectural bandwidth of a physical server should be large enough to allow each tenant of a shared server to have sufficient bandwidth relative to the communication activities generated by (one or more) related workloads. However, when several workloads attempt to perform large amounts of data transfer, it is likely that more than one tenant can utilize the full link bandwidth - even 100Gb / s and above. In order to address this scenario, it is necessary to complete the provision of multiple tenants to the same physical server in a manner that ensures that each tenant obtains at least a given minimum percentage of available bandwidth. However, for RDMA-based communications, the ability to impose restrictions on how much bandwidth a tenant can generate in the egress direction does not mean that the ingress bandwidth can be limited in the same way. That is, although each sender is limited by the maximum send bandwidth, multiple remote communication peers may send data to the same destination in a manner that completely overloads the receiver. In addition, RDMA read operations can be generated from local tenants using only a negligible egress bandwidth. This can potentially lead to destructive ingress bandwidth if a large number of RDMA read operations are generated for multiple remote peers. Therefore, in order to limit the total fabric bandwidth used by a single tenant on a single server, it is not sufficient to impose a maximum limit on the egress bandwidth.
[0463] According to an embodiment, the systems and methods can configure an average bandwidth limit for a tenant that will ensure that the tenant never exceeds its relative portion of the bandwidth of its associated link in either the ingress or egress direction, regardless of the use of RDMA read operations and regardless of the number of remote peers with active data traffic, and regardless of the bandwidth limits of the remote peers. (How this is achieved is discussed in the "Long-Term Goals" section below).
[0464] According to an embodiment, as long as the systems and methods cannot implement all aspects of communication bandwidth restrictions, a tenant's highest level communication SLA can only be achieved by restricting it from sharing a physical server with other tenants, or potentially, it will not share a physical HCA with other tenants. (i.e., in the case of a server with multiple physical HCAs). Restrictions can also be used in cases where the physical HCA can operate in active-active mode with full link bandwidth utilization of both HCA ports, where a given tenant is granted exclusive access to one of the HCA ports under normal circumstances. Nevertheless, due to HA constraints, the failure of a complete HCA (in the case of multiple HCAs per server) or a single HCA port may mean that reconfiguration and sharing of the expected communication SLA for a given tenant is no longer guaranteed.
[0465] According to an embodiment, in addition to the restrictions on the overall bandwidth utilization of a single link, the ability of each tenant to achieve QOS between different communication flows or flow types depends on it not experiencing severe congestion conflicts due to the communication activities of other tenants relative to the fabric level buffer resources or arbitration. In particular, this means that if a tenant is using a specific "QOS ID" to achieve low latency messaging, then it should not find itself "competing" with large amounts of data traffic from another tenant due to how another tenant is using the "QOS ID" and / or due to how the fabric implementation forces the use of the "QOS ID" and / or how it maps to packet buffer allocations and / or bandwidth arbitration within the fabric. Therefore, if the tenant communication SLA means that the tenant intra-QOS assumption cannot be achieved without relying on other tenants sharing the same fabric link(s) to "behave well", then this may force the tenant to be provisioned without sharing with other tenants' HCAs (or HCA ports).
[0466] According to an embodiment, for the basic bandwidth allocation and QOS issues described above, sharing constraints apply to fabric internal links as well as server local HCA port links. Thus, depending on the nature and stringency of a given tenant's communication SLA, deployment of VMs for a tenant may have restrictions on sharing of physical servers and / or HCAs as well as sharing of fabric internal ISLs. To avoid ISL sharing, routing restrictions may be applied as well as restrictions on where VMs can be provisioned relative to each other within a private fabric topology.
[0467] Considerations regarding topology, routing, and blocking scenarios:
[0468] According to an embodiment, as described above, in order to ensure that tenants can achieve the expected communication performance between a group of communication VMs without relying on the operation of VMs belonging to other tenants, there may not be multiple HCAs / one HCA port or any architectural ISL shared with other tenants. Therefore, the highest SLA category provided will usually implement this as an implicit implementation. This is in principle the same scheme as the current provisioning model of many traditional systems in the cloud. However, for shared leaf switches, this SLA will need to ensure that there is no ISL shared with other tenants. In addition, in order for tenants to achieve the best possible balance of flows and utilization of available architectural resources, it needs to be able to "optimize non-blocking" in an explicit way (i.e., the communication software infrastructure must provide tenants with a method to ensure that communication occurs in a way that different flows do not compete for the same link bandwidth). This will include a method to ensure that communication that can occur via a single leaf switch is actually implemented in this way. In addition, in the case where communication must involve ISLs, it should be possible to balance the traffic between available ISLs to maximize throughput.
[0469] According to an embodiment, from a single HCA port, as long as the maximum bandwidth available is the same for all links in the fabric, it does not make sense to try to balance traffic between multiple ISLs. From this perspective, it makes sense to use a dedicated "next hop" ISL for each sending HCA port as long as the available ISLs represent a non-blocking sub-topology with respect to the sender. However, unless the relevant ISL represents a connection between only two leaf switches, the scheme of having a dedicated next hop ISL for each sender port is not really sustainable because at some point, if communication is with multiple remote peer HCA ports connected to different leaf switches, then more than one ISL must be used.
[0470] According to an embodiment, in a non-blocking Infiniband fat tree topology, a popular routing algorithm uses a "dedicated downstream path", which means that in a non-blocking topology, there are the same number of switch ports in each layer of the fat tree. This means that each end port can have a dedicated port chain from a root switch, through each intermediate switch layer, to the egress leaf switch port that connects the relevant HCA port. Therefore, all traffic for a single HCA port will use this dedicated downstream path, and there will be no traffic to any other destination port (in the downstream direction) on these links. However, in the upstream direction, there cannot be a dedicated path to each destination, with the result that some links in the upstream direction will have to be shared by traffic to different destinations. In the next round, when different flows to different destinations all try to utilize the full bandwidth on the shared intermediate link, this may cause congestion. Similarly, if multiple senders are sending to the same destination at the same time, this may cause congestion in the dedicated downstream path, which may then quickly spread to other unrelated flows.
[0471] According to an embodiment, as long as a single destination port belongs to a single tenant, there is no risk of congestion between multiple tenants in a dedicated downlink path. However, it is still a problem that different tenants may need to use the same link in the uplink direction in order to reach the root switch (or intermediate switch) representing the dedicated downlink path. By dedicating as many different root switches as possible to specific tenants, the system and method will reduce the need for different tenants to share paths in the uplink direction. However, from a single leaf switch, the scheme can reduce the number of available uplinks toward the associated (one or more) root switches. Therefore, in order to maintain non-blocking binary bandwidth between servers (or more precisely, HCA ports) belonging to the same tenant, the number of servers allocated to a single tenant on a specific leaf switch (i.e., in a single rack) will need to be less than or equal to the number of uplinks toward the (one or more) root switches used by the tenant. On the other hand, in order to maximize the ability to communicate via a single cross switch, it makes sense to allocate as many servers as possible to the same tenant in the same rack.
[0472] Depending on the embodiment, this essentially implies a conflict between being able to utilize guaranteed bandwidth within a single leaf switch and guaranteed bandwidth to communication peers in different racks. To resolve this dilemma, the best approach may be to use a scheme in which tenant VMs are grouped based on which leaf switches (i.e., leaf switch pairs) they are directly connected to, and then properties that define the available bandwidth between these groups are needed. However, again, there is a trade-off between being able to maximize bandwidth between two such groups (e.g., between the same tenant in two racks) and being able to guarantee bandwidth to multiple remote groups. Nevertheless, in the special case of only two layers of switches (i.e., the leaf layers are interconnected by a single spine layer), a non-blocking topology means that there can always be N dedicated uplinks between the leaf switch and the N spine ports of the N HCA ports belonging to the same tenant. Therefore, as long as these N spine ports represent a spine that "owns" all dedicated downstream paths for all relevant remote peer ports, the configuration is non-blocking for that tenant. However, if the associated remote peer represents a dedicated downlink path from more than N spine switches, or if the N uplinks are not distributed among all of the associated spine switches, the systems and methods have possible contention conflicts with respect to other tenants.
[0473] According to an embodiment, among the VMs of a single tenant, independent of non-blocking or blocking connections, there is still potential contention between flows from different sources connected to the same leaf switch. That is, if the destination has a dedicated downlink path from the same trunk, and the number of uplinks from the source leaf switch to that trunk is less than the number of such concurrent flows, then as long as all senders operate at full link speed, some kind of blocking / congestion on the uplink cannot be avoided. In this case, in order to maintain bandwidth, the only option is to use a secondary path to one of the destinations via a different trunk. This would then represent a potential conflict with another dedicated downlink path, since a standard non-blocking fat tree can only have one dedicated downlink per end port.
[0474] According to an embodiment, in the case of some conventional systems, there may be a blocking factor of three (3) between the leaf switches and the spine switches. Therefore, in a multi-rack scenario where the workload is distributed in a way that means more than one-third of the communication traffic is between racks rather than within a rack, the resulting segment bandwidth will be blocked. For example, in an 8-rack system, the most general scenario of even distribution of traffic between any node pair means that 7 / 8 of the communication is between racks, and the blocking effect will be huge.
[0475] According to an embodiment, if the cost of over-provisioning cabling can be tolerated in the system (i.e., given a fixed switch unit cost), then the additional links can be used to provide "backup" downlinks for each leaf switch, and also to provide spare uplink capacity from each leaf to each spine. That is, in both cases, at least some potential remedies are provided for dynamic workload distribution that represents an uneven distribution of traffic, and thus cannot take advantage of a topology that is inherently non-blocking in the first place.
[0476] According to an embodiment, a higher radix full crossbar switch also has the potential to increase the size of each single "leaf domain", as well as reduce the number of spine switches required for a given system size. For example, in the case of a 128-port switch, two full racks with 32 servers can be contained in a single full crossbar switch leaf domain and still provide non-blocking uplink connections. Similarly, only 8 spines are needed to provide non-blocking connections between 16 racks (512 servers, 1024 HCA ports). Therefore, there are still only 8 uplinks from each leaf to each spine (i.e., in the case of a single fully connected network). In the extreme case where all HCA ports on a leaf are sent to a single remote leaf via a single spine, this still means a blocking factor of 8. On the other hand, assuming that the dedicated downstream paths of each leaf switch are evenly distributed among all spines, the likelihood of this extreme scenario should be negligible.
[0477] According to an embodiment, in the case of dual independent networks / rails where each leaf switch in a redundant leaf switch pair belongs to a single rail with dedicated spines. The same 8 spines will be divided into two groups of 4 (one for each rail), so in this case, each leaf in the rail only needs to be connected to 4 spines. Therefore, in this case, the worst case blocking factor will be only 4. On the other hand, in this scenario, it becomes even more important to select the rail for each communication operation to provide load balancing across the two rails.
[0478] Dynamic and static packet routing / forwarding and multipathing:
[0479] According to an embodiment, although standard InfiniBand uses static routing for each destination address, there are several standard and proprietary schemes for dynamic routing selection in Ethernet switches. For InfiniBand, there are also various proprietary schemes for "adaptive routing" (some of which may be standardized).
[0480] According to an embodiment, one advantage of dynamic routing is a higher likelihood of optimal utilization of the relevant bisection bandwidth within the fabric, and thus higher overall throughput. However, a potential disadvantage is that ordering may be disturbed, and congestion in one area of the fabric may more easily propagate to other areas (i.e., in a manner that could have been avoided if static routing had been used).
[0481] According to an embodiment, while "dynamic routing" or "dynamic route selection" is generally used for forwarding decisions that occur within and between switches, "multipath" is the term used when traffic to a single destination can be distributed over multiple paths based on explicit addressing from the sender(s). Such multipathing can include "striping" (i.e., the complete message is divided into multiple sub-messages, each representing a separate transmission operation) of a single message across multiple local HCA ports, and it can mean that different transmissions to the same destination are set up to use different paths through the fabric in a dynamic manner.
[0482] According to an embodiment, in general, if all transmissions from all sources targeting destinations outside the local leaf domain are divided into (smaller) chunks, which are then distributed over all possible paths / routes towards the destination, then the system will achieve optimal utilization of the available bisection bandwidth, and will also maximize "inter-leaf throughput". However, this is only true if the communication workload is also evenly distributed across all possible destinations. If not, then the impact is that any congestion towards a single destination will quickly affect all concurrent flows.
[0483] According to an embodiment, the impact of congestion on dynamic routing and multipathing is that it makes sense to restrict traffic to a single destination to use only a single path / route, as long as that route / path is not a victim of congestion for other destinations or any intermediate links. In a two-tier fat tree topology with dedicated downstream paths, this means that the only possible congestion that is not related to the end ports will exist on the uplinks to the same backbone switch. This means that it makes sense to treat all uplinks to the same backbone as a group of ports sharing the same static route, except that individual ports will be dynamically selected for specific destinations. Alternatively, individual ports can be selected based on tenant association.
[0484] According to an embodiment, the use of tenant association to select uplink ports within such a group can be based on a fixed association or based on a scheme where different tenants have a "first priority" to use a certain (certain) port but can also use other ports. The ability to use another port will then depend on this not conflicting with the "first priority" traffic of another port. In this way, as long as there is no conflict, the tenant has the possibility to use all the relevant binary bandwidth, but when there is a conflict, there will be a guaranteed minimum bandwidth. This minimum guaranteed bandwidth can then reflect all the bandwidth of a single or several links, or a percentage of the bandwidth of one or more links.
[0485] According to an embodiment, in principle, the same dynamic scheme can also be used in the downward path from the backbone to a specific leaf. On the one hand, this increases the risk of congestion caused by sharing the downlink between flows targeting different end ports, but on the other hand, it can provide a way to utilize additional alternative paths between two sets of nodes connected to two different leaf switches, but still provide a way to prevent congestion propagation between different tenants.
[0486] According to an embodiment, in a scenario where different dedicated downstream paths from a spine to a leaf already represent specific tenants, it would be relatively straightforward to have a scheme that allows these links to be used as a "backup" for traffic (belonging to the same tenant) to end ports on the associated leaf switches that have their (primary) dedicated downstream paths from another spine.
[0487] In accordance with an embodiment, one possible model would be to have the switch handle dynamic routing between parallel ISLs connecting a single spine or leaf switch, but have host-level decisions about using explicit multipathing via (one or more) spines that do not represent (primarily) dedicated downstream paths to the associated destinations.
[0488] Per-tenant bandwidth licensing control:
[0489] According to an embodiment, in the case where a single HCA is used by only a single tenant, the system and method can limit the bandwidth that can be generated from the HCA port. In particular, this applies to the case where there is limited bisection bandwidth for traffic flowing to (one or more) remote leaf switches to the tenant.
[0490] According to an embodiment, one aspect of such bandwidth limitation is to ensure that the limitation is applied only to targets affected by the limited bisection bandwidth. In principle, this would involve a scheme where different target groups are associated with specific bandwidth quotas (i.e., strict maximum rates and / or average bandwidth across a certain amount of transmitted data).
[0491] According to an embodiment, such restrictions, by definition, must be implemented at the HCA level. Furthermore, such restrictions would more or less map directly to a virtualized HCA scenario, where VMs belonging to different tenants share an HCA via different virtual functions. In this case, the various "shared bandwidth quota groups" introduced above would require an additional dimension in terms of being associated with a group of one or more VFs rather than just a complete physical HCA port.
[0492] Per-tenant bandwidth reservation on ISL:
[0493] According to an embodiment, as described above, it may make sense to reserve some guaranteed bandwidth for a tenant (or group of tenants) across one or more ISLs. In one scenario, the entire link may be reserved for the tenant (or tenants) by limiting which tenants are allowed to use the link. However, in order to obtain a more flexible and fine-grained solution, an alternative approach is to use a switch arbitration mechanism to ensure that a certain ingress port (or ports) will be allowed to use up to X% of the bandwidth of one or more egress ports, regardless of which other ingress ports are competing for bandwidth on the same egress port.
[0494] According to an embodiment, in this way, all ingress ports may use up to 100% of the bandwidth of the associated egress port(s), as long as this does not conflict with any traffic from the priority ingress ports.
[0495] According to an embodiment, in a scenario where different tenants "own" different ingress ports (e.g., leaf switch ports connected to HCA ports), then the scheme will facilitate a flexible and fine-grained scheme for allocating uplink bandwidth to one or more spine switches.
[0496] According to an embodiment, the usefulness of such a scheme will depend on the extent to which a scheme with strictly dedicated downlink paths is used or not used in the downlink path from the spine to the leaf switch. If a strictly dedicated downlink path is used and the target end port represents a single tenant, then by default there is no potential conflict between different tenants trying to use the downlink. Therefore, in this case, access to the relevant downlink should generally be set to use a round-robin arbitration scheme with equal access rights to all relevant ingress ports.
[0497] According to an embodiment, since an ingress port can represent traffic belonging to different tenants, it should never be a problem that a packet belonging to one tenant can be sent to an egress port that the related tenant is not allowed to send to and consume bandwidth on that egress port. In this case, it is assumed that strict access control (e.g., VLAN-based restrictions on various ports) is adopted instead of arbitration strategy to prevent such packets from wasting any bandwidth.
[0498] According to an embodiment, on a leaf switch, downstream ports from the trunk may be given more bandwidth toward various end ports relative to other local end ports, because the downstream link can in principle represent multiple sender HCA ports, while the local end port represents only a single HCA port. If this is not the case, then a scenario may arise where several remote servers share a single downstream path to a target leaf switch, but then in the next round, 1 / N of the bandwidth toward a single destination will be shared on that leaf in the case where N-1 HCA ports directly connected to the leaf switch also try to send to the same local target port.
[0499] According to an embodiment, in the case of virtualized HCAs representing different tenants, the issue of reserving bandwidth within the fabric (i.e., across various) ISLs can become much more complex. For the ingress / uplink path, a simplified approach is that the HCA provides bandwidth arbitration between different tenants, and then anything sent out on the HCA port will be handled by the ingress leaf switch based on the port level arbitration policy. Therefore, in this case, there is no change from the perspective of the leaf switch.
[0500] According to an embodiment, in the downlink path (from the trunk to the leaf and from the leaf inlet to the end port), the situation is different because the arbitration decision may depend not only on the port trying to forward the packet, but also on which tenant the various pending packets belong to. One possible solution is to (again) limit some ISLs to represent only specific tenants (or tenant groups), which are then reflected in the port level arbitration scheme. Alternatively (or additionally), different priorities or QOS IDs can be used to represent different tenants, as described below. Finally, using the "tenant ID" or any related access control header field (such as VLAN ID or partition ID) as part of the arbitration logic will help the granularity level required for arbitration. However, this may significantly increase the complexity of the arbitration logic in switches that already have significant "time and space" complexity. In addition, since such schemes involve an overload of information that may already work in the end-to-end wired protocol, it is important that such additional complexity does not conflict with any existing use or assumptions about such header field values.
[0501] Different priorities, flow types and QOS IDs / classes across ISL and end-port links:
[0502] According to an embodiment, in order for different flow types to make progress simultaneously on the same link, it is critical that they do not compete for the same packet buffers in the switch and HCA. In addition, in order to distinguish the relative priorities between the different flow types, the arbitration logic that determines what packet to send next on the various switch egress ports must take into account which packet type queues have something to send out on which egress port. The result of the arbitration should be that all active flows progress forward according to their relative priorities and the extent to which the flow control conditions (if any) of the associated flow types for the associated downstream ports currently allow any packets to be sent.
[0503] According to an embodiment, in principle, different QOS IDs can be used to make traffic flows from different tenants independent of each other, even if they are using the same link. However, since the number of packet queues and independent buffer pools that can be supported by each port is usually limited to less than 10, the scalability of this approach is very limited. In addition, when a single tenant wants to use different QOS IDs to make different flow types independent of each other, the scalability is further reduced.
[0504] According to an embodiment, as described above, by logically combining multiple ISLs between a single pair of switches, the system and method can then restrict some links to some tenants, and then ensure that different tenants can use different QOS IDs on different ISLs independently of each other. However, again, if independence from other tenants is to be 100% guaranteed, this imposes a limit on the total bandwidth available to any single tenant.
[0505] According to an embodiment, in an ideal case, HCA ingress (receive) packet processing can always occur at a rate higher than the associated link speed, regardless of what transport level operation the incoming packet represents. This means that different flow types do not need to be flow controlled on the egress port on the leaf switch that is the last link (i.e.) connected to the HCA port. However, the scheduling of different packets in different queues in the leaf switch must still reflect the relevant policies of priority, fairness, and forward progress. For example, if a small high priority packet is targeted to an end port, and N ports are also trying to send a "bulk transfer packet" of the maximum MTU size to the same target port, then the high priority packet should be scheduled before any others.
[0506] In the egress path, the sending HCA may schedule and mark packets in many different ways, depending on the embodiment. In particular, using an overlay protocol as a "bump in the wire" between the VM+virtual HCA and the physical fabric will allow fabric-specific information that may be relevant to the switch to be encoded without messing with any aspects of the end-to-end protocol between tenant virtual HCA instances.
[0507] According to an embodiment, the switch can provide more buffering and internal queuing than what is assumed by current wire protocols. In this way, buffering, queuing, and arbitration policies can be set that take into account that a link is shared by traffic representing multiple tenants with different SLAs, and that different QOS classes are used for different flow types.
[0508] In this way, different high priority tenants may also have more private packet buffer capacity within the switch, according to an embodiment.
[0509] Lossless vs. Lossy Packet Forwarding:
[0510] According to an embodiment, high performance RDMA traffic relies heavily on individual packets not being lost due to lack of buffer capacity in the switch, and also on packets arriving in the correct order for each individual RDMA connection. In principle, the higher the potential bandwidth, the more critical these aspects are to achieving optimal performance.
[0511] According to an embodiment, lossless operation requires explicit flow control, and very high bandwidth implies a trade-off between buffer capacity, MTU size, and flow control update frequency.
[0512] According to an embodiment, a disadvantage of lossless operation is that it will cause congestion when the total bandwidth generated is higher than the downstream / receiving capacity. The congestion will then (most likely) propagate and will eventually slow down all flows competing for the same buffer somewhere within the architecture.
[0513] According to an embodiment, as described above, the ability to provide flow separation based on independent buffer pools is a major scalability issue, which for a switch implementation depends on the number of ports, the number of different QOS classes, and (as described above) also possibly the number of different tenants.
[0514] According to an embodiment, an alternative approach may be to make true lossless operation (ie, lossless based on guaranteed buffer capacity) a "premium SLA" attribute, and thereby limit this feature to tenants who have purchased such a premium SLA.
[0515] According to an embodiment, the key issue here is to be able to "oversubscribe" the available buffer capacity so that the same buffers can be used for lossy and lossless streams, but the buffers allocated to the lossy stream can be preempted at any time a packet from the lossless stream arrives and requires the use of a buffer from the same pool. A set of very small buffers can be provided to allow the lossy stream to progress forward, but at a (much) lower bandwidth than can be achieved with optimized buffer allocation.
[0516] According to an embodiment, different classes of hybrid lossless / lossy flow classes may also be introduced based on differences in the maximum time a buffer can be occupied before having to be preempted and given a higher level SLA type flow type (when this is required). This works best in the context of an architecture implementation with link level credits, but can also potentially be adapted to work with xon / xoff type flow control (i.e., Ethernet pause-based flow control schemes for RoCE / RDMA).
[0517] Strict vs. (more) loose grouping ordering:
[0518] According to an embodiment, an HCA implementation can achieve reliable connectivity and RDMA with minimal state overhead at the transport level through strict ordering and lossless packet forwarding within the architecture. However, in order to better tolerate a certain amount of out-of-order packet delivery due to occasional changes in routing (due to adaptive / dynamic forwarding decisions within the architecture), and also to minimize the overhead and latency associated with the loss of (one or more) packets due to lossy or "hybrid lossless / lossy" mode forwarding within the architecture, an efficient transport implementation will need to maintain sufficient state to allow a large number of individual packets (sequence numbers) to arrive out of order and be retried individually while other packets with later sequence numbers are being accepted and acknowledged.
[0519] According to an embodiment, the key point here is to avoid long delays and average bandwidth loss when lost or out-of-order packets cause retries using current default transport implementations. In addition, by avoiding subsequent packets in a series of issued packets from being dropped, the system and method also significantly reduce the waste of fabric bandwidth that might otherwise consume a lot of bandwidth that could have been consumed by other flows.
[0520] Shared Services and Shared HCAs:
[0521] According to an embodiment, a shared service on an infrastructure (e.g., a backup appliance) used by multiple tenants means that some end-port links will be shared by different tenants, unless the service can provide end-port(s) that can be dedicated to a specific tenant (or a restricted group of tenants). A similar scenario exists when VMs belonging to multiple tenants are sharing the same server and the same HCA port(s).
[0522] According to an embodiment, finely tuned server and HCA resources may be allocated to different tenants, and it may also be ensured that outgoing data traffic bandwidth from the HCA is fairly divided among different tenants according to the relevant SLA levels.
[0523] According to an embodiment, within the architecture, packet buffer allocation and queuing priority and arbitration policies may also be set that reflect relative importance and, thereby, fairness between data traffic belonging to different tenants. However, even with very finely tuned buffer allocation and arbitration policies within the architecture, the granularity may not be fine enough to ensure that the relative priorities and bandwidth quotas of different tenants are accurately reflected in terms of ingress bandwidth to the shared HCA ports.
[0524] According to an embodiment, to achieve such fine-grained bandwidth allocation, a dynamic end-to-end traffic control scheme is needed that can effectively divide and schedule available ingress bandwidth among multiple remote communication peers belonging to one or more tenants.
[0525] According to an embodiment, the goal of such a scheme would be that at any point in time, the set of related active remote clients is able to utilize a fair (not necessarily equal) share of their available ingress bandwidth. Furthermore, such bandwidth utilization should occur without creating congestion in the fabric due to attempts to use excessive bandwidth at the end ports. (Although, fabric-level congestion may still occur due to overloading of shared links within the rest of the fabric).
[0526] According to an embodiment, a high-level model for achieving this goal would be for the receiver to be able to dynamically allocate and update available bandwidth for a set of related remote clients. The current bandwidth value for each remote client needs to be calculated based on the content currently being provided to each client and what will be needed next.
[0527] According to an embodiment, this means that if a single client is currently allowed to use all available bandwidth and another client then also needs to use the ingress bandwidth, then an update instruction must be passed to the currently active client informing it of the new reduced maximum bandwidth, and an instruction must be passed to the new client indicating that it can use the maximum bandwidth corresponding to the reduction of the current client.
[0528] Depending on the embodiment, the same scheme will then in principle apply to "any" number of concurrent clients. However, there is of course a huge trade-off between being able to ensure that the available bandwidth is not "oversubscribed" at any point in time and ensuring that the available bandwidth is always fully utilized when needed.
[0529] According to an embodiment, an additional challenge of such an approach is to ensure that it interoperates well with dynamic congestion control, and also to ensure that congestion associated with a shared path to multiple destinations is handled in a coordinated manner within each sender.
[0530] High availability and failover:
[0531] According to an embodiment, in addition to performance, a key attribute of a private architecture may be redundancy and the ability to failover communications after any single point of failure without loss of service to any client application. Nevertheless, while "loss of service" represents a binary condition (i.e., service is present or lost), some equally important but more scalar attributes are the extent to which there is any outage time during a failover, and if so, how long. Another key aspect is the extent to which expected performance is provided (or re-established) during and after the failover operation is completed.
[0532] According to an embodiment, while from a single node (server) perspective, the goal is that no single point of failure in the fabric communications infrastructure outside of the server itself (i.e., including a single local HCA) should mean that a node becomes unable to communicate. However, from a complete fabric perspective, there is also a question about the extent to which the loss of one or more components means that fabric-wide throughput and performance will be impacted. For example, in a case where the topology is sized to operate using only two spine switches. Then, is it acceptable in terms of bisecting bandwidth and increased risk of congestion if 50% of the leaf-leaf communication capacity is lost if one of the spines is out of service?
[0533] According to an embodiment, another question related to per-tenant SLAs is, to what extent should the ability to reserve and / or prioritize fabric resources for tenants with premium SLAs be reflected in the fact that such tenants are getting a proportionally larger share of the remaining available resources after a failure and subsequent failover operation? That is, in such a way that the impact of a failure on premium SLA tenants will be smaller, but at the expense of a larger impact on other tenants.
[0534] Depending on the embodiment, it may also be a "super premium SLA attribute" in terms of redundancy, i.e. the initial provisioning of resources for such tenants will ensure that no single point of failure will mean that the relevant performance / QOS SLAs will not be met during or after a failure. However, the fundamental problem with such over-provisioning is that there must be extremely fast failover (and failback / rebalancing) in order to ensure that the available resources are always utilized in an optimal manner and that no communication is stopped for more than a very negligible period of time due to any single point of failure.
[0535] According to an embodiment, an example of such a "super advanced" setup could be a system with dual HCA based servers where both HCAs are operating in active-active mode and where both HCA ports are also utilized in active-active mode using an APM (Automatic Path Migration) scheme where the delay before trying an alternative path is very short.
[0536] Path selection:
[0537] According to an embodiment, when multiple possible paths exist between two endpoints, ideally, selection of the best or "correct" path for the associated RDMA connection should be automatic, such that the communication workload experiences the best possible performance within the constraints of the associated SLA, and also such that system-level architectural resources are utilized in an optimized manner.
[0538] According to an embodiment, ideally this would mean that application logic within the VM does not have to deal with which local HCAs and which local HCA ports can or should be used for what communications. This also means an addressing scheme at the node level rather than the port level, and the underlying architecture infrastructure is used transparently to the application.
[0539] In this manner, related workloads may be more easily deployed on different infrastructures without explicitly dealing with different system types or system configurations, according to an embodiment.
[0540] feature:
[0541] According to an embodiment, it is assumed that features in this category are supported by existing HCA and / or switch hardware using current firmware and software.
[0542] According to the embodiment, the main goals in this category are as follows:
[0543] Ability to limit the total egress bandwidth generated by VFs belonging to a single VM or tenant per local physical HCA instance.
[0544] • Ability to ensure that a single VM or tenant's VF belonging to a local physical HCA will be able to utilize at least a minimum percentage of the available local link bandwidth.
[0545] Ability to limit the network (Enet) priority that VF can use.
[0546] o This may mean that multiple VFs must be allocated in order for a single VM to use multiple priorities (i.e., as long as a single VF is only allowed to use a single priority when priority limiting is enabled).
[0547] Ability to restrict which ISLs can be used by groups of flows belonging to a single tenant or group of tenants.
[0548] According to an embodiment, in order to control HCA usage for tenants that are sharing a physical HCA with other tenants, an "HCA Resource Limit Group" (referred to herein as a "HRLG") is established for the tenant. The HRLG can be set up with a maximum bandwidth, which defines the actual data rate that the HRLG can generate, and the HRLG can also be set up with a minimum bandwidth share, which will ensure that the HRLG will achieve at least a specified percentage of the HCA bandwidth when there is contention with other tenants / HRLGs. As long as there is no contention with other HRLGs, the VFs in the HRLG can permanently use up to the specified rate (or link capacity if no rate limit is defined).
[0549] According to an embodiment, the HRLG may contain up to the number of VFs that an HCA instance can support. Within the HRLG, it is expected that each VF will receive a fair share of the "quota" that has been allocated to the HRLG. For each VF, the associated QP will also receive their fair share of access to the local link based on the available HRLG quota and any current flow control limits of the QP. (i.e., a QP will not be considered for local link access if it has received congestion control feedback indicating that it is throttling itself, or if it currently has no "credits" to send at the associated priority level).
[0550] According to an embodiment, within the HRLG, restrictions may be implemented on which priorities a VF may use. As long as this restriction can only be defined in terms of a single priority allowed for a VF, it means that a VM that should use multiple priorities (but is still limited to only some priorities) will have to use multiple VFs - one for each required priority. (Note: using multiple VFs means that sharing local memory resources between multiple QPs using different priorities may be problematic, as it means that the ULPs / applications in the VM must be assigned and use different VFs depending on which priority restrictions / execution policies are defined).
[0551] According to an embodiment, within a single HRLG, there is no difference in bandwidth allocation depending on the priority currently being used by the VF / QP. They all share the associated quota in a fair / equal manner. Therefore, in order to associate different bandwidth quotas with different priorities, one or more dedicated HRLGs need to be defined, which will only include VFs that are restricted to using the priority associated with the shared quota represented by the associated HRLG. In this way, different bandwidth quotas can be given to VMs or tenants with multiple VMs sharing the same physical HCA for different priorities.
[0552] According to an embodiment, the current hardware priority limit prevents data that is attempted to be sent with an illegal priority from being sent to the external link, but does not prevent the relevant data from being obtained from the local memory. Therefore, in the case where the local memory bandwidth that the HCA can maintain in the egress direction is approximately the same as the available external link bandwidth, there is still a waste of the entire HCA link bandwidth. However, if the relevant memory bandwidth is (significantly) greater than the external link bandwidth, then as long as the HCA pipeline operates at optimal efficiency, attempting to use illegal priorities will waste less external link bandwidth. Nevertheless, as long as there is not much to save in terms of external link bandwidth, a possible alternative to prevent the use of illegal priorities can be to implement ACL rules in the switch inlet port. If the relevant tenants can be effectively identified without any possibility of deception, then this can be used to implement tenant / priority association without assigning a separate VF to each priority of the same VM. However, ensuring that the packet / tenant association is always well defined and it is impossible to spoof from the sending VM, and dynamically updating the relevant switch port to perform the relevant implementation whenever the VF is set for use by the VM / tenant shows negligible complexity. One possible solution is to use each VF port MAC to represent an unspoofable ID that can be associated with the VM / tenant. However, if VxLAN or another overlay protocol is in use, this is not straightforward - especially as long as the external switch should not be involved in (or aware of) the overlay scheme being used.
[0553] According to an embodiment, in order to restrict which flows can use which ISLs, the switch forwarding logic needs to have a policy that identifies the relevant flows and sets the forwarding accordingly. One example is to use VLAN IDs to represent flow groups. In the case where different tenants are mapped to different VLAN IDs on the fabric, one possible solution is that the switch can dynamically implement LAG type balancing based on which VLAN IDs are allowed for each port in any LAG or other port group. Another option would involve explicitly forwarding packets based on a combination of destination address and VLAN ID.
[0554] According to an embodiment, where VxLAN-based overlays are used transparently to the physical switch architecture, different overlays may then be mapped to different VLAN IDs to allow the switch to map the VLAN IDs to ISLs as described above.
[0555] According to an embodiment, another possible solution is to set up forwarding for each endpoint address according to a routing scheme that takes into account VLAN membership or some other concept of "tenant" association. However, as long as the same endpoint address value is allowed in different VLANs, the VLAN ID needs to be part of the forwarding decision.
[0556] Depending on the embodiment, distributing each tenant flow to a shared or exclusive ISL may require an overall routing solution to distribute traffic in a globally optimized manner within the fabric (fat tree) topology. The implementation of such a solution usually depends on the SDN type management interface of the switch, but the implementation of overall routing is not easy.
[0557] Short-term and medium-term SLA categories:
[0558] According to an embodiment, the following assumes that a non-blocking two-tier fat-tree topology is being used for a system size (physical node count) that exceeds the cardinality of a single leaf switch. Furthermore, it is assumed that a single VM on a physical server will be able to use all fabric bandwidth (via one or more vHCAs / VFs). Therefore, from an HCA / fabric perspective, the number of VMs per tenant per physical server is not a parameter that needs to be considered a tenant-level SLA factor.
[0559] According to an embodiment, the top level (e.g., Premium Plus)
[0560] · Dedicated servers are available.
[0561] • As many as possible can be allocated on the same leaf domain, unless the number and size of VMs (or HA policy) implies additional distance.
[0562] If a tenant (ie, relative to the number of servers allocated to that tenant within the same leaf domain) is using more than one leaf domain, then on average it may get non-blocking uplink bandwidth from the local leaf, but will not have dedicated uplinks or uplink bandwidth.
[0563] • May be able to use all "flow groups" (ie, representing different buffer pools within the architecture and priorities of arbitration groups, etc.).
[0564] According to an embodiment, a lower tier (e.g., Premium)
[0565] Dedicated servers can be used, but the same leaf domain is not guaranteed.
[0566] • May have on average (ie, relative to the number of servers allocated to this tenant within the same leaf domain) at least 50% non-blocked uplink bandwidth, but will have no dedicated uplinks or uplink bandwidth.
[0567] May be able to use all "stream groups".
[0568] According to an embodiment, the third layer (e.g., Economy Plus)
[0569] May use shared servers but will have dedicated "stream groups".
[0570] ο These resources will be dedicated to the local HCA and switch ports, but will be shared within the fabric.
[0571] May have the ability to use all available egress bandwidth from the local server, but is guaranteed to have at least 50% of the total egress bandwidth.
[0572] Each physical server has only one Economy Plus tenant.
[0573] • On average, can have at least 25% non-blocking leaf uplink bandwidth relative to the number of servers being used by this economy plus tenant.
[0574] According to an embodiment, the fourth layer (eg, economy)
[0575] Can use shared servers
[0576] No dedicated priority
[0577] May be allowed to use up to 50% of the server's egress bandwidth, but can share that bandwidth with up to 3 other economic tenants
[0578] On average, up to 25% of the available leaf uplink bandwidth can be shared with other economic tenants within the same leaf domain.
[0579] According to an embodiment, the lowest layer (e.g., backup)
[0580] Can use spare capacity without guaranteed bandwidth
[0581] Long-term characteristics:
[0582] According to the embodiments, the main features discussed in this section are as follows:
[0583] • Priority restrictions can be enforced per VF in a manner that allows restricting a single VF to use any subset of the total set of supported priorities.
[0584] Whenever a data transfer is attempted using a priority level that was not allowed by the initial VF, the wasted memory bandwidth or link bandwidth is zero.
[0585] • Ability to limit egress rates for different individual priorities across multiple HRLGs so that each VF in an individual HRLG will get their fair share of the associated HRLG total minimum bandwidth and / or maximum rate, but within the constraints defined by the various associated per-priority quotas.
[0586] • Ability to perform sender bandwidth control and congestion shaping on a per-destination and per-shared path / route basis, and do this aggregation at the VM / VF (ie, vHCA port) level and the HCA port level.
[0587] • Ability to limit total average send and RDMA write ingress bandwidth to a VF based on receiver throttling of cooperating remote senders.
[0588] Ability to limit average RDMA read ingress bandwidth to a VF without relying on cooperative remote RDMA read responders.
[0589] When limiting the total average ingress bandwidth to a VF based on receiver throttling of cooperative remote senders, RDMA reads can be included in addition to sends and RDMA writes.
[0590] • Tenant VMs can observe different groups of available segmented bandwidth relative to peer VMs.
[0591] SDN features for routing control and arbitration policies within the architecture.
[0592] In accordance with an embodiment, the HCA VF context may be extended to include a list of legal priorities (similar to the legal SL set for an IBTA IBvPort). Whenever a work request is attempting to use a priority that is not legal for the VF, the work request should fail before any local or remote data transfer is initiated. Additionally, priority mappings may be used to give the application the illusion that any priority can be used. However, the disadvantage of such a mapping, where multiple priorities can be mapped to the same value before sending a packet, is that the application may no longer be able to control its own QOS policy in terms of associating different flow types with different "QOS categories". This restricted mapping represents SLA attributes (i.e., more privileged SLAs mean more actual priorities after mapping). However, it is always important that applications can decide which flow types are associated with which QOS categories (priorities) in a way that will also represent independent flows in the architecture.
[0593] Ingress RDMA read BW quota is implemented via target groups and dynamic BW quota updates:
[0594] According to an embodiment, as long as the target group association for flows from a "producer / sender" node means bandwidth throttling for all outgoing data packets - including UD Sends, RDMA Writes, RDMA Sends, and RDMA Reads (i.e., RDMA Read Responses with data) - all ingress bandwidth for a vHCA port can be fully controlled. This is independent of whether the VM owning the target vHCA port is generating an "excessive" number of RDMA Read Requests to multiple peer nodes.
[0595] According to an embodiment, as described above, the coupling of target groups with flow-specific and "active" BECN signaling means that each vHCA port ingress bandwidth can be dynamically throttled to any number of remote peers.
[0596] According to an embodiment, the above-mentioned "active BECN" message can also be used to convey specific rate values in addition to pure CE marking / unmarking for different phase numbers. In this way, it is possible to have a scenario where an initial incoming packet (e.g., a CM packet) from a new peer can trigger the generation of one or more "active BECN" messages to the HCA (i.e., the relevant firmware / super-privileged software) from which the incoming packet came, as well as to the current communicating peer.
[0597] According to an embodiment, in the case where both ports on the HCA are used simultaneously (i.e., active-active scenario), it may make sense to share the target group between the local HCA ports if there are possible concurrent flows that are sharing some ISLs or may even target the same destination ports.
[0598] According to an embodiment, another reason to share a target group between HCA ports is if the HCA local memory bandwidth cannot sustain the full link speed of both (all) HCA ports. In this case, the target group can be set so that the total aggregate link bandwidth never exceeds the local memory bandwidth, regardless of which port is involved on the source or destination HCA.
[0599] According to an embodiment, in the case of a fixed route towards a specific destination, any (one or more) intermediate target groups will typically represent only a single ISL at a specific stage in the path. However, when dynamic forwarding is active, both the target group and the ECN treatment must take this into account. In the case where dynamic forwarding decisions will only occur to balance traffic between parallel ISLs between a pair of switches (e.g., uplinks from a single leaf switch to a single spine switch), then all treatments are in principle very similar to when only a single ISL is used. FECN notifications will occur based on the state of all ports in the relevant group, and signaling can be "aggressive" in that it is signaled based on congestion indications from any port, or it can be more conservative and based on the size of a shared output queue for all ports in the group. The target group configuration typically represents the aggregate bandwidth of all links in the group, as long as forwarding allows any packet to select the best output port at that point in time. However, if each flow has a strict concept of packet order preservation, then the evaluation of bandwidth quotas will be more complicated, because some flows may "have to" use the same ISL at a certain point in time. If such a flow order scheme is based on a well-defined header field, then it is best to represent each port in the group as an independent target group. In this case, the selection of the target group by the sender-side HCA must be able to perform the same evaluation of the header fields that will be associated with the RC QP connection or address handle as the switch will perform for each packet at runtime.
[0600] According to an embodiment, the initial target group rate for new remote targets may be conservatively set low by default. In this way, there is inherent throttling before the targets have a chance to update the associated rates. Thus, all such rate controls are independent of the VMs themselves involved, but the VMs will be able to request that the hypervisor update quotas for ingress and egress traffic for different remote peers, but this can only be granted within the total constraints defined for the local and remote vHCA ports.
[0601] Associate peer nodes, paths, and target groups:
[0602] According to an embodiment, in order for a VM to identify bandwidth limits associated with different peer nodes and different groups of peer nodes, there must be a way to query which target groups are associated with various communication peers (and associated address / path information). Based on associating a set of communication peers with various target groups and the rate limits represented by the various target groups, the VM will be able to track what bandwidth can be achieved relative to the various communication peers. In principle, this will allow the VM to schedule communication operations in a manner where the best possible bandwidth utilization is achieved over time by having as many concurrent transmissions as possible that do not involve conflicting target groups.
[0603] The relationship between the HCA resource limit group and the target group:
[0604] According to an embodiment, the HRLG concept and the target group concept overlap in several ways, as they both represent bandwidth limits that can be defined and shared between VMs and tenants in a flexible manner. However, while the primary focus of the HRLG is on defining how to allocate different portions of the local HCA / HCA port capacity to different VFs (and thus to VMs and tenants), the target group concept focuses on flow control constraints and bandwidth limits that exist outside of the local HCA in terms of the final destination as well as the intermediate fabric topology.
[0605] In this way, according to an embodiment, it makes sense to use HRLG as a way to control what share of the local HCA capacity the various VFs can use, but ensure that the granted capacity can only be used in a way that does not conflict with any architectural or remote target limits or congestion conditions. These external constraints are then dynamically controlled and reflected via the relevant target groups.
[0606] According to an embodiment, in terms of implementation, the state of all relevant target groups will define which pending work requests the local QPs will have in a flow control state that allows them to generate more egress data traffic at any point in time. This state, as well as the state as to what QP actually has to send, can then be aggregated at the VF / vHCA port level in terms of which VFs are candidates for next to send. The decision as to which VF to schedule for next to send on the HCA port will be based on the state and policies of the various HRLGs in the HRLG hierarchy, the set of VFs that are "ready to send", and the recent history in terms of what egress traffic the VF has generated. For the selected VF, the VF-specific arbitration policy will define which QP will be selected for data transmission.
[0607] According to an embodiment, since the set of QPs with pending data transfers includes QPs with local work requests as well as QPs with pending RDMA read requests from associated remote peers, the above-described scheduling and arbitration will take care of all pending egress data traffic.
[0608] According to an embodiment, ingress traffic (including incoming RDMA read responses) will be controlled by the current state of all relevant target groups in the remote peer node. This (remote) state will include dynamic flow control state based on congestion conditions and explicit updates from the HCA reflecting changes in the ingress bandwidth quotas of the local VFs on the HCA. Such ingress bandwidth quotas will be based on the policies reflected by the HRLG hierarchy. In this way, various VMs can have "fine-tuned" independent bandwidth quotas for ingress and egress, and on a per-priority basis for ingress and egress.
[0609] SLA Category:
[0610] According to an embodiment, the following proposal assumes that a non-blocking two-layer fat-tree topology is being used for a system size (physical node count) that exceeds the cardinality of a single leaf switch. Furthermore, it is assumed that a single VM on a physical server will be able to use all fabric bandwidth (via one or more HCA VFs). Therefore, the number of VMs per tenant per physical server is not a parameter that needs to be considered as an SLA factor.
[0611] According to an embodiment, the highest level tier (e.g., Advanced Plus)
[0612] · You can use only dedicated servers.
[0613] • As many as possible can be allocated on the same leaf domain, unless the number and size of VMs (or HA policy) implies additional distance.
[0614] If a tenant is using more than one leaf domain, it can always have non-blocking uplink bandwidth from the local leaf.
[0615] May be able to use all "stream groups".
[0616] According to an embodiment, a lower level layer (e.g., high level) may be provided.
[0617] · It is possible to use only dedicated servers, but the same leaf domain is not guaranteed.
[0618] • Can be guaranteed (ie, relative to the number of servers allocated to this tenant in the same leaf domain) at least 50% of non-blocking uplink bandwidth.
[0619] May be able to use all "stream groups".
[0620] According to an embodiment, a third level tier (e.g., Economy Plus) may be provided.
[0621] Shared servers may be used, but there will be 4 dedicated "stream groups" (ie, representing different buffer pools within the architecture and priorities of arbitration groups, etc.).
[0622] ο These resources will be dedicated to the local HCA and switch ports, but will be shared within the fabric.
[0623] Able to use all available bandwidth (egress and ingress) of the local server, but guaranteed to have at least 50% of the total bandwidth.
[0624] Each physical server is limited to one Economy Plus tenant.
[0625] • Can be guaranteed at least 25% of non-blocking leaf uplink bandwidth relative to the number of servers being used by the Economy Plus tenant.
[0626] According to an embodiment, a fourth layer (e.g., economy) may be provided.
[0627] Only shared servers can be used
[0628] No dedicated priority
[0629] May be allowed to use up to 50% of the server bandwidth (egress and ingress), but may share this bandwidth with up to 3 other economic tenants
[0630] - Up to 25% of the available leaf uplink bandwidth can be shared with other economic tenants within the same leaf domain.
[0631] According to an embodiment, an underlay (eg, spare) may be provided that may use spare capacity without guaranteed bandwidth.
[0632] Although various embodiments of the present teachings have been described above, it should be understood that they are presented by way of example and not limitation. The embodiments are selected and described in order to explain the principles of the present teachings and their practical application. The embodiments illustrate systems and methods in which the present teachings are utilized to improve the performance of the systems and methods by providing new and / or improved features and / or providing benefits such as reduced resource utilization, increased capacity, improved efficiency, and reduced latency.
[0633] In some embodiments, the features of the present teachings are implemented in whole or in part in a computer including a processor, a storage medium such as a memory, and a network card for communicating with other computers. In some embodiments, the features of the present teachings are implemented in a distributed computing environment, where one or more computer clusters are connected via a network such as a local area network (LAN), a switch fabric network (e.g., InfiniBand), or a wide area network (WAN). A distributed computing environment can have all computers located in a single location, or have clusters of computers at different remote geographic locations connected by a WAN.
[0634] In some embodiments, the features of the present teachings are implemented in whole or in part in the cloud as part of or as a service of a cloud computing system based on shared elastic resources delivered to users in a self-service, metered manner using Web technologies. There are five characteristics of the cloud (defined by the National Institute of Standards and Technology: on-demand self-service; extensive network access; resource pooling; rapid elasticity; and measured service). For example, see "The NIST Definition of Cloud Computing", Special Publication 800-145 (2011), which is incorporated herein by reference. Cloud deployment models include: public, private, and hybrid. Cloud service models include software as a service (SaaS), platform as a service (PaaS), database as a service (DBaaS), and infrastructure as a service (IaaS). As used herein, a cloud is a combination of hardware, software, network, and Web technologies that delivers shared elastic resources to users in a self-service, metered manner. Unless otherwise noted, as used herein, cloud includes public cloud, private cloud, and hybrid cloud embodiments, as well as all cloud deployment models, including but not limited to cloud SaaS, cloud DBaaS, cloud PaaS, and cloud IaaS.
[0635] In some embodiments, the features of this teaching are implemented using hardware, software, firmware or a combination thereof or with the assistance of hardware, software, firmware or a combination thereof. In some embodiments, the features of this teaching are implemented using a processor configured or programmed to perform one or more functions of this teaching. In some embodiments, the processor is a single-chip or multi-chip processor, a digital signal processor (DSP), a system on a chip (SOC), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a state machine, a discrete gate or transistor logic, a discrete hardware component or any combination thereof designed to perform the functions described herein. In some implementations, the features of this teaching can be implemented by circuits specific to a given function. In other implementations, the features can be implemented in a processor that is configured to perform a specific function using instructions stored on, for example, a computer-readable storage medium.
[0636] In some embodiments, the features of this teaching are incorporated into software and / or firmware for controlling the hardware of processing and / or networking systems, and for enabling processors and / or networks to utilize the features of this teaching to interact with other systems. Such software or firmware may include, but is not limited to, application code, device drivers, operating systems, virtual machines, hypervisors, application programming interfaces, programming languages, and execution environments / containers. Based on the teachings of this disclosure, appropriate soft coding pieces may be easily prepared by skilled programmers, as will be clear to those skilled in the art of software.
[0637] In some embodiments, the present method includes a computer program product, such as a computer-readable medium that conveys instructions that can be used to make this teaching effective. In some examples, a computer-readable medium is a storage medium or a computer-readable medium (multiple media), which conveys instructions by storing instructions thereon / therein, and these instructions can be used to program a system such as a computer or otherwise configure to perform any process or function of this teaching. Storage media or computer-readable media can include but are not limited to any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs) or any type of medium or device suitable for storing instructions and / or data. In a particular embodiment, a storage medium or a computer-readable medium is a non-transient storage medium or a non-transient computer-readable medium. A computer-readable medium can also include or alternatively include a transient medium, such as a carrier or a transmission signal that conveys such instructions.
[0638] Thus, from one perspective, systems and methods for providing RDMA (remote direct memory access) read requests as a restricted feature in a high performance computing environment have been described. An example method may provide a first subnet at one or more microprocessors, the first subnet comprising: a plurality of switches; a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via a plurality of switches; and a plurality of end nodes, comprising a plurality of virtual machines. The method may associate a host channel adapter with a selective RDMA restriction. The method may host a virtual machine of a plurality of virtual machines at a host channel adapter comprising the selective RDMA restriction.
[0639] The foregoing description is not intended to be exhaustive or to limit the scope to the precise form disclosed. In addition, in the case of describing the embodiments of the present teaching using a specific series of affairs and steps, it should be clear to those skilled in the art that the scope is not limited to the described affairs and step series. In addition, in the case of having described the embodiments using a specific combination of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of the present teaching. In addition, although various embodiments describe specific combinations of features of the present invention, it should be understood that it will be clear to those skilled in the relevant art that different combinations of features are within the scope of the present teaching, so that the features of one embodiment can be combined in another embodiment. Moreover, it will be clear to those skilled in the relevant art that various additions, reductions, deletions, changes and other modifications and changes can be made in form, details, implementation and application without departing from the spirit and scope. The present invention is intended to be limited by the appropriate interpretation of the following claims.
Claims
1. A system for providing remote direct memory access (RDMA) read requests as a restricted feature in a high performance computing environment, comprising: one or more microprocessors; as well as A first subnet, wherein the first subnet includes Multiple switches, a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via the plurality of switches, a plurality of end nodes, the plurality of end nodes comprising a plurality of virtual machines, wherein each end node of the plurality of end nodes comprises at least one virtual machine of the plurality of virtual machines; wherein a host channel adapter of the plurality of host channel adapters is associated with a selective RDMA restriction; as well as wherein one or more of the plurality of virtual machines are hosted at the host channel adapter including the selective RDMA restriction, wherein the selective RDMA restriction comprises a trusted end node restriction, wherein the trusted end node restriction is configured to restrict issuance of RDMA read requests to a set of virtual machines among the one or more virtual machines hosted on the host channel adapter that are marked as trusted within the selective RDMA restriction, wherein the trusted end nodes are virtual machines that do not generate excessive RDMA read-based ingress bandwidth at the host channel adapter in response to the RDMA read requests, wherein another virtual machine of the one or more virtual machines hosted at the host channel adapter is marked as untrusted within the selective RDMA restriction, and When another virtual machine among the one or more virtual machines hosted at the host channel adapter initiates an RDMA read request for another end node among the multiple end nodes, the host channel adapter prevents the RDMA read request from leaving the host channel adapter based on the selective RDMA restriction.
2. The system according to claim 1, The selective RDMA restriction includes an ingress bandwidth restriction for responding to RDMA read requests issued from the one or more virtual machines.
3. The system according to claim 2, Wherein, when a response to the RDMA read request issued from the one or more virtual machines among the plurality of virtual machines is received, the response is limited by the ingress bandwidth limit of the selective RDMA limit.
4. The system according to claim 3, Wherein the first subnet also includes a subnet manager; and The subnet manager sets an ingress bandwidth limit for the selective RDMA restriction.
5. A method for providing remote direct memory access (RDMA) read requests as a restricted feature in a high performance computing environment, comprising: A first subnet is provided at one or more microprocessors, the first subnet comprising Multiple switches, a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via the plurality of switches, a plurality of end nodes, the plurality of end nodes comprising a plurality of virtual machines, wherein each end node of the plurality of end nodes comprises at least one virtual machine of the plurality of virtual machines; associating a host channel adapter of the plurality of host channel adapters with a selective RDMA restriction; as well as hosting one or more of the plurality of virtual machines at the host channel adapter including the selective RDMA restriction, wherein the selective RDMA restriction comprises a trusted end node restriction, wherein the trusted end node restriction is configured to restrict issuance of RDMA read requests to a set of virtual machines among the one or more virtual machines hosted on the host channel adapter that are marked as trusted within the selective RDMA restriction, wherein the trusted end nodes are virtual machines that do not generate excessive RDMA read-based ingress bandwidth at the host channel adapter in response to the RDMA read requests, wherein another virtual machine of the one or more virtual machines hosted at the host channel adapter is marked as untrusted within the selective RDMA restriction, and When another virtual machine among the one or more virtual machines hosted at the host channel adapter initiates an RDMA read request for another end node among the multiple end nodes, the host channel adapter prevents the RDMA read request from leaving the host channel adapter based on the selective RDMA restriction.
6. The method according to claim 5, The selective RDMA restriction includes an ingress bandwidth restriction for responding to RDMA read requests issued from the one or more virtual machines.
7. The method according to claim 6, Wherein, when a response to the RDMA read request issued from the one or more virtual machines among the plurality of virtual machines is received, the response is limited by the ingress bandwidth limit of the selective RDMA limit.
8. The method according to claim 7, Wherein the first subnet also includes a subnet manager; and The subnet manager sets an ingress bandwidth limit for the selective RDMA restriction.
9. A computer readable medium transmitting instructions for providing remote direct memory access (RDMA) read requests as a restricted feature in a high performance computing environment, the instructions, when read and executed, causing a computer to perform the steps comprising: A first subnet is provided at one or more microprocessors, the first subnet comprising Multiple switches, a plurality of host channel adapters, wherein each of the host channel adapters comprises at least one host channel adapter port, and wherein the plurality of host channel adapters are interconnected via the plurality of switches, a plurality of end nodes, the plurality of end nodes comprising a plurality of virtual machines, wherein each end node of the plurality of end nodes comprises at least one virtual machine of the plurality of virtual machines; associating a host channel adapter of the plurality of host channel adapters with a selective RDMA restriction; as well as hosting one or more of the plurality of virtual machines at the host channel adapter including the selective RDMA restriction, wherein the selective RDMA restriction comprises a trusted end node restriction, wherein the trusted end node restriction is configured to restrict issuance of RDMA read requests to a set of virtual machines among the one or more virtual machines hosted on the host channel adapter that are marked as trusted within the selective RDMA restriction, wherein the trusted end nodes are virtual machines that do not generate excessive RDMA read-based ingress bandwidth at the host channel adapter in response to the RDMA read requests, wherein another virtual machine of the one or more virtual machines hosted at the host channel adapter is marked as untrusted within the selective RDMA restriction, and When another virtual machine among the one or more virtual machines hosted at the host channel adapter initiates an RDMA read request for another end node among the multiple end nodes, the host channel adapter prevents the RDMA read request from leaving the host channel adapter based on the selective RDMA restriction.
10. The computer readable medium of claim 9, The selective RDMA restriction includes an ingress bandwidth restriction for responding to RDMA read requests issued from the one or more virtual machines.
11. The computer readable medium of claim 10, Wherein, when receiving a response to the RDMA read request issued from the one or more virtual machines among the plurality of virtual machines, the response is limited by the ingress bandwidth limit of the selective RDMA limit.
