System and method for efficient virtualization in lossless networks

Abstract

To provide a system and a method that support efficient virtualization in lossless interconnect networks.SOLUTION: A network switching environment 600 includes: one or more switches including at least a leaf switch; a plurality of host channel adapters each including at least one virtual function (VF), a virtual switch, and at least one physical function (PF); a plurality of hypervisors; and a plurality of virtual machines (VMs), and each of the plurality of virtual functions associated with at least one VF. A method deploys a plurality of host channel adapters comprising one or more virtual switches with a prepopulated local identifier (LID) architecture or a virtual switch with a dynamic LID assignment architecture, assigns the LID to each virtual switch, and calculates one or more linear forwarding tables based at least on the LIDs assigned to each of the virtual switches.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] Copyright notice: Parts of the disclosure in this patent document include material protected by copyright. Since this patent document or patent disclosure is contained in the patent files or records of the Patent and Trademark Office, the copyright holder will not object to any copying or reproduction thereof by any other party, but otherwise retains all copyrights at all times.

[0002] Field of invention: This invention generally relates to computer systems, and more particularly to supporting computer system virtualization and live migration using the SR-IOV vSwitch architecture. [Background technology]

[0003] background: As cloud computing architectures become larger in scale, performance and management limitations related to traditional networks and storage are becoming increasingly serious problems. There is growing interest in using high-performance, lossless interconnects such as InfiniBand® (IB) technology as the foundation for cloud computing fabrics. This is a general area that embodiments of the present invention are intended to address. [Overview of the project] [Means for solving the problem]

[0004] overview: A system and method for supporting virtual machine migration in a subnet are described in this specification. An exemplary method may include one or more switches in one or more computers, each containing one or more microprocessors, the one or more switches including at least one leaf switch, each of the one or more switches including multiple ports, and the method may further include multiple host channel adapters. Each of the multiple host channel adapters includes at least one virtual function, at least one virtual switch, and at least one physical function. The multiple host channel adapters are interconnected via the one or more switches. The method may further include multiple hypervisors, each of which is associated with at least one of the multiple host channel adapters. The method may further include multiple virtual machines, each of which is associated with at least one virtual function. The method may further include multiple host channel adapters, each having one or more virtual switches with a prepopulated local identifier (LID) architecture or a dynamic LID assignment architecture. The method allows each virtual switch to be assigned a LID, and the assigned LID corresponds to the LID of the associated physical function. The method can calculate one or more linear forwarding tables (LFTs) based on at least the LIDs assigned to each virtual switch. Each of the one or more LFTs is one or more switches It is associated with one of the switches.

[0005] According to one embodiment, the method is a computer comprising one or more microprocessors, comprising one or more microprocessors and at least one leaf switch. The method can provide multiple switches, each of which includes multiple ports, and further, multiple host channel adapters, each of which includes at least one virtual function, at least one virtual switch, and at least one physical function, and the multiple host channel adapters are interconnected via one or more switches, and further, multiple hypervisors can be provided, each of which is associated with at least one host channel adapter from among the multiple host channel adapters, and further, multiple virtual machines can be provided, each of which is associated with at least one virtual function. The method can provide multiple host channel adapters, each of which has one or more virtual switches with a prepopulated local identifier (LID) architecture or a dynamic LID assignment architecture. The method can assign one pLID from among multiple physical LIDs (pLIDs) to each virtual switch, and the assigned pLID corresponds to the pLID of the associated physical function. The method also assigns multiple virtual LIDs (vLIDs) to each of the multiple virtual machines. One of the vLIDs can be assigned, and the LID space contains multiple pLIDs and multiple vLIDs.

[0006] According to one embodiment, each pLID value can be represented using the standard SLID field and standard DLID field in the local route header of an InfiniBand packet. Similarly, each vLID value can be represented using a combination of the standard SLID field and standard DLID field, combined with two or more additional bits representing extensions. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of an InfiniBand environment according to one embodiment. [Figure 2]This figure shows an example of a tree topology in a network environment according to one embodiment. [Figure 3] This figure shows an exemplary shared port architecture according to one embodiment. [Figure 4] This figure shows an exemplary vSwitch architecture according to one embodiment. [Figure 5] This figure shows an exemplary vPort architecture according to one embodiment. [Figure 6] This figure shows an exemplary vSwitch architecture with pre-populated LIDs according to one embodiment. [Figure 7] This figure shows an exemplary vSwitch architecture with dynamic LID assignment according to one embodiment. [Figure 8] This figure shows an exemplary vSwitch architecture, which includes a vSwitch with dynamic LID assignment and prepopulated LIDs, according to one embodiment. [Figure 9] This figure shows an extended local route header according to one embodiment. [Figure 10] This figure shows two exemplary linear forwarding tables according to one embodiment. [Figure 11] This figure shows an example of efficient virtualization support in a lossless interconnect network according to one embodiment. [Figure 12] This figure shows an example of efficient virtualization support in a lossless interconnect network according to one embodiment. [Figure 13] This figure shows an example of efficient virtualization support in a lossless interconnect network according to one embodiment. [Figure 14] This figure shows an example of efficient virtualization support in a lossless interconnect network according to one embodiment. [Figure 15] This figure shows a potential virtual machine migration according to one embodiment. [Figure 16]Figure illustrating a switch tuple in accordance with one embodiment. [Figure 17] Figure illustrating a reconfiguration process in accordance with one embodiment. [Figure 18] Flowchart illustrating a method for supporting efficient virtualization in a lossless interconnect network in accordance with one embodiment. [Figure 19] Flowchart illustrating a method for supporting efficient virtualization in a lossless interconnect network in accordance with one embodiment. [Description of Embodiments]

[0008] Detailed explanation: The present invention is described by way of illustration, not limitation, in the figures of the accompanying drawings in which like reference numerals refer to like elements. Note that references to "an", "one" or "some" embodiments in this disclosure are not necessarily all referring to the same embodiment, and such references mean at least one. While specific implementations are described, it is understood that these specific implementations are provided for illustrative purposes only. Those skilled in the art will recognize that other components and configurations may be used without departing from the scope and spirit of the present invention.

[0009] Common reference numerals may be used throughout the drawings and detailed description to denote like elements. Accordingly, a reference numeral used in one figure may or may not be referenced in the detailed description specific to such figure when the element is described elsewhere.

[0010] Systems and methods for supporting efficient virtualization in lossless interconnect networks are described in this specification.

[0011] The following description of this invention uses an InfiniBand (IB) network as an example of a high-performance network. It will be apparent to those skilled in the art that other types of high-performance networks can be used without limitation. The following description also uses a fat tree topology as an example of a fabric topology. It will be apparent to those skilled in the art that other types of fabric topologies can be used without limitation.

[0012] To meet the demands of the cloud in the modern era (for example, the Exascale era), virtual machines can utilize low-overhead network communication paradigms such as Remote Direct Memory Access (RDMA). This is desirable. RDMA bypasses the OS stack and communicates directly with the hardware, enabling pass-through technologies such as Single-Root I / O Virtualization (SR-IOV) network adapters. According to one embodiment, a virtual switch (vSwitch) SR-IOV architecture can be provided for applicability in high-performance lossless interconnect networks. Since network reconfiguration time is critical to make live migration a viable option, in addition to the network architecture, a scalable and topology-independent dynamic reconfiguration mechanism can be provided.

[0013] According to one embodiment, a routing strategy for a virtualized environment using vSwitch can be provided, and an efficient routing algorithm for a network topology (e.g., a fat tree topology) can be provided. The dynamic reconfiguration mechanism can be further adjusted to minimize the overhead imposed in a fat tree.

[0014] According to one embodiment of the present invention, virtualization can be beneficial for efficient resource utilization and flexible resource allocation in cloud computing. Live migration can optimize resource usage by moving virtual machines (VMs) between physical servers in an application-transparent manner. To enable this, virtualization can allow for integration through live migration, on-demand provisioning of resources, and flexibility.

[0015] InfiniBand (registered trademark) InfiniBand (IB) is a product of the InfiniBand Trade Association. TM It is an open standard lossless networking technology developed by the Trade Association. This technology is based on serial point-to-point full-duplex interconnect, which provides high-throughput and low-latency communication, particularly for high-performance computing (HPC) applications and data centers.

[0016] The InfiniBand Architecture (IBA) is a two-layer architecture. It supports polology partitioning. At the lower level, IB networks are called subnets, and one subnet may contain a set of hosts interconnected using switches and point-to-point links. At a higher level, one IB fabric consists of one or more subnets that can be interconnected using routers.

[0017] Within a subnet, hosts may be connected using switches and point-to-point links. In addition, there may be a single master management entity, or subnet manager (SM), residing on a designated device within the subnet. The subnet manager is responsible for configuring, starting, and maintaining IB subnets. Furthermore, the subnet manager (SM) may perform routing table calculations within the IB fabric. Here, for example, routing in an IB network aims for proper load balancing between all source-destination pairs within the local subnet.

[0018] Through the subnet management interface, the subnet manager exchanges control packets called subnet management packets (SMPs) with the subnet management agent (SMA). The agent resides on all IB subnet devices. By using SMP, the subnet manager can discover the fabric, configure end nodes and switches, and receive notifications from the SMP.

[0019] According to one embodiment, routing within a subnet in an IB network may be based on a LFT stored in the switch. The LFT is calculated by the SM according to the routing mechanism in use. In a subnet, the Host Channel Adapter (HCA) ports on the end nodes and the switch are addressed using local identifiers (LIDs). Each entry in the LFT consists of a destination LID (DLID) and an output port. One entry per LID in the table Only entries are supported. When a packet arrives at a switch, its output port is determined by looking up the DLID in the switch's forwarding table. Routing is deterministic because packets travel the same path in a network between given source-destination pairs (LID pairs).

[0020] Generally, all subnet managers except the master subnet manager are: It operates in standby mode for fault tolerance. However, in the event of a master subnet manager failure, the standby subnet managers will arrange for a new master subnet manager. The master subnet manager also performs periodic sweeps of subnets to detect any topology changes and adjusts accordingly. Reorganize the network.

[0021] Furthermore, hosts and switches within a subnet can be addressed using local identifiers (LIDs), and a single subnet may be limited to 49,151 unicast LIDs. In addition to LIDs, which are valid local addresses within a subnet, each IB device may have a 64-bit global unique identifier (GUID). GUIDs can be used to form global identifiers (GIDs), which are IB Layer 3 (L3) addresses.

[0022] The SM may compute a routing table (i.e., connections / routes between each pair of nodes in a subnet) during network initialization time. Furthermore, whenever the topology changes, the routing table may be updated to ensure connectivity and optimal performance. During normal operation, the SM may perform periodic light sweeps of the network to check for topology changes. If a change is detected during a light sweep, Alternatively, if an SM receives a message (trap) that signals a network change, the SM may reconfigure the network according to the detected change.

[0023] For example, a Smart Module (SM) can reconfigure the network when the network topology changes, such as when a link goes down, a device is added, or a link is removed. The reconfiguration step may include steps performed during network initialization. Furthermore, the reconfiguration may have a local scope, limited to the subnet where the network change occurred. Additionally, segmenting a large fabric using routers may limit the scope of the reconfiguration.

[0024] According to one embodiment, an IB network may support partitioning as a security mechanism to provide separation of logical groups of systems sharing a network fabric. Each HCA port on a node in the fabric may be a member of one or more partitions. Partition membership is managed by a centralized partition manager, which may be part of the SM. The SM can organize partition membership information for each port as a table of 16-bit partition keys (P_ keys). The SM also manages these ports Switch ports and router ports can be configured using partition enforcement tables that contain P_Key information associated with end nodes sending or receiving data traffic via them. In addition, in typical cases, the partition membership of a switch port may represent the set of all memberships indirectly associated with LIDs routed through the port toward the outbound direction (towards the link).

[0025] According to one embodiment, for communication between nodes, queue pairs (QP) and end-to-end contexts (EEC), excluding the management queue pair (QP0 and QP1), can be assigned to specific partitions. Next, P_key information can be added to all transmitted IB transport packets. When a packet arrives at an HCA port or switch, its P_key value can be checked against a table configured by the SM. If an invalid P_key value is found, the packet is immediately discarded. In this way, communication is permitted only between ports that share partitions.

[0026] Figure 1 shows an example of an InfiniBand environment 100 according to one embodiment, and an example of an InfiniBand fabric is shown. In the example shown in Figure 1, nodes A101 to E105 communicate using the InfiniBand fabric 120 via their respective host channel adapters 111 to 115. According to one embodiment, various nodes (for example, nodes A101 to E105) can be represented by various physical devices. According to one embodiment, various nodes (for example, nodes A101 to E105) can be represented by various virtual devices such as virtual machines.

[0027] Virtual Machines in InfiniBand Over the past decade, CPU overhead has been virtually eliminated by hardware virtualization support, memory overhead has been significantly reduced by virtualizing the memory management unit, storage overhead has been reduced by the use of high-speed SAN storage or distributed network file systems, and device passthrough technologies such as Single Root Input / Output Virtualization (SR-IOV) have been introduced. As network I / O overhead has been reduced by using [this technology], the future prospects for virtualized High Performance Computing (HPC) environments have improved significantly. Currently, the cloud supports virtual HPC (vHPC) clusters using high-performance interconnect solutions, providing the necessary performance. We can provide this.

[0028] However, when connected to lossless networks such as InfiniBand (IB), some cloud functions, such as live migration of virtual machines (VMs), still present challenges due to the complex addressing and routing schemes used in these solutions. IB is an interconnect network technology that provides high bandwidth and low latency, making it very well suited for HPC and other communication-intensive workloads.

[0029] The traditional approach to connecting IB devices to VMs involves using directly assigned SR-IOVs. However, achieving live migration of VMs assigned to IB host channel adapters (HCAs) using SR-IOVs has proven difficult. Each node to which an IB is connected has three different addresses (i.e., LID, GUID, and GID). When live migration occurs, one or more of these addresses change. Other nodes communicating with the VM in migration (VM-in-migration) may lose connectivity. When this occurs, the IB Subnet Manager (SM) should send a Subnet Administration (SA) routing query to reconnect. By determining the virtual machine's new address, we can attempt to restore the lost connection.

[0030] IB uses three different types of addresses. The first type of address is a 16-bit Local Identifier (LID). At least one unique LID is assigned by the SM to each HCA port and each switch. LIDs are used to route traffic within the subnet. Since LIDs are 16 bits long, 65,536 unique address combinations can be formed, of which only 49,151 (0×0001-0×BFFF) can be used as unicast addresses. As a result, the number of available unicast addresses defines the maximum size of the IB subnet. The second type of address is a 64-bit Globally Unique Identifier (GUID) assigned by the manufacturer to each device (e.g., HCA and switch) and each HCA port. The SM may assign additional subnet-specific GUIDs to HCA ports, which is useful when SR-IOV is used. The third type The address of a port is a 128-bit Global Identifier (GID). A GID is a valid IPv6 unicast address, and at least one is assigned to each HCA port. The GID is formed by combining a globally unique 64-bit prefix assigned by the Fabric Administrator with the GUID address of each HCA port.

[0031] Fat Tree (FTree) topology and routing According to one embodiment, some IB-based HPC systems employ a fat tree topology to leverage the beneficial properties that fat trees offer. These properties include full bisection bandwidth and inherent fault tolerance, resulting from the availability of multiple paths between each source-destination pair. The initial concept behind fat trees was to employ fatter links between nodes using more available bandwidth as the tree approaches the root of the topology. These fatter links can help avoid congestion at higher-level switches while maintaining bisection bandwidth.

[0032] Figure 2 shows an example of a tree topology in a network environment according to one embodiment. As shown in Figure 2, one or more end nodes 201-204 may be connected in the network fabric 200. The network fabric 200 may be based on a fat tree topology including multiple leaf switches 211-214 and multiple spine switches or root switches 231-234. In addition, the network fabric 200 may include one or more intermediate switches such as switches 221-224.

[0033] Furthermore, as shown in Figure 2, each of the end nodes 201 to 204 may be a multi-homed node, that is, a single node connected to two or more parts of the network fabric 200 via multiple ports. For example, node 201 may include ports H1 and H2, node 202 may include ports H3 and H4, node 203 may include ports H5 and H6, and node 204 may include ports H7 and H8.

[0034] In addition, each switch may have multiple switch ports. For example, root switch 231 may have switch ports 1-2, root switch 232 may have switch ports 3-4, root switch 233 may have switch ports 5-6, and root switch 234 may have switch ports 7-8.

[0035] According to one embodiment, the fat tree routing mechanism is one of the most popular routing algorithms for IB-based fat tree topologies. The fat tree routing mechanism is also used in OFED (Open Fabric Enterprise). Distribution: Standard software for building and deploying IB-based applications. This is implemented in the WearStack subnet manager, i.e., in OpenSM.

[0036] The purpose of the fat tree routing mechanism is to generate LFTs that uniformly spread the shortest path routes across links in the network fabric. This mechanism traverses the fabric in an indexed order, assigning the target LID of each end node, and consequently the corresponding route, to each switch port. For end nodes connected to the same leaf switch, the indexed order may depend on the switch port to which the end node is connected (i.e., the port numbering sequence). For each port, the mechanism can maintain a port usage counter, and each time a new route is added, it can use the port usage counter to select the least used port.

[0037] According to one embodiment, in a partitioned subnet, nodes that are not members of a common partition are not allowed to communicate. In practice, this is fat This means that some of the routes allocated by the tree routing algorithm are not used for user traffic. Problems arise when the fat tree routing mechanism generates LFTs for those routes in the same way as other functional routes. This behavior can degrade balancing on the link because nodes are routed in indexed order. Fat tree-routed subnets generally result in poor isolation between partitions because routing is performed without regard for partitions.

[0038] According to one embodiment, a fat tree is a hierarchical network topology that can scale with available network resources. Furthermore, fat trees are easily constructed using commodity switches arranged at various levels of hierarchy. In addition, various modifications of fat trees, including k-ary-n-tree, Extended Generalized Fat-Tree (XGFT), Parallel Ports Generalized Fat-Tree (PGFT), and Real Life Fat-Tree (RLFT), are generally available.

[0039] Furthermore, a k-ary-n-tree is an n-level fat tree, and k n End node and n·k n-1 It has a switch and each has 2k ports. A switch has an equal number of connections vertically in the tree. The XGFT fat tree extends the k-ary-n-tree by allowing a different number of vertical connections for switches and a different number of connections at each level in the tree. The PGFT definition further extends the XGFT topology to allow multiple connections between switches. A wide variety of topologies can be defined using XGFT and PGFT. However, for practical use, RLFT, a restricted version of PGFT, has been introduced to define the fat trees commonly found in modern HPC clusters. RLFT uses switches with the same port count at all levels in the fat tree.

[0040] Input / Output (I / O) virtualization In one embodiment, I / O virtualization (IOV) can make I / O available by allowing virtual machines (VMs) to access the underlying physical resources. The combination of storage traffic and inter-server communication can place an unbearably high load on a single server's I / O resources, potentially resulting in backlogs and processor idle periods while data is waiting. As the number of I / O requests increases, IOV can provide availability and improve the performance, scalability, and flexibility of (virtualized) I / O resources to match the performance levels seen in modern CPU virtualization.

[0041] In one embodiment, an IOV is desired that can enable the sharing of I / O resources and protect access to those resources from the VM. The IOV decouples the logical units exposed to the VM from their physical implementation. Currently, emulation, paravirtualization, direct assignment (DA), and single-root I / O are available. Various types of IOV technologies can exist, such as virtualization (SR-IOV).

[0042] According to one embodiment, software emulation is a type of IOV technology. Software emulation can enable a separate front-end / back-end software architecture. The front-end may be a device driver deployed in the VM and can communicate with the back-end, which is implemented by the hypervisor to provide I / O access. The physical device sharing ratio is high, and live migration of VMs can be achieved with only a few milliseconds of network downtime. However, the software... Wear emulation introduces further undesirable computational overhead.

[0043] In one embodiment, another type of IOV technology is direct device assignment. In direct device assignment, I / O devices need to be coupled to VMs, but the devices are not shared between VMs. Direct assignment, or device passthrough, provides nearly intrinsic performance with minimal overhead. Physical devices bypass the hypervisor and are mounted directly to the VMs. However, a drawback of such direct device assignment is that scalability is limited because there is no sharing between virtual machines, such as one physical network card being coupled to one VM.

[0044] According to one embodiment, a Single Root IOV (SR-IOV) is, Hardware virtualization can allow a physical device to appear as multiple independent, lightweight instances of the same device. These instances can be assigned to VMs as pass-through devices and accessed as virtual functions (VFs). The hypervisor accesses the devices through unique, fully functional physical functions (PFs) (for each device). SR-IO V mitigates scalability issues when allocating purely directly. However, the problem presented by SR-IOV is that it can impair VM migration. Among these IOV technologies, SR-IOV can extend the PCI Express (PCIe) standard by providing a means to allow multiple VMs to directly access a single physical device while maintaining nearly intrinsic performance. This allows SR-IOV to offer superior performance and scalability.

[0045] SR-IOV allows a PCIe device to expose multiple virtual devices that can be shared among multiple guests by assigning one virtual device to each guest. Each SR-IOV device has at least one physical function (PF) and one or more associated virtual functions (VFs). The PF is controlled by a virtual machine monitor (VMM) or hypervisor. While PCIe is a standard PCIe function, VF is a lightweight PCIe function. Each VF has its own base address (BAR) and is assigned a unique requester ID. The unique requester ID is assigned to the I / O memory management unit (I / O memory). The management unit (IOMMU) enables the differentiation of traffic streams to and from various VFs. The IOMMU also applies memory to interrupt the translation between PFs and VFs.

[0046] Unfortunately, direct device allocation technology presents a barrier for cloud providers in situations where transparent live migration of virtual machines is desired for data center optimization. The essence of live migration is that the memory contents of the VM are copied to the remote hypervisor. Furthermore, the VM is suspended on the source hypervisor and its operation is resumed at the destination. When using software emulation methods, network interfaces are virtual so that their internal state is stored in memory and then copied. As a result, downtime can be reduced to a few milliseconds.

[0047] However, migration becomes more difficult when direct device assignment technologies such as SR-IOV are used. In such situations, the entire internal state of the network interface cannot be copied because it is tied to the hardware. Instead, the SR-IOV VF assigned to the VM is isolated, and live migration... A migration will be performed, and a new VF will be assigned to the destination. In the case of InfiniBand and SR-IOV, this process can result in downtime on the order of a few seconds. Furthermore, in the SR-IOV shared port model, the VM's address will change after the migration, which adds overhead to the SM and negatively impacts the performance of the underlying network fabric.

[0048] InfiniBand SR-IOV Architecture - Shared Ports There can be various types of SR-IOV models (e.g., shared port model, virtual switch model, and virtual port model).

[0049] Figure 3 shows an exemplary shared port architecture according to one embodiment. As shown in the figure, host 300 (e.g., host channel adapter) can interact with hypervisor 310. Hypervisor 310 can assign various virtual functions 330, 340, and 350 to several virtual machines. Similarly, physical functions can be handled by hypervisor 310.

[0050] According to one embodiment, when using a shared port architecture as shown in Figure 3, the host (e.g., HCA) appears as a single port in a network where there is a single shared LID and shared queue pair (QP) space between the physical function 320 and the virtual functions 330, 350, 350. However, each function (i.e., the physical function and the virtual function) may have its own GID.

[0051] As shown in Figure 3, according to one embodiment, various GIDs can be assigned to virtual and physical functions, and a special queue pair, QP0 and QP1 (i.e., a dedicated queue pair used for InfiniBand management packets), is owned by the physical function. These QPs are also exposed to the VF, but the VF is not permitted to use QP0 (all incoming SMPs from the VF toward QP0 are discarded), and QP1 can function as a proxy for the actual QP1 owned by the PF.

[0052] According to one embodiment, a shared port architecture can enable a highly scalable data center that is not limited by the number of VMs (which are associated with the network by being assigned to virtual functions), because only the physical machines and switches in the network consume LID space.

[0053] However, a drawback of the shared port architecture is its inability to provide transparent live migration, which hinders the possibility of flexible VM placement. Since each LID is associated with a specific hypervisor and shared among all VMs residing on that hypervisor, migrating VMs (i.e., virtual machines migrating to a destination hypervisor) must change their LID to that of the destination hypervisor. Furthermore, as a result of restricted QP0 access, the subnet manager cannot be run inside a VM.

[0054] InfiniBand SR-IOV Architecture Model - Virtual Switch (vSwitch) Figure 4 shows an exemplary vSwitch architecture according to one embodiment. As shown in the figure, host 400 (e.g., host channel adapter) can interact with hypervisor 410, which can assign various virtual functions 430, 440, and 450 to several virtual machines. Similarly, physical functions can This can be handled by the hypervisor 410. The virtual switch 415 can also be handled by the hypervisor 401.

[0055] In one embodiment, in the vSwitch architecture, each virtual function 430, 440, and 450 is a complete virtual host channel adapter (vHCA), meaning that in hardware, the VMs assigned to the VF are assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. For the rest of the network and SM, the HCA 400 appears as a switch to which additional nodes are connected via the virtual switch 415. The hypervisor 410 can use the PF 420, and the VMs (assigned to the virtual functions) use the VF.

[0056] In one embodiment, the vSwitch architecture provides transparent virtualization. However, since each virtual function is assigned a unique LID, the available LIDs are consumed rapidly. Similarly, if many LID addresses are in use (i.e., one for each physical function and each virtual function), more communication paths must be computed by the SM, and more subnet management packets (SMPs) must be sent to the switch to update those LFTs. For example, calculating communication paths can take several minutes in large networks. Since the LID space is limited to 49,151 unicast LIDs, and each VM (via VF) occupies one LID each for the physical node and switch, the number of active VMs is limited by the number of physical nodes and switches in the network, and vice versa.

[0057] InfiniBand SR-IOV Architecture Model - Virtual Port (vPort) Figure 5 shows an exemplary vPort concept according to one embodiment. As shown in the figure, host 300 (e.g., host channel adapter) can interact with hypervisor 410, which can assign various virtual functions 330, 340, and 350 to several virtual machines. Similarly, physical functions can be handled by hypervisor 310.

[0058] In one embodiment, the vPort concept is loosely defined to give vendors implementation freedom (for example, the definition does not stipulate that implementations should be SRIOV-specific), and the purpose of vPort is to standardize how VMs are handled within a subnet. The vPort concept can define both SR-IOV shared port-like architectures and vSwitch-like architectures, or a combination of these architectures, which can be more scalable in both the spatial and performance domains. In addition, vPort supports optional LIDs, and unlike shared ports, the SM can recognize all vPorts available in the subnet, even if the vPort does not use a dedicated LID.

[0059] InfiniBand SR-IOV Architecture Model - LID pre-populated vSwitch According to one embodiment, the disclosure provides a system and method for providing a vSwitch architecture with prepopulated LIDs.

[0060] Figure 6 shows an exemplary vSwitch architecture with pre-populated LIDs according to one embodiment. As shown in the figure, several switches 501-504 can establish communication between members of a fabric, such as an InfiniBand fabric, within a network switching environment 600 (e.g., an IB subnet). This may include several hardware devices such as host channel adapters 510, 520, and 530. Furthermore, host channel adapters 510, 520, and 530 can interact with hypervisors 511, 521, and 531, respectively. Each hypervisor, in conjunction with the host channel adapters, can further interact with, configure, and assign to several virtual functions 514, 515, 516, 524, 525, 526, 534, 535, and 536 to several virtual machines. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. In addition, hypervisor 511 can assign virtual machine 2 551 to virtual function 2 515, and virtual machine 3 552 can be assigned to virtual function 3 516. The hypervisor 531 can further assign virtual machine 4 553 to virtual function 1 534. The hypervisor can access the host channel adapters via physical functions 513, 523, and 533, each of which are fully functional on the host channel adapters.

[0061] According to one embodiment, each of switches 501 to 504 may include several ports (not shown). Some of these ports are used to configure a linear forwarding table to direct traffic within the network switching environment 600.

[0062] In one embodiment, virtual switches 512, 522, and 532 can be handled by their respective hypervisors 511, 521, and 531. In such a vSwitch architecture, each virtual function is a complete virtual host channel adapter (vHCA), meaning that in hardware, the VMs assigned to the VF are assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. For the rest of the network and SM (not shown), HCAs 510, 520, and 530 appear as switches to which additional nodes are connected via virtual switches.

[0063] In one embodiment, the disclosure provides a system and method for providing a vSwitch architecture with pre-populated LIDs. Referring to Figure 5, LIDs are pre-populated for various physical functions 513, 523, and 533, as well as for virtual functions 514-516, 524-526, and 534-536 (even virtual functions not associated with any currently active virtual machines). For example, physical function 513 is pre-populated with LID1, and virtual function 534 is pre-populated with LID10. When the network is booted, LIDs are pre-populated in the SR-IOV vSwitch-enabled subnet. Even if not all VFs are occupied by VMs in the network, the populated VFs are assigned LIDs as shown in Figure 5.

[0064] According to one embodiment, many similar physical host channel adapters may have two or more ports (two ports shared for redundancy), and a virtual HCA may also be represented by two ports and connected to an external IB subnet via one or more virtual switches.

[0065] According to one embodiment, in a vSwitch architecture where LIDs are prepopulated, each hypervisor consumes one LID for itself via the PF, and can consume one or more LIDs for each additional VF. The sum of all available VFs across all hypervisors in an IB subnet gives the maximum number of VMs that can run in the subnet. For example, in an IB subnet with 16 virtual functions per hypervisor, each hypervisor consumes 17 LIDs in the subnet (one LID for each of the 16 virtual functions and one LID for the physical function). In this case, the theoretical hypervisor limit for a single subnet is defined by the number of available unicast LIDs, which is 2891 (obtained by dividing the 49151 available LIDs by 17 LIDs per hypervisor), and the total number of VMs (i.e., the limit) is 46256 (obtained by multiplying the 2891 hypervisors by 16 VFs per hypervisor) (in practice, these numbers are actually smaller because each switch, router, or dedicated SM node in the IB subnet consumes LIDs similarly). Note that vSwitches do not need to occupy additional LIDs because they can share LIDs with PF.

[0066] In one embodiment, in a vSwitch architecture where LIDs are prepopulated, once the network is booted, communication paths are calculated for all LIDs. If a new VM needs to be started, the system does not need to add a new LID in the subnet. Otherwise, operations that could completely reconfigure the network, including recalculating paths, would be the most time-consuming. Instead, available ports for VMs reside in one of the hypervisors (i.e., available virtual functions), and virtual machines are assigned to available virtual functions.

[0067] According to one embodiment, a vSwitch architecture with pre-populated LIDs also enables the ability to compute and use different paths to reach different VMs hosted by the same hypervisor. Essentially, this is achieved by using LID-Mask-Control-like (LMC-like) features in such subnets and networks to provide alternative paths to a single physical machine without being constrained by the LMC constraints that require LIDs to be contiguous. This makes it possible. The ability to freely use discontinuous LIDs is particularly useful when migrating a VM and needing to deliver its associated LID to a destination.

[0068] In one embodiment, along with the aforementioned advantages of a vSwitch architecture with pre-populated LIDs, several considerations can be taken into account. For example, because LIDs are pre-populated in SR-IOV vSwitch-enabled subnets when the network is booting, the initial routing calculation (e.g., at startup) may take longer than if LIDs were not pre-populated.

[0069] InfiniBand SR-IOV Architecture Model - vSwitch with Dynamic LID Assignment According to one embodiment, the present disclosure provides a system and method for providing a vSwitch architecture with dynamic LID assignment.

[0070] Figure 7 shows an exemplary vSwitch architecture with dynamic LID assignment according to one embodiment. As shown in the figure, several switches 501-504 can establish communication between members of a fabric, such as an InfiniBand fabric, within a network switching environment 700 (e.g., an IB subnet). The fabric may include several hardware devices, such as host channel adapters 510, 520, and 530. The host channel adapters 510, 520, and 530 can further interact with hypervisors 511, 521, and 531, respectively. Each hypervisor can further interact with, configure, and assign to several virtual functions 514, 515, 516, 524, 525, 526, 534, 535, and 536 together with the host channel adapters. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 also assigns virtual machine 2 551 to virtual function 2 515, and virtual machine 3 5 52 can be assigned to virtual function 3 516. Hypervisor 531 can further assign virtual machine 4 553 to virtual function 1 534. The hypervisor can access the host channel adapters via physical functions 513, 523, and 533, each of which are fully functional on each of the host channel adapters.

[0071] According to one embodiment, each of switches 501 to 504 may include several ports (not shown). Some of these ports are used to configure a linear forwarding table to direct traffic within the network switching environment 700.

[0072] In one embodiment, virtual switches 512, 522, and 532 can be handled by their respective hypervisors 511, 521, and 531. In such a vSwitch architecture, each virtual function is a complete virtual host channel adapter (vHCA), meaning that in hardware, the VMs assigned to the VF are assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. For the rest of the network and SM (not shown), HCAs 510, 520, and 530 appear as switches to which additional nodes are connected via the virtual switches.

[0073] In one embodiment, the disclosure provides a system and method for providing a vSwitch architecture with dynamic LID assignment. Referring to Figure 7, LIDs are dynamically assigned to various physical functions 513, 523, and 533, with physical function 513 receiving LID1, physical function 523 receiving LID2, and physical function 533 receiving LID3. Those virtual functions associated with active virtual machines can also receive dynamically assigned LIDs. For example, since virtual machine 1 550 is active and associated with virtual function 1 514, virtual function 514 may be assigned LID5. Similarly, virtual functions 2 515, 3 516, and 1 534 are each associated with an active virtual function. Therefore, LIDs are assigned to these virtual functions, with LID7 assigned to virtual function 2 515, LID11 assigned to virtual function 3 516, and LID9 assigned to virtual function 1 534. Unlike vSwitches where LIDs are prepopulated, virtual functions that are not currently associated with an active virtual machine do not receive an LID assignment.

[0074] According to one embodiment, if dynamic LID assignment is performed, the initial routing calculation can be substantially reduced. When the network is booting for the first time and no VMs exist, a relatively small number of LIDs can be used for the initial routing and LFT distribution.

[0075] According to one embodiment, many similar physical host channel adapters may have two or more ports (two ports shared for redundancy), and a virtual HCA may also be represented by two ports and connected to an external IB subnet via one or more virtual switches.

[0076] According to one embodiment, when a new VM is created in a system using a vSwitch with dynamic LID assignment, a free VM slot is discovered, and a unique unused unicast LID is also discovered, in order to determine which hypervisor the newly added VM should boot on. However, there is no known route in the switch's LFT and network to handle the newly added LID. Calculating a new set of routes to handle the newly added VM is undesirable in a dynamic environment where several VMs may be booted every minute. In large IB subnets, calculating a new set of routes can take several minutes, and this procedure would have to be repeated every time a new VM is booted.

[0077] Advantageously, according to one embodiment, since all VFs in the hypervisor share the same uplink as the PF, there is no need to compute new sets of routes. It is only necessary to iterate through the LFTs of all physical switches in the network, copy the forwarding ports from the LID entries belonging to the PF of the hypervisor (where the VMs are created) to the newly added LIDs, and send a single SMP to update the corresponding LFT blocks of the specific switches. This eliminates the need for the system and method to compute new sets of routes.

[0078] According to one embodiment, the assigned LIDs in a vSwitch with a dynamic LID assignment architecture do not need to be contiguous. Comparing the LIDs assigned on VMs on each hypervisor between a vSwitch with pre-populated LIDs and a vSwitch with dynamic LID assignment, it can be seen that the assigned LIDs in the dynamic LID assignment architecture are discontinuous, while the pre-populated LIDs are essentially contiguous. Furthermore, in the vSwitch dynamic LID assignment architecture, when a new VM is created, the next available LID is used throughout the VM's lifetime. Conversely, in a vSwitch with pre-populated LIDs, each VM inherits the LID already assigned to its corresponding VF, and in a network without live migration, VMs continuously assigned to a given VF receive the same LID.

[0079] In one embodiment, a vSwitch with a dynamic LID assignment architecture can overcome the shortcomings of a vSwitch with a pre-popular LID architecture model, at the expense of some additional network and runtime SM overhead. Each time a VM is created, the LFT of the physical switch in the subnet is updated with the newly added LID associated with the created VM. This operation requires sending one subnet management packet (SMP) per switch. Features such as LMC are also unavailable because each VM uses the same path as its host hypervisor. However, there is no limit to the total number of VFs present across all hypervisors, and the number of VFs may exceed the unicast LID limit. In such cases, it is naturally not possible for all VFs to be assigned simultaneously on an active VM, and having more spare hypervisors and VFs adds flexibility to recover from and optimize fragmented network failures when operating near the unicast LID limit.

[0080] InfiniBand SR-IOV Architecture Model - A vSwitch with Dynamically Assigned and Pre-populated LIDs Figure 8 shows an exemplary vSwitch architecture with a vSwitch that has dynamic LID assignment and prepopulated LIDs, according to one embodiment. As shown in the figure, several switches 501-504 can establish communication between members of a fabric, such as an InfiniBand fabric, within a network switching environment 800 (e.g., an IB subnet). The fabric may include several hardware devices, such as host channel adapters 510, 520, and 530. The host channel adapters 510, 520, and 530 can further interact with hypervisors 511, 521, and 531, respectively. Each hypervisor can further interact with, configure, and assign to several virtual functions 514, 515, 516, 524, 525, 526, 534, 535, and 536 together with the host channel adapters. For example, virtual machine 1 550 can be assigned to virtual function 1 514 by hypervisor 511. Hypervisor 511 also supports 2 virtual machines. 551 can be assigned to virtual function 2 515. Hypervisor 521 is virtual Machine 3 552 can be assigned to virtual function 3 526. The hypervisor 531 can further assign virtual machine 4 553 to virtual function 2 535. The hypervisor can access the host channel adapters via physical functions 513, 523, and 533, each of which are fully functional on the host channel adapters.

[0081] According to one embodiment, each of switches 501 to 504 may include several ports (not shown). These ports are used to configure a linear forwarding table to direct traffic within the network switching environment 800.

[0082] According to one embodiment, virtual switches 512, 522, and 532 can be handled by their respective hypervisors 511, 521, and 531. In such a vSwitch architecture, each virtual function is a complete virtual host channel adapter (vHCA), meaning that in hardware, the VMs assigned to the VF are assigned a set of IB addresses (e.g., GID, GUID, LID) and a dedicated QP space. For the rest of the network and SM (not shown), HCAs 510, 520, and 530 appear as switches to which additional nodes are connected via the virtual switches.

[0083] According to one embodiment, the present disclosure provides a system and method for providing a hybrid vSwitch architecture in which dynamic LID assignment is performed and LIDs are prepopulated. Referring to Figure 7, a hypervisor 511 may be configured with a vSwitch having a prepopulated LID architecture, a hypervisor 521 may be configured with a vSwitch in which LIDs are prepopulated and dynamic LID assignment is performed, and a hypervisor 531 may be configured with a vSwitch in which dynamic LID assignment is performed. Therefore, the physical functions 513 and virtual functions 514-516 have their LIDs prepopulated (i.e., LIDs are assigned even to virtual functions that are not assigned to an active virtual machine). Physical function 523 and virtual function 1 524 may have their LIDs prepopulated, while virtual function 2 525 and virtual function 3 526 have their LIDs dynamically assigned (i.e., virtual function 2 525 is available for dynamic LID assignment, and virtual function 3 526 is dynamically assigned the LID 11 because virtual machine 3 552 is assigned to it). Finally, functions (physical and virtual functions) associated with hypervisor 3 531 can have their LIDs dynamically assigned. As a result, virtual function 1 534 and virtual function 3 536 become available for dynamic LID assignment, and virtual function 2 535 is dynamically assigned the LID 9 because virtual machine 4 553 is assigned to it.

[0084] According to one embodiment, as shown in Figure 8, where both vSwitches with pre-populated LIDs and vSwitches with dynamic LID assignments are used (independently or in combination within any given hypervisor), the number of pre-populated LIDs per host channel adapter can be defined by the fabric administrator and can be within the range of 0 <= pre-populated VF <= total VF (per host channel adapter). The VFs available for dynamic LID assignment can be found (per host channel adapter) by subtracting the number of pre-populated VFs from the total number of VFs.

[0085] According to one embodiment, many similar physical host channel adapters can have two or more ports (two ports shared for redundancy), and a virtual HCA is also represented by two ports, and an external IB subnetwork is connected via one or more virtual switches. It can be connected to a net.

[0086] vSwitch scalability According to one embodiment, the problem with using the vSwitch architecture is the limited LID space. To overcome the scalability problem with respect to LID space, the following three alternatives (each described in more detail below) can be used independently or in combination: namely, using multiple subnets; introducing backward-compatible LID space extensions; and combining the vPort architecture and the vSwitch architecture to form a lightweight vSwitch.

[0087] In one embodiment, multiple IB subnets can be used. The LID is a Layer 2 address and must be unique within a subnet. When the IB topology spans multiple subnets, the LID is no longer a limitation, however, if a VM needs to be migrated to a different subnet, its LID address may change because that address may already be in use in the new subnet. While spanning multiple subnets solves the LID limitations of a single subnet topology, it also means that Layer 3 GID addresses must be used for inter-subnet routing, adding additional overhead and latency to the routing process because the Layer 2 header must be modified by the router located at the edge of the subnet. Furthermore, under current hardware and software implementations and loose IBA (InfiniBand Architecture) standards, SMs in individual subnets are unable to recognize the global topology in order to provide optimized routing paths for clusters spanning multiple subnets.

[0088] According to one embodiment, backward-compatible LID space extensions can be introduced in IBA. Problems can arise when increasing insufficient LID space by increasing the number of LID bits, for example, to 24 bits or 32 bits. Increasing the LID space by such an amount can break backward compatibility because the IB Local Route Header (LRH) would have to be overhauled, and legacy hardware would no longer be able to function with the new standard. According to one embodiment, the LID space can be extended while maintaining backward compatibility, so that new hardware can still take advantage of the extensions. The LRH has 7 spare bits that are transmitted as 0 and ignored by the receiver. Two of these spare bits in the LRH for the Source LID (SLID) and 2 bits for the Destination LID (DLID) are used. This allows for expanding the LID space to 18 bits (quadruple the LID space), enabling the creation of schemes using physical LIDs (pLIDs) assigned to physical devices and virtual LIDs (vLIDs) assigned to VMs.

[0089] In one embodiment, if the additional two bits are sent as 0, the LID is used as defined at that time in the IBA (48K unicast LID and 16K multicast LID), and the switch can look up their primary LFT for packet forwarding. In other cases, the LID is a vLID and can be forwarded based on a secondary LFT having a size of 192K. Since the vLID belongs to a VM, and the VM shares an uplink with the physical node having the pLID, the vLID can be excluded from the routing stage when configuring (e.g., initial configuration) or reconfiguring (e.g., after a topology change), but the secondary LFT table in the switch can be updated as described above. When the SM boots up and discovers the network, the SM can It is possible to identify whether all the software supports the extended LID space. If not, the SM can fall back to legacy compatibility mode, and the VM should occupy LIDs from the pLID space.

[0090] Figure 9 shows an extended local route header according to one embodiment. As shown in the figure, within the local route header, the virtual lane (VL) 900 contains 4 bits, the link version (Lver) 901 contains 4 bits, the service level (SL) 902 contains 4 bits, and the LID extension flag (LID The extension flag (LEXTF)903 contains 1 bit and the first spare bit (R1) 9 04 contains 1 bit, link next header (LNH) 905 contains 2 bits, destination local ID (DLID) 906 contains 16 bits, DLID prefix extension (DPF) 907 contains 2 bits Including SLID prefix extension (SPF) 908, 2B The second spare bit (R2) 909 contains 1 bit, the packet length (PktLen) 910 contains 11 bits, and the source local ID (SLID) 911 contains 16 bits. According to one embodiment, both spare bits 904 and 909 can be set to zero.

[0091] According to one embodiment, as described above, the LRH shown in Figure 9 utilizes four of the seven (original) spare bits as prefix extensions for destination local ID 906 and source local ID 908. This signals that, when used, the LRH is used in association with a vLID that can be routed via a secondary LFT in the switch, in association with the LID extension flag. Alternatively, if extensions 907 and 908 are transmitted as zero (ignored by the receiver), the LID is associated with a pLID and used as defined at that time in the IBA.

[0092] Figure 10 shows two exemplary linear forwarding tables according to one embodiment. As shown in Figure 10, linear forwarding table 916 is a forwarding table associated with pLID. The LFT spans from entry 912 (entry 0, indexed by DLID=0) to entry 913 (entry 48K-1, indexed by DLID=48K-1). In this case, each entry in the LFT is indexed by a standard 16-bit DLID and contains a standard IB port number. In contrast, linear forwarding table 917 is a secondary forwarding table associated with vLID. The LFT spans from entry 914 (entry 0, indexed by 18-bit DPF+DLID=0) to entry 915 (entry 256K-1, indexed by 18-bit DPF+DLID=256K-1). In this case, each entry is indexed by an extended 18-bit DPF+DLID and contains a standard IB port number.

[0093] According to one embodiment, a hybrid architecture can be used to form a lightweight vSwitch architecture. A vSwitch architecture that can migrate LIDs along with migrated VMs scales well for subnet management because, in contrast to a shared LID scheme where LIDs will change, there is no requirement for additional signaling to rebuild connectivity with peers after migration. On the other hand, a shared LID scheme scales well for LID space. A hybrid vSwitch + shared vPort model can be realized if the SM is aware of the available SR-IOV virtual functions in the subnet, but certain VFs may receive dedicated LIDs while others are routed in a shared LID manner based on their GIDs. If there is some information about the VM node roles (e.g., calculating routes and load balancing in the network), To ensure they are considered separately while running, popular VMs with many peers (e.g., servers) may be assigned a dedicated LID, while other VMs that do not interact with many peers or run stateless services (which do not need to be migrated and can be regenerated) may share an LID.

[0094] Routing strategies for vSwitch-based subnets In one embodiment, to achieve higher performance, the routing algorithm can take the vSwitch architecture into account when calculating routes. In a fat tree, a vSwitch can be identified in the topology discovery process by its unique characteristic that it has only one uplink to its corresponding leaf switch. Once a vSwitch is identified, the routing function can generate LFTs for all switches so that traffic from each VM can find a path to all other VMs in the network. Each VM has its own address, and therefore each VM can be routed independently of other VMs assigned to the same vSwitch. This results in a routing function that generates multiple independent paths that lead to the vSwitch in the topology, each carrying traffic to a specific VM. One drawback of this approach is that if VM distribution is not uniform among vSwitches, vSwitches with more VMs may be allocated potentially larger network resources. However, the single uplink from the vSwitch to the corresponding leaf switch remains a bottleneck link shared by all VMs assigned to that particular vSwitch. As a result, the network may be utilized suboptimally. The simplest and fastest routing strategy is to generate routes between all vSwitch-vSwitch pairs and route VMs with the same routes assigned to their corresponding vSwitches. Given prepopulated and dynamic LID assignment schemes, each vSwitch has a LID defined by the PF in the SR-IOV architecture. These PF LIDs for the vSwitches can be used to generate LFTs in the first stage of routing, and in the second stage, the VM's LID can be added to the generated LFT. In prepopulated LID schemes, entries to VF LIDs can be added by copying the output ports of the corresponding vSwitches.Similarly, in the case of dynamic LID assignment when a new VM is booted, a new entry is added to all LFTs with the VM's LID and the corresponding output port determined by the vSwitch. The problem with this strategy is that VMs belonging to different tenants sharing a vSwitch may interfere with each other because they share the same complete path on the network. To solve this problem while maintaining high network utilization, a weighted routing scheme for virtualized subnets can be used.

[0095] In one embodiment, a weighted routing scheme can be used for vSwitch-based virtualized subnets. In such a mechanism, each VM on a vSwitch is assigned a parameter weight that can be taken into consideration when calculating routes. The value of the weight parameter reflects the ratio of the vSwitch to the leaf switch link capacity allocated to the VMs on that vSwitch. For example, in a simple configuration, each VM may be assigned a weight equal to 1 / num_vms, where num_vms is the number of booted VMs on the corresponding vSwitch hypervisor. Another possible implementation is to allocate a higher percentage of vSwitch capacity to the most important VMs in order to prioritize the traffic flowing to these VMs. However, since the cumulative weight of VMs per vSwitch can be equal across all vSwitches, the topology... The links can be balanced independently of actual VM distribution. Simultaneously, the scheme eliminates interference between the same vSwitch VMs on intermediate links in the topology, enabling multiple paths where each VM can be routed independently within the network. This scheme can be combined with implementation on each vSwitch for each VM rate cap to ensure that VMs do not exceed their allocated capacity. Furthermore, if multiple tenant groups exist in the network, techniques such as tenant-aware routing can be integrated with the proposed routing scheme to isolate the entire network between tenants.

[0096] According to one embodiment, weighted routing for an IB-based fat tree topology is described below. As a fat tree routing algorithm, vSwitchFatTree recursively traverses the fat tree topology to set LFTs on all switches for the LID associated with each VM in the subnet. This mechanism is deterministic and supports destination-based routing where back-calculation for all routes starts from the destination node.

[0097] Weighted fat tree routing algorithm for virtualized subnets

[0098]

number

[0099] In one embodiment, the vSwitchFatTree routing mechanism works as follows: Each VM is assigned a proportional weight. This proportional weight is calculated by dividing the weight of the vSwitch node (which is obtained, for example, as a constant of 1) by the total number of VMs running on it. Various weighting schemes can also be implemented. For example, one could choose an implementation to assign weights based on VM type. However, for the sake of brevity, this explanation will focus on the proportional weighting scheme. For each leaf switch, the routing mechanism sorts the connected vSwitches in decreasing order based on the connected VMs (line 3). This order ensures that higher-weighted VMs are routed first, thus balancing the routes assigned to the links. The routing mechanism recursively assigns paths to VMs within the tree by calling ROUTEDOWNGOINGBYGOINGUP (line 10), traversing the tree from each VM, through all leaf switches and their corresponding vSwitches. Downlink ports on each switch are selected based on the minimum cumulative downlink weight among all available uplink ports (ROUTEDOWNGOING GBYGOINGUP; line 16). When a downlink port is selected, the mechanism can increase the cumulative downlink weight for the corresponding port by the weight of the routed VM (ROUTEDOWNGOINGBYGOINGUP; line 19). After the downlink port is configured, the routing mechanism can assign an uplink port for the route to the VM on all connected downlink switches by descending the tree (updating the corresponding uplink weight for the port). )(ROUTEUPGOINGBYGOINGDOWN; line 20). The process is then repeated by moving up to the next level in the tree. Once all VMs are routed, the algorithm also routes the physical LIDs of the vSwitch in the same way as the VMs, even though equal weighting is used to balance the vSwitch routes in the topology (not illustrated in the pseudocode). This is desirable to improve balancing when a minimal reconfiguration method is used in the context of live migration. Also, the routing paths on the underlying physical LIDs of the vSwitch can be used as predetermined paths to quickly deploy new VMs without requiring reconfiguration. However, over a period of time, the overall routing performance will be slightly reduced compared to the original vSwitchFatTree routing. To limit the performance degradation, a reconfiguration based on vSwitchFatTree may be performed offline when a certain performance threshold is exceeded.

[0100] In one embodiment, the routing mechanism described above can offer various improvements over conventional / legacy routing mechanisms. Unlike the original fat tree routing algorithm, which does not take into account vSwitches or VMs in the topology, vSwitchFatTree marks vSwitches and routes each VM independently of other VMs connected to them. Similarly, to ensure uneven distribution of VMs across vSwitches, each VM is assigned a weight corresponding to the proportion of links allocated on the vSwitch. The weights are used to maintain port counters to balance the route distribution in the fat tree. The scheme also enables generalized weighted fat tree routing, in which case each VM can be assigned a weight based on its traffic profile or role priority in the network.

[0101] Figures 11 to 14 illustrate an example of supporting efficient virtualization in a lossless interconnect network according to one embodiment. Specifically, Figure 11 shows a two-level fat tree topology with four switches: root switches 925 and 926, leaf switches 920 and 921, and four virtual switches VS1 931, VS2 941, VS3 951, and VS4 961. The four virtual switches VS1 931, VS2 941, VS3 951, and VS4 961 are associated with four hosts / hypervisors 930, 940, 950, and 960, respectively. In this case, the four virtual switches are associated with eight virtual machines VM1 932, VM2 933, VM3 942, VM4 943, VM5 952, VM6 953, VM7 954, and VM Provides connectivity for 8962.

[0102] To further illustrate vSwitchFatTree routing, consider a simple virtualized fat tree topology with four end nodes (vSwitches), as shown in Figure 11. Each of the vSwitches connected to leaf switch 920, VS1, and VS2 has two running VMs (VM1 and VM2 for VS1, and VM3 and VM4 for VS2). The second leaf switch 921 has VS3 with three VMs (VM5, VM6, VM7), one of which is running on host vSwitch VS4. Since each leaf switch is connected to both root switches 925 and 926, there are two alternative paths available to set up routes to each VM via the roots. Routing for the VMs connected to VS1 is shown in Figure 12, using circles indicating the selected downlink paths from the root switches. VM1 is routed using 925→920, and VM2 is routed from 926→920. The corresponding downlink load counters are updated on the selected links, adding 0.5 for each VM. Similarly, as shown in Figure 13, after adding a route for VS2, VM3 and VM4 are routed via links 925→920 and 926→920, respectively. Note that after routing all VMs connected to leaf switch 920, the total downlink load on both links will be equal, even if the VMs were routed individually. The VM distribution on the vSwitch connected to leaf switch 921 is different, and therefore the vSwitch with one VM (VS4) will be routed first. Route 925→921 is assigned to VM8, and all three VMs connected to VS3 are routed from 926→921 to maintain a balanced cumulative load on both downlinks. In the final routing shown in Figure 14, assuming a VM distribution in the topology, the load is balanced on each link with independent routes to the VMs wherever possible.

[0103] Minimal overhead reconfiguration on virtual machine live migration In one embodiment, it is abbreviated as ItRC (Iterative Reconfiguration). The dynamic reconfiguration mechanism involves repeatedly switching and updating routes as needed when VMs are migrated. However, only a subset of switches actually need to be updated, depending on the existing LFTs in the subnet (i.e., LFTs that have already been calculated and are present on each switch within the subnet).

[0104] Figure 15 illustrates a potential virtual machine migration according to one embodiment. More specifically, Figure 15 shows a special case of VM migration within a leaf switch where, despite the network topology, only the corresponding leaf switch requires LFT updates.

[0105] As shown in Figure 15, the subnet consists of several switches, namely switch 1 Switches 1301 through 1312 may be included. Some of these switches may include leaf switches such as switch 1 1301, switch 2 1302, switch 11 1311, and switch 12 1312. The subnet may additionally include some hosts / hypervisors 1330, 1340, 1350, and 1360, and some virtual switches VS1 1331, VS2 1341, VS3 1351, and VS4 1361. Various hosts / hypervisors can host virtual machines within the subnet via virtual functionality, such as VM1 1332, VM2 1333, VM3 1334, VM4 1342, VM5 1343, and VM6 1352.

[0106] According to one embodiment, VM3 is associated with (as indicated by the bold arrow) When migrating from hypervisor 1330 to a free virtual function in hypervisor 1340, only the LFT on leaf switch 1301 needs to be updated. This is because both hypervisors are connected to the same leaf switch, and local changes do not affect the rest of the network. For example, the initial routing algorithm determines that traffic going from hypervisor 1360 to hypervisor 1330 follows a first path marked by a solid line (i.e., 12→9→5→3→1). Similarly, traffic going from hypervisor 1360 to hypervisor 1340 follows a second path marked by a dashed line (i.e., 12→10→6→4→1). If VM3 is migrated and ItRC is used to reconfigure the network, traffic heading to VM3 will follow a first route to hypervisor 1330 before the migration, and a second route to hypervisor 1340 after the migration. In this situation, assuming a fat tree routing algorithm was used for the initial routing, the ItRC method would update half (6 / 12) of the total number of switches. However, only one leaf switch needs to be updated to keep the migrated VM connected.

[0107] According to one embodiment, the network can be reconfigured more quickly by limiting the number of switch updates after VM migration, reducing the time and overhead required for traditional routing updates. This is based on topology-agnostic skyline technique. This can be achieved by a topology-aware, high-speed reconfiguration method called TreeMinRC, which supports VM migration on fat trees.

[0108] Subtrees and switch tuples in fat trees According to one embodiment, the following description utilizes FTreeMinRC, a minimal overhead network reconfiguration method, with XGFT as an exemplary fat-tree network. However, the concepts presented herein are also valid for PGFT and RLFT. XGFT(n;m1,...,m n ;w1,...,w n ) is a fat-tree with n+1 levels of nodes. The levels are denoted from 0 to n, with compute nodes at level n and switches at all other levels. Except for the leaf compute nodes that have no children, all nodes at level i, where 0≦i≦n-1, have m i child nodes. Similarly, except for the root switch that has no parent, all other nodes at level i, where 1≦i≦n, have w i +1 parent nodes.

[0109] [Formula]

[0110] According to one embodiment, each switch in the n+1-level XGFT can be represented by a unique n-tuple (l,x1,x2,...,x n ). The leftmost tuple value (l) represents the level where the tree is located, and the remaining values (x1,x2,...,x n ) represent the position of the switch in the tree relative to other switches. In particular, switch A(l,a1,...,a at level l l ,...,a n ) has a for all values except when i=l+1 i =b i if and only if it is connected to switch B at level l+1, (l+1,b1,...,b l ,b l+1 ,...,b n ).

[0111] Figure 16 shows a switch tuple according to one embodiment. More specifically, the figure shows a switch tuple as it would be allocated by the OpenSM fat tree routing algorithm implemented for the exemplary fat tree XGFT(4;2,2,2,2;2,2,2,1). Fat tree 1400 may include switches 1401-1408, 1411-1418, 1421-1428, and 1431-1438. Since the fat tree has n=4 switch levels (marked as column 0 at the root level up to column 3 at the leaf level), the fat tree consists of first-level subtrees of m1=2, each of which has n'=n-1=3 switch levels. This is shown in the figure by two boxes defined by dashed lines enclosing the switches from levels 1 to 3. Each first-level subtree receives an identifier of 0 or 1. Each of the first-level subtrees consists of a second-level subtree of m2=2, which is a switch level of n″=n′-1=2, each above a leaf switch. This is shown in the diagram by four boxes defined by dotted lines enclosing the switches from level 2 to 3. Each of the second-level subtrees receives an identifier of 0 or 1. Similarly, each leaf switch can also be considered a subtree, shown in the diagram by eight boxes defined by dashed lines. Each of these subtrees receives an identifier of 0 or 1.

[0112] According to one embodiment, as illustrated in the figure, a tuple, such as a tuple of four numbers, can be assigned to various switches, where each number in the tuple indicates a specific subtree correspondence for the position of each value in the tuple. For example, switch 1413 (which may be referred to as switch 1_3) can be assigned to tuple 1.0.1.1, which represents its position at level 1 and the 0th first-level subtree.

[0113] Fat tree-recognized minimal reconstruction using FTreeMinRC in the context of live migration In one embodiment, a switch tuple encodes information about the location of switches corresponding to subtrees in the topology. FTreeMinRC can use this information to enable rapid reconfiguration in the case of live VM migration. The tuple information can be used to discover the skyline with the fewest number of switches that need to be reconfigured by the SM when the VM is migrated. In particular, when a VM is migrated between two hypervisors in a fat tree topology, the skyline representing the minimum number of switches that need to be updated is formed by all top-level switches of all subtrees involved in the migration.

[0114] According to one embodiment, when a VM is live-migrated, the switch marking mechanism can be initiated from both leaf switches. In this case, the source and destination hypervisors are connected and compare tuples of switches. If the tuples match, the mechanism can determine that the VM has migrated within the leaf switch. This ensures that only the corresponding leaf switch is marked for reconfiguration. However, if the tuples do not match, the uplink is traced from both the source and destination leaf switches. The switch located one level above is the top-level switch in the immediate supertree to which the leaf-level subtrees are connected, and is the only hop that may occur before reaching a leaf switch when traversing the tree downwards. The mechanism can then compare the source leaf switch tuple and destination leaf switch tuple with the newly traced switch, adjust the tuple values ​​to reflect the current level, and then use the value corresponding to the subtree of the tree at that point as a wildcard. Furthermore, the traced switch (which is the top-level switch for the corresponding subtree) is marked to be updated, and the trace is stopped if the comparison from both the source and destination switch tuples matches the tuples of all the traced switches. Otherwise, the same procedure is repeated until the mechanism identifies a common ancestor switch from both ends. In the worst case, the mechanism can be stopped after reaching the root switch of the fat tree topology. Since all uplink traces start at the leaf level and mark consecutive subtree skyline switches, when the mechanism reaches the top-level subtree affected by the migration, it has already selected all switches that are potential traffic gateways to lower-level switches, as well as hypervisors involved in the live migration, along the way. As a result, the mechanism has marked all switches that form the skyline of the portion of the network affected by the live migration.

[0115] According to one embodiment, the switch marking mechanism discovers the minimum number of switches that need to be updated in terms of physical connectivity. However, not all of these switches may contain active routes calculated by the routing algorithm for LIDs affected by the reconfiguration. Therefore, switches containing active routes are prioritized in the update procedure, while the remaining switches with secondary routes can be updated later.

[0116] In one embodiment, the fat tree routing mechanism always routes traffic to a given destination through the same route switch. Since there is only a single path between the route switch and the end node in the topology, once the route switch selected to represent a given end node is located, the intermediate switches used to route traffic to the end node can be found. To discover the active route, the route can be traced from the source LID to the destination LID of the hypervisors involved, or vice versa. A subset of switches that have already been selected for reconfiguration can be marked, and the LFT updates of those switches can be prioritized. The remaining selected switches can then be updated to keep all LFTs active.

[0117] Figure 17 shows a reconstruction process according to one embodiment. The fat tree 1400 may include switches 1401-1408, 1411-1418, 1421-1428, and 1431-1438. Since the fat tree has n=4 switch levels (marked as column 0 at the root level, up to column 3 at the leaf level), the fat tree consists of first-level subtrees m1=2, each of which is a switch level n′=n-1=3. Each of these first-level subtrees consists of a second-level subtree m2=2, each of which is a switch level n″=n′-1=2, each of which is above the leaf switches. Similarly, each leaf switch can also be considered a subtree.

[0118] In one embodiment, Figure 17 shows a VM being migrated between two hypervisors connected to a leaf switch with tuples 3.0.0.0 and 3.0.1.1. These two tuples are used as a reference for comparison when the mechanism traces the path upward from the selected leaf switch. In this example, a common ancestor switch is found on level 1. Level 0 is the root level, and level 3 is the leaf level. Links between switches with the displayed tuple information are links that can be traced throughout the execution of the mechanism, and all of those same switches can be marked for updates. The five highlighted switches (switches 1431, 1421, 1411, 1423, and 1434) and the links between them represent active routes, and their LFT updates can be prioritized.

[0119] According to one embodiment, FTreeMinRC minimizes the number of LFT updates that need to be sent to the switch in order to provide rapid connectivity with minimal overhead in a virtualized data center that supports live migration.

[0120] Figure 18 is a flowchart of a method for supporting efficient virtualization in a lossless interconnect network according to one embodiment. In step 1810, the method can provide one or more computers, each including one or more microprocessors, with one or more switches, each including at least one leaf switch, each of the one or more switches including multiple ports, and further, a plurality of host channel adapters, each of which includes at least one virtual function, at least one virtual switch, and at least one physical function, and the plurality of host channel adapters are interconnected via one or more switches, and further, a plurality of hypervisors can be provided, each of which is associated with at least one host channel adapter from the plurality of host channel adapters, and further, a plurality of virtual machines can be provided, each of which is associated with at least one virtual function.

[0121] In step 1820, the method can deploy multiple host channel adapters, each having one or more virtual switches with a prepopulated local identifier (LID) architecture or a dynamic LID assignment architecture.

[0122] In step 1830, the method can assign a LID to each virtual switch, and the assigned LID corresponds to the LID of the associated physical function.

[0123] In step 1840, the method can compute one or more linear forwarding tables based on at least the LIDs assigned to each of the virtual switches, and each of the one or more LFTs is associated with one of the one or more switches.

[0124] Figure 19 is a flowchart of a method for supporting efficient virtualization in a lossless interconnect network according to one embodiment. In step 1910, the method provides one or more computers, each containing one or more microprocessors, and one or more switches, where one or more switches include at least one leaf switch, each of the one or more switches includes multiple ports, and furthermore, multiple host channel adapters, each of the host channel adapters including at least one virtual function, at least one virtual switch, and at least one physical function, and the multiple host channel adapters interconnect via one or more switches. The system is connected and can have multiple hypervisors, each of which is associated with at least one host channel adapter from among multiple host channel adapters, and can have multiple virtual machines, each of which is associated with at least one virtual function.

[0125] In step 1920, the method can deploy multiple host channel adapters, each having one or more virtual switches with a prepopulated local identifier (LID) architecture or a dynamic LID assignment architecture.

[0126] In step 1930, the method can assign one of several pLIDs to each virtual switch. The assigned pLID corresponds to the pLID of the associated physical function.

[0127] In step 1940, the method may assign one of several vLIDs to each of several virtual machines, and the LID space may include several pLIDs and several vLIDs.

[0128] Many features of the present invention can be implemented using or with hardware, software, firmware, or a combination thereof. Therefore, features of the present invention can be realized using a processing system (for example, including one or more processors).

[0129] The features of this invention can be realized by using or with the help of a computer program product which is a storage medium or computer-readable medium storing instructions that can be used to program a processing system to perform any of the features presented herein. The storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0130] Features of this invention can be incorporated into software and / or firmware, stored on any machine-readable medium, to control the hardware of a processing system and to enable the processing system to interact with other mechanisms that utilize the results of this invention. Such software or firmware may include, but is not limited to, application code, device drivers, operating systems, and execution environments / containers.

[0131] Another feature of this invention is that it uses hardware components such as application-specific integrated circuits (ASICs) to enable hardware This may be implemented in the software. Implementing a hardware state machine to perform the functions described herein will be obvious to those skilled in the art.

[0132] In addition, this invention can be conveniently implemented using one or more conventional general-purpose or specialized digital computers, computing devices, machines, or microprocessors, including one or more processors, memory and / or computer-readable storage media programmed in accordance with the teachings of this disclosure. As will be apparent to those skilled in the art of software technology, appropriate software coding based on the teachings of this disclosure can be performed by skilled It can be easily prepared by a programmer.

[0133] While various embodiments of this invention have been described above, it should be understood that they are presented for illustrative purposes only and not for limitation. It will be apparent to those skilled in the art that various modifications of shape and detail can be made without departing from the spirit and scope of this invention.

[0134] This invention has been described above with the help of functional building blocks that demonstrate the execution of specified functions and their relationships. For convenience of explanation, the boundaries of these functional building blocks have often been arbitrarily defined in this specification. Alternative boundaries can be defined as long as the specified functions and their relationships are adequately performed. For this reason, any such alternative boundaries are included within the scope and spirit of this invention.

[0135] The foregoing description of this invention has been provided for illustrative and explanatory purposes only. It is not intended to be exhaustive or to limit the invention to the disclosed forms themselves. The scope and breadth of this invention should not be limited by any of the exemplary embodiments described above. Many modifications and variations will be apparent to those skilled in the art. These modifications and variations include any relevant combination of the disclosed features. The embodiments have been selected and described to best illustrate the principles of this invention and its practical applications, thereby enabling those skilled in the art to understand the invention, with respect to various embodiments suitable for specific conceivable uses, and with respect to various modifications. The scope of this invention is intended to be defined by the claims and their equivalents.

Claims

1. A system for supporting efficient virtualization in lossless interconnected networks, One or more microprocessors, A subnet including multiple switches, wherein the multiple switches provide point-to-point connectivity between multiple host channel adapters, each host channel adapter has a corresponding hypervisor running on it from among multiple hypervisors, and each of the multiple switches is assigned a switch tuple that encodes information about the location of the switch. The virtual machine performs a migration from the first hypervisor among the multiple hypervisors to the second hypervisor among the multiple hypervisors. As a result of the migration of the virtual machine, the subnet manager compares the first switch tuple of the leaf switch to which the first hypervisor is connected with the second switch tuple of the leaf switch to which the second hypervisor is connected, and if the first switch tuple and the second switch tuple match, it determines that the virtual machine has been migrated within the same leaf switch and updates only one linear forwarding table on the same leaf switch within the subnet.

2. The system according to claim 1, wherein the subnet manager updates only one linear forwarding table on the same leaf switch among the plurality of switches by sending a command via a subnet management packet to the same leaf switch within the subnet.

3. The system according to claim 2, wherein the first hypervisor runs on a first host channel adapter, and the second hypervisor runs on a second host channel adapter.

4. The system according to claim 3, wherein the first host channel adapter and the second host channel adapter are connected to the leaf switch.

5. A method for supporting efficient virtualization in a lossless interconnect network, The steps include providing a computer that includes one or more microprocessors, The method further includes the step of providing a subnet that includes a plurality of switches, wherein the plurality of switches provide point-to-point connectivity between a plurality of host channel adapters, each host channel adapter has a corresponding hypervisor running on it from a plurality of hypervisors, and each of the plurality of switches is assigned a switch tuple that encodes information about the location of the switch, and the method further includes The steps include migrating the virtual machine from a first hypervisor among the plurality of hypervisors to a second hypervisor among the plurality of hypervisors, As a result of the migration of the virtual machine, the subnet manager compares the first switch tuple of the leaf switch to which the first hypervisor is connected with the second switch tuple of the leaf switch to which the second hypervisor is connected, and if the first switch tuple and the second switch tuple match, it determines that the virtual machine has been migrated within the same leaf switch, and updates only one linear forwarding table on the same leaf switch within the subnet.

6. The method according to claim 5, wherein the subnet manager updates only the one linear forwarding table on the same leaf switch among the plurality of switches by sending a command via a subnet management packet to the same leaf switch among the plurality of switches.

7. The method according to claim 6, wherein the first hypervisor runs on a first host channel adapter, and the second hypervisor runs on a second host channel adapter.

8. The method according to claim 7, wherein the first host channel adapter and the second host channel adapter are connected to the leaf switch.

9. A computer-readable program for causing one or more microprocessors to perform the method according to any one of claims 5 to 8.

Images