Virtual device composition in scalable input / output (I / O) virtualization (S-IOV) architecture

Abstract

Virtual device composition in scalable input / output (I / O) virtualization (S-IOV) architectures is provided. An example can include a method to instantiate a virtual machine; instantiate a virtual device to transmit data to and receive data from an allocated resource of a shared physical device; and assign the virtual device to the virtual machine, the virtual machine to transmit data to and receive data from the physical device via the virtual device.

Description

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Application No. 62 / 721,483, filed on August 22, 2018. Background Technology

[0002] The introduction of the Single Root I / O Virtualization (SR-IOV) and Shared Specification (Version 1.1), released by the Peripheral Component Interconnect (PCI) Special Interest Group (PCI-SIG) on January 20, 2010, was a significant advancement towards hardware-assisted high-performance I / O virtualization and sharing for PCI Express devices. The PCI Express Foundation Specification (Revision 4.0, Version 1.0), released on October 5, 2017, defines PCI Express (PCIe). Since then, in addition to deploying virtual machines (VMs) for computer server consolidation, the computing landscape has evolved to include hyperscale data centers requiring seamless resource addition and dynamic provisioning of containers. These new computing environments demand increased scalability and flexibility for I / O virtualization. Attached Figure Description

[0003] Figure 1 The example computing system is illustrated.

[0004] Figure 2 The diagram illustrates some of the high-level differences between the SR-IOV architecture and the Scalable I / O Virtualization (IOV) architecture.

[0005] Figure 3 The illustration shows some of the differences between an endpoint device with SR-IOV capability and an endpoint device with Scalable IOV capability.

[0006] Figure 4 The diagram illustrates an example of a high-level software architecture for scalable IOV.

[0007] Figure 5 The diagram illustrates an example of a logical view of virtualization software that utilizes a variable number of device backend resources, including an Assignable Device Interface (ADI), and one or more ADIs to form a Virtual Device (VDEV) instance.

[0008] Figure 6 The diagram illustrates a sample Scalable IOV specification vendor-specific scalability (DVSEC) architecture.

[0009] Figure 7 The diagram illustrates an example high-level translation structure organization for scalable mode address translation.

[0010] Figure 8The diagram illustrates an example of virtual device composition.

[0011] Figure 9 The diagram illustrates an example flowchart of virtual device construction.

[0012] Figure 10 The diagram illustrates an example of a virtual device configuration that utilizes physical functionality ADI and interrupt message service (IMS) resources.

[0013] Figure 11 The diagram illustrates an example flowchart for modifying virtual device resources.

[0014] Figure 12 The diagram illustrates an example flowchart for virtual device decomposition.

[0015] Figure 13 An example of a storage medium is illustrated.

[0016] Figure 14 Another example computing platform is illustrated. Detailed Implementation

[0017] Embodiments of this invention disclose a Scalable I / O Virtualization (Scalable IOV) architecture and associated host computing platforms and endpoint device capabilities. Scalable IOV defines a scalable and flexible approach to hardware-assisted I / O virtualization targeting hyperscale usage. Scalable IOV builds upon the existing set of PCI Express capabilities, enabling the Scalable IOV architecture to be easily supported by compatible PCI Express endpoint device designs and the existing software ecosystem.

[0018] Virtualization allows system software called a Virtual Machine Monitor (VMM) (also known as a hypervisor) to create multiple isolated execution environments called Virtual Machines (VMs), in which operating systems (OS) and applications can run. Virtualization is widely used in modern enterprises and cloud data centers as a mechanism to consolidate multiple workloads onto a single physical machine while still keeping the workloads isolated from each other. In addition to VMs, containers provide another type of isolated environment for encapsulating and deploying applications and running them in isolated processing environments. Containers are built as bare-metal containers instantiated as groups of OS processes or as machine containers that utilize hardware-supported, increased isolation properties for virtualization. Compared to VMs, containers are lighter and can be deployed at much higher densities, potentially increasing the number of container instances on a computing platform by an order of magnitude.

[0019] Modern processors offer features to reduce virtualization overhead, which can be leveraged by the VMM to allow the VM direct access to hardware resources. (For IA-32 Intel)® Architecture (Intel) ® Intel VT-x ® Virtualization technology (Intel) ® Intel Virtualization (VT) defines the overhead for reducing processor and memory virtualization. ® Processor hardware capabilities. Used for targeted I / O (Intel) ® Intel VT-d) ® Virtualization technology (Intel) ® VT defines computing platform hardware features for direct memory access (DMA) and interrupt remapping and isolation, which can be used to minimize the overhead of I / O virtualization. I / O virtualization refers to the virtualization and sharing of I / O devices across multiple VMs or container instances. Several methods exist for I / O virtualization, which can be broadly classified as software-based or hardware-assisted.

[0020] By leveraging software-based I / O virtualization, the Virtual Machine Model (VMM) exposes virtual devices (such as network interface controller (NIC) functionality) to the Virtual Machine (VM). The software device model within the VMM, or the host OS, emulates the behavior of the virtual devices. The software device model translates virtual device commands into physical device commands before forwarding them to the physical device. This software emulation of devices provides compatibility with software running within the VM but incurs significant performance overhead, especially for high-performance devices. Beyond performance limitations, emulating virtual device access in software can be too complex for programmable devices such as graphics processing units (GPUs) and field-programmable gate arrays (FPGAs) because these devices perform multiple functions compared to a single fixed set. Variants of software-based I / O virtualization, such as “device paravirtualization” and “mediated pass-through,” allow computing platforms to mitigate some of the performance and complexity drawbacks of device emulation.

[0021] To avoid the overhead of software-based I / O virtualization, VMMs can leverage platform support for DMA and interrupt remapping capabilities (such as Intel). ® VT-d) supports "direct device allocation," allowing guest software to directly access the allocated device. This direct device allocation provides the best I / O virtualization performance because the VMM no longer hinders most guest software from accessing the device. However, this method requires the exclusive allocation of the device to the VM and does not support sharing the device across multiple VMs.

[0022] Single Root I / O Virtualization (SR-IOV) is a PCI-SIG specification for hardware-assisted I / O virtualization, defining a standard way to partition endpoint devices for direct sharing across multiple VMs or containers. SR-IOV-enabled endpoint devices can support one or more Physical Functions (PFs), each of which can support multiple Virtual Functions (VFs). The PF acts as a resource management entity for the device and is managed by the PF driver in the host OS. Each VF can be assigned to a VM or container for direct access. SR-IOV is supported through multiple high-performance I / O devices (such as network and storage controller devices) and programmable or reconfigurable devices (such as GPUs, FPGAs, and other emerging accelerators).

[0023] In some embodiments, SR-IOV is implemented using PCIe. In other embodiments, interconnects other than PCIe may be used.

[0024] With the proliferation of hyperscale computing models, coupled with an increasing number of processing elements (e.g., processing cores) on modern processors, high-capacity computing platforms (e.g., compute servers) are being used to host an order of magnitude more bare metal or machine containers than traditional VMs. Many of these uses, such as Network Functions Virtualization (NFV) or heterogeneous computing leveraging accelerators, require high-performance hardware-assisted I / O virtualization. These dynamically provisioned, high-density uses (i.e., approximately 1,000 domains) require more scalable and fine-grained I / O virtualization solutions than those offered by traditional virtualization uses supported by devices with SR-IOV capabilities.

[0025] Scalable IOV, as described in the embodiments of the invention herein, is a novel approach to hardware-assisted I / O virtualization that enables highly scalable and high-performance sharing of I / O devices across isolated domains, while incorporating the costs and complexities of enabling endpoint device hardware to support such scalable sharing. Depending on the use model, the isolated domain can be a traditional VM, machine container, bare-metal container, or application process. While embodiments of the invention primarily refer to the isolated domain as a VM, the general principles are broadly applicable to other domain abstractions, such as containers.

[0026] Figure 1 The illustration shows an example computing system 100 that supports a scalable IOV architecture. Figure 1 As shown, computing system 100 includes a computing platform 101 coupled to network 170 (which may be, for example, the Internet or a network within a data center). In some examples, such as Figure 1As shown, computing platform 101 is coupled to network 170 via network communication channel 175 and through at least one network I / O device 110 (e.g., network interface controller (NIC)). The at least one network I / O device 110 has one or more ports connected to or coupled to network communication channel 175. In an embodiment, network I / O device 110 is an Ethernet NIC. Network I / O device 110 transmits data packets from computing platform 101 to other destinations through network 170 and receives data packets from other destinations for forwarding to computing platform 101.

[0027] Based on some examples, such as Figure 1 As shown, the computing platform 101 comprises a circuitry 120, main memory 130, network (NW) I / O device drivers 140, an operating system (OS) 150, a virtual machine manager (VMM) 180 (also referred to as a hypervisor), at least one application 160, and one or more storage devices 165. In one embodiment, the OS 150 is Linux. TM In another embodiment, OS 150 is a Windows® server. In this embodiment, application 160 represents one or more applications executed by one or more guest VMs (not shown). Network I / O device driver 140 operates to initialize and manage I / O requests executed by network I / O device 110. In this embodiment, packets and / or packet metadata transmitted to and / or received from network I / O device 110 are stored in one or more of main memory 130 and / or storage devices 165. In at least one embodiment, storage device 165 may be one or more of hard disk drives (HDDs) and / or solid-state drives (SSDs). In this embodiment, storage device 165 may be non-volatile memory (NVM). In some examples, such as... Figure 1 As shown, circuit system 120 can be communicatively coupled to network I / O device 110 via communication link 155. In one embodiment, communication link 155 is a Fast Peripheral Component Interconnect (PCIe) bus conforming to PCIe revision 4.0 or other versions of the standard. In some examples, operating system 150, NW I / O device driver 140, and application 160 are implemented at least in part via cooperation between one or more memory devices included in main memory 130 (e.g., volatile or non-volatile memory devices), storage device 165, and elements of circuit system 120 (such as processing cores 122-1 to 122-m), wherein " m" is any positive integer greater than 2. In an embodiment, OS 150, NW I / O device driver 140, and application 160 are executed by one or more processing cores 122-1 to 122-m. In other embodiments, there are other endpoint devices coupled to communication link 155 (e.g., PCIe interconnect) that support scalable IOV capabilities.

[0028] In some examples, computing platform 101 includes, but is not limited to, computer servers, server arrays or server farms, web servers, network servers, internet servers, workstations, microcomputers, mainframes, supercomputers, network appliances, web appliances, distributed computing systems, multiprocessor systems, processor-based systems, laptop computers, tablet computers, smartphones, or combinations thereof. In one example, computing platform 101 is a decomposed server. A decomposed server is a server that distributes components and resources across subsystems and connects them via network connections (e.g., network sleds). A decomposed server can adapt to changing storage or computing loads as needed without replacing or interrupting the entire server for extended periods. For example, the server can be divided into modular computing, I / O, and storage modules that can be shared among other nearby servers.

[0029] The circuit system 120 having processing cores 122-1 to 122-m may include a variety of commercially available processors, including but not limited to Intel® Atom®, Celeron®, Core(2) Duo®, Core i3, Core i5, Core i7, Itanium®, Pentium®, Xeon® or Xeon Phi® processors, ARM processors, and similar processors. The circuit system 120 may include at least one cache 135 for storing data.

[0030] According to some examples, main memory 130 may include one or more memory devices or dies, which may include various types of volatile and / or non-volatile memory. Volatile memory may include, but is not limited to, dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (TRAM), or zero-capacitor RAM (ZRAM). Non-volatile memory may include byte- or block-addressable non-volatile memory with a three-dimensional (3D) crosspoint memory structure, including chalcogenide phase change materials (e.g., chalcogenide glass) hereinafter referred to as "3D crosspoint memory". Non-volatile memory may also include other types of byte- or block-addressable non-volatile memory, such as, but not limited to, multi-threshold level NAND flash memory, NOR flash memory, single-level or multi-level phase change memory (PCM), resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), magnetoresistive random access memory (MRAM) incorporating memristor technology, spin-transfer torque MRAM (STT-MRAM), or combinations of any of the above. In another embodiment, main memory 130 may include one or more hard disk drives within and / or accessible by computing platform 101.

[0031] Figure 2 Example diagram 200 illustrates some high-level differences between the SR-IOV 202 and Scalable IOV 222 architectures. Unlike the coarse-grained device partitioning and allocation method 208 of SR-IOV used to create multiple VF 214s on PF 212, the Scalable IOV architecture 222 allows software to flexibly construct virtual devices with finer granularity, leveraging hardware assistance for device sharing. Frequent (i.e., performance-critical) operations on the constructed virtual devices are directly mapped to the underlying device hardware (e.g., Scalable IOV device 230), while infrequent operations are simulated by device-specific constructing software 226 in the VMM / host OS 228. This differs from the existing architecture used for SR-IOV device 202, where only software is used to virtualize the device-independent PCI Express architecture resources of the virtual device (such as configuration space registers and message signal interrupt extension (MSI-X) capability registers), and the rest of the virtual device resources (including all other memory-mapped I / O (MMIO)) are directly mapped to the underlying VF 214 hardware resources (e.g., SR-IOV device 216).

[0032] In the SR-IOV architecture 202 using hardware replication, multiple VMs and containers 204 run on top of the VMM and / or host OS 206. Device partitioning and allocation logic 208 assigns I / O requests to PF drivers 210 coupled to physical functions (PF) 212 in the SR-IOV device 216, or virtual functions (VF) 214 in the SR-IOV device 216. In contrast, in the scalable IOV architecture 222 of this embodiment of the invention using replication and configuration, a much larger number of VMs and containers 224 are supported. VMs and / or containers 224 invoke device configuration logic 226 in the VMM / host OS 228 to fulfill I / O requests. Device configuration logic 226 assigns I / O requests to PF drivers 210 coupled to fine-grained supplied device resources (which also include physical functions) in the scalable IOV device 230, or directly to fine-grained supplied device resources 232 in the scalable IOV device 230.

[0033] The scalable IOV architecture offers advantages over SR-IOV. Fine-grained provisioning of device resources to VM 224, along with software emulation of infrequent device access, allows devices to increase shared scalability with lower hardware cost and complexity. The scalable IOV architecture provides system software such as VMM / host OS 228 with the flexibility to share device resources using different abstractions across different address domains (e.g., application 160 processes accessed via system calls and / or VM / container 224 accessed via virtual device interfaces). Through dynamic mapping of virtual devices (VDEVs) to device resources via software control, embodiments of the scalable IOV architecture of this invention also enable the VMM to overprovision device resources to VM 224.

[0034] This approach also enables VMM 228 to easily maintain generational compatibility in data centers. For example, in a data center with physical machines of different generations (e.g., versions) containing the same I / O devices, the VMM can use software emulation to virtualize the VDEV's MMIO-based capability registers to represent the same VDEV capabilities regardless of the different generations of physical I / O devices. This is to ensure that the same guest OS image as the VDEV driver can be deployed or migrated to any of the physical machines.

[0035] The Scalable IOV architecture comprises the following components. This architecture supports PCI Express endpoint device requirements and capabilities. This architecture supports host platform (e.g., root federation) requirements, including enhancements to Direct Memory Access (DMA) remapping hardware. In embodiments, these requirements are implemented as part of Intel® virtualization technologies for directed I / O, Rev 3.0, or higher, on Intel processor-based computing platforms. This architecture also supports a reference software architecture envisioned to enable Scalable IOV, including host system software (OS and / or VMM 228) that enables the implementation of infrastructure and endpoint device-specific software components such as host drivers, guest drivers, and Virtual Device Constituent Modules (VDCMs).

[0036] PCI Express endpoint devices can support the requirement to operate using Scalable IOV without being tied to their support for SR-IOV. This enables device implementations that already support SR-IOV to maintain that capability for backward compatibility while adding additional capabilities to support Scalable IOV.

[0037] In embodiments of the present invention, the endpoint physical function enables SR-IOV and Intel Scalable IOV to operate in one mode or the other, rather than simultaneously.

[0038] The PCI Express SR-IOV architecture follows a near-full hardware replication of the physical function (PF) 212 hardware used for its virtual functions (VF) 214. This is achieved through the majority of SR-IOV device implementations via the hardware / software interface that replicates the PF for each of its VFs, including resources such as memory-mapped resources, MSI-X storage, and capabilities such as function-level reset (FLR). Such a function replication approach can increase device complexity and impose limitations on scaling to a large number of VMs.

[0039] Hardware-software interfaces for I / O controller implementations can be categorized into (a) less frequent, slow-path control / configuration operations with minimal impact on overall device performance; and (b) frequent, fast-path command / completion operations with a significant impact on overall device performance. This distinction between slow and fast-path operations is practiced in many high-performance I / O devices that support direct user-mode access. Scalable IOV architectures extend such device designs to define software-configurable approaches to I / O virtualization and sharing.

[0040] The Scalable IOV architecture requires endpoint devices (i.e., the Scalable IOV device 230) to organize their hardware / software interfaces into fast (frequent) and slow (infrequent) access paths. Which operations and accesses are categorized as slow or fast paths is controlled by the device implementation. Slow path accesses typically include initialization, control, configuration, management, error handling, and reset operations. Fast path accesses typically involve data path operations related to job submission and job completion processing. Using this organization, slow path accesses from guest VMs to the virtual device are captured and simulated via device-specific host software, while fast path accesses are directly mapped to the physical device. This approach enables simplified device design (compared to a full functional replication of SR-IOV) without compromising I / O virtualization scalability or performance. Furthermore, this hybrid approach provides increased software flexibility to construct virtual devices through fine-grained provisioning of device resources.

[0041] High-performance I / O devices support a large number of command / completion interfaces for efficient multiplexing / demultiplexing of I / O requests and, in some uses, support user-mode I / O requests. Several examples of such devices are: a) high-bandwidth network controllers supporting thousands of transmit / receive (TX / RX) requests across a large number of Virtual Switch Interfaces (VSIs); b) memory controllers supporting numerous command and completion queue pairs, such as NVM Express (as described in the Non-Volatile Memory (NVM) Express specification (version 1.3c) available at nvmexpress.org); c) accelerator devices supporting a large number of graphics and / or computing contexts, such as GPUs; d) reconfigurable FPGA devices with Accelerator Functional Units (AFUs) supporting a large number of execution contexts; and e) devices with Remote Direct Memory Access (RDMA) capabilities supporting thousands of queue pairs (QP) interfaces.

[0042] The Scalable IOV architecture leverages high-performance I / O device designs with multi-queue / multi-context capabilities and defines methods for sharing these devices at a finer granularity (queues, queue bundles, contexts, etc.) than SR-IOV VF. To achieve this finer-grained sharing, embodiments of the Scalable IOV architecture define the granularity of device sharing as an "Assignable Device Interface" (ADI) on the device. According to embodiments, an ADI is an allocation unit for devices with Scalable IOV capabilities. Each ADI instance on the device encompasses a set of resources on the device that are software-distributed to support fast-path operations for Virtual Devices (VDEVs).

[0043] Conceptually, ADI is similar to SR-IOV Virtual Functions (VF), except that it is more fine-grained and maps to fast path operations for virtual devices. Unlike VF, all ADIs on a Physical Function (PF) share the PF's requester ID (e.g., bus / device / function number), have no PCI configuration space registers, share the same Base Address Register (BAR) resources of the PF (i.e., no Virtual Function Base Address Register (VFBAR)), and do not require copied MSI-X stores. Instead of an MSI-X table store for each ADI, the PF implements a device-specific Interrupt Message Store (IMS). The IMS is intentionally similar to the MSI-X table store, but not architecturally, and is instead implemented in a device-specific manner for maximum flexibility. Furthermore, compared to the implementation of VF... Unlike some SRIOV devices with PF communication channels and their on-device "resource remapping logic," ADI uses slow path emulation to provide this functionality. ADI's memory-mapped register space is laid out so that fast path registers reside in a separate system page-size region from slow path registers. The host driver for devices with Scalable IOV capabilities defines a set of grouped device backend resources that form ADI's capabilities.

[0044] Figure 3 Example diagram 300 illustrates some differences between endpoint devices with SR-IOV capability and those with Scalable IOV capability. SR-IOV device 216 includes a Physical Function (PF) Base Address Register (PF BAR) 302, a PF Configuration (PFconfig) circuitry 304, and a PF Message Signal Interrupt Extension (MSI-X) circuitry 306. PF MSI-X 306 provides MSI-X capabilities as defined by the PCI Express Basic Specification. SR-IOV device 216 also includes multiple sets of Virtual Function (VF) BAR 312, VF config 314, and MSI-X 316. PF BAR 302 and VF BAR 312 are coupled to device resource remapping logic and VF... PF mailbox logic 308 invokes device backend resource 310. In an embodiment, device backend resource 310 includes multiple queues for storing packets. Device backend resource 310 may include command / status registers, queues on the device, references to queues in memory, local memory on the device, or any other device-specific internal configuration.

[0045] The scalable IOV device 230 includes a PF BAR 320, which also includes multiple ADI MMIO components 322. The PF BAR 320 is coupled with a PF config 324, a PF MSI-X 326, and an Interrupt Message Store (IMS) for ADI 328. The PF MSI-X 326 provides MSI-X capabilities as defined by the PCI Express Basic Specification. The IMS 328 enables the device to store interrupt messages for ADI in a device-specific optimized manner without the scalability constraints of the PCI Express-defined MSI-X capabilities. The PF BAR 320 and ADI MMIO components 322 are coupled with device backend resources 330. Device backend resources 330 may include command / status registers, queues on the device, references to queues in memory, local memory on the device, or any other device-specific internals.

[0046] ADI's device-specific and lightweight nature, along with the flexibility in emulating various parts of virtual device functionality in device-specific host software, enables device hardware implementations to construct a large number of virtual devices for scalable sharing at a lower device cost and complexity compared to equivalent extensions of SR-IOV VF.

[0047] Utilizing the SR-IOV architecture, each VF 214 in the SR-IOV device 216 is identified by a PCI Express requester identifier (ID) (RID), thereby allowing DMA remapping hardware support (such as Intel) in the root federation. ® VT-d) applies a unique address translation function to upstream requests from VF. RID is a bus, device, and function number identity used for PCI Express PF or VF. RID is also used to route transactions at the PCI Express device level (such as read completions), and can therefore be a rare resource on some platform topologies with large I / O fanout designs. This can impose scalability limitations on the number of isolation domains that SR-IOV devices can support.

[0048] The scalable IOV architecture of embodiments of the present invention addresses platform scalability issues by sharing the RID of the Physical Function (PF) 232 with all its ADIs and, alternatively, assigning Process Address Space Identifiers (PASIDs) to the ADIs. The PASIDs are communicated in upstream transactions using the PCI Express PASID Transaction Layer Packet (TLP) prefix. Refer to the PCI Express specification for details regarding the PASID TLP prefix. Support for the scalable IOV architecture by the computing platform 101 enables unique address translation for upstream requests at the PASID granularity. Unlike the RID, the PASID is not used for transactions routed on I / O structures, but instead is used solely to communicate the address space targeted by memory transactions. Furthermore, the PASID is a 20-bit ID compared to the 16-bit RID, providing 16x more identifiers. This use of PASIDs in the scalable IOV architecture allows scalable IOV devices to support significantly more domains.

[0049] Figure 4 The diagram illustrates an example of a high-level software architecture 400 for scalable IOV. Figure 4 The diagram illustrates the components used to describe the scalable IOV architecture and is not intended to illustrate all virtualization software or specific implementation choices. To support a broad range of device classifications and implementations, software responsibilities are abstracted between system software (OS 150 / VMM 180) and device-specific driver software components.

[0050] thus, Figure 4 This describes the hardware-assisted virtual devices, along with the system software (host OS150 and VMM 180) that manage device operation, and the device-specific software roles and interactions. The described software architecture focuses on I / O virtualization for virtual machines and machine containers. However, the principles can be applied using appropriate software support for other domains, such as I / O sharing across bare metal containers or application processes.

[0051] The Scalable IOV architecture of embodiments of the present invention introduces a device-specific software component called Virtual Device Construction Module (VDCM) 402, which is responsible for constructing one or more Virtual Device (VDEV) 404 instances using one or more Assignable Device Interfaces (ADIs) 406, 408. The VDCM does this by emulating slow path operations / accesses of the VDEVs and mapping fast path accesses of the VDEVs to ADI instances allocated and configured on the physical devices. Unlike SRIOV VF 214, VDCM 402 allows the Scalable IOV device 230 to avoid implementing slow path operations in hardware and instead centralizes the device hardware for efficiently scaling ADIs.

[0052] Additionally, virtualization management software (e.g., VMM 180) utilizes the VDCM 402 software interface for enhanced virtual device resource and state management, enabling capabilities such as suspending, restarting, resetting, and migrating virtual devices. Depending on the specific VMM implementation, VDCM 402 can be instantiated as a separate user or kernel module, or it can be encapsulated as part of a host driver.

[0053] The host driver 412 for device 230 with Scalable IOV capabilities is conceptually equivalent to the SR-IOV PF driver 210. Host driver 412 is loaded and executed as part of the host OS 150 or VMM (hypervisor) software 180. In addition to its normal device driver role, host driver 412 implements software interfaces as defined by the host OS 150 or VMM 180 infrastructure to support the enumeration, configuration, instantiation, and management of multiple ADIs 428, 430, 432, and 434. Host driver 412 is responsible for configuring each ADI, such as its PASID identity, the device-specific interrupt message store (IMS) 328 for storing ADI interrupt messages, the MMIO register resource 322 for fast path access to the ADI, and any device-specific resources.

[0054] Table 1 illustrates a set of example high-level operations supported by host driver 412 for managing ADI. These operations are invoked through software interfaces implemented by specific system software, such as host OS 150 or VMM 180.

[0055] Table 1 – Host Driver Interfaces for Scalable IOV describe Scalable IOV capability for reporting PF. The types and maximum number of ADI / VDEVs are enumerated. An enumeration of resource requirements for each ADI type. Enumerations and settings for ADI deployment compatibility. The allocation, configuration, reset, exhaustion, termination, and release of ADI and its constituent resources. Set up and manage ADI's PASID identity. Manage the Device-Specific Interrupt Message Store (IMS) for ADI. This allows visitors to host communication channels (if supported). Configure ADI's device-specific QoS attributes. Enumerate and manage ADI migration compatibility. Suspend / save the state of ADI, and restore / restart the state of ADI.

[0056] The Virtual Device Construction Module (VDCM) 402 is a device-specific component responsible for constructing one or more instances of Virtual Devices (VDEVs) 404 using one or more ADIs 406, 408 dispatched by the host driver 412. VDCM 402 implements software-based virtualization for slow-path operations of the VDEV 404 and arranges fast-path operations to be submitted directly to the backing ADIs 428, 430, 432, 434. The host OS 150 or VMM 180 implementation supporting such hardware-assisted virtual device construction may require the VDCM to be implemented and packaged in different ways by the device vendor. For example, in some OS or VMM implementations, VDCM 402 is packaged as a user-space module or library installed as part of the device's host driver 412. In other implementations, VDCM 402 is a kernel module. If implemented as a library, VDCM 402 can be statically or dynamically linked to the VMM-specific Virtual Machine Resource Manager (VMRM) responsible for creating and managing VM resources. If implemented in the host OS kernel, VDCM 402 can be part of host driver 412.

[0057] The guest driver 424 for device 230 with Scalable IOV capabilities is conceptually equivalent to the SR-IOV device VF driver. In an embodiment, the guest driver 424, residing in the guest VM 422, manages the VDEV instance 404 comprised of VDCM 402. Fast path access 426 via the guest driver 424 is directly published to the ADI 432, 434 behind the VDEV 404, while slow path access 420 is intercepted and virtualized by the VM Resource Manager (VMRM) 416 and VDCM 402. Similar to the implementation choices available for SR-IOV PF 212 and VF 214 drivers, for the target OS 150, the guest driver 424 is deployed as a separate driver or a unified driver supporting the functionality of both the host OS 150 and the guest VM 422. For the existing SR-IOV device 216, if the VDEV 404 is configured to operate like the existing VF 214, then for backend compatibility, the scalable IOV guest driver 424 can even be the same as the SR-IOV VF 214 driver.

[0058] In embodiments of the invention, a Virtual Device (VDEV) 404 is an abstraction of software that exposes shared physical devices (e.g., Scalable IOV device 230) to a guest VM 422. The VDEV 404 is exposed to the guest VM 422 as a virtual PCI Express enumeration device using virtual resources such as Virtual Requester IDs, Virtual Configuration Space Registers, Virtual Memory Bases (BARs), Virtual MSI-X Tables, etc. Each VDEV 404 may be supported by one or more ADIs 428, 430, 432, 434. The ADIs supporting the VDEV 404 typically belong to the same PF 232, but implementations that distribute them across multiple PFs (e.g., to support device fault tolerance or load balancing) are possible.

[0059] PF 232 can support multiple types of ADIs, both in terms of the number and functionality of the backend resources 330. Similarly, it is possible to construct multiple types of VDEVs on the scalable IOV device 230 (regarding the number of supported ADIs, the functionality of the ADIs, etc.). VDCM 402 can publish support for constructing multiple "VDEV types," enabling the Virtual Machine Resource Manager (VMRM) 416 to request different types of VDEV instances for allocation to virtual machines (VMs). VDCM 402 uses the interfaces defined by the host OS 150 and VMM 180 to allocate and configure the resources required to construct multiple VDEV 404 instances. VDEV instances can be allocated to VMs 422 in the same manner as SR-IOV VF 214.

[0060] A VDEV 404 can be composed of a static number of ADIs pre-assigned at the time of VDEV instantiation, or dynamically composed by the VDCM 402 in response to a guest-driven 424 request to allocate / release resources. An example of a statically assigned ADI is a Virtual NIC (vNIC) with a fixed number of RX / TX queues. An example of a dynamically assigned ADI is a Virtual Accelerator device, where the VDCM 402 virtualizes context assignment requests to dynamically create the accelerator context as an ADI.

[0061] For any system page-sized region of the VDEV MMIO space, the VDEV MMIO register 322 can be constructed using any of the following methods.

[0062] 1) Direct Mapping to ADI MMIO. As part of the VDEV instance 404, VDCM 402 defines the system page size range in the VDEV virtual BAR within the Guest Physical Address (GPA) space, which needs to be mapped to the ADI-enabled MMIO page range in the Host Physical Address (HPA) space. VDCM 402 can request VMM 180 to establish a GPA-HPA mapping in the host processor 122-1…122-M virtualization page tables, enabling direct access to ADI via guest driver 424. These directly mapped MMIO ranges support fast path operations 426 to ADI 432, 434.

[0063] 2) VDEV MMIO interception and emulation via VDCM. Slow-path registers for the VDEV are virtualized by VDCM 402 by requesting VMM 180 not to map these MMIO regions 322 in the host processor's virtualization page table, thus forcing host interception when guest driver 424 accesses these registers. This interception is provided to the VDCM module 402 constituting the VDEV instance 404, allowing VDCM 402 to virtualize such intercepted accesses either on its own or through interaction with the host driver 412. To minimize software complexity in slow-path access emulation, the host OS 150 or virtualization provider can constrain guest driver 424 to access the VDEV's slow-path MMIO resources using simple memory movement operations of eight bytes or less. VDEV registers that are frequently read and have no read side effects but require write access can be mapped as read-only to memory-backed pages provided by VDCM. This enables high-performance read access to these registers, along with virtualization of write side effects, by intercepting guest write access. In the VDEV MMIO layout, the "write-only intercept" registers must be hosted in a separate system page-size region from the "read-write intercept" registers.

[0064] 3) VDEV MMIO 322 mapped to memory 130. VDEV registers without read or write side effects can be mapped to main memory 130 using read and write access. These registers can contain parameters or data for subsequent operations performed by writing to intercepted registers. Device implementations can also use this method to define virtual registers for a VDEV-specific communication channel between the guest driver 424 and the VDCM 402. The guest driver 424 writes data to memory-backed virtual registers without host interception, followed by mailbox register access intercepted by the VDCM. This optimization reduces the host interception and instruction emulation costs for transferring data between the guest and the host. This approach allows the guest driver 424 to be implemented more generally than hardware-based communication doorbells (such as those often implemented between SR-IOV VF 214 and PF 212) or to implement such a channel with respect to the VDCM without relying on the guest OS or VMM-specific paravirtualized software interface.

[0065] VDEV 404 exposes Virtual MSI or Virtual MSI-X capabilities emulated by VDCM 402. Guest driver 424 typically requests VDEV interrupt resources through the guest VM 422 interface, and the guest VM can service this by programming one or more interrupt messages via the Virtual MSI or Virtual MSI-X capabilities of VDEV 404.

[0066] For a typical virtual device configuration, there are two sources of interrupts delivered as VDEV interrupts to the guest driver 424. One source is the VDCM software 402 itself, which can generate virtual interrupts to be delivered to the guest driver on behalf of the VDEV. These are software-generated interrupts through the slow path operation of the VDEV simulated by the VDCM. The other source of interrupts is the ADI instances 432, 434 on the device that support the fast path operation of the VDEV 404. The interrupts generated by the ADI use interrupt messages stored in the Interrupt Message Store (IMS) 328.

[0067] When the guest VM 422 programs the virtual MSI or MSI-X registers, the operation is intercepted and virtualized by the VDCM 402. For slow-path virtual interrupts, the VDCM requests virtual interrupt injection for the guest VM via the VMM 180 software interface. For fast-path interrupts from ADI, the VDCM invokes the host driver 412 to dispatch and configure the interrupt message address and data required in the IMS. This is conceptually similar to how MSI-X interrupts for the SR-IOV VF are virtualized via some kind of virtualization software, except that the interrupt messages are programmed in the IMS via the host driver 412, as opposed to programming interrupt messages in the MSI-X table via the PCI driver.

[0068] For specific use and reasons, the device 230 with scalable IOV capabilities may choose to establish a communication channel between the guest driver 424 and the VDCM 402. These communication channels can be established in a manner independent of the guest and host system software using any of the following methods.

[0069] 1) Software-simulated communication channel. Such a channel is constructed by VDCM 402 using one or more system page-sized regions within the VDEV MMIO space, which is configured to be fully memory-supported to enable data sharing between the guest VM 422 and the host OS 150. The system page-sized regions within the VDEV MMIO space intercepted by the host are also configured to signal guest actions to the host. Optionally, a virtual interrupt can also be established by the VDCM to signal the guest about the completion of asynchronous communication channel actions.

[0070] 2) Hardware mailbox-based communication channel. If communication between the guest driver 424 and the host driver 412 is frequent and the overhead of a software-simulated communication channel is significant, the scalable IOV device 230 can implement a communication channel based on a software mailbox. This is similar to the communication channel between the SR-IOV VF 214 and PF 212 in some existing designs.

[0071] Shared Virtual Memory (SVM) refers to the use of a device operating within the shared CPU virtual address space of an application that uses that device. SVM is enabled by system software that programs DMA remapping hardware to reference the CPU page table to obtain requests using a PASID, which represents the target application's virtual address space. Devices supporting such SVM capabilities do not require the pages accessed by the device to be pinned and instead support PCI Express Address Translation Service (ATS) and Page Request Service (PRS) capabilities to support recoverable device page faults. Refer to the PCI Express specification for details on ATS and PRS capabilities.

[0072] Devices supporting Scalable IOV architecture can independently support SVM usage on ADIs assigned to host applications, or SVM usage on ADIs assigned to guest applications via VDEV instances assigned to guest VM 422. Host and guest SVM usage is transparent to ADI operations. One difference lies in the address translation functionality programming of the root federation DMA remapping hardware. Address translation programmed for PASIDs representing host SVM usage refers to the translation of the corresponding CPU virtual address to physical address, while address translation programmed for PASIDs representing guest SVM usage refers to the translation of the corresponding nested addresses (guest virtual address to guest physical address and further to host physical address).

[0073] The set of requirements and capabilities for enabling endpoint devices to support scalable IOV architectures will now be described. These requirements apply to both Root Consortium Integrated Endpoint (RCIEP) and PCI Express Endpoint (PCIEP) devices. In embodiments, the endpoint device may be a NIC, memory controller, GPU, FPGA, application-specific integrated circuit (ASIC), or other circuitry system.

[0074] As previously described, the Scalable IOV architecture defines the fine-grained sharing constructs used on endpoint devices (i.e., Scalable IOV device 230) as Assignable Device Interfaces (ADIs) 428, 430, 432, and 434. ADIs form units for the allocation and isolation of devices 230 with Scalable IOV capabilities and are software-configured to form virtual devices. The requirements for enumerating, dispatching, configuring, managing, and isolating endpoint devices for ADIs are as follows.

[0075] Resources on the endpoint devices associated with fast path job submission, execution, and completion operations are referred to as device backend resources.

[0076] Assignable Device Interfaces (ADIs) 428, 430, 432, and 434 refer to collections of device backend resources 330 that are assigned, configured, and organized into isolated units (forming units shared by devices). The type and number of backend resources grouped to constitute an ADI are device-specific. For example, for network controller devices (such as Ethernet NICs), an ADI may consist of a collection of TX / RX queues and resources associated with a Virtual Switch Interface (VSI). An ADI on a storage controller may be a collection of command queues and completion queues associated with a storage namespace. Similarly, an ADI on a GPU may be organized as a collection of graphics or compute contexts created representing instances of virtual GPU devices. Depending on the design, an ADI on an FPGA device may be an entire Accelerator Functional Unit (AFU) or a context on an AFU with multi-context capabilities.

[0077] The SR-IOV architecture specifies the allocation of PCI Express architecture resources through VF construction, but leaves the details of how device backend resources are allocated and how these backend resources are associated with a specific VF 214 to the device implementation. Devices that want to flexibly supply a varying number of backend resources to a VF 214 (i.e., one queue pair going to one VF and another queue pair going to another VF) need to implement another level of "resource remapping logic" within the endpoint device (as in...). Figure 3 (As shown in block 308) to map which device backend resources 310 are accessible through a specific VF 214 and isolated from access through other VFs. As the number of VFs and backend resources increases, such resource remapping logic 308 in the endpoint device increases device complexity.

[0078] The SR-IOV software architecture provides virtual device instances consisting of a single VF, while the scalable IOV software architecture of embodiments of the present invention allows software to construct virtual device (VDEV) instances using one or more ADIs. This enables endpoint device hardware design to avoid the need for complex resource remapping logic within the endpoint device.

[0079] For example, consider a device using queue pairs (QPs) 436, 438, 440, and 442 as backend resources 330, and a VM 422 requiring eight QPs in a VDEV for its workload. In an SR-IOV architecture, the design would have to map VF 214 to eight QPs, either through a static partitioning of eight QPs per VF or through a dynamic partitioning of eight QPs to VFs using resource remapping logic 308 in the endpoint device. An equivalent device design with scalable IOV capabilities treats each QP as an ADI 428, 430, 432, or 434, and uses VDCM software 402 to construct a VDEV using eight ADIs. In this case, resource remapping functionality is implemented in VDCM 402.

[0080] Figure 5Example diagram 500 illustrates a logical view of virtualization software utilizing a varying number of device backend resources 330 (ADI) and virtualization software comprising one or more ADIs 520, 522, 524, 526 to form virtual device instances 508, 510, 512. One or more guest partitions exist, such as guest partition 1 502, guest partition 2 504… guest partition J 506, where J is a natural number, executed by computing platform 101. One or more virtual devices (VDEVs) exist, such as virtual device 1 508, virtual device 2 510… virtual device K 512, where K is a natural number, executed by computing platform 101. Each guest partition can invoke one or more virtual devices for I / O requests. For example, guest partition 502 invokes virtual device 1 508, guest partition 2 504 invokes virtual device 2 510, and so on, with guest partition J invoking virtual device K 512. Any number of guest partitions may exist. Any number of virtual devices may exist. The maximum number of virtual devices invoked by any guest partition is implementation-dependent. Within the endpoint device hardware (i.e., scalable IOV device 230), there are one or more ADIs, such as ADI 1 520, ADI 2 522, ADI 3 524…ADI M 526, where M is a natural number. Any number of ADIs may exist in scalable IOV device 230 (i.e., it is implementation-dependent), and one or more scalable IOV devices (e.g., network I / O device 110) exist in computing platform 101. The number of scalable IOV devices used in the computing platform is implementation-dependent. Each ADI uses one or more device backend resources 330. For example, ADI 1 520 uses backend resource 1528, ADI 2 522 uses backend resource 2 530, ADI 3 524 uses backend resources 3 532, 4 534, and 5 536, and ADI M 526 uses backend resource N 538. There may be any number of backend resources in a scalable IOV device 230. The number of backend resources in a scalable IOV device is implementation-dependent.

[0081] Any virtual device 508, 510, or 512 can use either a slow path or a fast path for I / O requests. For example, virtual device 1 508 invokes either a slow path software emulation 514 or a fast path direct mapping 540 to ADI 1 520. For example, virtual device 1 508 also invokes ADI 2 522 via fast path direct mapping 540. For example, virtual device 2 510 invokes either a slow path software emulation 516 or a fast path direct mapping 542 to ADI 3 524. For example, virtual device K 512 invokes either a slow path software emulation 518 or a fast path direct mapping 544 to ADI M 526.

[0082] Unlike SR-IOV VFs, where requests are marked using the unique requester ID (RID) of each VF, in embodiments of this invention, all ADI requests from a PF are marked using the PF's RID. Alternatively, requests from ADIs are distinguished by the process address space identifier (PASID) in the end-to-end PASID TLP prefix. The PCI Express specification defines the process address space identifier (PASID) in the PASID TLP prefix of a transaction, which, in conjunction with the RID, identifies the address space associated with the request.

[0083] The definition of the address space targeted by the PASID value depends on the root union's DMA remapping hardware capabilities and the programming of such hardware via software. This utilizes Intel's Rev 3.0 or higher for directional I / O. ® The virtualization computing platform supports scalable IOV architecture through PASID-level address translation capabilities. Depending on the programming of such DMA remapping hardware, the address space targeted by a request with a PASID can be the Host Physical Address (HPA), Host Virtual Address (HVA), Host I / O Virtual Address (HIOVA), Guest Physical Address (GPA), Guest Virtual Address (GVA), Guest I / O Virtual Address (GIOVA), etc. All these address space types can coexist on the computing platform 101 for different PASID values, and ADI from one or more scalable IOV devices can be configured to use these PASIDs.

[0084] When an ADI is assigned to an address domain (e.g., VM 422, container, or process), the ADI is configured using the address domain's unique PASID, and the memory request for the ADI is identified using the PASID value in the PASID TLP prefix. If multiple ADIs are assigned to the same address domain, they can be assigned the same PASID. If an ADI belonging to a virtual device (VDEV) assigned to a VM is further mapped to a secondary address domain within the VM (e.g., an application process), a unique PASID corresponding to that secondary address domain is assigned to each such ADI. This enables uses such as shared virtual memory (SVM) within a VM, where ADIs are assigned to guest application processes, and requests from guest ADIs undergo the same nested translation (GVA to GPA to HPA) performed via DMA remapping hardware, similar to the nested address translation used for CPU access via guest applications. Depending on the use model, it is also possible for an ADI to use more than one PASID value, and in this case, the semantics of which PASID value is used with which request are related to the scalable IOV device. For example, an ADI can be configured to access metadata, commands, and completion using a PASID that represents a constrained control domain, with data access associated with the PASID of the domain to which the ADI is assigned.

[0085] In this embodiment, the device 230 supporting the Scalable IOV architecture supports PASID capabilities as defined by the PCI Express specification and complies with all associated requirements. The PASID capability on the PF 232 is enabled via software before enabling ADI on the PF 232. The ADI is configured using a PASID value before activation. All upstream memory requests generated by the ADI (except for requests translated by Address Translation Service (ATS)) are marked using the PASIDTLP prefix with the assigned PASID value. In this embodiment, the ADI cannot generate memory requests (except for ATS-translated requests) without a PASID and cannot generate memory requests using a PASID value in the PASID TLP prefix that is not a PASID value assigned to the ADI.

[0086] Each ADI's memory-mapped I / O (MMIO) 322 register is managed within one or more of the PCIExpress base address registers (BARs) 320 that manage the PF 232. Each ADI's MMIO 322 register is contained within one or more system page-sized aligned regions, and these can be contiguous or scattered regions within the PF's MMIO space 322. The number and location of the system page-sized regions within the PF's MMIO are device-specific to a particular ADI. The system page size supported by Scalable IOV devices is reported via the Intel Scalable IOV enumeration capability described below. In this embodiment, for Intel... ® The 64-bit computing platform has a system page size of four kilobytes (KB).

[0087] Devices supporting Scalable IOV architectures divide their ADI MMIO 322 registers into two categories: (a) MMIO registers frequently accessed for fast-path operations; and (b) MMIO registers less frequently accessed for slow-path operations (control, configuration, management, etc.). The definition of which operations are designated as slow-path and fast-path is device-specific. PF 232 locates registers in these two categories within distinct system page-size regions. This allows virtualization software such as the host OS / VMM 228 to directly map fast-path operations to one or more component ADIs, while emulating slow-path operations in software 514, 516, and 518.

[0088] In one embodiment, the device implements a 64-bit BAR 320 such that an address space of more than four billion bytes (4 GB) can be used to expand the ADI MMIO 322 resources. Additionally, since non-fetchable BARs use less than 4 GB of MMIO space even with a 64-bit BAR, in one embodiment, the device implements a prefetchable 64-bit BAR.

[0089] An ADI capable of generating interrupts only generates Message Signaled Interrupts (MSIs) (not traditional interrupts). An ADI does not share interrupt resources / messages with a Power Provider (PF) or another ADI. An ADI can support one or more interrupt messages. For example, an ADI consisting of N queues on a PF can support N interrupt messages to distinguish between work arrival and completion for each queue, where N is a natural number.

[0090] The Scalable IOV architecture enables device implementations to support a large number of ADIs, with each ADI capable of using multiple interrupt messages. To support a large interrupt message store for all ADIs, an Interrupt Message Store (IMS) invoked by the device-specific construct for ADI 328 is defined. IMS 328 allows devices to store interrupt messages for ADIs in a device-specific optimized manner, without the scalability constraints of the MSI-X capabilities defined by PCI Express.

[0091] Although the IMS storage organization is device-specific, in one embodiment, IMS entries use the same interrupt message address and data format as PCI Express MSI-X table entries to store and generate interrupts. Interrupt messages stored in the IMS 328 consist of a DWORD-sized data payload and a 64-bit address. Similar to the per-vector mask and pending bit array in PCI Express MSI-X capabilities, the IMS 328 may also optionally support per-message mask and pending bit states. In this embodiment, the IMS resources are programmed via host driver 412.

[0092] The PF hosting ADI can support the MSI or MSI-X capabilities defined by PCI Express. Interrupts generated by PF 232 can use the MSI or MSI-X capability 326 of the PF as specified by the PCI Express specification, while interrupts generated by ADI can use the device-specific IMS 328. A specific host OS 150 / VMM 180 implementation according to an embodiment of the present invention supports the use of IMS 328 for interrupts of the PF and / or the use of the MSI-X table of the PF for interrupts of ADI.

[0093] The size, location, and storage format of IMS 328 are device-specific. For example, some devices may implement IMS as on-device storage, while other stateful devices that manage the context saved to and restored from main memory 130 may implement IMS as part of the context's privileged state. In either approach, a device may implement IMS 328 as a uniform storage structure or as a distributed per-ADI storage structure. If IMS 328 is implemented in host main memory 130, ADI may cache IMS entries on the Scalable IOV device. If the Scalable IOV device implements IMS caching, it also implements a device-specific interface for device-specific drivers to invalidate IMS cache entries.

[0094] IMS 328 is managed by host driver software 412 and cannot be directly accessed from guest or user-mode drivers in guest partitions 502, 504, and 506. Within the Scalable IOV device, IMS storage cannot be directly accessed from ADI; instead, ADI can request interrupt generation solely through the PF's "interrupt message generation logic." This ensures that ADI cannot modify the IMS content, and ADI can indirectly generate interrupts using only the IMS entries allocated to the corresponding ADI by host driver software 412.

[0095] On a 64-bit Intel architecture (IA)-based computing platform, a message signal interrupt is issued to the address range 0xFEExxxxx without the PASID TLP prefix, as an untranslated DWORD-sized memory write. Since all memory requests generated by ADI include the PASID TLP prefix, and the interrupt message is generated without the PASID TLP prefix, it is impossible to generate a DMA write to the interrupt message address (0xFEExxxxx on an IA-based 64-bit computing platform) via ADI and have the platform interpret the DMA write as an interrupt message.

[0096] The operation or functioning of one ADI must not affect the operation of another ADI or the PF 232. Every memory request from an ADI (except for ATS translation requests) must be accompanied by a PASID TLP prefix containing the ADI's assigned PASID value. The PASID identity used by an ADI is accessed or modified by privileged software via host driver 412.

[0097] Because the ADIs on the Scalable IOV device 230 are part of the PF 232, PCI Express Access Control (ACS) capabilities are not applicable to isolation between ADIs. Instead, the device disables peer-to-peer access between ADIs and between ADIs and the PF (either within the device or at I / O fabric exits). Regardless of Scalable IOV architecture support, the PF 232 can support ACS guidelines for isolation across endpoint functions or devices in accordance with the PCI Express specification.

[0098] The Quality of Service (QoS) for ADI is defined specific to a given Scalable IOV device. ADI QoS attributes are managed by the host driver 412 and controlled by the VDCM 402 through the host driver 412 interface.

[0099] ADI-specific errors are errors attributable to a specific ADI, such as malformed commands or address translation errors. Such errors do not affect the operation of other ADI components or PF 232 devices. ADI-specific error handling is implemented in a device-specific manner.

[0100] Each ADI can be reset independently without affecting the operation of other ADIs. However, unlike the SR-IOV VF 214, ADIs do not support function-level reset (FLR) capability. Instead, ADI resets are performed via ADI reset configuration 410 through a software interface to host driver 412, such as... Figure 4 As shown in the diagram. To support ADI reset, Scalable IOV devices implement interfaces to abort (e.g., discard) in-flight and accepted operations on an ADI via a specific domain (or PASID). In an embodiment, a VDEV 404 composed of ADIs can expose virtual FLR capabilities, which can be simulated by VDCM 402 by requesting host driver 412 to perform an ADI reset for the constituent ADIs 428, 430, 432, or 434 for VDEV 404.

[0101] An ADI reset will not report a reset as complete until the following conditions are met: a) all DMA write operations performed via ADI have been exhausted or aborted; b) all DMA read operations performed via ADI have been completed or aborted; c) all interrupts from ADI have been generated; d) if ADI is capable of Address Translation Service (ATS), all ATS requests performed via ADI have been completed or aborted; and e) if ADI is capable of Page Request Service (PRS), no further page requests will be generated by ADI. Alternatively, either page responses have been received for all page requests generated by ADI, or ADI will discard page responses for any incomplete page requests.

[0102] In this embodiment, PF 232 supports Function Level Reset (FLR) and may optionally support additional device-specific global reset control. A global reset operation on PF 232 and FLR reset all its ADIs and return the PF to a state where no ADI is configured.

[0103] In this embodiment, PF 232 supports saving and restoring ADI state to facilitate operations such as live migration and suspension / restart of virtual devices composed of such ADIs. For example, to support ADI suspension, Scalable IOV device 230 implements an interface to exhaust (i.e., complete) in-flight and accepted operations on an ADI via a specific domain (or PASID). In this embodiment, ADI suspension, ADI state saving, ADI state restoration, and ADI restart from the restored state are also implemented via host driver 412 interface.

[0104] PF 232 reports support for the Scalable IOV architecture to system software such as VDCM 402 via the host driver 412 interface. If host driver 412 reports support for the Scalable IOV architecture, it supports an extended set of interfaces to enumerate, provision, instantiate, and manage ADIs on the PF. System software, such as VDCM 402, performs all Scalable IOV-specific operations on the Scalable IOV device 230 via host driver 412.

[0105] Additionally, in one embodiment, PCIExpress vendor-specific extension capabilities (DVSEC) are defined for system software and software tools such as VDCM 402 to detect devices supporting the Scalable IOV architecture, without host driver dependency. Host driver 412 remains responsible for enabling the Scalable IOV architecture and related operations through system software-specific interfaces.

[0106] Figure 6 The diagram illustrates an example of a Scalable IOV DVSEC structure 600. The fields up to offset 0xA contain standard DVSEC capability header information. Refer to the PCI Express DVSEC header for a detailed description of these fields. The remaining fields are described below.

[0107] The Function Dependency (DEP) Link (Read-Only (RO)) field 602 is located at offset 0xA and has a size of one byte. The programming model used for the device may have vendor-specific dependencies between sets of functions. The Function Dependency Link field 602 is used to describe these dependencies. This field describes the dependencies between PF 232. ADI dependencies are the same as the dependencies of their respective PFs. If PF 232 is independent of other PFs of the Scalable IOV device 230, this field contains the PF's own function number. If a PF depends on other PFs of the Scalable IOV device, this field contains the function number of the next PF in the same Function Dependency List (FDL). The last PF in the FDL contains the function number of the first PF in the FDL.

[0108] Dependencies between PFs are described by flag field 604 at offset 0xB. Flag field 604 (read-only) is located at offset 0xB and has a size of one byte. In an embodiment, flag field 604 includes a homogeneity (H) flag in bit 0 of this byte, and bits 1 through 7 are reserved. When the H flag is reported as set, the H flag indicates that all PFs in the FDL must be enabled (in a device-specific manner) for Scalable IOV operation. If some, but not all, of the PFs in the FDL are already enabled for Scalable IOV operation, the behavior is undefined (i.e., if the H flag is reported as set, one PF cannot be in Scalable IOV operation mode and another cannot be in SR-IOV operation mode). If the H flag is not set, the PFs in the FDL can be in different modes.

[0109] The supported page size (read-only) field 606 is located at offset 0xC and has a size of four bytes. The supported page size field 606 indicates the page size supported by PF 232. If bit n is set, PF supports 2... n+12 The page size is specified. For example, if bit 0 is set, the PF supports 4 KB pages. The page size describes the minimum alignment requirement for ADI MMIO 322 pages, allowing them to be allocated independently to different address domains. In this embodiment, the PF is required to support a 4 KB page size. For broad compatibility across host platform architectures, the PF may support additional system page sizes.

[0110] The System Page Size (Read-Write (RW)) field 608 is located at offset 0x10 and has a size of four bytes. System Page Size field 608 defines the page size used by the system to map MMIO 322 pages to ADI. The software sets the value of the System Page Size to one of the page sizes set in the Supported Page Size fields. Similar to the Supported Page Sizes, if bit n is set in the System Page Size, then the ADI associated with this PF supports 2... n+12 The system page size. For example, if bit 1 is set, the device uses an 8 KB page size. If the system page size is zero, more than one bit is set, or a bit in the system page size field that is not set among the supported page sizes is set, the behavior is undefined.

[0111] When the system page size field 608 is written, PF 232 aligns all ADIMMIO 322 resources on the system page size boundary. The system page size must be configured before the memory space enable bit in the PF's PCI command register is set. If the system page size is modified after the memory space enable bit has been set, the behavior is undefined. The default value is 0000 0001h, indicating a 4 KB system page size.

[0112] The capability (read-only) field 610 is located at offset 0x14 and has a size of four bytes. In this embodiment, the capability field 610 includes an IMS support flag in bit 0, and bits 1 through 31 are reserved. The IMS support flag indicates support for Interrupt Message Storage (IMS) in the device. When the IMS support flag is 0, the device does not support IMS. When the IMS support flag is 1, the device supports IMS.

[0113] If the virtualization software (such as VDCM 402) does not support IMS use via the PF itself (IMS use is supported only for PFADI), then for compatibility, the virtualization software can expose the Virtual Extensible IOV capability to domains with an IMS support flag that is reported as 0 when the PF is directly assigned to a domain.

[0114] In this embodiment, the scalable IOV architecture relies on the following platform-level capabilities: a) support for the PCI Express PASID TLP prefix when supporting root ports (RP), root unions (RC), and DMA remapping hardware units (refer to the PCIExpress Revision 4.0 specification for details on PASID TLP prefix support); and b) PASID-granular address translation via DMA remapping hardware, such as through directional I / O (Intel... ® Intel VT-d) ® Virtualization technology (Intel) ® PASID-granular address translation as defined in Scalable Mode Address Translation (VT) Revision 3.0 or later.

[0115] Such as Intel ®The Scalable Mode address translation defined by VT-d involves three-stage address translation. Other embodiments may use different methods. First, the Requester ID (RID) (bus / device / function number) in the upstream request is used to consult the root and context structures that specify translation behavior at the RID (PF or SR-IOV VF) granularity. The context structure refers to the PASID structure. Second, if the request includes a PASID TLP prefix, the PASID value from the TLP prefix is ​​used to consult the PASID structure that specifies translation behavior at the PASID (target address field) granularity. If the request does not have an PASID TLP prefix, the PASID value programmed by the software in the context structure is used instead. For each PASID, the corresponding PASID structure entry can be programmed to specify a first-level, second-level, pass-through, or nested translation function, along with references to the first-level and second-level page table structures. Finally, depending on the type of translation function, the address in the request undergoes address translation using the first-level, second-level, or both page table structures.

[0116] PASID granular address translation enables each upstream request from an ADI on the PF to have a unique address translation. Any such ADI 428, 430, 432, or 434 on the PF 232 can be used by the VDCM 402 to construct a virtual device 404 that can be assigned to any type of address domain, such as the guest physical address space of a VM or machine container, the I / O virtual address for a bare metal container, the shared CPU virtual address for an application process, or such a guest container or process operating within a VM.

[0117] For interrupt isolation across devices, host system software, such as host OS 150, VMM 180, and / or VDCM402, enables interrupt mapping and allows for the use of remappable interrupt message formats for all interrupt messages programmed in the MSI, MSI-X326, or IMS 328 on the device. Refer to Intel Virtualization Technology for Directed I / O Specifications for details on interrupt remapping.

[0118] In various embodiments, computing platforms supporting scalable IOV architectures also support posted interrupt capabilities. Posted interrupts enable scalable interrupt virtualization by allowing interrupt messages to operate in the guest interrupt vector space without consuming host processor interrupt vectors. It also enables direct delivery of virtual interrupts to the active virtual processor without hypervisor processing overhead. Refer to Intel Virtualization Technology for the Directed I / O Architecture Specification for details on posted interrupts. Posted interrupt operations are transparent to endpoint devices.

[0119] Figure 7 An example high-level translation structure organization 700 for Scalable Mode address translation is illustrated. In an embodiment, Scalable Mode address translation is implemented using a Scalable Mode root table 702. The Scalable Mode root table 702 includes multiple entries. In this example, the number of entries in the Scalable Mode root table is 256, the maximum number of possible buses in the computing platform 101. Example entry N 708 in the Scalable Mode root table 702 points to an entry in the Scalable Mode Middle Context Table 710 and to an entry in the Scalable Mode Upper Context Table 712. In an embodiment, there are 128 entries in the Scalable Mode Middle Context Table 710, numbered from Device (DEV) = 0 entry 714 to Device = 15 entry 716 (each of these 16 devices has eight entries). Each entry in the Scalable Mode Middle Context Table specifies a function. The function ID in one embodiment is a number between 0 and 7. In this embodiment, there are 128 entries in the Scalable Mode Upper Context Table 712, numbered from Device=16 Entry 718 to Device=31 Entry 720 (each of these 16 devices has eight entries). Each entry in the Scalable Mode Upper Context Table specifies a function. Selected entries in either the Scalable Mode Upper Context Table 710 or the Scalable Mode Upper Context Table 712 point to the Scalable Mode PASID directory 722. In this embodiment, there are 2^14 entries in the Scalable Mode PASID directory 722, from entry number 0 724 to entry number 2^14-1 726. The index to the directory in one embodiment is formed by bits 6 to 19 of the PASID. Selected entries in the Scalable Mode PASID directory 722 point to selected entries in the Scalable Mode PASID table 728. In this embodiment, there are 64 entries in the Scalable Mode PASID table 728, from entry number 0 730 to entry number 63 732. In one embodiment, the index to the scalable mode PASID table 728 is formed by bits 0 to 5 of the PASID. The selected entry in the scalable mode PASID table 728 points to the first-level page table structure 734 and to the second-level page table structure 736.

[0120] Figure 8The diagram illustrates an example of virtual device configuration 800. The host OS 150 includes a VDCM 402 for instantiating virtual device VDEV 1 816 using a VDEM template 404 to utilize guest driver 424 operations within guest VM 422. VDEV 404 includes a PASID 802 that uniquely identifies the VDEV instance (e.g., VDEV 1 816). VDEV 404 includes a Virtual PCI Configuration Space (VCS) 804 that defines parameters for the VDEV instance, such as capabilities (e.g., MSI-X, BAR, and other PCI capabilities), error handling, Quality of Service (QoS) settings, and reset indicators. VDEV 404 includes an MSI-X table 806 for storing multiple MSI-X interrupt entries. VDEV 404 includes an MMIO field for mapping the address of registers in PF 232 to the fast path ADI MMIO 808, the software-emulated hypervisor intercept (HI) MMIO 810, and the memory support (MB) MMIO 812. ADI MMIO 808 provides the fast path resources for VDEV 404. VM Resource Manager 416 serves VDCM 402, assigns PASIDs to VDEVs, and triggers VDEV configuration. IOMMU driver 813 configures the translation table for the IOMMU using the extensible IOV extension 414 for the PASID assigned to VDEV 404.

[0121] Physical function (PF) 232 includes PF base address register (BAR) 320, which includes one or more ADI MMIO registers 322, in Figure 8These are represented as ADI MMIO 828, ADI MMIO 830, ..., ADI MMIO 832, and ADI MMIO 834. The Base Address Register (BAR) specifies how much memory a device (e.g., a PF) wants to map into main memory 130, and after device enumeration, the BAR holds the (base) address at which the mapped memory block begins. ADI MMIO registers 828, 830...832, and 834 are used by VDEV to communicate directly with hardware queues 436, 438...440, and 442. PF 232 includes PF config 324, which defines PCI configuration parameters for the PF, such as device vendor ID, BAR, error reporting, etc. PF 232 stores the PASID for ADI 814, and PF MSI-X 326 provides MSI-X capabilities as defined by the PCI Express Basic Specification. The IMS for ADI 328 enables PF 232 to store interrupt messages for ADI in a PF-specific optimized manner, without the scalability constraints of the MSI-X capabilities defined by PCI Express. In one embodiment, the interrupt message is used to indicate that data (e.g., packets) is available in queues 436, 438...440, 442 for processing by VDEV 1 816.

[0122] The VDEV lifecycle is divided into four phases: 1) VM and VDEV definition; 2) VM instantiation and VDEV construction; 3) runtime VDEV resource modification; and 4) VM shutdown and VDEV decomposition.

[0123] In the first phase, the user (such as a system administrator, remotely running scripts, or other control authority) selects the resources for the VMs to be instantiated (e.g., guest VM 422 and other guest VMs). VM resources include virtual processors, memory, and physical devices such as network devices, storage devices, graphics processing units (GPUs), field-programmable gate arrays (FPGAs), accelerator devices, etc. Depending on the VDEV type (such as network, storage, GPU, FPGA, accelerator, etc.), the user also selects the resources for each VDEV to be instantiated. For example, for a network VDEV, the user can select initial resources, such as several receive (Rx) and transmit (Tx) queues and several IMS entries, as well as the maximum number of resources of each type that the VDEV can request during the lifetime of the VDEV instance. In one example, a network VDEV is instantiated using two ADIs and N entries in the IMS for ADI 328, where N is a natural number, but additional ADI and IMS entries can be requested and added during runtime due to processing workloads. In this embodiment, the number of ADIs used for VDEV is the same as the number of interrupt entries in the IMS used for ADI 328. That is, there is a one-to-one correspondence between the ADIs and the interrupt entries in the IMS used for ADI 328.

[0124] In the second phase, a VM is instantiated (e.g., guest VM 422) and together with a PF host driver 412 with S-IOV capability, a VDCM 402 constitutes one or more associated VDEVs. Figure 9A sample flowchart 900 illustrating the configuration of a virtual device is shown. After instantiating a VDEV (such as VDEV 1 816) via VDCM 402, at block 902, VDCM 402 creates a virtual PCI configuration space (VCS) 804 in VDEV 1 816. VDCM 402 then populates values ​​into the VCS 804 fields in the following three blocks. At block 904, VDCM 402 assigns the device vendor ID of PF 232 to VCS 804 in VDEV 1 816. This associates PF 232 with VDEV 816. At block 906, VDCM 402 prepares the MSI-X BAR in VCS 804 of VDEV 1 816 by setting a location representing the location in the main memory 130 of the VDEV. At block 908, VDCM 402 prepares one or more MMIO bars in VCS 804 of VDEV 1 816 by setting locations in main memory. In this embodiment, MMIO memory space is supplied to the VDEV up to the maximum amount of resources the VDEV can have during its lifetime, but initially, fewer resources are configured for the VDEV. At block 910, VDCM 402 obtains one or more ADIs and associated register pages from PF host driver 412 and maps the ADIs and associated register pages to the VDEV MMIO space. At block 912, VDCM 402 specifies the MMIO space for HI MMIO register 810 for software emulation and memory-supported MB MMIO register 812. At block 914, VDCM 402 assigns the instantiated VDEV (e.g., VDEV 1 816) to guest VM 422.

[0125] Figure 10The illustration shows an example diagram of a virtual device configuration 1000 using Physical Functions Adaptive Design (ADI) and Interrupt Message Service (IMS) resources. For example, a network VDEV is instantiated and configured with Adaptive Design (ADI), where each ADI represents a set of receive / transmit queues, and PF IMS resources are used to communicate with the VDEV's MSI-X configuration. Additionally, the VDEV MMIO is configured by the VDCM for the emulated HI MMIO register 810 (e.g., a slow path interface), where high-performance processing is not required. In this embodiment, the behavior of the emulated HI MMIO register 810 is implemented by the VDCM 402 and the PF host driver 412. A guest VM 422 accesses the BAR for MMIO 1002 in the VDEV config space (VCS 804). The BAR for MMIO 1002 points to a location in VDEV memory page 1006. VDEV memory page 1006 resides in main memory 130 on the host computing platform 101. Locations in VDEV memory page 1006 are mapped to locations in the physical functional address space represented as PF and ADI fast path register page 1012 on PF232. Software-emulated HI MMIO or memory-supported register page 1010 also points to VDEV memory page 1006. Guest VM 422 also accesses the BAR for MSI-X 1004 in VCS 804. The BAR for MSI-X 1004 points to locations in VDEV MSI-X table 1008, which is an instance of MSI-X table 806. Locations in VDEV MSI-X table 1008 point to locations in PF and IMS MSI-X vector 1004. In this embodiment, PF and IMS MSI-X vector 1004 are identical to PF MSI-X 326.

[0126] In this embodiment, the IMS resides in the PF MMIO space, which may include MSI-Z table entries for both the PF and VDEV. One advantage of the IMS is that, unlike the MSI-X tables, it is not constrained to a maximum size of 2K. Because the IMS is in the PF MMIO space, it can support many MSI-X-like entries (e.g., more than 2K).

[0127] In the third phase, the resources available for VDEV can be modified at runtime. Figure 11A sample flowchart 1100 illustrating virtual device resource modification is illustrated. In some cases, the user can select the maximum amount of resources (e.g., ADI, IMS entries) for the VDEV, but the VDEV can initially be constructed using only the minimum amount of resources. In this embodiment, the maximum and minimum values ​​are implementation-dependent. During runtime, depending on the processing workload requirements, an event can be triggered to request the dynamic addition or modification of resources to an existing VDEV. In this embodiment, the event is triggered by an action of a system administrator. In another embodiment, the event is triggered programmatically (e.g., via a controller component).

[0128] At block 1102, VDCM 402 receives an external request to add or modify VDEV resources. In this case, "external" refers to an entity outside of VDCM 402, guest VM 422, and VDEV 1 816. At block 1104, in response to receiving the request, VDCM 402 requests additional ADI and IMS interrupt entries from PF host driver 412. At block 1106, VDCM 402 enables the implementation of a new MMIO BAR mapping. At block 1108, VDCM 402 notifies guest driver 424 in guest VM 422 of the most recently available resources.

[0129] In the fourth stage, the guest VM can be shut down and its VDEV can be decomposed. Figure 12 A sample flowchart 1200 illustrates virtual device decomposition. This process is performed for each VDEV when the guest VM is shut down. At block 1202, VDCM 402 initiates a Functional Level Reset (FLR) for the VDEV (e.g., VDEV 1 816) as described in the PCI Express specification. At block 1204, VDCM 402 unmaps the VDEV's MMIO BAR from the guest VM's memory space. At block 1206, VDCM 402 returns the ADI and IMS entries to the PF host driver 412. The PF host driver 412 makes these ADIs available for reuse by other VDEVs. In the case of network VDEVs, the PF host driver disables Rx / Tx queues 436, 438…440, 442 and makes these queues available for use by other VDEVs. The PF host driver 412 also makes the corresponding IMS entries available for use by other VDEVs. The PF host driver 412 also releases internal resources supplied to the VDEVs.

[0130] Figure 13An example of storage medium 1300 is illustrated. Storage medium 1300 may include article of manufacture. In some examples, storage medium 1300 may include any non-transitory computer-readable or machine-readable medium, such as optical, magnetic, or semiconductor storage. Storage medium 1300 may store various types of computer-executable instructions, such as instructions 1302 that implement the logical flow of S-IOV architecture components, including... Figure 9 , 11 And those described in 12. Examples of computer-readable or machine-readable storage media can include any tangible medium capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of computer-executable instructions can include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc. Examples are not limited in this context.

[0131] Figure 14 The illustration shows an example computing platform 1400. In some examples, such as... Figure 14 As shown, the computing platform 1400 may include a processing component 1402, other platform components 1404, and / or a communication interface 1406.

[0132] According to some examples, processing component 1402 can execute processing operations or logic for instructions stored on storage medium 1300. Processing component 1402 may include various hardware elements, software elements, or combinations thereof. Examples of hardware elements may include devices, logic devices, components, processors, microprocessors, circuits, processor circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), memory cells, logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. Examples of software elements may include software components, programs, applications, computer programs, application programs, device drivers, system programs, software development programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application programming interfaces (APIs), instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. The determination of whether to use hardware and / or software components to implement the example can vary depending on any number of factors, such as desired computation speed, power level, thermal tolerance, processing cycle budget, input data rate, output data rate, memory resources, data bus speed, and other design or performance constraints as expected for a given example.

[0133] In some examples, other platform components 1404 may include common computing elements such as one or more processors, multi-core processors, coprocessors, memory cells, chipsets, controllers, peripherals, interfaces, oscillators, timing devices, video cards, audio cards, multimedia input / output (I / O) components (e.g., digital displays), power supplies, etc. Examples of memory cells may include, but are not limited to, various types of computer-readable and machine-readable storage media in the form of one or more high-speed memory cells, such as read-only memory (ROM), random access memory (RAM), dynamic RAM (DRAM), double data rate DRAM (DDRAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and non-volatile memory types, such as byte- or block-addressable 3D cross-point memory. Non-volatile memory types may also include other types of byte- or block-addressable non-volatile memory, such as, but not limited to, multi-threshold level NAND flash memory, NOR flash memory, single-level or multi-level PCM, resistive memory, nanowire memory, FeTRAM, MRAM incorporating memristor technology, STT-MRAM, or any combination of the above. Other types of computer-readable and machine-readable storage media may also include magnetic or optical cards, arrays of devices such as redundant array of independent disks (RAID) drives, solid-state storage devices (e.g., USB storage), solid-state drives (SSDs), and any other type of storage media suitable for storing information.

[0134] In some examples, communication interface 1406 may include logic and / or features supporting the communication interface. For these examples, communication interface 1406 may include one or more communication interfaces that operate according to various communication protocols or standards to communicate via direct or network communication links or channels. Direct communication may occur via the use of communication protocols or standards described in one or more industry standards (including their successors and variants), such as those associated with the PCIe specification. Network communication may occur via the use of communication protocols or standards, such as those described in one or more Ethernet standards published by IEEE. For example, one such Ethernet standard may include IEEE 802.3. Network communication may also occur according to one or more OpenFlow specifications, such as the OpenFlow switch specification.

[0135] The components and features of the computing platform 1400, including the logic represented by instructions stored on the storage medium 1300, can be implemented using any combination of discrete circuit systems, ASICs, logic gates, and / or single-chip architectures. Additionally, where appropriate, the features of the computing platform 1400 can be implemented using microcontrollers, programmable logic arrays, and / or microprocessors, or any combination thereof. It should be noted that hardware, firmware, and / or software elements can be collectively or individually referred to herein as "logic" or "circuit".

[0136] It should be understood that, Figure 14 The exemplary computing platform 1400 shown in the block diagrams may represent a functionally descriptive example of many potential implementations. Accordingly, the segmentation, omission, or inclusion of functional blocks depicted in the accompanying figures does not imply that hardware components, circuitry, software, and / or elements used to implement these functions will necessarily be segmented, omitted, or included in the embodiments.

[0137] The present invention also relates to the following examples: Example 1. A method that includes: Instantiate the virtual machine; Instantiate a virtual device to transfer data to and receive data from the resources allocated to the shared physical device; and The virtual device is assigned to the virtual machine, which transmits data to and receives data from the physical device via the virtual device.

[0138] Example 2. The method according to Example 1, wherein assigning the virtual device to the virtual machine includes exposing the virtual device to the virtual machine as a virtual Fast External Component Interconnect (PCI) enumeration device.

[0139] Example 3. The method according to Example 1, wherein the allocated resources include shared physical device resources allocated to the virtual device for data transmission.

[0140] Example 4. According to the method of Example 3, the shared physical device includes a network controller device, the shared physical device resources include a receive and transmit queue for storing the data, and the data includes packets.

[0141] Example 5. The method according to Example 1, wherein the shared physical device includes a storage controller device.

[0142] Example 6. The method according to Example 1, wherein instantiating the virtual device includes: Assign the identifier of the shared physical device to the virtual device; Prepare the base address register for message signaling interrupts for the virtual device; Prepare one or more memory-mapped input / output (MMIO) base address registers; Obtain one or more allocated resources and associated register pages, and map the allocated resources and associated register pages to the MMIO memory space for the fast path interface to the virtual device; and For the slow path interface to the virtual device, specify the MMIO memory space for the simulated and memory-supported registers.

[0143] Example 7. The method according to Example 6 includes decomposing the virtual device by initiating a functional level reset of the virtual device and unmapping the allocated resources and associated register pages from the MMIO memory space for the virtual device.

[0144] Example 8. The method according to Example 1 includes allocating additional resources to the virtual device based at least in part on requests received during runtime.

[0145] Example 9. At least one tangible machine-readable medium, comprising a plurality of instructions, said plurality of instructions causing the processor to: Instantiate the virtual machine; Instantiate a virtual device to transfer data to and receive data from the resources allocated to the shared physical device; and The virtual device is assigned to the virtual machine, which transmits data to and receives data from the physical device via the virtual device.

[0146] Example 10. At least one tangible machine-readable medium according to Example 9, wherein the instructions for assigning the virtual device to the virtual machine include instructions for exposing the virtual device to the virtual machine as a virtual Fast External Component Interconnect (PCI) enumeration device.

[0147] Example 11. At least one tangible machine-readable medium according to Example 9, wherein the allocated resources include shared physical device resources allocated to the virtual device for data transmission.

[0148] Example 12. At least one tangible machine-readable medium according to Example 9, wherein the instructions for instantiating a virtual device include instructions for: Assign the identifier of the shared physical device to the virtual device; Prepare the base address register for message signaling interrupts for the virtual device; Prepare one or more memory-mapped input / output (MMIO) base address registers; Obtain one or more allocated resources and associated register pages, and map the allocated resources and associated register pages to the MMIO memory space for the fast path interface to the virtual device; and For the slow path interface to the virtual device, specify the MMIO memory space for the simulated and memory-supported registers.

[0149] Example 13. At least one tangible machine-readable medium according to Example 9, comprising decomposing the instructions of the virtual device by initiating a functional level reset of the virtual device and unmapping the allocated resources and associated register pages from the MMIO memory space for the virtual device.

[0150] Example 14. At least one tangible machine-readable medium according to Example 9, including instructions to allocate additional resources to the virtual device based at least in part on requests received during runtime.

[0151] Example 15. An apparatus comprising: Virtual machines; and A virtual device constitutive module, coupled to the virtual machine, is configured to instantiate the virtual device to transmit data to and receive data from the resources allocated to the shared physical device; and to allocate the virtual device to the virtual machine, which transmits data to and receives data from the physical device via the virtual device.

[0152] Example 16. The apparatus according to Example 15 includes the virtual device constituting module for assigning the virtual device to the virtual machine by exposing the virtual device as a virtual Fast External Component Interconnect (PCI) enumeration device to the virtual machine.

[0153] Example 17. The apparatus according to Example 15, wherein the allocated resources include shared physical device resources allocated to the virtual device for data transmission.

[0154] Example 18. The apparatus of Example 17, wherein the shared physical device includes a network controller device, the shared physical device resources include a receive and transmit queue for storing the data, and the data includes packets.

[0155] Example 19. The apparatus according to Example 15, wherein the virtual device constituting module is used for Assign the identifier of the shared physical device to the virtual device; Prepare the base address register for message signaling interrupts for the virtual device; Prepare one or more memory-mapped input / output (MMIO) base address registers; Obtain one or more allocated resources and associated register pages, and map the allocated resources and associated register pages to the MMIO memory space for the fast path interface to the virtual device; and For the slow path interface to the virtual device, specify the MMIO memory space for the simulated and memory-supported registers.

[0156] Example 20. The apparatus according to Example 19 includes a virtual device constituting module that decomposes the virtual device by initiating a functional level reset of the virtual device and unmapping the allocated resources and associated register pages from the MMIO memory space for the virtual device.

[0157] Example 21. The apparatus according to Example 15, including the virtual device constituting module, allocates additional resources to the virtual device at least in part based on requests received during runtime.

[0158] Various examples can be implemented using hardware components, software components, or a combination of both. In some examples, hardware components may include devices, components, processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, ASICs, programmable logic devices (PLDs), digital signal processors (DSPs), FPGAs, memory cells, logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. In some examples, software components may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application programming interfaces (APIs), instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. The determination of whether to use hardware components and / or software components to implement an example can vary depending on any number of factors, such as desired computational speed, power levels, thermal tolerance, processing cycle budget, input data rate, output data rate, memory resources, data bus speed, and other design or performance constraints, as desired for a given implementation.

[0159] Some examples may include an article of writing or at least one computer-readable medium. A computer-readable medium may include a non-transitory storage medium used to store logic. In some examples, a non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. In some examples, logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, APIs, instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof.

[0160] Some examples may be described using the phrase "in one example" or "example" along with its derivatives. These terms mean that a particular feature, structure, or property described in connection with the example is included in at least one example. The appearance of the phrase "in one example" in various places in the specification does not necessarily refer to the same example.

[0161] This document includes logical flows or schemes representing example methodologies for implementing novel aspects of the disclosed architecture. For the sake of simplicity of explanation, although one or more methodologies shown herein are illustrated and described as a series of actions, those skilled in the art will understand and appreciate that the methodologies are not restricted by the order of the actions. Accordingly, some actions may occur in an order different from that shown and described herein and / or occur concurrently with other actions. For example, those skilled in the art will understand and appreciate that methodologies may alternatively be represented as a series of interleaved states or events (such as in a state diagram). Furthermore, not all actions described in the methodologies are required by the novel implementation.

[0162] The logic flow or scheme can be implemented in software, firmware, and / or hardware. In software and firmware embodiments, the logic flow or scheme can be implemented by computer-executable instructions stored on at least one non-transitory computer-readable medium or machine-readable medium (such as optical, magnetic, or semiconductor storage). Embodiments are not limited in this context.

[0163] The terms “coupled” and “connected” along with their derivatives are used to describe some examples. These terms are not necessarily intended to be synonyms with each other. For example, a description using the terms “connected” and / or “coupled” may indicate that two or more elements are in direct physical or electrical contact with each other. However, the term “coupled” can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0164] It is important to emphasize that this abstract of disclosure is provided to comply with 37 CFR Section 1.72(b), requiring an abstract that will allow the reader to quickly grasp the nature of the technical disclosure. It is argued, where understood, that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, in the foregoing detailed description, it can be seen that various features are grouped together in a single example for the purpose of simplification of this disclosure. This method of disclosure should not be construed as reflecting an intention that the claimed example requires more features than are expressly recited in each claim. More precisely, as reflected in the following claims, the inventive subject lies in fewer features than all of the individual disclosed examples. Therefore, the following claims are thus incorporated into the detailed description, wherein each claim is an independent, separate example. In the appended claims, the terms “comprising” and “therein” are used as concise English equivalents of the respective terms “comprising” and “therein.” Furthermore, the terms “first,” “second,” “third,” etc., are used merely as labels and are not intended to impose numerical requirements on their objects.

Claims

1. A calculation method, comprising: In operating systems compatible with S-IOV (Scalable Input / Output I / O Virtualization): Virtual Device Components (VDCM) execution: The Virtual Device (VDEV) is used to transfer data to and receive data from resources of a shared physical device by accessing the Assignable Device Interface (ADI). The virtual device is assigned to a virtual machine, which is used to access the resources of the shared physical device by accessing the ADI.

2. The method of claim 1, wherein the ADI memory-mapped I / O MMIO register is contained in one or more of an address range mapped by the peripheral component interface PCI Express base address register BAR of the virtual device.

3. The method according to claim 1, comprising: The Virtual Machine Monitor (VMM) creates the virtual machine.

4. The method of claim 1, wherein the virtual machine accessing the resources of the shared physical device by accessing the ADI includes accessing one or more of the following: a transmit / receive queue, a command and completion queue, or a compute context.

5. The method of claim 1, wherein the request originates from the ADI: Associated with the requester ID of the Physical Function PF shared with at least one other ADI. Associated with the process address space identifier PASID used to distinguish the at least one other ADI, and Associated with the Interrupt Message Storage IMS for storing interrupt messages for the ADI.

6. At least one non-transitory computer-readable medium, including instructions stored thereon, which, if executed by one or more processors, cause the one or more processors to: Execute an operating system (OS) compatible with Scalable Input / Output I / O Virtualization (S-IOV) to execute Virtual Device Components (VDCM) to: The Virtual Device (VDEV) is used to transfer data to and receive data from resources of a shared physical device by accessing the Assignable Device Interface (ADI). The virtual device is assigned to a virtual machine, which is used to access the resources of the shared physical device by accessing the ADI.

7. The at least one non-transitory computer-readable medium of claim 6, wherein the ADI memory-mapped I / O MMIO register is contained in one or more of an address range mapped by the peripheral component interface PCI Express base address register BAR of the virtual device.

8. The at least one non-transitory computer-readable medium of claim 6, wherein the virtual machine for accessing the resources of the shared physical device by accessing the ADI includes accessing one or more of the following: a transmit / receive queue, a command and completion queue, or a computing context.

9. The at least one non-transitory computer-readable medium of claim 6, wherein the request from the ADI is associated with: The requester ID of the Physical Function PF shared with at least one other ADI. The process address space identifier PASID used to distinguish the at least one other ADI, and An interrupt message storage IMS is used to store interrupt messages for the ADI.

10. The at least one non-transitory computer-readable medium of claim 6, comprising instructions stored thereon, which, if executed by one or more processors, cause the one or more processors to: Execute the Virtual Machine Monitor (VMM) to create the virtual machine.

11. A computing device, comprising: A device includes resources that can be accessed by a virtual machine as a scalable input / output I / O virtualization S-IOV-compliant virtual device VDEV and are accessible by an assignable device interface ADI.

12. The apparatus of claim 11, wherein the apparatus includes a graphics processing unit and the ADI is associated with a graphics context of a virtual GPU device instance.

13. The apparatus of claim 11, wherein the apparatus includes a network controller device and the ADI is associated with a receive and transmit queue for storing packets.

14. The apparatus of claim 11, wherein the apparatus includes a storage controller device and the ADI is associated with a command and completion queue pair.

15. The apparatus of claim 11, wherein the apparatus includes an accelerator and the ADI is associated with an execution context.

Images