MANAGEMENT OF VIRTUAL I / O DEVICES

Abstract

A network interface device runs an input / output (I / O) virtualization manager to identify a virtual device defined as containing resources of a specific virtual function within a variety of virtual functions associated with a physical function of the device. It identifies an operation to be performed between the virtual device and a system image hosted by a host system connected to the network interface device. The network interface device emulates the virtual device in operation through the I / O virtualization manager.

Description

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority under 35 USC § 119(b) to International Application No. PCT / CN2023 / 118655, filed September 13, 2023, which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0002] This disclosure relates generally to the field of distributed computing systems and, more particularly, to I / O virtualization. BACKGROUND

[0003] A data center may include one or more platforms, each containing at least one processor and associated memory modules. Each data center platform may facilitate the execution of any number of processes associated with various applications running on the platform. These processes may be performed by the processors and other associated logic on the platforms. Each platform may additionally include I / O controllers, such as network adapters, that can be used to send and receive data over a network for use by the various applications.

[0004] Some platforms leverage I / O virtualization to improve data center performance. The Peripheral Component Interconnect (PCI) Special Interest Group (PCI-SIG) Single Root I / O Virtualization (SR-IOV) and Sharing specification, version 1.0 (2007), provides hardware-assisted high-performance I / O virtualization and sharing of PCI Express devices. Intel® Scalable IOV (SIOV) and Application Defined Infrastructure (ADI) are additional I / O virtualization specifications that can significantly expand the current limit on the number of Peripheral Component Interconnect Express (PCIe) devices to increase the number of containers or services that can utilize a PCIe device. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows a block diagram of components of a data center in accordance with certain embodiments. Fig. Figure 2 is a simplified block diagram of an example system implementing I / O virtualization. Fig. Figure 3 is a simplified block diagram showing an example of a computing system. Fig. Figure 4 is a simplified block diagram showing an example of a computing system with an enhanced network interface. Fig. Figure 5A is a simplified block diagram showing an example of an SR-IOV-capable device. Fig. Figure 5B is a simplified block diagram showing an example definition of a virtual I / O device. Fig. Figure 6 is a flowchart showing an example of assigning virtual devices to an application or other process. Fig. Figure 7 is a flowchart showing an example of using an enhanced network interface device to perform I / O virtualization of a virtual function. Fig. 8 shows a block diagram of an example processor unit according to certain embodiments.

[0005] Like reference numbers and designations in the different drawings indicate like elements. DETAILED DESCRIPTION

[0006] Fig. 1 shows a block diagram of the components of a data center 100 according to certain embodiments. In the illustrated embodiment, the data center 100 includes a plurality of platforms 102, a data analytics engine 104, and a data center management platform 106 interconnected via a network 108. A platform 102 may include platform logic 110 including one or more central processing units (CPUs) 112, memory 114 (which may include any number of different modules), chipsets 116, communication interfaces 118, and other suitable hardware and / or software to execute a hypervisor 120 or other operating system capable of executing processes associated with the applications running on the platform 102. In some embodiments, a platform 102 may act as a host platform for one or more guest systems 122 that invoke these applications.The platform can be divided logically or physically into clusters, and these clusters can be enhanced, among other things, by specialized network accelerators and the use of CXL (Compute Express Link) storage semantics to increase the efficiency of such clusters.

[0007] A platform 102 may include platform logic 110. The platform logic 110 includes, among other things, logic that enables the functionality of the platform 102, one or more CPUs 112, memory 114, one or more chipsets 116, and communication interfaces 118. Although three platforms are illustrated, the data center 100 may include any number of platforms. In various embodiments, a platform 102 may be located on a circuit board installed in a chassis, a rack, possibly servers, disaggregated servers, or other suitable structures that include multiple platforms interconnected via a network 108 (which may include, for example, a rack or backplane switch).

[0008] The CPUs 112 may consist of any number of processor cores. The cores may be coupled to each other, to memory 114, to at least one chipset 116, and / or to a communications interface 118 through one or more controllers located on the CPU 112 and / or the chipset 116. In certain embodiments, a CPU 112 is formed in a socket that is permanently or removably coupled to the platform 102. Although four CPUs are illustrated, a platform 102 may include any suitable number of CPUs.In some implementations, the applications to be executed with the CPU (or other processors) may include physical layer management applications that enable customized software-based configuration of the physical layer of one or more interconnects used to couple the CPU (or associated processor devices) to one or more other devices in a data center system.

[0009] The memory 114 may include, without limitation, any form of volatile or non-volatile storage, magnetic media (e.g., one or more tape drives), optical media, random access memory (RAM), read-only memory (ROM), flash memory, removable media, or any other suitable local or remote storage component or components. The memory 114 may be used for short-, medium-, and / or long-term storage by the platform 102. The memory 114 may store any suitable data or information utilized by the platform logic 110, including software embedded in a computer-readable medium and / or coded logic (e.g., firmware) built into or otherwise stored in hardware. The memory 114 may store data utilized by cores of the CPUs 112.In some embodiments, memory 114 may also store instructions that can be executed by the cores of CPUs 112 or other processing elements (e.g., logic on chipsets 116) to provide the functions associated with the components of platform logic 110. Additionally or alternatively, chipsets 116 may include memory having all of the characteristics of memory 114 described herein. Memory 114 may also store the results and / or intermediate results of the various calculations and determinations performed by CPUs 112 or the processing elements on chipsets 116. In various embodiments, memory 114 may consist of one or more modules of system memory connected to the CPUs via memory controllers (which may be external or integrated with CPUs 112).In various embodiments, one or more particular modules of memory 114 may be associated with a particular CPU 112 or other processing device, or may be shared by multiple CPUs 112 or other processing devices.

[0010] A platform 102 may also include one or more chipsets 116, which may include any suitable logic to support the operation of the CPUs 112. In various embodiments, the chipset 116 may be located on the same package as the CPU 112 or on one or more different packages. A chipset may support any suitable number of CPUs 112. A chipset 116 may also include one or more controllers to couple other components of the platform logic 110 (e.g., communication interface 118 or memory 114) to one or more CPUs. Additionally or alternatively, the CPUs 112 may include integrated controllers. For example, the communication interface 118 could be coupled directly to the CPUs 112 via integrated I / O controllers located on the respective CPUs.

[0011] Chipsets 116 may include one or more communication interfaces 128. Communication interface 128 may be used to communicate signals and / or data between chipset 116 and one or more I / O devices, one or more networks 108, and / or one or more devices connected to network 108 (e.g., data center management platform 106 or data analytics engine 104). For example, communication interface 128 may be used to send and receive network traffic, such as data packets. In a particular embodiment, communication interface 128 may be implemented by one or more I / O controllers, such as one or more physical network interface controllers (NICs), also known as network interface cards or network adapters.An I / O controller may include electronic circuitry to communicate using any suitable physical layer and data link layer standard, such as Ethernet (e.g., according to the IEEE 802.3 standard), Fibre Channel, InfiniBand, Wi-Fi, or other suitable standards. An I / O controller may include one or more physical ports that can be connected with a cable (e.g., an Ethernet cable). An I / O controller may enable communication between any suitable element of chipset 116 (e.g., switch 130) and another device connected to network 108. In some embodiments, network 108 may include a switch with bridging and / or routing capabilities located external to platform 102 and used to interconnect various I / O controllers (e.g., NICs) distributed throughout data center 100 (e.g., on different platforms).In various embodiments, an I / O controller may be integrated into the chipset (i.e., it may be located on the same integrated circuit or circuit board as the rest of the chipset logic), or it may be located on a different integrated circuit or circuit board that is electromechanically coupled to the chipset. In some embodiments, communication interface 128 may also enable communication from platform-integrated or external I / O devices (e.g., hard disk drives, other NICs, etc.) with the CPU cores.

[0012] Switch 130 may couple to various ports (e.g., of NICs) of communications interface 128 and mediate data between those ports and various components of chipset 116 according to one or more interconnect or link protocols, such as Peripheral Component Interconnect Express (PCIe), Compute Express Link (CXL), HyperTransport, GenZ, OpenCAPI, and others, each of which may alternatively or jointly apply the general principles and / or specific features discussed herein. Switch 130 may be embodied as a physical or virtual switch (i.e., software).

[0013] The platform logic 110 may include an additional communication interface 118. Similar to the communication interfaces 128, the communication interfaces 118 may be used to communicate signals and / or data between the platform logic 110 and one or more networks 108 and one or more devices coupled to the network 108. For example, the communication interface 118 may be used to send and receive network traffic, such as data packets. In a particular embodiment, the communication interface 118 includes one or more physical I / O controllers (e.g., NICs). These NICs may enable communication between any suitable element of the platform logic 110 (e.g., the CPUs 112) and another device connected to the network 108 (e.g., elements of other platforms or remote nodes connected to the network 108 through one or more networks).In certain embodiments, the communication interface 118 may enable devices external to the platform (e.g., hard disk drives, other NICs, etc.) to communicate with the CPU cores. In various embodiments, the NICs of the communication interface 118 may be connected to the CPUs via I / O controllers (which may be external or integrated with the CPUs 112). Furthermore, as described herein, I / O controllers may include a power manager 125 to implement functions to manage power consumption at the I / O controller (e.g., by automatically implementing power savings on one or more interfaces of the communication interface 118 (e.g., a PCIe interface connecting a network card to another element of the system), among other example functions.

[0014] The platform logic 110 can receive and execute any suitable type of processing request. A requested processing may include any request to utilize one or more resources of the platform logic 110, such as one or more cores or connected logic. A processing request may include, for example, a processor core interrupt, a request to instantiate a software component, such as an I / O device driver 124 or a virtual machine 132, a request to process a network packet received from a virtual machine 132 or a device external to the platform 102 (such as a network node connected to the network 108), a request to execute a workload (e.g.,a process or thread) associated with a virtual machine 132, an application running on the platform 102, a hypervisor 120, or another operating system running on the platform 102, or any other suitable requirement.

[0015] In various embodiments, request processing may be associated with guest systems 122. A guest system may include a single virtual machine (e.g., virtual machine 132a or 132b) or multiple virtual machines operating together (e.g., a virtual network function (VNF) 134 or a service function chain (SFC) 136). As illustrated, various embodiments may include a variety of guest system types 122 on the same platform 102.

[0016] A virtual machine 132 may emulate a computer system with its own dedicated hardware. A virtual machine 132 may run a guest operating system on the hypervisor 120. The components of the platform logic 110 (e.g., CPUs 112, memory 114, chipset 116, and communications interface 118) may be virtualized so that the virtual machine 132 appears to the guest operating system to have its own dedicated components.

[0017] A virtual machine 132 may include a virtualized network interface (vNIC) used by the virtual machine as its network interface. A vNIC may be assigned a Media Access Control (MAC) address, allowing multiple virtual machines 132 on a network to be individually addressable.

[0018] In some embodiments, a virtual machine 132b may be paravirtualized. For example, the virtual machine 132b may include enhanced drivers (e.g., drivers that provide higher performance or have higher-bandwidth interfaces to the underlying resources or capabilities provided by the hypervisor 120). For example, an enhanced driver may have a faster interface to the underlying virtual switch 138 to achieve higher network performance compared to standard drivers.

[0019] VNF 134 may comprise a software implementation of a functional building block with defined interfaces and behavior that can be deployed in a virtualized infrastructure. In certain embodiments, a VNF 134 may comprise one or more virtual machines 132 that collectively provide specific functionalities (e.g., wide area network (WAN) optimization, virtual private network (VPN) termination, firewall operations, load balancing operations, security functions, etc.). A VNF 134 running on platform logic 110 may provide the same functionality as traditional network components implemented by dedicated hardware. For example, a VNF 134 may include components to perform any suitable NFV workloads, such as virtualized Evolved Packet Core (vEPC) components, mobility management entities, 3rd Generation Partnership Project (3GPP) control, and data plane components, etc.

[0020] SFC 136 is a group of VNFs 134 organized as a chain to perform a series of operations, such as network packet processing operations. Service function chaining can provide the ability to define an ordered list of network services (e.g., firewalls, load balancers) stitched together in the network to create a service chain.

[0021] A hypervisor 120 (also known as a virtual machine monitor) may include logic to create and execute guest operating systems 122. The hypervisor 120 may provide guest operating systems executed by virtual machines with a virtual operating platform (i.e., it appears to the virtual machines that they are running on separate physical nodes when they are actually consolidated on a single hardware platform) and manage the execution of the guest operating systems through platform logic 110. Services of the hypervisor 120 may be provided by virtualization in software or through hardware-assisted resources, or both, requiring minimal software intervention. Multiple instances of a plurality of guest operating systems may be managed by the hypervisor 120. A platform 102 may have a separate instantiation of a hypervisor 120.

[0022] The hypervisor 120 may be a native or software-independent hypervisor that runs directly on the platform logic 110 to control the platform logic and manage the guest operating systems. Alternatively, the hypervisor 120 may be a hosted hypervisor that runs on a host operating system and abstracts the guest operating systems from the host operating system. Various embodiments may include one or more non-virtualized platforms 102, in which case any suitable features or functions of the hypervisor 120 described herein may apply to an operating system of the non-virtualized platform. For advanced I / O virtualization, other implementations may be supported, such as that described above. A host operating system may identify conditions and configurations of a system and determine that functions (e.g.,SIOV-based virtualization of SR-IOV-based devices) can be enabled or disabled, and can use corresponding application programming interfaces (APIs) to send and receive information about such activation or deactivation, among other example functions.

[0023] The hypervisor 120 may include a virtual switch 138 that may provide virtual switching and / or routing functions to the virtual machines of guest systems 122. The virtual switch 138 may include a logical switching fabric that couples the virtual network interfaces (VNICs) of the virtual machines 132 together to create a virtual network through which the virtual machines can communicate with each other. The virtual switch 138 may also be connected to one or more networks (e.g., network 108) via physical NICs of the communication interface 118 to enable communication between virtual machines 132 and one or more network nodes external to the platform 102 (e.g., a virtual machine running on another platform 102 or a node connected to the platform 102 via the Internet or another network).The virtual switch 138 may include a software element executed using components of the platform logic 110. In various embodiments, the hypervisor 120 may communicate with any entity (e.g., an SDN controller), which may cause the hypervisor 120 to reconfigure the parameters of the virtual switch 138 in response to changing conditions in the platform 102 (e.g., the addition or deletion of virtual machines 132 or the identification of optimizations that can be made to increase the performance of the platform).

[0024] Hypervisor 120 may include any number of I / O device drivers 124. I / O device driver 124 represents one or more software components that enable hypervisor 120 to communicate with a physical I / O device. In various embodiments, the underlying physical I / O device may be connected to one of CPUs 112 and may send data to and receive data from CPUs 112. The underlying I / O device may use any suitable communication protocol, such as PCI, PCIe, Universal Serial Bus (USB), Serial Attached SCSI (SAS), Serial ATA (SATA), InfiniBand, Fibre Channel, an IEEE 802.3 protocol, an IEEE 802.11 protocol, or another current or future signaling protocol.

[0025] The underlying I / O device may include one or more ports that can be used to communicate with the cores of CPUs 112. In one example, the underlying I / O device is a physical network card or a physical switch. In one embodiment, the underlying I / O device of I / O device driver 124 is, for example, a NIC of communication interface 118 with multiple ports (e.g., Ethernet ports). In some implementations, I / O virtualization may be supported within the system and utilize the techniques described in more detail below. I / O devices may support I / O virtualization based on SR-IOV, SIOV, and other example techniques and technologies.

[0026] In other embodiments, the underlying I / O devices may include any suitable device capable of transmitting and receiving data to and from the CPUs 112, such as an audio / video device controller (e.g., a graphics accelerator or audio controller), a data storage controller such as flash memory, a magnetic disk or optical disk controller, a wireless transceiver, a network processor, or a controller for another input device such as a monitor, printer, mouse, keyboard, or scanner, or any other suitable device.

[0027] In various embodiments, upon receiving a processing request, the I / O device driver 124 or the underlying I / O device may send an interrupt (e.g., a message-signaled interrupt) to one of the cores of the platform logic 110. For example, the I / O device driver 124 may send an interrupt to a core selected to perform an operation (e.g., on behalf of a virtual machine 132 or a process of an application). Before the interrupt is forwarded to the core, incoming data (e.g., network packets) destined for the core may be cached in the underlying I / O device and / or an I / O block connected to the core's CPU 112. In some embodiments, the I / O device driver 124 may configure the underlying I / O device with instructions regarding where to send interrupts.

[0028] In some embodiments, when distributing workloads across cores, hypervisor 120 may direct a greater number of workloads to the more powerful cores than to the less powerful cores. In certain cases, cores experiencing issues such as overheating or heavy load may be assigned fewer tasks than other cores or avoided entirely (at least temporarily). Workloads associated with applications, services, containers, and / or virtual machines 132 may be distributed across cores based on network load and traffic patterns, not just CPU and memory utilization metrics.

[0029] The elements of platform logic 110 may be coupled together in any suitable manner. For example, a bus may couple any of these components together. A bus may include any known interconnect, such as a multi-station bus, a mesh interconnect, a ring interconnect, a point-to-point interconnect, a serial interconnect, a parallel bus, a coherent bus (e.g., cache coherent), a layered protocol architecture, a differential bus, or a Gunning Transceiver Logic (GTL) bus.

[0030] The elements of data system 100 may be interconnected in any suitable manner, for example, via one or more networks 108. A network 108 may be any suitable network or a combination of one or more networks operating using one or more suitable network protocols. A network may represent a series of nodes, points, and connected communication paths for receiving and sending information packets transmitted through a communication system. For example, a network may include one or more firewalls, routers, switches, security appliances, antivirus servers, or other useful network devices.A network provides communication interfaces between sources and / or hosts and may include a local area network (LAN), a wireless local area network (WLAN), a metropolitan area network (MAN), an intranet, an extranet, the Internet, a wide area network (WAN), a virtual private network (VPN), a cellular network, or any other suitable architecture or system that facilitates communication in a network environment. A network may consist of any number of hardware or software elements that are interconnected (and communicate with each other) via a communication medium. In various embodiments, guest systems 122 may communicate with nodes external to data center 100 via network 108.

[0031] Single Root I / O Virtualization (SR-IOV) is a PCI-SIG specification for hardware-assisted I / O virtualization that defines a standard method for partitioning endpoint devices for direct sharing by multiple VMs or containers. An SR-IOV-enabled endpoint device provides one physical function (PF) and multiple virtual functions (VF). A device's PF in SR-IOV provides resource management for the device and is managed by a host driver running within the host operating system (OS). A deployed VF can be assigned to a VM or container for direct access. SR-IOV-enabled devices can provide high-performance I / O, including I / O devices such as network and storage controller devices, as well as programmable or reconfigurable devices such as GPUs, FPGAs, and other accelerators, among other examples.

[0032] Scalable IOV (SIOV) also aims to define an approach for virtualizing I / O, e.g., within a data center. SIOV provides hardware-assisted I / O virtualization, enabling a higher degree of scalability and performance when sharing I / O devices in isolated domains (e.g., VMs and containers). SIOV enables the flexible composition of virtual devices for device sharing. Access between a VM and a virtual device is defined in SIOV as either "direct path" access or "broken path" access. Direct path operations on the virtual device are mapped directly to the underlying device hardware for performance reasons, while intercept path operations are at least partially emulated in software by a Virtual Device Composition Module (VDCM) to enable this greater flexibility in I / O virtualization.Which operations and accesses are processed as intercepted paths and which are processed as direct paths can vary depending on the device implementation and application. For example, slow-path operations (e.g., initialization, control, configuration, management, QoS, error processing, and reset) are treated as intercepted path accesses, and fast-path operations (e.g., work submission and work completion processing) are treated as direct path accesses, to name a few examples.

[0033] Similar to SR-IOV, the resources of a given physical device can be assigned to individual VMs. SIOV takes a more adaptable and granular approach by enabling the flexible definition of virtual devices (VDEVs) that can be assigned to a specific VM. High-performance I / O devices can contain a large number of command / completion interfaces for efficient I / O multiplexing / demultiplexing. SIOV platforms can enable the assignment of such interfaces to isolated domains with fine granularity. A SIOV architecture defines the granularity of sharing a device or device resource as an "Assignable Device Interface" (ADI). Each ADI instance on the device can comprise the set of resources on the device allocated by software to support direct-path operations for a virtual device.For example, resources on a device associated with the submission, execution, and completion of operations may implement device backend resources (e.g., command / status registers, in-device queues, references to in-memory queues, local storage on the device, or other device-specific internal constructs). An ADI may identify a set (e.g., all or part of the total device resources, or even a combination of resources from two or more individual devices) of device backend resources that are allocated, configured, and organized as an isolated unit and form the unit of device sharing. The type and number of backend resources grouped into an ADI may be device-specific. Each SIOV ADI on a device function may use the same PCIe requester ID (Bus / Device / Function (BDF) number) that corresponds to the device's PCIe function.Process Address Space Identifiers (PASID) can be used to distinguish upstream memory transactions for different ADIs and to convey the address space targeted by the transaction.

[0034] ADIs form the unit of allocation and isolation for devices and are software-assembled into virtual devices (VDEVs). A Virtual Device Composition Module (VDCM) is responsible for managing virtual device instances. For direct-path accesses, for example, a VMM can map the guest's direct-path accesses directly to the ADIs provided to the VDEV. For intercepted-path accesses, the VMM identifies the guest's intercepted-path accesses and forwards them to the VDCM for emulation. The VDCM emulates the intercepted accesses to the VDEV. In some cases, the VDCM can access the underlying physical device corresponding to the ADI (e.g., to read a corresponding device register, determine the ADI status, configure the ADI's PASID, etc.).Virtual device composition, among other benefits, enables greater scalability and sharing flexibility with lower hardware cost and complexity. SIOV uses software to define and share device resources with different address ranges using various VDEV abstractions. For example, application processes can access a device through system calls, and VMs can access a device through virtual device interfaces. Virtual device composition can also enable dynamic allocation of VDEVs to device resources, allowing a VMM to provide VMs with a surplus of device resources. For example, the resources of one or more physical devices can be mapped to a specific VDEV. VDEVs can therefore be defined to achieve specific system goals.For example, in a data center with different physical machines containing different generations (e.g., versions) of the same I / O device, VDEVs can be defined to provide the same VDEV functionality regardless of the different generations of physical I / O devices used in the VDEV definitions. Such a solution can enable the same guest operating system image with a specific VDEV driver to be deployed or migrated to different combinations or implementations of physical machines.

[0035] During operation, upstream memory requests from all ADIs (within the respective VDEV associated with different VMs or containers) can be tagged with the requester ID of the device (or device function) hosting the ADIs. Requests from different ADIs of the device function can be differentiated using a process address space identifier (PASID). The requester ID and / or the PASID can be used to identify (e.g., in a TLP prefix) the address space associated with the request. Accordingly, when assigning an ADI to an address domain (e.g., VM, container, or process), the ADI can be configured with a unique PASID of the address domain, and its memory requests can be tagged with the PASID value (e.g., in a PASID TLP prefix).

[0036] As mentioned previously, a VDEV in SIOV can serve as an abstraction through which a shared physical device is exposed to guest software. In some implementations, a VDEV can be exposed to a guest operating system as a virtual PCI Express device. A VDEV can be defined to have virtual resources such as a virtual requester ID, virtual configuration space registers, virtual memory BARs, a virtual MSI-X table, etc. Each VDEV can be mapped to, or formed from, one or more ADIs (corresponding to different devices or device functions). The ADIs supporting a VDEV can belong to the same physical function or be distributed across multiple functions (e.g., to support fault tolerance or load balancing).

[0037] Fig. 2 is a simplified block diagram 200 illustrating a conventional implementation of an example operating environment 200 that supports the SIOV architecture to virtualize one or more devices (e.g., 205) such as component devices on a particular computing platform or other packages such as accelerators, I / O devices, network processing devices, etc. In this example, the operating environment may include a host operating system (OS) 202, a guest operating system (OS) 208, a VMM 212, an input / output memory management unit (IOMMU) 214, and one or more devices (e.g., 205) that have I / O resources that can be virtualized (e.g., based on SR-IOV or SIOV, etc.). The host operating system 202, the guest operating system 208, and / or the VMM 212 may execute on the host hardware 104. The host operating system 202 may include a host driver 220 and the guest operating system 208 may include a guest driver 210.

[0038] As shown, in conventional embodiments of SIOV environments, the host operating system 202 may include software 204 that may form a virtual device (VDEV) 222 for the guest operating system 208. In some embodiments, the VDEV 222 may include virtual capability registers configured for one or more components of the operating environment 200 to expose device (or "device-specific") capabilities. In various embodiments, the guest driver 210 of the device 205 may access virtual capability registers to determine the device capabilities associated with the VDEV 222. The VDEV 222 may include one or more assignable device interfaces (ADIs) (also referred to as "assignable interfaces"), including an ADI 206a and an ADI 206b. In some embodiments, an ADI may be assigned, e.g., B. by assigning the ADIs 206a-206b to an MMIO room of VDEV 222.An ADI generally refers to the set of backend resources 218 of device 205 that are allocated, configured, and organized as an isolated unit and constitute the unit of shared use of device 205. The type and number of backend resources 218 that are grouped into a particular ADI 206a, 206b may be specific to device 205. An ADI 206a, 206b may be associated with a device context rather than specific device resources. As another example, the backend resources 218 of ADIs 206a-206b may include one or more shared work queues. A repository (not shown) or other data structure may store a plurality of different ADIs and the respective attributes of each ADI.

[0039] For example, if device 205 is a network controller, ADIs 206a-206b may provide backend resources 218 that include send and receive queues associated with a virtual switch interface. As another example, if device 205 is a storage device, ADIs 206a-206b may provide backend resources 218 that include command queues and completion queues associated with a memory namespace. If device 205 is a graphics processing unit (GPU), ADIs 206a-206b may provide backend resources 218 that include, among other things, dynamically created graphics or compute contexts.

[0040] The IOMMU 214 may be configured to perform memory management operations, including address translations between virtual memory regions and physical memory. As shown, the IOMMU 214 may support translations at the process address space ID (PASID) level. Generally, each of a plurality of processes executing on the host hardware 104 (e.g., processes associated with the guest operating system 208 and / or VMs) may be assigned a PASID. In this manner, the device 205 may be shared among multiple processes, with each process having a full virtual address space available.

[0041] In some implementations, the software 204 may implement a VDCM. In some cases, a separate instance of the software 204 (or VDCM) may be provided for each device to be virtualized. For example, a VDCM may be implemented as a device-specific component responsible for assembling and implementing VDEV instances 222 using one or more ADIs allocated, for example, by a host driver 220. The VDCM implements software-based virtualization of operations on intercepted paths and ensures that operations on direct paths are submitted directly to the supporting ADIs. The host driver 220 may be loaded, and DCMs may be implemented and packaged by device vendors in various ways, for example, as user-space modules or libraries installed as part of the host driver. In other implementations, the VDCM may be a kernel module.If the VDCM is implemented as a library, it can be statically or dynamically linked to the hypervisor-specific virtual machine resource manager, which is responsible for creating and managing VM resources. If implemented in the host kernel, the VDCM can be part of the host driver. The host driver is loaded and executed as part of the host operating system or hypervisor software. The host driver can report support for SIOV (and / or SR-IOV) to the system software through the driver interface. In addition to traditional device driver functions, the host driver can implement 220 software interfaces (e.g., as defined by the host operating system or hypervisor infrastructure) to support the enumeration, configuration, instantiation, and management of ADIs.The host driver may be responsible for configuring the ADIs, including aspects such as PASID identity, interrupt message storage entries, MMIO register resources for direct-path access to the ADI, and any device-specific resources, among other example functions and features. A SIOV-compliant guest driver 210 may manage the VDEV instances assembled by the VDCM. Direct-path accesses by the guest driver 210 may be issued directly to the ADIs (e.g., 206a-b) associated with the VDEV, while interrupted-path accesses are intercepted and virtualized by the VDCM (e.g., 204). In some implementations, the guest and host drivers may be implemented as a unified driver supporting both host and guest functions, or as two separate drivers.For existing SR-IOV devices, the Intel Scalable IOV guest driver can be used like the SR-IOV VF driver if the VDEV can be composed to behave like an existing VF, to name just one example.

[0042] In Fig. Figure 3 illustrates a simplified block diagram 300 illustrating an example of a network interface device 305, such as an infrastructure processing unit (IPU), an intelligent network interface controller (Smart NIC), or other network device that includes network interfaces, network circuitry, memory, and a microcontroller (e.g., CPU). In this example, a network interface device 305 is coupled to host hardware 304 and device 205 via a PCIe interface 306. The network interface device 305, host hardware 304, device 205, and PCIe interface 306 may be implemented in circuitry. For example, the network interface device 305, the host hardware 304, the device 205, and the PCIE interface 306 may be communicatively coupled components of a compute node, a server blade, a server rack, or other computer hardware.Host hardware 304 includes at least processor circuitry and memory (each not shown). Network interface device 305 also includes at least a processor 312, memory 314, an accelerator 318, and a network interface device 316 (e.g., a wired and / or wireless Ethernet network interface). Network interface device 316 may provide an interface to other devices over a network (e.g., a local area network (LAN), a wide area network (WAN), the Internet, etc.) and may support Institute of Electrical and Electronics Engineers (IEEE) Ethernet standards (e.g., 802.1, 802.3, etc.). Device 205 may be any type of peripheral device, such as a PCIe-compliant device. Although PCIe interface 306 is used as a reference example of an interface, other interfaces may be used in the operating environment.For example, a Compute Express Link® (CXL) interface, a Peripheral Component Interconnect (PCI) interface, a Universal Serial Bus (USB) interface, a Serial Peripheral Interconnect (SPI), an Integrated Interconnect (I2C), or a Universal Chiplet Interconnect Express (UCIe) interface may be used instead of the PCIe interface 306. Therefore, the device 205 may be a USB device, a PCI device, a PCIe device, a CXL device, a UCIe device, an I2C device, and / or an SPI device and may provide I / O resources (e.g., for virtualization) in accordance with the appropriate I / O protocol or technology. The network interface device 305 also includes a Direct Memory Access (DMA) engine to facilitate DMA transactions, among other example features.

[0043] The host hardware 304 may consist of one or more processors and memory to run one or more virtual machines (VMs), such as VM 308a, VM 308b, and VM 308c (or other containers or other isolated domains). The network interface device 305 includes one or more programmable or fixed function processors to offload operations that could have been performed by the processors of the host hardware 304. The network interface device 305 may therefore be considered an "offload device." In general, the network interface device 305 may perform virtual switch operations, manage memory transactions (e.g., compression, cryptography, virtualization), and manage operations running on other IPUs, compute nodes, servers, and / or devices.An enhanced network interface device 305 may include SIOV and SR-IOV logic that enables SIOV functionality to be transferred from the host to the network interface device 305 (e.g., the VDCM or other ADI management logic).

[0044] An example network interface device 305 may handle I / O, initialization, resource management, security, error handling, quality of service (QoS) handling, and control. Conventionally, I / O, resources, security, and control may be performed by host hardware 304. These functions include virtualization of devices such as device 205. Device 205 may be any type of device, such as a network interface device, an acceleration device, a storage device, and the like. Although depicted as external to network interface device 305, in some embodiments, device 205 may be a component of network interface device 305. Similarly, although depicted as external to host hardware 304, in some embodiments, device 205 is a component of host hardware 304, among other example implementations.In some implementations, network interface device 305 may include I / O virtualization logic (implemented in hardware and / or software) to virtualize aspects of device 205. For example, device 205 may be virtualized by network interface device 305 for VMs 308a-308c based on the SIOV architecture. Similarly, accelerator 318, network interface device 316, and other components (and device functions) of network interface device 305 may be virtualized using the S-IOV architecture (e.g., if device 205, network interface device 316, and accelerator 318 are SIOV-compliant or compatible).

[0045] By offloading SIOV functionality from the host (e.g., the host kernel) to a network interface device (e.g., 305), host resources can be conserved, additional security can be provided (e.g., through enhanced security features implemented on the network interface device), and additional functionality and enhancements can be provided via the network interface device 305, among other examples of uses and benefits. For example, VDEVs comprising multiple physical functions can be enabled, where the multiple physical functions can be provided by one or more devices 205 and / or components of the network interface device 305. Furthermore, the network interface device 305 can also support SR-IOV and include SR-IOV-based physical functions and virtual functions.In some implementations, SR-IOV resources and functions can be mapped to SIOV ADIs by equipping the network interface device with SIOV logic (e.g., an I / O virtualization manager, ADI manager, VDCM, etc.) to backport SIOV functions to SR-IOV devices and functions.

[0046] SR-IOV is a primarily hardware-implemented standard that allows a PCIe device, such as a network interface card (NIC), to present itself to a hypervisor as multiple virtual NICs. This capability is particularly advantageous for efficiently allocating PCIe devices, including widely deployed Ethernet devices, between virtual machines (VMs) in cloud service systems. While SIOV provides an alternative, software-enhanced I / O virtualization scheme, SR-IOV is considerably more widespread and widely deployed. Furthermore, SR-IOV remains an important option for VM-level use, even when using a virtual I / O memory management unit (vIOMMU), when the VM kernel, rather than an application, directly accesses the device (e.g., due to SR-IOV's relative simplicity and slightly better performance (e.g., without the need for a Process Address Space ID (PASID) header in each packet).It would therefore be beneficial if some of the additional features (e.g., address translation, live migration, etc.) and finer-grained virtualization provided by SIOV could be ported to SR-IOV devices to take advantage of both technologies.

[0047] As is often the case with hardware-defined functions, SR-IOV is expensive to manufacture and difficult to upgrade. This can be particularly challenging as cloud infrastructure continues to evolve, with additional high-value features being developed to meet the needs of cloud providers and customers. For example, there have been advances in address translation (e.g., to improve PCIe address translation performance) that may not be supported by some SR-IOV devices, requiring many new features that depend on SR-IOV to be added quickly and in bulk. One example is the Address Translation Engine (ATE) component in the Intel IPU. It is a hardware design to improve PCIe address translation performance.To make it acceptable to the open-source community and usable for the general market, some design changes related to SR-IOV are required, which are not feasible for the current generation of IPUs and expensive for future IPUs following the hardware-defined SR-IOV approach. This paper uses it as an example to illustrate the problem and the proposed solution.

[0048] Indeed, address translation has emerged as one of the most significant challenges for high-performance PCIe devices. Adding such functionality within the hardware-defined SR-IOV paradigm may be impractical without an update facilitated by a corresponding hardware development cycle. However, a software-based approach, such as in SIOV, can enable the deployment and update of such address translation functions (e.g., Address Translation Service (ATS) 2.0). An ADI subsystem and its associated software-defined SIOV capabilities (e.g., in VDCM-implemented slow-path software implementations) can be leveraged to effectively add this functionality to a device virtualized with SIOV.In an enhanced implementation, a network interface device can be equipped with SIOV logic that allows mapping SR-IOV virtual functions (VFs) to a new SIOV ADI type. Accordingly, these VFs can be treated as a subordinate device, from which the physical function's I / O virtualization manager (e.g., VDCM) can leverage hardware resources to compose a hybrid VDEV based on the SR-IOV device. Applying software-based SIOV virtualization to an SR-IOV device allows additional software-based enhancements to be applied to the SR-IOV device, such as advanced address translation, live migration, advanced encryption, etc.

[0049] As presented here, an enhanced network interface device can be provided with SIOV management logic to implement a general framework that provides SR-IOV devices with a “software-defined” capability and enables the convenient and rapid enhancement of such devices (e.g., adopting the latest technologies such as ATS 2.0 without the typical lengthy hardware upgrade cycle), among other example benefits. Fig. 4 is a simplified block diagram 400 illustrating an exemplary operating environment for assembling SIOV-based virtual devices using hybrid resources, including SR-IOV-based resources. In this example, a VMM 212 (also referred to as a hypervisor) may run or manage a VM, such as VM 212. VM 308 may include a device driver 405 for a device (e.g., network interface device 305 or associated devices, SR-IOV device 410, etc.). VMM 212 may also include a Virtual Function I / O (VFIO) PCIe emulator 412. A container 415 may execute a user application 418 and a minidriver 420. VMM 212, VM 308, and / or container 415 may execute on host hardware 425 and in user space 430.

[0050] In one example, kernel area 420 may include a VFIO ADI driver 432, a UACCE ADI driver 434, an ADI subsystem 435, and a PCI driver 440 with an ADI operation driver 445. The VFIO ADI driver 432 may correspond to a driver for a pass-through device, such as the network interface device 305 and / or the device 450. Generally, the VFIO ADI driver 432 uses a device template to assemble a virtual AVF device using the mapping between the ADI and the register addresses in the ADI entry. Therefore, the VFIO ADI driver 432 maintains the attributes of the device and allows access to the device via the same driver as the corresponding host driver. The Unified / User-Space Access-Intended Accelerator Framework (UACCE) ADI driver 434 provides Shared Virtual Addressing (SVA) between accelerators and processes, allowing an accelerator device (e.g.The device 450 and / or an accelerator component of the IPU 305 can access any data structures in the host hardware. Due to the unified address space provided by the UACCE ADI driver 434, hardware and user-space processes can use the same virtual addresses when communicating. Furthermore, the VFIO ADI driver 432 implements VFIO user-space interfaces based on various ADIs. Therefore, such VFIO user-space interfaces can be a standard PCIe device with a standard PCIe configuration space. The UACCE ADI driver 434 can be paired with the mini-driver 420, allowing the UACCE ADI driver 434 to access user space through the ADI hardware via the mini-driver 420.

[0051] The PCI driver 440 represents all PCI drivers that support virtualization, including standard drivers that comply with the PCI or PCIe specification. In one example, a PCI driver 440 may be implemented as a PCI stub driver. An example ADI PCI driver (e.g., 440) can bind to any type of device (e.g., SIOV and SR-IOV devices). If the device has a valid PCIe "ADI Extended Capability," the ADI subsystem will attempt to initialize it as an ADI primary device and read the ADI manager profile (of a sample I / O virtualization manager 470) via appropriate registers. Devices (PFs or VFs) without "ADI Extended Capability" are treated as subordinate devices, providing hardware resources to the ADI manager, which it can use in VDEVs, among other things.The PCI driver 440 may include ADI operations (ops) 445 that provide application program interfaces (APIs) for at least one driver to interface with the ADI subsystem 435. The ADI subsystem 435 may be implemented as a kernel-scope component that interfaces with and interacts with the embedded I / O virtualization manager 470 of the network interface device 305. The ADI subsystem 435 may be used to identify various VDEVs and mappings of VDEVs to various isolated domains (e.g., VM 308, container 415, etc.). As previously mentioned, the embedded I / O virtualization manager 470 of the network interface device 305 may be implemented as a VDCM, an embedded application (or other executable code) on the network interface device 305 that is configured to assemble VDEVs and emulate intercepted path operations involving the VDEVs.The resources accessed through a VDEV generally include resources such as control registers, status registers, BAR registers, one or more interrupt message buffers (IMS), and message-signaled interrupts (MSI-X) of the underlying physical (and virtual) function(s), among other examples.

[0052] To assemble a VDEV, the embedded I / O virtualization manager 470 may determine information associated with a particular device function, such as PCI Base Address Register (BAR) ranges of the connected device. In some cases, the function may be a function of a device provided on the network interface device 305 itself (as one of its internal components (e.g., LAN block, accelerator blocks, etc.), and the embedded I / O virtualization manager 470 may read the information (e.g., the BAR ranges and any associated values) directly from the devices (e.g., in one or more registers) using internal connections (e.g., PCIe connections, peer-to-peer PCIe translations, DMA, etc.). In embodiments where the device function is associated with a device other than the network interface device 305 (e.g.,When a device is connected to a processor (SR-IOV device 450), the embedded I / O virtualization manager 470 may access the other device's registers via peer-to-peer inter-device communication (e.g., peer-to-peer PCIe translations, DMA, etc.). In some embodiments, the embedded I / O virtualization manager 470 may include a cloud agent for, among other things, querying and receiving device capability information from external (e.g., cloud-based) sources.

[0053] Once the embedded I / O virtualization manager 470 receives the information for each device function to be used in the desired VDEV, the I / O virtualization manager 470 can assemble the corresponding VDEV using ADIs defined for one or more of the device functions. For example, the VDEV can contain one or more ADIs, where each ADI can contain a mapping between virtualized registers and the BARs of the underlying hardware. After the registers of the respective device function(s) are created in a corresponding ADI entry (and passed, for example, to the ADI OPs 445 of the PCI driver 440), the registers can be used by software as physical MMIO host addresses.The host or VMM can detect when operations involving a VDEV (and an associated isolated domain) represent direct access or intercepted access, and can cause intercepted accesses to be forwarded to the network interface device 305 for handling by the I / O virtualization manager 470.

[0054] As previously mentioned, SR-IOV devices (e.g., 450) can be natively configured to be compatible with SR-IOV hardware-based I / O virtualization. Such devices, whether connected to a network interface device 305 enhanced with an embedded I / O virtualization manager 470 or internally to the network interface device 305 itself, can benefit from the I / O virtualization manager functionality to allow the device's SR-IOV-based virtual functions (e.g., 460a-460n) to be assigned to SIOV-based ADIs and included in SIOV-based VDEVs for use in a SIOV operating environment. This allows legacy SR-IOV devices to be enhanced with SIOV and other functions that can be defined in software and emulated using the I / O virtualization manager 470. Fig. 5A is a simplified block diagram 500a showing an example of an SR-IOV device 450. An SR-IOV device (e.g., 450) may include one or more physical functions (PFs), where a PF (e.g., 505) has one or more (likely multiple) virtual functions (VFs) (e.g., 510, 515, 520) connected to the PF 505. PFs are device functions (e.g., PCIe functions) that support SR-IOV Extended Capability, and VFs may be "lightweight" versions of the same function that can be directly accessed by an isolated domain (e.g., a container or VM). The VF may contain its own resources (e.g., control registers, status registers, BAR registers, one or more interrupt message buffers (IMS), and message-signaled interrupts (MSI-X)) associated with the main data movement of the function, and these resources may be made available to an isolated domain assigned to the VF.A VF can be sequentially shared by different isolated domains (e.g., first assigned to a VM, then reset and assigned to another VM, etc.). A VF connected to a corresponding PF has the same device type as the PF (e.g., the same network device type, the same storage device type, the same encryption device type, etc.). The SR-IOV device 450 shares a common PCIe connection 525 (between the PF and VF devices (e.g., using internal routing circuitry 526)). The link and PCIe functions shared by all functions on device 450 can be managed through physical function 0 (e.g., 505). A PCIe function has a unique set of physical resources, including a separate configuration space and a BAR. However, a VF shares some other resources with its associated PF, such as:sharing a number of common configuration space fields with the PF, using the same configuration mechanisms and header types as a PF, etc. The associated Extended SR-IOV Capability register can be programmed to specify or identify the maximum number of VFs that can be associated with a particular PF on device 450. Each function, PF, and VF can be assigned a unique routing ID. The routing ID for each VF can be defined using the routing ID of the associated PF and the fields in that PF's Extended SR-IOV Capability. The non-shared physical resources of a VF are resources used to provide function-specific services (e.g., resources like worklists, data buffers, etc.) for the assigned isolated domain. In some cases, a single address translation cache (ATC) is used (e.g., 530). The routing ID described in . Fig. While the example shown in Figure 5A shows a single PF with multiple associated VFs, it should be noted that other SR-IOV devices, among other examples, may also include multiple different PFs (of different device types), each with its own set of VFs.

[0055] In the simplified block diagram 500b of Fig. 5B illustrates an example embedded I / O virtualization manager 470, including the definition of an example VDEV 550 supported by I / O virtualization manager 470. An example VDEV 550 includes ADIs 555a and 555b. ADIs 555a-b may be similar to ADIs 206a-206b. However, in this example, ADIs 555a-b may not map to SIOV-based PFs, but rather to one or more resources (e.g., 565, 570) of an SR-IOV device's VFs (e.g., 510, 515) (e.g., all or a subset of the resources of a particular VF) connected to an SR-IOV-based PF 505 (and using virtualizations of its physical resources 560). Each of the ADIs 555a-b can be mapped to the respective physical backend resources (e.g., 565, 570) of the corresponding VFs 510, 515.For example, when creating an ADI, a mapping is created between a virtual register and a physical register (and / or a mapping between a virtual register and an emulated register). The I / O virtualization manager 470 may assemble the VDEV 550 using one or more of the ADIs 555a-b associated with the VFs 510, 515. In some implementations, a network interface device with an embedded I / O virtualization manager 470 may enable VDEV definition and emulation involving VFs of several different SR-IOV devices coupled to or integrated with the network interface device (e.g., to form "hybrid" VDEVs based on these SR-IOV devices), among other examples. Furthermore, the device's VFs may be provided with additional VDEVs through the emulation of the I / O virtualization manager 470 (e.g.,during intercepted operations) features not included in the SR-IOV device hardware, such as advanced address translation, live migration, and enhanced security.

[0056] In Fig. 6 shows an example flowchart 600 illustrating an example general approach for composing virtual devices using SR-IOV-based VFs of one or more physical devices. In general, in the timing diagram 600, points 601-608 may correspond to system initialization steps, 609-616 to ADI creation, and points 615-626 to ADI usage, with points 617-622 corresponding to software intercepted and / or emulated registers. Embodiments are not limited in these contexts.

[0057] As shown, at 601, the cloud orchestrator software executing on a cloud system may launch the system, including the host hardware, a network interface device 305 housing an embedded I / O virtualization manager 470, and one or more devices that may include PFs and corresponding VFs. At 602, the I / O virtualization manager (e.g., a VDCM) may initialize an extended ADI capability structure and may also cause the initialization of a primary physical function (e.g., 650) that implements at least a portion of the I / O virtualization framework (e.g., the VDCM). At 603, the host system may deploy a driver 440 (e.g.,A PCIe driver (e.g., a PCIe driver) may be loaded into the host kernel 425 for the primary PF 650, which enables an interface between an ADI subsystem 435 (corresponding to the virtualization manager 470) running in the host kernel 425 and the I / O virtualization manager 470 running on the network interface device 305. For example, the primary PF 650 may be an accelerator device (e.g., a virtualization accelerator (e.g., based on SIOV)) on the network interface device, and the PCI driver 440 may correspond to this accelerator device. At 604, the driver 440 may read a register (e.g., an extended ADI capability structure) of the network interface device (e.g., connected to the primary PF 650) to identify (at 605) one or more VFs or another subordinate PF, as well as profile information for the I / O virtualization manager. While the primary PF 650 is on the same device 305 (e.g.,an IPU) that hosts the embedded I / O virtualization manager 470, subordinate PFs (e.g., 655) and their virtual functions (implementing subordinate VFs) may be hosted on other, separate devices (e.g., connected to the network interface 305). The driver 440 may, at 606, initialize the resources of one or more VFs (e.g., in conjunction with a subordinate PF 655 on the device 305 or another SR-IOV device), such as control registers, status registers, BAR registers, one or more interrupt message buffers, and one or more message-signaled interrupts. At 607, the driver 440 may initialize the ADI subsystem 435 in the host kernel 425 to enumerate capabilities associated with a particular ADI (e.g.,all or a subset of resources of a particular VF on the network interface device) that can be used by a container or VM in user space of the host system. Enumerated capabilities may include, for example, virtual devices or ADI entries. This may result in one or more ADIs, such as ADIs, being created in ADI subsystem 435. At 608, driver 440 may cause ADI driver 434 (e.g., the VFIO ADI driver and / or the UACCE ADI driver) to initialize the ADI template for the ADI(s) created at 607. In general, steps 605-608 may be performed for each child PF or VF identified at 604.

[0058] At 609, the cloud orchestrator software 605 may instruct the I / O virtualization manager 470 to create an ADI. In this example, the ADI may correspond to an SR-IOV-based VF, and the I / O virtualization manager 470 may cause the VF to be initialized at 610 (e.g., based on an SR-IOV protocol and using the corresponding PF of the device (e.g., a device integrated with the network interface device or another SR-IOV device coupled to the network interface device, etc.). In the SR-IOV context, a subordinate PF (e.g., 655) may (e.g., in response to a request from the I / O virtualization manager 470) create corresponding subordinate VF(s) (e.g., 460) associated with its resources by writing to registers in a PCIe Extended Capability structure corresponding, among other things, to an SR-IOV Extended Capability.The I / O Virtualization Manager 470 can then assemble and activate the ADI corresponding to the VF (at 611). At 612, the primary PF 650 can generate an interrupt, which is passed to the driver 440. At 613, the VF 460 signals the driver 440 that it can recognize the VF and connect to the driver 440. At 614, the driver 440 adds the VF-based ADI to the ADI repository of the I / O Virtualization Manager 470. This creates an entry for the ADI in the repository, containing the register mappings and all other information describing the ADI. At 615, the ADI subsystem 435 issues a request to the ADI driver 434 (e.g., the VFIO ADI driver and / or the UACCE ADI driver) to indicate that a virtual device is available for use. At 616, the corresponding ADI driver 434 may create a user interface using the data in the ADI repository entry associated with the ADI.An example of a user interface is a VFIO interface. However, the application 418 may not be able to use the ADI directly. Instead, the application 418 may access the ADI through the user interface (e.g., the VFIO interface), among other example implementations. In the case of a VF-based ADI, the VF's IOMMU domain may be used instead of the commonly used PASID IOMMU domain, since the IOMMU domain of SR-IOV devices differs from the domain assumed and otherwise defined for SIOV, among other examples.

[0059] At 617, the cloud orchestrator 605 may assign the user-space interface created at 616 to an application such as application 418 and launch the user-space interface. At 618, application 418 may open and use the user-space interface. Application 418 may also set up an MMIO and / or queues. Application 418 may further configure the device implementing the VF. For example, application 418 may set an interrupt vector through VFIO_DEVICE_SET_IRQS. At 619, application 418 may read from and / or write to emulated control status registers (CSRs) and / or BAR registers (e.g., the primary PF 650) (e.g., for an intercepted path transaction). For example, the application 418 may issue a request to read the emulated CSRs and / or BAR registers of the primary PF 650 (e.g.,as emulated by the I / O virtualization manager 470 to conform to the VDEV based on the subordinate VF 460, which includes any software-emulated advanced functionality (e.g., address translation, encryption or other advanced security features, live migration, etc.) facilitated by these emulated registers). At 620, the primary PF 620 may convert the request into one or more translation layer packets (TLPs) in one or more hardware queues of the I / O virtualization manager 470. At 621, the I / O virtualization manager 470 processes the request and returns the result to a hardware response queue of the primary PF 650. For example, the I / O virtualization manager 470 may read the emulated CSRs and / or BAR registers of the primary PF 650 and store the resulting data in the hardware response queue. At 622, the primary PF 650 returns the result of the request to the application 418.For example, the primary PF 650 may return the data read from the emulated CSRs and / or bar registers of the primary PF 650 to the application 418. At 623, the application 418 may access one or more hardware registers of the VF 460 (e.g., in a direct-path transaction). For example, the application may read from and / or write to one or more hardware registers of the device implementing the VF 460.

[0060] Therefore, when application 312 requests access to hardware registers, application 418 can access the hardware registers directly via VF 460 (see 623 in Fig. 6). However, if the application 418 requests access to emulated registers (see 619-622), the application 418 submits the request to the primary PF 650, which uses the I / O virtualization manager 470 to process the request and return a result. Advantageously, the registers of various subordinate functions are arranged so that these hardware registers can be directly accessed by the application 418. However, since multiple subordinate functions (including multiple VFs) may be supported by multiple different devices, the I / O virtualization manager 470 can distinguish them based on their respective BAR addresses. The cloud orchestrator can create a virtual device map between the VDEV's emulated registers and the physical registers of the associated device.In this way, the I / O virtualization manager 470 learns which virtual (or emulated) registers are mapped to which physical registers. When an ADI is created, a mapping is created between a virtual register and a physical register (and / or a mapping between a virtual register and an emulated register).

[0061] Fig. 7 is a simplified flowchart illustrating an example technique for using an example network interface device to support I / O virtualization of one or more resources of a device that supports hardware-based I / O virtualization (e.g., SR-IOV) using software emulation of the device resources (e.g., based on SIOV). For example, a network interface device may be equipped with an I / O virtualization manager, which may be implemented as software or firmware code executable by a processor of the network interface device to perform software-based emulation of a virtual I / O device within a system. For example, a set of virtual functions associated with a physical function may be identified in a device coupled to the network interface (e.g.,as a component of the network interface or coupled to a port of the network interface). The device may be a device compatible with an SR-IOV-based protocol. A virtual I / O device (e.g., a SIOV VDEV) may be defined 710 or created to contain one or more interfaces or references (e.g., ADIs) to resources of one or more physical devices. For example, some of the interfaces may be associated with a set or subset of resources of one of the virtual functions. The virtual function may be made available for assignment to various system images, e.g., applications running in containers or virtual machines of a host system coupled to the network interface device. Operations between a virtual device and the system image may include at least two types of operations: direct path operations and intercepted path operations.In direct path operations, the system image can directly access the device's registers to access virtual function resources. In intercepted path operations, the network interface device's I / O virtualization manager can be forwarded to an operation and access 720 virtual function resources (based on the mapping of the interface within the virtual device to the virtual function resources) and emulate the operation of the virtual I / O device based on the accessed resources, for example, by generating a response within the operation / transaction. In some cases, the I / O virtualization manager can enhance the underlying device's capabilities through this emulation, for example, by incorporating a feature not natively present or supported by the underlying device, such as an enhanced address translation scheme (e.g., ATE 2.0).0), live migration, improved security, and other examples.

[0062] Note that the devices, methods, and systems described above can be implemented in any electronic device or system as mentioned above. For illustration purposes, Fig. 8 shows an example implementation of a processing device such as may be included in a network interface device. It should be noted that other processor architectures may also be available to implement the functionality and request processing by an example network interface device, including implementing the components and functions of the example network interface device described above. While the above examples focus on enhancements to an Ethernet subsystem and connections compatible with an Ethernet-based protocol, it is important to note that the principles discussed herein are protocol-independent and can be applied to connections based on a variety of other technologies, such as PCIe, CXL, UCIe, CCIX, Infinity Fabric, and other examples.

[0063] In Fig. Figure 8 illustrates a block diagram 800 of an exemplary computing device (e.g., a central processing unit (CPU)) 812 coupled to various other components of a platform, according to certain embodiments. Although the figure of CPU 812 illustrates a particular configuration, the cores and other components of CPU 812 may be arranged in any suitable manner. CPU 812 may include any processor or processing device, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, an application processor, a coprocessor, a system on a chip (SOC), or other device for executing code. CPU 812 in the illustrated embodiment includes four processing elements (cores 802 in the illustrated embodiment), which may include asymmetric processing elements or symmetric processing elements.However, the CPU 812 may include any number of processing elements, which may be symmetric or asymmetric.

[0064] In one embodiment, a processing element refers to hardware or logic for supporting a software thread. Examples of hardware processing elements include: a thread unit, a thread slot, a thread, a process unit, a context, a context unit, a logical processor, a hardware thread, a core, and / or any other element capable of maintaining state for a processor, such as execution state or architectural state. In other words, in one embodiment, a processing element refers to any hardware capable of independently interacting with code, such as a software thread, an operating system, an application, or other code. A physical processor (or processor socket) typically refers to an integrated circuit that can potentially support any number of other processing elements, such as:cores or hardware threads.

[0065] A core may refer to logic located on an integrated circuit capable of maintaining an independent architectural state, where each independently maintained architectural state is associated with at least some dedicated execution resources. A hardware thread may refer to any logic located on an integrated circuit capable of maintaining an independent architectural state, where the independently maintained architectural states share access to execution resources. As can be seen, the line between the nomenclature of a hardware thread and a core overlaps when certain resources are shared and others are dedicated to an architectural state.However, a core and a hardware thread are often treated by an operating system as individual logical processors, with the operating system being able to schedule operations on each logical processor individually.

[0066] The physical CPU 812 includes, as shown in Fig.8, four cores—cores 802A, 802B, 802C, and 802D—are illustrated, although a CPU may include any number of cores. Here, cores 802 may be considered symmetric cores. In other embodiments, cores may include one or more unordered processor cores or one or more ordered processor cores. However, cores 802 may be individually selected from any core type, such as a native core, a software-managed core, a core configured to execute a native instruction set architecture (ISA), a core configured to execute a translated ISA, a co-designed core, or any other known core. In a heterogeneous core environment (e.g., asymmetric cores), some form of translation, such as binary translation, may be utilized to schedule or execute code on one or both cores.

[0067] A core 802 may include a decode module coupled to a fetch unit for decoding fetched elements. In one embodiment, the fetch logic includes individual sequencers, each connected to thread slots of the cores 802. Typically, the core 802 is connected to a first ISA that defines / specifies instructions executable on the core 802. Often, machine code instructions that are part of the first ISA include a portion of the instruction (referred to as an opcode) that references / specifies an instruction or operation to be executed. The decode module may include circuitry that recognizes these instructions by their opcodes and forwards the decoded instructions down the pipeline for processing as defined by the first ISA. For example, in one embodiment, decoders may include logic designed or adapted to recognize certain instructions, such as transaction instructions.As a result of the decoders' detection, the architecture of core 802 performs specific, predefined actions to execute the tasks associated with the corresponding instruction. It is important to note that each of the tasks, blocks, operations, and methods described herein may be performed in response to a single or multiple instructions; some of which may be new or old instructions. In one embodiment, the decoders of cores 802 detect the same ISA (or a subset thereof). Alternatively, in a heterogeneous core environment, a decoder of one or more cores (e.g., core 802B) may detect a second ISA (either a subset of the first ISA or a different ISA).

[0068] In various embodiments, cores 802 may also include one or more arithmetic logic units (ALUs), floating point units (FPUs), caches, instruction pipelines, interrupt handling hardware, registers, or other suitable hardware to facilitate operation of cores 802.

[0069] Bus 808 may represent any connection coupled to CPU 812. In one example, bus 808 may connect CPU 812 to another CPU of the platform logic (e.g., via UPI). I / O blocks 804 represent interface logic to couple I / O devices 810 and 815 to cores of CPU 812. In various embodiments, an I / O block 804 may include an I / O controller integrated into the same package as cores 802, or it may simply include interface logic for coupling to an I / O controller located off-chip. For example, I / O blocks 804 may include PCIe interface logic. Similarly, memory controller 806 represents interface logic to connect memory 814 to the cores of CPU 812. In various embodiments, the memory controller 806 is integrated on the same package as the cores 802.In alternative embodiments, a memory controller could be located off-chip.

[0070] As various examples, in the illustrated embodiment, core 802A may have relatively high bandwidth and lower latency to devices coupled to bus 808 (e.g., other CPUs 812) and NICs 810, but relatively low bandwidth and higher latency to memory 814 or core 802D. Core 802B may have relatively high bandwidths and low latencies to both NICs 810 and PCIe solid-state drives (SSDs) 815, and moderate bandwidths and latencies to devices connected to bus 808 and core 802D. Core 802C would have relatively high bandwidths and low latencies to memory 814 and core 802D. Finally, core 802D would have relatively high bandwidth and low latency to core 802C, but relatively low bandwidth and high latency to NICs 810, core 802A, and devices coupled to bus 808.

[0071] "Logic" (e.g., in I / O controllers, power managers, latency managers, etc., and other references to logic in this application) may refer to hardware, firmware, software, and / or combinations thereof for performing one or more functions. In various embodiments, the logic may include a microprocessor or other processing element for executing software instructions, discrete logic such as an application-specific integrated circuit (ASIC), a programmed logic device such as a field-programmable gate array (FPGA), a memory device containing instructions, combinations of logic devices (e.g., as found on a printed circuit board), or other suitable hardware and / or software. The logic may include one or more gates or other circuit components. In some embodiments, the logic may also be embodied entirely as software.

[0072] A design can go through various stages, from generation to simulation to manufacturing. Data representing a design can represent the design in several ways. First, which is useful in simulations, the hardware can be represented using a hardware description language (HDL) or another functional description language. Additionally, at some stages of the design process, a circuit-level model containing logic and / or transistor gates can be produced. Furthermore, most designs eventually reach a stage of data representing the physical placement of various devices in the hardware model.In the case where conventional semiconductor manufacturing techniques are used, the data representing the hardware model may be the data indicating the presence or absence of various features at different mask layers for masks used to create the integrated circuit. In some implementations, such data may be stored in a database file format such as Graphic Data System II (GDS II), Open Artwork System Interchange Standard (OASIS), or a similar format.

[0073] In some implementations, software-based hardware models, as well as HDL and other functional description language objects, may include, among other things, register transfer language (RTL) files. Such objects may be machine-parsed, allowing a design tool to accept the HDL object (or model), analyze the HDL object for attributes of the described hardware, and determine a physical circuit and / or on-chip layout from the object. The results of the design tool may be used to manufacture the physical device. For example, a design tool may determine configurations of various hardware and / or firmware elements from the HDL object, such as bus widths, registers (including sizes and types), memory blocks, physical interconnect paths, fabric topologies, and other attributes implemented to realize the system modeled in the HDL object.Design tools can include tools for determining the topology and fabric configurations of System on Chip (SoC) and other hardware devices. In some cases, the HDL object can be used as a basis for developing models and design files that can be used by manufacturing facilities to produce the described hardware. Indeed, an HDL object itself can be provided as input to the manufacturing system's software to generate the described hardware.

[0074] In any representation of the design, the data may be stored in any form of machine-readable medium. A storage device or a magnetic or optical storage medium, such as a disc, may be the machine-readable medium for storing information transmitted via optical or electrical waves that are modulated or otherwise generated to transmit such information. When an electrical carrier wave indicating or carrying the code or design is transmitted, to the extent that copying, buffering, or retransmission of the electrical signal is performed, a new copy is made. Therefore, a communications provider or network operator may store on a tangible, machine-readable medium, at least temporarily, an item, such as information encoded in a carrier wave, embodying techniques of embodiments of the present disclosure.

[0075] A module, as used herein, refers to any combination of hardware, software, and / or firmware. As one example, a module includes hardware, such as a microcontroller, coupled to a non-volatile storage medium to store code adapted to be executed by the microcontroller. Therefore, in one embodiment, reference to a module refers to the hardware specifically configured to recognize and / or execute the code to be held on a non-volatile storage medium. Furthermore, in another embodiment, use of a module refers to the non-volatile storage medium containing code specifically adapted to be executed by the microcontroller to perform predetermined operations.And as can be seen, in yet another embodiment, the term module (in this example) may refer to the combination of the microcontroller and the non-volatile storage device. Often, module boundaries depicted as separate may generally vary and potentially overlap. For example, a first and a second module may share hardware, software, firmware, or a combination thereof, while potentially maintaining independent portions of hardware, software, or firmware. In one embodiment, the use of the term logic includes hardware such as transistors, registers, or other hardware such as programmable logic devices.

[0076] The use of the phrase "to" or "configured to," in one embodiment, refers to arranging, assembling, manufacturing, offering for sale, importing, and / or designing a device, hardware, logic, or element to perform a specified or particular task. In this example, a device or element thereof that is non-functional is nevertheless "configured to" perform a specified task if it is designed, coupled, and / or connected to perform the specified task. As a purely illustrative example, a logic gate may provide a 0 or a 1 during operation. However, a logic gate "configured to" provide an enable signal for a clock does not include every potential logic gate that can provide a 1 or 0.Instead, the logic gate is one that is coupled in such a way that the output of a 1 or 0 during operation serves to activate the clock. It should be noted again that the use of the term "designed to" does not require an operation, but instead focuses on the latent state of a device, hardware, and / or element, where in the latent state, the device, hardware, and / or element is designed to perform a specific task when the device, hardware, and / or element is operating.

[0077] Furthermore, the use of the terms "capable of" and / or "operable to" in one embodiment refers to a device, logic, hardware, and / or element that is designed in a manner to enable use of the device, logic, hardware, and / or element in a particular manner. Note, as above, that the use of "to," "capable of," or "operable to" in one embodiment refers to the latent state of a device, logic, hardware, and / or element where the device, logic, hardware, and / or element is not in operation, but is designed in a manner to enable use of a device in a particular manner.

[0078] A value, as used herein, includes any known representation of a number, a state, a logical state, or a binary logical state. Often, the use of logic levels, logic values, or logical values ​​is also referred to as ones and zeros, which simply represent binary logical states. For example, a 1 refers to a high logic level, and a 0 refers to a low logic level. In one embodiment, a memory cell, such as a transistor or flash cell, may be capable of holding a single logical value or multiple logical values. However, other representations of values ​​have been used in computer systems. For example, the decimal number ten can be represented as the binary value 418A0 and the hexadecimal letter A. Therefore, a value includes any representation of information that can be stored in a computer system.

[0079] Furthermore, states can be represented by values ​​and portions of values. As an example, a first value, such as a logical one, can represent a default or initial state, while a second value, such as a logical zero, can represent a non-default state. Furthermore, in one embodiment, the terms reset and set refer, respectively, to a default and an updated value or state. For example, a default value potentially includes a high logical value, i.e., reset, while an updated value potentially includes a low logical value, i.e., set. Note that any combination of values ​​can be used to represent any number of states.

[0080] The above-described embodiments of methods, hardware, software, firmware, or code may be implemented via instructions or code stored on a machine-accessible, machine-readable, computer-accessible, or computer-readable medium and executable by a processing element. A non-transitory machine-accessible / readable medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form readable by a machine, such as a computer or electronic system. For example, a non-transitory machine-accessible medium includes random access memory (RAM), such asstatic RAM (SRAM) or dynamic RAM (DRAM), ROM, a magnetic or optical storage medium, flash memory devices, electrical storage devices, optical storage devices, acoustic storage devices, other forms of storage devices for holding information received from transitory (propagated) signals (e.g., carrier waves, infrared signals, digital signals), etc., which are to be distinguished from the non-volatile media that can receive information therefrom.

[0081] Instructions used to program logic to perform embodiments of the disclosure may be stored within a memory in the system, such as DRAM, cache, flash memory, or other storage element. Furthermore, the instructions may be distributed over a network or by means of other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a format supported by a machine (e.g., aa computer) readable form, but is not limited to floppy disks, optical discs, compact discs, read-only memories (CD-ROMs) and magneto-optical discs, read-only memories (ROMs), random access memories (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or any tangible, machine-readable storage used in the transmission of information over the Internet by electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Accordingly, computer-readable medium includes any type of tangible, machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0082] The following examples relate to embodiments consistent with this specification. Example 1 is an apparatus including: a network interface device including: a processor; a first port for connecting to a network; a second port for connecting to a host system; circuitry for implementing a physical function and supporting a plurality of virtual functions associated with the physical function; and an input / output (I / O) virtualization manager executable by the processor to cause: identifying a virtual device defined as containing resources of a particular one of the plurality of virtual functions; identifying an operation to be performed between the virtual device and a system image hosted by the host system; and emulating the virtual device in the operation.

[0083] Example 2 includes the subject matter of Example 1, wherein the physical function and the plurality of virtual functions are based on Peripheral Component Interconnect Express (PCIe) Single Root I / O Virtualization (SR-IOV).

[0084] Example 3 includes the subject matter of Example 2, wherein the virtual device emulation is based on Scalable I / O Virtualization (SIOV).

[0085] Example 4 includes the subject matter of Example 3, wherein the I / O virtualization manager includes a virtual device composition manager (VCDM).

[0086] Example 5 includes the subject matter of any of Examples 2 to 4, wherein the network interface device further comprises a third port for coupling with another device, the other device supporting SR-IOV and supporting a second physical function and a set of virtual functions associated with the second physical function, wherein the I / O virtualization manager is to define one or more virtual devices based on one or more of the set of virtual functions of the other device.

[0087] Example 6 includes the subject matter of Example 5, wherein the virtual device is defined to map to both the resources of the particular virtual function of the network interface device and the resources of one of the set of virtual functions of the other device.

[0088] Example 7 includes the subject matter of any of Examples 1-6, wherein the resources are mapped to a first assignable device interface (ADI), and the virtual device is defined to include a set of ADIs including the first ADI.

[0089] Example 8 includes the subject matter of any of Examples 1-7, wherein the resources include at least one of the following: instruction / status registers, in-device queues, in-memory queue references, and local storage.

[0090] Example 9 includes the subject matter of any of Examples 1-8, wherein the system image is running in an isolated domain, and the isolated domain contains either a virtual machine or a container.

[0091] Example 10 includes the subject matter of any of Examples 1-9, wherein the network interface device includes local storage.

[0092] Example 11 includes the subject matter of any of Examples 1-10, wherein, for direct path operations, the system image directly accesses the resources of the particular virtual function, and for intercepted path operations, the system accesses the resources of the particular virtual function via emulation by the I / O virtualization manager.

[0093] Example 12 includes the subject matter of any of Examples 1-11, wherein emulating the virtual device includes emulating a feature that is not native to the plurality of virtual functions or the physical functions in operation.

[0094] Example 13 includes the subject matter of Example 12, wherein the feature includes support for live migration.

[0095] Example 14 includes the subject matter of Example 12, wherein the feature includes a particular virtual address to physical address translation service.

[0096] Example 15 is a non-transitory computer-readable storage medium having instructions stored thereon, the instructions executable by a processor to cause the processor to: identify, in a host system, a virtual I / O device defined by an I / O virtualization manager executing in a network interface device, wherein the host system is coupled to the network interface device, and the virtual I / O device is to provide access to resources of a physical device connected to or contained within the network interface device; identify the association of the virtual I / O device with a system image in an isolated domain hosted by the host system;Determining that a first operation between the system image and the virtual I / O device comprises a direct path operation, wherein the system image directly accesses the resources in the direct path operation; determining that a second operation between the system image and the virtual I / O device comprises an intercepted path operation; and redirecting intercepted path operations for handling by the I / O virtualization manager, wherein the I / O virtualization manager is to emulate the operation by the virtual I / O device in software in intercepted path operations, wherein the resources include resources of a virtual function associated with a physical function of the physical device.

[0097] Example 16 includes the subject matter of Example 15, wherein the virtual function is based on a Single Root I / O Virtual (SR-IOV) protocol and the virtual I / O device emulation is based on a Scalable I / O Virtualization (SIOV) protocol.

[0098] Example 17 includes the subject matter of any of Examples 15-16, wherein the network interface device comprises the apparatus of any of Examples 1-14.

[0099] Example 18 is a method comprising: defining a virtual device on a network interface device to include resources of a particular one of a plurality of virtual functions, the plurality of virtual functions associated with a physical function of a device coupled to the network interface device; identifying an operation to be performed between the virtual device and a system image hosted by a host system coupled to the network interface device; and emulating the virtual device on the network interface device in the operation.

[0100] Example 19 includes the subject matter of Example 18, wherein the network interface device comprises the apparatus of any of Examples 1-14.

[0101] Example 20 is a system having means for performing the method of any one of Examples 18-19.

[0102] Example 21 includes the subject matter of Example 20, wherein the means comprises a non-transitory storage medium having instructions stored thereon, the instructions executable by a processor to perform the method of any of Examples 18-19.

[0103] Example 22 is a system comprising: a host system for hosting a system image in an isolated domain; a first device, the first device comprising a physical function and a set of virtual functions corresponding to the physical function, the set of virtual functions comprising corresponding sets of resources; and a network interface device comprising: a processor; memory; a first port for connecting to a network; a second port for connecting to the host system; a third port for connecting to the first device;and an I / O virtualization manager executable by the processor to cause: generating a virtual device definition, the virtual device defined to include a set of assignable device interfaces, one of the set of assignable device interfaces being mapped to at least a subset of the set of resources of one of the set of virtual functions of the first device; identifying an operation to be performed between the virtual device and the system image hosted by the host system; and emulating the virtual device in the operation, wherein the I / O virtualization manager accesses the subset of the set of resources of the virtual function to emulate the virtual device in the operation.

[0104] Example 23 includes the subject matter of Example 22, wherein the isolated domain comprises either a virtual machine or a container.

[0105] Example 24 includes the subject matter of Example 22, wherein the network interface device comprises either an infrastructure processing unit (IPU) or an intelligent network interface controller (NIC).

[0106] Example 25 includes the subject matter of Example 22, wherein the first device comprises an SR-IOV compatible device and the first device does not natively support SIOV.

[0107] When reference is made to "a single embodiment" or "an embodiment" in this specification, it means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of the phrases "in a single embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Moreover, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0108] In the foregoing specification, a detailed description has been given with respect to specific exemplary embodiments. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure as set forth in the appended claims. The specifications and drawings are, accordingly, to be considered in an illustrative rather than a restrictive sense. Moreover, the foregoing use of embodiment and other exemplary language does not necessarily refer to the same embodiment or example, but may refer to different and separate embodiments, as well as potentially the same embodiment. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] CN 2023 / 118655

[0001]

Claims

[1] Device comprising: a network interface device comprising: a processor; a first port for coupling to a network; a second port for coupling with a host system; a circuit for implementing a physical function and for supporting a plurality of virtual functions associated with the physical function; and an input / output (I / O) virtualization manager that can be executed by the processor to do the following: identifying a virtual device defined to include resources of a particular one of the plurality of virtual functions; Identifying an operation to be performed between the virtual device and a system image hosted by the host system; and Emulate the virtual device in the operation. [2] The apparatus of claim 1, wherein the physical function and the plurality of virtual functions are based on Peripheral Component Interconnect Express (PCIe) Single Root I / O Virtualization (SR-IOV). [3] The apparatus of claim 2, wherein the emulation of the virtual device is based on Scalable I / O Virtualization (SIOV). [4] The apparatus of claim 3, wherein the I / O virtualization manager comprises a virtual device composition manager (VCDM). [5] The apparatus of any one of claims 2 to 4, wherein the network interface device further comprises a third port for coupling to another device, the other device supporting SR-IOV and supporting a second physical function and a set of virtual functions associated with the second physical function, wherein the I / O virtualization manager is to define one or more virtual devices based on one or more of the set of virtual functions of the other device. [6] The apparatus of claim 5, wherein the virtual device is defined to map both the resources of the particular virtual function of the network interface device and the resources of one of the set of virtual functions of the other device. [7] The apparatus of any of claims 1 to 6, wherein the resources are mapped to a first assignable device interface (ADI) and the virtual device is defined to include a set of ADIs including the first ADI. [8] The apparatus of any of claims 1-7, wherein the resources comprise at least one of the following: instruction / status registers, in-device queues, in-memory queue references, and local memory. [9] The apparatus of any of claims 1-8, wherein the system image is executed in an isolated domain, and the isolated domain comprises either a virtual machine or a container. [10] The apparatus of any one of claims 1 to 9, wherein the network interface device comprises a local memory. [11] The apparatus of any of claims 1-10, wherein the system image directly accesses the resources of the particular virtual function for direct path operations, and the system accesses the resources of the particular virtual function via emulation by the I / O virtualization manager for intercepted path operations. [12] The apparatus of any one of claims 1 to 11, wherein emulating the virtual device comprises emulating a feature that is not native to the plurality of virtual functions or the physical functions in operation. [13] The apparatus of claim 12, wherein the feature includes support for live migration. [14] The apparatus of claim 12, wherein the feature comprises a particular virtual address to physical address translation service. [15] Procedure comprising: Identifying a virtual I / O device in a host system defined by an I / O virtualization manager executing in a network interface device, wherein the host system is coupled to the network interface device and the virtual I / O device is to provide access to resources of a physical device connected to or contained within the network interface device; Identifying the virtual I / O device's mapping to a system image in an isolated domain hosted by the host system; Determining that a first operation between the system image and the virtual I / O device comprises a direct path operation, wherein the system image directly accesses the resources in the direct path operation; Determining that a second operation between the system image and the virtual I / O device includes an intercept path operation; and Redirecting intercept path operations for handling by the I / O virtualization manager, wherein the I / O virtualization manager is to emulate the operation by the virtual I / O device in software in intercept path operations, wherein the resources comprise resources of a virtual function associated with a physical function of the physical device. [16] The method of claim 15, wherein the virtual function is based on a Single Root I / O Virtual (SR-IOV)-based protocol and the emulation of the virtual I / O device is based on a Scalable I / O Virtualization (SIOV)-based protocol. [17] The method of any of claims 15-16, wherein the resources are mapped to a first assignable device interface (ADI) and the virtual device is defined to include a set of ADIs including the first ADI. [18] The method of any of claims 15-17, wherein the resources comprise at least one of the following: instruction / status registers, in-device queues, in-memory queue references, and local memory. [19] The method of any of claims 15-18, wherein the system image accesses the resources of the particular virtual function directly for direct path operations and the system accesses the resources of the particular virtual function via emulation by the I / O virtualization manager for intercepted path operations. [20] The method of any one of claims 15 to 19, wherein emulating the virtual device comprises emulating a feature that is not native to the plurality of virtual functions or the physical functions in operation. [21] System comprising means for carrying out the method according to any one of claims 15-20.

Images